D4science

Apr 13, 2016

08:03 PM D4Science Infrastructure Task #3262 (Closed): Change the ldap authentication for the d4science subversion repository

From the _old_ ldap server to the new. Possible problems that need to be managed:
* Different usernames
* Users t... Andrea Dell'Amico

08:02 PM D4Science Infrastructure Task #3261 (Closed): Manage more ldap groups from the portal so that we can setup different roles on various services

Needed right now:
* An admin group for redmine
* The 'infrastructure managers' that is now a manually managed gro... Andrea Dell'Amico

07:55 PM D4Science Infrastructure Task #3257: Create a letsencrypt callback for each service that will serve letsencrypt certificates

Inside the ansible playbooks, the following variables need to be defined or set to `True`:
~~~
letsencrypt_instal... Andrea Dell'Amico

07:49 PM D4Science Infrastructure Task #3257: Create a letsencrypt callback for each service that will serve letsencrypt certificates

An example (it's the `haproxy` one):
~~~
#!/bin/bash
LE_SERVICES_SCRIPT_DIR=/usr/local/lib/letsencrypt
LE_CER... Andrea Dell'Amico

07:48 PM D4Science Infrastructure Task #3257 (In Progress): Create a letsencrypt callback for each service that will serve letsencrypt certificates

The scripts need to be installed in the /usr/local/lib/letsencrypt directory and marked as executable. Andrea Dell'Amico

07:41 PM D4Science Infrastructure Task #3257 (Closed): Create a letsencrypt callback for each service that will serve letsencrypt certificates

The letsencrypt scripts take care of the certificates requests and renewal. After a certificate has been renewed, the... Andrea Dell'Amico

07:46 PM D4Science Infrastructure Task #3260 (Closed): letsencrypt script for nginx

nginx needs to be reloaded after a certificate renewal.
Andrea Dell'Amico

07:45 PM D4Science Infrastructure Task #3259 (Closed): letsencrypt script for apache

apache needs to be reloaded after a certificate renewal. Andrea Dell'Amico

07:45 PM D4Science Infrastructure Task #3258 (Closed): letsencrypt script for haproxy

The script has been committed to the ansible haproxy role.
Andrea Dell'Amico

07:44 PM D4Science Infrastructure Task #3258 (Closed): letsencrypt script for haproxy

The script needs to create the `/etc/pki/certs/haproxy.pem` file from the fullchain.pem and key.pem, and set the corr... Andrea Dell'Amico

05:10 PM D4Science Infrastructure Support #3252 (Feedback): Please add me to the svn for the d4science

Andrea Dell'Amico

05:08 PM D4Science Infrastructure Support #3252: Please add me to the svn for the d4science

Please follow the email instructions to activate your account. Ignore the certificate security warnings.
Andrea Dell'Amico

05:03 PM D4Science Infrastructure Support #3252 (In Progress): Please add me to the svn for the d4science

Andrea Dell'Amico

04:17 PM D4Science Infrastructure Task #3164 (Feedback): Investigate the procedures needed to obtain valid X.509 TLS certificates from letsencrypt

Andrea Dell'Amico

04:16 PM D4Science Infrastructure Task #3164: Investigate the procedures needed to obtain valid X.509 TLS certificates from letsencrypt

We now have a script to request/renew the certificate, and a cron job that starts the renewal procedure if the certif... Andrea Dell'Amico

03:31 PM D4Science Infrastructure Task #3243 (Closed): Remove monitoring checks from openBio VMs

Andrea Dell'Amico

12:26 PM D4Science Infrastructure Task #3243 (In Progress): Remove monitoring checks from openBio VMs

Nagios checks removed. The ganglia ones, if any, expire by themselves.
Andrea Dell'Amico

12:54 PM D4Science Infrastructure Task #3109 (Feedback): VREModeler: nightly restart

The cron job will restart the service at 2 am on Saturdays night. Andrea Dell'Amico

12:51 PM D4Science Infrastructure Task #3109 (In Progress): VREModeler: nightly restart

Andrea Dell'Amico

Apr 11, 2016

06:15 PM D4Science Infrastructure Task #3181: Update RStudio on rstudio.d4science.org

GP is asking to complete the upgrade not later than Friday 15th.
Andrea Dell'Amico

06:11 PM D4Science Infrastructure Task #3181: Update RStudio on rstudio.d4science.org

Gianpaolo does not need an updated rstudio.d4science.org for his teaching. Let's proceed this way
* update R and r... Andrea Dell'Amico

Apr 08, 2016

07:27 PM D4Science Infrastructure Task #2227 (In Progress): test replication tools for postgres

The replication scripts need more testing. And we need some automated tools to remove the WAL archives regularly beca... Andrea Dell'Amico

06:14 PM D4Science Infrastructure Task #3104 (Feedback): Please install a reverse proxy over dev2.d4science.org

The haproxy configuration is complete, and the SSL certificates installed.
I used the letsencrypt tools for the SS... Andrea Dell'Amico

06:13 PM D4Science Infrastructure Task #3164 (In Progress): Investigate the procedures needed to obtain valid X.509 TLS certificates from letsencrypt

We have a preliminary playbook that installs the tools and requests a certificate (with the machine hostname, if some... Andrea Dell'Amico

06:12 PM D4Science Infrastructure Task #3164 (Closed): Investigate the procedures needed to obtain valid X.509 TLS certificates from letsencrypt

Letsencrypt, https://letsencrypt.org is an initiative that has the goal to guarantee free (and valid, where _valid_... Andrea Dell'Amico

Apr 07, 2016

04:34 PM D4Science Infrastructure Task #3104: Please install a reverse proxy over dev2.d4science.org

The VM has been reconfigured and ready to answer on port 8080.
I'm starting the work on the haproxy configuration an... Andrea Dell'Amico

03:45 PM D4Science Infrastructure Task #3104: Please install a reverse proxy over dev2.d4science.org

A complete reinstall is ongoing for `dev2.d4science.org`. It will be configured in the same way of infra-gateway and ... Andrea Dell'Amico

03:46 PM D4Science Infrastructure Task #3157 (Closed): Improve the nagios check for the Smartgears (not the smart executor ones) nodes

Andrea Dell'Amico

03:40 PM D4Science Infrastructure Task #3157: Improve the nagios check for the Smartgears (not the smart executor ones) nodes

OK.
Andrea Dell'Amico

02:58 PM D4Science Infrastructure Task #3157: Improve the nagios check for the Smartgears (not the smart executor ones) nodes

Could it be a problem caused by the whn-manager version? I see that `node31.p.d4science.research-infrastructures.eu` ... Andrea Dell'Amico

02:51 PM D4Science Infrastructure Task #3157: Improve the nagios check for the Smartgears (not the smart executor ones) nodes

Roberto Cirillo wrote:
> Andrea Dell'Amico wrote:
> > The check is now active. The following hosts seems not to be ... Andrea Dell'Amico

01:35 PM D4Science Infrastructure Task #3157 (In Progress): Improve the nagios check for the Smartgears (not the smart executor ones) nodes

The check is now active. The following hosts seems not to be running a smartgear service:
~~~
dewn03.madgik.di.uo... Andrea Dell'Amico

01:24 PM D4Science Infrastructure Task #3157 (Closed): Improve the nagios check for the Smartgears (not the smart executor ones) nodes

The check needs to get the `/whn-manager/gcube/resource/` URL and search for the `The resource is active` string
Andrea Dell'Amico

03:42 PM D4Science Infrastructure Task #850 (Closed): Investigate a new way for check smartgears container by nagios

Never mind. The war installation name was the problem, see #3159 Andrea Dell'Amico

03:02 PM D4Science Infrastructure Task #850 (In Progress): Investigate a new way for check smartgears container by nagios

@lucio.lelii@isti.cnr.it Can you thell when the /whn-manager/gcube/resource/ URL was added to the whn-manager? There ... Andrea Dell'Amico

12:37 PM D4Science Infrastructure Task #850 (Closed): Investigate a new way for check smartgears container by nagios

Andrea Dell'Amico

03:41 PM D4Science Infrastructure Task #3140 (Closed): Improve nagios checks for gCore container

The new check is active for all the gCore nodes.
Andrea Dell'Amico

02:52 PM D4Science Infrastructure Task #3140: Improve nagios checks for gCore container

Roberto Cirillo wrote:
> Andrea Dell'Amico wrote:
> > OK, there is one node that's failing the gCore check:
> >
... Andrea Dell'Amico

01:21 PM D4Science Infrastructure Task #3140 (Feedback): Improve nagios checks for gCore container

OK, there is one node that's failing the gCore check:
~~~
dewn10.madgik.di.uoa.gr
~~~ Andrea Dell'Amico

12:48 PM D4Science Infrastructure Task #3140 (In Progress): Improve nagios checks for gCore container

My fault, I used the check that should be used to test the gcore nodes. Andrea Dell'Amico

12:44 PM D4Science Infrastructure Task #3140: Improve nagios checks for gCore container

Some more:
~~~
node51.p.d4science.research-infrastructures.eu
dewn08.madgik.di.uoa.gr
dl17.di.uoa.gr
~~~ Andrea Dell'Amico

12:39 PM D4Science Infrastructure Task #3140 (Feedback): Improve nagios checks for gCore container

It seems that the following nodes aren't smartgears nodes:
~~~
dewn03.madgik.di.uoa.gr
dewn09.madgik.di.uoa.gr
... Andrea Dell'Amico

12:38 PM D4Science Infrastructure Task #3140 (In Progress): Improve nagios checks for gCore container

I switched to the new check all the nodes described as _SMART_ instances on the attachment.
Andrea Dell'Amico

Apr 05, 2016

01:38 PM D4Science Infrastructure Task #3133 (Closed): Install monitoring checks on access.d4science.org

nagios is also active.
Andrea Dell'Amico

12:48 PM D4Science Infrastructure Task #3133 (In Progress): Install monitoring checks on access.d4science.org

Roberto Cirillo wrote:
> I think we could progressively migrate all checks "port based" to checks "wsdl based" , fo... Andrea Dell'Amico

11:55 AM D4Science Infrastructure Task #3133: Install monitoring checks on access.d4science.org

If the WSDL lives at a fixed position, the nagios check could try to fetch it.
A call to the IS to check a service t... Andrea Dell'Amico

11:34 AM D4Science Infrastructure Task #3133: Install monitoring checks on access.d4science.org

Munin will be dismissed in a not so distant future, I hope. I'll add the node to the gcore ganglia cluster.
About ... Andrea Dell'Amico

12:30 AM D4Science Infrastructure Task #3131 (Feedback): Create a suitable R environment on statistical-pre.d4science.org

The R installation is complete. Andrea Dell'Amico

Apr 04, 2016

07:41 PM D4Science Infrastructure Task #3131: Create a suitable R environment on statistical-pre.d4science.org

It failed because an old version of R, with locally compiled additional packages was installed.
I'm running the pl... Andrea Dell'Amico

07:04 PM D4Science Infrastructure Task #3131 (In Progress): Create a suitable R environment on statistical-pre.d4science.org

The installation is on going.
Andrea Dell'Amico

05:43 PM D4Science Infrastructure Task #3131: Create a suitable R environment on statistical-pre.d4science.org

Define *suitable*. Does it mean that it needs to be the same of the dev/production statistical manager nodes?
Andrea Dell'Amico

06:58 PM D4Science Infrastructure Task #3132 (Feedback): Increase RAM on access.d4science.org VM

Let me know when it can be restarted. Andrea Dell'Amico

06:54 PM D4Science Infrastructure Task #3132 (In Progress): Increase RAM on access.d4science.org VM

Andrea Dell'Amico

Apr 01, 2016

04:06 PM D4Science Infrastructure Task #536: Thredds: tomcat runs as root

Any other progress?
Andrea Dell'Amico

Mar 31, 2016

01:45 PM D4Science Infrastructure Task #3105 (Closed): The dev portals need to be able to send email

Andrea Dell'Amico

01:45 PM D4Science Infrastructure Task #3105 (In Progress): The dev portals need to be able to send email

Reconfigured the local postfix instances to use smtp-relay.research-infrastructures.eu, with authentication.
Andrea Dell'Amico

01:44 PM D4Science Infrastructure Task #3105 (Closed): The dev portals need to be able to send email

Right now they are configured to directly talk with the isti SMTP server without authentication, so they can only del... Andrea Dell'Amico

01:13 PM D4Science Infrastructure Task #2705: Create new VM for production Infrastructure Gateway (Liferay 6.2)

Done.
Andrea Dell'Amico

12:42 PM D4Science Infrastructure Task #2705: Create new VM for production Infrastructure Gateway (Liferay 6.2)

Massimiliano Assante wrote:
> I don't have access to infra-gateway via ssh, not as root, nor as life user
Sorry,... Andrea Dell'Amico

Mar 30, 2016

02:54 PM D4Science Infrastructure Task #2192 (Closed): Gcc and boost installation

Andrea Dell'Amico

02:52 PM D4Science Infrastructure Task #2192: Gcc and boost installation

May I close it? A new one should be opened with the installation instructions into the definitive place.
Andrea Dell'Amico

02:50 PM D4Science Infrastructure Support #2134 (Closed): Enhancing Statistical Manager performance

An agreement was reached, see https://support.d4science.org/issues/3070
Andrea Dell'Amico

02:49 PM D4Science Infrastructure Task #3070 (Closed): Provide a way of starting and stopping a list of VMs by some users

A web page with a fixed - and checkable - list of hosts should be sufficient.
Associated actions (via a simple cgi... Andrea Dell'Amico

Mar 26, 2016

10:47 PM D4Science Infrastructure Task #2705 (In Progress): Create new VM for production Infrastructure Gateway (Liferay 6.2)

On db-gateway.d4science.org postgresql 9.4 is running. There's a db defined named `infra_liferay`. The db username is... Andrea Dell'Amico

Mar 25, 2016

06:43 PM D4Science Infrastructure Task #3064: Investigate the installation of an identity manager

A possibile candidate: https://evolveum.com/midpoint/ Andrea Dell'Amico

06:42 PM D4Science Infrastructure Task #3064 (Closed): Investigate the installation of an identity manager

It could be a way to eliminate manage.research-infrastructures.eu and integrate the ldap data from the portals with r... Andrea Dell'Amico

Mar 24, 2016

01:51 PM D4Science Infrastructure Support #2134: Enhancing Statistical Manager performance

Gianpaolo Coro wrote:
> You're right, this is not the place to talk about the mandates of the projects we are involv... Andrea Dell'Amico

01:18 PM D4Science Infrastructure Support #2134: Enhancing Statistical Manager performance

I lied. The last question is: why you expose the _mandatory_ requirement as argument only after days of discussion?
... Andrea Dell'Amico

01:13 PM D4Science Infrastructure Support #2134: Enhancing Statistical Manager performance

Gianpaolo Coro wrote:
> So we are stuck. Our platforms has been involved in projects that require embedding software... Andrea Dell'Amico

12:49 PM D4Science Infrastructure Support #2134: Enhancing Statistical Manager performance

Gianpaolo Coro wrote:
> On the dataminer nodes (both dev and prod, see my comment above). Most of the jobs run in th... Andrea Dell'Amico

12:38 PM D4Science Infrastructure Support #2134: Enhancing Statistical Manager performance

On what? prodution dataminer nodes? devel dataminer nodes? and what about the JDK heap? Do the jobs run inside or out... Andrea Dell'Amico

12:22 PM D4Science Infrastructure Support #2134: Enhancing Statistical Manager performance

Gianpaolo Coro wrote:
> 16 CPUs or cores should be sufficient if these were averagely free.
(16 * (2 dataminer no... Andrea Dell'Amico

Mar 23, 2016

06:14 PM D4Science Infrastructure Support #2134: Enhancing Statistical Manager performance

Waiting for Tom for the operations.
But we really need some data, because your `top` output is completely useless.
... Andrea Dell'Amico

02:53 PM D4Science Infrastructure Task #536 (In Progress): Thredds: tomcat runs as root

It seems that the switch did not happen, the old machine is still active.
Andrea Dell'Amico

02:29 PM D4Science Infrastructure Task #2946 (Closed): Install ganglia and the local nagios checks on thredds-p-d4s.d4science.org

What cluster on Ganglia? Andrea Dell'Amico

Mar 22, 2016

07:12 PM D4Science Infrastructure Task #2921 (Feedback): Please reduce log on couchdb

The log level has been reset to `info` on `couchdb01-d-d4s.d4science.org`
Andrea Dell'Amico

07:11 PM D4Science Infrastructure Task #2921 (In Progress): Please reduce log on couchdb

The log level is already set as info by the playbook.
On the production servers, were the logs are at `info` level... Andrea Dell'Amico

07:00 PM D4Science Infrastructure Task #2922 (Closed): Dismiss couchdb instance available at http://accounting-d4s.d4science.org:5984/

Done. Andrea Dell'Amico

07:00 PM D4Science Infrastructure Task #2922 (In Progress): Dismiss couchdb instance available at http://accounting-d4s.d4science.org:5984/

Andrea Dell'Amico

06:00 PM D4Science Infrastructure Task #2892 (Feedback): Provide SoBigData colleagues with details for interfacing with LDAP

Andrea Dell'Amico

05:59 PM D4Science Infrastructure Task #2892 (In Progress): Provide SoBigData colleagues with details for interfacing with LDAP

Andrea Dell'Amico

04:52 PM D4Science Infrastructure Task #2227 (Feedback): test replication tools for postgres

Andrea Dell'Amico

04:52 PM D4Science Infrastructure Task #2192 (Feedback): Gcc and boost installation

Andrea Dell'Amico

04:52 PM D4Science Infrastructure Task #2192 (In Progress): Gcc and boost installation

Andrea Dell'Amico

04:09 PM D4Science Infrastructure Task #2192: Gcc and boost installation

Gianpaolo Coro wrote:
> Just one note: to make quickrank compile correctly, the required compilator version is g++ 5... Andrea Dell'Amico

Mar 21, 2016

06:37 PM D4Science Infrastructure Task #2784 (Closed): nagios, ganglia: add credentials to mongodb checks

Ganglia is working again too.
Andrea Dell'Amico

05:34 PM D4Science Infrastructure Task #2784: nagios, ganglia: add credentials to mongodb checks

For the future: the arbiter does not work against authentication in the same way the other mongo servers do. So I dis... Andrea Dell'Amico

05:17 PM D4Science Infrastructure Task #2784 (In Progress): nagios, ganglia: add credentials to mongodb checks

The nagios part is OK. Andrea Dell'Amico

04:17 PM D4Science Infrastructure Task #2784: nagios, ganglia: add credentials to mongodb checks

I see that mongo1 (the arbiter?) now fails for every check.
Andrea Dell'Amico

04:16 PM D4Science Infrastructure Task #2784: nagios, ganglia: add credentials to mongodb checks

About nagios: the checked functionalities are:
~~~
if action == "connections":
elif action == "replicati... Andrea Dell'Amico

02:49 PM D4Science Infrastructure Task #2784: nagios, ganglia: add credentials to mongodb checks

How? Is there a user/password that nagios (and ganglia) can use to only access the data they need to read?
Andrea Dell'Amico

02:49 PM D4Science Infrastructure Task #2261: Better check for data.d4science.org availability

Andrea Dell'Amico wrote:
> @roberto.cirillo@isti.cnr.it @francesco.mangiacrapa@isti.cnr.it can you provide such URLs... Andrea Dell'Amico

02:25 PM D4Science Infrastructure Task #2774 (Feedback): Configure the mysql and postgresql local backups to not run if the remote backup is active

Both the postgresql and mysql backup cron jobs have been changed. Andrea Dell'Amico

02:06 PM D4Science Infrastructure Task #2774 (In Progress): Configure the mysql and postgresql local backups to not run if the remote backup is active

Andrea Dell'Amico

02:23 PM D4Science Infrastructure Task #2775: Configure backuppc to run the local DB backup scripts and write the backup status on the remote host

Andrea Dell'Amico wrote:
> BackupPC can be globally configured to run mysql/postgresql backups as a pre-step.
>
>... Andrea Dell'Amico

Mar 18, 2016

07:03 PM D4Science Infrastructure Task #2775 (Closed): Configure backuppc to run the local DB backup scripts and write the backup status on the remote host

BackupPC can be globally configured to run mysql/postgresql backups as a pre-step.
When the backuppc operations ar... Andrea Dell'Amico

06:59 PM D4Science Infrastructure Task #2774 (Closed): Configure the mysql and postgresql local backups to not run if the remote backup is active

The local scripts can check the existance and the state of a local file, eg `/var/log/backuppc.log` to check if a r... Andrea Dell'Amico

06:41 PM D4Science Infrastructure Task #1854 (Feedback): Tool for usage statistics on DataMiner

I've imported the logs of the last two days. The procedure will run every hour from now on.
The imported informati... Andrea Dell'Amico

01:57 PM D4Science Infrastructure Task #1854: Tool for usage statistics on DataMiner

I'm going to change the dataminer{1,2}-p-d4s.d4science.org nginx configuration to let the access logs be available fo... Andrea Dell'Amico

01:35 PM D4Science Infrastructure Task #2754 (Closed): Activate backup for some analytics.d4science.org, cm.research-infrastructures.eu, code-repo.d4science.org

Of analytics.d4science.org we need a backup o /etc, /usr/share and /data
The others need a complete backup Andrea Dell'Amico

Mar 17, 2016

09:34 PM D4Science Infrastructure Task #1854: Tool for usage statistics on DataMiner

The basic configuration is ready. The server answers at http://analytics.d4science.org
It authenticates against lda... Andrea Dell'Amico

05:00 PM D4Science Infrastructure Task #2720: Explore rundeck as a better alternative to foreman

A possibile way to ansible integration: https://github.com/srknc/rundeck_ansible_sync Andrea Dell'Amico

04:59 PM D4Science Infrastructure Task #2720 (Closed): Explore rundeck as a better alternative to foreman

See http://rundeck.org
Andrea Dell'Amico

01:22 PM D4Science Infrastructure Task #2705: Create new VM for production Infrastructure Gateway (Liferay 6.2)

Massimiliano Assante wrote:
> Using (a customised) liferay bundle as in the past.
OK

> > * Can we start from ... Andrea Dell'Amico

12:20 PM D4Science Infrastructure Task #2705: Create new VM for production Infrastructure Gateway (Liferay 6.2)

Questions:
* Plans to deploy the application? Using the liferay bundle as in the past, or using the war(s) on an a... Andrea Dell'Amico

12:39 PM D4Science Infrastructure Task #1285: Explore Zulip as a free alternative to Slack for team collaboration

Alternatives: mattermost http://www.mattermost.org/ and rocket.chat https://rocket.chat/
The first seems the more ma... Andrea Dell'Amico

Mar 16, 2016

03:46 PM D4Science Infrastructure Task #1854: Tool for usage statistics on DataMiner

Hostname is going to be: analytics.d4science.org, IP 146.48.122.20 Andrea Dell'Amico

03:37 PM D4Science Infrastructure Task #1854 (In Progress): Tool for usage statistics on DataMiner

Andrea Dell'Amico

03:26 PM D4Science Infrastructure Support #2620 (Closed): Access to D4Science svn

Andrea Dell'Amico

12:53 PM D4Science Infrastructure Task #2275 (Closed): Try foreman with the ansible plugin as an ansible console

An alternative is to ask me or @tommaso.piccioli@isti.cnr.it for an access to `shell.research-infrastructures.eu`, w... Andrea Dell'Amico

12:50 PM D4Science Infrastructure Task #2275: Try foreman with the ansible plugin as an ansible console

What's needed to enable the foreman callback on your own desktop:
1. Edit the ansible global configuration file. `... Andrea Dell'Amico

Mar 15, 2016

09:06 PM D4Science Infrastructure Task #2275: Try foreman with the ansible plugin as an ansible console

I configured authentication against ldap, so everyone can authenticate. After the first authentication I (or another ... Andrea Dell'Amico

09:03 PM D4Science Infrastructure Task #2275: Try foreman with the ansible plugin as an ansible console

Foreman is working in some way. An ansible plugin is needed, instructions will follow.
shell.research-infrastructure... Andrea Dell'Amico

04:32 PM D4Science Infrastructure Task #2508 (Feedback): No firewall on the cassandra hosts

Firewall rules deployed on the production cluster too.
Andrea Dell'Amico

03:56 PM D4Science Infrastructure Support #2620 (Feedback): Access to D4Science svn

Andrea Dell'Amico

03:55 PM D4Science Infrastructure Support #2620: Access to D4Science svn

User created and access to the D4Science SVN granted. It will be operative when @mnikolopoulos@cite.gr will complete ... Andrea Dell'Amico

03:45 PM D4Science Infrastructure Support #2620 (In Progress): Access to D4Science svn

Andrea Dell'Amico