D4science

Jan 03, 2018

07:18 PM D4Science Infrastructure Task #9213: enable calDAV authentication on D4Science Gateways

Andrea Dell'Amico wrote:
> I've fixed a discrepancy in the web server and haproxy versions between the preprod and t... Andrea Dell'Amico

Jan 02, 2018

03:21 PM D4Science Infrastructure Task #10816 (Feedback): TLS Error on services.d4science.org when accessed by Firefox

It seems a openssl bug. Why it started showing now I've yet to understand. I worked around it for the time being and ... Andrea Dell'Amico

02:19 PM D4Science Infrastructure Task #10816 (In Progress): TLS Error on services.d4science.org when accessed by Firefox

I'm investigating the issue. I do not have a clue yet, the same exact configuration is working on the dev and preprod... Andrea Dell'Amico

Dec 29, 2017

06:19 PM D4Science Infrastructure Task #9213 (Feedback): enable calDAV authentication on D4Science Gateways

I've fixed a discrepancy in the web server and haproxy versions between the preprod and the production servers. Now t... Andrea Dell'Amico

Dec 22, 2017

02:46 PM D4Science Infrastructure Incident #10791 (Closed): php session files on wiki.d4science.org are filling up the disk

I added a cron job that cleanup the empty session files and the ones older than three days.
Some space should be mad... Andrea Dell'Amico

11:57 AM D4Science Infrastructure Incident #10791 (Closed): php session files on wiki.d4science.org are filling up the disk

They are never removed, we need a script to cleanup the old files.
Andrea Dell'Amico

01:52 PM D4Science Infrastructure Task #10748 (Closed): Reinstall 2 GARR dataminer instances

The two dataminer have been added back to the load balancer.
Andrea Dell'Amico

Dec 21, 2017

08:30 PM D4Science Infrastructure Task #10666 (Closed): DataMiner - Install Eartrack software for support DataMiner algorithms

I don't see a difference between this ticket and #10653, I'm closing this one. Andrea Dell'Amico

07:17 PM D4Science Infrastructure Task #10666 (In Progress): DataMiner - Install Eartrack software for support DataMiner algorithms

On a second look, it should be useful to us in the long run. Andrea Dell'Amico

06:52 PM D4Science Infrastructure Task #10666: DataMiner - Install Eartrack software for support DataMiner algorithms

Well, conda is a real mess for us.
It installs its own python environment that's totally incompatible with the alrea... Andrea Dell'Amico

08:22 PM D4Science Infrastructure Task #10664: DataMiner - Create a list of the names of Python packages installed on the DataMiner machines

Another consideration: the above lists are completely useless to the people that is going to use conda. conda is a co... Andrea Dell'Amico

06:08 PM D4Science Infrastructure Task #10664: DataMiner - Create a list of the names of Python packages installed on the DataMiner machines

I will not track the packages that are not explicitly required for the dataminers btw, because they can be triggered ... Andrea Dell'Amico

06:06 PM D4Science Infrastructure Task #10664 (In Progress): DataMiner - Create a list of the names of Python packages installed on the DataMiner machines

The python environment that's currently installed on all the dataminer servers is based on python 2.7.6 (Ubuntu 14.04... Andrea Dell'Amico

04:31 PM D4Science Infrastructure Task #10748: Reinstall 2 GARR dataminer instances

Reconfigured. I'll add them again to the corresponding cluster tomorrow, so that I'll sure that all the R dependencie... Andrea Dell'Amico

03:03 PM D4Science Infrastructure Task #10748: Reinstall 2 GARR dataminer instances

The configuration is ongoing.
Andrea Dell'Amico

03:47 PM D4Science Infrastructure Task #10783 (Closed): Reinstall worker8-hadoop.d4science.org

The VM is ready to be reconfigured. Andrea Dell'Amico

03:40 PM D4Science Infrastructure Task #10783 (In Progress): Reinstall worker8-hadoop.d4science.org

Andrea Dell'Amico

03:25 PM D4Science Infrastructure Task #10783 (Closed): Reinstall worker8-hadoop.d4science.org

On a more performant hypervisor.
Andrea Dell'Amico

03:12 PM D4Science Infrastructure Task #10758: R Packages Lists for RStudio - Distinguish between Prototyping and Production Environments

I see. If we can wait, there's a chance that the GARR cloud people will enable CephFS for us in some weeks. With it w... Andrea Dell'Amico

02:13 PM D4Science Infrastructure Task #10769 (Closed): Enable ldap authentication on grafana.d4science.org

Done. I also added the hadoop dashboards to show some data.
Andrea Dell'Amico

12:34 PM D4Science Infrastructure Task #10769: Enable ldap authentication on grafana.d4science.org

Now I'll move the configuration to a postgresql database.
Andrea Dell'Amico

12:33 PM D4Science Infrastructure Task #10769: Enable ldap authentication on grafana.d4science.org

Done. There's what I guess it's a bug, because if I set the mail field correctly the authentication fails because it ... Andrea Dell'Amico

Dec 20, 2017

09:15 PM D4Science Infrastructure Task #10769: Enable ldap authentication on grafana.d4science.org

Yet to find how to match the groups
Andrea Dell'Amico

06:07 PM D4Science Infrastructure Task #10769 (In Progress): Enable ldap authentication on grafana.d4science.org

Andrea Dell'Amico

12:43 PM D4Science Infrastructure Task #10769 (Closed): Enable ldap authentication on grafana.d4science.org

Natively in the grafana configuration Andrea Dell'Amico

05:29 PM D4Science Infrastructure Task #10748 (In Progress): Reinstall 2 GARR dataminer instances

VM creation is failing right now. Andrea Dell'Amico

12:11 PM D4Science Infrastructure Task #10317 (Closed): Increase the resources of dataminer[1:5]-p-d4s.d4science.org

Andrea Dell'Amico wrote:
>
> dataminer5-proto.d4science.org is being configured (I missed it).
dataminer5-proto... Andrea Dell'Amico

12:08 PM D4Science Infrastructure Task #10758: R Packages Lists for RStudio - Distinguish between Prototyping and Production Environments

It's doable, the R packages management is the same on both the dataminer and rstudio servers.
Renaming a node is a p... Andrea Dell'Amico

Dec 19, 2017

06:08 PM D4Science Infrastructure Task #10317: Increase the resources of dataminer[1:5]-p-d4s.d4science.org

dataminer[1:5]-p-d4s.d4science.org are now running with 32 GB of RAM and 16 CPUs. Added to the dataminer.garr haproxy... Andrea Dell'Amico

11:47 AM D4Science Infrastructure Task #9132 (Closed): Dismiss the PC used as a server by the D-Net team

Andrea Dell'Amico

11:46 AM D4Science Infrastructure Task #9134 (Closed): Move everything from dlib34.isti.cnr.it to playground-t.dnet.d4science.org

Done weeks ago. Andrea Dell'Amico

11:46 AM D4Science Infrastructure Incident #10717 (Closed): Tagme stops responding because is going out of heap space

Andrea Dell'Amico

11:46 AM D4Science Infrastructure Incident #10717: Tagme stops responding because is going out of heap space

Closing the ticket. If those accesses were legit, they should rate limit it before removing the firewall rules. Andrea Dell'Amico

Dec 18, 2017

05:01 PM D4Science Infrastructure Task #10748 (Closed): Reinstall 2 GARR dataminer instances

The one with hostname ip-90-147-167-236.ct1.garrservices.it fails executing jobs correctly even if there's apparently... Andrea Dell'Amico

01:37 PM D4Science Infrastructure Task #10729: Broken disk on dlib33x.dom0.research-infrastructures.eu

That's worrisome, the omsa tools should have reported them to nagios. Or maybe the problem lasted too little time to ... Andrea Dell'Amico

09:29 AM D4Science Infrastructure Task #10729 (Closed): Broken disk on dlib33x.dom0.research-infrastructures.eu

From the nagios check:
~~~
CRITICAL: mdstat:[md3(931.39 GiB raid1):F:sdg1:_U, md2(931.39 GiB raid1):UU, md1(931... Andrea Dell'Amico

10:19 AM D4Science Infrastructure VM Creation #10686 (Closed): New VM with Resource Registry instance

There was a double `/` on the nginx configuration. Fixed now. Andrea Dell'Amico

09:55 AM D4Science Infrastructure Task #10666: DataMiner - Install Eartrack software for support DataMiner algorithms

I see that there's a conda module for ansible, here: https://github.com/UDST/ansible-conda Andrea Dell'Amico

Dec 17, 2017

04:00 PM D4Science Infrastructure VM Creation #10686 (Feedback): New VM with Resource Registry instance

The VM is up. The resource registry configuration should be verified. @luca.frosini@isti.cnr.it can you do that? I ga... Andrea Dell'Amico

02:54 PM D4Science Infrastructure VM Creation #10686: New VM with Resource Registry instance

Hostname and IP address are going to be: manuele-registry.dev.d4science.org 146.48.122.19 Andrea Dell'Amico

02:50 PM D4Science Infrastructure VM Creation #10686 (In Progress): New VM with Resource Registry instance

Andrea Dell'Amico

Dec 15, 2017

05:51 PM D4Science Infrastructure Task #10662: orientdb01-d4s.d4science.org keeps crashing badly

@luca.frosini@isti.cnr.it just remove the two nodes from the inventory file. When we will be able to use them we'll a... Andrea Dell'Amico

03:18 PM D4Science Infrastructure Incident #10717 (Closed): Tagme stops responding because is going out of heap space

It seems that a lot of accesses started today from 141.115.102.180 (many requests per second, often) and the servic... Andrea Dell'Amico

Dec 14, 2017

05:34 PM D4Science Infrastructure Task #10274 (Closed): The IRDTunaAtlas R package does not build anymore

I'm removing it from the list. When a working version will be available, we will reinstate it.
Andrea Dell'Amico

05:10 PM D4Science Infrastructure Task #10317: Increase the resources of dataminer[1:5]-p-d4s.d4science.org

Update: we need 6 VMs in proto, all of them hosted at CNR because we do not have the choice to instantiate a 16 GB RA... Andrea Dell'Amico

04:42 PM D4Science Infrastructure Task #10685 (Feedback): Add new cronjob on accounting-aggregator

Done.
Andrea Dell'Amico

04:34 PM D4Science Infrastructure Task #10685 (In Progress): Add new cronjob on accounting-aggregator

Andrea Dell'Amico

04:16 PM D4Science Infrastructure VM Creation #10686: New VM with Resource Registry instance

I'll create a new VM, better to have the same standard everywhere.
Andrea Dell'Amico

03:47 PM D4Science Infrastructure Incident #10701 (Closed): accounting bloat on workspace-repository-prod1.d4science.org

I removed the files after stopping the services. Now it's behaving correctly.
Andrea Dell'Amico

02:40 PM D4Science Infrastructure Incident #10701 (Closed): accounting bloat on workspace-repository-prod1.d4science.org

The fallback accounting data is filling up the disk. I removed the files, but they do not stop growing again.
~~~
... Andrea Dell'Amico

Dec 13, 2017

07:28 PM D4Science Infrastructure Task #10662: orientdb01-d4s.d4science.org keeps crashing badly

I just increased the RAM of the three orientdb production servers to 6GB each. You can comfortably add another GB of ... Andrea Dell'Amico

06:37 PM D4Science Infrastructure VM Creation #10681 (Closed): Instance a new VM to host phpPgAdmin

The VM is ready. Andrea Dell'Amico

02:43 PM D4Science Infrastructure VM Creation #10681 (In Progress): Instance a new VM to host phpPgAdmin

hostname and IP are going to be pg-admin.d4science.org 146.48.122.5
Andrea Dell'Amico

02:41 PM D4Science Infrastructure VM Creation #10681 (Closed): Instance a new VM to host phpPgAdmin

Reserved to db-tuna.d4science.org right now. Andrea Dell'Amico

05:27 PM D4Science Infrastructure Task #10661 (Closed): Dismiss the old jenkins cluster

Andrea Dell'Amico

05:19 PM D4Science Infrastructure Task #10317: Increase the resources of dataminer[1:5]-p-d4s.d4science.org

Moreover, dataminer6-p-d4s.d4science.org and dataminer7-p-d4s.d4science.org must be reassigned to the proto backend (... Andrea Dell'Amico

11:26 AM D4Science Infrastructure Task #10671 (Closed): Move the nagios checks to the new Jenkins instance

Andrea Dell'Amico

Dec 12, 2017

07:52 PM D4Science Infrastructure Task #10671 (Closed): Move the nagios checks to the new Jenkins instance

It is not monitored yet. Andrea Dell'Amico

07:40 PM D4Science Infrastructure Task #10662: orientdb01-d4s.d4science.org keeps crashing badly

So they say that we need more heap and more memory. The tuning doc seems to have some good advice, you should try som... Andrea Dell'Amico

12:19 PM D4Science Infrastructure Task #10662 (Closed): orientdb01-d4s.d4science.org keeps crashing badly

It stops in state that it's impossible for a nagios handler to restart it: the process must be brutally killed. The s... Andrea Dell'Amico

06:08 PM D4Science Infrastructure Task #10669 (Closed): Reconfigure the openaire dev solr nodes

Andrea Dell'Amico

06:06 PM D4Science Infrastructure Task #10669 (Feedback): Reconfigure the openaire dev solr nodes

Andrea Dell'Amico

06:05 PM D4Science Infrastructure Task #10669 (In Progress): Reconfigure the openaire dev solr nodes

Andrea Dell'Amico

06:03 PM D4Science Infrastructure Task #10669: Reconfigure the openaire dev solr nodes

Done. Andrea Dell'Amico

06:01 PM D4Science Infrastructure Task #10669 (Closed): Reconfigure the openaire dev solr nodes

The HDP cluster hostname changed, so the solr nodes must adapt their configuration.
Andrea Dell'Amico

06:07 PM D4Science Infrastructure Task #10460 (Closed): Upgrade resources for worker[1-3]-hadoop-test.d4science.org

The cluster is working. Andrea Dell'Amico

12:06 PM D4Science Infrastructure Task #10661: Dismiss the old jenkins cluster

The involved hosts are:
ci.research-infrastructures.eu
slave[1:3].ci.research-infrastructures.eu
The related n... Andrea Dell'Amico

12:04 PM D4Science Infrastructure Task #10661 (Closed): Dismiss the old jenkins cluster

All the work was moved on the new one.
If needed we can add a slave with a JDK 7 tu run the builds of old versions.
Andrea Dell'Amico

10:58 AM D4Science Infrastructure Task #10656: Restart hosting nodes to get new buckets configuration

I'm reassigning the task to @roberto.cirillo@isti.cnr.it because most of them are gCore nodes. Andrea Dell'Amico

Dec 11, 2017

07:41 PM D4Science Infrastructure Incident #10651 (Closed): dataminer3-p-d4s.d4science.org filled the disk

Under Smartgears/state there are 17GB worth of files:
~~~
-rw-r--r-- 1 gcube gcube 0 Dec 11 19:34 _d4science.... Andrea Dell'Amico

07:26 PM D4Science Infrastructure Task #10460: Upgrade resources for worker[1-3]-hadoop-test.d4science.org

The cluster is up and running. worker8-hadoop.d4science.org has been removed for the time being. Andrea Dell'Amico

07:12 PM D4Science Infrastructure Task #10460: Upgrade resources for worker[1-3]-hadoop-test.d4science.org

worker8-hadoop.d4science.org still creates problems. Maybe it should be installed on a better hypervisor.
Andrea Dell'Amico

02:44 PM D4Science Infrastructure Task #10460: Upgrade resources for worker[1-3]-hadoop-test.d4science.org

Reinstallation started. The up to date node mapping:
~~~
ambari-hadoop.d4science.org (dlib34x)
rm1-hadoop.d4scie... Andrea Dell'Amico

04:44 PM D4Science Infrastructure Task #10478 (Feedback): Dismiss old message-broker

I disabled the nagios check for activemq and shut down all the involved VMs. Andrea Dell'Amico

Dec 07, 2017

07:25 PM D4Science Infrastructure Task #10460: Upgrade resources for worker[1-3]-hadoop-test.d4science.org

Erasing the nodes without reinstalling them wasn't a good choice. We will reinstall them from scratch, and then recon... Andrea Dell'Amico

04:59 PM D4Science Infrastructure Task #10460: Upgrade resources for worker[1-3]-hadoop-test.d4science.org

All the cluster have been reconfigured, HDP 2.6.3 has been installed.
There's a problem on worker8-hadoop.d4science... Andrea Dell'Amico

06:24 PM D4Science Infrastructure Support #10607 (Closed): access to geonetwork-sdi.dev.d4science.org

You can access now as user `gcube` Andrea Dell'Amico

12:45 PM D4Science Infrastructure Task #10606 (Closed): Disable nagios check on old services

Done. Andrea Dell'Amico

Dec 06, 2017

07:31 PM D4Science Infrastructure Task #9857 (Closed): Evacuate the hypervisors from dlib14x to dlib19x

The nodes are empty. Andrea Dell'Amico

06:54 PM D4Science Infrastructure Task #10270: Configure the xen hypervisor to let the guests talk to the storage network

The test made on dlib18x went bad, so a full reboot of every hypervisor is needed to permit the VMs to talk to the pr... Andrea Dell'Amico

06:06 PM D4Science Infrastructure Task #10597 (Closed): Give the ability to enable and configure the tomcat manager when we use the tomcat-multiple-instances role

It's working, just tested on thredds-d-d4s.d4science.org.
Andrea Dell'Amico

05:04 PM D4Science Infrastructure Task #10597: Give the ability to enable and configure the tomcat manager when we use the tomcat-multiple-instances role

The smartgears side is also ready to be tested. The template part is:
~~~
{% if tomcat_m_manager_install or t... Andrea Dell'Amico

04:30 PM D4Science Infrastructure Task #10597: Give the ability to enable and configure the tomcat manager when we use the tomcat-multiple-instances role

The tomcat part should be complete. Andrea Dell'Amico

03:59 PM D4Science Infrastructure Task #10597: Give the ability to enable and configure the tomcat manager when we use the tomcat-multiple-instances role

The configuration will be global. It will affect all the tomcat instances installed on a node.
Andrea Dell'Amico

02:30 PM D4Science Infrastructure Task #10597 (In Progress): Give the ability to enable and configure the tomcat manager when we use the tomcat-multiple-instances role

Andrea Dell'Amico

02:02 PM D4Science Infrastructure Task #10597 (Closed): Give the ability to enable and configure the tomcat manager when we use the tomcat-multiple-instances role

Also, arbitrary additional users and roles must be accepted and managed.
Also, the nginx template used by smartgears... Andrea Dell'Amico

05:40 PM D4Science Infrastructure Support #10602: access to geonetwork-sdi.dev.d4science.org

? You can add your public ssh key to your profile on the portal, under `My Account -> Custom fields`.
Andrea Dell'Amico

01:33 PM D4Science Infrastructure Support #10595 (Closed): access to geofence-d-d4s.d4science.org

Done. Andrea Dell'Amico

01:31 PM D4Science Infrastructure Support #10594 (Closed): access to geoserver1-spatial-dev.d4science.org

You had access already. Andrea Dell'Amico

01:29 PM D4Science Infrastructure Support #10592 (Closed): access to geonetwork-sdi.dev.d4science.org

Done. Andrea Dell'Amico

12:49 PM D4Science Infrastructure Task #10460: Upgrade resources for worker[1-3]-hadoop-test.d4science.org

The DB server has been configured. Andrea Dell'Amico

Dec 05, 2017

07:29 PM D4Science Infrastructure Task #10460: Upgrade resources for worker[1-3]-hadoop-test.d4science.org

The VMs have been renamed, moved and reinstalled when needed. All the data disk volumes renamed to match the VM host... Andrea Dell'Amico

12:03 PM D4Science Infrastructure Task #10460 (In Progress): Upgrade resources for worker[1-3]-hadoop-test.d4science.org

The involved VMs are:
ambari-hadoop-test.d4science.org (dlib34x) ambari-hadoop.d4science.org
rm1-hadoop-test.d4sc... Andrea Dell'Amico