D4science

Jul 06, 2018

10:07 AM D4Science Infrastructure Upgrade #12023 (In Progress): /gcube/preprod upgrade to gCube 4.12

Roberto Cirillo

Jul 05, 2018

03:36 PM D4Science Infrastructure Task #12118 (Closed): MongoDB: upgrade production cluster to version 3.4

The upgrade will be done in the following way:
* Upgrade secondary members of the replica set;
* Step down the re... Roberto Cirillo

03:33 PM D4Science Infrastructure Task #12117 (Closed): MongoDB: upgrade production cluster to version 3.2

The upgrade will be done in the following way:
* Upgrade secondary members of the replica set;
* Step down the re... Roberto Cirillo

03:25 PM D4Science Infrastructure Upgrade #12116: D4Science Root VO upgrade to gCube 4.12

**HostingNode**
*WHN to upgrade*
* **POSTPONED** catalogue-ws.d4science.org
* **(CNR) DONE** all the instanc... Roberto Cirillo

03:02 PM D4Science Infrastructure Upgrade #12116 (Completed): D4Science Root VO upgrade to gCube 4.12

Roberto Cirillo

Jul 04, 2018

11:36 AM D4Science Infrastructure Task #12058: IS-Collector: Restart all the smartgears services with too many open connections to IS-Collector

socialnetworking1.d4science.org restarted for connections to collector-parthenosvo.d4science.org Roberto Cirillo

Jul 03, 2018

12:29 PM D4Science Infrastructure Incident #12103 (Rejected): workspace in dev: upload doesn't work anymore

It's not possible to upload any file from my ws.
In attachment the exception found on dev portal.
Please, could yo... Roberto Cirillo

11:41 AM D4Science Infrastructure Task #12102 (Closed): Preproduction infrastructure: create an alias for enabling services

The enabling services of the root-VO will be recycled from enabling services of Edison VO.
It's needed to create an... Roberto Cirillo

Jul 02, 2018

05:22 PM D4Science Infrastructure Task #12087: Remove services from Edison VO

scope removed from ansible-playbook.
We should also remove the Edison key from the playbook. Roberto Cirillo

04:54 PM D4Science Infrastructure Task #12087: Remove services from Edison VO

node44.p.d4science.research-infrastructures.eu removed Roberto Cirillo

04:45 PM D4Science Infrastructure Task #12087 (In Progress): Remove services from Edison VO

On node44.p.d4science.research-infrastructures.eu (VREModeler service, not provisioned), the scope will be removed ma... Roberto Cirillo

02:32 PM D4Science Infrastructure Task #12087 (Closed): Remove services from Edison VO

It's needed to remove the Edison scope from the following services:
~~~
accounting-service1.d4science.org
accoun... Roberto Cirillo

03:11 PM D4Science Infrastructure Task #12090 (Closed): Remove Edison VO scope from infrastructure-monitor

In order to dismiss the Edison VO, it's needed to remove the Edison VO from all the production and preproduction port... Roberto Cirillo

03:01 PM D4Science Infrastructure Task #11862: Dismiss EDISON Virtual Organisation and related enabling services

The services to dismiss are the following:
~~~
collector-edison.d4science.org:8080 (IS-Collector)
collector-edis... Roberto Cirillo

10:00 AM D4Science Infrastructure Upgrade #12019: D4Science Infrastructure Upgrade to gCube 4.12.0

due to changes in a related task Roberto Cirillo

Jun 29, 2018

03:13 PM D4Science Infrastructure Upgrade #12023 (Paused): /gcube/preprod upgrade to gCube 4.12

Roberto Cirillo

11:08 AM D4Science Infrastructure Upgrade #12023 (In Progress): /gcube/preprod upgrade to gCube 4.12

Roberto Cirillo

10:29 AM D4Science Infrastructure Task #12058 (Paused): IS-Collector: Restart all the smartgears services with too many open connections to IS-Collector

Roberto Cirillo

Jun 28, 2018

10:34 AM D4Science Infrastructure Task #12058: IS-Collector: Restart all the smartgears services with too many open connections to IS-Collector

Services restarted. Roberto Cirillo

10:14 AM D4Science Infrastructure Task #12058 (In Progress): IS-Collector: Restart all the smartgears services with too many open connections to IS-Collector

Roberto Cirillo

09:56 AM D4Science Infrastructure Task #12058 (Closed): IS-Collector: Restart all the smartgears services with too many open connections to IS-Collector

Roberto Cirillo

Jun 27, 2018

06:18 PM D4Science Infrastructure Task #12031: gCubeApps - IS-Collector: too many open connections

OK you are still not convinced but I say that because all the threads in the logs are named "AccountingScheduledThrea... Roberto Cirillo

03:02 PM D4Science Infrastructure Task #12031: gCubeApps - IS-Collector: too many open connections

@luca.frosini@isti.cnr.it , @lucio.lelii@isti.cnr.it the log seems to be clear to me:
There were a lot of accountin... Roberto Cirillo

02:56 PM D4Science Infrastructure Task #12031: gCubeApps - IS-Collector: too many open connections

After the restart of the containers identified yesterday, the situation seems to be returned to more normal levels.
... Roberto Cirillo

09:48 AM D4Science Infrastructure Task #12031: gCubeApps - IS-Collector: too many open connections

Are you agree with my analysis? if not why there are a lot of accounting exception on the log in attachment? Have you... Roberto Cirillo

04:13 PM D4Science Infrastructure Upgrade #12023 (Planned): /gcube/preprod upgrade to gCube 4.12

Roberto Cirillo

04:13 PM D4Science Infrastructure Upgrade #12019: D4Science Infrastructure Upgrade to gCube 4.12.0

due to changes in a related task Roberto Cirillo

02:39 PM StocksAndFisheriesKB Bug #12040: Error while publishing GRSF record

The service has been upgraded, now it is up and running. Roberto Cirillo

02:23 PM D4Science Infrastructure Support #12032 (Closed): ip-90-147-166-112.ct1.garrservices.it: too many opened connection to node66.p.d4science

The services have been restarted. Now I see 0 opened connections from 90.147.166.112. I'm going to close this ticket. Roberto Cirillo

02:05 PM D4Science Infrastructure Support #12032 (In Progress): ip-90-147-166-112.ct1.garrservices.it: too many opened connection to node66.p.d4science

OK. I'm going to restart the services one at a time Roberto Cirillo

12:34 PM D4Science Infrastructure Support #12032 (Feedback): ip-90-147-166-112.ct1.garrservices.it: too many opened connection to node66.p.d4science

I've verified the logs on all the dataminer instances with private ip. I see the same exceptions found on ckan-bb1, ... Roberto Cirillo

11:24 AM D4Science Infrastructure Support #12034 (Closed): ckan-bb1.d4science.org: too many opened connections to node66.p.d4science

The service has been restarted with log level set to DEBUG/TRACE. Now, the high number of opened connections to node6... Roberto Cirillo

Jun 26, 2018

05:54 PM D4Science Infrastructure Support #12034 (Feedback): ckan-bb1.d4science.org: too many opened connections to node66.p.d4science

@luca.frosini@isti.cnr.it could I restart the container or do you want to check the behavior of this host?
I think t... Roberto Cirillo

03:10 PM D4Science Infrastructure Support #12034: ckan-bb1.d4science.org: too many opened connections to node66.p.d4science

@andrea.dellamico@isti.cnr.it I guess the logback reloading doesn't work. I've set the log to DEBUG three hours ago b... Roberto Cirillo

12:09 PM D4Science Infrastructure Support #12034: ckan-bb1.d4science.org: too many opened connections to node66.p.d4science

Andrea Dell'Amico wrote:
> The first line of the `logback.xml` file that automatically reload itself is:
>
> ~~~
... Roberto Cirillo

11:47 AM D4Science Infrastructure Support #12034: ckan-bb1.d4science.org: too many opened connections to node66.p.d4science

In this host there is the smartgears distribution 4.11 but I don't remember if the logback version released in 4.11 a... Roberto Cirillo

11:32 AM D4Science Infrastructure Support #12034 (In Progress): ckan-bb1.d4science.org: too many opened connections to node66.p.d4science

I see a lot of opened connection to node66.p, sometimes over 50. This is not normal.
From logs (INFO level) I don't ... Roberto Cirillo

11:21 AM D4Science Infrastructure Support #12034 (Closed): ckan-bb1.d4science.org: too many opened connections to node66.p.d4science

I see a lot of concurred opened connections from ckan-bb1.d4science.org to node66.p.d4science.research-infrastructure... Roberto Cirillo

05:50 PM D4Science Infrastructure Support #12033 (Closed): rstudio2.d4science.org: too many opened connetions to node66.p.d4science

After the restart I see again 0 opened connections. I'm going to close this ticket Roberto Cirillo

11:00 AM D4Science Infrastructure Support #12033: rstudio2.d4science.org: too many opened connetions to node66.p.d4science

For the moment, after the restart I see 0 opened connections Roberto Cirillo

10:42 AM D4Science Infrastructure Support #12033 (In Progress): rstudio2.d4science.org: too many opened connetions to node66.p.d4science

I'm going to decrease the log level in order to analyze the logs
Roberto Cirillo

10:36 AM D4Science Infrastructure Support #12033 (Closed): rstudio2.d4science.org: too many opened connetions to node66.p.d4science

I see a lot of concurred opened connections from rstudio2.d4science.org.
This host should be monitored.
Roberto Cirillo

04:40 PM D4Science Infrastructure Task #12044 (Closed): Remove the component storagehub-client-wrapper (with groupid org.gcube.portal') from Nexus

Done Roberto Cirillo

04:31 PM D4Science Infrastructure Task #12044 (In Progress): Remove the component storagehub-client-wrapper (with groupid org.gcube.portal') from Nexus

Roberto Cirillo

04:01 PM D4Science Infrastructure Task #12031 (Feedback): gCubeApps - IS-Collector: too many open connections

I think, finally, I've localize the problem.
Yesterday, at 6 p.m. I've restarted the IS-Collector and Registry se... Roberto Cirillo

10:12 AM D4Science Infrastructure Task #12031 (In Progress): gCubeApps - IS-Collector: too many open connections

Roberto Cirillo

10:10 AM D4Science Infrastructure Task #12031 (Closed): gCubeApps - IS-Collector: too many open connections

There are too many open connections on IS-Collector of gCubeApps VO hosted on:
~~~
node66.p.d4science.research-in... Roberto Cirillo

02:48 PM D4Science Infrastructure Incident #12043 (Rejected): ckan-grsf-admin2: more memory is needed

The smartgears container has been killed by system this night at 3 a.m.
I've tried to restart the container but th... Roberto Cirillo

11:41 AM D4Science Infrastructure Support #12032: ip-90-147-166-112.ct1.garrservices.it: too many opened connection to node66.p.d4science

Andrea Dell'Amico wrote:
> It's the IP address that's used by the hosts without a public IP. So there's one (or more... Roberto Cirillo

10:16 AM D4Science Infrastructure Support #12032 (Closed): ip-90-147-166-112.ct1.garrservices.it: too many opened connection to node66.p.d4science

Who is this host?
I'm not able to access to this host: ip-90-147-166-112.ct1.garrservices.it
Please could you gi... Roberto Cirillo

Jun 25, 2018

04:19 PM D4Science Infrastructure Upgrade #12023: /gcube/preprod upgrade to gCube 4.12

**Hosting Nodes**
*WHN to upgrade*
* **(CNR) DONE** catalogue-ws-t.pre.d4science.org
* **(CNR) DONE** datamine... Roberto Cirillo

03:41 PM D4Science Infrastructure Upgrade #12023 (Completed): /gcube/preprod upgrade to gCube 4.12

Roberto Cirillo

03:04 PM D4Science Infrastructure Upgrade #12019 (Completed): D4Science Infrastructure Upgrade to gCube 4.12.0

Roberto Cirillo

12:15 PM D4Science Infrastructure Incident #12015: News Feed search function returns result up to mid April 2018

The smart-executor service is up and running and all the social-indexer plugin resources are correctly published on I... Roberto Cirillo

Jun 21, 2018

10:11 AM D4Science Infrastructure Task #12008 (Rejected): grsf-publisher deployed on gcube-staging repo doesn't work

The latest grsf-publisher service version (grsf-publisher-ws-1.6.0-4.12.0-169302) deployed on staging repo doesn't wo... Roberto Cirillo

10:03 AM D4Science Infrastructure Task #12007 (New): Storage-MongoDB: Disable nagios alert when a backup is performed

If a backup is performed the nagios alert related to mongo5 instance should be disabled otherwise I cannot understand... Roberto Cirillo

Jun 20, 2018

02:59 PM D4Science Infrastructure Incident #11988 (Closed): Copy doesn't work prepoerly

The test has been passed. I'm going to close this ticket. Thanks. Roberto Cirillo

11:14 AM D4Science Infrastructure Incident #11988: Copy doesn't work prepoerly

Thanks @francesco.mangiacrapa@isti.cnr.it
Please @massimiliano.assante@isti.cnr.it could you deploy the new jar o... Roberto Cirillo

Jun 19, 2018

05:56 PM D4Science Infrastructure Incident #11988: Copy doesn't work prepoerly

Please @francesco.mangiacrapa@isti.cnr.it could you add this case on workspace test plan ? Roberto Cirillo

02:37 PM D4Science Infrastructure Incident #11988 (Feedback): Copy doesn't work prepoerly

I've performed some tests on porduction workspace and I can confirm that the storage id of the copied file is the sam... Roberto Cirillo

12:54 PM D4Science Infrastructure Incident #11988 (In Progress): Copy doesn't work prepoerly

Roberto Cirillo

12:49 PM D4Science Infrastructure Incident #11988 (Closed): Copy doesn't work prepoerly

In order to test the new copy implementation I have performed some tests by our workspace on copy operation and I've ... Roberto Cirillo

11:26 AM D4Science Infrastructure Task #11964 (In Progress): MongoDB: upgrade dev cluster to version 3.4

Roberto Cirillo

09:57 AM D4Science Infrastructure Task #11982 (Closed): Increase RAM on ckan-grsf-admin2.d4science.org

As discussed by mail, it's needed to add 1 GB to the following VM:
~~~
ckan-grsf-admin2.d4science.org
~~~
Ple... Roberto Cirillo

Jun 18, 2018

05:35 PM D4Science Infrastructure Upgrade #11970 (Completed): Please update GRSF service in production with latest versions available

Done Roberto Cirillo

05:31 PM D4Science Infrastructure Upgrade #11970 (In Progress): Please update GRSF service in production with latest versions available

Roberto Cirillo

05:30 PM D4Science Infrastructure Upgrade #11970 (Planned): Please update GRSF service in production with latest versions available

I'm going to install the following version:
~~~
<groupId>org.gcube.data-catalogue</groupId>
<artifactId>grsf-p... Roberto Cirillo

10:44 AM D4Science Infrastructure Task #9122: Upgrade MongoDB to 3.4 version

due to changes in a related task Roberto Cirillo

10:24 AM D4Science Infrastructure Task #11964 (Closed): MongoDB: upgrade dev cluster to version 3.4

The upgrade will be done in the following way:
* Upgrade secondary members of the replica set.
* Step down the re... Roberto Cirillo

10:19 AM D4Science Infrastructure Task #9123 (Closed): MongoDB: upgrade dev cluster to version 3.2

The storage library works fine with 3.2 version. I'm going to close this ticket and open a new one for upgrading the ... Roberto Cirillo

Jun 13, 2018

11:32 AM D4Science Infrastructure Task #11944: Add 1 GB of RAM to accounting service instances

OK I agree for adding 1 GB of RAM for each instance.
I don't know why accounting-service3 is down, maybe @luca.frosi... Roberto Cirillo

10:33 AM D4Science Infrastructure Task #11944 (Feedback): Add 1 GB of RAM to accounting service instances

Roberto Cirillo

10:33 AM D4Science Infrastructure Task #11944 (In Progress): Add 1 GB of RAM to accounting service instances

I don't know if it's better to decrease the heap size or to increase the RAM. At this moment both the instances are s... Roberto Cirillo

09:59 AM D4Science Infrastructure Task #11944 (Closed): Add 1 GB of RAM to accounting service instances

The accounting instances have the following (default) heap limits:
~~~
-Xms2048m -Xmx2048m
~~~
but the VM hav... Roberto Cirillo

Jun 12, 2018

03:49 PM D4Science Infrastructure Upgrade #11834 (Completed): D4Science Root VO upgrade to gCube 4.11.1

Roberto Cirillo

03:49 PM D4Science Infrastructure Upgrade #11834: D4Science Root VO upgrade to gCube 4.11.1

@luca.frosini@isti.cnr.it have you notice that this is an already closed maintenance release?
Please, if you need t... Roberto Cirillo

03:44 PM D4Science Infrastructure VM Creation #11852 (Closed): Create a VM to deploy the new storage hub application.

Roberto Cirillo

11:49 AM D4Science Infrastructure Task #9123: MongoDB: upgrade dev cluster to version 3.2

Please @tommaso.piccioli@isti.cnr.it could you plan this activity? Roberto Cirillo

11:40 AM D4Science Infrastructure Task #11880 (Closed): Workspace in dev doesn't delete correctly the files stored on MongoDB when a delete from trash is performed

In order to reproduce your test, I've tried to delete my Trash directly by StorageManager libraries, after that, I've... Roberto Cirillo