Project

General

Profile

Actions

Incident #4200

closed

IS not responding

Added by Gianpaolo Coro almost 9 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Immediate
Category:
Other
Target version:
-
Start date:
Jun 08, 2016
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Development

Description

The IS/Registry is not responding on https://dev.d4science.org/web/guest/monitor
This is needed to add and manage the many requests for algorithms integration that arrive every day.

Actions #1

Updated by Roberto Cirillo almost 9 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 100

The problem seems to be the same happened in production: a memory leak on accounting libs.
There were a lot of accounting exceptions but I cannot see the cause because the log level was set to ERROR:

2016-06-01 18:02:35,843 ERROR persistence.PersistenceBackendConfiguration [pool-1017-thread-1,getInstance:57] AccountingPersistenceConfiguration not initialized correctly. It will not be used. Trying the next one if any.
java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedConstructorAccessor123.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.gcube.documentstore.persistence.PersistenceBackendConfiguration.getInstance(PersistenceBackendConfiguration.java:51)
        at org.gcube.documentstore.persistence.PersistenceBackendFactory.discoverPersistenceBackend(PersistenceBackendFactory.java:117)
        at org.gcube.documentstore.persistence.PersistenceBackendFactory.rediscoverPersistenceBackend(PersistenceBackendFactory.java:169)
        at org.gcube.documentstore.persistence.PersistenceBackendRediscover.run(PersistenceBackendRediscover.java:38)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.rangeCheck(ArrayList.java:635)
        at java.util.ArrayList.get(ArrayList.java:411)
        at org.gcube.accounting.persistence.AccountingPersistenceConfiguration.getServiceEndpoint(AccountingPersistenceConfiguration.java:68)
        at org.gcube.accounting.persistence.AccountingPersistenceConfiguration.<init>(AccountingPersistenceConfiguration.java:42)
        ... 14 more

I've restarted the container with the log level set to debug. Now the IS-Collector is up and running but it's need to monitoring this container because the problem is not solved. I guess there are several VRE scope without accounting service endpoint that cause a memory leak in accounting lib.

Actions #2

Updated by Roberto Cirillo almost 9 years ago

  • Status changed from In Progress to Resolved
Actions #3

Updated by Gianpaolo Coro almost 9 years ago

  • Status changed from Resolved to In Progress

The problem is again there, just after few minutes.
The devsec environment has never worked, instead.

Actions #4

Updated by Gianpaolo Coro almost 9 years ago

  • % Done changed from 100 to 0
Actions #5

Updated by Roberto Cirillo almost 9 years ago

  • Status changed from In Progress to Feedback

Could you give me more details, please?

From monitor I see that the IS-Collector is working and also the registry service. In addition the ghn containers are correctly updated

https://preprod.d4science.org/web/guest/monitor
https://dev.d4science.org/web/guest/monitor
Actions #6

Updated by Gianpaolo Coro almost 9 years ago

  • Status changed from Feedback to Closed
  • % Done changed from 0 to 100

it seems to be working now, but just after a restart of the portal. I don't know if this is related.

Actions #7

Updated by Pasquale Pagano over 8 years ago

  • Target version deleted (gCube related support)
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)