Actions
Incident #12380
closed
node22.p.d4science.research-infrastructures.eu is not responding
Start date:
Sep 04, 2018
Due date:
% Done:
100%
Estimated time:
Infrastructure:
Pre-Production
Description
From the nohup.out file:
Exception in thread "Thread-11647282" java.lang.OutOfMemoryError: Java heap space
at org.gcube.common.searchservice.searchlibrary.resultset.elements.StreamWorker.run(StreamWorker.java:52)
Exception in thread "Thread-173" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-14996838" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-14997091" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-14997471" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "GHNConsumer" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "GHNConsumer$<anon>" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-14997999" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "GHNConsumer" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "cb-computations-2" java.lang.IllegalStateException: Fatal Exception thrown on Scheduler.Worker thread.
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "GHNConsumer" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "EventSweeper" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-14998199" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "cb-computations-1" java.lang.IllegalStateException: Fatal Exception thrown on Scheduler.Worker thread.
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "ServiceDispacherThread-53" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "GHNConsumer" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "GHNConsumer" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "GHNConsumer$<anon>" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-14998267" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Updater" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "GHNConsumer$<anon>" java.lang.OutOfMemoryError: GC overhead limit exceeded
As I do not know if it's one of the exist backed services, I did not restart it because I don't know the procedure to follow if the exist database fails.
Related issues
Updated by Roberto Cirillo almost 7 years ago
- Status changed from New to In Progress
Updated by Roberto Cirillo almost 7 years ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
- Infrastructure Pre-Production added
- Infrastructure deleted (
Production)
This is the VREModeler service instance of preproduction infrastructure. It's a gCore based service and there is a memory leak in this service, so sometimes it's needed a restart.
I've just restarted it, now it is up.
Updated by Andrea Dell'Amico almost 7 years ago
I assumed that it was production, from the hostname (it's still a Ubuntu 12.04 FYI).
About the restart, a cron job can be added in the same way we do on other gCore services.
Updated by Roberto Cirillo almost 7 years ago
Andrea Dell'Amico wrote:
I assumed that it was production, from the hostname (it's still a Ubuntu 12.04 FYI).
About the restart, a cron job can be added in the same way we do on other gCore services.
OK. I'm going to open a separate ticket for this activity
Updated by Roberto Cirillo almost 7 years ago
- Related to Task #12384: Provide a cronjob for restarting VREModeler service added
Actions
