Actions
Task #9980
closed
The accounting-aggregator exhausted all the available memory
Status:
Closed
Priority:
Normal
Assignee:
Category:
System Application
Target version:
Start date:
Oct 18, 2017
Due date:
% Done:
100%
Estimated time:
Infrastructure:
Production
Description
And was killed by the OOM.
There's now a nagios handler that automatically restarts it, but an investigation is required to understand if there's a leak somewhere or more memory is needed. Both the available RAM and the heap size were increased some days ago.
Updated by Andrea Dell'Amico over 7 years ago
In the logs (catalina.out, btw) there's the following exception, multiple times:
Oct 18, 2017 6:07:54 PM com.orientechnologies.common.log.OLogManager log
INFO: Caught Network I/O errors on orientdb01-d4s.d4science.org:2424/smartexecutor, trying an automatic reconnection... (error: Error during exception deserialization: java.lang.ClassNotFoundException: com.orientechnologies.o
rient.server.distributed.ODistributedException)
Oct 18, 2017 6:07:54 PM com.orientechnologies.common.log.OLogManager log
SEVERE: Error during exception deserialization
java.lang.ClassNotFoundException: com.orientechnologies.orient.server.distributed.ODistributedException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1718)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1569)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:677)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:552)
at java.lang.Throwable.readObject(Throwable.java:914)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:429)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:393)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:275)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:167)
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2225)
at com.orientechnologies.orient.client.remote.OStorageRemote$12.execute(OStorageRemote.java:793)
at com.orientechnologies.orient.client.remote.OStorageRemote$12.execute(OStorageRemote.java:789)
at com.orientechnologies.orient.client.remote.OStorageRemote$1.execute(OStorageRemote.java:166)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:243)
at com.orientechnologies.orient.client.remote.OStorageRemote.asyncNetworkOperation(OStorageRemote.java:158)
at com.orientechnologies.orient.client.remote.OStorageRemote.updateRecord(OStorageRemote.java:773)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2183)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:181)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2707)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:102)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1805)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1796)
at org.gcube.vremanagement.executor.persistence.orientdb.OrientDBPersistenceConnector.releaseScheduledTask(OrientDBPersistenceConnector.java:364)
at org.gcube.vremanagement.executor.persistence.orientdb.OrientDBPersistenceConnector.releaseScheduledTask(OrientDBPersistenceConnector.java:380)
at org.gcube.vremanagement.executor.persistence.orientdb.OrientDBPersistenceConnector.reserveScheduledTask(OrientDBPersistenceConnector.java:325)
at org.gcube.vremanagement.executor.SmartExecutorInitializator.onInit(SmartExecutorInitializator.java:354)
at org.gcube.smartgears.RegisterApplicationManagerObserver$InitAppManager.call(RegisterApplicationManagerObserver.java:157)
at org.gcube.smartgears.RegisterApplicationManagerObserver$InitAppManager.call(RegisterApplicationManagerObserver.java:138)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Updated by Luca Frosini over 7 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
This issue happen when the plug-in trying to aggregate a day having thousand of records which are not aggregable .The plugin need more than one day to terminate. Infact, it has to compare thousand of records and at the end keep in in memory.
We should investigate instead why this situation occur.
I'm going to close this ticket.
Actions
