Project

General

Profile

Actions

Task #4617

closed

Smart-executor- node24.d4science.org: Too Many Open Files

Added by Roberto Cirillo almost 9 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Costantino Perciante
Category:
Data Management
Target version:
Start date:
Jul 04, 2016
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Production

Description

On node24.d4science.org is running the social-data-indexer plugin of SmartExecutor service.
We have the following exceptios:
catalina.out:

java.lang.NullPointerException
com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=cassandra2-p-d4s.d4science.org(146.48.123.140):9160, latency=6000(6000), attempts=3]Timed out waiting for conne
ction
        at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:218)
        at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:185)
        at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:66)
        at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:67)
        at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:253)
        at com.netflix.astyanax.thrift.ThriftClusterImpl.describeKeyspaces(ThriftClusterImpl.java:155)
        at com.netflix.astyanax.thrift.ThriftClusterImpl.describeKeyspace(ThriftClusterImpl.java:174)
        at org.gcube.portal.databook.server.CassandraClusterConnection.SetUpKeySpaces(CassandraClusterConnection.java:157)
        at org.gcube.portal.databook.server.CassandraClusterConnection.<init>(CassandraClusterConnection.java:101)
        at org.gcube.portal.databook.server.DBCassandraAstyanaxImpl.<init>(DBCassandraAstyanaxImpl.java:201)
        at org.gcube.socialnetworking.socialdataindexer.SocialDataIndexerPlugin.launch(SocialDataIndexerPlugin.java:95)
        at org.gcube.vremanagement.executor.pluginmanager.RunnablePlugin.run(RunnablePlugin.java:67)
        at org.gcube.vremanagement.executor.scheduler.SmartExecutorTask.execute(SmartExecutorTask.java:214)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
java.lang.NullPointerException

ghn.log

00:00:11.560 [pool-2-thread-1] WARN  ProfileBuilder: unable to detect the uptime of this machine
java.io.IOException: Cannot run program "uptime": error=24, Too many open files
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047) ~[na:1.7.0_80]
        at java.lang.Runtime.exec(Runtime.java:617) ~[na:1.7.0_80]
        at java.lang.Runtime.exec(Runtime.java:450) ~[na:1.7.0_80]
        at java.lang.Runtime.exec(Runtime.java:347) ~[na:1.7.0_80]
        at org.gcube.smartgears.handlers.container.lifecycle.ProfileBuilder.uptime(ProfileBuilder.java:297) [common-smartgears-1.2.7-3.11.0-128702.jar:na]
        at org.gcube.smartgears.handlers.container.lifecycle.ProfileBuilder.update(ProfileBuilder.java:228) [common-smartgears-1.2.7-3.11.0-128702.jar:na]
        at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager$2$1.run(ProfileManager.java:266) [common-smartgears-1.2.7-3.11.0-128702.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_80]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_80]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_80]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
Caused by: java.io.IOException: error=24, Too many open files
        at java.lang.UNIXProcess.forkAndExec(Native Method) ~[na:1.7.0_80]
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:187) ~[na:1.7.0_80]
        at java.lang.ProcessImpl.start(ProcessImpl.java:130) ~[na:1.7.0_80]
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028) ~[na:1.7.0_80]
        ... 13 common frames omitted

The plugin was not working and I've restarted the container. It's need further analysis.


Related issues

Related to D4Science Infrastructure - Task #4647: node24.d4science.org : Increase the maximum number of file descriptorsClosed_InfraScience Systems EngineerJul 06, 2016

Actions
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)