Task #9210: The SPD service stopped working - D4Science Infrastructure - D4science

Actions

Copy link

Task #9210

closed

The SPD service stopped working

Added by Andrea Dell'Amico over 8 years ago. Updated over 8 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Lucio Lelii

Category:

Application

Target version:

UnSprintable

Start date:

Jul 07, 2017

Due date:

Jul 12, 2017

% Done:

100%

Estimated time:

Infrastructure:

Production

Description

This morning GP tried to access the service, but it did not work. A service restart was needed apparently, and the gCore endpoint was absent from the scope because the ghn profile update failed.

Actions

Copy link

Updated by Andrea Dell'Amico over 8 years ago

The exception in the smartgears logs:

10:09:15.533 [catalina-exec-3] WARN  ConfigurationFactory: No configuration found. Configuring ehcache from ehcache-failsafe.xml  found in the classpath: jar:file:/home/gcube/tomcat/webapps/species-products-discovery/WEB-INF/lib/ehcache-core-2.5.1.jar!/ehcache-failsafe.xml
10:09:15.539 [catalina-exec-3] ERROR Manager: error submitting search
java.lang.NullPointerException: null
        at org.gcube.data.spd.utils.ExecutorsContainer.execSearch(ExecutorsContainer.java:20) ~[ExecutorsContainer.class:na]
        at org.gcube.data.spd.manager.search.Search.search(Search.java:117) ~[Search.class:na]
        at org.gcube.data.spd.resources.Manager.search(Manager.java:142) ~[Manager.class:na]
        at sun.reflect.GeneratedMethodAccessor151.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.23.2.jar:na]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.23.2.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.23.2.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.23.2.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.23.2.jar:na]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.23.2.jar:na]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.23.2.jar:na]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.23.2.jar:na]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.23.2.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.23.2.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.23.2.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.23.2.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.gcube.smartgears.managers.RequestManager.doFilter(RequestManager.java:95) [common-smartgears-2.1.2-4.4.0-146443.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [tomcat-catalina-7.0.55.jar:7.0.55]
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) [tomcat-coyote-7.0.55.jar:7.0.55]
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) [tomcat-coyote-7.0.55.jar:7.0.55]
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) [tomcat-coyote-7.0.55.jar:7.0.55]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote-7.0.55.jar:7.0.55]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote-7.0.55.jar:7.0.55]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]

Actions

Copy link

Updated by Andrea Dell'Amico over 8 years ago

After the service started working again, a search requested by me failed. I requested the search again, and it went well. The failed one generated the exception:

11:37:48.142 [catalina-exec-6] INFO  RequestAccounting: REQUEST SERVED ON SpeciesProductsDiscovery:DataAccess(UNKNOWN) CALLED FROM andrea.dellamico@146.48.122.51 IN SCOPE /d4science.research-infrastructures.eu/gCubeApps/Biodiversit
yLab FINISHED IN 25 millis
11:37:48.970 [spd-search-thread-14] ERROR HavingContext: Error converting item to XML
java.lang.NullPointerException: null
        at com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.startElement(AbstractDOMParser.java:1120) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1363) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2786) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347) ~[na:1.7.0_80]
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) ~[na:1.7.0_80]
        at org.gcube.dataaccess.spd.havingengine.exl.HavingContext.buildDocument(HavingContext.java:90) [having-engine-1.0.1-4.4.0-142495.jar:na]
        at org.gcube.dataaccess.spd.havingengine.exl.HavingContext.getDocument(HavingContext.java:76) [having-engine-1.0.1-4.4.0-142495.jar:na]
        at org.gcube.dataaccess.spd.havingengine.exl.HavingFunctions.xpath(HavingFunctions.java:61) [having-engine-1.0.1-4.4.0-142495.jar:na]
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
        at org.apache.commons.jexl2.internal.MethodExecutor.execute(MethodExecutor.java:64) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.internal.AbstractExecutor$Method.invoke(AbstractExecutor.java:327) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.Interpreter.call(Interpreter.java:1068) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1100) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.parser.ASTMethodNode.jjtAccept(ASTMethodNode.java:18) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1317) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.parser.ASTReference.jjtAccept(ASTReference.java:18) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.Interpreter.interpret(Interpreter.java:232) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.ExpressionImpl.evaluate(ExpressionImpl.java:65) [commons-jexl-2.1.1.jar:2.1.1]
        at org.gcube.dataaccess.spd.havingengine.exl.HavingStatementFactoryEXL$HavingStatementJEXL.accept(HavingStatementFactoryEXL.java:68) [having-engine-1.0.1-4.4.0-142495.jar:na]
        at org.gcube.data.spd.manager.search.workers.HavingFilterWorker.execute(HavingFilterWorker.java:19) [HavingFilterWorker.class:na]
        at org.gcube.data.spd.manager.search.Worker.run(Worker.java:44) [Worker.class:na]
        at org.gcube.common.authorization.library.AuthorizedTasks$2.run(AuthorizedTasks.java:75) [common-authorization-2.0.2-4.4.0-144378.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
11:37:48.990 [spd-search-thread-0] ERROR HavingContext: Error converting item to XML
org.xml.sax.SAXException: FWK005 parse may not be called while parsing.
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:271) ~[na:1.7.0_80]
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347) ~[na:1.7.0_80]
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) ~[na:1.7.0_80]
        at org.gcube.dataaccess.spd.havingengine.exl.HavingContext.buildDocument(HavingContext.java:90) [having-engine-1.0.1-4.4.0-142495.jar:na]
        at org.gcube.dataaccess.spd.havingengine.exl.HavingContext.getDocument(HavingContext.java:76) [having-engine-1.0.1-4.4.0-142495.jar:na]
        at org.gcube.dataaccess.spd.havingengine.exl.HavingFunctions.xpath(HavingFunctions.java:61) [having-engine-1.0.1-4.4.0-142495.jar:na]
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
        at org.apache.commons.jexl2.internal.MethodExecutor.execute(MethodExecutor.java:64) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.internal.AbstractExecutor$Method.invoke(AbstractExecutor.java:327) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.Interpreter.call(Interpreter.java:1068) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1100) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.parser.ASTMethodNode.jjtAccept(ASTMethodNode.java:18) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1317) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.parser.ASTReference.jjtAccept(ASTReference.java:18) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.Interpreter.interpret(Interpreter.java:232) [commons-jexl-2.1.1.jar:2.1.1]
        at org.apache.commons.jexl2.ExpressionImpl.evaluate(ExpressionImpl.java:65) [commons-jexl-2.1.1.jar:2.1.1]
        at org.gcube.dataaccess.spd.havingengine.exl.HavingStatementFactoryEXL$HavingStatementJEXL.accept(HavingStatementFactoryEXL.java:68) [having-engine-1.0.1-4.4.0-142495.jar:na]
        at org.gcube.data.spd.manager.search.workers.HavingFilterWorker.execute(HavingFilterWorker.java:19) [HavingFilterWorker.class:na]
        at org.gcube.data.spd.manager.search.Worker.run(Worker.java:44) [Worker.class:na]
        at org.gcube.common.authorization.library.AuthorizedTasks$2.run(AuthorizedTasks.java:75) [common-authorization-2.0.2-4.4.0-144378.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
[...]

Actions

Copy link

Updated by Pasquale Pagano over 8 years ago

Due date set to Jul 12, 2017

Please analyse the incident and then close the ticket. Remember that we have SLA on incident ticket and therefore the incident has to be resolved by July 12.

Actions

Copy link