Actions
Task #9210
closed
The SPD service stopped working
Added by Andrea Dell'Amico over 8 years ago. Updated over 8 years ago.
Start date:
Jul 07, 2017
Due date:
Jul 12, 2017
% Done:
100%
Estimated time:
Infrastructure:
Production
Description
This morning GP tried to access the service, but it did not work. A service restart was needed apparently, and the gCore endpoint was absent from the scope because the ghn profile update failed.
Updated by Andrea Dell'Amico over 8 years ago
The exception in the smartgears logs:
10:09:15.533 [catalina-exec-3] WARN ConfigurationFactory: No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/home/gcube/tomcat/webapps/species-products-discovery/WEB-INF/lib/ehcache-core-2.5.1.jar!/ehcache-failsafe.xml
10:09:15.539 [catalina-exec-3] ERROR Manager: error submitting search
java.lang.NullPointerException: null
at org.gcube.data.spd.utils.ExecutorsContainer.execSearch(ExecutorsContainer.java:20) ~[ExecutorsContainer.class:na]
at org.gcube.data.spd.manager.search.Search.search(Search.java:117) ~[Search.class:na]
at org.gcube.data.spd.resources.Manager.search(Manager.java:142) ~[Manager.class:na]
at sun.reflect.GeneratedMethodAccessor151.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.23.2.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.23.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.23.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.23.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.23.2.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.gcube.smartgears.managers.RequestManager.doFilter(RequestManager.java:95) [common-smartgears-2.1.2-4.4.0-146443.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [tomcat-catalina-7.0.55.jar:7.0.55]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) [tomcat-coyote-7.0.55.jar:7.0.55]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) [tomcat-coyote-7.0.55.jar:7.0.55]
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) [tomcat-coyote-7.0.55.jar:7.0.55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote-7.0.55.jar:7.0.55]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote-7.0.55.jar:7.0.55]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
Updated by Andrea Dell'Amico over 8 years ago
After the service started working again, a search requested by me failed. I requested the search again, and it went well. The failed one generated the exception:
11:37:48.142 [catalina-exec-6] INFO RequestAccounting: REQUEST SERVED ON SpeciesProductsDiscovery:DataAccess(UNKNOWN) CALLED FROM andrea.dellamico@146.48.122.51 IN SCOPE /d4science.research-infrastructures.eu/gCubeApps/Biodiversit
yLab FINISHED IN 25 millis
11:37:48.970 [spd-search-thread-14] ERROR HavingContext: Error converting item to XML
java.lang.NullPointerException: null
at com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.startElement(AbstractDOMParser.java:1120) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1363) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2786) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347) ~[na:1.7.0_80]
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) ~[na:1.7.0_80]
at org.gcube.dataaccess.spd.havingengine.exl.HavingContext.buildDocument(HavingContext.java:90) [having-engine-1.0.1-4.4.0-142495.jar:na]
at org.gcube.dataaccess.spd.havingengine.exl.HavingContext.getDocument(HavingContext.java:76) [having-engine-1.0.1-4.4.0-142495.jar:na]
at org.gcube.dataaccess.spd.havingengine.exl.HavingFunctions.xpath(HavingFunctions.java:61) [having-engine-1.0.1-4.4.0-142495.jar:na]
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
at org.apache.commons.jexl2.internal.MethodExecutor.execute(MethodExecutor.java:64) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.internal.AbstractExecutor$Method.invoke(AbstractExecutor.java:327) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.Interpreter.call(Interpreter.java:1068) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1100) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.parser.ASTMethodNode.jjtAccept(ASTMethodNode.java:18) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1317) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.parser.ASTReference.jjtAccept(ASTReference.java:18) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.Interpreter.interpret(Interpreter.java:232) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.ExpressionImpl.evaluate(ExpressionImpl.java:65) [commons-jexl-2.1.1.jar:2.1.1]
at org.gcube.dataaccess.spd.havingengine.exl.HavingStatementFactoryEXL$HavingStatementJEXL.accept(HavingStatementFactoryEXL.java:68) [having-engine-1.0.1-4.4.0-142495.jar:na]
at org.gcube.data.spd.manager.search.workers.HavingFilterWorker.execute(HavingFilterWorker.java:19) [HavingFilterWorker.class:na]
at org.gcube.data.spd.manager.search.Worker.run(Worker.java:44) [Worker.class:na]
at org.gcube.common.authorization.library.AuthorizedTasks$2.run(AuthorizedTasks.java:75) [common-authorization-2.0.2-4.4.0-144378.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
11:37:48.990 [spd-search-thread-0] ERROR HavingContext: Error converting item to XML
org.xml.sax.SAXException: FWK005 parse may not be called while parsing.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:271) ~[na:1.7.0_80]
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347) ~[na:1.7.0_80]
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) ~[na:1.7.0_80]
at org.gcube.dataaccess.spd.havingengine.exl.HavingContext.buildDocument(HavingContext.java:90) [having-engine-1.0.1-4.4.0-142495.jar:na]
at org.gcube.dataaccess.spd.havingengine.exl.HavingContext.getDocument(HavingContext.java:76) [having-engine-1.0.1-4.4.0-142495.jar:na]
at org.gcube.dataaccess.spd.havingengine.exl.HavingFunctions.xpath(HavingFunctions.java:61) [having-engine-1.0.1-4.4.0-142495.jar:na]
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
at org.apache.commons.jexl2.internal.MethodExecutor.execute(MethodExecutor.java:64) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.internal.AbstractExecutor$Method.invoke(AbstractExecutor.java:327) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.Interpreter.call(Interpreter.java:1068) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1100) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.parser.ASTMethodNode.jjtAccept(ASTMethodNode.java:18) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1317) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.parser.ASTReference.jjtAccept(ASTReference.java:18) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.Interpreter.interpret(Interpreter.java:232) [commons-jexl-2.1.1.jar:2.1.1]
at org.apache.commons.jexl2.ExpressionImpl.evaluate(ExpressionImpl.java:65) [commons-jexl-2.1.1.jar:2.1.1]
at org.gcube.dataaccess.spd.havingengine.exl.HavingStatementFactoryEXL$HavingStatementJEXL.accept(HavingStatementFactoryEXL.java:68) [having-engine-1.0.1-4.4.0-142495.jar:na]
at org.gcube.data.spd.manager.search.workers.HavingFilterWorker.execute(HavingFilterWorker.java:19) [HavingFilterWorker.class:na]
at org.gcube.data.spd.manager.search.Worker.run(Worker.java:44) [Worker.class:na]
at org.gcube.common.authorization.library.AuthorizedTasks$2.run(AuthorizedTasks.java:75) [common-authorization-2.0.2-4.4.0-144378.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
[...]
Updated by Pasquale Pagano over 8 years ago
- Due date set to Jul 12, 2017
Please analyse the incident and then close the ticket. Remember that we have SLA on incident ticket and therefore the incident has to be resolved by July 12.
Updated by Pasquale Pagano over 8 years ago
- Tracker changed from Incident to Task
- Status changed from New to Closed
Actions
