D4Science Infrastructure

I've checked the container hosted on "node27.d.d4science.research-infrastructures.eu": now, it is correctly registered to the infrastructure after the restart

I've found an exception after the migration of IS-Collector from java 7 to java8.

The following exception seems to be related to an incompatibility between java8 and the XercesImpl library (dependency of IS-Collector service).

java.lang.RuntimeException: could not load authorization endpoints
        at org.gcube.common.authorization.library.enpoints.AuthorizationEndpointScanner.endpoints(AuthorizationEndpointScanner.java:75)
        at org.gcube.common.authorization.client.proxy.DefaultAuthorizationProxy.<init>(DefaultAuthorizationProxy.java:47)
        at org.gcube.common.authorization.client.Constants.authorizationService(Constants.java:22)
        at org.gcube.common.handlers.GCUBEHandler.retrieveTokenInfo(GCUBEHandler.java:221)
        at org.gcube.common.handlers.GCUBEHandler.invoke(GCUBEHandler.java:165)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.server.AxisServer.invoke(AxisServer.java:248)
        at org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:664)
        at org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:382)
        at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:291)
Caused by: java.lang.IllegalStateException: org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
        at com.sun.xml.internal.bind.v2.util.XmlFactory.createParserFactory(XmlFactory.java:128)
        at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.getXMLReader(UnmarshallerImpl.java:139)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:157)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:162)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:171)
        at org.gcube.common.authorization.library.enpoints.AuthorizationEndpointScanner.endpoints(AuthorizationEndpointScanner.java:53)
        ... 11 more
Caused by: org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
        at org.apache.xerces.parsers.AbstractSAXParser.setFeature(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl.setFeatures(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParserImpl(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserFactoryImpl.setFeature(Unknown Source)
        at com.sun.xml.internal.bind.v2.util.XmlFactory.createParserFactory(XmlFactory.java:121)
        ... 16 more

it needs further analysis.

obis2.i-marine.research-infrastructures.eu alias geoserver-dev2.d4science-ii.research-infrastructures.eu migrated (no java app running on this host, only postgresql)
dev.d4science.org migrated (no java app running on this host, only postgresql)
...

Problems with java8 on node6.d4science.org too:

...
[PUBLISHER] 16:28:09,382 ERROR startup.ContextConfig [localhost-startStop-1,processAnnotationsJar:1937] Unable to process Jar entry [javassist/util/proxy/SerializedProxy.class] from Jar [jar:file:/home/gcube/tomcat/webapps/smart-executor/WEB-INF/lib/javassist-3.19.0-GA.jar!/] for annotations
java.io.EOFException
    at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
    at org.apache.tomcat.util.bcel.classfile.Utility.swallowMethodParameters(Utility.java:797)
    at org.apache.tomcat.util.bcel.classfile.Attribute.readAttribute(Attribute.java:171)
    at org.apache.tomcat.util.bcel.classfile.FieldOrMethod.<init>(FieldOrMethod.java:57)
    at org.apache.tomcat.util.bcel.classfile.Method.<init>(Method.java:71)
    at org.apache.tomcat.util.bcel.classfile.ClassParser.readMethods(ClassParser.java:267)
    at org.apache.tomcat.util.bcel.classfile.ClassParser.parse(ClassParser.java:127)
    at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2058)
    at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1934)
    at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1900)
    at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1885)
    at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1317)
    at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:876)
    at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:374)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5378)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1247)
    at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1898)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized
INFO: WSSERVLET12: JAX-WS context listener initializing
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getRequestWrapperClass
INFO: Dynamically creating request wrapper Class org.gcube.vremanagement.executor.api.jaxws.GetIterationState
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getResponseWrapperClass
INFO: Dynamically creating response wrapper bean Class org.gcube.vremanagement.executor.api.jaxws.GetIterationStateResponse
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getRequestWrapperClass
INFO: Dynamically creating request wrapper Class org.gcube.vremanagement.executor.api.jaxws.GetStateEvolution
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getResponseWrapperClass
INFO: Dynamically creating response wrapper bean Class org.gcube.vremanagement.executor.api.jaxws.GetStateEvolutionResponse
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getRequestWrapperClass
INFO: Dynamically creating request wrapper Class org.gcube.vremanagement.executor.api.jaxws.GetIterationStateEvolution
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getResponseWrapperClass
INFO: Dynamically creating response wrapper bean Class org.gcube.vremanagement.executor.api.jaxws.GetIterationStateEvolutionResponse
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getRequestWrapperClass
INFO: Dynamically creating request wrapper Class org.gcube.vremanagement.executor.api.jaxws.UnSchedule
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getResponseWrapperClass
INFO: Dynamically creating response wrapper bean Class org.gcube.vremanagement.executor.api.jaxws.UnScheduleResponse
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getRequestWrapperClass
INFO: Dynamically creating request wrapper Class org.gcube.vremanagement.executor.api.jaxws.Launch
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getResponseWrapperClass
INFO: Dynamically creating response wrapper bean Class org.gcube.vremanagement.executor.api.jaxws.LaunchResponse
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getRequestWrapperClass
INFO: Dynamically creating request wrapper Class org.gcube.vremanagement.executor.api.jaxws.GetState
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getResponseWrapperClass
INFO: Dynamically creating response wrapper bean Class org.gcube.vremanagement.executor.api.jaxws.GetStateResponse
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getRequestWrapperClass
INFO: Dynamically creating request wrapper Class org.gcube.vremanagement.executor.api.jaxws.Stop
Jan 23, 2017 4:28:12 PM com.sun.xml.ws.model.RuntimeModeler getResponseWrapperClass
INFO: Dynamically creating response wrapper bean Class org.gcube.vremanagement.executor.api.jaxws.StopResponse
Jan 23, 2017 4:28:13 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate <init>
INFO: WSSERVLET14: JAX-WS servlet initializing
[PUBLISHER] 16:28:17,706 FATAL jasper.EmbeddedServletOptions [localhost-startStop-1,<init>:658] The scratchDir you specified: /home/gcube/tomcat/work/Catalina/localhost/smart-executor is unusable.

there was a problem with some file access rights on node6, now solved.
At the moment this is the only problem in catalina.out:

...
[PUBLISHER] 18:07:28,086 ERROR startup.ContextConfig [localhost-startStop-1,processAnnotationsJar:1937] Unable to process Jar entry [javassist/util/proxy/SerializedProxy.class] from Jar [jar:file:/home/gcube/tomcat/webapps/smart-executor/WEB-INF/lib/javassist-3.19.0-GA.jar!/] for annotations

Tommaso Piccioli wrote:

there was a problem with some file access rights on node6, now solved.
At the moment this is the only problem in catalina.out:

...
[PUBLISHER] 18:07:28,086 ERROR startup.ContextConfig [localhost-startStop-1,processAnnotationsJar:1937] Unable to process Jar entry [javassist/util/proxy/SerializedProxy.class] from Jar [jar:file:/home/gcube/tomcat/webapps/smart-executor/WEB-INF/lib/javassist-3.19.0-GA.jar!/] for annotations

We have these errors since before the upgrade, see #2472

I see again the following exception in ghn.log and the container is not correctly published on the infrastructure:

Caused by: java.lang.IllegalStateException: org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
        at com.sun.xml.internal.bind.v2.util.XmlFactory.createParserFactory(XmlFactory.java:128)
        at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.getXMLReader(UnmarshallerImpl.java:139)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:157)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:162)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:171)
        at org.gcube.common.authorization.library.enpoints.AuthorizationEndpointScanner.endpoints(AuthorizationEndpointScanner.java:53)
        ... 11 more

could you upgrade the tomcat version to 7.0.72 on node6.d4science.org? Maybe the problem is due to tomcat version

Roberto Cirillo wrote:

could you upgrade the tomcat version to 7.0.72 on node6.d4science.org? Maybe the problem is due to tomcat version

I don't know of any incompatibility between java 8 and our tomcat packages.
tomcat on node6 is still running with java 7, btw:

gcube 21237 3.2 22.0 3581640 675836 ? Sl 19:03 0:35 /usr/lib/jvm/java-7-oracle/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m -server -Djava.awt.headless=true -Xms2048m -Xmx2048m -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/home/gcube/tomcat -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/home/gcube/tomcat/tmp org.apache.catalina.startup.Bootstrap start

Was the playbook run without passing the 'tomcat' tag?

Andrea Dell'Amico wrote:

Roberto Cirillo wrote:

could you upgrade the tomcat version to 7.0.72 on node6.d4science.org? Maybe the problem is due to tomcat version

I don't know of any incompatibility between java 8 and our tomcat packages.

@lucio.lelii@isti.cnr.it please, could you provide the details of incompatibility between Xerces library and tomcat/java8 ?

tomcat on node6 is still running with java 7, btw:

gcube 21237 3.2 22.0 3581640 675836 ? Sl 19:03 0:35 /usr/lib/jvm/java-7-oracle/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m -server -Djava.awt.headless=true -Xms2048m -Xmx2048m -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/home/gcube/tomcat -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/home/gcube/tomcat/tmp org.apache.catalina.startup.Bootstrap start

Was the playbook run without passing the 'tomcat' tag?

@tommaso.piccioli@isti.cnr.it please could you enable java8 on tomcat in the right way?

Now tomcat is running with java8

gcube 24074 1 16 10:02 ? 00:00:51 /usr/lib/jvm/java-8-oracle/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -XX:+UseConcMarkSweepGC -server -Djava.awt.headless=true -Xms2048m -Xmx2048m -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/home/gcube/tomcat -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/home/gcube/tomcat/tmp org.apache.catalina.startup.Bootstrap start

there is an issue with javaassist and java8 in tomcat with version 7.0.55, this is the link i have found : https://github.com/jboss-javassist/javassist/issues/29.
We have to update the tomcat to the latest version.

Lucio Lelii wrote:

there is an issue with javaassist and java8 in tomcat with version 7.0.55, this is the link i have found : https://github.com/jboss-javassist/javassist/issues/29.
We have to update the tomcat to the latest version.

Moving to the latest tomcat version is not so easy. We want to stuck with the distribution provided tomcat for a lot of reasons:

• backport of bug and security fixes
• compatibility with the apr libraries

In addition to that, the javassist exceptions were a cosmetic only problem in the past, and an upgrade to a newer version of javassist was planned. #2472

Because during the release 4.3.0 all components should be migrated to Java8, it's important to understand if the migration to the developments nodes can be considered closed (please, update the ticket) .
We are starting with the writing of the documentation (https://wiki.gcube-system.org/gcube/GCube_Java_8_migration_guide) about this activity.

This question should be asked to @tommaso.piccioli@isti.cnr.it

we have found another issue on registry service (node9-d-d4s): after the replacing of Xerces and xml-api libraries, the gCore container doesn't start with the following exception:

2017-02-16 17:20:20,037 FATAL contexts.GHNContext [Initialiser,fatal:76] [0.663s] GHNContext: gHN could not complete initialisation
java.lang.NoSuchMethodError: org.apache.xerces.impl.xs.XMLSchemaLoader.loadGrammar([Lorg/apache/xerces/xni/parser/XMLInputSource;)V
        at org.apache.xerces.jaxp.validation.XMLSchemaFactory.newSchema(Unknown Source)
        at javax.xml.validation.SchemaFactory.newSchema(SchemaFactory.java:638)
        at org.gcube.common.resources.kxml.KGCUBEResource.getSchema(KGCUBEResource.java:170)
        at org.gcube.common.resources.kxml.KGCUBEResource.load(KGCUBEResource.java:80)
        at org.gcube.common.resources.kxml.node.KGCUBEHostingNode.load(KGCUBEHostingNode.java:18)
        at org.gcube.common.core.contexts.GHNContext.loadGHNResource(GHNContext.java:405)
        at org.gcube.common.core.contexts.GHNContext.configureGHNResource(GHNContext.java:384)
        at org.gcube.common.core.contexts.GHNContext.initialise(GHNContext.java:291)
        at org.gcube.common.core.contexts.GHNContext.<clinit>(GHNContext.java:261)
        at org.gcube.common.core.contexts.GCUBEServiceContext$Initialiser.onRIDeployed(GCUBEServiceContext.java:677)
        at org.gcube.common.core.contexts.service.Consumer.onEvent(Consumer.java:30)
        at org.gcube.common.core.utils.events.GCUBEProducer$2.run(GCUBEProducer.java:176)

we are investigating on this issue

Me and @costantino.perciante@isti.cnr.it experienced the javaassit issue on a smart-executor node (node22.d4science.org a development machine).
Giving that the tomcat on the machine cannot be upgraded and we urge to use it, the only way to solve it for the moment was to use smartgears-distribution-bundle which already provide apache-tomcat-7.0.73.

Luca Frosini wrote:

Me and @costantino.perciante@isti.cnr.it experienced the javaassit issue on a smart-executor node (node22.d4science.org a development machine).
Giving that the tomcat on the machine cannot be upgraded and we urge to use it, the only way to solve it for the moment was to use smartgears-distribution-bundle which already provide apache-tomcat-7.0.73.

As was stated more than once, the problem afflitcs Ubuntu installations < 14.04. The solution is to get rid of the older Ubuntu installations.