Task #5169
closed
Prepare environment to test Generic Worker on Dataminer
100%
Description
Some machines are required to finally test the porting of GW on Dataminer.
First, the installation of Dataminer running on the latest version of the GHN should be tested, since both the GHN libraries and the DM pom have changed.
The GWs can be two machines behind a proxy. Dataminer2-d can be reused for this aim and one/two Generic Workers can be recycled.
Updated by Gianpaolo Coro over 8 years ago
I have organised the DataMiner pom in order to limit the spread of dependencies and to detect the libraries that should be deleted after the installation.
The new pom is here:
https://svn.research-infrastructures.eu/public/d4science/gcube/trunk/data-analysis/DataMiner/pom.xml
The libraries to be deleted after the installation are:
52n-wps-algorithm-3.3.1 52n-wps-algorithm-3.3.2-SNAPSHOT 52n-wps-client-lib-3.2.0 52n-wps-commons-3.3.1 52n-wps-database-3.3.1 52n-wps-io-3.3.1 52n-wps-io-impl-3.3.1 52n-wps-server-3.3.1 52n-wps-server-3.3.2-SNAPSHOT common-configuration-scanner-1.1.0-SNAPSHOT common-encryption-1.0.2-SNAPSHOT common-gcore-stubs-1.2.2-SNAPSHOT commons-beanutils-1.7.0 common-scope-1.2.1-SNAPSHOT common-scope-maps-1.0.4-SNAPSHOT jackrabbit-jcr-commons-1.4 jackrabbit-api-1.4 jaxen-1.1.1 jcl-over-slf4j-1.6.4 jcr-1.0 slf4j-api-1.7.5
The libraries to leave in the WEB-INF/lib after the installation are:
ic-client discovery-client common-gcore-resource
Finally, downloading dataminer-algorithms.jar is not necessary anymore.
Instead, downloading the files inside the files in trunk/data-analysis/DataMinerConfiguration/algorithms is still necessary.
Updated by Gianpaolo Coro over 8 years ago
The GW-Dataminer machines should be placed behind a proxy named dataminer-d-workers.d4science.org
Updated by Andrea Dell'Amico over 8 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 80
I tried using the zip file at last.
The deploy fails with the following exception:
2016-10-04 19:38:04,614 [localhost-startStop-1] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/wps]- Exception sending context initialized eve│ 19│nt to listener instance of class org.n52.security.service.config.support.SecurityConfigContextListener │no t │org.n52.security.service.config.ConfigurationException: unexpected exception occured during configuration processing:java.lang.NoClassDefFoundError: org/apache/commo│ 19│ns/beanutils/BeanPredicate │no t │ at org.n52.security.service.config.XmlSecurityConfig.init(XmlSecurityConfig.java:235) │ 19│ at org.n52.security.service.config.XmlSecurityConfig.setConfiguration(XmlSecurityConfig.java:147) │no t │ at org.n52.security.service.config.XmlSecurityConfig.<init>(XmlSecurityConfig.java:105) │ 19│ at org.n52.security.service.config.support.SecurityConfigContextListener.contextInitialized(SecurityConfigContextListener.java:95) │no t │ at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4992) │ 19│ at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5490) │no t │ at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) │ --│ at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) │-- --│ at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) │-- --│ at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649) │-- --│ at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1247) │-- │ at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1898) │ 00│ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) │26 20│ at java.util.concurrent.FutureTask.run(FutureTask.java:262) │ol -1│ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) │ 20│ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) │ol -1│ at java.lang.Thread.run(Thread.java:745) │ 20│Caused by: java.lang.NoClassDefFoundError: org/apache/commons/beanutils/BeanPredicate │g. ap│ at org.n52.security.service.config.XmlSecurityConfig.initService(XmlSecurityConfig.java:318) │va lu│ at org.n52.security.service.config.XmlSecurityConfig.initServices(XmlSecurityConfig.java:274) │pr ob│ at org.n52.security.service.config.XmlSecurityConfig.init(XmlSecurityConfig.java:226) │ 20│ ... 16 more │g. ap│Caused by: java.lang.ClassNotFoundException: org.apache.commons.beanutils.BeanPredicate │of tR│ at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720) │ 20│ at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571) │g. ap│ ... 19 more │ef .S│2016-10-04 19:38:04,616 [localhost-startStop-1] ERROR org.apache.catalina.core.StandardContext- Error listenerStart │. 20│2016-10-04 19:38:04,617 [localhost-startStop-1] ERROR org.apache.catalina.core.StandardContext- Context [/wps] startup failed due to previous errors
It seems that something is missing.
Updated by Andrea Dell'Amico over 8 years ago
Gianpaolo Coro wrote:
The GW-Dataminer machines should be placed behind a proxy named dataminer-d-workers.d4science.org
haproxy is ready.
Updated by Gianpaolo Coro over 8 years ago
Could you retry now please?
Anyway, I fear that some snapshots of the authorization libraries are not updated. Please, see if you got an error in the catalina.out when calling this URL (after installing the war):
http://dataminer2-d.d4science.org/wps/WebProcessingService?Request=GetCapabilities&Service=WPS&gcube-token=f9d49d76-cd60-48ed-9f8e-036bcc1fc045-98187548
Updated by Andrea Dell'Amico over 8 years ago
Gianpaolo Coro wrote:
Could you retry now please?
Anyway, I fear that some snapshots of the authorization libraries are not updated. Please, see if you got an error in the catalina.out when calling this URL (after installing the war):http://dataminer2-d.d4science.org/wps/WebProcessingService?Request=GetCapabilities&Service=WPS&gcube-token=f9d49d76-cd60-48ed-9f8e-036bcc1fc045-98187548
The dataminer app now starts correctly. That URL fails indeed:
10:09:22.698 [catalina-exec-5] INFO RequestValidator: accessing validator with token f9d49d76-cd60-48ed-9f8e-036bcc1fc045-98187548 10:09:22.796 [catalina-exec-5] WARN RequestValidator: rejecting call to DataMiner in invalid context /gcube/devNext/NextNext
But the token request returned with success. The command was
java TokenGenerator <TOKEN_GIVEN_BY_LUCIO> dataminer2-d-d4s.d4science.org 80 /home/gcube/.containerxml/2-container.xml /gcube/devNext /gcube/devNext/devNext
You can see the token got in the answer in /home/gcube/SmartGears/container.xml
Updated by Andrea Dell'Amico over 8 years ago
A question about the http://thredds.d4science.org/thredds/fileServer/public/netcdf/gebco_08_OCEANS_CLIMATOLOGY_METEOROLOGY_ATMOSPHERE_.nc file that we need to download at each installation.
Can it live outside tomcat/webapps/wps? if so, it should be downloaded once, and not at every upgrade.
Updated by Lucio Lelii over 8 years ago
java TokenGenerator dataminer2-d-d4s.d4science.org 80 /home/gcube/.containerxml/2-container.xml /gcube/devNext /gcube/devNext/devNext
The problem is the last context in the command, it is a not existing vre and not the one in wich gianpaolo is trying to contact the dataminer
Updated by Andrea Dell'Amico over 8 years ago
Lucio Lelii wrote:
java TokenGenerator dataminer2-d-d4s.d4science.org 80 /home/gcube/.containerxml/2-container.xml /gcube/devNext /gcube/devNext/devNext
The problem is the last context in the command, it is a not existing vre and not the one in wich gianpaolo is trying to contact the dataminer
Interesting, it was configured that way since months. Changing it right now.
Updated by Andrea Dell'Amico over 8 years ago
It seems it's working now. The correct scopes are /gcube/devNext /gcube/devNext/NextNext
Updated by Andrea Dell'Amico over 8 years ago
- Status changed from In Progress to Feedback
Updated by Gianpaolo Coro over 8 years ago
I'm calling the service with the NextNext token, but I see these errors on the GHN referring to the gCubeApps scope:
This is the test url:
http://dataminer2-d-d4s.d4science.org/wps/WebProcessingService?Request=GetCapabilities&Service=WPS&gcube-token=f9d49d76-cd60-48ed-9f8e-036bcc1fc045-98187548
10:55:48.092 [pool-1-thread-5] ERROR ProfileManager: cannot publish container (see details)
java.lang.IllegalStateException: a map for /gcube/gCubeApps is undefined
at org.gcube.common.scope.impl.ScopedServiceMap.currentMap(ScopedServiceMap.java:63) ~[common-scope-1.2.0-4.0.0-125717.jar:na]
at org.gcube.common.scope.impl.ScopedServiceMap.endpoint(ScopedServiceMap.java:44) ~[common-scope-1.2.0-4.0.0-125717.jar:na]
at org.gcube.resources.discovery.icclient.ICClient.getStub(ICClient.java:74) ~[ic-client-1.0.1-4.0.0-126578.jar:na]
at org.gcube.resources.discovery.icclient.ICClient.submit(ICClient.java:39) ~[ic-client-1.0.1-4.0.0-126578.jar:na]
at org.gcube.resources.discovery.client.impl.DelegateClient.submit(DelegateClient.java:50) ~[discovery-client-1.0.1-4.0.0-125857.jar:na]
at org.gcube.informationsystem.publisher.utils.RegistryStubs.getEndPoints(RegistryStubs.java:49) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.informationsystem.publisher.utils.RegistryStubs.getStubs(RegistryStubs.java:62) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.informationsystem.publisher.RegistryPublisherImpl.registryUpdate(RegistryPublisherImpl.java:180) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.informationsystem.publisher.RegistryPublisherImpl.update(RegistryPublisherImpl.java:128) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.informationsystem.publisher.ScopedPublisherImpl.update(ScopedPublisherImpl.java:54) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfilePublisher.update(ProfilePublisher.java:81) ~[common-smartgears-1.2.7-4.0.0-128702.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager.publish(ProfileManager.java:224) [common-smartgears-1.2.7-4.0.0-128702.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager.access$300(ProfileManager.java:50) [common-smartgears-1.2.7-4.0.0-128702.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager$1.publishAfterChange(ProfileManager.java:122) [common-smartgears-1.2.7-4.0.0-128702.jar:na]
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
at org.gcube.common.events.impl.Observer.onEventImmediate(Observer.java:99) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.Observer.onEvent(Observer.java:93) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.DefaultHub.notifyObservers(DefaultHub.java:171) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.DefaultHub.fire(DefaultHub.java:93) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager$1.onChanged(ProfileManager.java:97) [common-smartgears-1.2.7-4.0.0-128702.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_80]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_80]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
at org.gcube.common.events.impl.Observer.onEventImmediate(Observer.java:99) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.Observer.onEvent(Observer.java:93) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.DefaultHub$1.run(DefaultHub.java:194) [common-events-1.0.1-4.0.0-126364.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
10:55:48.094 [localhost-startStop-2] WARN BasicUsageRecord: Unable to automaticcally set the scope using scope provider. The record will not be valid if the scope will not be explicitly set.
10:55:48.102 [pool-1-thread-5] ERROR ProfileManager: cannot publish container (see details)
java.lang.IllegalStateException: a map for /gcube/gCubeApps is undefined
at org.gcube.common.scope.impl.ScopedServiceMap.currentMap(ScopedServiceMap.java:63) ~[common-scope-1.2.0-4.0.0-125717.jar:na]
at org.gcube.common.scope.impl.ScopedServiceMap.endpoint(ScopedServiceMap.java:44) ~[common-scope-1.2.0-4.0.0-125717.jar:na]
at org.gcube.resources.discovery.icclient.ICClient.getStub(ICClient.java:74) ~[ic-client-1.0.1-4.0.0-126578.jar:na]
at org.gcube.resources.discovery.icclient.ICClient.submit(ICClient.java:39) ~[ic-client-1.0.1-4.0.0-126578.jar:na]
at org.gcube.resources.discovery.client.impl.DelegateClient.submit(DelegateClient.java:50) ~[discovery-client-1.0.1-4.0.0-125857.jar:na]
at org.gcube.informationsystem.publisher.utils.RegistryStubs.getEndPoints(RegistryStubs.java:49) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.informationsystem.publisher.utils.RegistryStubs.getStubs(RegistryStubs.java:62) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.informationsystem.publisher.RegistryPublisherImpl.registryUpdate(RegistryPublisherImpl.java:180) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.informationsystem.publisher.RegistryPublisherImpl.update(RegistryPublisherImpl.java:128) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.informationsystem.publisher.ScopedPublisherImpl.update(ScopedPublisherImpl.java:54) ~[registry-publisher-1.2.6-4.0.0-126697.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfilePublisher.update(ProfilePublisher.java:81) ~[common-smartgears-1.2.7-4.0.0-128702.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager.publish(ProfileManager.java:224) [common-smartgears-1.2.7-4.0.0-128702.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager.access$300(ProfileManager.java:50) [common-smartgears-1.2.7-4.0.0-128702.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager$1.publishAfterChange(ProfileManager.java:122) [common-smartgears-1.2.7-4.0.0-128702.jar:na]
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
at org.gcube.common.events.impl.Observer.onEventImmediate(Observer.java:99) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.Observer.onEvent(Observer.java:93) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.DefaultHub.notifyObservers(DefaultHub.java:171) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.DefaultHub.fire(DefaultHub.java:93) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.smartgears.handlers.container.lifecycle.ProfileManager$1.onChanged(ProfileManager.java:97) [common-smartgears-1.2.7-4.0.0-128702.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_80]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_80]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
at org.gcube.common.events.impl.Observer.onEventImmediate(Observer.java:99) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.Observer.onEvent(Observer.java:93) [common-events-1.0.1-4.0.0-126364.jar:na]
at org.gcube.common.events.impl.DefaultHub$1.run(DefaultHub.java:194) [common-events-1.0.1-4.0.0-126364.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
Updated by Gianpaolo Coro over 8 years ago
The issue was due to a common-scope library left in the web application. The newly installed service works. All the regression tests were passed.
I'm going to test the cloud computations using one machine as orchestrator and another one as worker.
Updated by Gianpaolo Coro over 8 years ago
- Status changed from Feedback to Closed
- % Done changed from 80 to 100
Tests have been done with Cloud computing algorithms using two DataMiners as Worker nodes. Everything works fine with medium-time-running computations.
The production environment or the EGI infrastructure will give the occasion to test larger jobs.
The execution time reduction is more than 85%, but we are doing more precise measurements.
