D4Science Infrastructure

I just learned that Ubuntu trusty can install an R version >= 3.0. Reinstalling as Ubuntu 12.04.

The dev machine is running, R and the dataminer webapp are installed. @gianpaolo.coro@isti.cnr.it is testing the installation.

The pom dependencies of DataMiner are not in the wps application folder. E.g. the following call fails because it cannot find a class in the ecological-engine jar:

http://dataminer2-d-d4s.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=4ccc2c35-60c9-4c9b-9800-616538d5d48b&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN&DataInputs=OccurrencePointsClusterLabel=OccClustersTest;epsilon=10;min_points=1;OccurrencePointsTable=http://goo.gl/VDzpch;FeaturesColumnNames=depthmean|sstmnmax|salinitymean;

Only the main DataMiner jar is there.

I see. I used maven to install that jar, with the command

mvn org.apache.maven.plugins:maven-dependency-plugin:2.8:copy -Dartifact=org.gcube.dataanalysis:dataminer:1.0.0-SNAPSHOT -DoutputDirectory=/home/gcube/tomcat/webapps/wps/WEB-INF/lib

There are a lot of jars under the maven repository directory, but it seems stuff related to maven itself:

/home/gcube/.m2/repository/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.jar
/home/gcube/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.0/maven-reporting-api-3.0.jar
/home/gcube/.m2/repository/org/apache/maven/reporting/maven-reporting-impl/2.0.5/maven-reporting-impl-2.0.5.jar
/home/gcube/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.0/doxia-sink-api-1.0.jar
/home/gcube/.m2/repository/org/apache/maven/doxia/doxia-core/1.0/doxia-core-1.0.jar
/home/gcube/.m2/repository/org/apache/maven/doxia/doxia-site-renderer/1.0/doxia-site-renderer-1.0.jar
/home/gcube/.m2/repository/org/apache/maven/doxia/doxia-decoration-model/1.0/doxia-decoration-model-1.0.jar
/home/gcube/.m2/repository/org/apache/maven/doxia/doxia-module-apt/1.0/doxia-module-apt-1.0.jar
/home/gcube/.m2/repository/org/apache/maven/doxia/doxia-module-fml/1.0/doxia-module-fml-1.0.jar
/home/gcube/.m2/repository/org/apache/maven/doxia/doxia-module-xdoc/1.0/doxia-module-xdoc-1.0.jar
/home/gcube/.m2/repository/org/apache/maven/doxia/doxia-module-xhtml/1.0/doxia-module-xhtml-1.0.jar
/home/gcube/.m2/repository/org/apache/maven/shared/maven-doxia-tools/1.0.2/maven-doxia-tools-1.0.2.jar
/home/gcube/.m2/repository/org/apache/maven/shared/file-management/1.2.1/file-management-1.2.1.jar
/home/gcube/.m2/repository/org/apache/maven/shared/maven-shared-io/1.1/maven-shared-io-1.1.jar
/home/gcube/.m2/repository/org/apache/maven/shared/maven-dependency-analyzer/1.4/maven-dependency-analyzer-1.4.jar
/home/gcube/.m2/repository/org/apache/maven/shared/maven-dependency-tree/2.1/maven-dependency-tree-2.1.jar
/home/gcube/.m2/repository/org/apache/maven/shared/maven-common-artifact-filters/1.4/maven-common-artifact-filters-1.4.jar
/home/gcube/.m2/repository/org/apache/maven/shared/maven-invoker/2.0.11/maven-invoker-2.0.11.jar
/home/gcube/.m2/repository/org/apache/velocity/velocity/1.5/velocity-1.5.jar
/home/gcube/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.9/plexus-utils-3.0.9.jar
/home/gcube/.m2/repository/org/codehaus/plexus/plexus-interactivity-api/1.0-alpha-4/plexus-interactivity-api-1.0-alpha-4.jar
/home/gcube/.m2/repository/org/codehaus/plexus/plexus-i18n/1.0-beta-7/plexus-i18n-1.0-beta-7.jar
/home/gcube/.m2/repository/org/codehaus/plexus/plexus-velocity/1.1.7/plexus-velocity-1.1.7.jar
/home/gcube/.m2/repository/org/codehaus/plexus/plexus-archiver/2.3/plexus-archiver-2.3.jar
/home/gcube/.m2/repository/org/codehaus/plexus/plexus-io/2.0.6/plexus-io-2.0.6.jar
/home/gcube/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar
/home/gcube/.m2/repository/org/eclipse/aether/aether-util/0.9.0.M2/aether-util-0.9.0.M2.jar
/home/gcube/.m2/repository/org/gcube/dataanalysis/dataminer/1.0.0-SNAPSHOT/dataminer-1.0.0-20150923.103012-4.jar
/home/gcube/.m2/repository/org/gcube/dataanalysis/dataminer/1.0.0-SNAPSHOT/dataminer-1.0.0-SNAPSHOT.jar
/home/gcube/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar
/home/gcube/.m2/repository/commons-cli/commons-cli/1.0/commons-cli-1.0.jar
/home/gcube/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
/home/gcube/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
/home/gcube/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar
/home/gcube/.m2/repository/commons-io/commons-io/1.4/commons-io-1.4.jar
/home/gcube/.m2/repository/commons-validator/commons-validator/1.2.0/commons-validator-1.2.0.jar
/home/gcube/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar
/home/gcube/.m2/repository/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar
/home/gcube/.m2/repository/commons-digester/commons-digester/1.6/commons-digester-1.6.jar
/home/gcube/.m2/repository/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar
/home/gcube/.m2/repository/asm/asm/3.3.1/asm-3.3.1.jar
/home/gcube/.m2/repository/rapidminer-custom/52n-wps-server/3.3.2/52n-wps-server-3.3.2.jar

Please try using "dependency:copy-dependencies" as maven goal

There is still something to adjust, which prevents the services to work.
From the guide, you should download also the following dependency (please confirm this is included in your installation script):

 <dependency>
   <groupId>rapidminer-custom</groupId>
   <artifactId>52n-wps-server</artifactId>
   <version>3.3.2</version>
 </dependency>

and substitute it to the corresponding one in the war.
Furthermore, I see both 3.3.1 and 3.3.2 52 north libraries in the WEB-INF/lib directory, e.g.
52n-wps-server-3.3.1.jar
52n-wps-server-3.3.2.jar

Those of the original war should be left and only the one above should be overwritten.

Gianpaolo Coro wrote:

There is still something to adjust, which prevents the services to work.
From the guide, you should download also the following dependency (please confirm this is included in your installation script):

 <dependency>
   <groupId>rapidminer-custom</groupId>
   <artifactId>52n-wps-server</artifactId>
   <version>3.3.2</version>
 </dependency>

Yes.

and substitute it to the corresponding one in the war.
Furthermore, I see both 3.3.1 and 3.3.2 52 north libraries in the WEB-INF/lib directory, e.g.
52n-wps-server-3.3.1.jar
52n-wps-server-3.3.2.jar

Those of the original war should be left and only the one above should be overwritten.

There are a lot of jar files from the 52n 3.3.1 distribution that are downloaded as org.gcube.dataanalysis dataminer dependencies. Until the dependency mess is solved I'm adding a task that removes the following files:

52n-wps-io-impl-3.3.1.jar
52n-wps-database-3.3.1.jar
52n-wps-server-3.3.1.jar
52n-wps-algorithm-3.3.1.jar
52n-wps-commons-3.3.1.jar
52n-wps-io-3.3.1.jar

There are still a number of problems:
1 - Two files are missing, which enable the web application as a gCube web application. I missed to report them in the Wiki but I have just updated the page as follows:

Download the following xml files and copy them into the web application WEB-INF folder, to make the application be enabled on a gCube container:
http://data.d4science.org/id?fileName=web.xml&smp-id=56615ae0e4b0158fcb561817&contentType=application%2Fxml
http://data.d4science.org/id?fileName=gcube-app.xml&smp-id=56615ae0e4b0158fcb561815&contentType=application%2Fxml

2 - I see some issues with the libraries in the WEB-INF/lib folder:
(i) snapshot libraries are present as well as 3.8.0 libraries, thus they are not at the latest version (3.9.0). Furthermore, snapshots should not appear at all.
(ii) the following libraries are present along with a newer version in the lib folder and should be deleted:

commons-beanutils-1.7.0.jar
dom4j-1.6.jar
jaxen-1.1.1.jar
jcl-over-slf4j-1.6.4.jar
jdom-1.0.jar
logback-classic-1.0.13.jar
logback-core-1.0.13.jar
slf4j-api-1.7.5.jar
spring-beans-2.0.6.jar
spring-context-2.0.6.jar
spring-core-2.0.6.jar

I think that the points above depend on the settings.xml, in which only the gcube-releases and the gcube-externals repository should have the flag "enabled" set to true.
@andrea.dellamico@isti.cnr.it could you please verify this configuration file and repeat the downloading from Maven?

Gianpaolo Coro wrote:

There are still a number of problems:
1 - Two files are missing, which enable the web application as a gCube web application. I missed to report them in the Wiki but I have just updated the page as follows:

Download the following xml files and copy them into the web application WEB-INF folder, to make the application be enabled on a gCube container:
http://data.d4science.org/id?fileName=web.xml&smp-id=56615ae0e4b0158fcb561817&contentType=application%2Fxml
http://data.d4science.org/id?fileName=gcube-app.xml&smp-id=56615ae0e4b0158fcb561815&contentType=application%2Fxml

OK

2 - I see some issues with the libraries in the WEB-INF/lib folder:
(i) snapshot libraries are present as well as 3.8.0 libraries, thus they are not at the latest version (3.9.0). Furthermore, snapshots should not appear at all.
(ii) the following libraries are present along with a newer version in the lib folder and should be deleted:

commons-beanutils-1.7.0.jar
dom4j-1.6.jar
jaxen-1.1.1.jar
jcl-over-slf4j-1.6.4.jar
jdom-1.0.jar
logback-classic-1.0.13.jar
logback-core-1.0.13.jar
slf4j-api-1.7.5.jar
spring-beans-2.0.6.jar
spring-context-2.0.6.jar
spring-core-2.0.6.jar

I think that the points above depend on the settings.xml, in which only the gcube-releases and the gcube-externals repository should have the flag "enabled" set to true.
@andrea.dellamico@isti.cnr.it could you please verify this configuration file and repeat the downloading from Maven?

OK, so what I need to do is to completely disable the gcube-snapshots repository, while the others remain configured as follows:

                                        <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                    <releases>
                        <enabled>true</enabled>

Right?

Yes, but both in the repositories and the plugins sections.

Gianpaolo Coro wrote:

Yes, but both in the repositories and the plugins sections.

Of course.

Done, and run the playbook again, tomcat restarted.
There are some SNAPSHOT named jars files:

52n-wps-algorithm-3.3.2-SNAPSHOT.jar
52n-wps-algorithm-impl-3.3.2-SNAPSHOT.jar
52n-wps-client-lib-3.3.2-SNAPSHOT.jar
52n-wps-commons-3.3.2-SNAPSHOT.jar
52n-wps-database-3.3.2-SNAPSHOT.jar
52n-wps-io-3.3.2-SNAPSHOT.jar
52n-wps-io-impl-3.3.2-SNAPSHOT.jar
52n-wps-mc-3.3.2-SNAPSHOT.jar
52n-wps-webadmin-3.3.2-SNAPSHOT.jar
commons-vfs2-2.1-SNAPSHOT.jar
dataminer-1.0.0-SNAPSHOT.jar

All but common-vfs2 and dataminer come from the 52 north webapp archive.

Now the libs and the configuration seem fine. Just logback-classic-1.0.13.jar remains as an old lib and should be deleted.
Unfortunately, the scope passed to the application is still null. I'm using the following request and token:

http://dataminer2-d-d4s.d4science.org/wps/WebProcessingService?Request=GetCapabilities&Service=WPS&gcube-token=d7a4076c-e8c1-42fe-81e0-bdecb1e8074a

We need @lucio.lelii@isti.cnr.it help here to check the WHN configuration on that machine.

Gianpaolo Coro wrote:

Now the libs and the configuration seem fine. Just logback-classic-1.0.13.jar remains as an old lib and should be deleted.

Cleaned up this one too.

The problem is that the authorization libraries cannot be in the webapp folder too.
@andrea.dellamico@isti.cnr.it could you create a rule to delete the following libraries from the WEB-INF/lib?

common-scope-1.2.0-3.9.0.jar
common-authorization-1.0.0-3.9.0.jar

Currently, they have been deleted by hand just to test.

In order to properly work, the rule should delete all the common-scope-x and common-authorization-x libraries, and generally all the libraries already present in the SmartGears distribution. Note that these are declared as "provided" in the dataminer pom. I don't know why they are in the WEB-INF/lib then.

The following are the dependencies of SmartGears which should not go in the webapp folder (taken from https://svn.d4science-ii.research-infrastructures.eu/gcube/branches/distributions/maven-smartgears-bom/1.0/pom.xml)

<dependencies>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-smartgears</artifactId>
                <version>[1.0.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-smartgears-app</artifactId>
                <version>[1.0.0-SNAPSHOT,)</version>
                <scope>compile</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-encryption</artifactId>
                <version>[1.0.2-SNAPSHOT, 2.0.0-SNAPSHOT)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-events</artifactId>
                <version>[1.0.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-configuration-scanner</artifactId>
                <version>[1.0.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.resources</groupId>
                <artifactId>common-gcore-resources</artifactId>
                <version>[1.0.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.resources</groupId>
                <artifactId>registry-publisher</artifactId>
                <version>[1.0.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-gcore-stubs</artifactId>
                <version>[1.1.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-scope</artifactId>
                <version>[1.2.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.resources.discovery</groupId>
                <artifactId>ic-client</artifactId>
                <version>[1.0.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.resources.discovery</groupId>
                <artifactId>discovery-client</artifactId>
                <version>[1.0.0-SNAPSHOT,)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-validator</artifactId>
                <version>[1.0.0-SNAPSHOT,2.0.0-SNAPSHOT)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
                <version>1.7.5</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.core</groupId>
                <artifactId>common-validator</artifactId>
                <version>[1.0.0-SNAPSHOT,2.0.0-SNAPSHOT)</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.gcube.common</groupId>
                <artifactId>authorization-client</artifactId>
                <version>[1.0.0-SNAPSHOT,2.0.0-SNAPSHOT)</version>
                <scope>provided</scope>
            </dependency>

            <dependency>
                <groupId>org.gcube.common</groupId>
                <artifactId>common-authorization</artifactId>
                <version>[1.0.0-SNAPSHOT,2.0.0-SNAPSHOT)</version>
                <scope>provided</scope>
            </dependency>

            <dependency>
                <groupId>org.gcube.accounting</groupId>
                <artifactId>accounting-lib</artifactId>
                <version>[1.0.0-SNAPSHOT,2.0.0-SNAPSHOT)</version>
                <scope>provided</scope>
            </dependency>

        </dependencies>

This is getting crazy.
I'm going to remove all the webapps and reinstall, to see if it's a conseguence of the old wrong maven settings. Otherwise I'll find and delete all the duplicate jars, but we need a reliable way to install only the needed jars. I cannot (read "do not want") manually check a list of files at every release, only because we don't know what we are going to install.

the smartest thing to do is to include a dependency to the smartgear-bom in the webapp.

Lucio Lelii wrote:

the smartest thing to do is to include a dependency to the smartgear-bom in the webapp.

Is it possibile to solve this before going to production (and generating an EGI image for the dataminer?)

I've reinstalled the webapps, in the meantime. I still had to remove the pom dependencies. The complete list of duplicates from the 3.9.0 release is:

  - accounting-lib-1.1.0-3.9.0.jar
  - authorization-client-1.0.0-3.9.0.jar
  - common-authorization-1.0.0-3.9.0.jar
  - common-clients-2.1.0-3.9.0.jar
  - common-configuration-scanner-1.0.0-3.9.0.jar
  - common-encryption-1.0.2-3.9.0.jar
  - common-gcore-resources-1.3.1-3.9.0.jar
  - common-gcore-stubs-1.2.1-3.9.0.jar
  - common-generic-clients-1.0.0-3.9.0.jar
  - common-scope-maps-1.0.2-3.9.0.jar
  - common-validator-1.0.0-3.9.0.jar
  - discovery-client-1.0.1-3.9.0.jar
  - ic-client-1.0.1-3.9.0.jar

In your list common-scope-1.2.0-3.9.0.jar is missing.

Lucio Lelii wrote:

In your list common-scope-1.2.0-3.9.0.jar is missing.

That file isn't in wps/WEB-INF/lib. Or did anyone remove it just after I reinstalled?

In the first installation it was there and I have removed it. I don't know why in this new installation it is disappeared.

I tried again and it was there. I now removed only the two jars

common-scope-1.2.0-3.9.0.jar
common-authorization-1.0.0-3.9.0.jar

Can you check if it works now?

There is still something wrong in the installation. I find inconsistent libraries both in the tomcat/lib and in the WEB-INF/lib that prevent the application to run (e.g. not updated slf4j and logback in tomcat libs, not updated libraries in the webapp).
This is the list of libraries in the webapp that make the application work (tested both on dataminer1 and 2):

52n-security-authentication-2.2-M2.jar
52n-security-authn-web-2.2-M2.jar
52n-security-config-2.2-M2.jar
52n-security-core-2.2-M2.jar
52n-security-decision-2.2-M2.jar
52n-security-enforcement-2.2-M2.jar
52n-security-licensing-2.2-M2.jar
52n-security-precondition-2.2-M2.jar
52n-security-support-net-2.2-M2.jar
52n-security-xml-schema-2.1.0.jar
52n-security-xml-secconfig-2.1.0.jar
52n-wps-algorithm-3.3.2-SNAPSHOT.jar
52n-wps-algorithm-impl-3.3.2-SNAPSHOT.jar
52n-wps-client-lib-3.2.0.jar
52n-wps-client-lib-3.3.2-SNAPSHOT.jar
52n-wps-commons-3.3.2-SNAPSHOT.jar
52n-wps-config-1.2.1.jar
52n-wps-database-3.3.2-SNAPSHOT.jar
52n-wps-io-3.3.2-SNAPSHOT.jar
52n-wps-io-impl-3.3.2-SNAPSHOT.jar
52n-wps-mc-3.3.2-SNAPSHOT.jar
52n-wps-server-3.3.2.jar
52n-wps-webadmin-3.3.2-SNAPSHOT.jar
52n-xml-ows-v110-1.1.0.jar
52n-xml-wps-v100-1.1.0.jar
52n-xml-xlink-v110-1.0.0.jar
abdera-bundle-1.1.2.jar
accounting-lib-1.1.0-3.9.0.jar
activation-1.1.jar
activemq-core-5.6.0.jar
activemq-protobuf-1.1.jar
agent-library-2.0.0-3.9.0.jar
ant-1.6.5.jar
antlr-2.7.7.jar
antlr-runtime-3.4.jar
aopalliance-1.0.jar
asm-3.3.1.jar
axiom-api-1.2.10.jar
axiom-impl-1.2.10.jar
bcprov-jdk15-124.jar
bounce-0.14.0.jar
c3p0-0.9.1.2.jar
cglib-2.2.2.jar
common-clients-2.1.0-3.9.0.jar
common-configuration-scanner-1.0.0-3.9.0.jar
common-encryption-1.0.2-3.9.0.jar
common-fw-clients-1.1.0-3.9.0.jar
common-gcore-clients-2.2.0-3.9.0.jar
common-gcore-resources-1.3.1-3.9.0.jar
common-gcore-stubs-1.2.1-3.9.0.jar
common-gcube-calls-1.0.1-3.9.0.jar
common-generic-clients-1.0.0-3.9.0.jar
common-jaxws-calls-1.0.1-3.9.0.jar
commons-beanutils-1.8.3.jar
commons-beanutils-core-1.8.0.jar
commons-codec-1.5.jar
commons-collections-3.2.jar
commons-configuration-1.6.jar
common-scope-maps-1.0.2-3.9.0.jar
commons-digester-1.8.jar
commons-exec-1.3.jar
commons-httpclient-3.1.jar
commons-httpclient-contrib-ssl-3.1.jar
commons-io-2.0.jar
commons-lang-2.5.jar
commons-lang3-3.3.2.jar
commons-logging-1.0.4.jar
commons-net-3.1.jar
commons-pool-1.5.4.jar
commons-vfs2-2.1-SNAPSHOT.jar
common-uri-1.0.1-3.9.0.jar
common-validator-1.0.0-3.9.0.jar
concurrent-1.3.4.jar
cors-filter-1.9.3.jar
dataminer-1.0.0-SNAPSHOT.jar
derby-10.2.2.0.jar
discovery-client-1.0.1-3.9.0.jar
dom-2.3.0-jaxb-1.0.6.jar
dom4j-1.6.1.jar
ecological-engine-1.8.6-3.9.0.jar
ecological-engine-external-algorithms-1.1.7-3.9.0.jar
ecological-engine-geospatial-extensions-1.3.5-3.9.0.jar
ecological-engine-smart-executor-1.1.1-3.9.0.jar
ecological-engine-wps-extension-1.0.4-3.9.0.jar
ehcache-core-2.5.1.jar
executor-service-1.2.2-3.6.0.jar
executor-stubs-1.2.2-3.6.0.jar
ezmorph-1.0.4.jar
fusemq-leveldb-1.1.jar
gcf-1.6.3-3.9.0.jar
gdal-1.0.0.jar
geoapi-pending-3.1-M04.jar
geonetwork-2.1.1-3.9.0.jar
geonetwork-manager-1.1.jar
geopeo-1.0.2.jar
geoserver-manager-1.5.2.jar
geotk-analytics-3.20.jar
geotk-coverage-3.20.jar
geotk-coverageio-3.20.jar
geotk-coverageio-store-3.20.jar
geotk-coverage-sql-3.20.jar
geotk-datastore-core-3.20.jar
geotk-display-3.20.jar
geotk-engine-core-3.20.jar
geotk-feature-3.20.jar
geotk-filter-3.20.jar
geotk-filter-cql-3.20.jar
geotk-geometry-3.20.jar
geotk-go2-3.20.jar
geotk-go2-style-extension-3.20.jar
geotk-index-tree-3.20.jar
geotk-jtswrapper-3.20.jar
geotk-metadata-3.20.jar
geotk-metadata-sql-3.20.jar
geotk-processing-core-3.20.jar
geotk-processing-coverage-3.20.jar
geotk-referencing-3.20.jar
geotk-render-3.20.jar
geotk-storage-3.20.jar
geotk-style-3.20.jar
geotk-temporal-3.20.jar
geotk-utility-3.20.jar
geotk-utility-pending-3.20.jar
geotk-xml-base-3.20.jar
geotk-xml-gml-3.20.jar
geotk-xml-ogc-3.20.jar
geotk-xml-ows-3.20.jar
geotk-xml-se-3.20.jar
geotk-xml-service-3.20.jar
geotk-xml-sld-3.20.jar
geotk-xml-wms-3.20.jar
geronimo-activation_1.1_spec-1.0.2.jar
geronimo-j2ee-management_1.1_spec-1.0.1.jar
geronimo-javamail_1.4_spec-1.6.jar
geronimo-jms_1.1_spec-1.1.1.jar
geronimo-stax-api_1.0_spec-1.0.1.jar
ghn-core-runtime-1.0.0.jar
gis-interface-2.1.2-3.9.0.jar
gmlpacket-2.0-0.4.jar
graphlayout-1.2.1.jar
grs2library-2.1.2-3.9.0.jar
gsay-service-1.0.0.jar
gson-1.7.1.jar
guava-13.0.1.jar
guava-annotations-r03.jar
guava-base-r03.jar
guava-collections-r03.jar
guava-primitives-r03.jar
guice-3.0.jar
guice-multibindings-3.0.jar
hadoop-core-1.0.0.jar
hamcrest-core-1.3.jar
hawtbuf-1.9.jar
hawtbuf-proto-1.9.jar
hawtdispatch-1.9.jar
hawtdispatch-scala-1.9.jar
hawtdispatch-transport-1.9.jar
hawtjni-runtime-1.5.jar
hibernate3-3.0.0.jar
hibernate-spatial-1.0-M2.jar
hibernate-spatial-postgis-1.0-M2.jar
htmlparser-1.0.5.jar
httpclient-4.2.1.jar
httpcore-4.2.1.jar
ibatis-sqlmap-2.3.0.jar
ic-client-1.0.1-3.9.0.jar
is-client-1.5.1-3.6.0.jar
is-collector-stubs-3.0.0-3.6.0.jar
jackrabbit-api-1.4.jar
jackrabbit-core-1.4.jar
jackrabbit-jcr-commons-1.4.jar
jackrabbit-spi-1.4.jar
jackrabbit-spi-commons-1.4.jar
jackrabbit-text-extractors-1.4.jar
jackrabbit-webdav-2.3.3.jar
jackson-annotations-2.3.3.jar
jackson-core-2.3.3.jar
jackson-core-asl-1.9.2.jar
jackson-databind-2.3.3.jar
jackson-mapper-asl-1.9.2.jar
jahmm-0.6.1.jar
jai_imageio-1.1.jar
jama-1.0.2.jar
jasypt-1.8.jar
java-cup-0.11a.jar
java-property-utils-1.9.1.jar
javax.inject-1.jar
jaxen-1.1.4.jar
jaxen-limited-1.1.2.jar
jcip-annotations-1.0.jar
jcl-over-slf4j-1.7.5.jar
jcommon-1.0.16.jar
jcr-1.0.jar
jdom-1.1.jar
jetty-6.1.26.jar
jetty-util-6.1.26.jar
jfreechart-1.0.13.jar
jgrapht-0.8.2.jar
jgrapht-jdk1.6-0.8.2.jar
joda-time-2.2.jar
json-lib-2.2.2-jdk15.jar
jsp-2.1-6.1.14.jar
jsp-api-2.1-6.1.14.jar
jsr-275-0.9.3.jar
jstl-1.2.jar
jta-1.1.jar
jts-1.12.jar
jug-2.0.0.jar
jul-to-slf4j-1.7.5.jar
junit-4.11.jar
kahadb-5.6.0.jar
kxml2-2.3.0.jar
leveldb-0.2.jar
leveldb-api-0.2.jar
leveldbjni-1.2.jar
leveldbjni-linux32-1.2.jar
leveldbjni-linux64-1.2.jar
leveldbjni-osx-1.2.jar
log4j-1.2.16.jar
log4j-over-slf4j-1.7.5.jar
logback-classic-1.1.2.jar
logback-core-1.1.2.jar
lucene-core-2.2.0.jar
madgikcommonslibrary-1.5.0-3.9.0.jar
mail-1.4.jar
mapscript-6.0.3.jar
marytts-d4science-5.0.0.jar
maxent-princeton-3.3.3.jar
mc-runtime-1.1-20150212.235551-8.jar
mc-schema-1.1-20150212.235552-13.jar
mongo-java-driver-3.0.4.jar
mqtt-client-1.0.jar
netcdf-4.2.0.jar
objenesis-1.2.jar
ogckml2.2-1.0.0.jar
opendap-2.2.0.jar
org.osgi.core-4.1.0.jar
pentaho-package-manager-1.0.0.jar
pjl-comp-filter-1.6.4.jar
postgis-2.0.0SVN.jar
postgis-driver-1.0.jar
postgresql-8.4-702.jdbc4.jar
quartz-2.2.1.jar
rapidMiner-1.0.0.jar
rapidminer-plugintext-1.0.0.jar
registry-publisher-1.2.5-3.9.0.jar
REngine-0.6-8.jar
RserveEngine-0.6-8.jar
scala-library-2.9.1.jar
serializer-2.7.1.jar
servlet-api-2.5-20081211.jar
servlet-api-2.5-6.1.14.jar
slf4j-api-1.7.6.jar
slf4j-log4j12-1.7.10.jar
smart-executor-api-1.2.0-3.9.0.jar
smart-executor-client-1.1.0-3.9.0.jar
snappy-java-1.0.3.jar
spd-client-library-3.1.1-3.9.0.jar
spd-model-3.0.1-3.9.0.jar
spring-aop-3.0.6.RELEASE.jar
spring-asm-3.0.6.RELEASE.jar
spring-beans-3.0.6.RELEASE.jar
spring-context-3.0.6.RELEASE.jar
spring-core-3.0.6.RELEASE.jar
spring-expression-3.0.6.RELEASE.jar
spring-web-2.0.6.jar
stax2-api-3.0.1.jar
stax-utils-20060502.jar
storage-manager-core-2.3.0-3.9.0.jar
storage-manager-wrapper-2.2.1-3.9.0.jar
streams-2.0.2-3.9.0.jar
stringtemplate-3.2.1.jar
taxamatchpostgres-1.2.0.jar
transfer-library-1.0.0-3.9.0.jar
trees-1.4.2-3.9.0.jar
vecmath-1.5.2.jar
weka-dev-3.7.6.jar
woodstox-core-asl-4.0.6.jar
xalan-2.7.1.jar
xercesImpl-2.7.1.jar
XMeans-1.0.3.jar
xml-apis-1.0.b2.jar
xmlbeans-2.4.0.jar
xmlpull-1.1.3.1.jar
xpp3_min-1.1.4c.jar
xstream-1.4.6.jar

This is the list of libraries in the tomcat/lib when the application works:

accounting-lib-1.1.0-3.9.0.jar
accounting-lib-persistence-couchdb-1.0.0-3.9.0.jar
authorization-client-1.0.0-3.9.0.jar
common-authorization-1.0.0-3.9.0.jar
common-clients-2.1.0-3.9.0.jar
common-configuration-scanner-1.0.0-3.9.0.jar
common-encryption-1.0.2-3.9.0.jar
common-events-1.0.1-3.9.0.jar
common-gcore-resources-1.3.1-3.9.0.jar
common-gcore-stubs-1.2.1-3.9.0.jar
common-generic-clients-1.0.0-3.9.0.jar
common-scope-1.2.0-3.9.0.jar
common-scope-maps-1.0.2-3.9.0.jar
commons-io-2.4.jar
common-smartgears-1.2.4-3.9.0.jar
common-smartgears-utils-1.0.1-3.9.0.jar
common-validator-1.0.0-3.9.0.jar
couchdb-connector-1.0.0-3.9.0.jar
discovery-client-1.0.1-3.9.0.jar
ic-client-1.0.1-3.9.0.jar
jackson-core-asl-1.9.13.jar
jackson-mapper-asl-1.9.13.jar
logback-classic-1.1.2.jar
logback-core-1.1.2.jar
registry-publisher-1.2.5-3.9.0.jar
slf4j-api-1.7.10.jar
slf4j-log4j12-1.7.10.jar
slf4j-nop-1.7.10.jar

We should understand together why the automatic installation is not creating the above set of libraries, by precisely analyzing the generation steps. In particular, let's check the versions of slf4j-api, sl4j-log4j, sl4j-nop, ic-client, is-client, is-collector-stubs, logback-classic and logback-core

The libraries you did find installed in /home/gcube/tomcat/lib are the ones installed by the latest smartgears installation. You can find the same files in all the smart executor production nodes, and I don't want to add mess to an already messy situation manually changing them.

On the other side, we have a pom definition that downloads way too many files.

A clean solution would be having maven download and install a set of dependencies that do not conflict with the smartgears libraries, as I hope that those libraries are external to the smartgears war file because they can be used by more apps than smartgears itself.
Otherwise it should be better put those jars inside the smartgears war and exploit the native tomcat class isolation functionality to avoid conflicts.

By the way, before changing the pom that installs the dataminer dependencies, the application worked without having to touch the jars inside /home/gcube/tomcat/lib, so it can work with older versions of logback and slf4j.

Could you try using the following project, containing an updated pom (which should select specific slf4j and logback), for the copy-dependency?

https://svn.d4science.research-infrastructures.eu/gcube/trunk/data-analysis/DataMiner

Let's rely on this also for further modifications.

Gianpaolo Coro wrote:

Could you try using the following project, containing an updated pom (which should select specific slf4j and logback), for the copy-dependency?

https://svn.d4science.research-infrastructures.eu/gcube/trunk/data-analysis/DataMiner

Let's rely on this also for further modifications.

I reinstalled using it. Note that the project is private, so I manually got the pom file.
The only difference from the last try is that there's a new jar: slf4j-api-1.7.6.jar inside wps/WEB-INF/lib

The latest configuration before you last re-installation had been hand-modified by me. If you refer to that, you will see little difference with the current installation.
In fact, in the current installation the libraries are consistent and the service works!

Gianpaolo Coro wrote:

The latest configuration before you last re-installation had been hand-modified by me. If you refer to that, you will see little difference with the current installation.

The diff was against the dependencies-list.txt file I attached.

In fact, in the current installation the libraries are consistent and the service works!

That's good news, at last :).

I'd like to provision dataminer1-d-d4s.d4science.org too so that we have two completely identical services.

Please go on!

I had more than one problem caused by the modified pom. They didn't show up on dataminer2-d-d4s because of the local maven cache.
I now worked around them and now the only difference between the two installations should be this one:

dataminer1-d-d4s: dataminer-1.0.0-3.9.0.jar
dataminer2-d-d4s: dataminer-1.0.0-SNAPSHOT.jar

It the test of dataminer1-d-d4s ends successfully I'll install dataminer-1.0.0-3.9.0.jar on dataminer2-d-d4s too. After that we need to test that the service still works when queried through the load balancer.

dataminer1 works.

Thanks to @lucio.lelii@isti.cnr.it we now have a new maven command to install the dependencies that does not import the ones already provided by smartgears:

mvn  -DexcludeScope=provided org.apache.maven.plugins:maven-dependency-plugin:2.8:copy-dependencies -Dartifact=org.gcube.dataanalysis:dataminer:1.0.0-3.9.0 -DoutputDirectory=/home/gcube/tomcat/webapps/wps/WEB-INF/lib

We still need to manually remove the wrong 52n dependencies but the situation is less messed up now.

Both the dataminer services work well now. I'm going to test the balancer.

The load balancer works well now. We are ready to put Dataminer in the production environment.