Project

General

Profile

Actions
Copy link

Task #4865

closed

Installing data analysis machine to develop prototypes

Added by Gianpaolo Coro over 8 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
High
Assignee:
Andrea Dell'Amico
Category:
High-Throughput-Computing
Target version:
Dataminer and StatMan provisioning
Start date:
Aug 22, 2016
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Production

Description

An all-in-one machine is required by the developers of Dataminer algorithms. This machine should be a production Dataminer instance to be placed under the "dataminer-prototype.d4science.org" proxy group. The same machine should be endowed with an RStudio instance.
Developers will use the RStudio instance to develop their R script and then the Dataminer instance to run their prototypes.

The installation of the R packages will be still at root level and managed in the current way. The installation of the algorithm will pass from our management but will be soon automatised through the Statistical Algorithms Importer.

Related issues

Related to D4Science Infrastructure - Task #4371: same R configuration (R version and packages) on R-Studio as the one on machine(s) used by Dataminer. ClosedAndrea Dell'AmicoJun 23, 2016Jul 23, 2016

Actions
Related to D4Science Infrastructure - Task #4842: Make the ldap configuration on rstudio(-dev).d4science.org point to ldap.d4science.orgClosedAndrea Dell'AmicoAug 02, 2016

Actions
Blocked by D4Science Infrastructure - Task #4917: Update Dataminer Ansible installationClosedAndrea Dell'AmicoSep 01, 2016

Actions
Actions
Copy link
 #2

Updated by Pasquale Pagano over 8 years ago

  • Tracker changed from Support to Task
Actions
Copy link
 #3

Updated by Gianpaolo Coro over 8 years ago

  • Priority changed from Normal to High

Do you have updates on this ticket? Please this is a requirement of the BlueBRIDGE project.

Actions
Copy link
 #4

Updated by Leonardo Candela over 8 years ago

  • Parent task set to #4894
Actions
Copy link
 #5

Updated by Pasquale Pagano over 8 years ago

Please @tommaso.piccioli@isti.cnr.it start creating a new VM. Then @andrea.dellamico@isti.cnr.it will run the ansible playbook to configure it.

Actions
Copy link
 #6

Updated by Tommaso Piccioli over 8 years ago

dataminer1-proto.d4science.org created, to be configured as dataminer.

Initial resources are 40 GB disk, 8 cpu and 8 GB ram, they can be modified.

Actions
Copy link
 #7

Updated by Andrea Dell'Amico over 8 years ago

I have some questions:

  • rstudio server needs to authenticate in the same way of rstudio.d4science.org?
  • If so, against the new ldap server? (rstudio(-d).d4science.org authenticate against the old one, when I tried to switch it didn't work)
  • is the new installation procedure for the dataminer needed? (See #4917 )
Actions
Copy link
 #8

Updated by Andrea Dell'Amico over 8 years ago

  • Blocked by Task #4917: Update Dataminer Ansible installation added
Actions
Copy link
 #9

Updated by Andrea Dell'Amico over 8 years ago

  • Related to Task #4371: same R configuration (R version and packages) on R-Studio as the one on machine(s) used by Dataminer. added
Actions
Copy link
 #10

Updated by Andrea Dell'Amico over 8 years ago

  • Blocked by Task #4842: Make the ldap configuration on rstudio(-dev).d4science.org point to ldap.d4science.org added
Actions
Copy link
 #11

Updated by Andrea Dell'Amico over 8 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #12

Updated by Andrea Dell'Amico over 8 years ago

  • Assignee changed from _InfraScience Systems Engineer to Andrea Dell'Amico
Actions
Copy link
 #13

Updated by Andrea Dell'Amico over 8 years ago

@lucio.lelii@isti.cnr.it @roberto.cirillo@isti.cnr.it I need your help with the smartgears contexts configuration. The actual production dataminers have /wps as context:

  <application mode="online" context="/wps">

        <scope>/d4science.research-infrastructures.eu</scope>

            <scope>/d4science.research-infrastructures.eu/gCubeApps</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/BiodiversityLab</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/BiOnym</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ScalableDataMining</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/StockAssessment</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/SoBigData.eu</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/PerformanceEvaluationInAquaculture</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/FAO_TunaAtlas</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ProtectedAreaImpactMaps</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ENVRIPlus</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ICCAT_BFT-E</scope>
  </application>

While rstudio.d4science.org has /r-connector:

  <application mode="online" context="/r-connector">
    <scope>/d4science.research-infrastructures.eu</scope>
            <scope>/d4science.research-infrastructures.eu/gCubeApps</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/TabularDataLab</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/FAO_TunaAtlas</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/BOBLME_HilsaAWG</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/StockAssessment</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/BiodiversityLab</scope>
    <scope>/d4science.research-infrastructures.eu/gCubeApps/ICCAT_BFT-E</scope>
          </application>

Do we need two contexts (it means that I need to refactor the container.xml templating heavily)? or only one is permitted? if so, which one?

Actions
Copy link
 #14

Updated by Lucio Lelii over 8 years ago

you can put both contexts in the container.xml with the same list of scopes.

Actions
Copy link
 #15

Updated by Andrea Dell'Amico over 8 years ago

Thanks.

Actions
Copy link
 #16

Updated by Andrea Dell'Amico over 8 years ago

  • % Done changed from 0 to 70

I think only the LDAP configuration for rstudio is missing. In the meantime could you check if the dataminer jars are correctly installed and that the smartgear configuration is correct?

I also see those logs:

Connecting...
Updating IS on scope /d4science.research-infrastructures.eu
echo  | voms-proxy-init --voms fedcloud.egi.eu --rfc --dont-verify-ac -cert /tmp/x509up_u1000 -out /tmp/proxy417816595.pem -pwstdin
Process started !
/bin/sh: 1: voms-proxy-init: not found
Process ended !
Proxy Certificate created
java.io.FileNotFoundException: /tmp/proxy417816595.pem (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at org.gcube.vomanagement.occi.utils.ScriptUtil.getScriptFromFile(ScriptUtil.java:26)
        at org.gcube.resources.federation.fhnmanager.impl.ConnectorFactory.createConnector(ConnectorFactory.java:115)
        at org.gcube.resources.federation.fhnmanager.impl.ConnectorFactory.getConnector(ConnectorFactory.java:30)
        at org.gcube.resources.federation.fhnmanager.is.ISProxyImpl.updateIs(ISProxyImpl.java:491)
        at org.gcube.resources.federation.fhnmanager.impl.ISSynchronizer.run(ISSynchronizer.java:47)
        at java.lang.Thread.run(Thread.java:745)

voms-proxy-init is an EGI cloud thing, don't know why it's been executed.

Actions
Copy link
 #17

Updated by Andrea Dell'Amico over 8 years ago

The smartgear configuration is

<container mode='online'>

  <hostname>dataminer1-proto.d4science.org</hostname>
      <port>80</port>
      <infrastructure>d4science.research-infrastructures.eu</infrastructure>
      <vo>gCubeApps</vo>

  <site>
    <country>it</country>
    <location>pisa</location>
    <latitude>41.9</latitude>
    <longitude>12.5</longitude>
  </site>

  <property name='SmartGearsDistributionBundle' value='UnBundled' />
  <property name='SmartGearsDistribution' value='1.2.8-4.0.0-129615' />
  <publication-frequency>60</publication-frequency>

  <application mode="online" context="/wps">
        <scope>/d4science.research-infrastructures.eu</scope>
                <scope>/d4science.research-infrastructures.eu/gCubeApps</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/BiodiversityLab</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/BiOnym</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ScalableDataMining</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/StockAssessment</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/SoBigData.eu</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/PerformanceEvaluationInAquaculture</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/FAO_TunaAtlas</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ProtectedAreaImpactMaps</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ENVRIPlus</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ICCAT_BFT-E</scope>
          </application>

  <application mode="online" context="/r-connector">
        <scope>/d4science.research-infrastructures.eu</scope>
                <scope>/d4science.research-infrastructures.eu/gCubeApps</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/BiodiversityLab</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/BiOnym</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ScalableDataMining</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/StockAssessment</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/SoBigData.eu</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/PerformanceEvaluationInAquaculture</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/FAO_TunaAtlas</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ProtectedAreaImpactMaps</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ENVRIPlus</scope>
        <scope>/d4science.research-infrastructures.eu/gCubeApps/ICCAT_BFT-E</scope>
          </application>
    </container>
Actions
Copy link
 #18

Updated by Andrea Dell'Amico over 8 years ago

Andrea Dell'Amico wrote:

I think only the LDAP configuration for rstudio is missing. In the meantime could you check if the dataminer jars are correctly installed and that the smartgear configuration is correct?

I also see those logs:

Connecting...

And those logs go into catalina.out. That should never happen.

Actions
Copy link
 #19

Updated by Andrea Dell'Amico over 8 years ago

The ldap authentication needed by the r-connector is in place. Against the new ldap server, so this has to be tested as well.

Actions
Copy link
 #20

Updated by Andrea Dell'Amico over 8 years ago

  • Status changed from In Progress to Feedback
  • % Done changed from 70 to 100
Actions
Copy link
 #21

Updated by Gianpaolo Coro over 8 years ago

It seems that the only way to completely test this service is to have a new VRE in the production environment.

Actions
Copy link
 #22

Updated by Andrea Dell'Amico over 8 years ago

Andrea Dell'Amico wrote:

Connecting...
Updating IS on scope /d4science.research-infrastructures.eu
echo  | voms-proxy-init --voms fedcloud.egi.eu --rfc --dont-verify-ac -cert /tmp/x509up_u1000 -out /tmp/proxy417816595.pem -pwstdin

About this one: a service was installed by mistake (wrong defaults into the ansible role). I'm removing it and restarting tomcat.

Actions
Copy link
 #23

Updated by Pasquale Pagano over 8 years ago

  • Status changed from Feedback to In Progress
Actions
Copy link
 #24

Updated by Pasquale Pagano over 8 years ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #25

Updated by Andrea Dell'Amico over 8 years ago

  • Blocked by deleted (Task #4842: Make the ldap configuration on rstudio(-dev).d4science.org point to ldap.d4science.org)
Actions
Copy link
 #26

Updated by Andrea Dell'Amico over 8 years ago

  • Related to Task #4842: Make the ldap configuration on rstudio(-dev).d4science.org point to ldap.d4science.org added
Actions
Copy link
 #27

Updated by Andrea Dell'Amico over 8 years ago

  • Status changed from Feedback to Closed

rstudio server is working, and the ldap authentication against the new ldap server too.

Actions
Copy link

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)