Project

General

Profile

Actions

Task #176

closed

Configure DSS to support EUDAT-D4Science integration

Added by Pasquale Pagano almost 10 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Category:
Data Management
Target version:
Start date:
May 28, 2015
Due date:
Jun 08, 2015
% Done:

100%

Estimated time:
Infrastructure:
Production

Description

According to what reported here: http://eudat.eu/User%20Documentation%20-%20Data%20Staging%20Script.html#UserDocumentation-DataStagingScript-Installationnotesforend-user the following steps should be performed to setup DSS

Installation notes for end-user

The following requirements need to be satisfied in order to install the Data Staging Script:

Python 2.7 (pip is suggested)
grid-proxy-utils (in particular grid-proxy-init)
python-m2crypto (script version > 1.1). You should also replace the file m2.py installed with the GO API with a new one, available on the SVN (see below).
or myproxy commands (in particular myproxy-init)
(optionally) icommands
The DSS uses the GlobusOnline service via its API to manage the transfers. For this reason, you also need the following provisions:

Acquire an account on GO and configure the needed GO end points.
Associate your X.509 certificate to your GO account under https://www.globusonline.org/account/ManageIdentities.
If you would like to transfer data from your laptop where no GridFTP server is available, please consider installing GlobusConnect. Thanks to GO you will be able to look at your data transfers transfer in the GO web GUI.
The installation is quite simple:

get the Script from the GitHub repository.
install globus online api
for example: pip install globusonline-transfer-api-client
configure your irods environment in order to point to the iRODS server enabled with the needed rules
store the path of icommands; in datastagerconfig.py, iPATH stores the the path of the icommands, like in the following line:

            iPATH='<IRODS-PATH>/clients/icommands/bin/'

In order to use the DSS software, you should adapt it to the path of your iRODS installation.

The DSS software distribution consists of the following files:

datastagerconfig.py.example template for the required (cp to) datastagerconfig.py file (script >1.1)
PIDselecter.r the rule which gives you the PID of a file given its URL
URLselecter.r the rule which gives you the URL of a file given its PID

datastager.py the main script
datamover.py the script which invokes GO

example some concrete example
pid.file.example an example of PID file, i.e. a list of PIDs (to be used with -PF option)
task.file.example an example of path file, i.e. a list of file names (to be used with -pF option)
url.file.example an example of URL file, i.e. a list of URLs (to be used with -UF option)
README with some further information

Actions #1

Updated by Andrea Dell'Amico almost 10 years ago

We had a conference call with Giovanni Morelli, he asked to wait for more precise guidelines. He will send them in the next days.

Actions #2

Updated by Andrea Dell'Amico almost 10 years ago

  • Status changed from New to In Progress
Actions #3

Updated by Andrea Dell'Amico about 9 years ago

  • Status changed from In Progress to Paused
Actions #4

Updated by Andrea Dell'Amico over 6 years ago

Can we close this one?

Actions #5

Updated by Pasquale Pagano over 6 years ago

  • Status changed from Paused to Closed
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)