Project

General

Profile

Actions

Support #11021

closed

pb algo data miner

Added by Paul Taconet over 7 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Start date:
Jan 23, 2018
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Development

Description

I have the attached pb (logs) when I execute this algo:

https://i-marine.d4science.org/group/rprototypinglab/data-miner?OperatorId=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.CREATE_YOUR_OWN_GEOREFERENCED_CATCH_TUNA_ATLAS_DATASET

Seems to be linked to the following R script:

cat("setting working directory\n")
working_directory_init=getwd()
cat("setting working directory",working_directory_init,"\n")
setwd(working_directory_init)
system("chmod -R 777 ./
")*

Thanks,
Paul


Files

Actions #1

Updated by Giancarlo Panichi over 7 years ago

what's the problem?

Actions #2

Updated by Paul Taconet over 7 years ago

I do not know. If you try to execute the algo through the DM you will see that it does not work (it returns an error).
The execution of the script seems to stop at its very beginning, ie after the line system("chmod -R 777 ./")

Actions #3

Updated by Giancarlo Panichi over 7 years ago

  • Assignee changed from Giancarlo Panichi to Paul Taconet

Please @paul.taconet@ird.fr , add some logs to your code because I can not figure out where the algorithm stops, and republish it.

Actions #4

Updated by Paul Taconet over 7 years ago

Ok, I have added logs and republished the script

Actions #5

Updated by Paul Taconet over 7 years ago

Following the logs, the problem seems to be located in these lines of the R script (i.e. where I set the default values for the parameters):

cat("setting values to parameters\n")
include_IOTC<-TRUE
include_ICCAT<-TRUE
include_IATTC<-TRUE
include_WCPFC<-TRUE
include_CCSBT<-TRUE
datasets_year_release<-"2017"
iccat_include_type_of_school<-TRUE
iattc_raise_flags_to_schooltype<-TRUE
iattc_dimension_to_use_if_no_raising_flags_to_schooltype<-"flag"
mapping_map_code_lists<-TRUE
mapping_csv_mapping_datasets_url<-"http://data.d4science.org/ZWFMa3JJUHBXWk9NTXVPdFZhbU5BUFEyQnhUeWd1d3lHbWJQNStIS0N6Yz0"
mapping_keep_src_code<-FALSE
gear_filter<-"NULL"
unit_conversion_convert<-FALSE
unit_conversion_csv_conversion_factor_url<-"http://data.d4science.org/Z3V2RmhPK3ZKVStNTXVPdFZhbU5BTTVaWnE3VFAzaElHbWJQNStIS0N6Yz0"
unit_conversion_codelist_geoidentifiers_conversion_factors<-"areas_conversion_factors_numtoweigth_ird"
raising_georef_to_nominal<-TRUE
aggregate_on_5deg_data_with_resolution_inferior_to_5deg<-TRUE
disaggregate_on_5deg_data_with_resolution_superior_to_5deg<-"none"
disaggregate_on_1deg_data_with_resolution_superior_to_1deg<-"none"
spatial_curation_data_mislocated<-"remove"
overlapping_zone_iattc_wcpfc_data_to_keep<-"IATTC"
SBF_data_rfmo_to_keep<-"CCSBT"
metadata_title<-"Global monthly catch of tuna, tuna-like and shark species (%year_start%-%year_end%) aggregated by %spatial_resolution%° squares"
metadata_contact_owner<-"ird@ird.fr"
metadata_contact_metadata<-"paul.taconet@ird.fr@@julien.barde@ird.fr@@emmanuel.blondel@fao.org"
metadata_contact_pointofcontact<-"paul.taconet@ird.fr@@julien.barde@ird.fr@@emmanuel.blondel@fao.org"
metadata_contact_principalinvestigator<-"emmanuel.chassot@ird.fr"
metadata_contact_publisher<-"ird@ird.fr"
metadata_contact_processor<-"paul.taconet@ird.fr"
metadata_description<-"This dataset lists global catch of tuna, tuna-like and shark species from %year_start% to %year_end%. Catches are stratified by month, species, gear, vessel flag reporting country, fishing mode (i.e. type of school used), area (mainly 5° square) and unit of catch (weight or number). This dataset was computed using public domain catch-and-effort datasets released by the five tuna Regional Fisheries Management Organizations: the Indian Ocean Tuna Commission (IOTC), the International Commission for the Conservation of Atlantic Tunas (ICCAT), the Inter-American Tropical Tuna Commission (IATTC), The Western and Central Pacific Fisheries Commission (WCPFC), the Commission for the Conservation of Southern Bluefin Tuna (CCSBT)."
metadata_source<-"IRD"
metadata_supplemental_information<-"IRD"

cat("setting values to parameters OK\n")

Actions #6

Updated by Emmanuel Blondel over 7 years ago

If i remember well, there is a weird adhoc "workaround" to make Dataminer understand the boolean parameters... https://wiki.gcube-system.org/gcube/Statistical_Algorithms_Importer:_R_Project_FAQ#Managing_Boolean_values Maybe it is what you are looking for...

Actions #7

Updated by Paul Taconet over 7 years ago

Thanks Emmanuel, the pb was actually the one you have pointed out.

I have another question: I would like to use within my script the functions to interact with the WS (the ones located here: https://wiki.gcube-system.org/gcube/Workspace_Interaction_From_R). For this I need to retrieve somewhere the username and the token.
When I look at the logs (see attached) I see that the first lines are:

gcube_token<<-"0b999a4f-f160-4697-9166-b68e0071f128-843339462"
gcube_username<<-"paultaconet"
gcube_context<<-"/d4science.research-infrastructures.eu/gCubeApps/RPrototypingLab"

So I have assumed that these variables are created and filled with the right values. However when I try to use them in my script, it seems they do not exist (see logs attached)

Error in push_R_script_to_server(metadata_and_parameterization, virtual_repository_with_R_files, :
object 'gcube_username' not found

Can you indicate me how I can retrieve the gcube username and token and use them in the scripts that I import in the SAI?

Thanks,
Paul

Actions #8

Updated by Paul Taconet over 7 years ago

The line in my R script is:

metadata_and_parameterization<-push_R_script_to_server(metadata_and_parameterization,virtual_repository_with_R_files,gcube_username,gcube_token)

here I need as values for gcube_username and gcube_token the username and the token of the user that is executing the algo through the Data miner in the VRE

Actions #9

Updated by Giancarlo Panichi over 7 years ago

  • Tracker changed from Incident to Support
Actions #10

Updated by Paul Taconet over 7 years ago

Thanks for the documentation.
However, the logs indicate that the object "gcube_username" does not exist (object 'gcube_username' not found) (https://support.d4science.org/attachments/download/2105/RLOGb8345554-8a50-446c-bf59-aec05145675a.txt , while according to the documentation, this object should exist.

Actions #11

Updated by Gianpaolo Coro over 7 years ago

Hi Paul, I guarantee the variables are there because they are used extensively by other algorithms (also some we are currently running every day).

However, global variables are deleted if you use cleaning instruction such as rm(list=ls(all=TRUE)) at the beginning of your code. These instructions are not necessary on the DataMiner since when an algorithm runs it is always executed in a completely new environment.

Actions #12

Updated by Paul Taconet over 7 years ago

Thanks GianPaolo. I removed the rm(list=ls(all=TRUE)) at the beginning of my code and I do not have that pb anymore.

However, I have a new pb linked to this.

This is what I do in my script:

in this function, I have (among others) the following lines:
source("http://svn.research-infrastructures.eu/public/d4science/gcube/trunk/data-analysis/RConfiguration/RD4SFunctions/workspace_interaction.r")
username<-vre_username
token<-vre_token
uploadWS(virtual_repository_with_R_files,RFileName,overwrite=T)
RFileURL <- getPublicFileLinkWS(paste(virtual_repository_with_R_files,RFileName,sep="/"))

  • Then I execute the function with the following parameters: metadata_and_parameterization<-push_R_script_to_server(metadata_and_parameterization,virtual_repository_with_R_files,gcube_username,gcube_token)

In the logs (see attached) I get: Error in authenticate(username, token, type = "basic") :
object 'username' not found

It seems that the error comes from when I source the workspace_interaction script, i.e. : source("http://svn.research-infrastructures.eu/public/d4science/gcube/trunk/data-analysis/RConfiguration/RD4SFunctions/workspace_interaction.r")

Can you indicate where I am doing something wrong?
Thanks,
Paul

Actions #13

Updated by Gianpaolo Coro over 7 years ago

Hi Paul,
I think you just miss to use a "<<-" instead of "<-":

Since they are global variables, I would use directly gcube_username and gcube_token. Thus, I would write

username<<-gcube_username
token<<-gcube_token

before calling the WS functions. Anyway, also in your code (with the vre_* variables) this should work. You should be done with the lines above.

Overall, I'm referring to the instructions in this page: https://wiki.gcube-system.org/gcube/Workspace_Interaction_From_R

Actions #14

Updated by Paul Taconet over 7 years ago

Thanks GianPaolo.
I have modified my code as you indicated and it is working now.
Thank you!

Actions #15

Updated by Pasquale Pagano over 7 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)