Project

General

Profile

Actions

Support #17934

closed

Provide a GRSF enviroment on preproduction to test the publishing of GRSF records by grsf-services

Added by Francesco Mangiacrapa over 5 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Start date:
Nov 06, 2019
Due date:
% Done:

100%

Estimated time:
VREName:

Description

The GRSF-PRE catalogue provided for testing is https://ckan-grsf.pre.d4science.org (accessible only to a list of public IP, the public access is denied). The catalogue is reachable by preproduction gateway at https://pre.d4science.org/group/dorne/data-catalogue (the VRE is /pred4s/preprod/Dorne).

We have already generated a user (the grsf.publisher) and a token (ddfda8a9-8594-4344-MASKED-TOKEN) that have to be used for testing by grsf-services of preproduction available at http://smart-grsf-d-d4s.d4science.org.

The FORTH (by @marketak@ics.forth.gr) has notified the following error:

2019-11-06 10:24:39 DEBUG GrsfPublisher:84 - publishing URL: https://smart-grsf-d-d4s.d4science.org/grsf-publisher-ws/rest/firms/stock/publish-product?gcube-token=MASKED-TOKEN
2019-11-06 10:24:39 DEBUG GrsfPublisher:88 - publishing content: {"stock_name":"Mackerel icefish - South Georgia","license_id":"CC-BY-SA-4.0","version":1.0,"d...."}
2019-11-06 10:24:39 ERROR GrsfPublisher:99 - Method failed: HTTP/1.1 500 Internal Server Error; Response body: {"id":null,"knowledge_base_id":null,"product_url":null,"error":"Error while performing GET. Request url was: https://ckan-grsf.pre.d4science.org//api/3/action/organization_show?id=firms&include_datasets=false&include_users=true  CkanResponse{error=Ckan error of type: Not Found Error  message:Not found  Other fields:{}, success=false, help=https://ckan-grsf.pre.d4science.org/api/3/action/help_show?name=organization_show}  CkanClient{catalogURL=https://ckan-grsf.pre.d4science.org, ckanToken=*****MASKED_TOKEN*******}"}
2019-11-06 10:24:39 ERROR GrsfPublisher:111 - Fatal transport error:
java.lang.Exception: Method failed: HTTP/1.1 500 Internal Server Error; Response body: {"id":null,"knowledge_base_id":null,"product_url":null,"error":"Error while performing GET. Request url was: https://ckan-grsf.pre.d4science.org//api/3/action/organization_show?id=firms&include_datasets=false&include_users=true  CkanResponse{error=Ckan error of type: Not Found Error  message:Not found  Other fields:{}, success=false, help=https://ckan-grsf.pre.d4science.org/api/3/action/help_show?name=organization_show}  CkanClient{catalogURL=https://ckan-grsf.pre.d4science.org, ckanToken=*****MASKED_TOKEN*******}"}

The reported issue is on the CKAN-side. I believe that is due to the missing organizations (at the moment) on the GRSF-PRE catalogue, the forth is trying to publish into FIRMS organization (that does not exist).
I'm going to create the list of GRSF organizations needed for publishing (like the GRF-ADMIN organizations, see at http://ckan-grsf-admin.d4science.org/api/action/organization_list)


Files

DorneVRE-capture.jpg (84.7 KB) DorneVRE-capture.jpg Yannis Marketakis, Nov 08, 2019 10:30 AM
grsf-publish.log (3.13 KB) grsf-publish.log Yannis Marketakis, Feb 26, 2020 10:58 AM
000a9211-ebcf-3b3e-a72b-11c0f3ce887a.json (1.48 KB) 000a9211-ebcf-3b3e-a72b-11c0f3ce887a.json Yannis Marketakis, Feb 26, 2020 04:05 PM
Actions #1

Updated by Francesco Mangiacrapa over 5 years ago

  • Status changed from In Progress to Feedback
  • % Done changed from 50 to 100

I just created the GRSF-ADMIN organizations also in the GRSF-PRE (see at https://ckan-grsf.pre.d4science.org/api/action/organization_list) and added the user grsf.publisher to each one.
Now, on the CKAN side, the configurations should be OK.

@marketak@ics.forth.gr, could you retry publishing?

Actions #2

Updated by Yannis Marketakis over 5 years ago

  • Status changed from Feedback to Closed

Thanks @francesco.mangiacrapa@isti.cnr.it .
I've started publishing without errors

Actions #3

Updated by Yannis Marketakis over 5 years ago

I noticed that although the records are published without any issues, I cannot browse them in the catalogue, since they seem to be private to me.

Actions #4

Updated by Francesco Mangiacrapa over 5 years ago

Hi @marketak@ics.forth.gr,

now you (with your personal account) should be able to access the items published on that catalogue, in the organizations:

FIRMS
FishSource
GRSF Admin
RAM

let me know...

Actions #5

Updated by Yannis Marketakis over 5 years ago

  • Status changed from In Progress to Closed

I confirm I do.
Many thanks!

Actions #6

Updated by Yannis Marketakis about 5 years ago

It seems that we're facing some issues while publishing GRSF records in https://pre.d4science.org/group/dorne/data-catalogue

Although legacy records (records from FIRMS, RAM, FishSource) are published without any issues, for GRSF records, we're facing an Access Denied error.
The detailed log from the GRSF publishing client is attached.

Actions #7

Updated by Francesco Mangiacrapa about 5 years ago

  • Status changed from Closed to In Progress
  • % Done changed from 100 to 80

Hi Yannis,

could you kindly attach a JSON that you are trying to publish? I suppose that the organisation ID (as field in the JSON) which you are passing to Catalogue does not exist or you are not authorized to publish into it. Should be the organisation ID of "GRSF Admin", right?

Moreover, I just noticed an error returned by gCat on preprod (see at http://gcat.pre.d4science.org/gcat), @roberto.cirillo@isti.cnr.it or @luca.frosini@isti.cnr.it could you check it asap? Many thanks.

Actions #8

Updated by Roberto Cirillo about 5 years ago

I see several calls coming from Dorne VRE:

rejecting call to gcat in invalid context /pred4s/preprod/Dorne, allowed context are [/pred4s/preprod, /pred4s/preprod/preVRE]

gcat is running only in preVRE. @francesco.mangiacrapa@isti.cnr.it if you want I could add gcat to Dorne VRE as well.

Actions #9

Updated by Yannis Marketakis about 5 years ago

@francesco.mangiacrapa@isti.cnr.it please find attached an indicative json file.

I am not sure what you mean with Organization ID. I never used this field.

Actions #10

Updated by Francesco Mangiacrapa about 5 years ago

Yannis Marketakis wrote:

@francesco.mangiacrapa@isti.cnr.it please find attached an indicative json file.

I am not sure what you mean with Organization ID. I never used this field.

Hi Yannis,
you're right, you do not use the field "Organization ID" but the rest interface to push the items in the proper organization directly. That field is used into another context, sorry for this.
In this period, I'm out of office, thanks to @pasquale.pagano@isti.cnr.it, he just added the user "grsf_publisher" to "GRSF PRE" organisation. Now you should be authorized to publish in such organization. Let me know...

Actions #11

Updated by Yannis Marketakis about 5 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 80 to 100

Thanks @francesco.mangiacrapa@isti.cnr.it and @pasquale.pagano@isti.cnr.it .
I confirm it works.

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)