Task #23670
closedSupport of a new Source of records in GRSF VREs
100%
Description
As already discussed GRSF VREs and the corresponding services (i.e. GRSF Publisher) will have to be updated to support a new type of Source.
The new Source will be FAO SDG 14.4.1 Questionnaire with both stock and fishery records, and will complement the existing sources: FIRMS, RAM, FishSource, and GRSF.
This new Source will be treated as a legacy source, therefore the corresponding records will handled in a similar manner as records from those sources (i.e. FIRMS)
From the GRSF KB perspective, the new source and the records are ready to be published. Therefore when a first draft of the GRSF Publisher is available we can directly test it.
In order to maintain GRSF Admin and GRSF Public VREs clean, we could test the addition of records from the new source in GRSF PRE VRE.
Files
Related issues
Updated by Luca Frosini almost 3 years ago
- Status changed from New to In Progress
@marketak@ics.forth.gr is it ok for you if use sdg
as path part?
E.g. for firms we have:
/grsf-publisher-ws/rest/firms/stock/publish-product
for FAO SDG 14.4.1 Questionnaire we will have
/grsf-publisher-ws/rest/sdg/stock/publish-product
If this ok for you?
Updated by Luca Frosini almost 3 years ago
- Status changed from In Progress to Feedback
Updated by Yannis Marketakis almost 3 years ago
- Status changed from Feedback to In Progress
Absolutely fine by me
Updated by Luca Frosini almost 3 years ago
I just found a note about the URL should be:
/sdg14.4.1
Sorry, I forget it
Updated by Luca Frosini almost 3 years ago
- Status changed from In Progress to Paused
Updated by Luca Frosini over 2 years ago
- Status changed from Paused to In Progress
Updated by Luca Frosini over 2 years ago
- Status changed from In Progress to Paused
Updated by Yannis Marketakis over 2 years ago
Hi @luca.frosini@isti.cnr.it.
Do we have any news with the updated publisher service?
Updated by Luca Frosini over 2 years ago
- Status changed from Paused to In Progress
Yannis Marketakis wrote in #note-8:
Hi @luca.frosini@isti.cnr.it.
Do we have any news with the updated publisher service?
I'm going to work on this
Updated by Luca Frosini over 2 years ago
- % Done changed from 0 to 50
@marketak@ics.forth.gr I have changed the code to support this new source but I have to properly test it.
Can you please provide me 2 or 3 json of records from this source?
Updated by Luca Frosini over 2 years ago
@roberto.cirillo@isti.cnr.it can you upgrade the preproduction with version 1.13.3-SNAPSHOT
Updated by Luca Frosini over 2 years ago
- Status changed from In Progress to Feedback
- Assignee changed from Luca Frosini to Roberto Cirillo
@roberto.cirillo@isti.cnr.it to properly manage this new feature and the required fix for #23679 I have branched grsf-publisher-ws and grsf-common-library.
I also needed to upgrade the major version of grsf-common-library to 2.0.0-SNAPSHOT because we need to break the backward compatibility to support the new source.
For this reason, I have also created dedicated Jenkins jobs
https://jenkins.d4science.org/job/grsf-common-library-23670
https://jenkins.d4science.org/job/grsf-publisher-ws-23670
Jenkins compiled successfully this new versions.
Updated by Yannis Marketakis over 2 years ago
- File 70ae6895-7d3d-4f4a-86f9-bcb17d41bff6.json 70ae6895-7d3d-4f4a-86f9-bcb17d41bff6.json added
- File aebf21a0-7d0d-4fd6-a528-15a6f95897a6.json aebf21a0-7d0d-4fd6-a528-15a6f95897a6.json added
- File efa5c7b4-7072-4212-bed5-4be9bb292932.json efa5c7b4-7072-4212-bed5-4be9bb292932.json added
Luca Frosini wrote in #note-10:
@marketak@ics.forth.gr I have changed the code to support this new source but I have to properly test it.
Can you please provide me 2 or 3 json of records from this source?
@luca.frosini@isti.cnr.it attached you will find 3 indicative stock records for that source
Updated by Roberto Cirillo over 2 years ago
- Assignee changed from Roberto Cirillo to Luca Frosini
Luca Frosini wrote in #note-11:
@roberto.cirillo@isti.cnr.it can you upgrade the preproduction with version 1.13.3-SNAPSHOT
preproduction upgraded with 1.13.3-SNAPSHOT version as requested. Let me know if everything works for you now.
Updated by Luca Frosini over 2 years ago
Trying to create the record I get:
{ "success": false, "message": "'name' field of a resource cannot be null", "result": null }
I don't get this error trying to create a GRSF stock.
I'm going to debug the code to check if there is a bug or something which I didn't consider adding the new source of record.
@marketak@ics.forth.gr do you confirm that the attached json contain all the required fields?
Updated by Yannis Marketakis over 2 years ago
@luca.frosini@isti.cnr.it Please consider that this is not a GRSF stock record.
It is a legacy stock record.
Updated by Luca Frosini over 2 years ago
I found the issue.
It is related to the path source which does not correspond (ignoring cases) to the 'name' of database_source entry
"database_sources" : [ {
"name" : "FAO SDG 14.4.1 questionnaire",
"description" : "FAO SDG 14.4.1 questionnaire",
"url" : "https://www.fao.org/sustainable-development-goals/indicators/14.4.1/en/"
} ],
"FAO SDG 14.4.1 questionnaire" != 'sdg14.4.1'
Instead, there were matches for the other sources. I had already considered these differences but I didn't fix the JSON deserializer.
@roberto.cirillo@isti.cnr.it jenkins has already built the service. Can you please upgrade again the preproduction with version 1.13.3-SNAPSHOT.
Updated by Luca Frosini over 2 years ago
@francesco.mangiacrapa@isti.cnr.it I needed to modify the enum org.gcube.datacatalogue.common.enums.Sources
in grsf-common-library
Can you please check if it has any impact on your side?
The new version is 2.0.0-SNAPSHOT. As you can see I upgraded the major version because ti breaks backward compatibility of the library.
Updated by Luca Frosini over 2 years ago
@francesco.mangiacrapa@isti.cnr.it grsf-manage-widget
uses grsf-common-library
Updated by Luca Frosini over 2 years ago
- Assignee changed from Luca Frosini to Roberto Cirillo
Updated by Roberto Cirillo over 2 years ago
- Assignee changed from Roberto Cirillo to Luca Frosini
Luca Frosini wrote in #note-18:
@roberto.cirillo@isti.cnr.it jenkins has already built the service. Can you please upgrade again the preproduction with version 1.13.3-SNAPSHOT.
Done right now.
Updated by Francesco Mangiacrapa over 2 years ago
Luca Frosini wrote in #note-19:
@francesco.mangiacrapa@isti.cnr.it I needed to modify the enum
org.gcube.datacatalogue.common.enums.Sources
ingrsf-common-library
Can you please check if it has any impact on your side?
The new version is 2.0.0-SNAPSHOT. As you can see I upgraded the major version because ti breaks backward compatibility of the library.
Hi @luca.frosini@isti.cnr.it
I confirm that grsf-manage-widget
uses the grsf-common-library
at line https://code-repo.d4science.org/gCubeSystem/grsf-manage-widget/src/branch/master/src/main/java/org/gcube/datacatalogue/grsf_manage_widget/server/manage/GRSFNotificationService.java#L323
Upgrading the major version of the grsf-common-library, I need to release the grsf-manage-widget
. If it is urgent, please, create a specific ticket for this activity. Thanks.
Updated by Francesco Mangiacrapa over 2 years ago
- Related to Task #23809: The grsf-manage-widget must pass to grsf-common-library.2.0.0[-SNAPSHOT] added
Updated by Luca Frosini over 2 years ago
- Status changed from Feedback to In Progress
- Assignee changed from Luca Frosini to Roberto Cirillo
@marketak@ics.forth.gr I need to switch to 'sdg' as path because in the code there were to much assumption on the used path.
@roberto.cirillo@isti.cnr.it can you please upgrade again the preproduction with version 1.13.3-SNAPSHOT?
Updated by Roberto Cirillo over 2 years ago
- Status changed from In Progress to Feedback
- Assignee changed from Roberto Cirillo to Luca Frosini
Done right now.
Updated by Luca Frosini over 2 years ago
- Status changed from Feedback to In Progress
@marketak@ics.forth.gr I made a test for SDG in the preproduction environment. It seems working now.
You can check if the new source properly works at:
Updated by Luca Frosini over 2 years ago
- Status changed from In Progress to Feedback
- Assignee changed from Luca Frosini to Yannis Marketakis
Updated by Yannis Marketakis over 2 years ago
Thanks @luca.frosini@isti.cnr.it
I see that the record has been published. However, it seems to be private and I am unable to check its contents (see screenshot)
Is the new service in place for GRSF PRE right now? Should I try to publish records?
Updated by Luca Frosini over 2 years ago
I think we are ready to test it.
@roberto.cirillo@isti.cnr.it can you please upgrade again GRSF_Pre in preproduction as weel as GRSF_Pre in production with version 1.13.3-SNAPSHOT?
@marketak@ics.forth.gr as soon as the GRSF_Pre in production will be updated you can make any tests.
Updated by Luca Frosini over 2 years ago
- Status changed from Feedback to In Progress
- Assignee changed from Yannis Marketakis to Roberto Cirillo
Updated by Francesco Mangiacrapa over 2 years ago
- Related to deleted (Task #23809: The grsf-manage-widget must pass to grsf-common-library.2.0.0[-SNAPSHOT])
Updated by Francesco Mangiacrapa over 2 years ago
- Blocks Task #23809: The grsf-manage-widget must pass to grsf-common-library.2.0.0[-SNAPSHOT] added
Updated by Roberto Cirillo over 2 years ago
- Status changed from In Progress to Feedback
- Assignee changed from Roberto Cirillo to Luca Frosini
Luca Frosini wrote in #note-30:
I think we are ready to test it.
@roberto.cirillo@isti.cnr.it can you please upgrade again GRSF_Pre in preproduction as weel as GRSF_Pre in production with version 1.13.3-SNAPSHOT?
both the instances upgraded right now
Updated by Yannis Marketakis over 2 years ago
@luca.frosini@isti.cnr.it are the any updates on this?
Updated by Aureliano Gentile over 2 years ago
Dear @luca.frosini@isti.cnr.it and @marketak@ics.forth.gr, thanks for the work done so far, please this is a kind reminder to complete whatever is still pending so we can start loading records from this new source of data (FAO SDG 14.4.1 questionnaire).
With thnaks in advance, this would be a major enhancement and we would need it now since we are going to dispatch the new SDG questionnaires to countries and it would be great to add the UUIDs when these are generated.
Updated by Yannis Marketakis over 2 years ago
- % Done changed from 60 to 100
Dear @aureliano.gentile@fao.org
I have started testing the new facility on GRSF PRE and you will notice that we have already published some records from SDG (more than 200) https://blue-cloud.d4science.org/group/grsf_pre/data-catalogue
We are currently testing some features. I will ask you and @anne.elise.nieblas@gmail.com to test it, hopefully by the end of this week.
Updated by Luca Frosini over 2 years ago
The service with the new source of record is under release. It will be deployed ASAP.
Updated by Luca Frosini over 2 years ago
- Status changed from Feedback to Closed