StocksAndFisheriesKB

If we remove them they are will be re-created when we publish data. Is that correct?

Yannis Marketakis wrote:

If we remove them they are will be re-created when we publish data. Is that correct?

Yes, It is correct. Except for bug, when records will be re-published the groups will be created on-the-fly (and the records added to them) for the fields that have the property isGroup = Yes (see at https://wiki.gcube-system.org/gcube/GCube_Data_Catalogue_for_GRSF), so may I proceed with removing the groups?

Yes please do.
Thanks

As per interaction with @francesco.mangiacrapa@isti.cnr.it I would need assistance from:

@marketak@ics.forth.gr for confirming that such approved GRSF records are associated to groups
@luca.frosini@isti.cnr.it to check why groups are missing.

@aureliano.gentile@fao.org grouping is a facility offered by the catalogue. Such information is not stored in the GRSF KB. The published records are added in groups during publishing.

thanks, @francesco.mangiacrapa@isti.cnr.it you have your answer from FORTH.

Yannis Marketakis wrote:

@aureliano.gentile@fao.org grouping is a facility offered by the catalogue. Such information is not stored in the GRSF KB. The published records are added in groups during publishing.

Sure and thanks @marketak@ics.forth.gr
The question is: have you already published some records that had to be added to groups during the publishing? If yes, could you attach (to this ticket) a GRSF record (as JSON source of input) already published? It will be used by @luca.frosini@isti.cnr.it to check why the groups were not created...

@francesco.mangiacrapa@isti.cnr.it the records published in GRSF VRE are replicas of GRSF records found in GRSF Admin VRE. As such all of them fall under certain groups.

You will find the JSON contents of all these GRSF records (597 in total) at https://goo.gl/wQDAV2

Yannis Marketakis wrote:

@francesco.mangiacrapa@isti.cnr.it the records published in GRSF VRE are replicas of GRSF records found in GRSF Admin VRE. As such all of them fall under certain groups.

You will find the JSON contents of all these GRSF records (597 in total) at https://goo.gl/wQDAV2

Thanks a lot, @marketak@ics.forth.gr.
@luca.frosini@isti.cnr.it could you check this issue (why the groups were not created during the publishing) asap?

As soon as there's no other way we will republish them

We cleaned the catalogue.
@marketak@ics.forth.gr can you republish the records? Thanks a lot

Thanks Luca.
I am re-publishing right away

thanks, no one was imaging the need to purge the grsf vre again, or at least nobody warned me. Anyway thanks a lot and sorry for that.
At the moment the group of source records are empty (i.e. Groups Stock - FIRMS or Groups Stock - FIRMS RAM, etc.). But maybe it is due to the publishing still in progress?

The publishing of the records was completed a couple of hours after updating the ticket.
However, I still see that some groups are empty. @francesco.mangiacrapa@isti.cnr.it and @luca.frosini@isti.cnr.it can you please check?

Yannis Marketakis wrote:

The publishing of the records was completed a couple of hours after updating the ticket.
However, I still see that some groups are empty.

Hi @marketak@ics.forth.gr, @luca.frosini@isti.cnr.it,

checking the published records (at https://ckan-grsf.d4science.org/dataset) that were added to groups (at https://ckan-grsf.d4science.org/group), the situation seems to me the following:

1. the groups for legacy records ('Stock - FIRMS', 'Stock FIRMS FishSource' and so on) are empty because no legacy records have been published... and they will not be published. Is it right? @aureliano.gentile@fao.org, do we want to remove such groups from GRSF Catalogue?

2. no fishery records have been published, then the "GRSF Fishery" group is empty (see at https://ckan-grsf.d4science.org/group/grsf-fishery)

3. We should investigate on it (see attached screenshot)... 595 published records are "Assessment Unit" and they all had to be added to:
   ** "GRSF Assessment Unit", they are 593 - 2 records are missing;
   ** GRSF Stock", they are 592 - 3 records are missing;

If this analysis sounds for you, I can go immediately with Luca to find missing published records in those groups...

Thanks, sorry but I think there is a misunderstanding, that list is for GRSF records and those groups should be there and populated with the current numbers of approved records. None of those items should be removed. if you want tomorrow we can have a brief call on that.

Hi all,

The field refers_to is used to automatically create the field Database Source and to add the GRSF record to some groups.

As the wiki report (see https://wiki.gcube-system.org/gcube/GCube_Data_Catalogue_for_GRSF#Common_Metadata), the field refers_to contains:

"A list of objects of the format {"url": "http://", "id": "..."} that allows the aggregated GRSF records to point to their source records **already published in the catalogue. The url and the id are both mandatory and are the ones returned by the services when a source record is published."

The code retrieves the referred records and uses their information to create the field Database Source and to add the record to the appropriates groups.
Unfortunately, the referred records (which are legacy records) are not present in GRSF hence the Database Source is not present and the groups are not added.

Waiting for @aureliano.gentile@fao.org, please see below my personal opinion.

The alternatives I see are:

• Add only the source of the legacy record that contributed for this GRSF record (i.e. FIRMS, RAM, FishSource)

I think this may confuse the user accessing GRSF.

• Add the URL of the original legacy record (i.e. http://firms.fao.org/firms/resource/13792/en) or the URI if such a URL does not exist (i.e. for RAM records http://www.bluebridge-vres.eu/ram/ANCHMEDGSA16)

Not so elegant as well since for RAM we have not a persistent URL (BB domain is related to the project and it will not be maintained for so many additional years)

• Add the catalogue URL of the legacy records from the GRSF_Admin VRE

I think this is safe since we maintain both GRSF Admin and GRSF. Those URLs are persistent and we can properly advise the user that their access require specific privileges.

• Publish the legacy records in GRSF VRE as well

This is clearly fine if FAO decides to select this option.

Personally speaking, I think that options 3 and 4 are not so elegant.

As you can see above, I think that only 3 and 4 are viable.

Thanks for your answers Lino. See some comments below:

Pasquale Pagano wrote:

Waiting for @aureliano.gentile@fao.org, please see below my personal opinion.

The alternatives I see are:

• Add only the source of the legacy record that contributed for this GRSF record (i.e. FIRMS, RAM, FishSource)

I think this may confuse the user accessing GRSF.

I do not see why they will be confused.

• Add the URL of the original legacy record (i.e. http://firms.fao.org/firms/resource/13792/en) or the URI if such a URL does not exist (i.e. for RAM records http://www.bluebridge-vres.eu/ram/ANCHMEDGSA16)

Not so elegant as well since for RAM we have not a persistent URL (BB domain is related to the project and it will not be maintained for so many additional years)

I agree with this.

• Add the catalogue URL of the legacy records from the GRSF_Admin VRE

I think this is safe since we maintain both GRSF Admin and GRSF. Those URLs are persistent and we can properly advise the user that their access require specific privileges.

I agree it is safe however the problem here is that users registered in GRSF VRE should also register in GRSF-ADMIN VRE to check them out.

• Publish the legacy records in GRSF VRE as well

This is clearly fine if FAO decides to select this option.

Personally speaking, I think that options 3 and 4 are not so elegant.

As you can see above, I think that only 3 and 4 are viable.

Thanks to all, appreciated. I discussed the matter also with Anton and I showed him also the citation aspect. We think that it would be enough to have under the box "Data and Resources" simply the list of the data source(s) as appropriate. In the public VRE the link to the legacy record was envisaged as confusing and indeed it was asked to be omitted. if you consider the citation https://support.d4science.org/issues/12278 , following the GRSF record citation we are envisaging something like this "Database sources: [FIRMS]" which then is followed by the original source record citation.

In conclusion, if feasible, at this stage it would be enough to have listed the sources and the "Groups" populated with that information. Admin users have the opportunity to browse legacy records and make all the checks while for the public could be enough like that. The citation, when completed, will give access to the source URL in the data owner websites.

Opening this pilot release to other users and the discussion at FSC11 will give further directions, if needed.

Does it make sense for you? Thanks.

I am OK with that.

Technically speaking this means that we should include the database sources in the JSON serialization (as we do with legacy records). For example:

"database_sources" : [ {
    "name" : "FIRMS",
    "description" : "Fisheries and Resources Monitoring System aims to ...",
    "url" : "http://firms.fao.org/firms/en"
  }, 
  {
    "name" : "FishSource",
    "description" : "FishSource is an online information resource about ...",
    "url" : "http://www.fishsource.com"
  },
  {
    "name" : "RAM",
    "description" : "RAM Legacy Stock Assessment Database is ...",
    "url" : "http://ramlegacy.org"
  }
],

CNR colleagues is this OK with you?

The solution is ok for me.
Please take into account that this behaviour will occur also on GRSF_Admin VREfor any records containing the field database_sources.

If this is ok for everyone, I'll modify the code.

I Understand this is an additional information added in the json of the grsf record, so at worst it won't be used in specific contexts. So it is fine with me. Fyi, this afternoon we'll have the first call with RAM colleagues to start validating GRSF VRE and approving new records in GRSF VRE Admin. let em know if these modifications implies erase/republish or other drastic actions in the GRSF KB.

@@luca.frosini@isti.cnr.it we already have this feature when publishing records in the GRSF_Admin (in particular when publishing legacy records)

@aureliano.gentile@fao.org
I think we can simply update the existing records in GRSF VRE, so nothing will be removed.

Yannis Marketakis wrote:

@@luca.frosini@isti.cnr.it we already have this feature when publishing records in the GRSF_Admin (in particular when publishing legacy records)

Hi @marketak@ics.forth.gr,

sorry I lost your comment.

Looking the code seems that database_sources** filed is only used to create additional resources.

Legacy records are added to organizations not to group.

So in that case, the field name should be different. Right?

Yannis Marketakis wrote:

So in that case, the field name should be different. Right?

It could. But if it is easier for you use the database_sources field I can use it.
In GRSF_admin should not cause any issues because the record is just added twice to the same group.

Just let me know what do you prefer.

I thought it would create issues if the field name is the same.
Since it does not, then use database_sources. It is fine.

The new feature is available in the production instance.
@marketak@ics.forth.gr you can update the records when you want.
Please be sure that the update rate is limited as was agreed with Costantino.

Hi Luca. Thanks a lot.
What exactly do you mean with the following?
Luca Frosini wrote:

Please be sure that the update rate is limited as was agreed with Costantino.

Hi Yannis,

the publishing rate should be limited to avoid failures on async operations caused by workloads.
Costantino told me you agreed on a delay between invocations.
If you are in trouble, giving that they are few records you could try to use 60 seconds.

Is that feasible?

As far as I remember we did not have any issues with updates.
However, it is fine by me to add an idle period between updates.

Thanks

I checked the GRSF VRE, groups are now available for source databases (ram, firms, fishsource) and also record pages are enriched with the box "Data and resources", similarly to GRSF Admin vre. Many thanks