Project

General

Profile

GRSF ยป

5th May 2020 - Discussing GRSF topics with FishSource

Meeting Notes

Participants:

FishSource (Susana Segurado, Patricia Amorim, Merul Patel)
FAO (Bracken van Niekerk, Anton Ellenbroek, Aureliano Gentile)

Main topics

The participants discussed the following topics.

  1. We understand there are some technical issues with record management but we noticed that Susana's role appears as "Catalogue Editor" but Patricia's not. Could this be behind the message she receives saying she is not permitted to manage records?

  2. Following up on Patricia's email, we wondered what are the criteria that are behind the "Short Name" as the rules are not apparent to us

  3. Regarding the traceability identifier discussed in https://support.d4science.org/projects/stocksandfisherieskb/wiki/17-09-26_GRSF_validation, could you remind us if is this in the works, or what is the status?

  4. Another question that arose is whether we will need to merge fisheries too? And is this possible?

  5. We've noticed among the approved records, http://data.d4science.org/ctlg/GRSF/03c0241d-2b9a-3404-b226-eb3495830d3f should have area FU5 in the semantic id, as there are multiple stocks in 27.4.b and 27.4.c and they appear indistinguishable. Examples like http://data.d4science.org/ctlg/GRSF/7ebdba30-f325-3d38-b900-15c74dce88da and http://data.d4science.org/ctlg/GRSF/e6012752-e0cf-36ae-b2bb-877b61c88690 show the specific area in the semantic id - perhaps we could revisit the rules for the areas displayed in the ID when there is a merge?

  6. We were also wondering when we can expect the new harvest from FishSource?

  7. We are still pending a resolution on how to structure Resources. I will revisit the proposal I made back in https://support.d4science.org/projects/stocksandfisherieskb/wiki/18-05-09-GRSF_validation and will send you an update soon, for discussion.

Answers

  1. Patricia's role has been modified, now with same roles of Susana's (Catalogue Editor, Approver). (done by Aureliano on 10 may 2020).

  2. The GRSF short name comes directly from the source record, i.e. FIRMS short names. Within the management panel, the short name is the only field that can be edited manually. The GRSF standard foresees the short name to follow the FIRMS naming convention (e.g. adopting species ASFIS English name). It has been noticed that the stocks and fisheries short names are absent and they are just copies of the GRSF name. In case of merged records, the short name should be edited in agreement with the data provider, e.g. with Michael Melnychuk if a FishSource record has been merged with the RAM database.
    ACTION: @merul to verify that the FishSource name is properly passed in the FishSource web service, if so then raise a ticket for FORTH to investigate and fix. MP: Yes, the fisheries endpoint does return the name of the fishery as shown on FishSource, with the icons in the name replaced by '|', eg:

    {
    "fishery_name": "Capelin | Barents Sea | Russia/Norway | Russian Federation | Purse seines",
    "exploiting_stocks": 1830,

  3. A Stock and a Fishery can be connected in the Management Panel. So far there is no specific id for such connection, however ad hoc web service can return this information indicating the two UUIDS, the GRSF names, the flag, etc.
    An issue was raised by FishSource regarding trying to link a stock and fishery when the fishing area does not match the assessment area, as there would be a loss of information in the semantic identifier and a concatenation of the two would be required (possible as a webservice on the end user side).
    ACTION: FishSource to make a pilot list of these potential concatenations within the FishSource website.
    With reference to this discussion https://support.d4science.org/projects/stocksandfisherieskb/wiki/17-09-26_GRSF_validation the concatenation should be of this type "(Fishery ID + Assessment area) e.g. asfis:COD + assessarea:xxxx + fao:21.3.M + grsf-org:INT:NAFO + rfb_comp:NAFO + isscfg:OTB + iso3:LTU"
    This pilot will serve SFP purposes and will support any future decisions for GRSF further development on this aspect.

  4. In principle it is possible to merge fisheries, but not manually at the moment.
    ACTION: FishSource to create an excel file of these merges in a ticket to be sent to FORTH.
    ACTION: FishSource made a note that it may be better to wipe all FishSource records from the GRSF Admin that haven't been approved or merged and to be replaced with data from new data harvest due to new API rules making such a drastic changes to the DB. - To be discussed with FORTH.

  5. This is a long standing issues where these areas (Functional Units, e.g. FU33) need to be more specific. FIRMS Secretariat to discuss this issue with ICES as the last interaction was unclear whether the areas where indeed finalized. FishSource to adopt these area codes once finalized.
    For merges, the RAM records indicate a "more correct" ICES area code, hence should be the dominant record when merging to retain the area details.
    Finally, if confirmed by ICES, such FU areas should become part of the GRSF standards.
    ACTION: Aureliano & Bracken to contact ICES and get guidelines on FUs.

  6. The new data harvest is in progress, about 500 new records were loaded into a test environment and reviewed by Aureliano.
    ACTION: FORTH to follow up on the feedback provided by FAO and inform when the new data harvest is complete.
    Furthermore, CNR was asked to configure a new VRE "GRSF PRE" for hosting the new data harvest before moving into GRSF Admin. This will facilitate the workflow for refreshing data.

  7. ACTION: Susana to revert to the GRSF team with any proposal/feedback regarding https://support.d4science.org/projects/stocksandfisherieskb/wiki/18-05-09-GRSF_validation

Answers FORTH

Q#1 - FORTH: The issues with the management panel have been resolved some time ago. Indeed currently it is not working. I've investigated this and found out that the service which is responsible for communicating with the GRSF KB cannot reach it. This seems to be a network-related problem, due to some updates in the firewall policy since the service invoked from the management panel is deployed in CNR premises while GRSF KB is hosted in FORTH. Last week, I've asked our System Administrators to update it. I reminded it them today.

Q#6 - FORTH: During the last period, we've made several major updates on this which apart from resolving issues including implementing new facilities for supporting the new construction workflow (e.g. preserving UUIDs, statuses of resources, catalog IDs and URLs, annotation messages and relevant information, the manual merging of stock records). This is fully implemented and tested for all records coming from their initial databases (i.e. FIRMS, RAM, FishSource) and for GRSF Stock Records. Currently, we're working on constructing GRSF Fishery records.
While we're are working with the GRSF construction workflow, we are using the latest version of the actual data from their original sources. As soon as we finish with step 2 above, we'll add the fresh contents in GRSF KB and subsequently in the catalogs.

In terms of time consumption for refreshing the GRSF KB:

  • As regards steps the 2nd and 3rd, taking into consideration that we still need to implement the GRSF construction part for fisheries and test it, I would say that it is reasonable to publish them by next week (May 11-14). If we can make it sooner, then we'll let you know obviously. Please find some statistics as regards the time required for constructing GRSF (as they stand right now)
  • Data Harvesting and Normalization: approx. 1 hour
  • Data transformation and values generation: approx. 3 minutes
  • Data Ingestion (into GRSF KB): approx 1 hour
  • Data Reconciliation within GRSF KB: approx 30 minutes
  • Construction of Legacy Records in GRSF KB with information from the previous version of GRSF KB (i.e. UUIDs, catalog URLs): 30 minutes
  • Construction of GRSF Records in GRSF KB with information from the previous version of GRSF KB (i.e. UUIDs, catalog URLs [admin and public], manual merges, annotations): 2 hours (only for stocks)
  • Update GRSF Stock records in VRE Catalogues: approx. 8 hours
  • Update GRSF Fishery records in VRE Catalogues: approx. 15 hours
  • Publish New GRSF Resources in GRSF Admin catalog: approx 10 seconds per records

Resources

Add picture from clipboard (Maximum size: 8.91 MB)