Task #21213
closed
Refresh GRSF KB (April 2021)
100%
Description
Refresh GRSF KB with updated contents in terms of
- Updated contents from FIRMS using the observation IDs found at https://data.d4science.net/nnXm (FIRMS-MR-F_ID-OID-lang-16Apr2021.xlsx)
- new version of RAM (V4.494) and updated geo information. The bounding boxes information can be found at https://data.d4science.net/qS3A and the file is named RAMLDB_4-494_bbox_compilation_2021-03-16.xlsx
- new harvest from FishSource database source using the dedicated endpoint V5.
Refresh the contents of GRSF PRE VRE, after refreshing GRSF KB.
Files
Related issues
Updated by Aureliano Gentile about 4 years ago
Thank You Yannis, please let me recall what we drafted in this page for our memory during the refresh:
Scientific names vs fao3alpha code
https://support.d4science.org/projects/stocksandfisherieskb/wiki/GRSF_Maintenance#Scientific-name-vs-standard-codes-ASFIS-WoRMS
Updated by Yannis Marketakis almost 4 years ago
- File Obsolete GRSF Records.xlsx Obsolete GRSF Records.xlsx added
- File Obsolete Legacy Records.xlsx Obsolete Legacy Records.xlsx added
- Status changed from New to In Progress
- % Done changed from 0 to 80
The refreshed contents of GRSF KB have been published on GRSF PRE VRE (https://blue-cloud.d4science.org/group/grsf_pre/grsf_pre)
You'll notice that we have performed
- Normalization of species w.r.t. ASFIS coding system as requested
- Connect resources as they are indicated from FishSource
- We've prepared the list of obsolete records which refer to records that do not exist anymore in the original data sources
We haven't computed the similarities between resources because of some errors occured with the GRSF Intersection Method. I'll open a new issue assigned to Emmanuel to investigate this. Let me clarify that this does not block the refresh process, we can add the similarities (and update all the records accordingly) as soon as we resolve this issue.
Please feel free to check the records in GRSF PRE VRE. Unless you spot any problems, we can start updating GRSF Admin and GRSF Public VREs.
Updated by Yannis Marketakis almost 4 years ago
- Related to Bug #21380: GeoGRSF Intersects Batch added
Updated by Aureliano Gentile almost 4 years ago
Dear Yannis, thanks a lot.
I made some checks for the obsolete records and the normalization of species vs FAO3Alpha code.
I confirm the obsolete records for FIRMS. And I will write to RAM and FishSource colleagues asking for confirming their obsolete records. Particularly those which were already approved.
For the normalization of species by 3Alpha, it seems fine. I noticed still some "spp." in tags, this were due to this FIRMS record http://firms.fao.org/fishery/xml/resource/13887/en which I have now fixed. Is it possible to re-harvest it again? (sorry for that)
The googledoc (for our memory): https://docs.google.com/spreadsheets/d/1mDw3PjS5QKOyNWFeBMu_QWGXqY7Rp6CcE8kXx-s9pNo/edit#gid=736416633
Updated by Yannis Marketakis almost 4 years ago
Thanks @aureliano.gentile@fao.org for your revision.
As regards the wrong species, I'll re-harvest them and re-publish in GRSF PRE
Updated by Aureliano Gentile almost 4 years ago
With thanks, I am now starting reviewing the content of the GRSF Pre.
Updated by Yannis Marketakis almost 4 years ago
The record has been reharvested and the corresponding records have been re-published in GRSF PRE. More specifically the following:
- https://data.d4science.org/ctlg/GRSF_Pre/4850d2ef-d421-3eea-b434-fffbff2619a3
- https://data.d4science.org/ctlg/GRSF_Pre/3c4fdb36-8b6a-359e-a5cf-b1208e1c2862
In addition, I found that in the following records some FAO 3-alpha codes are missing.
- http://firms.fao.org/fishery/xml/resource/13447/en (SQUALIFORMES and RAJIFORMES)
- http://firms.fao.org/fishery/xml/resource/13455/en (SQUALIFORMES and RAJIFORMES)
- http://firms.fao.org/fishery/xml/resource/13460/en (SQUALIFORMES and RAJIFORMES)
The corresponding codes exist in other FIRMS records (e.g. http://firms.fao.org/fishery/xml/resource/13426/en or http://firms.fao.org/fishery/xml/fishery/954/en). So, could you update the aforementioned three records as well so that we fix them in GRSF KB?
Updated by Aureliano Gentile almost 4 years ago
Thanks again, I fixed these records as you spotted:
http://firms.fao.org/fishery/xml/resource/13447/en (SQUALIFORMES and RAJIFORMES)
http://firms.fao.org/fishery/xml/resource/13455/en (SQUALIFORMES and RAJIFORMES)
http://firms.fao.org/fishery/xml/resource/13460/en (SQUALIFORMES and RAJIFORMES)
fi:SpeciesRef - fi:ForeignID Code="SRX" CodeSystem="fao3alpha" - fi:SpeciesRef
fi:SpeciesRef - fi:ForeignID Code="SHX" CodeSystem="fao3alpha" - fi:SpeciesRef
I took also the opportunity to add CData elements where missing.
13455 can be also set as "Archived".
Updated by Yannis Marketakis almost 4 years ago
Many thanks for your prompt update.
I've updated the corresponding records. The GRSF records that have been updated are:
Updated by Aureliano Gentile almost 4 years ago
Thanks, I will review the PRE a little more. And in parallel the mail discussion on dominant records and data for fishery records.
Updated by Aureliano Gentile almost 4 years ago
@marketak@ics.forth.gr I guess we can close this ticket. Since the refresh is complete (apart republishing in Admin VRE and update map viewer), we are going to start again in reviewing records for new approvals, then in case of issues we will open new tickets (e.g. on misplaced polygons etc). What do you think?
Updated by Yannis Marketakis almost 4 years ago
Hi @aureliano.gentile@fao.org .
I agree. As soon as we'll complete updating the internal IDs of the public GRSF records, we will close this issue.
As regards the latter, please let me know if you are fine updating those records.
I would like to remind you that we need to delete and re-publish the public GRSF records in GRSF Admin. During this process the figures from the GRSF Admin catalogue will not be consistent.
Updated by Aureliano Gentile almost 4 years ago
Yes please we said Monday is fine. We have another internal meeting on Wednesday morning, I hope by then the Admin will be back at work.
Updated by Yannis Marketakis almost 4 years ago
- Status changed from In Progress to Closed
- % Done changed from 80 to 100
And we are done.
We can close the issue.
