17 June 2022 Discussing GRSF maps, polygons, codes etc.¶
Participants:
FAO (Emmanuel Blondel, Aureliano Gentile, Bracken van Niekerk)
FORTH (Yannis Marketakis)
Meeting Notes¶
From: Blondel, Emmanuel (NFISI)
Sent: 16 June 2022 8:06 PM
To: Gentile, Aureliano (NFISI) ; Giannis Marketakis
Cc: Ellenbroek, Anton (NFISI) ; Nieblas, Anne (NFISI) ; Bracken van Niekerk
Subject: RE: RE: Updating the GRSF map viewer
Dear Aureliano,
As I told you in our chat discussion:
the work to move from bounding boxes has not been completed, because of classical issues when an exercice is moved to handle complex geometries, issues that we know and for which I always warn when such kind of move is wished. Typical issues: 1) heavy processing - resouce comsuption --> process crash due to memory issues, 2) geometry issues, more complex the geometries are, more risk you have to get random - unexpected geometry processing issues, for which there is NO solution, unless applying cleaning algorithms which are themselves more consuming and do not guarantee a perfect cleaning (eg. you can get a valid geometry at the end but with loss of complete part of your multipolygon..)
I'm even surprised to see that these services were used with the detailed polygon mode, because it has not been finalized.
As I told you, Yannis could probably use it for Fishsource because very low resolution polygons, but which is not the case for FIRMS probably that i'm pretty sure it crashed due to changes in geometry resolutions we had to handle in our NFI systems for priority reasons external to the GRSF specific needs (ie alignment on UN Geospatial standards). If this is the case, this confirms why you still have records with bounding boxes, if these records are based on FIRMS
In general the processes to inherit spatial information are NOT based on the area standard codes you use to annotate and build the semantic identifier, for historical reasons first, because FishSource and RAM were never annotated with known area codes, FishSource delivering their own custom polygons in KML through API; and RAM with bounding boxes listed in Excel with a lot of invalid bounding boxes in addition. For this reason, the processes designed to inherit spatial information from the 3 sources were designed and still are as follow:
FIRMS: use of standard area codes as referenced as WaterAreaRefs in FIRMS factsheets XML, and multi-data fetching based on the area codes using Geoserver OGC data services, next: final intersecting /merging algorithm to get one single multipolygon.
FishSource: use of KML polygons as provided by their API
RAM: use of bounding boxes
So far all resulting polygons even if detailed (in case of FIRMS and FishSource) were summarized as bounding boxes which are then used as input to compare spatially the records and derive all possible combinations. A work has been started to keep detailed polygons, but clearly with limitations. As I said this activity has not been completed (so in principle, i'd say no attempt should be done to use it, or at least we need to acknowledge to understand why you get certain results and not others)
Hence, if you get:
FIRMS with bbox in place of standard polygons: this is normal, we get standard polygons, merge them into a single geometry and still - a bounding box -. The use of polygons is not robust and in place to be used exploited for the GRSF, so you get a resulting bounding box. Let me re-highlight that the work on polygons is a R&D work under BlueCloud research project. To make it clear: it's not a classical development that we want, we implement and that's it we use it. No, this work requires investigation to overcome heavy processing issues and make the work as batch process posssible. On this, clearly we are not there, there is work to do, and at now there is no straighforward solution. Cf. eg. crashes mentioned by Yannis.
FishSource - Polygon not consistent with semantic identifier: this is normal, spatial data processing is done based on KML polygons provided by Fishsource APIs; and not annotating area codes. If this KML polygon is not consistent with the area codes defined, you may see discrepancies.
RAM - bounding box only: that's normal too, no process has been defined to exploit annotating known area codes that are inserted in time with GRSF, and were not part of the source we process. We process the only spatial info we have: bounding boxes
FIRMS + RAM - bounding box only: still normal, we didn't move to detailed polygons for FIRMS, we still rely on summarized bounding boxes, we don't have a similar area-code based process for RAM; so you get bounding boxes only.
FIRMS + RAM + Fishsource: polygon; yes because Fishsource provide detailed polygons that might have been possible to consume in GRSF quite easily: KML data provided by Fishsource is light-weighted. Resulting geometry will be a polygon; here probably not because of intersection resulting from the comparison work, but more because between 2 bboxes and a detailed polygon from Fishsource, the latter was the finest polygon defined, included within the 2 former bounding boxes, so it was retained.
From what I see, there is nothing weird as long you acknowledge the fact that
implementation of detail polygons handling is not completed and operational;
RAM data has been enriched in time in same spirit as FIRMS marineresources/fisheries have WaterArearRef, but still with no process to handle this enriched information and derived polygons in the same way we do from FIRMS for both FIRMS Stocks/Fisheries and GRSF maps;
Hope this clarifies,
Emmanuel
De : Gentile, Aureliano (NFISI)
Envoyé : jeudi 16 juin 2022 17:51
À : Blondel, Emmanuel (NFISI) ; Giannis Marketakis
Cc : Ellenbroek, Anton (NFISI) ; Nieblas, Anne (NFISI) ; Bracken van Niekerk
Objet : Re: RE: Updating the GRSF map viewer
Some examples for the discussion.
My first question would be, why the Competency Queries are returning BBox coordinates in place of detailed polygons in case of standard area codes?
(Note the last example in which the polygon is displayed also in the map viewer)
FIRMS - BBox in place of standard polygon
Short Name: Emperor red snapper - Seychelles (Mahe Plateau)
GRSF Semantic Identifier: asfis:LUB+fao:51.5
Record URL: https://data.d4science.org/ctlg/GRSF_Admin/eab1afce-01f5-3350-8af0-bf93d7bd6583
CQ:
{"type":"Polygon","coordinates":[38.7761,-10.47],[38.7761,10],[65,10],[65,-10.47],[38.7761,-10.47]}
FishSource - Polygon (not consistent with semantic identifier)
Short Name: Brazilian sardinella - Southeastern Brazil
GRSF Semantic Identifier: asfis:BSR+fao:41.1;fao:41.2
Record URL: https://data.d4science.org/ctlg/GRSF_Admin/6da2242c-52f7-3426-a128-a6eaa896318b (some FAO area codes are missing, content related issue)
CQ:
{"type":"Polygon","coordinates":[[[-48.7400843034261,3.62940685435964],[-49.7138771201283,4.61389068675328],....
RAM - BBox
Short Name: Chum Salmon Yukon River Summer Run
GRSF Semantic Identifier: asfis:CHU+unk:USA-AKSTATE-YUKONRSR
Record URL: https://data.d4science.org/ctlg/GRSF_Admin/0ac3d97e-3c0c-3c60-884c-8677a07f3991
CQ:
{"type":"Polygon","coordinates":[177.5,35.7],[177.5,69.6],[-112.5,69.6],[-112.5,35.7],[177.5,35.7]}
FIRMS + RAM - merged records BBOx in place of polygon
Short Name: Red Seabream - Portuguese Waters
GRSF Semantic Identifier: asfis:SBR+fao:27.9
Record URL: https://data.d4science.org/ctlg/GRSF_Admin/491c7e71-951c-3474-8b4a-679c4d977b82
CQ:
{"type":"Polygon","coordinates":[-20.4,35.7],[-20.4,43.4],[-4,43.4],[-4,35.7],[-20.4,35.7]}
FIRMS + RAM + FishSource - polygon ! And noting that the area GSA 17 is smaller than the polygon represented in the GRSF map (see https://www.fao.org/gfcm/data/maps/gsas/en/)
Short Name: Sardine - Northern Adriatic Sea
GRSF Semantic Identifier: asfis:PIL+gfcm:17
Record URL: https://data.d4science.org/ctlg/GRSF_Admin/5fa92224-2f93-3271-9877-afcea94aba8d
CQ:
{"type":"Polygon","coordinates":[[[13.50248737515096,45.65797651560309].....}
Map viewer: https://grsf-admin.d4science.org/map-viewer/?dataset=grsf_resource_points&mode=2D&baseview=UN%20Clear%20Map&views=%5B%22pid%3Dgrsf_resource_points%2Clyr%3Dgrsf_resource_points%2Cstrategy%3Dogc_filters%2Cpar%3D(uuid%20IN('5fa92224-2f93-3271-9877-afcea94aba8d'))%2Cgeom%3Dgeom%2Cgeomtype%3Dgml%3APointPropertyType%2Cq%3Dtrue%22%2C%22pid%3Dgrsf_fishery_points%2Clyr%3Dgrsf_fishery_points%2Cstrategy%3Dogc_filters%2Cq%3Dfalse%22%2C%22pid%3Dgrsf_fishery_points%2Clyr%3Dgrsf_fishery_points%2Cstrategy%3Dogc_filters%2Cq%3Dfalse%22%5D&extent=-53.5117830526963,4.429265514493309,74.04353653786063,65.54824950613535¢er=10.265876742582165,34.98875751031433&zoom=4.524088725099892&popup_id=grsf_resource_points&popup_coords=15.52145516,42.80203297
Actions¶
- FAO - Provide to FORTH examples of GeoJson geometries: multipolygon (sure to work) multi-features - feature to collection (to test in CKAN map extension, not sure to work). Depending on that geospatial data will be postprocessed or not into a single multipolygon geometry (in any case needed for record comparison) DONE
- FAO - Prepare GIS web layers ONGOING
- in FAO Geoserver: need to set-up secondary data management workflow with publication of FAO areas as low resolution
- in GRSF Geoserver: publish new layers based on shapefiles identified by Arturo
- FORTH to provide web-service to list all records, and for each records list the area codes used to identify semantically the records
- FAO to build a new geo-service based on 3. that will collect/process geometries for each record, by contacting each Geoserver/layer
- FORTH to use the new geoservice developed in 4. at GRSF refreshing time
After that checkpoint, and see how to move forward with analysis of spatial similarities between records
