Task #9583
closed
Represent ASFIS AquaMaps distribution with NetCDF format
Added by Gianpaolo Coro about 8 years ago.
Updated almost 8 years ago.
Category:
High-Throughput-Computing
Infrastructure:
Production
Description
The current AquaMaps distributions for ASFIS species will be also represented under NetCDF format through an automatic process.
This operation has a number of goals:
1 - increase the KPIs of the number of maps available on the catalogue #972;
2 - make these maps more exportable and reusable;
3 - discuss with FAO about the possibility to attach the produced information to the ASFIS official page (http://www.fao.org/fishery/collection/asfis/en);
4 - discuss with the AquaMaps Consortium about the possibility to invoke DataMiner from their website to get a NetCDF representation of their maps on-the-fly.
The official list of species to process is http://www.fao.org/fishery/static/ASFIS/ASFIS_sp.zip
Files
Re: 1; why only for ASFIS Species? That suggests a link to FAO that is (currently) not established
Re: 3: it will be difficult to add AquaMaps to FAO species fact-sheets, and I do not believe this can be done on short notice. It would seem e.g. that FAO endorses AquaMaps over other methods, and we want to avoid that. We will help with identifying other contexts to use the map generation processes.
- Status changed from New to In Progress
- % Done changed from 0 to 30
At the moment I am developing an automated process to collect all the maps generated using the AquaMaps NativeRange algorithm. This algorithm downloads the layers as CSV files and converts the polygonal representation into the standard Longitude-Latitude one using the centroid of the polygons themselves. The second step will consist of a csv to netcdf conversion.
re 1: if we talk about catalogue, we refer more to the production of accurate and exhaustive metadata rather than a specific data format available for download. Accurate and exhaustive metadata for Aquamaps distributions, with simple links to WMS / WFS output formats would be of added value for the catalogue, but not necessarily its NetCDF representation
re 2: Formats supported through WFS are more than enough, we are dealing with bi-dimensional data. A potential usability of NetCDF, maybe, but which use case exactly. Hence i don't really understand the focus on NetCDF, except onr way to test and exploit the NetCDF simple generator developed recently.
re 3: can be removed from this ticket. It is not ever on table for FAO aquatic distributions, although the later are a product of FAO; so I doubt that Aquamaps would be considered, and more doubt about the format, linking to a page like ASFIS. If some day a link is attached to ASFIS for distributions, it will probably be linking to a FAO product, and endorsed by FAO.
As for point 1 and 2, generating a different format of an already existing layer may not seem as useful as generating more accurate metadata. Nevertheless, the NetCDF is way more reusable and portable than the other formats already supported through WFS and it’s naturally suited to represent raster information (like uniformly spaced species distributions). In fact, on top of being self-describing and designed to represent n-dimensional data with n >= 2, it is also widely used by many communities and research institutions as a standard and there is plenty of tools to visualize and manipulate such format. Moreover, additional information about the data can be included in the file itself as attributes, creating more complex objects that don't need any external reference to be fully understandable, and thus reusable and portable. Obviously my claims are backed-up by the documentation (http://www.unidata.ucar.edu/software/netcdf/docs/user_guide.html). For all these reason we started by implementing the very general NetCDF converter I presented at the last TCOM, and we are currently trying to exploit all the features of this format. These has been discussed with Nicolas Bailly on July.
As for point 3, sure we can remove this one from the goals of this activity as also suggested by Anton. Thank you for pointing that out.
Paolo
- % Done changed from 30 to 70
I am currently generating the CSV for all the AQUAMAPS layers, to be subsequently converted in NetCDF.
This task is completed, I summarize here all the steps:
- Download all the Aquamaps native layers in CSV format
- Convert the polygon map into a latitude-longitude one, using the center of mass for every square
- Use the recently developed CSV-to-NetCDF converter to obtain the NetCDF version of each map
- Upload the new maps onto GeoNetwork enriching them with the needed metadata
During this process I spot several doubled layers, I attach to the ticket a list containing the names of the species and the number of duplicates I found for each one.
10,835 maps have been produced by this ticket and are indexed on the data catalogue.
@leonardo.candela@isti.cnr.it , since these come from a huge processing of the infrastructure data, can they be considered in the KPIs?
Yes, why they should not be counted?
#972 KPI updated accordingly.
I'm wondering whether we should improve the quality of the accompanying metadata. I just open #9975 for that
Also available in: Atom
PDF