Project

General

Profile

Actions

Task #9583

closed

Represent ASFIS AquaMaps distribution with NetCDF format

Added by Gianpaolo Coro about 8 years ago. Updated almost 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Paolo Scarponi
Category:
High-Throughput-Computing
Target version:
Start date:
Sep 04, 2017
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Production

Description

The current AquaMaps distributions for ASFIS species will be also represented under NetCDF format through an automatic process.

This operation has a number of goals:

1 - increase the KPIs of the number of maps available on the catalogue #972;
2 - make these maps more exportable and reusable;
3 - discuss with FAO about the possibility to attach the produced information to the ASFIS official page (http://www.fao.org/fishery/collection/asfis/en);
4 - discuss with the AquaMaps Consortium about the possibility to invoke DataMiner from their website to get a NetCDF representation of their maps on-the-fly.

The official list of species to process is http://www.fao.org/fishery/static/ASFIS/ASFIS_sp.zip


Files

doubled_layers (17 KB) doubled_layers Paolo Scarponi, Oct 12, 2017 11:03 AM
Actions #1

Updated by Anton Ellenbroek about 8 years ago

Re: 1; why only for ASFIS Species? That suggests a link to FAO that is (currently) not established
Re: 3: it will be difficult to add AquaMaps to FAO species fact-sheets, and I do not believe this can be done on short notice. It would seem e.g. that FAO endorses AquaMaps over other methods, and we want to avoid that. We will help with identifying other contexts to use the map generation processes.

Actions #2

Updated by Paolo Scarponi almost 8 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30

At the moment I am developing an automated process to collect all the maps generated using the AquaMaps NativeRange algorithm. This algorithm downloads the layers as CSV files and converts the polygonal representation into the standard Longitude-Latitude one using the centroid of the polygons themselves. The second step will consist of a csv to netcdf conversion.

Actions #3

Updated by Emmanuel Blondel almost 8 years ago

re 1: if we talk about catalogue, we refer more to the production of accurate and exhaustive metadata rather than a specific data format available for download. Accurate and exhaustive metadata for Aquamaps distributions, with simple links to WMS / WFS output formats would be of added value for the catalogue, but not necessarily its NetCDF representation
re 2: Formats supported through WFS are more than enough, we are dealing with bi-dimensional data. A potential usability of NetCDF, maybe, but which use case exactly. Hence i don't really understand the focus on NetCDF, except onr way to test and exploit the NetCDF simple generator developed recently.
re 3: can be removed from this ticket. It is not ever on table for FAO aquatic distributions, although the later are a product of FAO; so I doubt that Aquamaps would be considered, and more doubt about the format, linking to a page like ASFIS. If some day a link is attached to ASFIS for distributions, it will probably be linking to a FAO product, and endorsed by FAO.

Actions #4

Updated by Paolo Scarponi almost 8 years ago

As for point 1 and 2, generating a different format of an already existing layer may not seem as useful as generating more accurate metadata. Nevertheless, the NetCDF is way more reusable and portable than the other formats already supported through WFS and it’s naturally suited to represent raster information (like uniformly spaced species distributions). In fact, on top of being self-describing and designed to represent n-dimensional data with n >= 2, it is also widely used by many communities and research institutions as a standard and there is plenty of tools to visualize and manipulate such format. Moreover, additional information about the data can be included in the file itself as attributes, creating more complex objects that don't need any external reference to be fully understandable, and thus reusable and portable. Obviously my claims are backed-up by the documentation (http://www.unidata.ucar.edu/software/netcdf/docs/user_guide.html). For all these reason we started by implementing the very general NetCDF converter I presented at the last TCOM, and we are currently trying to exploit all the features of this format. These has been discussed with Nicolas Bailly on July.

As for point 3, sure we can remove this one from the goals of this activity as also suggested by Anton. Thank you for pointing that out.

Paolo

Actions #5

Updated by Paolo Scarponi almost 8 years ago

  • % Done changed from 30 to 70

I am currently generating the CSV for all the AQUAMAPS layers, to be subsequently converted in NetCDF.

Actions #6

Updated by Paolo Scarponi almost 8 years ago

This task is completed, I summarize here all the steps:

  1. Download all the Aquamaps native layers in CSV format
  2. Convert the polygon map into a latitude-longitude one, using the center of mass for every square
  3. Use the recently developed CSV-to-NetCDF converter to obtain the NetCDF version of each map
  4. Upload the new maps onto GeoNetwork enriching them with the needed metadata

During this process I spot several doubled layers, I attach to the ticket a list containing the names of the species and the number of duplicates I found for each one.

Actions #7

Updated by Gianpaolo Coro almost 8 years ago

10,835 maps have been produced by this ticket and are indexed on the data catalogue.
@leonardo.candela@isti.cnr.it , since these come from a huge processing of the infrastructure data, can they be considered in the KPIs?

Actions #8

Updated by Pasquale Pagano almost 8 years ago

Yes, why they should not be counted?

Actions #9

Updated by Leonardo Candela almost 8 years ago

#972 KPI updated accordingly.

I'm wondering whether we should improve the quality of the accompanying metadata. I just open #9975 for that

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)