D4Science Infrastructure

Hi,
well as usual the harvesting of fao GN metadata is set for the reserved group All (rights "view" and "interactive map"), and the category "fao". However there's no need to configure it again to make it run. At the moment it's set to run just once for performance issues we had in the past, but whoever has admin privileges can launch it and/or configure it. Anyway I just launched it.

What VRE/VO visibility is associated to the all group? All the root/VO/VRE levels?

According to GeoNetwork documentation "A public metadata is a metadata that has the view privilege for the group named all.", the harvested metadata is public.

it's ok in Geoexplorer, but not in the data catalogue which i was looking primarily (due to its search capacities). Can you clarify how the information is harvested in the latter? It's not clear why it is harvested in Geoexplorer and not in the data catalogue.

Question: i would have questions/feedback about the data catalogue, and the way our products are handled there. Can you please tell me how you want me to proceed? single support ticket? distinct set of support tickets? thanks

Well, simply put : the data catalog performs harvesting of GeoNetwork information, and until that the catalog content is not updated.
GeoExplorer performs direct queries on our GeoNetworks, so you directly access that information.

About further questions/feedback I think it really depends on what you mean. Probably @gianpaolo.coro@isti.cnr.it can tell which way would be better.

Re-Harvesting of GeoNetwork for FAO Layers on D4science Data Catalogue has finished. You can see new layers here:
https://ckan-d4s.d4science.org/organization/fao

FAO FAGIS is displaying 1,364 products instead of 1368 (layers harvested on GeoNetwork, see: http://geonetwork.d4science.org/geonetwork/srv/en/main.home and then click on "fao" category). This mismatch is due to 4 Dataset schema (gmx.xsd) Validation Error:

content: https://ckan-d4s.d4science.org/harvest/object/fbd7d354-de03-44df-a62e-23d8692c0bbd

4c9e8750-dbba-11dc-9d70-0017f293bd28
Dataset schema (gmx.xsd) Validation Error
Element '{http://www.isotc211.org/2005/gco}DateTime': '2008-02-15' is not a valid value of the atomic type 'xs:dateTime'. (line 113)
Element '{http://www.isotc211.org/2005/gco}Real': '0,004' is not a valid value of the atomic type 'xs:double'. (line 230)

content: https://ckan-d4s.d4science.org/harvest/object/92c2107c-332b-490b-bcbf-419b03dbd9af

vliz-eez-map-8314
Dataset schema (gmx.xsd) Validation Error
Element '{http://www.isotc211.org/2005/gmd}MD_TopicCategoryCode': [facet 'enumeration'] The value 'OCEAN' is not an element of the set 
{'farming', 'biota', 'boundaries', 'climatologyMeteorologyAtmosphere', 'economy', 'elevation', 'environment', 'geoscientificInformation', 
'health', 'imageryBaseMapsEarthCover', 'intelligenceMilitary', 'inlandWaters', 'location', 'oceans', 'planningCadastre', 
'society', 'structure', 'transportation', 'utilitiesCommunication'}. (line 472)
Element '{http://www.isotc211.org/2005/gmd}MD_TopicCategoryCode': 'OCEAN' is not a valid value of the atomic type 
'{http://www.isotc211.org/2005/gmd}MD_TopicCategoryCode_Type'. (line 472)

content: https://ckan-d4s.d4science.org/harvest/object/0aac85fa-22c9-4fda-b953-7df1b95f9a42

84017a9f-09bf-4b18-a346-dff0e661d86f
Dataset schema (gmx.xsd) Validation Error
Element '{http://www.isotc211.org/2005/srv}SV_ServiceIdentification': This element is not expected. Expected is one of 
( {http://www.isotc211.org/2005/gmd}AbstractMD_Identification, {http://www.isotc211.org/2005/gmd}MD_DataIdentification, 
{http://www.isotc211.org/2005/gmd}MD_ServiceIdentification ). (line 80)

content: https://ckan-d4s.d4science.org/harvest/object/3ac2ac42-7224-4455-82fa-02e3f4b4efdb

ac02a460-da52-11dc-9d70-0017f293bd28
Dataset schema (gmx.xsd) Validation Error
Element '{http://www.isotc211.org/2005/gco}Integer': '' is not a valid value of the atomic type 'xs:integer'. (line 120)
Element '{http://www.isotc211.org/2005/gco}DateTime': '2008-02-15' is not a valid value of the atomic type 'xs:dateTime'. (line 150) 

Thanks for your feedback on validation issues. FYI, these has been fixed, as follows:

• 4c9e8750-dbba-11dc-9d70-0017f293bd28 --> fixed numeric field
• vliz-eez-map-8314 --> deleted metadata (old draft)
• 84017a9f-09bf-4b18-a346-dff0e661d86f --> deleted metadata
• ac02a460-da52-11dc-9d70-0017f293bd28 --> add missing count of vector objects

Thanks for the update on data catalogue. It's not clear if the 2 harvesting (Geonetwork <- FAO Geonetwork, CKAN <- Geonetwork) are automatic, and if yes what is the periodicity. Thanks in advance if you can clarify (maybe documenting it somewhere would be useful, if not yet done).

NB: On the data catalogue (which is very useful in term of search capacity): my first reaction is that i see the products are under a "FAO" group, and "FIGIS" subgroup, but strangely i'm not part of them while i'm the individual owner/editor of the harvested metadata. I will look more into the data catalogue, and send feedback if needed.

The catalog is a new product released in August. There is not yet a scheduled harvesting so far. We run the harvesters and check the results to avoid problems (we are verifying how it works with metadata updates, with deleted metadata, and so on). In the future (release 4.1) we will schedule the harvesters and they will run every night. Documentation will be added as well.

Dont' know why but now FAO products are no longer available in the data catalogue, CKAN based, not the CSW one. I'm referring to the 'production' environment.
I see this as notification: "Francesco Mangiacrapa deleted the product Geonetwork Harvester for Fao Figis Layers"

Hi Emmanuel,
FAO products with its Geonetwork Harvester have been moved in the BB catalogue: https://ckan-bb.d4science.org/organization/fao
The main catalogue https://ckan-d4s.d4science.org/ will be only a "simple" container, that is a CKAN harvester for other CKAN instances (ckan-bb, ckan-sobigdata).
However, FAO products (with FAO Organization, etc.) will reappear again in the main catalogue when CKAN D4Science will restart harvesting for ckan-bb and ckan-sobigdata
ok?

Thanks, i understand this diff, but what are we supposed to have in the i-Marine portal https://i-marine.d4science.org/group/imarine-gateway/data-catalogue? BB or the whole D4S? IMHO it is the BB one, that's why it's not clear why FAO products are not available in the i-Marine data catalogue which should point to https://ckan-bb.d4science.org/ or no?

perfect, thanks

I still don't see products in the data catalogue https://i-marine.d4science.org/group/imarine-gateway/data-catalogue, there is no 'FAO' organization. If harvested through D4Science, the latter organization mentions '0 products'.

At the moment, Fao products are available on CKAN BB instance: https://ckan-bb.d4science.org/organization/fao.
CKAN BB is reachable in the i-marine/services portal via VRE's. For example: https://services.d4science.org/group/biodiversitylab/data-catalogue
The Fao Products will be available also on D4Science Data Catalogue after that ckan harvester cron will harvest from CKAN D4Science to CKAN BB.

Ok please let me know when done, afterwhat we will close this ticket.