StocksAndFisheriesKB

Before moving on with the actual merging, please clarify what do you mean with UUID (I suspect you are referring to Semantic ID)?

For example for the record http://data.d4science.org/ctlg/GRSF/abcc951a-cb4f-3ec4-ad48-b4a1726c136f, the Semantic ID is asfis:SKJ+fao:51;fao:57 while the UUID is abcc951a-cb4f-3ec4-ad48-b4a1726c136f.

The reason I am asking is because UUID are NOT preserved during the merging. In fact a new UUID is created for the new record to be created (i.e. the merged one)

Thanks Yannis,

Yes you are right, let's try to clarify and to identify operational rules.

In case of merging two GRSF records:

1. the former GRSF UUIDS are deleted/lost
2. a new UUID is generated
3. the semantic ID for the new merged record is built upon GRSF standards and inherits the standard values of the reference source record vs. the other record which it is supposed to be merged with. Ideally, the system should offer the way to indicate which species and area, and in which coding systems. But considering that this is so far not available, we may stick to the rule of "what" is to be merged "with".

The requested merges in this ticket are all based on GRSF records sourced from FIRMS for which RAM records are to be merged with by adding the time dependent data and ignore their species and area codes. But indeed this is not the only possible case, we may encounter situations where the merge may take species from one record and area from another source record, but maybe this is less frequent case. To be verified.

Thanks in advance for your feedback on the above.

Thanks for the confirmation Aureliano.

One more clarification. I ran the merging process for the first case shown in the description and the semantic ID that is created is the following "asfis:SKJ+fao:51;rfb:IOTC;fao:57".
This seems to be normal and compliant with the merging rules (#12856). Let me know what you think.
In addition, let me know if you wish to send you the full details of a merged record (one of them) so that you can inspect it.

Thanks Yannis.

In the example you provided there is indeed one of the main issue of the merge process: the merge is suggested because FAO areas 51/57 are the equivalent of IOTC area of competence, and the FAO areas would be the correct way to describe that stock (as per IOTC definition https://www.iotc.org/about-iotc/competence and FIRMS implementation).
Therefore, the merge should produce a semantic identifier by retaining the identity data from the first record and discarding the data from the second record:

"asfis:SKJ+fao:51;fao:57"

How to do that, based on which criteria? I do not know, unless you provide an interface where it can be specified what should be retained and what discarded. This only for the GRSF identity and similar issue for the fishery records.
For the time dependent data we said simply to merge all the data provided it is a collation with any datum (e.g. each row) is properly referenced.
We can have a skype on that if you want.

For the moment and as regards the records under merging that are described in this issue we could move on by manually editing the semantic IDs as needed.

However, we should find a more sustainable (and preferably automatic) solution for supporting the merging. It seems to me that merging requires human intervention after all (for resolving issues) like the one mentioned above. I see two potential solutions, which are described below.

Solution 1:¶

Your proposal could be an option. Technically, it requires to implement: (a) the corresponding services for GRSF KB and (b) GUI enhancements in the catalog to enable user feedback.

Solution 2¶

Another solution is to have association matrices that would map entities found in different systems (e.g. rfb:IOTC equalsTo fao:51&fao:57). Then we could have a prioritization on the standard codes to be used (i.e. in this case as regards assessment areas, fao precedes rfb) in order to construct the desired values

Thanks, solution 2 is pretty logic but then other factors drive the choice. For example for iattc and iccat, in place of the equivalent fao areas the preferred standard is the rfb coding system. Therefore, I think the best would be option 1 where somehow is possible to indicate what should be merged with. Waiting for such advanced interface, would it be enough for the time being implying a rule that the first record selected for merging is the one holding the right data? So the identity elements of the second one are ignored. What do you think?

Yes that would be an option.
I'll record this (in fact I'll create a wiki page with the workflow for the merging) and move on with the merging of these records.

In addition:

• should we apply this rule (about the dominant record) only for the semantic ID, or we should apply this for the assessment areas as well (and as a result omit rfb coded area from the assessment areas of the merged record)?
• should we apply this rule (about the dominant record) for other fields as well?

I think as a temporary rule, the "dominant" record is the one providing all and only the codes for the semantic identifier (stock or fishery). I am saying this to ensure a proper unique identification compatible with the competent authority (in this case IOTC), which need to be verified case by case. Hence this notion of "dominant record" sounds viable. While for the fields within the GRSF fact sheet, I understand your suggestion and indeed we could keep all the data (properly referenced) with the understanding of the above convention.

In conclusion, the dominant record is the one holding all the info for populating the semantic identifier.

If that is not the case, then a manual merge should be asked through a proper ticket. This as ad interim solution waiting for ad hoc services and interface.

Hi Yannis, my apologies for the delayed answer but I was sick the last few days.
I checked https://ckan-grsf-admin2.d4science.org/dataset/a7a7c506-dd40-33e7-bf05-5cba18181807 but it returns a 404 page, then I searched in the GRSF Admin vre by ""a7a7c506-dd40-33e7-bf05-5cba18181807" and I find 5 results, among those I guess the merged record is the following https://bluebridge.d4science.org/group/grsf_admin/data-catalogue?path%3D%2Fdataset%2Fa7a7c506-dd40-33e7-bf05-5cba18181807

This being said I notice:

GRSF name Katsuwonus pelamis Indian Ocean, Western Indian Ocean, Eastern OK

Short Name: Skipjack tuna Indian Ocean ; Skipjack tuna - Indian Ocean DUPLICATED, maybe we should apply the "dominant record rule" also here
GRSF Semantic identifier: asfis:SKJ+fao:51;fao:57 OK
Record URL: https://bluebridge.d4science.org/group/grsf_admin/data-catalogue?path%3D%2Fdataset%2Fa7a7c506-dd40-33e7-bf05-5cba18181807 OK, I understand it is a new URL

I noticed some issues on the catch and biomass data time series, but not related to the merge rather to the importing process and we can handle that in other tickets.

Hi Aureliano,
I hope you are better and have recovered.

Indeed, the URL of the record is https://bluebridge.d4science.org/group/grsf_admin/data-catalogue?path%3D%2Fdataset%2Fa7a7c506-dd40-33e7-bf05-5cba18181807
As regards the duplication in the short name, this has been derived from the concatenation of the records under merging. We have agreed to do it like this since the short name can be changed afterwards. However, I am perfectly fine in using the short name of the dominant record.

thanks, the week-end contributed to the full recovery ;-)

Regarding the short name, the dominant record is the one having all the good and right information on the identity of the stock or the fishery. Therefore, also considering from a maintenance view point, the identity data are all from the dominant and in case the reviewer can add more content afterwards.

Thanks for the update Aureliano.

I've updated the short name of the merged record and the corresponding rule in the merging process, so that it will re-use the short name of the dominant record.

I will proceed with the merging of the other records.

Thanks, I have approved the above listed 5 stocks/assessment unit and I have referenced this ticket in the annotation.
Please note that I do not see these records updated in the GRSF VRE, is it still a manual process or will it be synchronized automatically sometime?

https://bluebridge.d4science.org/group/grsf_admin/data-catalogue?path%3D%2Fdataset%2Fa7a7c506-dd40-33e7-bf05-5cba18181807
Skipjack tuna - Indian Ocean
asfis:SKJ+fao:51;fao:57

https://bluebridge.d4science.org/group/grsf_admin/data-catalogue?path%3D%2Fdataset%2F5bfc64f2-a4f5-36f2-b7c2-84a1f37496fd

Swordfish - Indian Ocean

asfis:SWO+fao:51;fao:57

https://bluebridge.d4science.org/group/grsf_admin/data-catalogue?path%3D%2Fdataset%2F7febfe97-2952-3392-bde2-dfc48d754ff5

Blue marlin - Indian Ocean

asfis:BUM+fao:51;fao:57

https://bluebridge.d4science.org/group/grsf_admin/data-catalogue?path%3D%2Fdataset%2F37f59a2f-8d43-3fa4-ad54-2c78318adf14

Striped Marlin - Indian Ocean

asfis:MLS+fao:51;fao:57

https://bluebridge.d4science.org/group/grsf_admin/data-catalogue?path%3D%2Fdataset%2F43c75b29-85b0-34aa-a725-f95b6d49090c
Black Marlin - Indian Ocean
asfis:BLM+fao:51;fao:57