D4Science Infrastructure

Indeed, we have not been banned because SPD is now able to generate the DWC-A of carcharodon group. Perhaps the failure for WoRMS-Animalia was an issue of the WoRMS service. We can go for the smaller jobs solution.

@valentina.marioli@isti.cnr.it, this is a support request and as such it deserves your analysis and resolution in a short timeframe. Could you check the logs and reports if the WoRMS plugin can do something better than fail?

@pasquale.pagano@isti.cnr.it we are talking about a job that downloads the whole WoRMS repository (starting from animalia and requesting all children).
This job retries every call to the plugin 10 times waiting 2 secs between the retries.
The job failed because we had 10 consecutive connection timeouts contacting the WoRMS webservice, It meas that something was not working very well in the WoRMS webservice.

I'm running the DWC-A generation process of Valentina on 1013 families of interest to FAO, by invoking the SPD service.

Here https://goo.gl/hai08h you can find all the direct children of Animalia in Darwin Core Archive format. The archives have been generated by the scripts in my private project "createDwCA" using threads.

39 out of 45 WoRMS Phylum are present in the Darwin Core Archives. We should understand which ones are missing and regenerate them. Meanwhile, the per-family process is still running.

A complete list of direct children of Animalia:
18814 - Acanthocephala
765220 - Animalia incertae sedis
882 - Annelida
391884 - Aschelminthes
1803 - Brachiopoda
146142- Bryozoa
22699 - Cephalorhyncha
2081 - Chaetognatha
1821 - Chordata
1267 - Cnidaria
1248 - Ctenophora
22586 - Cycliophora
14221 - Dicyemida
1806 - Echinodermata
1271 - Entoprocta
2078 - Gastrotricha
14262 - Gnathostomulida
1818 - Hemichordata
162564 - Nematomorpha
152391 - Nemertea
14220 - Orthonectida
1789 - Phoronida
14260 - Rotifera
1268 - Sipuncula
1276 - Tardigrada
592916 - Xenacoelomorpha
799 - Nematoda
51 - Mollusca *
1065 - Arthropoda *
22737 -Placozoa *
793 - Platyhelminthes *
558 - Porifera *

(*)DwCA missing

Not accepted Phylums:
(AphiaID - Scientific Name -> status -> Accepted name)
152230 - Coelenterata -> nomen nudum -> 1268 Sipuncula
2601 - Ectoprocta -> unaccepted -> 146142 Bryozoa
162561 - Kinorhyncha -> unaccepted -> 101060 Kinorhyncha (Class)
162577 - Lophophorata -> unaccepted
162562 - Loricifera -> unaccepted -> 101061 Loricifera (Class)
14218 - Mesozoa -> unaccepted -> 14221 Dicyemida (Phylum)
596753 - Nemata -> unaccepted -> 799 Nematoda (Phylum)
1053 - Nemertina -> alternate representation -> 152391 Nemertea
246812 - Nemertini -> unaccepted -> 152391 Nemertea
196279 - Pentastomida -> unaccepted -> 22602 Pentastomida (Subclass)
1270 - Pogonophora -> unaccepted -> 129096 Siboglinidae (Family)
163280 - Rhynchocoela -> unaccepted -> 152391 Nemertea
152283 - Sipunculida -> unaccepted -> 1268 - Sipuncula

The direct children of Animalia are 45 Phylums, but maybe we should not consider not accepted Phylums.

New DwCAs generated:

• 1789 - Phoronida
• 799 - Nematoda

There are still some missing Phylums:

• 51 - Mollusca
• 1065 - Arthropoda
• 22737 -Placozoa
• 793 - Platyhelminthes

The process to generate the DwCAs of the direct children of Arthropoda is still running.
The other processes to generate the missing Phylums have failed, maybe because of network issues.

New DwCA generated:

• 22737 - Placozoa

Missing Phylums:

• 51 - Mollusca
• 1065 - Arthropoda
• 793 - Platyhelminthes

The last three Phylum cannot be generated due to issues on the WoRMS service. I have asked Unesco people to give it a look.
Meanwhile, I have obtained the list of currently unmatched ASFIS species through BiOnym. Some of them are indeed contained in WoRMS but were not included in the DWCA for collateral issues related to the ones reported in #8040. Valentina is generating the DWCA for them, where possible.