D4Science Infrastructure

Then the algorithm is not being installed.

@roberto.cirillo@isti.cnr.it , it may depend on the installation of the SAI which needs to be updated.

typo before, I meant I'm now going to update Latest SAI on production.

Obviously I cannot re-publish all the algorithms, because I'm not the author of many of them. The example I have reported should work as a guidance to figure out the issue.

Gianpaolo Coro wrote:

If a new DM installation was done, the algorithms are erased and installed from scratch. Is this the case? Perhaps something went wrong in the process?

No.

@gianpaolo.coro@isti.cnr.it ,however there is something true in what you say given that now there are 93 algorithms in the category Others seems a bit 'much.
Maybe the userperspective.properties and transducerers.properties files were changed in the upgrade?

Giancarlo Panichi wrote:

@gianpaolo.coro@isti.cnr.it ,however there is something true in what you say given that now there are 93 algorithms in the category Others seems a bit 'much.
Maybe the userperspective.properties and transducerers.properties files were changed in the upgrade?

Obviously yes. There's a list of files that overwrite the dataminer defaults and that they are installed after every upgrade. The files are:

  - algorithms.properties
  - clusterers.properties
  - dynamictransducerers.properties
  - evaluators.properties
  - generators.properties
  - modelers.properties
  - models.properties
  - nodealgorithms.properties
  - transducerers.properties
  - userperspective.properties

If they are meant to change when some algorithms are installed, they must be managed in a different way. And moved outside the tomcat/webapps/wps directory.

@andrea.dellamico@isti.cnr.it , when you update the DMs, you also install the algorithms and then update those files.
If all went well, the files should contain the correct information.
In this case of LANGUAGE_RECOGNIZER the two files above do not contain the information, so something was not successful in the update.

Are all the algorithms reinstalled after a service update? Could this be due to the number of algorithms that are now so many? Maybe some conflict arises?

Perhaps in the event of an update it may be appropriate to choose a different policy.

@lucio.lelii@isti.cnr.it , it may be necessary to move these files externally as @andrea.dellamico@isti.cnr.it suggests, and to evaluate if perhaps it would be better if they were a resource on the IS.

Giancarlo Panichi wrote:

@andrea.dellamico@isti.cnr.it , when you update the DMs, you also install the algorithms and then update those files.

No algorithm is installed during an update, the algorithms file outside the application.
Only the files listed above are provisioned and they overwrite the ones provided by the dataminer war file. But if they are changed when algorithms are installed, this isn't the correct procedure.

If all went well, the files should contain the correct information.
In this case of LANGUAGE_RECOGNIZER the two files above do not contain the information, so something was not successful in the update.

Are all the algorithms reinstalled after a service update? Could this be due to the number of algorithms that are now so many? Maybe some conflict arises?

Perhaps in the event of an update it may be appropriate to choose a different policy.

Nobody ever told me that an upgrade must trigger a reinstall of all the algorithms. It's an easy operation btw, everybody can trigger it just removing or emptying the wps_algorithms_install_log/already_installed_algorithms.txt file.

Let me know if I have to do it.

These are the FACTS we should consider (I have done specific verification on dataminer3-proto):

1 - several algorithms were installed in March and they have disappeared from the configuration files. The only way to do this is to completely overwrite the files, which I have never done, obviously and has been done by some script. This script could be the installation, the update or something else I don't know. BUT, these files have been overwritten.

2 - I don't own all these algorithms, I don't have the complete list of the ones missing, thus I cannot install them through SAI. Several of them are also in other VREs than RProto, thus reinstallation through SAI is not feasible!

3 - The algorithms are correctly installed if I run the installation instruction contained in the DMPool Manager string. Thus no issue on the installer

Thus, after this rational analysis the most probable issues are in the fact that either the installation of all the algorithms was not run or the installation cron is not working. Thus, I would force the run of the cron job to exclude the second.

Generally, If I have correctly understood the new version produced by @lucio.lelii@isti.cnr.it , this starts from an empty DataMiner and requires all the algorithms to be installed from scratch by triggering the cron job.

Giancarlo Panichi wrote:

@lucio.lelii@isti.cnr.it , it may be necessary to move these files externally, and to evaluate if perhaps it would be better if they were a resource on the IS.

I guess that an IS resource would be the best option.
Other than that, everything that is not part of the original dataminer war file (only exception the basic configuration that never changes after the installation, like wps_config.xml) should live elsewhere.
There is a lot of stuff that happens inside tomcat/webapps/wps right now, and shouldn't:

• algorithms execution temporary files and results
• property files
• gcube keys (apparently the ones already present inside tomcat/lib are not enough)
• The gebco_08.nc (I managed to move it outside and use a symbolic link, otherwise every dataminer upgrade would download it again, and it's almost 2GB).
• Maybe other that I don't know about.

Gianpaolo Coro wrote:

These are the FACTS we should consider (I have done specific verification on dataminer3-proto):

1 - several algorithms were installed in March and they have disappeared from the configuration files. The only way to do this is to completely overwrite the files, which I have never done, obviously and has been done by some script. This script could be the installation, the update or something else I don't know. BUT, these files have been overwritten.

2 - I don't own all these algorithms, I don't have the complete list of the ones missing, thus I cannot install them through SAI. Several of them are also in other VREs than RProto, thus reinstallation through SAI is not feasible!

3 - The algorithms are correctly installed if I run the installation instruction contained in the DMPool Manager string. Thus no issue on the installer

Thus, after this rational analysis the most probable issues are in the fact that either the installation of all the algorithms was not run or the installation cron is not working. Thus, I would force the run of the cron job to exclude the second.

@gianpaolo.coro@isti.cnr.it read above: the property files are overwritten at each dataminer upgrade, because that's what I was told to do.
The cron job has nothing to do with that, it executes the addAlgorithm script and we know that it works correctly (but for the [ ] case).

Generally, If I have correctly understood the new version produced by @lucio.lelii@isti.cnr.it , this starts from an empty DataMiner and requires all the algorithms to be installed from scratch by triggering the cron job.

The part where all the algorithms to be installed from scratch never arrived to me. Do you confirm that running the script executed from the cron job should fix?

@lucio.lelii@isti.cnr.it do you confirm that the algorithms installation should be triggered just after each re-installation?

@andrea.dellamico@isti.cnr.it please, run the cron job and let's see if everything works, because this issue is blocking several tools that build on these functionalities.

Sorry, perhaps I lost some answer, since this incident ticket is wrongly collecting also SAI requests, suggestions for DM best practices, and lifestyle advice. Let's keep all of this in separate tickets, please.

Removed the file and forced a run of /usr/local/bin/algorithms-updater on all the dev and preprod dataminers. The job is currently running on the production ones four at a time, it should complete in a couple of minutes. Then you'll have to wait for the algorithms installation procedure to complete.

I confirm that svn must be updated to trigger the script execution. If it can't be done, I'm going to change the script so that it will run unconditionally when the /home/gcube/wps_algorithms_install_log/already_installed_algorithms.tx is absent.

I have changed one creation time in the algorithms list and committed both in prod and in proto. This should trigger the installation if I have correctly understood.

You missed the 15:00 cron job run by a minute, so I triggered a run of the script.