D4Science Infrastructure

@lars.valentin@bfr.bund.de I see that I can download a bundle that contains all the knime extensions, but not the community one.
I also did not find a way to install them programmatically. I can extract the relevant files from the zip bundle and from the bintray.com URLs that you provided, but it's not a task that can be automated easily and it does not solve the problem of the community extensions.

Andrea Dell'Amico wrote:

@lars.valentin@bfr.bund.de I see that I can download a bundle that contains all the knime extensions, but not the community one.
I also did not find a way to install them programmatically. I can extract the relevant files from the zip bundle and from the bintray.com URLs that you provided, but it's not a task that can be automated easily and it does not solve the problem of the community extensions.

Dear @andrea.dellamico@isti.cnr.it,

as a first step I suggest that you do it the KNIME way via the KNIME GUI, its the most easy way. After installing KNIME 3.6. to the server https://www.youtube.com/watch?v=wibggQYr4ZA , you just need to add the repositories in the menu, and add all the plugins afterwards https://www.youtube.com/watch?v=8HMx3mjJXiw . The plugins will be then downloaded from the repositories. And there is the mentioned FSK-lab setting were the R path needs to be checked (after installing all relevant R-packages like Rserve, miniCran ...).

In the past, we have offered to help you with maintaining your KNIME installation. I suggest we have a technical web meeting to talk about the options. By the way, there are ways to handle KNIME more automatically /programatically, I am using ANSIBLE.

Regarding R, I understand that now all required plugins from the list are present? Thank you!
What version of R is installed at the moment and how can we check that ourself in the future and make sure it will not be changed without informing us before?

Best wishes
Lars

Lars Valentin wrote:

In the past, we have offered to help you with maintaining your KNIME installation. I suggest we have a technical web meeting to talk about the options.

OK by me. Next week? Not on Monday morning.

By the way, there are ways to handle KNIME more automatically /programatically, I am using ANSIBLE.

We are using ansible too.
The main problem here is how to retrieve the plugins files, I know that I can do that from the desktop but I was hoping to be able to avoid that step.

Regarding R, I understand that now all required plugins from the list are present? Thank you!

Yes, they are there.

What version of R is installed at the moment and how can we check that ourself in the future and make sure it will not be changed without informing us before?

All the nodes have R 3.4. I guess that the next major step will be upgrading the servers distribution, but that will not happen in the short term and you will be informed many weeks in advance.

Best wishes
Lars

Andrea Dell'Amico wrote:

Lars Valentin wrote:

In the past, we have offered to help you with maintaining your KNIME installation. I suggest we have a technical web meeting to talk about the options.

OK by me. Next week? Not on Monday morning.

How about Tuesday then? Could you set something up and invite me?

By the way, there are ways to handle KNIME more automatically /programatically, I am using ANSIBLE.

We are using ansible too.
The main problem here is how to retrieve the plugins files, I know that I can do that from the desktop but I was hoping to be able to avoid that step.

Great that you use Ansible as well, makes things more easy. I found in the internet Ansible files from this guy and learned from it https://github.com/NLeSC/Chemical-Analytics-Platform/tree/master/roles/knime/tasks . There is an executor in KNIME which you can call by console and force it to download and install or uninstall plugins via repositories. It usually works well, but sometimes it doesn't work and I have to fix it manually.

Best wishes
Lars

Lars Valentin wrote:

Andrea Dell'Amico wrote:

Lars Valentin wrote:

In the past, we have offered to help you with maintaining your KNIME installation. I suggest we have a technical web meeting to talk about the options.

OK by me. Next week? Not on Monday morning.

How about Tuesday then? Could you set something up and invite me?

Tuesday at 12:00? Is it OK? I'll send you a google hangouts link 10 minutes before the start.

By the way, there are ways to handle KNIME more automatically /programatically, I am using ANSIBLE.

We are using ansible too.
The main problem here is how to retrieve the plugins files, I know that I can do that from the desktop but I was hoping to be able to avoid that step.

Great that you use Ansible as well, makes things more easy. I found in the internet Ansible files from this guy and learned from it https://github.com/NLeSC/Chemical-Analytics-Platform/tree/master/roles/knime/tasks . There is an executor in KNIME which you can call by console and force it to download and install or uninstall plugins via repositories. It usually works well, but sometimes it doesn't work and I have to fix it manually.

Nice! I fear that knime itself is not available anymore without any interaction with the knime site, but I'll give it a shot.

Dear Andrea,

Tuesday at 12:00? Is it OK? I'll send you a google hangouts link 10 minutes before the start.

Sounds fine to me! I just hope I can connect then, sometimes we have issues with our firewall.

Nice! I fear that knime itself is not available anymore without any interaction with the knime site, but I'll give it a shot.

We did not try, we solved it by downloading the file manually and stored it for the later use of ANSIBLE.

See you tomorrow then!

Best wishes
Lars

Dear Andrea,

in the future we want to allow the user to run there own R models in our service (workflow). The issue we face is that we do not know in adcance what R packages the user will use. Therefore we can't ask you in advance to install those packages for us. Do you have an idea how to handle that? We came up with the idea to just install all CRAN packages avaiable (>11.000 /28 GB).

What do you think?

Best wishes
Lars

Lars Valentin wrote:

Dear Andrea,

in the future we want to allow the user to run there own R models in our service (workflow). The issue we face is that we do not know in adcance what R packages the user will use. Therefore we can't ask you in advance to install those packages for us. Do you have an idea how to handle that? We came up with the idea to just install all CRAN packages avaiable (>11.000 /28 GB).

What do you think?

We cannot do it: we recompile all the packages every night to have the latest version of each one. Currently the task needs a couple of hours, with so many packages it could last days.

Right now we manage the R package requests via a cron job that takes a list from a subversion repository. New packages are installed on the dataminers after 10 minutes since the list on subversion has been updated. You can find the current lists here: http://svn.research-infrastructures.eu/public/d4science/gcube/trunk/data-analysis/RConfiguration/RPackagesManagement/ (not only R packages, but deb dependencies, and some R packages are installed from github). The list is not complete, because many packages install other ones as dependencies and we do not track them.

If your workflows are too dynamic to be handled by the above methods, we'll have to think about a different way to manage the R packages. That discussion should go into the VRE, it's not a tickets system argument.

I'll start with the knime installation next week.

Andrea Dell'Amico wrote:

I'll start with the knime installation next week.

Hey Andrea,

how is the installation going? When can we test the new versions?

About the R packages issue, I am trying to suggest CNR that the VRE automatically reads in a file with the needed packages if a new model was uploaded and these packages are then added to your install list, in order to avoid too many packages.

Best Lars

Lars, we are suffering problems with one of our infrastructures so I cannot complete the installation until they have been solved. I expect that in the next couple of days the situation will be back to normal.

About the R packages, the one you are proposing is the method that we are already following. Adding an additional source is not a big deal, I don't know if it's the best way possible. The alternative is giving you write access to our subversion repository I guess.

Hey Andrea,

I was wondering if there is the option to manually set up one datamainer with the new version of KNIME and the mentioned nodes. Since we need to develop the underlying KNIME workflows and to test them in the Dataminer to showcase the later procedures soon, it would be fine for us to target one specific dataminer via REST interface for now.

Maybe this could serve as workaround until the main setup is available?

Best wishes
Lars

Hi Lars, the infrastructure problems were solved yesterday evening. I'll start working on the KNIME installation next Monday.

Andrea Dell'Amico wrote:

Hi Lars, the infrastructure problems were solved yesterday evening. I'll start working on the KNIME installation next Monday.

Hey Andrea,

how is it going? Did you succeed? Do you need further help?
We are going to have a webmeeting with CNR on friday. Maybe it makes sense you join in https://doodle.com/poll/wsmq5yzdbum8kz53 ?

Best Lars

PS. I was wondering how you make sure each dataminer receives the updates of KNIME plugins and R packages in parallel, since there might be processes running. Do you update each dataminer individually via ANSIBLE each time there is no active process or do you handle that another way?

Lars Valentin wrote:

Andrea Dell'Amico wrote:

Hi Lars, the infrastructure problems were solved yesterday evening. I'll start working on the KNIME installation next Monday.

Hey Andrea,

how is it going? Did you succeed? Do you need further help?

I'm having trouble with the plugins installation. Only a couple install successfully, I'm trying to understand if there's something missing on my part and then I'll report.

We are going to have a webmeeting with CNR on friday. Maybe it makes sense you join in https://doodle.com/poll/wsmq5yzdbum8kz53 ?

Best Lars

PS. I was wondering how you make sure each dataminer receives the updates of KNIME plugins and R packages in parallel, since there might be processes running. Do you update each dataminer individually via ANSIBLE each time there is no active process or do you handle that another way?

The R packes are usually not a problem, because the old package is available until the installation of the new one has been completed.
As the KNIME plugins seems to be more problematic, I'm going to add a check so that the installation starts only when no dataminer job is running.

Hey Andrea,

could you tell me what is installed already and what is missing? I might have an idea then.

Best Lars

Only two installed correctly, I still have to look at the error logs of the other ones:

org.knime.features.base.feature.group.installed
org.knime.features.ext.svg.feature.group.installed

Never mind, I guess I've found at least one mistake from my part: the plugins repositories do not use the point version (I used 3.6.0 instead of 3.6).

One thing about the knime executable path: do you need an absolute path? because the knime command is available in the user's environment path, you have to just invoke knime.

@lars.valentin@bfr.bund.de any news? did you run your workflows?

Dear Andrea,

I was assuming that you tested with the provided testworkflow above if KNIME is working and our FSKX nodes properly installed, therefore I didn't do it too.

What I see now is that the workflow fails - please refer to ticket https://support.d4science.org/issues/12366#note-10

When I inspected the log file (its attached i the other ticket) I found these lines:

[604] "WARN \t main Node\t Node can't be executed - Node \"FSKX Reader\" not available from extension \"FSK Lab Feature\" (provided by \"BfR\"; plugin \"de.bund.bfr.knime.fsklab.nodes\" is not installed)"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

[605] "WARN \t main Node\t Node can't be executed - Node \"FSKX Reader\" not available from extension \"FSK Lab Feature\" (provided by \"BfR\"; plugin \"de.bund.bfr.knime.fsklab.nodes\" is not installed)"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

[606] "WARN \t main Node\t Node can't be executed - Node \"FSKX Reader\" not available from extension \"FSK Lab Feature\" (provided by \"BfR\"; plugin \"de.bund.bfr.knime.fsklab.nodes\" is not installed)"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

[607] "INFO \t main BatchExecutor\t Workflow execution done Finished in 9 secs (9882ms)"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

[608] "INFO \t main BatchExecutor\t Saved workflow availabe at: /home/gcube/tomcat/webapps/wps/ecocfg/rscr_e414ab4a-2ce1-4757-96a4-8cc233cc71ee/FSKX_Model_Runner/workflow.zip"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            

[609] "INFO \t main BatchExecutor\t ========= Workflow did not execute sucessfully ============"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          

In addition the log file is now extremely long. I don't know why that is. Maybe many other plugins are not properly installed yet?
Could you please open KNIME in the GUI and check if the plugins are shown there and if you get some error messages?

I also improved another KNIME workflow and it runs sometimes but often it failes - this behavior I don't see in my KNIME. I don't know whats sometimes going wrong in the DataMiner. Here you can find it if you are interested: https://aginfra.d4science.org/group/demeter/dataminer ->KNIME -> Get Pubmed Abstracts

Many thanks in advance
Lars

There's no GUI available on the servers, so the command line is the only possible way to run knime. On which nodes did the workflow run, so that I can check locally?

On one of our servers I can connect via ssh and -X window and start the KNIME in the GUI.
Without that you could check the plugin folder and may send me the list.

The Testworkflow https://support.d4science.org/attachments/download/2682/simple_FSK_testworkflow.knwf from above is only using KNIME nodes and should run on its own. The workflow I created for the other ticket is similiar but need an additional file as input. Therefore please test via the FSK_testworkflow. It is using the FSKX nodes only.

Thank you!

PS. I just see I did not answer questions of yours from the previous post:

Got it. I don't understand if it's needed when running knime as a standalone command too.

Ja, it forces the KNIME reassign all plugins. It helps to avoid unwanted side effects. If you never have any issue with KNIME you can leave it out and use it if you face issues. I decided I never want to face side effects again and I save a lot of time in trouble shooting with the -clean always on in plugins are installed (otherwise its not needed).

I don't understand this part. Do you expect us to run knime as a server?

No I don't. I was just mentioning that my ANSIBLE script has to differ in some parts since we have a real server and don't need to run KNIME in batch mode - it was just to make u aware, that for example the paths might differ.

Lars Valentin wrote:

On one of our servers I can connect via ssh and -X window and start the KNIME in the GUI.
Without that you could check the plugin folder and may send me the list.

The Testworkflow https://support.d4science.org/attachments/download/2682/simple_FSK_testworkflow.knwf from above is only using KNIME nodes and should run on its own. The workflow I created for the other ticket is similiar but needs an additional file as input. Therefore please test via the FSK_testworkflow. It is using the FSKX nodes only.

Thank you!

PS. I just see I did not answer questions of yours from the previous post:

Got it. I don't understand if it's needed when running knime as a standalone command too.

Ja, it forces KNIME to reassign all plugins. It helps to avoid unwanted side effects. If you never have any issue with KNIME you can leave it out and use it if you face issues. I decided I never want to face side effects again and I save a lot of time in trouble shooting with the '-clean' always on if plugins are installed (otherwise it is not needed).

I don't understand this part. Do you expect us to run knime as a server?

No I don't. I was just mentioning that my ANSIBLE script has to differ in some parts since we have a real server and don't need to run KNIME in batch mode - it was just to make u aware, that for example the paths might differ.

Lars Valentin wrote:

On one of our servers I can connect via ssh and -X window and start the KNIME in the GUI.

We do not install the Xwindow libraries, so there's no way to run the KNIME GUI.

Without that you could check the plugin folder and may send me the list.

Attached. I run the provisioning again on some hosts, and I see that at least one plugin installation always fails.

The Testworkflow https://support.d4science.org/attachments/download/2682/simple_FSK_testworkflow.knwf from above is only using KNIME nodes and should run on its own. The workflow I created for the other ticket is similiar but need an additional file as input. Therefore please test via the FSK_testworkflow. It is using the FSKX nodes only.

Thank you!

PS. I just see I did not answer questions of yours from the previous post:

Got it. I don't understand if it's needed when running knime as a standalone command too.

Ja, it forces the KNIME reassign all plugins. It helps to avoid unwanted side effects. If you never have any issue with KNIME you can leave it out and use it if you face issues. I decided I never want to face side effects again and I save a lot of time in trouble shooting with the -clean always on in plugins are installed (otherwise its not needed).

Now the workflow is run every time a plugin is removed or installed. I see that it fails because of a missing class:

Caused by: java.lang.ClassNotFoundException: org.apache.aries.blueprint.NamespaceHandler cannot be found by org.apache.cxf.cxf-rt-frontend-jaxrs_3.0.7
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:484)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387)
        at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:150)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 139 more

I don't know if it should be made available by some plugin, I see

/opt/knime/knime_distribution/plugins/org.apache.cxf_2.7.12.v201805071222
/opt/knime/knime_distribution/plugins/org.knime.ext.cxf.webservice_3.6.0.v201807090906.jar

No I don't. I was just mentioning that my ANSIBLE script has to differ in some parts since we have a real server and don't need to run KNIME in batch mode - it was just to make u aware, that for example the paths might differ.

OK.

Andrea Dell'Amico wrote:

Sorry, I forgot the attachment. It's there now. We are installing KNIME 4.6.0, FYI.

Thank you! I see the files are there, but it looks like KNIME itself is not aware of them?
As a test, you could change the repository from

https://dl.bintray.com/silebat/fsklab_icpmf

to

https://dl.bintray.com/silebat/lars

in order to uninstall and then install the fsklab plugin. If the whole process including '-clean' runs through without issues it should work.

What do you think about my other thoughts from above?

I'm out of office today, I'll elaborate in the next days.

The deployment variables for the plugins are the following:

knime_plugin_uninstall: [] 
  #- de.bund.bfr.knime.fsklab.feature.feature.group

knime_plugin_repository:  "http://update.knime.org/analytics-platform/{{ knime_update_version }},http://update.knime.org/store/{{ knime_update_version }},http://update.knime.org/community-contributions/trusted/{{ knime_update_version }},http://update.knime.org/community-contributions/{{ knime_update_version }},http://update.knime.org/community-contributions/{{ knime_update_version }},https://dl.bintray.com/silebat/fsklab_icpmf,https://github.com/SiLeBAT/Other/raw/master/de.bund.bfr.knime.other.aaw.feat/update,https://dl.bintray.com/silebat/pmmlab,http://dl.bintray.com/silebat/generic,http://dl.bintray.com/silebat/internal/"

knime_plugin_install:
    - org.knime.features.testingapplication.feature.group
    - org.knime.features.datageneration.feature.group  #KNIME Data Generation http://update.knime.com/analytics-platform/3.6 Lars      
    - org.knime.features.base.filehandling.feature.group  #KNIME File Handling Nodes http://update.knime.com/analytics-platform/3.6 Lars      
    - org.knime.features.r.feature.group #KNIME Interactive R Statistics Integration http://update.knime.com/analytics-platform/3.6 Lars      
    - org.knime.features.base.feature.group #KNIME Core http://update.knime.com/analytics-platform/3.6 Lars      
    - org.knime.features.ext.jfreechart.feature.group #KNIME JFreeChart http://update.knime.com/analytics-platform/3.6 Lars      
    - org.knime.features.ext.svg.feature.group #KNIME SVG Support http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.webservice.client.feature.group #KNIME Web Service Client http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.virtual.feature.group #KNIME Virtual Nodes http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.textprocessing.feature.group #KNIME Textprocessing http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.textprocessing.language.german.feature.group #KNIME Textprocessing German Language Pack http://update.knime.com/analytics-platform/3.6 Matthias
    - org.knime.features.ext.textprocessing.language.spanish.feature.group #KNIME Textprocessing Spanish Language Pack http://update.knime.com/analytics-platform/3.6 Matthias
    - org.knime.features.ext.textprocessing.language.french.feature.group #KNIME Textprocessing French Language Pack http://update.knime.com/analytics-platform/3.6 Matthias
    - org.knime.features.stats2.feature.group #KNIME Statistics Nodes (Labs) http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.socialmedia.feature.group #KNIME Social Media Connectors (Twitter && Google) http://update.knime.com/analytics-platform/3.6 Matthias      
    - com.knime.features.reporting.designer.feature.group #KNIME Report Designer http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.webanalytics.feature.group #KNIME Webanalytics http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.python2.feature.group #KNIME Python Integration http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.perl.feature.group #KNIME Perl Integration http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.optimization.feature.group #KNIME Optimization extension http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.osm.feature.group #KNIME Open Street Map Integration http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.network.feature.group #KNIME Network Mining http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.network.distmatrix.feature.group #KNIME Network Mining distance matrix support http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.neighborgram.feature.group #KNIME Neighborgram && ParUni http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.distmatrix.feature.group #KNIME Distance Matrix http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.itemset.feature.group #KNIME Itemset Mining http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.lucene.feature.group #KNIME Indexing and Searching http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.birt.feature.group #KNIME HTML/PDF Writer http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.exttool.feature.group #KNIME External Tool Support http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.arima.feature.group #KNIME Autoregressive integrated moving average (ARIMA) http://update.knime.com/analytics-platform/3.6 Matthias      
    - org.knime.features.ext.exttool.feature.group #KNIME External Tool Support http://update.knime.com/analytics-platform/3.6 Matthias      
    - de.bund.bfr.knime.fsklab.feature.feature.group #FSK Lab Feature https://dl.bintray.com/silebat/fsklab_icpmf Lars      
      #    - de.bund.bfr.knime.fsklab.legacy.feature.feature.group #Legacy FSK-Lab nodes https://dl.bintray.com/silebat/fsklab_icpmf Lars      
    - de.bund.bfr.knime.pmm.base.feature.group #PMM Nodes https://dl.bintray.com/silebat/pmmlab/ Lars      
    - de.bund.bfr.knime.gis.feature.feature.group #KNIME Nodes for GIS and Graph Visualization http://dl.bintray.com/silebat/generic/ Lars Warenstrominfo
    - de.bund.bfr.knime.other.aaw.feat.feature.group #ZoMoPlugin-OtherFeats_aaw https://github.com/SiLeBAT/Other/raw/master/de.bund.bfr.knime.other.aaw.feat/update Alex Mibi-Portal
      ###    - de.bund.bfr.knime.internal.feature.feature.group #Rhome and editable node http://dl.bintray.com/silebat/internal/ Alex RiskRanking
    - de.nbi.cibi.feature.feature.group #de.NBI - CIBI Extension http://update.knime.com/community-contributions/3.6 M. Flor      
    - org.knime.knip.feature.feature.group #KNIME Image Processing http://update.knime.com/community-contributions/3.6 M. Filter AGINFRA

I'm going to reinstall a single node and catch all the installation logs.

Hey Andrea,

if Ansible run flawlessly through all these steps it should work in the end - but unfortunately it does not.
In your log are several errors mentioned:

INFO     main BatchExecutor  ===== Executing workflow /opt/knime/knime_workflows/dummy =====
ERROR    main BatchExecutor  Workflow is locked by another KNIME instance
INFO     main BatchExecutor  ========= Workflow did not execute sucessfully ============

error: XDG_RUNTIME_DIR not set in the environment.

java.lang.NoClassDefFoundError: org/apache/aries/blueprint/NamespaceHandler

Since, there is no FSK mentioned I assume you did not change the dummy with the FSKX testworkflow I provided you above, in order to test if FSK works after the installation.

I did that and checked the workflow in the dataminer and I receive the old error message that the nodes are not available/installed.

In order to get a system running right now, I manually created for you a version of a working KNIME version with only FSK plugins installed by doing the following steps on a Linux computer to be finally placed in the VRE DM to replace your version for now:

1. Downloading the KNIME https://download.knime.org/analytics-platform/linux/knime-latest-linux.gtk.x86_64.tar.gz
2. Extracting the file tar -xvzf knime_3.6.1.linux.gtk.x86_64.tar.gz
3. changing the knime.ini from value -Xmx2048m to -Xmx8g (if you can offer more RAM in the DM tell here)
4. run KNIME GUI ./knime
5. set webproxy if needed I switched in General Network to manual and added webproxy 8080 for HTTP and HTTPS to meet BfR requirements
6. install our minimal requirement of plugins as explained here https://foodrisklabs.bfr.bund.de/fsk-lab_de/
   ~~~ (only!!!!) FSK Lab Feature from https://dl.bintray.com/silebat/fsklab_icpmf (needs the R 3.4 and the packages Rserve and miniCRAN) ~~~ and restart KNIME
7. import the workflows https://support.d4science.org/attachments/download/2682/simple_FSK_testworkflow.knwf and https://support.d4science.org/attachments/download/2713/dummy.knwf and check both if they run properly through!)
8. zipping with tar czf FSK_KNIME_3.6.1.tar.gz Downloads/knime_3.6.1

If this version works on your test machine you can check on the cluster infrastructure too. Hopefully this way we can see if it does work at all and then evolve later to a fully functional ANSIBLE driven KNIME.

Here is the link:

 https://download:xOFoblIr@download.bfr.bund.de/downloads/u2KoMIvZ/

Best Lars

The workflow still fails. I've found the following thread, that can maybe give some hints? https://forum.knime.com/t/excutable-in-batch-mode/10779/6

/opt/Downloads/knime_3.6.1/knime -nosplash -clean  -reset -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir='/opt/knime/knime_workflows/dummy'
CompilerOracle: exclude javax/swing/text/GlyphView.getBreakSpot
Sep 24, 2018 1:02:46 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensions
INFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-transports-http (127) [org.apache.cxf.transport.http.HTTPTransportFactory, org.apache.cxf.transport.http.HTTPWSDLExtensionLoader, org.apache.cxf.transport.http.policy.HTTPClientAssertionBuilder, org.apache.cxf.transport.http.policy.HTTPServerAssertionBuilder, org.apache.cxf.transport.http.policy.NoOpPolicyInterceptorProvider]
INFO     main BatchExecutor      ===== Executing workflow /opt/knime/knime_workflows/dummy =====
ERROR    main BatchExecutor      Workflow is locked by another KNIME instance
INFO     main BatchExecutor      ========= Workflow did not execute sucessfully ============
Sep 24, 2018 1:02:54 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener unregister
INFO: Removing the extensions for bundle 127
error: XDG_RUNTIME_DIR not set in the environment.
Knime: Cannot open display:
Knime:
JVM terminated. Exit code=3
/opt/Downloads/knime_3.6.1//plugins/org.knime.binary.jre.linux.x86_64_1.8.0.152-01/jre/bin/java
-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Xmx8g
-Dorg.eclipse.swt.internal.gtk.disablePrinting
-jar /opt/Downloads/knime_3.6.1//plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
-os linux
-ws gtk
-arch x86_64
-launcher /opt/Downloads/knime_3.6.1/knime
-name Knime
--launcher.library /opt/Downloads/knime_3.6.1//plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.551.v20171108-1834/eclipse_1630.so
-startup /opt/Downloads/knime_3.6.1//plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
--launcher.overrideVmargs
-exitdata 400002
-clean
-reset
-application org.knime.product.KNIME_BATCH_APPLICATION
-workflowDir=/opt/knime/knime_workflows/dummy
-vm /opt/Downloads/knime_3.6.1//plugins/org.knime.binary.jre.linux.x86_64_1.8.0.152-01/jre/bin/java
-vmargs
-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Xmx8g
-Dorg.eclipse.swt.internal.gtk.disablePrinting
-jar /opt/Downloads/knime_3.6.1//plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar

Hey Andrea,

looks promising for me. I don't have the locked workflow issue.

I tried /home/Valentin/Downloads/knime_3.6.1/knime -nosplash -clean -reset -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir='/home/Valentin/knime-workspace/dummy'

and I get a long weird thing about something missing with blueprint (I will write KNIME about that) and in the end:

INFO     main BatchExecutor  ===== Executing workflow /home/Valentin/knime-workspace/dummy =====
INFO     main BatchExecutor  Workflow execution done Finished in 1 sec (1163ms)
INFO     main BatchExecutor  ============= Workflow executed sucessfully ===============

same happens with the FSK-test-workflow - it works

I read about the workflow locked and they mentioned the lock-file ".knimeLock" which needs to be removed after execution from the workspace folder. Maybe something went wrong? I read it here https://forum.knime.com/t/how-to-remove-lock-all-batch-processes-suddenly-locked-by-another-knime-instance/4040/3
Sometimes in network structures it seems to be not removed automatically.

HTH
Lars

Here you can read about the ticket - seems to be a new issue since 3.6. https://forum.knime.com/t/knime-analytics-platform-v-3-6-error-in-batch-mode/11838

The only lock files I see are the ones listed (3.6.0 is from our installation, 3.6.1 is your tar file):

./knime/knime_3.6.0/p2/org.eclipse.equinox.p2.engine/profileRegistry/KNIMEProfile.profile/.lock
./knime/knime_3.6.0/configuration/org.eclipse.core.runtime/.manager/.fileTableLock
./knime/knime_3.6.0/configuration/org.eclipse.equinox.app/.manager/.fileTableLock
./knime/knime_3.6.0/configuration/org.eclipse.osgi/.manager/.fileTableLock
./Downloads/knime_3.6.1/p2/org.eclipse.equinox.p2.engine/profileRegistry/KNIMEProfile.profile/.lock
./Downloads/knime_3.6.1/configuration/org.knime.core/root.lock
./Downloads/knime_3.6.1/configuration/org.eclipse.core.runtime/.manager/.fileTableLock
./Downloads/knime_3.6.1/configuration/org.eclipse.equinox.app/.manager/.fileTableLock
./Downloads/knime_3.6.1/configuration/org.eclipse.osgi/.manager/.fileTableLock

I removed them, anyway. I'm going to run other tests with your workflows.

I run both the workflows, they still fail and there's always the message related to the lock file.

We are stalled, so that's my proposal: put your public ssh key into your user profile on the d4science portal (under My Account -> Custom fields) and I'll give you access to one of the dev dataminer (the installations are identical on all the dev, preprod and production environments) so that you can experiment directly on one of the target servers. Are you OK with that?

@matthias.filter@bfr.bund.de @lars.valentin@bfr.bund.de I'm wondering you can start a dialogue with the KNIME team, I assume you are in contact with them.

Such a technology is neither mastered from CNR nor from BfR.

We can make a try with a plain installation of KNIME and with your collaboration we can test it.

Leonardo Candela wrote:

This is definitely a good news.

Indeed!

If I get it correctly, the FSK lab files (including the encoding of a model as a KNIME workflow) need to be expanded before being executed. This is a very important detail to be known when implementing the workflow enacting the simulation.

That's not correct. The FSKX files are container we create, they contain all data and files around the model itself. This container should be only unzipped if you need to access the files in there.

What Andrea was referring to was the direct use of a .knwf file (KNime WorkFlow).

The line I was using for the batch mode looked like

/home/Valentin/Downloads/knime_3.6.1/knime -nosplash -clean -reset -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir='/home/Valentin/knime-workspace/dummy'

As I described above in #note-39 step 7 this should work flawlessy after IMPORTING the workflow.

If the workflow was not imported (or as Andrea stated simply extracted) the log file should give immedately response 'THIS IS NO DIRECTORY', since parameter -workflowDir= was used.

I was wondering about that, since in the DATAMINER you always perfom the batch mode on the file and I realized that you there do the correct line using the file mode '-workflowFile=':

Knime computation
/usr/local/bin/knime -reset -consoleLog -nosplash -application org.knime.product.KNIME_BATCH_APPLICATION -workflowFile=FSKX_Model_Runner.knwf -destFile=workflow.zip -workflow.variable=inputFile,"/home/gcube/tomcat/tmp/inputFile_[FSKX_MODEL_RUNNER_ID_54b3d065-3d14-4d61-b4e4-04ce9a177472].csv",String 
computation finished

Therefore, there is still no explanation for me for your locked workflow issue, but the knowledge to do it right was always on your side. I did spend a lot of time on that to help you with detailed explanations, even if we don't use the batch mode at all, since we use a KNIME server.

Regarding our FSK plugin, we just released today a new version, which might also help to solve your installation issue. Could you please uninstall and reinstall them through the KNIME installer and test the provided workflows with the batch mode and in the DATAMINER and give feedback on that?

@lars.valentin@bfr.bund.de / @matthias.filter@bfr.bund.de I guess we can close this ticket now and focus on the rest of activities aiming at exploiting the instances of this facility.

Please, let us know how you plan to handle future releases of the technology.

I would like to suggest, that you install the latest version 3.6.1 for now. Updating future versions as soon as they are available (bugfixes) would make sense to me, but you should not install a new major version 3.7 without checking in with us. Same as for R 4.4, since R 4.5 does not work with our plugins.

Regarding our FSK plugin, this one should be always updated automatically from the icpmf update site as soon as available. Since it is under development, we quite often release a new version.

It would be helpful for us if you could send a kind of a newsletter to us via the support email address, to inform us about all recent changes in the infrastructure we depend on (like new FSK build installed).

Many thanks in advance

Lars

I would like to add, that I just run my FSKX test workflows in the RAKIP Dataminer and it said the nodes are not installed. I hope this will be solved with our newest release of the FSK plugin.

Dear @lars.valentin@bfr.bund.de and @matthias.filter@bfr.bund.de

the effort and time needed to install another version of KNIME will be reported by @andrea.dellamico@isti.cnr.it (he is currently away from the office and will reply as soon as feasible). From a previous comment (i.e. #12249#note-22) I see that the installation is time consuming ... we have a number of machines to be configured.

I can only observe that it is really a pity to have spent some months to install a version of a technology without having the possibility / plan to make any use of it. This is not to blame someone, bad things happen.

BTW, my proposal is the following:

• create a new ticket to request the installation of the 3.6.1 version ... this way we can follow this new task in the future;
  • I hope that the rest of activities will not be blocked waiting for 3.6.1;
• every change to the version of KNIME supported / available by the infrastructure should be triggered by a new ticket ... CNR will not deploy automatically any minor release of KNIME since we do not know in advance whether the new version will prevent previously imported "algorithms" from working;
• regarding the automatic update of the FSK plugin, this is doable and I understand the rationale. We trust the retro-compatibility, i.e. we hope that new versions of the plugin will not prevent existing FSK models from working. Otherwise we risk to have an "unstable" environment;

Regarding the way KNIME has been integrated in DataMiner and the need to "import" the workflows to be executed. We can work together to improve the batch mode invocation of KNIME if this is needed / expected. Dunno if in the case of FSK files there are extra steps to do.

Leonardo Candela wrote:

the effort and time needed to install another version of KNIME will be reported. From a previous comment (i.e. #12249#note-22) I see that the installation is time consuming ... we have a number of machines to be configured.

I can only observe that it is really a pity to have spent some months to install a version of a technology without having the possibility / plan to make any use of it. This is not to blame someone, bad things happen.

BTW, my proposal is the following:

• create a new ticket to request the installation of the 3.6.1 version ... this way we can follow this new task in the future;
  • I hope that the rest of activities will not be blocked waiting for 3.6.1;
• every change to the version of KNIME supported / available by the infrastructure should be triggered by a new ticket ... CNR will not deploy automatically any minor release of KNIME since we do not know in advance whether the new version will prevent previously imported "algorithms" from working;

Yes, the ticket is needed. The operation is very time consuming, but I guess that for a minor release servers running the current and the new version can cohesist until the upgrade is finished.

• regarding the automatic update of the FSK plugin, this is doable and I understand the rationale. We trust the retro-compatibility, i.e. we hope that new versions of the plugin will not prevent existing FSK models from working. Otherwise we risk to have an "unstable" environment;

I'll add a cron job that reinstall the plugin every day.

Lars Valentin wrote:

What Andrea was referring to was the direct use of a .knwf file (KNime WorkFlow).

The line I was using for the batch mode looked like

/home/Valentin/Downloads/knime_3.6.1/knime -nosplash -clean -reset -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir='/home/Valentin/knime-workspace/dummy'

As I described above in #note-39 step 7 this should work flawlessy after IMPORTING the workflow.

If the workflow was not imported (or as Andrea stated simply extracted) the log file should give immedately response 'THIS IS NO DIRECTORY', since parameter -workflowDir= was used.

Nope, because the directory exists: the knime workflow file was placed inside that directory.
The error message was completely misleading, otherwise the

I was wondering about that, since in the DATAMINER you always perfom the batch mode on the file and I realized that you there do the correct line using the file mode '-workflowFile=':

Knime computation
/usr/local/bin/knime -reset -consoleLog -nosplash -application org.knime.product.KNIME_BATCH_APPLICATION -workflowFile=FSKX_Model_Runner.knwf -destFile=workflow.zip -workflow.variable=inputFile,"/home/gcube/tomcat/tmp/inputFile_[FSKX_MODEL_RUNNER_ID_54b3d065-3d14-4d61-b4e4-04ce9a177472].csv",String 
computation finished

I didn't know anything about the way dataminer runs a KNIME workflow. I assumed that the command was the same I used. I'm unarchiving your test workflow files in the next playbook run, so the test workflow should work from now on.