D4Science Infrastructure

Francesco Mangiacrapa wrote:

HI Lars,

for your purpose the facility 'Execute DM Task" is available from your Workspace. You can call it by opening the workspace context menu and selecting 'Execute DM Task".
A brief "how-to" on it is available at https://gcube.wiki.gcube-system.org/gcube/Workspace#Specific_actions

Dear Francesco,

thanks a lot for the fast response. I just had a quick look into it and it looks promising. But I think I need more support in how to set up a KNIME workflow to get it fully compatible with this feature and how do I get our KNIME workflows into the selection menu? I don't find there those we have already implemented (not yet optimised for this purpose).

I did suggest a technical meeting to talk about this new feature and was informed that september would be good for that. But since we have a midterm review at september the 12th and we would like to show case the feature then, we kindly ask for a more close date. I think we will need some time to fully understand how its working and to prepare the needed workflows for the dataminer. I just agreed with Andrea Dell'Amico to have a webmeeting next week in another (but related) issue and maybe you could join?

Many thanks again

Lars

Lars Valentin wrote:

Francesco Mangiacrapa wrote:

HI Lars,

for your purpose the facility 'Execute DM Task" is available from your Workspace. You can call it by opening the workspace context menu and selecting 'Execute DM Task".
A brief "how-to" on it is available at https://gcube.wiki.gcube-system.org/gcube/Workspace#Specific_actions

Dear Francesco,

thanks a lot for the fast response. I just had a quick look into it and it looks promising. But I think I need more support in how to set up a KNIME workflow to get it fully compatible with this feature and how do I get our KNIME workflows into the selection menu? I don't find there those we have already implemented (not yet optimised for this purpose).

Hi Lars,

just a clarification on the above point...
From your workspace by the "Execute DM Task" you are able to get (so create the configuration and run) only the algorithms of a VRE having a file as input parameter (the kind FILE must be explicitly declared as input parameter of the algorithm). In fact, from Workspace by the right-click you select a file to perform a task on it (so at the moment we do not provide the run of an algorithm that does not take at least a file in input).

Looking the DM of rakip_portal at https://services.d4science.org/group/rakip_portal/data-miner the algorithms available from KNIME do not seem to take a file as input parameter, if that's right... could you try to add the kind FILE as input parameter (to an existing algorithm or a new one) and try to search it again from your workspace?

About this last point, you can read the wiki page at https://wiki.gcube-system.org/gcube/Statistical_Algorithms_Importer:_Knime-Workflow_Project or take support by @g.panichi@isti.cnr.it on how-to make it if you need to

F.

Lars

Hi Francesco,

thanks for clarifying, that I just have to integrate a workflow which is defined to have a file as input parameter to make the workflow appear in the send to list of the "right click feature".

What I would like to ask is, how does the user then is informed in that "right click feature" which workflow will except what file format?
Let's say I upload three workflows. First one works with .txt, second with .xlsx and third with .fskx only.
In the "right click feature" menu there is only mentioned there is a file expected, but I assume there is no further information or better, a filter for the fitting file format, meaning if I click a .fskx I would get only those workflows in the selection box which are able to handle this file format.

Best wishes
Lars

Lars Valentin wrote:

What I would like to ask is, how does the user then is informed in that "right click feature" which workflow will except what file format?

Currently, the information has to be included in the workflow description. This description is visualized to the user and it can be used to help the user in the selection of the right workflow.

Let's say I upload three workflows. First one works with .txt, second with .xlsx and third with .fskx only.
In the "right click feature" menu there is only mentioned there is a file expected, but I assume there is no further information or better, a filter for the fitting file format, meaning if I click a .fskx I would get only those workflows in the selection box which are able to handle this file format.

The filter for the fitting file format is not a feature currently supported. We plan to evaluate it for the next release. Please consider that the current implementation is compliant with the WPS standard. In order to support that filter, we need to evaluate how to enrich the description of the WPS process with the required information while respecting the standard.

Dear Pasquale, Dear Francesco,

In order to test the new application for our use case, I need to wait until our KNIME ticket is solved.

In the meantime, I would like to ask about an upcoming issue for which I have no solution yet. Hopefully you can provide some ideas on that.
The idea of the DataMiner is, as I understand it, that the user has to define the parameter of a model before the workflow is excecuted and then the results are presented in files.

What we are working on is a workflow which is generic and can except all kind of models following our defined scheme (FSKX). The first part of the workflow is reading the model file (in the future provided by the so called "right-click-feature") and extracts the parameter of the uploaded model. Here, we need to establish a model parameter dependend user interaction somehow in order to allow the user to change the parameter. Afterwards, the model is executed and the results are presented as a file.

We hope you may have some ideas for us on how we can fulfill this next task. Our developers (Java, JavaScript ...) are willing to help in any way.

Best wishes
Lars

Dear Lars,

In order to test the new application for our use case, I need to wait until our KNIME ticket is solved.

CNR team is partially on holidays and we will be able to be more active only starting from the last week of August.

In the meantime, I would like to ask about an upcoming issue for which I have no solution yet. Hopefully you can provide some ideas on that.
The idea of the DataMiner is, as I understand it, that the user has to define the parameter of a model before the workflow is excecuted and then the results are presented in files.

The DM engine implements a standard WPS interface and therefore it cannot support human interactions along the execution time. So, you are right saying that the user has to define the parameters to pass to the engine before the execution.

What we are working on is a workflow which is generic and can except all kind of models following our defined scheme (FSKX). The first part of the workflow is reading the model file (in the future provided by the so called "right-click-feature") and extracts the parameter of the uploaded model.

Ok, this is fine and supported with the current implementation.

Here, we need to establish a model parameter dependent user interaction somehow in order to allow the user to change the parameter.

You need to support a two-steps process. The first one is started with the selection of the file and the right-click-execution. This first step generates a file. At this point, the right-click-execution on this generated file (second step) will allow changing the parameters.

If I correctly understood, this requires some enhancements on the right-click-feature that we need to discuss and evaluate.

Afterwards, the model is executed and the results are presented as a file.

Fine.

We hope you may have some ideas for us on how we can fulfill this next task. Our developers (Java, JavaScript ...) are willing to help in any way.

Can you clarify if my interpretation is correct?

Thanks

Best wishes
Lars

Dear all,

I would like to suggest the following changes on the text from Matthias to more fit my understanding of the way how the VRE does work:

Preconditions:

0.: a dataminer infrastructure which provides a queue list to accept parallel incoming execution request and distributes them ideally to one dataminer each which is using its full capability of 16 cores to run one KNIME workflow ("FSK model execution" which calculates one model at the time). The dataminer infrastructure needs to also provide:
a) a defined KNIME version (currently agreed on 3.6.0)and a list of plugins we provided - but always the newest version (updated automatically via ANSIBLE) - Andrea is working on that
b) a defined R version (currently agreed on R 3.4.4) which provides all the plugins used in the models (since we don't know yet which plugins the models will use in the future, we need to find a way to automatically and constanly add new R plugins to the datamainer - the information of the plugins needed is stored in a plugins.json inside of each FSKX file)
1.: a FSKX file in a local / shared workspace
2.: a DM workflow "FSK model execution" that is a KNIME + FSK-Lab workflow consuming a (link to a) FSKX file and execute a the model script inside the FSKX file with the simulation settings defined inside the FSKX file; results are written out into the user's workspace.

1. A https link (URI) is generated via the user interface of the VRE which allows to open a new browser tab with the BfR KNIME server webportal. This link needs to contain as parameter at least the user token and the information needed to download the selected model from the VRE via VRE REST interface. The user is then allowed to change the parameter of the model in the KNIME webportal and the new/resolving FSKX file is then sent directly to the VRE dataminer via REST interface to be executed in the workflow "FSK model execution" and the results are written out into the user's workspace.
2. If a FSXK file is uploaded to the VRE Catalogue the system does read in the parameter file (sim.sed.ml), the metadata file (metaData.json) and the plugin.json which are stored inside the FSKX file (its a zip container). The information are shown in the VRE catalogue view and detailed model view. Furthermore, the user can change the simulation settings here (parameter of the model) this will result in a new FSKX file. The user can sent the (adjusted) model directly to the dataminer workflow "FSK model execution" and the results are written out into the user's workspace.

Procedure:

User:
I.:right click on a FSKX file in the user's workspace or a shared workspace
System's "right click" feature action:

a.: "Model execution" which will send the FSKX file directly to the KNIME workflow ("FSK model execution") in the dataminer
b.: "Interactive model execution" which will open a link to our KNIME server webportal including the URI of the file to be downloaded via VRE REST interface and the token of the user - the user can adjust the simulation settings in the BfR KNIME webportal start the execution of the model in the VRE workflow "FSK model execution"
c.: as alternative to b.: "Interactive model execution" which will open the portlet which is used in the VRE Catalogue (see 4.) to show the user the metadata of the selected model and to allow to change the parameter of it - the (changed) model file can be sent from here directly to executed in the dataminer workflow "FSK model execution".

II. user selects in the VRE Catalogue
a.: "Model execution" which will send the FSKX file directly to the KNIME workflow ("FSK model execution") in the dataminer
b.: "Interactive model execution" which will open a link to our KNIME server webportal including the URI of the file to be downloaded via VRE REST interface and the token of the user - the user can adjust the simulation settings in the BfR KNIME webportal start the execution of the model in the VRE workflow "FSK model execution"
c.: as alternative to b.: "Interactive model execution" which will open the portlet which is used in the VRE Catalogue (see 4.) to show the user the metadata of the selected model and to allow to change the parameter of it - the (changed) model file can be sent from here directly to be executed in the dataminer workflow "FSK model execution".

HTH
Lars

Dear @g.panichi@isti.cnr.it,

Francesco mentioned you as the person who can provide me with a KNIME workflow which can interact with the new right-click feature "Execute DM task".

In the earlier answers I got two links on a simple KNIME workflow example and a brief overview on the view of the new right-click feature, which do not explain what happens in the background and how exactly the KNIME workflow needs to be designed to work with the new feature.

Many thanks in advance!

Lars

PS. When I spent some hours to find out how a KNIME workflow needs to be designed to finally work in the DM (the example workflow was not really helpful), I realized that a temporary folder node needs to be present, but the files stored inside, what I normally do, can't be found by the DM for the results section. I would like to understand why I then need the temp folder node and how to address files stored in there for the DM results section.

I have to admit that the proliferation of unrelated tickets is making the discussion challenging to follow. We created #11316 to keep track of the steps related with "models" integration.

BTW, this ticket seems to me no longer related with the right click feature and the willingness to pass the workspace file as a parameter ... this is working.

The issue you are discussing here is how to implement a DataMiner algorithm that instantiates and executes the model stored into the file... @lars.valentin@bfr.bund.de is this understanding correct?

If this understanding is correct I'll suggest first to change the subject of the title.

Then on how to implement a DM workflow "FSK model execution" that is a KNIME + FSK-Lab workflow consuming a (link to a) FSKX file and execute a the model script inside the FSKX file with the simulation settings defined inside the FSKX file; results are written out into the user's workspace.

The issue I see is related with the "setting of the simulation settings" ... I assume this is something model specific and thus, it is quite challenging to model it as parameters of the "FSK model execution" algorithm. To make it possible for a user to instantiate the values for the simulation settings an intermediary step is needed to read what are the settings from the FSK file and make it possible for a user to assign values for them ... once this is done the DM FSK model execution can be invoked by passing both the link to the FSK file and the assigned values for the simulation settings (as a file or as other form of parameters) ... right now it is not clear to me who can take care of this step (and how).

Please, do take into account that DM algorithms are conceived to be executed in batch mode, i.e. they can be invoked by passing the specified parameters and to produce the declared output(s) ... no interactivity during the execution is there (and will be there).

One option might be the following. Count on Jupyter and the fact that it is has been integrated with its own GUI within a VRE.

We can design a Jupyter notebook that performs the following steps:

1. accept as input parameter the URL of the model stored into the workspace;
2. extracts the simulation settings from such a file and make it possible for a user to give values to such parameters;
3. invoke the DM algorithm FSK model execution by passing as parameters (a) the url of the model and (b) the parameters;

This option is not counting on the workspace "right click facility". Moreover, we should investigate in whether it is possible to invoke a Jupyter notebook with a link. If this is doable we will use the "right click facility" to publish model metadata into the catalogue (as discussed in #11426) including a link to "execute" them as envisaged above.

In #12250#note-9 and #12250#note-10 below BFR colleagues suggested to rely on a completely new JS portlet hosted by the BFR KNIME server to instantiate the parameters. The implication of this is to go for another "service" to be integrated and monitored in order to guarantee the functioning of the VRE.

Dear @leonardo.candela@isti.cnr.it,

this ticket is about the "right click feature" you recently implemented and all topics discussed here are related to that.

When I opened the ticket I asked how to use this "right click feature" and it was made available for me. But unfortunately the provided information on how to make it work was missing/minimalistic. Therefore, the right-click feature is not yet working for us! Since we have no clue how the mechanism behind works and how exactly a KNIME workflow needs to be designed to accept a input file, we would highly appreciate if you could provide us with the needed information and a specific example workflow to get it running. Besides that we depend on Andrea who is installing the new KNIME version and the needed plugins for us in the dataminer.

The further points we discuss here are our idea on the future evolution of that feature, trying to add more and more functionalities to the "right click feature" to finally establish the service we need, as described above in https://support.d4science.org/issues/12250#note-10. I would like to avoid opening tickets for all these steps, since it would cause the need to explain everything again and again, since everything is related and its already challenging to find a common understanding. Therefore, I am really looking forward to talk to you about all of that in the webmeeting on friday. I think this will clarify many points. Maybe Andrea can also join?

As we realized, "setting of the simulation settings" in a generic way is nothing the dataminer can provide. Therefore, we thought of a temporary workaround via our KNIME server user interface (webportal) to have a working solution soon. However, the suggested use of the Jupyter notebook sounds promising. I am just afraid there are limitations in the use of such a notebook view within the "right click feature" and the model catalogue view. I am looking forward to learn more about in the webmeeting.

Best wishes
Lars

PS. Could you please send us an invitation for friday?

Dear @lars.valentin@bfr.bund.de

waiting for our call tomorrow I like to clarify why I'm insisting on the fact that the issue you are facing has noting to do with the workspace right click feature.

What we discussed and agreed in #11316 does not expect to use the workspace right click for launching a simulation of a model, rather the "agreed" workflow envisages two steps:

1. use the workspace for "publishing" available models with rich metadata into a catalogue;
2. use the catalogue for the discovery existing models and for launching a simulation by using each model of interest;

The reason why we opted for a catalogue for the second step is almost evident, you cannot expect to count on the workspace especially if there are hundreds of models available.

We have worked by keeping this agreement into account. In fact, #11426 and its sub-tickets are oriented to implement the publishing phase.

Having clarified that, the open issue is still there, i.e. we need an actionable URI to be added into the catalogue for a facility taking care of

1. enacting a user to specifying model-specific simulation parameters
2. invoke a simulation by using the model and the specified parameters via a DM algorithm that is expected to be implemented by a KNIME workflow. You posed two constraints here:

• to have only one DM algorithm rather than one per model;
• to implement the DM algorithm by using a KNIME workflow;
  

I hope we are on the same page now.