Project

General

Profile

Actions
Copy link

Incident #12894

closed

Update of FSK plugin failed

Added by Lars Valentin almost 7 years ago. Updated almost 7 years ago.

Status:
Closed
Priority:
High
Assignee:
Andrea Dell'Amico
Category:
Other
Target version:
KNIME
Start date:
Nov 16, 2018
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Development, Pre-Production, Production

Description

Hello Andrea,

there is at the moment no KNIME FSKlab plugin installed in the DataMiner.

I am sure this was caused by us due to an uploading issue to the update repository.

Could you please run Ansible over again. I just fixed the repository.

Thank you!

Lars

Files

d4science-ansible-knime-role.tar.gz (49 KB) d4science-ansible-knime-role.tar.gz ansible knime role Andrea Dell'Amico, Dec 09, 2018 12:35 PM
Actions
Copy link
 #1

Updated by Andrea Dell'Amico almost 7 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by Andrea Dell'Amico almost 7 years ago

  • Assignee changed from Andrea Dell'Amico to Lars Valentin

The installation still fails:

!SESSION 2018-11-16 13:17:40.660 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.8.0_152
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group
Command-line arguments:  -os linux -ws gtk -arch x86_64 -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group

!ENTRY org.eclipse.equinox.p2.director.app 4 0 2018-11-16 13:17:46.848
!MESSAGE The installable unit de.bund.bfr.knime.fsklab.feature.feature.group has not been found.

The command invoked to install the plugin is:

knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group
Actions
Copy link
 #3

Updated by Lars Valentin almost 7 years ago

Thank you Andrea for trying. I reported your findings to our developers and they seem to have an idea why it does not work via the commandline this time. It seems to be a missing dependency. As soon as we solved it, I will let you know.

Actions
Copy link
 #4

Updated by ahmad swaid almost 7 years ago

  • Assignee changed from Lars Valentin to Andrea Dell'Amico
  • Priority changed from Urgent to Normal

Hi Andrea,
The problem was in the update site of FSK_Lab which cause problems in the cache of P2 repository of KNIME.
We are investigating the problem so that it will not happen again in the future. So it will be very nice if you provide us with the ansible script or part of it which do the installation of the plugins in KNIME headless.

Best regards
Ahmad

Actions
Copy link
 #5

Updated by Andrea Dell'Amico almost 7 years ago

The daily update is performed by a cron job that runs a shell script:

#!/bin/bash
RETVAL=
RETVAL_WF=
DATE=$( date )
export PATH="/usr/local/bin:$PATH"

if [ ! -d /opt/knime/logs ] ; then
    mkdir /opt/knime/logs
fi

if [ -f "/opt/knime/scripts/lib_knime_fsk_updater" ] ; then
    . /opt/knime/scripts/lib_knime_fsk_updater
else
    logger "knime_fsk_updater: /opt/knime/scripts/lib_knime_fsk_updater file, exiting"
    exit 1
fi

for plugin in ${KNIME_FSK_PLUGINS} ; do
    if [ -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed" ] ; then
        knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination "${KNIME_WORKING_PATH}" -repository "${KNIME_PLUGINS_REPOSITORY}" -uninstallIU "${plugin}" > "$KNIME_PLUGINS_LOG_FILE" 2>&1
        RETVAL=$?
        if [ $RETVAL -eq 0 ] ; then
            rm -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed"
            logger "knime_fsk_updater: Plugin ${plugin} removed successfully"
        else
            logger "knime_fsk_updater: Plugin ${plugin} removal failed"
        fi
    fi
done

RETVAL=

for plugin in ${KNIME_FSK_PLUGINS} ; do
    if [ ! -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed" ] ; then
        knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination "${KNIME_WORKING_PATH}" -repository "${KNIME_PLUGINS_REPOSITORY}" -installIU "${plugin}" >> "$KNIME_PLUGINS_LOG_FILE" 2>&1
        RETVAL=$?
        if [ $RETVAL -eq 0 ] ; then
            echo "$DATE" > "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed"
            logger "knime_fsk_updater: Plugin ${plugin} installed successfully"
            knime -nosplash -clean -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="${KNIME_WORKFLOWS_DIR}/simple_FSK_testworkflow" >> "$KNIME_PLUGINS_LOG_FILE" 2>&1
            RETVAL_WF=$?
            if [ $RETVAL_WF -eq 0 ] ; then
                logger "knime_fsk_updater: simple FSK workflow run successfully"
            else
                logger "knime_fsk_updater: simple FSK workflow failed"
                exit $RETVAL_WF
            fi
        else
            logger "knime_fsk_updater: Plugin ${plugin} failed to install"
        fi
    fi
done

chown -R gcube:gcube ${KNIME_WORKING_PATH}/

exit $RETVAL

The file /opt/knime/scripts/lib_knime_fsk_updater contains

KNIME_PLUGINS_REPOSITORY="https://dl.bintray.com/silebat/fsklab_icpmf"
KNIME_FSK_PLUGINS="de.bund.bfr.knime.fsklab.feature.feature.group "
KNIME_WORKING_PATH="/opt/knime/knime_distribution"
KNIME_PLUGINS_LOG_FILE="/opt/knime/logs/knime_fsk_plugins.log"
KNIME_WORKFLOWS_DIR=/opt/knime/knime_workflows
Actions
Copy link
 #6

Updated by Andrea Dell'Amico almost 7 years ago

Here is the ansible knime role, anyway. I've not reworked it to supporto multiple knime versions yet.

Actions
Copy link
 #7

Updated by Lars Valentin almost 7 years ago

Thank you Andrea,

since we have combined our repositories, https://dl.bintray.com/silebat/fsklab_icpmf
does include now https://dl.bintray.com/silebat/pmmlab and http://dl.bintray.com/silebat/internal/. Therefore, https://dl.bintray.com/silebat/pmmlab and http://dl.bintray.com/silebat/internal/ need to be removed from all the scripts.

In addition https://dl.bintray.com/silebat/fsklab_icpmf has dependencies to the KNIME main repository http://update.knime.org/analytics-platform/{{ knime_update_version }} and therefore should be always used in combination. I think this needs to be added to your cron job. 

The file /opt/knime/scripts/lib_knime_fsk_updater contains

KNIME_PLUGINS_REPOSITORY="https://dl.bintray.com/silebat/fsklab_icpmf"

Best,
Lars

Actions
Copy link
 #8

Updated by Andrea Dell'Amico almost 7 years ago

  • Status changed from In Progress to Feedback

I changed the playbook and the cron job configuration. The plugin installation still fails:

eclipse.buildId=unknown
java.version=1.8.0_152
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf,http://update.knime.org/analytics-platform/3.6 -installIU de.bund.bfr.knime.fsklab.feature.feature.group
Command-line arguments:  -os linux -ws gtk -arch x86_64 -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf,http://update.knime.org/analytics-platform/3.6 -installIU de.bund.bfr.knime.fsklab.feature.feature.group

!ENTRY org.eclipse.equinox.p2.director.app 4 0 2018-12-10 12:09:50.891
!MESSAGE The installable unit de.bund.bfr.knime.fsklab.feature.feature.group has not been found.
Actions
Copy link
 #9

Updated by Lars Valentin almost 7 years ago

  • Assignee changed from ahmad swaid to Andrea Dell'Amico
  • Priority changed from Normal to High
  • % Done changed from 100 to 20

Hey Andrea,

could you please point me to the parts where you check if any KNIME process is running, before removing plugins? I did not see it in the ANSIBLE script nor in the CHRON job. We think it is needed that no job (KIME workflow) is running while the nodes used in this job are removed and installed again.

The solution to repair KNIME after the incident that KNIME can't find the nodes in the repository anymore is to backup and remove the content of the p2 cache folder, in order to force KNIME to recheck all plugins repositories after KNIME execution. The folder is located under /knimefolder.../p2/org.eclipse.equinox.p2.repository/cache/

You can check what plugins are installed in KNIME via

./knime -nosplash -application org.eclipse.equinox.p2.director -repository https://dl.bintray.com/silebat/fsklab_icpmf -listInstalledRoots

and to check if the repository does work you can use

./knime -nosplash -application org.eclipse.equinox.p2.director -repository https://dl.bintray.com/silebat/fsklab_icpmf -list

Hopefully this will help to get the FSK plugin running again.

HTH,
Lars

Actions
Copy link
 #10

Updated by Andrea Dell'Amico almost 7 years ago

  • Status changed from Feedback to In Progress
  • % Done changed from 20 to 80

Lars Valentin wrote:

Hey Andrea,

could you please point me to the parts where you check if any KNIME process is running, before removing plugins? I did not see it in the ANSIBLE script nor in the CHRON job. We think it is needed that no job (KIME workflow) is running while the nodes used in this job are removed and installed again.

This is the code that you're looking for. 

if [ -d /home/gcube/tomcat/tmp/dmlocks ] ; then
    DATAMINER_RUNNING=$( find /home/gcube/tomcat/tmp/dmlocks/ -type f )
fi

# 
if [ -d /home/gcube/tomcat/tmp/dmlocks ] ; then
   while [ ! -z "$DATAMINER_RUNNING" ] ; do
       DATAMINER_RUNNING=$( find /home/gcube/tomcat/tmp/dmlocks/ -type f )
   done
fi

The solution to repair KNIME after the incident that KNIME can't find the nodes in the repository anymore is to backup and remove the content of the p2 cache folder, in order to force KNIME to recheck all plugins repositories after KNIME execution. The folder is located under /knimefolder.../p2/org.eclipse.equinox.p2.repository/cache/

The cache removal did the trick. I'm adding a command to remove the cache after a installation failure, then the installation is going to be retried once.

Actions
Copy link
 #11

Updated by Andrea Dell'Amico almost 7 years ago

The installation of the plugin is ongoing.

Actions
Copy link
 #12

Updated by Andrea Dell'Amico almost 7 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 80 to 100

The plugin is installed everywhere.

Actions
Copy link
 #13

Updated by Lars Valentin almost 7 years ago

Andrea Dell'Amico wrote:

The plugin is installed everywhere.

Great! Thanks a lot!

Actions
Copy link

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)