Incident #12894: Update of FSK plugin failed - D4Science Infrastructure - D4science

Custom queries

03. VRE Plan
04. All Projects Deliverables
04. All Projects Milestones
06. D4Science Projects Activities
All open Issues
D4Science Infrastructure Upgrade
My InProgress Tasks
My Paused Tasks
My Watched Tasks
Not Closed Assigned To Me
Not Closed Assigned To Me (No Releases, No Project Tasks)
Not Closed Opened by Me
Open Issues
Upgrade Plan
Upgrade Plan Clear

Actions

Copy link

Incident #12894

closed

Update of FSK plugin failed

Added by Lars Valentin over 6 years ago. Updated over 6 years ago.

Status:

Closed

Priority:

High

Assignee:

Andrea Dell'Amico

Category:

Other

Target version:

KNIME

Start date:

Nov 16, 2018

Due date:

% Done:

100%

Estimated time:

Infrastructure:

Development, Pre-Production, Production

Description

Hello Andrea,

there is at the moment no KNIME FSKlab plugin installed in the DataMiner.

I am sure this was caused by us due to an uploading issue to the update repository.

Could you please run Ansible over again. I just fixed the repository.

Thank you!

Lars

Files

d4science-ansible-knime-role.tar.gz (49 KB) d4science-ansible-knime-role.tar.gz

ansible knime role

Andrea Dell'Amico, Dec 09, 2018 12:35 PM

History
Notes
Property changes

Actions

Copy link

Updated by Andrea Dell'Amico over 6 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Andrea Dell'Amico over 6 years ago

Assignee changed from Andrea Dell'Amico to Lars Valentin

The installation still fails:

!SESSION 2018-11-16 13:17:40.660 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.8.0_152
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group
Command-line arguments:  -os linux -ws gtk -arch x86_64 -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group

!ENTRY org.eclipse.equinox.p2.director.app 4 0 2018-11-16 13:17:46.848
!MESSAGE The installable unit de.bund.bfr.knime.fsklab.feature.feature.group has not been found.

The command invoked to install the plugin is:

knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group

Actions

Copy link

Updated by Lars Valentin over 6 years ago

Thank you Andrea for trying. I reported your findings to our developers and they seem to have an idea why it does not work via the commandline this time. It seems to be a missing dependency. As soon as we solved it, I will let you know.

Actions

Copy link

Updated by ahmad swaid over 6 years ago

Assignee changed from Lars Valentin to Andrea Dell'Amico
Priority changed from Urgent to Normal

Hi Andrea,
The problem was in the update site of FSK_Lab which cause problems in the cache of P2 repository of KNIME.
We are investigating the problem so that it will not happen again in the future. So it will be very nice if you provide us with the ansible script or part of it which do the installation of the plugins in KNIME headless.

Best regards
Ahmad

Actions

Copy link

Updated by Andrea Dell'Amico over 6 years ago

The daily update is performed by a cron job that runs a shell script:

#!/bin/bash
RETVAL=
RETVAL_WF=
DATE=$( date )
export PATH="/usr/local/bin:$PATH"

if [ ! -d /opt/knime/logs ] ; then
    mkdir /opt/knime/logs
fi

if [ -f "/opt/knime/scripts/lib_knime_fsk_updater" ] ; then
    . /opt/knime/scripts/lib_knime_fsk_updater
else
    logger "knime_fsk_updater: /opt/knime/scripts/lib_knime_fsk_updater file, exiting"
    exit 1
fi

for plugin in ${KNIME_FSK_PLUGINS} ; do
    if [ -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed" ] ; then
        knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination "${KNIME_WORKING_PATH}" -repository "${KNIME_PLUGINS_REPOSITORY}" -uninstallIU "${plugin}" > "$KNIME_PLUGINS_LOG_FILE" 2>&1
        RETVAL=$?
        if [ $RETVAL -eq 0 ] ; then
            rm -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed"
            logger "knime_fsk_updater: Plugin ${plugin} removed successfully"
        else
            logger "knime_fsk_updater: Plugin ${plugin} removal failed"
        fi
    fi
done

RETVAL=

for plugin in ${KNIME_FSK_PLUGINS} ; do
    if [ ! -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed" ] ; then
        knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination "${KNIME_WORKING_PATH}" -repository "${KNIME_PLUGINS_REPOSITORY}" -installIU "${plugin}" >> "$KNIME_PLUGINS_LOG_FILE" 2>&1
        RETVAL=$?
        if [ $RETVAL -eq 0 ] ; then
            echo "$DATE" > "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed"
            logger "knime_fsk_updater: Plugin ${plugin} installed successfully"
            knime -nosplash -clean -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="${KNIME_WORKFLOWS_DIR}/simple_FSK_testworkflow" >> "$KNIME_PLUGINS_LOG_FILE" 2>&1
            RETVAL_WF=$?
            if [ $RETVAL_WF -eq 0 ] ; then
                logger "knime_fsk_updater: simple FSK workflow run successfully"
            else
                logger "knime_fsk_updater: simple FSK workflow failed"
                exit $RETVAL_WF
            fi
        else
            logger "knime_fsk_updater: Plugin ${plugin} failed to install"
        fi
    fi
done

chown -R gcube:gcube ${KNIME_WORKING_PATH}/

exit $RETVAL

The file /opt/knime/scripts/lib_knime_fsk_updater contains

KNIME_PLUGINS_REPOSITORY="https://dl.bintray.com/silebat/fsklab_icpmf"
KNIME_FSK_PLUGINS="de.bund.bfr.knime.fsklab.feature.feature.group "
KNIME_WORKING_PATH="/opt/knime/knime_distribution"
KNIME_PLUGINS_LOG_FILE="/opt/knime/logs/knime_fsk_plugins.log"
KNIME_WORKFLOWS_DIR=/opt/knime/knime_workflows

Actions

Copy link

Updated by Andrea Dell'Amico over 6 years ago

File d4science-ansible-knime-role.tar.gz d4science-ansible-knime-role.tar.gz added
Assignee changed from Andrea Dell'Amico to ahmad swaid
% Done changed from 0 to 100

Here is the ansible knime role, anyway. I've not reworked it to supporto multiple knime versions yet.

Actions

Copy link

Updated by Lars Valentin over 6 years ago

Thank you Andrea,

since we have combined our repositories, https://dl.bintray.com/silebat/fsklab_icpmf
does include now https://dl.bintray.com/silebat/pmmlab and http://dl.bintray.com/silebat/internal/. Therefore, https://dl.bintray.com/silebat/pmmlab and http://dl.bintray.com/silebat/internal/ need to be removed from all the scripts.

In addition https://dl.bintray.com/silebat/fsklab_icpmf has dependencies to the KNIME main repository http://update.knime.org/analytics-platform/{{ knime_update_version }} and therefore should be always used in combination. I think this needs to be added to your cron job.

The file /opt/knime/scripts/lib_knime_fsk_updater contains

KNIME_PLUGINS_REPOSITORY="https://dl.bintray.com/silebat/fsklab_icpmf"

Best,
Lars

Actions

Copy link

Updated by Andrea Dell'Amico over 6 years ago

Status changed from In Progress to Feedback

I changed the playbook and the cron job configuration. The plugin installation still fails:

eclipse.buildId=unknown
java.version=1.8.0_152
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf,http://update.knime.org/analytics-platform/3.6 -installIU de.bund.bfr.knime.fsklab.feature.feature.group
Command-line arguments:  -os linux -ws gtk -arch x86_64 -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf,http://update.knime.org/analytics-platform/3.6 -installIU de.bund.bfr.knime.fsklab.feature.feature.group

!ENTRY org.eclipse.equinox.p2.director.app 4 0 2018-12-10 12:09:50.891
!MESSAGE The installable unit de.bund.bfr.knime.fsklab.feature.feature.group has not been found.

Actions

Copy link

Updated by Lars Valentin over 6 years ago

Assignee changed from ahmad swaid to Andrea Dell'Amico
Priority changed from Normal to High
% Done changed from 100 to 20

Hey Andrea,

could you please point me to the parts where you check if any KNIME process is running, before removing plugins? I did not see it in the ANSIBLE script nor in the CHRON job. We think it is needed that no job (KIME workflow) is running while the nodes used in this job are removed and installed again.

The solution to repair KNIME after the incident that KNIME can't find the nodes in the repository anymore is to backup and remove the content of the p2 cache folder, in order to force KNIME to recheck all plugins repositories after KNIME execution. The folder is located under /knimefolder.../p2/org.eclipse.equinox.p2.repository/cache/

You can check what plugins are installed in KNIME via

./knime -nosplash -application org.eclipse.equinox.p2.director -repository https://dl.bintray.com/silebat/fsklab_icpmf -listInstalledRoots

and to check if the repository does work you can use

./knime -nosplash -application org.eclipse.equinox.p2.director -repository https://dl.bintray.com/silebat/fsklab_icpmf -list

Hopefully this will help to get the FSK plugin running again.

HTH,
Lars

Actions

Copy link

#10

Updated by Andrea Dell'Amico over 6 years ago

Status changed from Feedback to In Progress
% Done changed from 20 to 80

Lars Valentin wrote:

Hey Andrea,

could you please point me to the parts where you check if any KNIME process is running, before removing plugins? I did not see it in the ANSIBLE script nor in the CHRON job. We think it is needed that no job (KIME workflow) is running while the nodes used in this job are removed and installed again.

This is the code that you're looking for.

if [ -d /home/gcube/tomcat/tmp/dmlocks ] ; then
    DATAMINER_RUNNING=$( find /home/gcube/tomcat/tmp/dmlocks/ -type f )
fi

# 
if [ -d /home/gcube/tomcat/tmp/dmlocks ] ; then
   while [ ! -z "$DATAMINER_RUNNING" ] ; do
       DATAMINER_RUNNING=$( find /home/gcube/tomcat/tmp/dmlocks/ -type f )
   done
fi

The solution to repair KNIME after the incident that KNIME can't find the nodes in the repository anymore is to backup and remove the content of the p2 cache folder, in order to force KNIME to recheck all plugins repositories after KNIME execution. The folder is located under /knimefolder.../p2/org.eclipse.equinox.p2.repository/cache/

The cache removal did the trick. I'm adding a command to remove the cache after a installation failure, then the installation is going to be retried once.

Actions

Copy link

#11