D4Science Infrastructure

I experiencing some issues on orientdb:

• Only orientdb02-d-d4s was started.
• trying starting the service manually I obtain bash: ../log/orientdb.log: Permission denied. IMHO is related to the way to start the server in /etc/init.d/orientdb script.

su $ORIENTDB_USER -c "cd \"$ORIENTDB_DIR/bin\"; /usr/bin/nohup ./server.sh 1>../log/orientdb.log 2>../log/orientdb.err &"

• logs are not available in the configured folder (i.e. ~/logs which is a symbolic link to /var/log/orientdb ). I only have the old logs from previous instance
• Some variable on json are upper case True instead of true in default-distributed-db-config.json
• I'm not sure if the distributed server is running. We have forced distributed mode by setting in tag. I don't understadn from documentation if we have to use dserver.sh to start it. So that we could try to use dserver.sh in /etc/init.d/orientdb instead of server.sh.

Here is the content of some files:

cat ~/orientdb-community/config/default-distributed-db-config.json 
{
  "autoDeploy": True,
  "readQuorum": 1,
  "writeQuorum": "majority",
  "executionMode": "undefined",
  "readYourWrites": True,
  "servers": {
    "*": "master"
  },
  "clusters": {
    "internal": {
    },
    "*": {
      "servers": ["<NEW_NODE>"]
    }
  }
}

cat ~/orientdb-community/bin/dserver.sh 
#!/bin/sh
#
# Copyright (c) Orient Technologies LTD (http://www.orientechnologies.com)
#

"${0%dserver.sh}server.sh" -Ddistributed=true $*

cat /etc/init.d/orientdb 
#!/bin/bash
# OrientDB service script
#
# Copyright (c) Orient Technologies LTD (http://www.orientechnologies.com)

# chkconfig: 2345 20 80
# description: OrientDb init script
# processname: orientdb.sh

# You have to SET the OrientDB installation directory here
if [ -f /etc/default/orientdb ] ; then
    . /etc/default/orientdb
else
    ORIENTDB_DIR="/home/orientdb/orientdb-community"
    ORIENTDB_USER="orientdb"
fi

usage() {
    echo "Usage: `basename $0`: <start|stop|status>"
    exit 1
}

start() {
    status
    if [ $PID -gt 0 ]
    then
        echo "OrientDB server daemon was already started. PID: $PID"
        return $PID
    fi
    echo "Starting OrientDB server daemon..."
    cd "$ORIENTDB_DIR/bin"
    su $ORIENTDB_USER -c "cd \"$ORIENTDB_DIR/bin\"; /usr/bin/nohup ./server.sh 1>../log/orientdb.log 2>../log/orientdb.err &"
}

stop() {
    status
    if [ $PID -eq 0 ]
    then
        echo "OrientDB server daemon is already not running"
        return 0
    fi
    echo "Stopping OrientDB server daemon..."
    cd "$ORIENTDB_DIR/bin"
    su $ORIENTDB_USER -c "cd \"$ORIENTDB_DIR/bin\"; /usr/bin/nohup ./shutdown.sh 1>>../log/orientdb.log 2>>../log/orientdb.err &"
}

status() {
    PID=` ps auxw | grep 'orientdb.www.path' | grep java | grep -v grep | awk '{print $2}'`
    if [ "x$PID" = "x" ]
    then
        PID=0
    fi

    # if PID is greater than 0 then OrientDB is running, else it is not
    return $PID
}

if [ "x$1" = "xstart" ]
then
    start
    exit 0
fi

if [ "x$1" = "xstop" ]
then
    stop
    exit 0
fi

if [ "x$1" = "xstatus" ]
then
    status
    if [ $PID -gt 0 ]
    then
        echo "OrientDB server daemon is running with PID: $PID"
        exit 0
    else
        echo "OrientDB server daemon is NOT running"
        exit 3
    fi
fi

usage

I fixed /etc/init.d/orientdb script on orientdb01-d-d4s like this:

cat /etc/init.d/orientdb 
#!/bin/bash
# OrientDB service script
#
# Copyright (c) Orient Technologies LTD (http://www.orientechnologies.com)

# chkconfig: 2345 20 80
# description: OrientDb init script
# processname: orientdb.sh

# You have to SET the OrientDB installation directory here
if [ -f /etc/default/orientdb ] ; then
    . /etc/default/orientdb
else
    ORIENTDB_DIR="/home/orientdb/orientdb-community"
    ORIENTDB_USER="orientdb"
fi

usage() {
    echo "Usage: `basename $0`: <start|stop|status>"
    exit 1
}

start() {
    status
    if [ $PID -gt 0 ]
    then
        echo "OrientDB server daemon was already started. PID: $PID"
        return $PID
    fi
    echo "Starting OrientDB server daemon..."
    cd "$ORIENTDB_DIR/bin"
    su $ORIENTDB_USER -c "cd \"$ORIENTDB_DIR/bin\"; /usr/bin/nohup ./server.sh 1>/home/orientdb/logs/orientdb.log 2>/home/orientdb/logs/orientdb.err &"
}

stop() {
    status
    if [ $PID -eq 0 ]
    then
        echo "OrientDB server daemon is already not running"
        return 0
    fi
    echo "Stopping OrientDB server daemon..."
    cd "$ORIENTDB_DIR/bin"
    su $ORIENTDB_USER -c "cd \"$ORIENTDB_DIR/bin\"; /usr/bin/nohup ./shutdown.sh 1>>/home/orientdb/logs/orientdb.log 2>>/home/orientdb/logs/orientdb.err &"
}

status() {
    PID=` ps auxw | grep 'orientdb.www.path' | grep java | grep -v grep | awk '{print $2}'`
    if [ "x$PID" = "x" ]
    then
        PID=0
    fi

    # if PID is greater than 0 then OrientDB is running, else it is not
    return $PID
}

if [ "x$1" = "xstart" ]
then
    start
    exit 0
fi

if [ "x$1" = "xstop" ]
then
    stop
    exit 0
fi

if [ "x$1" = "xstatus" ]
then
    status
    if [ $PID -gt 0 ]
    then
        echo "OrientDB server daemon is running with PID: $PID"
        exit 0
    else
        echo "OrientDB server daemon is NOT running"
        exit 3
    fi
fi

usage

Now the server start but I have the following error on ~/logs/orientdb.err

 cat ~/logs/orientdb.err
./server.sh: 108: ./server.sh: cannot create /home/orientdb/orientdb-community/bin/orient.pid: Permission denied

2016-10-26 15:03:43:696 INFO  Loading configuration from: /home/orientdb/orientdb-community/config/orientdb-server-config.xml... [OServerConfigurationLoaderXml]
2016-10-26 15:03:44:336 INFO  OrientDB Server v2.2.12-SNAPSHOT (build 2.2.x@r2a76ee1efc4487d9dc5f43e1ddc2fba1c5a0167c; 2016-10-20 13:55:08+0000) is starting up... [OServer]
2016-10-26 15:03:44:350 INFO  Databases directory: /home/orientdb/databases [OServer]
2016-10-26 15:03:44:486 WARNI Not enough physical memory available for DISKCACHE: 2,938MB (heap=1,820MB direct=524,288MB). Set lower Maximum Heap (-Xmx setting on JVM) and restart OrientDB. Now running with DISKCACHE=256MB [orientechnologies]
2016-10-26 15:03:44:487 INFO  OrientDB config DISKCACHE=256MB (heap=1,820MB direct=524,288MB os=2,938MB) [orientechnologies]
2016-10-26 15:03:45:236 INFO  Listening binary connections on 0.0.0.0:2424 (protocol v.36, socket=default) [OServerNetworkListener]
2016-10-26 15:03:45:244 INFO  Listening http connections on 0.0.0.0:2480 (protocol v.10, socket=default) [OServerNetworkListener]
2016-10-26 15:03:45:270 INFO  Installing dynamic plugin 'studio-2.2.zip'... [OServerPluginManager]
2016-10-26 15:03:45:281 INFO  ODefaultPasswordAuthenticator is active [ODefaultPasswordAuthenticator]
2016-10-26 15:03:45:283 INFO  OServerConfigAuthenticator is active [OServerConfigAuthenticator]
2016-10-26 15:03:45:285 INFO  OSystemUserAuthenticator is active [OSystemUserAuthenticator]
2016-10-26 15:03:45:337 INFO  Installed GREMLIN language v.2.6.0 - graph.pool.max=50 [OGraphServerHandler]
2016-10-26 15:03:45:657 WARNI {db=OSystem} Storage 'OSystem' was not closed properly. Will try to recover from write ahead log [OLocalPaginatedStorage]
2016-10-26 15:03:45:659 INFO  {db=OSystem} Looking for last checkpoint... [OLocalPaginatedStorage]
2016-10-26 15:03:45:942 INFO  {db=OSystem} FULL checkpoint found. [OLocalPaginatedStorage]
2016-10-26 15:03:45:944 INFO  {db=OSystem} Data restore procedure from full checkpoint is started. Restore is performed from LSN OLogSequenceNumber{segment=0, position=623350} [OLocalPaginatedStorage]
2016-10-26 15:03:45:949 WARNI {db=OSystem} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OCheckpointEndRecord{lsn=OLogSequenceNumber{segment=0, position=623374}} will be skipped during data restore [OLocalPaginatedStorage]
2016-10-26 15:03:45:950 INFO  {db=OSystem} 1 operations were processed, current LSN is OLogSequenceNumber{segment=0, position=623374} last LSN is OLogSequenceNumber{segment=0, position=623380} [OLocalPaginatedStorage]
2016-10-26 15:03:45:951 INFO  {db=OSystem} Storage data recover was completed [OLocalPaginatedStorage]com.orientechnologies.orient.core.exception.OConfigurationException: Error on deleting 'replicator' user
    at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.config(ODistributedAbstractPlugin.java:173)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.config(OHazelcastPlugin.java:98)
    at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1230)
    at com.orientechnologies.orient.server.OServer.activate(OServer.java:397)
    at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:46)
Caused by: java.io.FileNotFoundException: /home/orientdb/orientdb-community/config/orientdb-server-config.xml (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
    at java.io.FileWriter.<init>(FileWriter.java:90)
    at com.orientechnologies.orient.server.config.OServerConfigurationLoaderXml.save(OServerConfigurationLoaderXml.java:122)
    at com.orientechnologies.orient.server.config.OServerConfigurationManager.saveConfiguration(OServerConfigurationManager.java:107)
    at com.orientechnologies.orient.server.OServer.dropUser(OServer.java:786)
    at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.config(ODistributedAbstractPlugin.java:171)
    ... 4 more

Please note:

• It cannot write the pid file. The pid file location can be changed in ~/orientdb-community/bin/server.sh

./server.sh: 108: ./server.sh: cannot create /home/orientdb/orientdb-community/bin/orient.pid: Permission denied

• We need to set the proper amount of Maximum Heap

2016-10-26 15:03:44:486 WARNI Not enough physical memory available for DISKCACHE: 2,938MB (heap=1,820MB direct=524,288MB). Set lower Maximum Heap (-Xmx setting on JVM) and restart OrientDB. Now running with DISKCACHE=256MB [orientechnologies]

• The file /home/orientdb/orientdb-community/config/orientdb-server-config.xml need to be writable.

Caused by: java.io.FileNotFoundException: /home/orientdb/orientdb-community/config/orientdb-server-config.xml (Permission denied)

I just notice that root own orientdb directory (i.e. orientdb-community-2.2.12).
If there is no reason I ignore, I think is better to assign the ownership of orientdb directory to orientdb user because it runs as orientdb user.

@andrea.dellamico@isti.cnr.it fixed the ansible script. I run it. I'll check if everything is ok now.

The tags is created but at startime the xml is rewritten so that is not at the end as expected and I didn't notice it.

The last thing to be fixed in ansible script is that is a new version of hooks jas is available the new one is installed but the old one is not removed.

-rw-r--r--  1 root root 21006619 ott 26 12:15 resource-registry-orientdb-hooks-1.0.1-20161026.093908-29-jar-with-dependencies.jar
-rw-r--r--  1 root root  4253240 ott 26 18:14 resource-registry-orientdb-hooks-1.0.1-20161026.161036-30-jar-with-dependencies.jar

I'm going to remove the old one manually but it should be fixed in the ansible script.

Luca Frosini wrote:

The last thing to be fixed in ansible script is that is a new version of hooks jas is available the new one is installed but the old one is not removed.

-rw-r--r--  1 root root 21006619 ott 26 12:15 resource-registry-orientdb-hooks-1.0.1-20161026.093908-29-jar-with-dependencies.jar
-rw-r--r--  1 root root  4253240 ott 26 18:14 resource-registry-orientdb-hooks-1.0.1-20161026.161036-30-jar-with-dependencies.jar

I'm going to remove the old one manually but it should be fixed in the ansible script.

The only way I see is adding a task that removes a list of files. I added the task to the orientdb playbook, and the variable:

orientdb_hooks_to_be_removed:
  - resource-registry-orientdb-hooks-1.0.1-20161026.093908-29-jar-with-dependencies.jar  

I thought about this proposal but it is difficult to keep aligned.
Can we remove all files matching resource-registry-orientdb-hooks-*-jar-with-dependencies.jar before installing the new one.

Luca Frosini wrote:

I thought about this proposal but it is difficult to keep aligned.
Can we remove all files matching resource-registry-orientdb-hooks-*-jar-with-dependencies.jar before installing the new one.

In this way we are going to remove all the hook files at every playbook run, install the current one again, and then restart all the nodes. It seems more riskier that leaving an old file by mistake, to me.

This playbook is launched to upgrade orient or the hook. In both cases orientdb must be restarted to have effect.
Moreover if two hooks are present and orientdb is restarted we don't know which one is loaded.