D6.1 OpenAIRE-Connect Maintenance Report (v1.4, 2nd of July 2019)¶
Authors: Marek Horst -- ICM
Revisions¶
v1.1 (Jun 29, 2018)¶
First version.
v1.2 (Jan 24, 2019)¶
Updating Maintenance Report up to the M24 - December 2018.
v1.3 (Mar 05, 2019)¶
Updating "OpenAIRE Connect services contribution to Information Space" section with bulk tagging results obtained on beta infra in March 2019.
v1.4 (Jul 02, 2019)¶
Updated version with the following changes applied:
- updating generic description related to production deployment of OpenAIRE-Connect services
- Catch All Broker production deployment details
- Research Community Dashboard production deployment details
- updating Publishing APIs current state description
- updating monitoring description and links pointing to monitoring services
- adding incidents in downtimes section
- updating numbers in "OpenAIRE Connect services contribution to Information Space" section
Overview¶
This document contains information about the status of the deployment and operation of the OpenAIRE-Connect services and their contribution to the Information Space. The official maintenance of the OpenAIRE-Connect services began on January 1st, 2017 when the project started.
All OpenAIRE-Connect services are being deployed in the already operating OpenAIRE beta and production infrastructures located at the ICM OCEAN datacenter.
A maintenance report related to the core OpenAIRE infrastructure is available on OpenAIRE 2020 (covering 2015.01.01-2018.06.30 operation period) and OpenAIRE Advance (operation period starting from 2018.01.01) wiki pages.
Services¶
Services deployment¶
Both 6.1 and 6.2 milestones were reached in M16 (April 2018) resulting in the successful deployment of the Research Community Dashboard and Catch-All Broker Service in the BETA infrastructure. All services are compliant with TRL6.
2nd release deployment of M4.3 (RCD) and M5.3 (Catch-All Notification Broker) software products in BETA infrastructure was performed in November 2018 (M23) resulting in a 2nd testing round.
Final TRL8 services, M4.4 and M5.4 software products, deployment in the OpenAIRE production infrastructure was originally scheduled for M27 (March 2019) however:
- Catch-All Broker deployment was anticipated to M21 to better support the pilot activities with relevant content
- Research Community Dashboard deployment was slightly delayed and will be performed after introducing Big OpenAIRE Research Graph on production
More details on the rationale behind the deviation from the original schedule can be found in the relevant sections below.
Research Community Dashboard¶
The OpenAIRE Research Community Dashboard (RCD) suite consists of two portals: a dashboard and an administration one.
The Dashboard is the main research community entry point providing access to a rich set of functionalities for Open Science publishing:
- research impact management and monitoring
- search and browse of community artefacts
- claiming of community artefacts and links between them
- deposition of metadata and files for community artefacts
It was deployed on a dedicated machine and can be accessed at:
https://beta.connect.openaire.eu
A separate administration portal’s purpose is to adjust RCD’s configuration, tailoring functionalities to community specific needs. Due to its lightweight nature it was deployed on a demo.openaire.eu machine, sharing resources with a few other already deployed BETA services, and is available at:
https://beta.admin.connect.openaire.eu
Research Community Dashboard is going to be deployed in production after introducing the Big OpenAIRE Research Graph. The reason for this dependency is that we want to avoid the loss of records in the dashboards when moving RCD from beta infrastructure (already providing Big OpenAIRE Research Graph) to production. The new OpenAIRE Research Graph is planned to be released on production by the end of July 2019.
Both Research Community Dashboard portals were developed using the Angular framework.
The research Community Dashboard is supported by a set of newly deployed services:
- Administration Tool API
- written in java
- deployed on a new
openaire-services2-betamachine on a tomcat server - communicating with a Mongo database already deployed in the OpenAIRE beta infrastructure as the storage layer on
openaire-services-beta
- Statistics API
- written in java
- deployed on a new
openaire-services2-betamachine on a tomcat server - communicating with a Redis database already deployed in the OpenAIRE beta infrastructure on
openaire-stats-beta
- Login Service
- two instances for each portal
- written in java
- deployed on a new
openaire-services2-betamachine on a tomcat server - communicating with a Redis database already deployed in the OpenAIRE beta infrastructure on
openaire-stats-beta
- Mining Service
- written in python
- deployed on a new
openaire-webservices-betamachine on a tornado web server - uses madis, an extensible relational DB based on SQLite
Publishing APIs¶
Publishing APIs were originally designed to become OpenAIRE APIs extension to support 3rd-party services at publishing interlinked and packaged artefacts into OpenAIRE, conforming to the new interoperability guidelines defined in WP3.
Due to the existence of Zenodo Deposit API made by CERN and providing functionalities which are overlapping with Publishing APIs it was agreed to suspend the implementation of a new API and to push for the adoption of the already existing Zenodo API. A number of discussions followed with CERN and members of the research communities highlighted that Zenodo Deposit API was already offering a super set of the needed functionality and that the co-existence of an additional API could have led to duplication of effort, possible sustainability issues, and misunderstandings among the adopters.
Deliverable D4_5 on Publishing APIs implementation plan was already released and got updated at the end of the project with detailed description of current state and rationale behind pulling back from Publishing APIs further development.
Catch-All Broker Service¶
The Catch-All Notification Broker Service serves content providers with services enabling notification-based exchange of research artefacts. Such notifications could carry information related to extra metadata about records they already have, wrong metadata in the records they already have or new metadata records they were unaware of.
The frontend service was integrated into the OpenAIRE Content Provider Dashboard and is available on beta:
https://beta.provide.openaire.eu/
while production service is available at:
According to the release plan the CAB Service should have been deployed in the production infrastructure by M28. However, its deployment was anticipated to M21 to better support the pilot activities with relevant content. In fact beta and production environments are based on two independent aggregation systems, each comprising different data sources and content, with the latter including more than the first. As the focus was moved from the beta to the production environment, the resources allocated on the beta backend services (especially Elasticsearch) were redirected to support the development activities, freeing more resources for the production system.
The backend software was written in java and is deployed as a spring-boot application using linux systemd-based approach on a new dedicated openaire-lbs machine. It heavily relies on ElasticSearch 4-nodes cluster already deployed in the OpenAIRE OCEAN infrastructure. Broker sends notifications using an SMTP server which was also already available in the infrastructure located on bwnmail.icm.edu.pl.
Technologies behind services¶
The following technologies were used in WP4 and WP5 services development:
- Research Community Dashboard
- Java8
- Apache Tomcat
- Angular and Node.js
- NPM and PM2
- Redis db 2.8.4
- Python 2.7
- Tornado web server 2.1.1
- Madis
- MongoDB 3.2.6
- Catch-All Broker Service
- nginx
- Apache Tomcat
- Angular
- Java8
- Spring Framework
- ElasticSearch 5.5.1, 4 nodes cluster
- embedded H2 db
- SMTP server
Services monitoring¶
Production services were integrated with Prometheus and Grafana frameworks which were already deployed in the OpenAIRE infrastructure and are available at:
https://prometheus.openaire.eu
https://checkme.openaire.eu
First instance is focused on both whitebox and blackbox monitoring of OpenAIRE services while checkme was deployed outside ICM facilities (at CNR premises) and is focused on monitoring uptime of publicly deployed portals and dashboards.
Catch-All Broker service monitoring dashboard is available at:
https://prometheus.openaire.eu/d/dg-q2m_iz/lbs?orgId=1&var-env=production&var-job=literature-broker
while Research Community Dashboard site uptime is being monitored at:
Currently both Grafana instances have restricted access allowed for OpenAIRE developers and system administrators only. It will be made publicly open as soon as appropriate dashboards serving non-sensitive data are prepared.
All the other standard monitoring procedures are compliant with the ones already described in the OpenAIRE 2020 Maintenance Report which is available here:
OpenAIRE-Connect services contribution to Information Space¶
Details about the content policy (i.e. what are the criteria by which a record is assigned to a community) are available in each RCD instance at the URL template https://beta.[community].openaire.eu/content, example: https://beta.mes.openaire.eu/content
Number of claimed records for the OpenAIRE-Connect communities
| community name | count |
|---|---|
| Neuroinformatics | 37 |
| SDSN Greece | 0 |
| Marine Environmental Science | 11 |
| Fisheries and Aquaculture Management | 25 |
| Digital Humanities and Cultural Heritage | 20 |
| Total | 43 |
(table updated on July 1st, 2019)
Number of ingested records
Number of harvested records which already feature the community tag or that we can assume to contain the community tag. This is the case for research products collected from the content providers selected by community managers, research products linked to selected projects, those available from the specified Zenodo communities and research products whose metadata features at least one of the keyword specified by the community managers. More details on the content policy of the RCD is available at the URL template https://beta..openaire.eu/content, example: https://beta.mes.openaire.eu/content
Total number of products that have been related to the communities thanks to content providers, keywords, and Zenodo communities: 769 238 (March 2019), 1 084 029 (June 2019).
The following tables inform about the number of selected content providers, projects, zenodo communities and keywords, together with the number of research products that have been related to the community because of them.
| community name | number of selected providers | number of research products |
|---|---|---|
| Neuroinformatics | 6 | 4000 |
| SDSN Greece | 0 | 0 |
| Marine Environmental Science | 0 | 0 |
| Fisheries and Aquaculture Management | 13 | 2953 |
| Digital Humanities and Cultural Heritage | 5 | 88663 |
Note that for SDSN Greece and for Marine Environmental Science the counters above are 0 because community managers have not identified any content provider whose whole content is relevant for the community.
| community name | number of selected projects | number of research products |
|---|---|---|
| Neuroinformatics | 8 | 70 |
| SDSN Greece | 496 | 6038 |
| Marine Environmental Science | 662 | 14947 |
| Fisheries and Aquaculture Management | 186 | 8637 |
| Digital Humanities and Cultural Heritage | 29 | 3902 |
As of June 2019, 1303 research products have been added to the RCDs because they were deposited in one of the Zenodo communities selected by RCD managers, in details:
| community name | number of selected Zenodo communities | number of research products |
|---|---|---|
| Neuroinformatics | 16 | 100 |
| SDSN Greece | 0 | 0 |
| Marine Environmental Science | 48 | 886 |
| Fisheries and Aquaculture Management | 10 | 67 |
| Digital Humanities and Cultural Heritage | 12 | 250 |
| community name | number of selected keywords | number of research products |
|---|---|---|
| Neuroinformatics | 18 | 29199 |
| SDSN Greece | 17 | 0 |
| Marine Environmental Science | 5 | 12911 |
| Fisheries and Aquaculture Management | 18 | 10651 |
| Digital Humanities and Cultural Heritage | 51 | 525527 |
The 0 research products for SDSN Greece is due to the fact that the RCD managers types keywords in the form "SDGXX -" (example: SDG13 - Climate action) that cannot be found in the metadata and full-texts of publications.
Number of deposited artefacts
*Zenodo communities relative to OpenAIRE communities have been created in December 2018, apart from the one of Fisheries and Aquaculture (created on 2017).
| community name | number of deposited products | Zenodo community |
|---|---|---|
| Neuroinformatics | 1 | https://zenodo.org/communities/oac_ni/ |
| SDSN Greece | 0 | https://zenodo.org/communities/oac_sdsn-greece |
| Marine Environmental Science | 0 | https://zenodo.org/communities/oac_mes |
| Fisheries and Aquaculture Management | 242 | https://zenodo.org/communities/fisheries |
| Digital Humanities and Cultural Heritage | 0 | https://zenodo.org/communities/oac_dh-ch/ |
Backup and restore policies¶
The ElasticSearch cluster, utilized by the Catch-All Broker Service, relies on High Availability and replication core features, which are provided by the platform’s design.
The Research Community Dashboard and Publishing APIs underlying data sources obey to the generic backup procedures described in the OpenAIRE 2020 Maintenance Report.
Services downtime¶
There were two major incidents related to OpenAIRE contents deposition repository - Zenodo, both caused by database related failures. First one took place on August 17, 2018 and was resolved after 2.5 days, second one caused 27 hours of downtime starting from May 16, 2019 at 15:30 UTC. In both cases, the transactions logs of the database that is critical for ensuring point-in-time recovery got corrupted due to bugs in the database management infrastructure (not the database itself). In both cases, the long downtime was caused by a decision to avoid data loss instead of immediately restoring a backup which would have meant ~12 hours of data loss.
On December 18, 2018 an unplanned 2h downtime of all services took place in ICM infrastructure.
More details can be found in OpenAIRE Advance maintenance report, system dowtimes section.
List of administrative operations¶
Detailed list of all administrative operations undertaken @ICM is available on changelog for servers administration operations.
