Project

General

Profile

Wiki »

D7.1 Testing plan and assessment questionnaire

Authors: Miriam Baglioni, Alessia Bardi, Paolo Manghi, Claudio Atzori -- ISTI-CNR
Leo Mack -- JISC
Contributors: Argiro Kokogiannaki, Katerina Iatropoulou -- ARC
Pedro Principe -- UMinho

1. Introduction

The aim of this wiki is to describe in details the testing phases the Research Community Dashboard (RCD) service and the Catch-All Broker (CAB) service will undertake, and provide the assessment questionnaire to be made available online.

2. Research Community Dashboard Testing

The RCD service is intended for communities to provide researchers with the services giving access to facilities for collaboratively maintain an up-to-date knowledge graph of their interlinked or packaged research artefacts, e.g. literature, data, software, etc.
This research community graph will integrate and inherit (from OpenAIRE) links to funders, projects, literature and datasets as inferred from article full-texts or harvested by OpenAIRE from content providers.
Each community will be able to request and configure its own RCD according to its specific needs. The RCD features available to the users are provided in Deliverable 4.4

The RCD provides access for two distinct kind of users: the researchers (http://beta.connect.openaire.eu) and the community managers (http://beta.admin.connect.openaire.eu). The researchers are allowed to access and use the functionalities provided by the RCD, while the community managers can configure the dashboard associated to a community.
Both researchers and community managers contribute to the testing sessions.

2.1 Testing functionalities

This test plan is concerned with testing and refining the functionalities of the RCD as described in Deliverable 4.4. More specifically, tests will guide users at testing and collecting feedback on the following functionalities:

  • Researchers

    • Registration
    • Discovery, browse, and save (related to the functionalities in Deliverable 4.4 Sections 2.2.1 and 2.2.2)
    • Deposition (related to the functionalities in Deliverable 4.4 Section 2.2.4)
    • Claiming (related to the functionalities in Deliverable 4.4 Section 2.2.3)
    • Interlinking (related to the functionalities in Deliverable 4.4 Section 2.2.3)

  • Community managers

    • User Management (related to the functionalities in Deliverable 4.4 Sections 2.2.6 and 2.2.7)
    • Mining configuration (intuitiveness) (related to the functionalities in Deliverable 4.4 Section 2.2.5)
    • Statistics configuration and visualization (related to the functionalities in Deliverable 4.4 Section 2.2.8)
    • RCD configuration: (related to the functionalities in Deliverable 4.4 Section 2.2.9)
      • data sources,
      • projects,
      • vocabularies,
      • Zenodo communities to be mapped onto this community

The aim of the tests is to improve functionalities and their usability and provide advice on missing functionalities.

2.1.1 First testing session

The first testing session is related to the first beta release of the service. In this session, each community will identify 2 or 3 people belonging to the organisation involved in the project associated to each research community, which will participate to the tests.

The functionalities each user is called to test are shown in the following Test Template:

  • Researcher: deposition, claiming, interlinking tests
    • Deposition in Zenodo (literature, software, datasets, other) for a total of 20 products for research community, at least one of each kind (if any in the community)
      • Literature: if not already deposited somewhere else (otherwise they could become claims), valuable depositions may be project deliverables or preprints of published articles
      • Datasets: any dataset that has not yet been deposited and is related with an article
      • Software: any research software relevant to the community and possibly (not mandatorily) related to an article
      • Others: any community object that is not dataset is not a software and is related with an article (e.g. workflow, research object, etc.)
    • Claiming to community (literature, software, datasets, other) for a total 20 for research community, at least one of each kind (if any in the community)
      • Literature: claim of articles in OpenAIRE or via DOI or via ORCID that belong to the community
      • Datasets: claim of datasets in OpenAIRE or via DOI or via ORCID that belong to the community
      • Software: claim of software in OpenAIRE that belong to the community
      • Other products: claim of other products in OpenAIRE that belong to the community
    • Interlinking (literature, software, datasets, other) for a total 20 for research community
      • Interlinking literature with datasets or software objects
      • Interlinking other products with datasets, literature and software
  • Researcher: discovery, browse, and save test
    • For each user at least five operation of search and save
  • Community Managers
    • Statistics configuration
    • User management
    • RCD configuration:
      • data sources
      • projects
      • vocabularies
      • subjects
      • Zenodo communities to be mapped onto this community

For all the community two online meetings will be organised between M16 (April 2018) and M20 (August 2018) where the RCD portal will be shown. The first meeting will be dedicated to RCD managers, the second to RCD researcher type of users. The RDC users the can perform their tests under the supervision of ARC and/or CNR and provide direct feedback on the dashboard functionalities and usability. At the end of the testing period answers to the questions provided in the questionnaire will be collected. There will be two specific questionnaires, one for each kind of user of the RCD. The outcome of the questionnaires and the direct feedback gathered from the users will drive the development of the RCD for the second beta release.

2.1.2 Second testing session

This testing session is related to the second beta release of the service. In this session an extended number of testers belonging to different organisations will be employed to test the functionalities of the RCD. These testers will be instructed on how to use the RCD by webinars held to present the service and teach how to use it. Each community should engage at least five more people and should also disseminate the link to the dashboard via their existing networks in order to possibly attract additional researchers interested in Open Science publishing.
In this session the new testers and the ones from the first testing session will test the dashboard without the supervision of ARC/CNR and they will provide their feedback by filling the questionnaire derived by questionnaire that will be made available on-line.

The test template for the second testing session will be made available in Month 23 (November 2018) as an adaptation of the template for the first testing session so as to include the new dashboard functionalities, the configuration of the mining algorithms, and the feedback gathered from the previous testing session.
The second testing session will start in M23 (November 2018) and will end in M25 (January 2019). The end of the second testing session is prior to the one reported in the DoA to give one extra month to the developers for changes/updates to the dashboard.

2.2 Third testing session

This testing session is concerned with testing the production ready-ness of the RCD. The testers for this session will be RCD users from the previous sessions, and all the RCD users willing to test the service. The aim is to test user-friendly-ness, and effective acquisition of testing results. At M28 (April 2019) an adaptation of the test template will be delivered so as to include the new functionalities provided for the RCD release, and the results from the previous testing phase. At M28 (April 2019) the last testing session will start, and end at M30 (June 2019). The results of the final tests will constitute a public deliverable.

3. Catch-All Broker Testing

The “publish/deposit once” practice is a decisive mean in the overall Open Science roadmap, and will only be achieved when content providers seamlessly connect to the wider open scientific communication ecosystem. This will allow them to pro-actively exchange information about research artefacts and links of value to all interested communities or stakeholders, without the researchers having to worry about where, when, and how to publish in order to fulfil the numerous mandates.
The Catch-All Notification Broker Service connects all types of content providers (institutional repositories, publishers, data repositories, and CRIS systems).
Thanks to its functionality, providers can be notified of metadata records related to artefacts that are “of interest to them” (i.e. metadata records that should be in the content provider data base), or “linked to them” (i.e. a scholarly link exists between one of the provider artefact and the identified artefact). Notifications are sent only to subscribed providers, following a subscription and notification pattern.

The functionalities and events that CAB service users can exploit are reported in Deliverable 5.1 and in Deliverable 5.2. The first beta release of the CAB is accessible from https://beta.provide.openaire.eu.

The CAB will undertake three testing sessions: one for each release of the service.

3.1 First Testing Session

For the first beta release a webinar will be held in May for Portuguese providers and in June for Spanish providers which will test the CAB service. These providers will be asked to subscribe to events of the type:

  • ENRICH/MORE: ENRICH / MORE is a macro-category that groups events about publication field values that differ from those available in the repository. For example, a repository manager subscribing to the topic ENRICH / MORE / OPENACCESS_VERSION will be notified about Open Access versions of publications that are not already present in his/her repository.)
  • ENRICH/MISSING: ENRICH/MISSING is a macro-category that groups events about metadata field values that are not present in the metadata of the repository. For example, a repository manager subscribing to the topic ENRICH / MISSING / ABSTRACT will be notified when an abstract is found for the items in his/her repository which do not have an abstract in their associated metadata.

For this testing session only literature repositories will be asked to test the service. Once the testing phase will be completed they will be asked to reply to a questionnaire aimed at understanding their degree of satisfaction about the service, and how we could improve it. The first testing session for the CAB service will end at M20 (August 2018).

3.2 Second Testing Session

This testing session is related to the second beta release of the service. In this session an extended number of providers will be employed to test the service. A webinar will be held to present the service and teach how to use it.
In this session, providers will be asked to subscribe to at least one of all kind of events available and to provide their feedback by filling the questionnaire derived by questionnaire that will be made available on-line.

The second testing session will start in M23 (November 2018) and will end in M26 (January 2019) as for the RCD.

3.3 Third Testing Session

The last testing session is concerned with testing the production ready-ness of the CAB. The aim is to test user-friendly-ness, and effective acquisition of testing results from the previous sessions. The testers for this session will be the testers of the previous testing sessions and all the providers willing to test the service. On M28 (April 2019) the last testing session will start, and it will end on M30 (June 2019). The results of the final tests will constitute a public deliverable.

3.4 Broker Services Interoperability

In addition to the testing of the CAB with research community and content providers users, the interoperability between the CAB and Jisc’s Publications Router will be tested. This will assess the technical conditions, feasibility and utility of exchanging notifications between notification messaging systems (i.e. the CAB and Publications Router), which would enable the receiving systems to send additional notifications to their subscribers. The purpose of the testing phase is to demonstrate in a relevant environment (i.e. between the CAB and Publications Router) how notifications can be exchanged between systems. Tests will be conducted in a development environment, independent from the release and test cycles described above, and will not involve end-users/subscribers.

The interoperability pilot will test how the CAB can unilaterally push event notifications on literature metadata to Jisc’s Publications Router, which could then further disseminate these to relevant subscribers. To accommodate issues arising from the differences in how both services are designed and operate, the test will assess the following aspects affecting interoperability:

  • Notifications exchange: This will test the sending of notifications from the CAB to Publications Router as well as the suitability of different notifications classes.
  • Data payload and ingestion: This aspect concerns the data payload for notifications, the matching of data formats for both systems, and the data ingestion workflow on the side of Publications Router.
  • Non-circularity between systems: This will assess options to avoid circularity of notifications between systems. Crucially, Publications Router does not have a deduplication facility. With the CAB and Publications Router sourcing metadata from partly the same data providers and delivering notifications to the same subscribers, this could, if unmitigated, lead to circular notifications between systems.
  • Added notifications: This will aim to evaluate the scale of new notifications which would be created by Publications Router based on the CAB feeds, allowing an indicative assessment of the added value from a subscriber / end-user perspective.

The interoperability development and testing phase will commence in M21 (September 2018) and is expected to be completed in M28 (April 2019).

Add picture from clipboard (Maximum size: 8.91 MB)