D7.4 Assessment of the services¶

Authors: Katerina Iatropoulou (ARC), André Vieira (UMINHO)
Reviewers: Miriam Baglioni (CNR), Stéphane Pesant (UniHB)

Introduction¶

This deliverable consists of on-line reports relative to the final evaluation reported via the questionnaires by users involved in the Pilots via D7.1 for Research Community Dashboard (RCD) and Catch-All Broker Service (CAB).

Compared to the previous version “ https://support.d4science.org/projects/openaire-connect-wiki/wiki/D73”, this deliverable aims not only to test new functionalities for RCD but also to ask users who have visited the RCD before, to evaluate the improvements of the already existing ones. One other important aspect of the testing was to identify whether the information that is provided to users before visiting the actual Community Gateway and the Management Tool, succeeds to describe what exactly are the benefits of the RCD for the communities.
Regarding to the CAB, this testing phase was concerned with testing the production ready-ness of the CAB, and was important to get feedback and evaluation from the CAB users. In order to do that, the main aim of this phase was to organize interactive sessions with content providers in different contexts. OpenAIRE-Connect have engaged with COAR members and the Open Repositories conference community to run demonstrations of the service and test-drivers and collect feedback to improve the broker pilot. Two workshops were organized in May, attached to the annual conference of COAR, and in June within the Open Repositories conference program, and also a webinar organized within the Portuguese repository managers.

Methodology¶

For the evaluation of the Research Community Dashboard, the unmoderated remote usability assessment methodology was followed. More precisely two webinars took place, one for research managers and one for researchers. All users who participated in the webinars were asked to fill in online questionnaires. There were two sets of online questionnaires, one for researchers and one for research managers.

For the evaluation of the Catch-all Broker service, which is available in production since October 2018, through the Content Provider Dashboard, a set of activities of dissemination and training were performed, focusing on how content providers can enrich their collections using the Broker service. In addition, in each activity, participants were invited to give their feedback through interactive sessions - as for example within breakout sessions and answering simple questions using Mentimeter surveys - and also to assess the service answering an online questionnaire, which results will be shown below.

Working with Research Managers¶

For the third testing phase, nineteen (19) research managers attended the webinar where all the new functionalities of the RCD were presented, together with the improvements of the already existing ones. Ten (10) research managers answered the online questionnaire; two (2) research managers form Instruct-ERIC, two (2) form Sustainable Development SolutionsNetwork - Greece, one (1) from Neuroinformatics, one (1) from Aginfra - Agricultural and Food Sciences, one (1) from Digital Humanities and Cultural Heritage, one (1) from Elixir, one (1) from European Marine Science and one (1) from Fisheries and Aquaculture Management.

Communities of research managers who answered the questionnaires

Working with Researchers¶

For the third testing phase, nine (9) researchers attended the webinar where all the new functionalities of the RCD were presented, together with the improvements of the already existing ones. Eight (8) researchers answered the online questionnaire; three (3) form Sustainable Development SolutionsNetwork - Greece, one (1) from Aginfra - Agricultural and Food Sciences, one (1) from Digital Humanities and Cultural Heritage and three (3) from other communities - who are related to to Aginfra, European Marine Science and Dariah EU communities.

Communities of researchers who answered the questionnaires

Working with Repository Managers¶

During the first semester of 2019, UMINHO prepared two workshops and one webinar, aiming to provide detailed information on the main services and tools targeting content providers, with special focus on the Broker service, and in which OpenAIRE team foresee to collect contributions from the repository manager’s community to further develop the portfolio of services offered by OpenAIRE, namely the Content Provider Dashboard, Broker service, and others.
The workshops (in the Open Repositories Conference 2019, and in the COAR Annual Meeting 2019 ) provided an interactive demonstration of the dashboard for content providers tools (repositories registration and validation, collection monitor and content enrichments), detailing mainly the catch-all broker service functionalities, such as the metadata enrichments and the usage statistics service.

Additionally, a webinar for Portuguese repository managers about the OpenAIRE services for repository managers, highlighting the changes in the OpenAIRE Content Acquisition Policy (CAP), novelties in the OpenAIRE Guidelines for Literature Repositories and in the Content Provide Dashboard functionalities.

Assessment Results¶

RCD General Information¶

Over 60% of the users that took the test find it easy (scores 4 and 5) to understand what is a Community Gateway.

Managers answers on how easy was to find what a Community Gateway is (1:difficult, 5:very easy)

Researchers answers on how easy was to find what a Community Gateway is (1:difficult, 5:very easy)

80% of the managers find it rather easy (scores 3, 4 and 5) to learn how a community can be created.

1:difficult, 5:very easy

The functionalities that can be performed in the Community Gateway were rather easily spotted. The most common answer was the search functionality, followed by linking, statistics and deposit.

Managers responses on the Community Gateway functionalities (1:difficult, 5:very easy)

Researchers responses on the Community Gateway functionalities (1:difficult, 5:very easy)

Managers also find it rather easy to spot what are the functionalities they can do using the Management Tool. The management and enrichment of content was the most popular answer. The invitation of users and the user management, and the management of the statistics were also popular answers.

Community Gateway¶

For the testing of the gateway there were only 4 researchers that visited the gateways for the first time.

Researchers visiting the Community Gateway

Managers visiting the Community Gateway

For the users who have visited the Community Gateway before the impression of the changes on how the information is presented was either the same or positive, proving that the UX/UI redesign is on the correct direction.

Managers’ answers on the improvement of the presentation (1:worse, 5:better)

Researchers’ answers on the improvement of the presentation (1:worse, 5:better)

Home page¶

More than 80% of the users find easy (scores 4 and 5) to spot their Community Gateway.

Managers answer on how easy was to find their Community Gateway (1:difficult, 5:very easy)

Researchers answer on how easy was to find their Community Gateway (1:difficult, 5:very easy)

The graph analysis was easier for the managers to find rather than it was for the researchers.

Managers answers on how easy was to find the graph analysis (1:difficult, 5:very easy)

Researchers answers on how easy was to find the graph analysis (1:difficult, 5:very easy)

Search¶

The search functionality has also been improved, Almost 80% of the users gave a score of 4 or 5.

Managers’ answer on how easy the search functionality is (1:difficult, 5:very easy)

Researchers’ answer on how easy the search functionality is (1:difficult, 5:very easy)

Linking¶

Almost 70% evaluated positively the linking functionality and 80% of the users believe that the linking functionality has improved.

Managers’ answers on how easy was the linking functionality (1:difficult, 5:very easy)

Researchers’ answers on how easy was the linking functionality (1:difficult, 5:very easy)

Managers’ answers on how improved is the linking functionality

Researchers’ answers on how improved is the linking functionality (1: worse, 5: much easier)

Management tool¶

For the management tool the questions were focused on the new functionalities for the addition of personal and organization information. The overall impression was very positive, but the users pointed out that there is a need for a privacy statement for the personal information and an official agreement for the organizations.

Personal Info¶

1:difficult, 5:very easy

Organizations¶

1:difficult, 5:very easy

1:difficult, 5:very easy

Catch-all Broker Service¶

The Broker service evaluation and feedback during this final phase of assessment was made asking to answer an online questionnaire to all the participants in the two workshops and a webinar, being collected 18 answers which results are presented and analysed below.

Enrichment of events¶

The results obtained through the questionnaire show that for Content providers, the enrichment events are well presented and comprehensive however, despite the overall good results about ease search/refine, only two respondents considered it very easy to search/refine, so we should consider to improve this functionality to make the search process of events more efficient.

1: Insufficient, 5: Excellent

1: Very hard, 5: Very easy

Regarding the events relevance, the majority of content providers consider that are relevant or very relevant and the information contained on it complete, so according to this answers we can consider that the Broker service is deploying an added value for content providers helping them to enrich their collections.

1: Not relevant, 5: Very relevant

Enrichment Events¶

Despite of content providers being satisfied with enrichment events that Boker service provides, they demonstrated to be interested to have more event classes, considering valuable to have links between publications and author ORCIDs, software, datasets and links to projects for data repositories.

If yes, which one(s)?

• Licensing, ORCID
• About metadata integration with web services.

1. Author ORCID (references to author ORCID identifiers that can be associated to authors in your publications)
2. Software (references to research software projects that can be associated to your publications)
3. Dataset (references to research data that can be associated to your publications)
4. Links to projects for data repositories

Note: These events will be available in the upcoming service releases.

Events notification¶

Regarding to the events notification functionality, the majority of content providers never used it, and for those who have used, they considered that needs to be improved. Looking at these results, the events notification needs to be promoted and the usage benefits explained. The user interface also needs to be improved and new help texts to guide users on the usage of this functionality needs to be added.

Overall evaluation¶

Looking to the overall evaluation, the majority of content providers consider important or very important the utility of the content enrichment events feature, and good or very good their interface usability.

Is there any improvements you would like to suggest?

• CSV Import, API exposure of the results
• Truly it needs an API at the CRIS-level. It can be a great help.
• Add data export functionalities.
• It would be good if content enrichment becomes more automatic.

Contributions from the interactions with repository managers¶

The testing, training and dissemination activities targeting repository managers about the Broker service, contributed to have interactive moments with participants, being possible to collect useful feedback beyond the online questionnaire, namely about the service usage, user needs and requests for new functionalities. These interactions have represented a very enriching contribution to identify new improvements and new functionalities to develop and deploy in the Broker service, which will address the real user needs.

In the following table are listed the top requests collected from the repository managers.

Conclusions¶

The RCD received an overall positive feedback proving that the enhancements and improvements developed based on the results of all the testing phases were towards the right direction. The questionnaires reveal that although there are still some issues for both the Community Gateway and the Management Tool, they are very useful to the communities as tools that support Open Science for their communities. As the RCD becomes more complete and advanced in the information and the functionalities that it hosts, new requirements will arise and new opportunities and challenges for improvements will show up.

The feedback results from the CAB shows that the Broker service is deploying an added value service for the content providers, helping them to enrich the content of their repository collections. However, there is a way forward, continuing the service improvements, in order to offer new and relevant event enrichments, and also new functionalities to improve the user interface and the way that users can get information from the Broker service.