D7.3 Assessment of the services and recommendations v2¶

Authors: Katerina Iatropoulou (ARC), Argiro Kokogiannaki (ARC)
Editor: André Vieira (UMINHO)
Reviewers: Julien Barde (IRD)

Introduction¶

This deliverable aims to present the reports relative to the questionnaires filled in by users involved in the second phase of the Pilots via D7.1 “Testing plan and assessment questionnaire”, which include evaluation results and recommendations of improvements for the Research Community Dashboard - RCD, and the results of of the Catch-All Broker Service (CAB) testing sessions. The Broker Service is integrated within the Content Provider Dashboard - CPD.

Compared to the previous version D7.2 Assessment of the services and recommendations v1, aimed to test the functionalities of the first beta version of the RCD, this phase includes the improvements of the first phase, together with the following enhancements and new features:

• User invitations (all users)
• Zenodo communities (all users)
• Search general page addition (all users)
• Ordering in search pages (all users)
• Configurable number of results in search pages (all users)
• Sorting and search in filters (all users)
• Notifications (managers)
• Manage subscribers (managers)
• Manage Zenodo communities (managers)
• Manage notifications settings (managers)
• Edit HTML pages for Organization & About (managers)

The Catch-All Broker service, available through the Content Provider Dashboard, allows repositories, publishers or aggregators to exchange metadata and enrich their local metadata collection by subscribing to notifications of different types. The testing sessions with repository managers aim to test the service with real users and get useful feedback on how the service can be improved to meet the user needs.

Methodology¶

For the evaluation of the Research Community Dashboard, both remote and in person usability testings were carried out. More specifically the following sets of usability tests were designed:

• Unmoderated remote usability tests for research managers;
• In person moderated usability tests for researchers;
• Moderated and unmoderated remote usability tests for researchers.

More details on how the users were approached and how the testing was planned, can be found on deliverable D7.1 “Testing plan and assessment questionnaire”.

Working with Research Managers¶

For the second testing phase, a webinar was scheduled for the Research Community managers, where all the new functionalities of the RCD were presented. More specifically ARC made a demonstration of the new features of the portal and administration tool. During the webinar, the managers could interact with ARC, asking questions, sharing their first impressions and making remarks. Six managers from four research communities (“Environment and Economy”, “Fisheries and Aquaculture Management”, “Neuroinformatics”, “European marine science”) participated in the webinar.

After this first input, ARC invited all managers to navigate through the RCD, to use the new functionalities and give their comments and remarks in one week’s time. The feedback was given by e-mail exchange and by answering an updated questionnaire especially designed to focus on both the administration tool and the portal new features. Five managers (1 manager from each community) filled in the questionnaire as shown in the following graphic, while managers from two communities (“Neuroinformatics” & “Fisheries and Aquaculture Management”) send detailed feedback and suggestions through email.

Communities of administration tool users who answered the questionnaires

Working with researchers¶

For researchers, the Research Community Dashboard was tested in the following ways:

• In person moderated usability tests for researchers;
• Moderated and unmoderated remote usability tests for researchers.

The detailed interactions of both testing scenarios are presented below.

In person usability tests for researchers
The in person test took place on 28th of January 2019 at Cádiz, Spain, during a workshop specially organised by OpenAIRE Connect, within the OYSTER EuroMarine Young Scientist Working Group. Fifteen young scientists participated in this workshop, organized in three parts: a presentation of the research community portal, an individual testing session of the portal and a working group testing session of the portal.
During the presentation, the functionalities of Research Community Dashboard were presented to the users. Regarding the individual testing, users were asked to follow a testing scenario and to provide their feedback by filling an online comments form (15 users) about the functionalities of the portal. In the last part, users were divided in three groups. Each group within the given scenario followed the talk aloud protocol, and an evaluator was taking notes of what participants said and did, highlighting the encountered difficulties. In addition researchers provided feedback via a questionnaire (9 users), aiming to evaluate the research community portal.

Unmoderated remote usability tests for researchers
On the 8th of February 2019, a webinar was organised for all research communities users - during which the functionalities of the RCD have been presented. Users had the opportunity to ask questions, share their remarks and comments. This webinar had the participation of eleven users from four research communities (“Sustainable Development Solutions Network - Greece” 4 users, “Fisheries and Aquaculture Management” 1 user, “Neuroinformatics” 4 users, “Digital Humanities and Cultural Heritage” 2 users). After this webinar, ARC invited all users to navigate through the community portal, use its functionalities and provide feedback by answering a questionnaire specifically designed to capture the evaluation of the portal functionalities.

Moderated remote usability tests for researchers
The Research Community Dashboard was also tested by researchers through online sessions, during which they were invited to execute specific scenarios. These scenarios were sets of specified tasks specially designed to evaluate the functionalities of the community portal. Some of those researchers had already seen the webinar some others not.

ARC implicitly got feedback by observing the researchers as they were executing the given scenarios. This was possible as researchers agreed to share their screen and to follow the talk aloud protocol. Participants were requested to describe their actions while they were performing the given tasks and ARC evaluators were taking notes of what participants said and did, highlighting the encountered difficulties.
After the online sessions, ARC asked researchers to complete an off-line questionnaire9 especially designed to capture the evaluation of the portal functionalities.

The online sessions of the dashboard had the participation of six users from five research communities: 2 users from “Environment and Economy” community, 2 users from “Fisheries and Aquaculture Management”, 1 user from “Neuroinformatics” and 1 user from “Digital Humanities and Cultural Heritage”. The following graphic represents the percentage of researchers by research community that have answered the questionnaire during the testing phase.

Communities of researchers that answered the community portal questionnaire (online session & workshop)

In the next sections, the results of all the usability tests from questionnaires, comments form, e-mails and online session observations will be presented, accompanied by useful recommendations for the RCD.

Working with repository managers¶

The Content Provider Dashboard testing, with focus on the Broker Service functionalities, was carried out by real users of the system, i.e. repository managers for national networks of repositories, within a controlled environment.
Two webinars were organised, each of them targeted to Portuguese and Spanish repository managers, in which were presented the OpenAIRE services for repository managers included in the dashboard (Broker Service, Content Provider Dashboard), the metadata enrichment and curation (cases of RCAAP and RECOLECTA), and the OpenAIRE infrastructure and upcoming developments. All repository managers were invited to create an OpenAIRE account and access the Content Provider Dashboard in order to use and test the following services:

• the results from the Catch-All Broker service and the events generated for their repositories to enrich their collections.
• the subscription management service to receive events notifications.

Country	Attendees	Represented Repositories
Portugal	59	21
Spain	49	28

After the webinar, the support team guided 22 repository managers (12 from Portugal and 10 from Spain), invited them to use and test the Broker Service, namely the events generated to enrich their repository collections and the subscription of notifications for specific events. In order to evaluate the beta version, all repository managers participating in the webinars were asked to answer a questionnaire aimed to understand their degree of satisfaction about the service, and how it could be improved.

Results & Recommendations¶

Portal¶

The overall impression that we got for the community portal was mostly positive. More notably 60% of the users who have already visited the portal before found the updated version easier.

But there is also room for improvement. In the following sections, we report the specific weaknesses and we propose solutions to overcome them.

Linking¶

For the linking functionality many users requested step-by-step instructions displayed on screen, help texts and explanations. These requests made obvious that the whole procedure was not clear and needed to be redesigned.

The table below presents the main difficulties of the linking functionalities and the recommended solutions.

Problem Identified	Recommendations
not clear how to select a research result	new select and added buttons
not clear if the result is selected	basket with selected results always open on the right
results not well presented in search	new way to present metadata enhance with additional metadata when necessary
not clear how to finish the linking procedure	link button more visible (colour & placement on the page)

The following images are prototypes of the new updated version of the linking functionality. Following the previous recommendations, users can see the basket on the right, the updated layout for the buttons and the new appearance of metadata in the search results.

Link to projects prototype

Link to research results prototype

Search¶

The need to promote the search functionality in the initial screen of the community page, rather than have it as an option in the menu, was made clear from the testing. Users would like to have on the first screen the possibility to make keyword searches, as searching the contents of a community is the most anticipated action for researchers visiting the community portal.
Another search option that would be very useful for researchers, is to provide an easy way to search for a community’s research products by using the subjects/topics of this community to filter the results. One effective way to implement this is to add this functionality in the initial page, where all community subjects are displayed. Instead of simply presenting them, we include them as links that redirect to the related search pages.
In some cases the search between different entities (publications, research data, software, etc.) confused the users, as it was not clear to them for which entity they were searching. Legends mentioning the entity type will be added in each form. Additionally the option of having sticky forms that will always be visible on the top of the page, independently of the scrolling, will be adopted.

Based on the previous comments, the first page of the community portal was redesigned. The the following image is a screenshot of the new prototype.

Zenodo Communities¶

The basic issue identified for the Zenodo Communities is that most researchers are not aware of Zenodo, and as a result, they could not discern how Zenodo and its communities could be useful to them.

One solution can be to have help texts targeted to explain what Zenodo is and how Zenodo communities can enhance the contents of RCD. Another solution would be to make clear the Zenodo and Zenodo communities belong under the “Share” functionality.

Share¶

During the tests it was obvious that users didn’t understand the share pages (share publications/ research data), and that they weren’t sure how they could share their data and include them in the RCD. Thus, a page that will explain in simple words how a user can share their data is necessary to be created.

More specifically in share publications, there is a search form where users can locate repositories based on the institution they belong and a map where they can locate a repository on map. Users were lost on this page and couldn’t figure out what they were supposed to locate, the institution or the repository. Additionally it was noticed a confusion between their institution and their research community.

On both share publications and research data pages, users can search for all repositories available in OpenAIRE. By adjusting those pages to only include the repositories related to a community, the functionality will be less confusing. They will also focus more on the information that each community want to pass on to its researchers about the repositories they could use.

Administration tool¶

Managers had an overall positive impression of the Administration tool including the new functionalities (Manage Zenodo communities, manage subscribers, receive and manage notifications Edit HTML pages for Organisation & About).

Zenodo Communities¶

Some remarks were made regarding the management of Zenodo community in terms of usability. The buttons used to search and add a new Zenodo community weren’t intuitive enough to some users. Thus the layout and positioning of those buttons is expected to be improved.

HTML Information Pages¶

Regarding the configuration of the HTML pages, some users weren’t familiar with the WYSIWYG edition of an HTML form, but in general they managed it pretty well as most of them enabled the pages. Although, some users didn’t find them useful and decided not to enable them.

Email Notifications¶

Most users had received email notifications and they were satisfied with the content of the email.

Not all managers tried to manage the notifications settings yet, but they found it easy to configure the settings.

Managers are positive to receiving more notifications. For example, they would like to receive notifications when the content of RCD is updated.

Catch-all Broker Service¶

The answers to the questionnaire of the Catch-all Broker service (CAB) revealed that the service is very valuable to the repositories.

The testings of CAB had an overall positive evaluation from the repository managers, but it also revealed that improvements could be made to the completeness and the presentation of the information.

Based on these results, the creation of new help texts to guide users through each CAB functionality is recommended. Special focus should be given to:

• Improvement of the CAB interface, adding more information about aggregation history;
• Event enrichments: > ** Improvement of Content Provider Dashboard (CPD) user interface with enrichments of the information displayed about events; > ** Addition of help texts in CPD.

Conclusions¶

The Research Community Dashboard (RCD) received an overall positive feedback which reveals that the enhancements and improvements developed based on the results of the first testing phase are moving towards the right direction. The identified issues of the second testing phase will be addressed in the next version of the RCD, that will be available in PRODUCTION. The feedback results from the Catch-all Broker service (CAB) had an overall positive evaluation from the repository managers, contributing also to identify some recommendations to improve the service. These recommendations had been implemented contributing to improve the CAB, which is now available in PRODUCTION via the Content Provider Dashboard.