D7.2 Assessment of the services and recommendations v1¶

Authors: Katerina Iatropoulou (ARC)
Reviewers: Achilleas Vassilopoulos (icre8), Antonis Lempesis (ARC)

Introduction¶

This deliverable aims to present the reports relative to the questionnaires filled in by users involved in the first phase of the Pilots via D7.1 “Testing plan and assessment questionnaire”, which include evaluation and recommendations of improvements for the Research Community Dashboard.

Research Community Dashboard overview¶

The Research Community Dashboard is designed for research communities and aims to give them the ability to use services to access, discover, maintain, enrich and monitor an up-to-date knowledge graph that contains their research artefacts. The research artefacts include literature, research data, software, and other research products (e.g. workflows, protocols, scripts, algorithms, etc.). In the community’s knowledge graph the research artefacts are linked to funders, projects, publications and datasets, that are integrated and inherited from OpenAIRE.

The Research Community Dashboard offers to each community a dedicated portal and an administration tool. The portal presents the artefact knowledge graph and offers a portfolio of common functionalities that are configured and adapted to the community’s research production and practices. The administration tool is a web application that allows the configuration and the on-demand deploy of the dedicated portal, in a way that meets the community’s specific needs.

The deliverable D4.4 “Research Community Dashboard: specification and release plan” contains a detailed description of all the functional specifications of the Research Community Dashboard.

Users¶

The Research Community Dashboard serves the needs of research communities that wish share, re-use, promote, monitor, report and evaluate scientific results related to the subject of their research.

The communities participating in the OpenAIRE Connect Project are:

• Fisheries and Aquaculture Management
• Environment and Economy (Sustainable Development Solutions Network - Greece)
• Digital Humanities & Cultural Heritage
• Neuroinformatics
• Earth and Environmental Science (European Marine Science)

For the Research Community Dashboard, we have 2 kinds of users: the researchers and the research operators. In the context of this deliverable the research operators are called “research managers”.

• Researchers

Researchers need to share (for “discovery” and “transparent evaluation”) their scientific results and re-use (for “reproducibility”) those of other researchers.

• Research Managers

Managers need to promote, monitor, report and evaluate scientific results of research within their community.

Methodology¶

For the evaluation of the Research Community Dashboard remote synchronous and asynchronous usability testing was carried out. Two sets of usability tests were designed. The first aimed the research managers that were invited to use both the portal and the administration tool. The second was addressed to the researchers of each community who were requested to use only the community’s portal.
More details on how the users were approached and how the testing was planned can be found on deliverable D7.1 “Testing plan and assessment questionnaire”

Working with Research Managers¶

For the first phase of the testing, a webinar to present all the features of the Research Community Dashboard was scheduled. More specifically ARC made a demonstration of the portal and the administration tool. For the administration tool ARC gave instructions on how managers could use the available configuration possibilities to achieve the best results for their community. During the webinar, the managers could interact with ARC asking questions, sharing their first impressions and making remarks.

After this first input, ARC invited all managers to navigate through the Research Community Dashboard, use the functionalities and give their comments and remarks in two weeks’ time. The feedback was given by e-mail exchange and by answering a questionnaire (text version: https://docs.google.com/document/d/1Z81OF4pIbHkUJgJntaFyuiS_AJtgz6HYghQuNEDMpjg/edit#heading=h.8c5x5wsx83x8) especially designed to capture both the administration tool and the portal functionalities. ARC was also available for a virtual call if that was considered necessary.

Working with Researchers¶

For the first phase, researchers tested the Research Community Dashboard via online sessions, during which they were given scenarios to execute. These scenarios were sets of specified tasks specially designed to evaluate the functionalities of the community portal.

ARC implicitly got feedback for the Research Community Dashboard by observing the researchers as they were executing the given scenarios. This was possible as researchers agreed to share their screen and to follow the talk aloud protocol. Participants were requested to describe their actions while they were performing the given tasks and ARC evaluators were taking notes of what participants said and did, highlighting the encountered difficulties.

After the online sessions, ARC asked researchers to complete an off-line questionnaire (text version: https://docs.google.com/document/d/1Z81OF4pIbHkUJgJntaFyuiS_AJtgz6HYghQuNEDMpjg/edit#heading=h.8c5x5wsx83x8) especially designed to capture the portal functionalities.

9 users participated in the assessment of the portal and 5 users evaluated the administration tool: 1 manager and 2 users from “Sustainable Development Solutions Network - Greece (Environment and Economy)” community, 1 manager and 2 users from “Fisheries and Aquaculture Management”, 2 managers from “Neuroinformatics” and 1 manager from “Digital Humanities and Cultural Heritage”.

All the managers answered the questionnaires but not all researchers did, even though reminders were properly sent.

Figure 1. Distribution of administration tool users who answered the questionnaires by community

In the next sessions, the results of all the usability tests (questionnaires, e-mails exchanged, online session observations) will be presented accompanied by useful recommendations for the Research Community Dashboard.

Results and Recommendations¶

Portal¶

The general reception of the portal of the Research Community Dashboard was positive.

Figure 2. Researchers' general impression

Figure 3. Managers' general impression

Users could easily navigate through the pages and the information was clearly presented and therefore easily discovered.

Figure 4. Researchers information discovery impression

Figure 5. Managers information discovery impression

In the next paragraphs, we present you the most important results of the usability tests together with proposals that could help improve the identified problems.

Layout¶

The layout was clear and attractive. The only remark that ARC got from some of the managers is that it would be nice if the pages of each community could be distinguished and easily identified not only by their logo, but also by additional images that they could upload and assign to different pages.

Search & Browse¶

The use of the search and browse functionality was smooth. The way to make keywords searches and filtering of the results was clear.

Figure 6. Researchers' search & browse impression

Figure 7. Managers' search & browse impression

As the following graph reveals, most of the users feel that a general search page is missing. Currently they can query for each type of artefact separately. A general search page is one where search results for all different kinds of community’s artefacts are available under one search form, i.e. under a single keyword search. This approach gives a better overview of the community’s content and reveals, in an intuitive way, the richness of the community’s knowledge graph. Additionally, it facilitates the discovery of content and information.

Figure 8. Need for general search page

One last request from the users was to provide more options for the display of the search results. Sorting, selection of number of results per page and in some cases richer filters are some of the recommendations that can be adopted.

Linking¶

As the rather low rating reveals, the linking procedure was difficult both for researchers and for managers.

Figure 9. Linking scores for researchers

Figure 10. Linking scores for managers

One main problem is the terminology used. Terms like “concept” and “category” are not clear. They were connected to a more technical phrasing rather than the language that the researchers would use. A change of the phrasing is recommended as an effective way to make the procedure clearer.

Another difficulty was that users could not be sure what were the exact artefacts that they selected to link. Two were the reasons why: First they could not easily view all the metadata necessary to identify an artefact and second it was not clear when the artefact was selected and is made available for linking.

Richer information of the metadata of the artefacts that are candidate for selection and a more obvious visual notification when an artefact is selected will be the first actions to improve the linking functionality. A need to redesign the whole procedure might be necessary, but this will be re-evaluated after the application of the mentioned fixes, on the second phase of the assessment.

Monitoring¶

The charts presented were considered interesting but users expressed their will to see more charts related to research data.

One identified problem in the monitoring pages was that in many cases when there is a big number of charts, due to scrolling, users were missing the tabs of the various monitored artefacts. Sticky menus on the left will be added to help users to have a complete overview of the existing statistics together with easy access to all different artefacts charts.

Additional Comments¶

Users brought up the idea of having an automatic way of inviting other people to visit their community’s portal. A special icon that will automatically create and send an invitation to specified e-mail addresses can be provided.

Administration tool¶

Managers had an overall positive impression for the administration tool.

Figure 11. Configuration impression

Some remarks were made that in some cases the terminology was more technical, but that can be avoided by changing the phrasing to what they believe is more suitable.

Layout Configuration¶

As it is already mentioned in the previous section, some managers would like to be able to upload additional images to the pages of their community that will help them differentiate their portal from those of other communities.

Link Management¶

The management of the link was quite straightforward. There was one requested addition by most of the managers; to receive e-mail notifications when a new link related to their community is created. That will allow them to have better control of the created links to avoid any mistakes introduced by the users.

Figure 12. New link creation notification

They also confirmed that it is useful to be able to link the research products of their community to others communities also. In that way ARC verified that the linking functionality needs to keep its flexible character.

Figure 13. Need for linking with other communities

User Management¶

The user management part of the administration need to be enhanced with new functionalities.

Even though not all managers agree on having an open subscription possibility for all users, most of them would like to have the option of receiving notifications when a user is registered and all of them would like to have the option of controlling who is subscribed to their community and can at least remove users.

Figure 14. Subscription approval

Figure 15. User registration notification

Figure 16. Subscribers list management

For the special case where a user is a community manager most of them agree that there should be a notification sent to the other managers of the community when changes happen to the community managers list (a manager is added or removed)

Figure 17. Manager list change notification

Conclusions¶

The Research Community Dashboard received very positive and encouraging comments that its development is moving towards the right direction. There were identified issues and new ideas brought to the table, that will help to improve the functionalities and the overall user experience. Based on the results presented in this deliverable and the recommendations made, the D4.4 "Research Community Dashboard: specification and release plan" will be updated.