Learning agreement with “critical friends” as a form of assessing research units
This initiative, implemented by St. Pölten University of Applied Sciences (FHSTP), introduces a new, development-driven approach to assessing research units within the institution’s quality assurance (QA) procedures since 2023. The approach is inspired by the idea of “bench learning” used in the external QA systems of some northern European countries (like Finland and the Netherlands) and is based on the principles of CoARA (Coalition for Advancing Research Assessment). The process aims at the empowerment of the assessed research unit itself. External (international) reviewers participate as “critical friends,” and a learning agreement is initially stipulated between the research unit, the reviewers, and the internal QA unit.
The traditional challenge addressed by this initiative is the extensive involvement of external QA at the level of individual projects, proposals, deliverables, and publications, due to the dependency of Austrian universities of applied sciences on third-party funding for Applied Research and Development. In response, FHSTP chose a strategic institutional approach with a focus that is more development-driven for research units.
FHSTP being a signatory and member of CoARA, it adheres to the principles of the European Charter for Researchers, and implements the institutional White Paper on Quality in Research. The new approach “Learning Agreement with Critical Friends” draws inspiration from the European context, specifically the enhancement-oriented QA tool known as bench learning. Notably, this form of assessment focuses on research units (such as research institutes, groups, or centres) rather than individual researchers, who are already assessed for career progression.
The three main aims identified for this approach were:
- To open up research assessment to more diversity and plurality without creating an overwhelming multitude of indicators.
- To involve external experts while ensuring the focus remains on creating benefits for the assessed research units.
- To empower research units themselves in defining their priorities and contents for the research assessment, in agreement with the internal QA unit.
Implementation and impact
The implementation of this new approach began with concrete thought formulation in early 2023, building upon developments like CoARA and the institutional White Paper on Quality in Research. A pilot phase was conducted in 2023 and 2024, involving four research units.
The process currently consists of four major steps:
(1) Learning agreement: A signed agreement is concluded between the UAS’s research unit, the critical friends, and the evaluation unit/QM. This document details essential process dates/deadlines, a basic set of indicators, and discussion questions.
(2) Assessment workshop: This stage, described as the ‘centrepiece’ of the assessment, involves an open, transparent, professional exchange between the research institute and the critical friends.
(3) Assessment report: Following the workshop, the assessed unit drafts a report reflecting on the workshop content and formulating potential measures. The Critical Friends then read the report and have the opportunity to add comments.
(4) Reflection: An internal, institutional reflection process takes place. This reflection occurs at the process level (with the assessed units and the Committee for Quality Assurance in Research), and at the content level (a meeting between the units and the responsible managing director).
To date, four out of six research institutes at FHSTP have undergone assessment using this new method, with the remaining two expected to be completed by the end of 2025. Based on the workshop outcomes, every institute identified or is identifying a set of measures to be implemented in the following years, such as staff development support or the development of specific cooperations. These measures are reflected upon with the institutional leadership, and follow-ups are incorporated into the internal QA cycle. The open, intense, and empowering character of the activities has already led to the development of interesting cooperations with some of the critical friends. The approach has been presented to a broader community, including the CoARA Working Group Experiments in Assessment.
Transferability
This approach is highly adaptable and shows how the CoARA principles can be implemented in reality through an innovative, open, and transparent instrument. Feedback from the international QA community has been very positive. For ongoing use, each research unit will undergo this assessment every four to five years. There is currently a discussion about clustering the discussion questions (defined by the research units themselves for the workshop) into specified dimensions, allowing FHSTP to show progress over multiple QA cycles and encourage a broader approach.
Key lessons include:
- The approach is very inclusive and empowering because research units are involved from the beginning in defining the items for the learning agreement and choosing the critical friends.
- The institutional reflections at all levels were fruitful and interesting, supporting the enhancement potential of the institutes and their research activities.
Learn More
For further details, supporting materials include the paper, „Learning Agreement with Critical Friends als Pilotprojekt in der Forschungsevaluation“ (in German) published in Qualität in der Wissenschaft 2024.
We will be back next week with another inspiring good practice from one of our members. It is not too late to submit yours. If interested, please contact [email protected]
