This post has been cross-posted on the SLSA Blog and on Amanda’s blog Approaching Law.
The Hidden REF project is on a mission to get ‘diverse’ outputs to form five percent of all outputs submitted to REF. They note that REF rules have long allowed, and are now more actively encouraging, the submission of non-traditional research outputs (NTOs). Furthermore, ‘data released by the REF suggests that non-traditional outputs perform just as well as traditional outputs in terms of the proportion awarded a 4* rating’. Yet Higher Education Institutions remain reticent to submit them.
A recent episode of What the REF, a podcast series produced by the Hidden REF project, prompted me to wonder: What might output diversity look like in Law?
Allowing diversity in REF2021
REF2021 Guidance defined ‘outputs’ as the ‘products’ of research; and ‘research’ as a ‘process of investigation leading to new insights, effectively shared’. ‘Effectively shared’ was defined as meaning that the output is in ‘assessable form’, whether ‘published’, ‘publicly available’, or in a ‘confidential report’ (Annex C).
It emphasised that, ‘In addition to printed academic work, research outputs may include, but are not limited to: new materials, devices, images, artefacts, products and buildings; confidential or technical reports; intellectual property, whether in patents or other forms; performances, exhibits or events; and work published in non-print media.’ (Para 208)
Nevertheless, across all disciplines, 98 percent of output submitted to REF2021 were ‘publications’—that is books or articles (up from 97 precent in REF2014). Things appear to have been marginally worse than average in Law. By my count, 99.99 percent of the 5,855 outputs submitted to Panel 18 (Law) in REF2021 were entirely traditional—that is, journal articles (3752), book chapters (1121), books (872), or edited collections (26). Of the remaining .01 percent of submitted outputs, all but two were highly traditional in the sense that there were text-based (working papers, reports, or conference contributions). More on those exceptional two below.
Requiring ‘representative’ submissions in REF2029
The evolving REF2029 Guidance on the Contributions to Knowledge and Understanding places a special emphasis on ‘diversity’ of research outputs (5.6).
The REF2029 guidance retains the REF2021 definition of research outputs. It also follows REF2021 in noting that research outputs ‘go well beyond journal articles, conference papers, monographs and book chapters’ (5.6.4); and ‘may include innovative formats, accessible reporting formats, and modes of expression that may depart from historical conventions within specific fields and contexts’ (5.6.3).
However, it goes further than REF2021 in the sense that it promises to assess Units on the extent to which the outputs that they submit are ‘representative’—that is, ‘reflect the diversity of research practices and outputs produced within the disciplinary area’, ‘including the extent to which different and diverse types of research are included in the submission’ (7.1.1).
Submitting diverse Law outputs to REF2029
What kinds of outputs are eligible for submission to REF2029? The Guidance on Disciplinary Level Evidence Statements is still in development. For now we have an indicative list provided in section 5.6 of the general Guidance on the Contributions to Knowledge and Understanding. This mentions textual outputs of the kind which legal researchers have long undertaken, but which have not historically been celebrated in the wider research system (e.g. reviews, translations, policy summaries, and reports). It also mentions ‘non-textual (artefacts, audio, visual, multi-modal)’ and ‘creative, performance and practice-led outputs’.
As legal researchers increasingly turn to visual, material, and creative methods, Units will presumably need to include some of the resulting outputs if their REF2029 submissions are to be deemed ‘representative’. This turn seems to be especially prevalent among postgraduate and early career researchers. One recent example that comes to mind is (In)visible, a short choreopoetry film by PhD candidate Bhumika Billa, made in creative response to testimonies of injustice she had gathered through interviews with female legal professionals in India.
How might such diverse outputs be presented in order to showcase their quality (originality, significance, and rigour)? We can look to the two primarily non-textual submissions made to REF2021 for inspiration. Both were presented as, in REF terminology, ‘multi-component’ outputs—that is, comprised of several inter-connected pieces of work. In both cases, the quality of the non-textual research outputs was articulated through textual analysis, as well as a coherent (textual and visual) web-based narrative.
‘It stays with you’
‘It stays with you: Use of force by UN Peacekeepers in Haiti’ (led by Siobhan Wills, submitted by University of Ulster) was ‘a body of associated work demonstrating sustained research effort and a complex multi-layered process of creative investigation’. It emerged from two projects: one focusing on law enforcement by the Brazilian-led UN Stabilisation Mission in Haiti (MINUSTAH), the other on ‘militarised policing’ by Brazilian troops and police in Rio de Janeiro. It included two films co-produced with marginalised communities in Haiti and Brazil nine short videos produced by community members, and an article focused on the methodology, aims and outcomes of the project; all of which are gathered on a website with links to independent films reviews, festival screenings and prizes.
‘Display at your own risk’
‘Display at your own risk’ (led by Andrea Wallace, submitted by University of Lancaster), was ‘a research-led exhibition experiment’. It emerged from an investigation into the use by cultural heritage institutions of ‘digital surrogates’ (digital reproductions of material artefacts). It included gallery and open-source versions of an ‘exhibition intended for public use’, and a series of reflective essays; all gathered together in an engaging and well-presented website with accessible explanations of the project rationale and methodology.
Promoting the production of diverse outputs
The requirement that a submission be ‘representative’ might encourage the submission of diverse outputs. But it will not necessarily promote the production of diverse outputs.
One pathway to incentivising Units to proactively support the production of diverse practices and outputs is to highlight their potential to generate engagement and impact. Indeed, it is reasonable to expect ‘It stays with you’ and ‘Display at your own risk’ to feature in the Engagement and Impact dimension of REF2029 submissions.
Another pathway is the evolving People Culture and Environment (PCE) dimension of REF2029.
The PCE Pilot Exercise held in 2025 was ‘structured around five factors which enable positive research culture’, none of which focused directly on supporting diversity in research practices and outputs. However, the fifth factor, ‘Development’, begins with the words ‘Recognising and valuing the breadth of activities, practices and roles involved in research…’ (PCE pilot exercise guidance, para 17).
We shall see.