Research Excellence Framework 2018

EIS-ULA Memorandum of Comments on Proposals affecting the Research Excellence Framework

October 2018


The EIS is the largest education union in Scotland with approaching sixty thousand teacher members including over five thousand members in Further Education Colleges and around fifteen hundred members as academics and academic related staff within Higher Education Institutions across Scotland.

The EIS HE members form a Self-Governing Association called the ‘Educational Institute of Scotland University Lecturers’ Association’ (EIS-ULA) with its own Executive to deal with HE matters including determining HE policy for the Institute. The EIS is therefore unique amongst trade unions in having HE policy matters determined solely in Scotland.

The EIS-ULA welcomes the opportunity to comment on the latest proposals affecting the Research Excellence Framework (REF) and has the following comments to offer:

The EIS-ULA has in the past raised a number of concerns with the REF and predecessor exercises. It welcomed the recommendations of the Stern Review, with some important caveats, but cautioned that these recommendations would not go far enough in terms of making REF more cost-effective and less divisive.

The EIS-ULA has serious concerns, therefore, with the way in which the recommendations arising from the Stern Review have been implemented (or not, as the case may be) and with the impact of this implementation for academic staff. One overarching concern is that, despite the changes, REF will continue to be an expensive and divisive exercise that corrupts and distorts the practice it claims to measure, discourages innovation and difference, and impacts negatively on teaching.


REF is expensive and time-consuming

REF 2014 was estimated to have cost £250 million, which is 2.4% of the total spent on research by funding councils. In addition, academic staff spend countless hours in meetings, preparing REF submissions, writing “impact" case studies and conducting mini-REF exercises. This is time that could be spent in the laboratory, library or classroom - doing the things that tax-payers assume academics do. Based on experience so far of REF 2021, the EIS-ULA does not anticipate any significant cost savings on previous exercises.


REF distorts and corrupts the practice it claims to measure

The REF is intended as a means of quantifying an abstract quality, the excellence of research, but in a process of goal-displacement, it has become an end in-itself, often to the detriment of the ultimate ends those means were meant to serve.

The EIS-ULA believes that this happens in many ways. For example, REF has placed universities in a situation where they must consider the “impact” of their research. In 2014, the most common kinds of impact related to “informing government policy”, “parliamentary scrutiny”, “technology commercialisation”, and “print and media publishing”. Terry Eagleton has wryly observed, “Such impact is rather easier to gauge for aeronautical engineers than ancient historians. Pharmacists are likely to do better at this game than phenomenologists.”

The instrumental focus on a certain conception of economic and social impact in the short term is a danger to the quality of research in the long term. After all, much advancement has been made by blue skies, curiosity-driven research (such as the mathematical modelling central to contemporary computing). Critics have also argued that the emphasis on impact has turned HE institutions into the unofficial and unremunerated research and development arm of industry.

Further, REF encourages apparent novelty and clear cut, casual impacts, but offers no incentives to replicate studies. It is therefore leading to sloppy science in which claims made in scientific journals are not being retested or cross-checked by other researchers. Where cross-checking has occurred reproducibility is often lacking, leading to a crisis in scientific research which had become the subject of a parliamentary inquiry.

Game-playing is also widespread. Playing REF ranges from the relatively innocuous practice of generously attributing authorship to colleagues, so they can pad their CVs, to hyper-inflation in the number of outputs rated as “world leading” (four star rated) and “internationally excellent” (rated three star). The EIS-ULA also worry, that, despite Stern’s recommendations, Institutions will continue to cherry pick the staff and the impact case studies to be submitted in 2021. Though institutions are required to be transparent about the criteria for inclusion into a REF submission, experience suggests that many REF 2021 submissions will continue to over-exaggerate the research activity taking place within institutions and underestimate the number of staff who have responsibility for research but who are not currently research active (often because they are over-burdened with teaching).

But, worryingly, this game-playing goes even further. With REF targets to meet and careers on the line, research suggests “many academics admit to engaging in at least one questionable research practice in order to achieve publication.” Examples of this range from selectively reporting data to simply fabricating it. Evidence of this nature was also recently submitted to the Parliamentary inquiry into reproducibility and scientific integrity. REF therefore distorts and corrupts the practice it claims to measure.


The Relationship between REF and work-related stress

The EIS-ULA has concerns about the unrealistic and unreasonable expectations that are placed on many staff in HE. Though the 2021 REF requires Institutions to have just one output between 2014 and 2020 per research active member of staff, the EIS-ULA understands that many Institutions have placed additional requirements on staff in an attempt to boost their REF submissions. These additional targets are set, for example, through performance management reviews or through criteria for promotion or the ending of a probationary period. These additional REF requirements are likely to have a knock-on effect in terms of staff health and wellbeing, staff morale and ultimately, work-related stress.


REF is divisive

The EIS-ULA believes that the REF perpetuates inequalities between and within Institutions, further stratifying Institutional hierarchies and academic labour.

It appears that between Institutions, it rewards already successful institutions and helps ensure their continued dominance. Russell Group universities typically rank higher and are therefore more handsomely rewarded than post-92 institutions. A good result on the REF can sustain a department, a bad result can close one.

Within institutions, the REF can pit colleague against colleague, creating intense rivalries around who will be included within the REF submission, whose output is deemed to be of sufficient quality, and who will be allocated time to pursue research. Though Stern recognised this problem and recommended that Institutions should be required to enter all “active researchers”, therefore avoiding the divisive practice of Institutions cherry-picking from amongst those considered their best staff, this recommendation has not been implemented in full. Instead, Institutions will be required to be transparent about inclusion/exclusion criteria for REF.

The EIS-ULA is concerned that, despite this change, the REF continues to induce rivalry amongst staff and thus, presents problems in terms of collegiality. Mini-REF exercises conducted by Institutions, which in themselves are time-consuming and costly exercises, have played a particular role in this. Many staff who have been rating their colleague’s work as part of a mini-REF exercise are now finding that these ratings are being used against their colleagues in performance review processes, applications for promotion, the ending of a probationary periods, etc.. One possible consequence of this might be to undermine the ratings system itself, since staff conducting the ratings will exaggerate the quality of the outputs being rated because they may now feel a need to support their colleagues.


REF is not as rigorous as other review processes

Plans for a cheaper and less-time consuming metrics based research assessment exercise were abandoned by the Government prior to REA 2008 at the behest of leading academics, who argued that rigorous research evaluation required expert peer review. But, according to Derek Sayer, the REF “falls very far short of international peer reviewing standards in other academic contexts like publication, research funding or promotions.” 

Each of the thirty-six disciplinary sub-panels for the REF 2014 were, Sayer says, composed almost entirely of “in-house” academics from British Institutions. Though these panel members were eminent in their fields, they often lacked specialist expertise and the time required to assess all of the outputs falling under their remit.

One REF 2014 panelist told the Times Higher that: “I needed to peruse around 75 books and 360 articles or sections, and various different yields, including cross-referrals from different boards. I was not given any leave from my organization, and in spite of the fact that I spent the majority of spring and summer at my work area, I could frequently give just a hour or so to "perusing" books, and close to 20 minutes to articles or parts. A few associates had a significantly heavier appraisal load.”


REF impacts negatively on teaching

The EIS-ULA believes that the REF incentivises Institutions to prioritise research at the expense of teaching. It has resulted in a situation in which senior academics “are most likely to boost their institution’s status by taking leave of it, taking time off from teaching to further their research”; and has intensified the move in many Institutions to “teaching only” and casualised academic contracts. It is no surprise therefore that many of the best performing Institutions on the REF 2014 were inversely among the worst performing on the newly introduced Teaching Excellence Framework (TEF)1

Though some are hoping that the introduction of the TEF will help to redress this imbalance by "benchmarking” and “providing yardsticks” for teaching, there are reasons to be doubtful about this. The concern is that universities may play “the TEF game” and, as is perceived to happen with the REF, TEF will be used as a tool in performance management and in the disciplinary process of academic staff.