And the tapestry of lies, damned lies and statistics that is REF2014 keeps on unraveling.

I learn from this morning’s Twitterfeed that Dr Dan Lockton, of the Royal College of Art, and Professor Melissa Terras, Professor of Digital Humanities and Director of the UCL Centre for Digital Humanities, have received identical letters in response to requests under the Freedom of Information Act for HEFCE to disclose information held on them in connection with the REF.

Dr Lockton had asked to see “data held by HEFCE concerning myself or my work, as part of the REF, including any comments, assessments or other material.”

HEFCE responded that they did not hold the information he was seeking and referred him to a FAQ on the REF website:

“Can you provide the scores for my outputs that were submitted to the REF?

Individual outputs were assessed in order to produce the output sub-profiles for each submission. Once the sub-profiles were complete, the scores for individual outputs were no longer required and have been destroyed. In accordance with data protection principles, we no longer hold the scores for individual outputs as they constitute personal data, which should not be held for longer than required to fulfil their purpose.”

When it first emerged that RAE2008 was making the same use of such Orwellian memory holes, an (anonymous) panelist explained to Times Higher Education that “It is for our own good. The process could become an absolute nightmare if departmental heads or institutions chose to challenge the panels and this information was available.“[1]

HEFCE’s letter to Dr Lockton goes on to emphasize that:

“The purpose of the REF is to assess the quality of research and produce outcomes for each submission in the form of sub-profiles and an overall quality profile. These outcomes are then used to inform funding, provide accountability for public investment and provide benchmarking information. The purpose of the REF is not to provide a fine-grained assessment of each individual’s contribution to a submission and the process is not designed to deliver this.”

Yes, but. At the risk of appearing obtuse, I would have thought that when 65% of the overall quality profile rests on REF subpanels’ assessment of the quality of individuals’ outputs, we would expect “fine-grained assessment” of those outputs. Is this not why we have this cumbersome, time-consuming, expensive process of panel evaluation—as distinct, for instance, from using metrics—to begin with?

If it’s not fine-grained assessment, what sort of assessment is it?  And how can we trust it to provide a reliable basis for funding decisions, accountability for public investment, or benchmarking?

In the immortal words of Amy Winehouse, what kind of fuckery is this?

[1] Zoe Corbyn, ‘Panels ordered to shred all RAE records’. Times Higher Education, 17 April 2008.