Though some UK universities are playing by the rules in preparing their submissions for REF 2014 and treating their academic staff with respect and decency, Lancaster seems not to be alone in employing procedures to select staff for its REF submission that satisfy neither HEFCE’s own Guidelines for the REF nor the established norms of academic peer review.

This morning I received an email from Warwick University UCU, informing me of a recent survey of Warwick staff who were being excluded from the REF, and requesting that I publicize their document on my blog.  I am happy to do so.

 

Warwick survey shows REF rules being bypassed and selection guidelines ignored

Much good research excluded; Interdisciplinary research sidelined; Academics left in limbo

Warwick UCU has conducted a survey of all its members to ask if they had received a letter from the Deputy Vice-Chancellor informing them that they were to be excluded from the REF. Forty-four members replied, the largest response we have received to any of our surveys. (We also received responses from some members who had recently been moved to teaching-only contracts who are no longer eligible.)

We wanted to know the grounds on which they were excluded, the process by which the decision had been made and what the implications would be for them and for research within the university going forward. Above all we wanted to see if the guidelines and principles of the national Research Excellence Framework and Warwick’s own Code of Practice are in fact being upheld.

Main findings in brief:

• The majority of respondents are excluded on ‘quality’ grounds, meaning they have enough submissible research but the university judged it to be below some threshold (e.g. for some individuals/departments an ‘average’ of 3*, and for others 3.5*)

• Selection criteria are not transparent, are applied inconsistently and with little regard to inclusivity – in contravention of the stated REF principles.

• A small number of members have been granted a right of appeal on substantive, academic grounds (despite the university having said that it would only hear appeals within the context of Equality Legislation) and been reinstated.

• Research is often excluded on the subjective judgement of heads of department without having been independently appraised by experts external to the university, and in some cases, externals were asked to ‘confirm’ HoDs’ assessments, not to read and assess the work independently.

• Some research is excluded purely on ‘strategic grounds’.

• Interdisciplinary research is being excluded without proper appraisal: it seems to be routinely described as ‘below the standard required in terms of quality’.

• Some academics complain that the university would rather exclude them than ask for their research outputs to be cross-referred to a different REF panel.

• Many members are unclear as to the consequences of exclusion for their careers. They are unsure if they are in good standing or not.

THE FULL WARWICK UCU DOCUMENT CAN BE FOUND HERE.

I today received an email from Trevor McMillan, Pro Vice-Chancellor (Research) at Lancaster University, asking that I “address two very serious issues” connected with my previous posts about Lancaster’s selection of History staff for the 2014 REF on this blog.  The letter was not marked confidential.

Professor McMillan is concerned, first, that the identity of History’s Critical Friend “has been easily deducible from the information I have posted online.” He asks that I “modify the text online to remove this possibility and also write to Professor ________ to apologise for what has been posted.”

As Professor McMillan very well knows, the “removal of the relevant text from my blog” for which he asks will seriously weaken the presentation of the procedural issues at stake in my appeal, to the undoubted advantage of the University in what has become an increasingly public debate.

Two key arguments in my case are (1) that the professional experience that led the History Department to invite Professor ________ to become its Critical Friend did not qualify him for the role he came subsequently to occupy in Lancaster History’s REF process; specifically, (2) that Professor ________ ‘s particular field of academic expertise did not qualify him to make judgments of the “originality, rigor and significance” of many colleagues’ outputs within the Department, including my own.  Since these judgments played a key part in the University’s selection of staff for submission in the REF, Lancaster’s procedures did not meet HEFCE’s requirements.

Professor McMillan demands I now remove from my blog (1) my description of Professor ________ ‘s area of expertise together with (2) any reference to the professional experience that led us to invite him to become our Critical Friend in the first place, in the interests of maintaining his anonymity.  My case will thereby lose much of its evidential support.  I am permitted to make generic arguments, but not to publish the facts on which they rest.

This is an excellent example of how what Paolo Palladino, in his original Open Letter, referred to as the “culture of secrecy” surrounding the REF at Lancaster works to suppress free discussion and dissent.

I shall remove “the relevant text” from my blog, but only because I shall lay myself to disciplinary action for disregarding the direct instructions of a superior if I do not.  I am also happy to write to Professor ________ explaining my actions, but not to apologize for them.  To apologize would constitute an admission of a liability that I do not accept.

The second issue that concerns Professor McMillan is this:

Your implied comparison online of compliance with the REF process and attitudes to the Holocaust has caused some great distress among a number of your colleagues at Lancaster.  While we clearly support academic freedom for our staff I hope that you will see that this cannot go so far as real upset caused to colleagues and therefore this warrants the removal of the relevant text from your blog.

For those interested, the full text of the post concerned can be found here.

I did not “compare the REF to the Holocaust.”  The specific problem I raised was explaining how individuals involved in administering the REF are prepared to engage in actions that may damage the reputations and careers of colleagues, even though they know that the processes of evaluation involved are far less rigorous than those normally used in academia.  This might be seen as an instance of a more general problem of how ordinary decent folks sometimes do extraordinary and indecent things when required or empowered to do so by the bureaucratic organizations for whom they work.

I cited a number of classic studies (Weber, Foucault, Havel) in the sociological literature that variously address this problem.  I could have mentioned others: the Milgram experiments, for instance, in which randomly-selected subjects showed themselves willing to administer powerful doses of electric shock to fellow human beings when reassured by people in white coats that it was OK to do so, or Adorno and Horkheimer’s Authoritarian Personality.  Bauman’s Modernity and the Holocaust is a pivotal work in this tradition, and is seen as a seminal contribution to the field precisely because of the way it brings out the connections between the extreme evil of the Holocaust and the everyday routines and expectations of modern bureaucracies—like, in this case, universities.

Since I doubt Professor McMillan—or, possibly, some of those who claim to have been caused “great distress” by my reference to Modernity and the Holocaust—will find the time, as the REF cull at Lancaster nears its conclusion, to read one of the great classics of contemporary sociology, let me give you Wikipedia’s summary of Bauman’s argument:

Drawing upon Hannah Arendt and Theodor Adorno’s books on totalitarianism and the Enlightenment, Bauman developed the argument that the Holocaust should not simply be considered to be an event in Jewish history, nor a regression to pre-modern barbarism. Rather, he argued, the Holocaust should be seen as deeply connected to modernity and its order-making efforts.  Procedural rationality, the division of labour into smaller and smaller tasks, the taxonomic categorisation of different species, and the tendency to view rule-following as morally good all played their role in the Holocaust coming to pass.  And he argued that for this reason modern societies have not fully taken on board the lessons of the Holocaust; it is generally viewed—to use Bauman’s metaphor—like a picture hanging on a wall, offering few lessons.

I am not going to be intimidated out of applying the analysis of bureaucracy and conformity developed in the discipline in which I was trained, sociology, to my own workplace, or to the REF more generally.  I do not regard the fact that people claim to have been “upset” by my reference to Bauman as sufficient reason to remove the reference from the blog, as Professor McMillan demands.  It is a wholly unwarranted intrusion into the academic freedom to which all Lancaster University employees should be entitled.  I will therefore not remove or amend this post.

Let me finally say that several of my colleagues at Lancaster have also been caused “great distress” by their exclusion from University’s submission to the 2014 REF.  They have ample reason for feeling “real upset.”  As I put it in an earlier post on this blog,

I find it poignant that so cavalier an attitude toward evaluating the research of colleagues should be adopted in a university that requires external examiners for PhDs to be “an experienced member of another university qualified … to assess the thesis within its own field” and—unlike in North America—also requires all undergraduate work to be both second-marked internally and open to inspection by an external examiner before it can count toward a degree.   Why are those whose very livelihood depends on their research—and its reputation for quality—not given at least equivalent consideration as the students they teach?

I understand why the University might prefer discussion to focus on my alleged breaches of confidentiality and upsetting of colleagues rather than its own lamentably inadequate REF procedures.  But does Professor McMillan really want to go on record as saying that “academic freedom for our staff … cannot go so far as real upset caused to colleagues,” while at the same time maintaining that the freedom of administrators to evaluate people’s research in the way Lancaster has in History–with all its demonstrably distressing effects for individuals who are excluded from the REF as a consequence–should be above public debate or criticism?

A reader of my blogpost “Update from Wonderland: the Lancaster REF farce goes on” commented: “Brilliant but when will university historians mount a sustained campaign against the REF?”  Many who have commented on this and my earlier post “Kafkarna continues: REF gloves off at Lancaster University” on Twitter and other social media have voiced the same concern.  Why has there been so little resistance, at Lancaster or elsewhere, to an enterprise that makes a mockery of the values universities are meant to stand for?  Several have gone so far as to liken British academics’ collective acceptance of the REF exercise to Stockholm syndrome.    

I am not especially puzzled at colleagues’ reluctance to publicly resist a process many of them despise but feel they can do little about. Fear is a powerful disincentive to action, and British academics do not enjoy the protections of tenure (which was legislated away under Margaret Thatcher).  What does bother me, however, is the apparent willingness of many of those involved in the administration of the REF to participate in and vigorously defend an exercise they know will damage the reputations of colleagues, even if they are fully aware that the evaluative mechanisms used in their internal processes for selection of staff for entry into the REF are far less rigorous than those routinely used in academia (in peer review for journals, research grant funding competitions, or promotion appraisals, for instance).  

How to explain this willingness to throw one’s colleagues to the wolves, in some cases obviously relishing the sense of self-importance that comes with the power to do it?  Max Weber’s classic studies of how bureaucracy works “without regard for persons” might provide one starting-point; Zygmunt Bauman’s Modernity and the Holocaust another.  We might invoke Michel Foucault’s analyses of power and subjectivity.  But the text that goes to the heart of the matter, for me, is Václav Havel’s parable of the compliant Prague greengrocer.

*   *   *

The manager of a fruit-and-vegetable shop places in his window, among the onions and carrots, the slogan: “Workers of the world, unite!” Why does he do it? What is he trying to communicate to the world? Is he genuinely enthusiastic about the idea of unity among the workers of the world? Is his enthusiasm so great that he feels an irrepressible impulse to acquaint the public with his ideals? Has he really given more than a moment’s thought to how such a unification might occur and what it would mean?

I think it can safely be assumed that the overwhelming majority of shopkeepers never think about the slogans they put in their windows, nor do they use them to express their real opinions. That poster was delivered to our greengrocer from the enterprise headquarters along with the onions and carrots. He put them all into the window simply because it has been done that way for years, because everyone does it, and because that is the way it has to be. If he were to refuse, there could be trouble. He could be reproached for not having the proper decoration in his window; someone might even accuse him of disloyalty. He does it because these things must be done if one is to get along in life. It is one of the thousands of details that guarantee him a relatively tranquil life “in harmony with society,” as they say.

Obviously the greengrocer is indifferent to the semantic content of the slogan on exhibit; he does not put the slogan in his window from any personal desire to acquaint the public with the ideal it expresses. This, of course, does not mean that his action has no motive or significance at all, or that the slogan communicates nothing to anyone. The slogan is really a sign, and as such it contains a subliminal but very definite message. Verbally, it might be expressed this way: “I, the greengrocer XY, live here and I know what I must do. I behave in the manner expected of me. I can be depended upon and am beyond reproach. I am obedient and therefore I have the right to be left in peace.” This message, of course, has an addressee: it is directed above, to the greengrocer’s superior, and at the same time it is a shield that protects the greengrocer from potential informers. The slogan’s real meaning, therefore, is rooted firmly in the greengrocer’s existence. It reflects his vital interests. But what are those vital interests?

Let us take note: if the greengrocer had been instructed to display the slogan “I am afraid and therefore unquestioningly obedient,’ he would not be nearly as indifferent to its semantics, even though the statement would reflect the truth. The greengrocer would be embarrassed and ashamed to put such an unequivocal statement of his own degradation in the shop window, and quite naturally so, for he is a human being and thus has a sense of his own dignity. To overcome this complication, his expression of loyalty must take the form of a sign which, at least on its textual surface, indicates a level of disinterested conviction. It must allow the greengrocer to say, “What’s wrong with the workers of the world uniting?” Thus the sign helps the greengrocer to conceal from himself the low foundations of his obedience, at the same time concealing the low foundations of power. It hides them behind the facade of something high.  […]

[T]he real meaning of the greengrocer’s slogan has nothing to do with what the text of the slogan actually says. Even so, this real meaning is quite clear and generally comprehensible because the code is so familiar: the greengrocer declares his loyalty (and he can do no other if his declaration is to be accepted) in the only way the regime is capable of hearing; that is, by accepting the prescribed ritual, by accepting appearances as reality, by accepting the given rules of the game. In doing so, however, he has himself become a player in the game, thus making it possible for the game to go on, for it to exist in the first place.

Václav Havel, “The Power of the Powerless” (1978), translated by Paul Wilson

I’m sick of blogging about the self-important, pompous, and utterly provincial British REF (“Research Excellence Framework’).  Enough miserablism  (André Breton), at least for today.

Instead, I would like to celebrate the October publication by Chicago University Press of Allen C. Shelton’s Where the North Sea Touches Alabama, the follow-up to his widely praised Dreamworlds of Alabama (Minnesota UP, 2007).

I have known Allen since he first invited me, over ten years ago now, to participate in the remarkable festival of arts and ideas he organized annually on a shoestring budget in Buffalo, NY under the Benjaminian title Ethnographic Dreamworlds.  Some of the richness of those meetings, which brought together intellectuals across disciplines, performers, and visual artists, can be gleaned from the soft arcades website that survives as a ghostly record of an extraordinarily vibrant event.

It was through Allen and Ethnographic Dreamworlds that I met, among others, Kathleen Stewart, Patricia Clough, Danielle Egan, and Susan Lepselter—scholars doing courageous and pioneering work that make the standard disciplinary boundaries in the humanities and social sciences seem no more than quaint and rather incomprehensible antiques.  Allen was kind enough to visit Lancaster University last year as a participant in a cross-disciplinary writing workshop for PhD students, on whom he left a lasting impression.

This is what others are saying about Where the North Sea Touches Alabama:

Kathleen C. Stewart, author of Ordinary Affects

“This is a beautiful and brilliant book. . . . The lives of Allen Shelton, Patrik Keim, Walter Benjamin, and many others intersect in these pages, rubbing up against each other, drawing on each other to evoke layers on layers of worlds in which objects, color, and texture are everything. Shelton’s writing is masterful.”

Donna Haraway, author of A Cyborg Manifesto

“Allen C. Shelton is really special. From the layering and subtlety of his writing to his sense of geography, intimacy, and sensuous detail, I don’t know anyone who writes quite like him. These interwoven narratives of the dead and the living form a boundary-crossing work of worlding, a productive new type of critical engagement; Where the North Sea Touches Alabama is not just a remarkable book, but a fresh genre of writing.”

Howard S. Becker, author of Art Worlds

“Allen C. Shelton is a provocative writer whose prose grapples with a lot of ideas we don’t usually allow ourselves to think about. Readers will have to think hard, but their efforts will pay off in new knowledge and insight: I felt that I knew a whole lot more after reading his book than I did before and I don’t often feel that way, nor feel that way so strongly.”

Jonathan Fullmer | Booklist

“Dense, wildly digressive, and divided into topical microchapters that cite more than 100 endnotes sometimes only loosely connected to the text, Shelton’s singular blend of art-, lit-, and pop-infused intellectualism may not draw a wide readership, but those who enter will find an invigorating analysis of death, art, friendship, and self-discovery.”

Luis Jaramillo | The Coffin Factory

“The sometimes abrupt shifts in subject matter make this a book that has to be read slowly to take in Shelton’s arguments. Fortunately this close reading is rewarded, especially in the moments when Shelton moves from more analytical passages to personal reflections, synthesizing the theories he’s discussing. . . . What makes this book so strangely wonderful is how Shelton moves from the abstract to the personal.”

Check it out here.

Image

Update, October 16.  I have censored this post at the insistence of Professor Trevor McMillan, Pro Vice-Chancellor (research) at Lancaster University.  I indicate passages that have been altered or removed by angle brackets <>.

Since my last post on Lancaster University’s selection of History staff for inclusion in the 2014 REF, another case has come to light that is no less absurd–and troubling–than Professor Palladino’s.  After reading and evaluation of his four research outputs by the Department’s external assessor, another of my colleagues had an aggregate score of 2.75 (3 x 3 and 1 x 2).  He was nonetheless excluded from the REF as a result of one of his articles being re-read by a second external reader and given a lower grade of 2*.  The Dean of the Faculty of Arts and Social Sciences at that point refused to commission a further specialist review.

What is particularly troubling is that the article in question was published in Past and Present, which is regarded by most UK historians as one of the top historical journals, if not the leading historical journal in the English language.  My colleague had also been urged by the editor of a major specialist journal in his field to withdraw the article from Past and Present (who had not at that point definitively accepted it), with the promise that it would be published quickly as a lead article in this second journal, where it would form the centerpiece of a themed issue.  Once again the subjective judgments of University-appointed assessors have trumped the peer evaluations (in this case three) commissioned by distinguished professional journals.

My colleague is now appealing the decision, which is why I have kept his name anonymous (though I am posting this information with his permission).  But appeals are permitted only on procedural grounds—one cannot appeal the judgment of quality as such.  As I pointed out in an earlier post, the truly Kafkan reasoning behind this is that (to quote Lancaster University’s Code of Conduct for the 2014 REF): “The judgements are subjective, based on factual information. Hence, disagreement with the decision alone would not be appropriate grounds for an appeal.”

In the surreal spirit of the enterprise, but with the intent of questioning Lancaster University’s procedures for selecting History staff for the 2014 REF, I decided to submit a formal appeal of my own.  The appeal is against my being selected as part of the History Unit of Assessment (UoA) in the 2014 REF.

Here is the full text of my appeal.  The only change from the version sent to the Director of Human Resources at Lancaster is that I have removed names of individuals, except where I am quoting from already published material.

 

 

 REF 2014: APPEAL

1.  I wish to appeal against Lancaster University’s decision to include me as part of the History UoA in its 2014 REF submission.

2.  This is not an appeal against the judgments of quality of my four outputs, on whose basis this decision was taken.  It therefore does not fall foul of the requirement that “disagreement with the decision alone would not be appropriate grounds for an appeal” (LANCASTER UNIVERSITY REF 2014: Code of Practice V5 27 September 2013, p. 5).[1]

3.  This appeal is based exclusively upon “concerns about process,” one of the two grounds permitted by LU Code of Practice (p. 5).   These concerns are:

(i) that the procedures used to select staff for inclusion in the Lancaster University History UoA for the 2014 REF do not satisfy the criteria set out by HEFCE in its document Assessment Framework and Guidance on Submissions[2] in respect of either transparency or accountability;

(ii) that the procedures used for assessing my own work for inclusion in the 2014 REF were incompatible with LU Code of Conduct‘s objective of ensuring that “The primary factor [in selection] will be the quality of the research outputs as defined by the published REF criteria contained in the Guidance on Submission and Panel Criteria documents” (p. 2); and

(iii) that the procedures used in assessing the quality of my own work for inclusion in the 2014 REF were de facto discriminatory toward several of my colleagues in History, who are not being returned in the REF, breaching HEFCE requirements of equality, fairness, and consistency.

I outline these concerns more fully under (4), (5) and (6) below, respectively.

4.  HEFCE’s Assessment Framework and Guidance makes clear that while “It is a requirement of the REF that each submitting institution establishes a code of practice on the selection of staff for REF submissions … It is the responsibility of HEIs to ensure that their codes of practice, and the manner in which they participate in the REF, are lawful” (39).

I am aware that LU Code of Practice was submitted to and approved by HEFCE.  I believe, however, that the actual implementation of this code, at least as regards the History UoA, has not conformed to HEFCE’s requirements.

Specifically:

204.  a. Transparency: All processes for the selection of staff for inclusion in REF submissions should be transparent. Codes of practice should be drawn up and made available in an easily accessible format and publicised to all academic staff across the institution, including on the staff intranet, and drawn to the attention of those absent from work. We would expect there to be a programme of communication activity to disseminate the code of practice and explain the processes related to selection of staff for submission. This should be documented in the code (Assessment Framework and Guidance, p. 39).

While the LU Code of Practice was made available on the University intranet, that document contained only the most general account of the “processes for the selection of staff for inclusion in the REF.”  It is my contention that the processes actually employed in the case of the History UoA were anything but transparent.

Specifically, History staff knew that their four outputs would be read by the Department’s external assessor Professor ___________ , and that his evaluation would in some (unspecified) way feed into the University’s final decisions on inclusion and exclusion.  Individuals were also informed that their work might be sent out for further, specialist readings.

But History staff were not told, at least until the final decision on inclusion was communicated to them by the HoD in September 2013:

(i) the scores Professor _______ had given their individual outputs;[3]

(ii) the circumstances that would trigger a second reading of their work, or a “re-review” by an independent specialist of an item already read and scored by Professor _______ ;

(iii) the basis on which external assessors (other than Professor _______ , <removed>) were chosen, or who was responsible for selecting them;

(iv) the overall aggregate score needed to qualify for inclusion in the University’s submission to the 2014 REF.

When I asked my HoD to tell me where information on the specific evaluative procedures used with regard to selection of staff for the History UoA had been published on the University website, I was told: “As far as I’m aware, the evaluative procedures for arriving at decisions have not been published on the website, and in any case they will differ from Faculty to Faculty and department to department” (email from [HoD] , 3 October 2013).

In short, several key elements of the evaluative procedures that determined whether or not individuals were included in the 2014 REF submission were not transparent, and were never clearly communicated to the staff concerned.

This in turn makes it difficult to appeal the University’s decisions: Lancaster has constructed a Kafkan scenario in which grounds for appeal include “concerns about process (including if it is felt that procedure has not been followed),” but one cannot know whether or not procedures have been followed if the procedures concerned have not been clearly communicated in advance.

204. c. Accountability: Responsibilities should be clearly defined, and individuals and bodies that are involved in selecting staff for REF submissions should be identified by name or role … Operating criteria and terms of reference for individuals, committees, advisory groups and any other bodies concerned with staff selection should be made readily available to all individuals and groups concerned (Assessment Framework and Guidance, p. 39, emphasis added).

The LU Code of Practice says that “selection decisions regarding the University submission to the REF will lie with the Vice-Chancellor on the advice of the REFSG,” whose membership the document details.   While I accept that this may be the formal legal position, it is not a complete or accurate description of what has actually happened.  I dispute that these are the only individuals or bodies at Lancaster University “involved in selecting staff for REF submissions.”  Others who have been involved, at various stages of the process, include Heads of Department, Departmental Research Directors, Associate Deans for Research, Faculty Deans, and external assessors.  Though none of these may be formally responsible for the final decisions on inclusion or exclusion, their inputs have contributed to those decisions.  In the case of external assessors who read and scored individual outputs, that contribution may often have been decisive.

When I was HoD for History (2009-2012) I sat on what were in effect ad hoc committees involved in the early stages of making recommendations—though not final decisions—for selection of staff for the 2014 REF for both History and the Department of European Languages and Cultures.  In the case of History, the relevant meetings involved, at various points, the Dean of FASS _______ , the Associate Dean Professor _______ , the History Department Research Director _______ , and the external assessor Professor _______ .  In the case of DELC, I was personally asked by the Dean, Professor _______ , to assess DELC staff members’ outputs as an “internal/external” reader.  The context was the need for FASS to take a strategic decision or whether or not DELC should be entered in the 2014 REF as an independent UoA.  So far as I recall, also present at that meeting were Dean _______ , [the FASS AD for Research], the DELC HoD _______ , the DELC Research Director _______ , and DELC’s external assessor, whose name I no longer recall.

I respectfully submit that (i) these were “bodies involved in selecting staff for REF submissions” in the sense intended by HEFCE paragraph 204c above, and (ii) their operating criteria and terms of reference were not “made readily available to all individuals and groups concerned.”

Specifically, these advisory groupings, whose recommendations will have formed the initial Department- and Faculty-level bases for the decisions on individual inclusion or exclusion by the central REF Steering Group and V-C, did not satisfy HEFCE’s requirements that:

209. Where a committee or committees have designated REF responsibilities – whether it is at departmental, faculty, UOA or central level – these should be detailed in the code of practice, including, for each committee:

• how the committee has been formed

• its membership

• the definition of its position within the advisory or decision-making process  …

210. The following details should be provided about its mode of operation:

• the criteria that it will use in carrying out its functions …

(Assessment Framework and Guidance, p. 40, emphasis added).

To my knowledge such bodies have no formal standing in the process at all, though their recommendations may well turn out to be decisive for individual members of staff.  They are likely to have been the most important decision-making bodies in all cases where department-level initial readings of outputs yielded a score sufficient for inclusion, a category into which my own case falls.  These bodies are not detailed anywhere in LU Code of Practice (as the Assessment Framework and Guidance para 209 requires), nor are their existence, composition, terms of reference, or operating criteria publicized on the intranet.

In short, the LU Code of Practice that was endorsed by HEFCE does not sufficiently describe the processes of staff selection for REF 2014 that have actually been used—at least with regard to the History UoA—at Lancaster University, while the processes that have actually been used to select individuals do not satisfy HEFCE’s stated criteria for either transparency or accountability.

5.  HEFCE’s Assessment Framework and Guidance is clear that: “The purpose of the guidance in Part 4 is to support institutions in promoting equality and diversity when preparing submissions to the REF, through drawing up and implementing a code of practice on the fair and transparent selection of staff. This will aid institutions in including all their eligible staff in submissions who are conducting excellent research, as well as promoting equality, complying with legislation and avoiding discrimination” (para. 187, p. 34, emphasis added).

I draw two inferences from this:

(i) HEFCE very clearly does not intend institutions to exclude any “eligible staff … who are conducting excellent research” from their submissions;

(ii) institutions’ procedures for deciding which staff are included in or excluded from submission in REF 2014 must be capable of judging whether outputs do or do not in fact constitute “excellent research,” as well as of satisfying HEFCE’s concerns regarding the separate issue of “promoting equality, complying with legislation and avoiding discrimination.”

Lancaster University likewise claims that: “”The primary factor [in selection of staff for the 2014 REF] will be the quality of the research outputs as defined by the published REF criteria contained in the Guidance on Submission and Panel Criteria documents” (LU Code of Conduct, p. 2).  Again, the logical expectation would be that its procedures for judging quality of outputs are fit for purpose.

I believe that the procedures applied within History at Lancaster University are manifestly not capable of judging the excellence or otherwise of my four outputs, and that my inclusion in the REF, without further specialist review of my work, is therefore contrary both to HEFCE Assessment Framework and Guidance and to the LU Code of Conduct as quoted above.

“Excellent research” has a very precise meaning within the discourse of REF 2014.  The word “excellent” is used only in connection with items ranked as 4* (“Quality that is world-leading in terms of originality, significance and rigour”) and 3* (“Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence”) (Assessment Framework and Guidance, Annex A, p. 43).  The specific criteria used to judge a work’s degree of excellence are originality, rigor, and significance.

On inquiring of my Head of Department whether my four outputs were read and evaluated by anybody other than the History UoA’s external assessor Professor _______ , I was informed that: “my understanding is that in your case all four outputs were evaluated only by [Professor _______] – I’m not aware of any being sent out for additional review” (email from [HoD], 3 October 2013).

I have the utmost respect for Professor _______ , who in my experience as HoD did his job as a Critical Friend with the utmost integrity and conscientiousness.  He is, however, a historian whose interests and expertise are very far from my own.  His website defines his research interests as <removed in order to preserve the reviewer’s anonymity>”[4]  I do not believe that he is remotely qualified to judge the excellence or otherwise of my research.  I write about none of the things in which he claims a research interest and has a record of publication.  I research twentieth-century history, not <removed>; Czech history, not <removed>; modernism and surrealism, with a particular focus on architecture and the visual arts, not <removed>.  My primary sources are all in Czech and French, languages Professor _______ does not read—as is much of the secondary discussion in the field.  I do not see how Professor _______ can be expected to evaluate the originality, rigor, and significance of my research if he has little or no knowledge of the fields to which it contributes.  Frankly, this is absurd!

Professor _______ is not qualified to evaluate the quality of my contributions to a field of historical inquiry so distant from his own, let alone make the necessarily fine distinctions that separate a 3* from a 2* output (the latter being defined as “Quality that is recognised internationally in terms of originality, significance and rigour,” Assessment Framework and Guidance, Annex A, p. 43).  He would not be asked to evaluate these items as an expert reviewer for a journal or publishing house, or in connection with tenure or promotion proceedings, not because he is not an eminent historian, but because his expertise lies in a very different area of the discipline.  His appointment as the single external assessor for the entire History UoA at Lancaster University is in contradiction with all normal professional norms of peer assessment and review.  Since Professor _______ is the only one to have read my work for REF purposes, I therefore submit that the procedures used by Lancaster University to evaluate that work, on the basis of which I am being submitting in REF 2014, are not consistent with HEFCE’s stated objective of ensuring that universities include “all their eligible staff in submissions who are conducting excellent research.”

After I learned I was to be included in the REF submission, I wrote my HoD with the following request: “In the interests of upholding the integrity of this process of quality evaluation, as well as of ensuring equitable treatment between colleagues, I would like formally to request that my outputs be sent out for further appraisal by subject-matter experts before I am submitted in the 2014 REF” (email to [HoD], 30 September 2013).  I received the following response: “I can see why you are asking for this, but I am afraid that it is no longer in my power to comply with your request for specialist reading of your work. The REF process is now at a stage where appeals on procedural grounds can be made, but not on any other grounds, and I have therefore been advised to decline your request” (email from [HoD], 2 October 2013).  Further communication with my HoD established that “Even before I opened your email of 30th Sept I had already been received messages from the Dean, AD Research and Pro VC for Research advising me that there was no longer any option for specialist reading of any work” (email of 3 October 2013).  That is to say, Lancaster University has closed the possibility of further specialist readings of work even before the deadline for making appeals (October 8) has expired.

It is worth noting, finally, that Lancaster University senior managers have admitted the inadequacy of the procedures that were eventually used to evaluate my work for inclusion in the REF.  Writing at an earlier stage of the process, the Associate Dean (Research) for the Faculty of Science and Technology accepted that “some weaknesses in the mock REF exercise are apparent, for example in many cases there was only one external reviewer per department, no doubt with expert knowledge but not in all the relevant areas” (Michael Koch, “In preparation for REF2014 – Mock REF and Units of Assessment,” SciTech Bulletin #125).[5]  This was exactly my case, and forms part of the basis of this appeal.  HEFCE expressed the hope that: “Institutions that conduct mock REF exercises might consider using them as an opportunity to apply their draft code and refine it further” (Assessment Framework and Guidance, para 203, p. 39).  Evidently in this case Lancaster, or at least FASS, chose to ignore that very sensible advice.

6.  HEFCE’s Assessment Framework and Guidance enshrines both a repeatedly stated commitment to “equality and diversity” and “fairness” (see e.g. para 187), and a specific requirement of:

204.  b. Consistency: It is essential that policy in respect of staff selection is consistent across the institution and that the code of practice is implemented uniformly (Assessment Framework and Guidance, p. 39).

I do not know how consistent staff selection procedures within History are with those used for other UoA’s, because as stated above, the procedures employed at individual UoA level have never been made public.  Within the History UoA, however, staff members have not been treated equally.  While some, like myself, have been included in the 2014 REF submission on the basis of one external reading of all outputs (i.e., Professor _______’s)—even when that reading has not been by an expert in the relevant field—others have had some or all items of their work additionally read by further external assessors.   Since these assessors are claimed to be field specialists, one might expect them to set the bar higher in terms of their ability to recognize originality, rigor, and significance.  In some cases this second reading has involved outputs that were originally given a grade by Professor _______ being downgraded, with (in at least two cases known to me) the staff member being excluded from the REF submission as a direct result.  I submit that this is inequitable insofar as some members of staff within the same UoA have been subjected to double or triple jeopardy and others not.

This might be justifiable, had clear criteria governing when outputs are sent out for further specialist evaluation been articulated and published in advance.  They were not.  From what I can glean from what I have been told by colleagues who have been excluded from the REF, the circumstances in which second readings have been sought include (but may not be limited to):

(i) where the external assessor, Professor _______ , himself expressed doubts as to his competence to evaluate an output in a field with which he is not familiar; and

(ii) where the aggregate score of all four outputs left the individual on a borderline in terms of the University’s (as yet unpublished) threshold for inclusion in the History UoA.

Since I fell into neither of these categories, assessment of my outputs was much less rigorous than that of several of my colleagues.  My outputs were read by only one external reviewer, as opposed to two or more, and that reviewer was not a subject matter specialist.

This is not consistent treatment of Lancaster University staff, nor is it by any reasonable standards fair to the colleagues in question.  Nor does it encourage representation of “diversity” of research within the Department, since the opinion of one individual, the external assessor, Professor _______ , plays a disproportionate role in determining who is or is not included in Lancaster University’s History UoA as submitted to the 2014 REF.

Derek Sayer

3 October 2013


[1] http://www.lancaster.ac.uk/depts/research/lancaster/REF2014.html, accessed 1 October 2013.  Subsequently cited as LU Code of Practice.

[2] Assessment framework and guidance on submissions (updated to include addendum published in January 2012), available at http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf, accessed 1 October 2013.  Subsequently cited as Assessment Framework and Guidance.

[3] As then Head of Department, responsible for communicating to staff where they stood in terms of likelihood of being submitted in the 2014 REF in January 2012, I was repeatedly told by the History Research Director (who was acting, I assume, on instructions from above) that these scores should be kept confidential and not communicated to the staff involved.

[4]

Two days ago an open letter from Professor Paolo Palladino, former Head of the History Department at Lancaster University, appeared on the university’s internal History Staff list serve, with copies sent to senior administrators in the Faculty of Arts and Social Sciences (FASS) and Central Administration.  I responded the next day with a post supporting Professor Palladino’s position, sent to the same recipients. With Paolo’s permission, I am reproducing both of our letters here.  We both believe that the issues raised by Lancaster’s selective culling of research-active staff from submission in the 2014 REF–a practice in which it is not alone among British universities–deserve the widest possible public debate.  We would therefore urge anyone who shares our concerns to share this post through social media.

Professor Palladino’s letter:
Dear all,
Over the next few days, a number of colleagues across the university are to be informed that they will not be returned in REF. I am one of these colleagues and I wish to challenge the culture of secrecy around our situation, which blocks any form of engaged, collective response to the way in which REF, as it is being managed at Lancaster University, impoverishes us all.
I regard the evaluation of research as perfectly legitimate and also think that the university’s management is at liberty to mobilise academic assets in a manner that maximises the returns of the university’s investments in staff and resources. I think none the less that the situation in which I find myself is an indictment of this department, this faculty and this university. All three have now assented to the notion that the author of the required four academic outputs, each published in a peer-reviewed, internationally recognised journal and thus meeting the expectation that any outputs returned should be ‘recognised internationally in terms of originality, significance and rigour’ (grade 2), does not merit inclusion in the evaluation of academic research undertaken by staff at Lancaster University. The problem is that the internal preparations for REF reward disciplinary orthodoxy by resting, for reasons of economy, upon the evaluations of a single reader per Unit of Assessment (UoA) and by expecting that each member of staff returned should fit within the narrative of one of the UoA submissions. Consequently, it is not possible to accommodate a researcher whose output is ‘recognised internationally in terms of originality, significance and rigour’, but whose work cuts across the domains of a number of UoAs and thus must in all likelihood fall short of the most common assumptions about what constitute the ‘highest standards of excellence’ (grade 4) for each and every UoA involved. Acceptance of this situation calls into question this university’s commitment to supporting interdisciplinary research, the one asset that has served most successfully to distinguish Lancaster University in an increasingly competitive global market for higher education.
I greatly regret this situation and I hope that the one third of our colleagues who, it seems, are not to be returned in REF will challenge an internal institutional judgement that calls into question both the rationale of our diverse appointments and the meaning of academic freedom at Lancaster University.
Best wishes,
Paolo Palladino
Professor of History and Theory
Department of History
Lancaster University
Lancaster LA1 4YT
My response:

Regarding Paolo Palladino’s open letter

Recently I published a lengthy blog post entitled “The Kafkan World of the British REF” (August 13, 2013), which concluded as follows:

“For whatever reasons, the funding councils for British universities have chosen to allocate that portion of their budget earmarked for research in ways that (at least in the humanities) systematically ignore the normal and well-established international benchmarks for judging the quality of research and publications [in particular, peer reviews and citations].  Instead, they have chosen to set up their own panopticon of handpicked disciplinary experts, whose eminence is not in doubt, but whose ability to provide informed assessments of the vast range of outputs submitted to them may well be questioned.

The vagaries of this approach have been exacerbated in the 2014 REF by putting in place a financial reward regime that incentivizes individual universities to exclude potential 2* and 1* outputs from submission altogether.  The resulting internal university REF procedures [for trying to identify and exclude these outputs in advance] are not only enormously wasteful of time and money that could otherwise be spent on research.  More importantly, they compound the elements of subjectivity and arbitrariness already inherent in the RAE/REF system, ensuring that evaluations of quality on whose basis individuals are [included or] excluded from the REF are often not made by subject-matter experts at all.   Research that crosses disciplinary boundaries or challenges norms will find itself especially vulnerable in this context, because it is seen as especially “risky.”

Whatever this charade is, it is not a framework for research excellence.  If anything, it is likely to encourage conventional, “safe” research, while actively penalizing risk-taking innovation—above all, where that innovation crosses the disciplinary boundaries entrenched in the REF Sub-panels.  The REF is not even any longer a research assessment exercise, in any meaningful definition of that term, because so much of the assessment is now done within individual universities, in anything but rigorous ways, before it enters the REF proper at all.”

Interested colleagues may wish to inspect the evidence and reasoning behind this conclusion here:

https://coastsofbohemia.com/2013/08/13/the-kafkan-world-of-the-british-research-excellence-framework/

The exclusion of the work of a senior and well published colleague, engaged in serious interdisciplinary work, from Lancaster University’s 2014 REF submission despite having (more than) 4 outputs in respected international peer-reviewed journals during the census period confirms all my misgivings.

Professor Palladino will no doubt be advised of his right to appeal.  I would advise him not to bother, for reasons I also spelled out in my post:

“Though there is a right of appeal for individuals … Lancaster’s Code of Practice is crystal clear that: “The decision on the inclusion of staff to the REF is a strategic and qualitative process in which judgements are made about the quality of research of individual members of staff.  The judgements are subjective, based on factual information. Hence, disagreement with the decision alone would not be appropriate grounds for an appeal” (my emphasis).

This is the ultimate Kafkan twist.  The subjectivity of the process of evaluation is admitted, but only as a reason for denying any right of appeal against its decisions on substantive grounds.”

I do not object to the evaluation of research, but believe the way it has been done within Lancaster University is anything but rigorous, fair or objective.  I therefore share all Professor Palladino’s concerns, and applaud his courage in raising them through this open letter to colleagues.  As a senior professor of this university, a former Head of History at Lancaster (and of Sociology at Alberta), a former Canada Research Chair and recipient of research funding from SSRC, AHRC, the British Academy, the Royal Society, SSHRC (Canada), and the Canadian Foundation for Innovation, not to mention a “research active” scholar of more than 35 years standing (I got my first permanent university position in 1978), I feel morally obliged to write publicly and emphatically in his support.  I would urge others who care about the future of British universities in general and Lancaster in particular to do the same.

Derek Sayer

September 27, 2013

Update, October 16.  I have censored this post at the insistence of Professor Trevor McMillan, Pro Vice-Chancellor (research) at Lancaster University.  I indicate passages that have been altered or removed by angle brackets <>.  

Times Higher Education recently reported that at Leicester University, “The position of all staff eligible for the [2014] REF but not submitted will be reviewed. Those who cannot demonstrate extenuating circumstances will have two options. Where a vacancy exists and they can demonstrate ‘teaching excellence’, they will be able to transfer to a teaching-only contract. Alternatively, they may continue on a teaching and research contract subject to meeting ‘realistic’ performance targets within a year.  If they fail to do so, ‘the normal consequence would be dismissal on the ground of unsatisfactory performance’” (08/08/2013, see full article here).   We live in interesting times.

1.   How it works—the context and the stakes

Having worked in North America from 1986 to 2006, when I took up a Chair in Cultural History at Lancaster University, I have spent most of my academic career in blissful ignorance of the peculiarly British institution that used to be called the RAE (Research Assessment Exercise) but has recently—and in good Orwellian fashion—been renamed the REF (Research Excellence Framework).   Many of my UK colleagues have known no academic life without such state surveillance.  But in most other countries, decisions on university funding are made without going through this time-consuming, expensive, and intellectually questionable audit of every university’s “research outputs” every five or six years.   Their research excellence has not conspicuously suffered in its absence.

The United States—whose universities currently occupy 7 of the top 10 slots in the THE World University Rankings—have no equivalent of the RAE/REF. This does not mean that research quality, whether of individuals or of schools and departments, is not evaluated.   It is continually evaluated, as anybody who has been through the tenure and promotion processes at a decent North American university, which are generally far more arduous (and rigorous) than their UK counterparts, will know.  But the relevant mechanisms of evaluation are those of the profession itself, not the state.  The most important of these are integrally bound up with peer-reviewed publication in top-drawer journals or (in the case of monographs) with leading university presses.  Venue of publication can be treated as an indicator of quality because of the rigor of the peer review processes of the top academic journals and publishers, and their correspondingly high rates of rejection.  Good journals typically use at least two reviewers (to counteract possible bias) per manuscript, who are experts in their fields, and the process of review is “double-blind”—i.e., the reviewer does not know the author’s identity and vice versa.  After an article or book has been published, citations and reviews provide further objective indicators of a work’s impact on an academic field.   The upshot is a virtuous (or, depending on your point of view, vicious) circle in which schools like CalTech, MIT, Princeton, or Harvard can attract the best researchers as evidenced mainly by their publication records, who will in turn bring further prestige and research income to those schools, maintaining their pre-eminence.

What immediately strikes anyone accustomed to the North American academy about the British RAE/REF is that at least in the humanities—which are my concern here—such quasi-objective indicators of research quality have been purposely ignored, in favor of entirely subjective evaluative procedures at every level.[1]  All those entered in the 2014 REF by their universities are required to submit four published “outputs” to a disciplinary sub-panel, whose members will then read and grade these outputs on a 4-point scale.   These scores account for 60 per cent of the overall ranking given to each “unit of assessment” (UoA), which will usually, but not always, be a university department.   The other 40 per cent comes from scores for “environment” (which includes PhD completions, external research income, conferences and symposia, marks of “esteem,” etc.) and “impact” (on the world at large, whose measurement has been the subject of considerable more or less entertaining debate), assigned by the same sub-panel.

Disciplinary sub-panels have some latitude in how they evaluate outputs, and in the natural sciences standing of journals and numbers of citations are likely to be seen as important indicators of quality.  The History Sub-panel has made it clear that it will not take into account venue of publication, citations, or reviews—so an article published in American Historical Review will be treated exactly the same as something posted on a personal website, as long as it is a published work.  The panel has also indicated that as a rule—and unlike with a typical book or journal article in the “real-world” peer review process that the panel has chosen to ignore—only one member of the panel will read each output.  While attempts will be made to find the best fit between the submitted outputs and the panel members’ personal scholarly expertise, in view of the size of the panel and the volume of material to be read this cannot always by any means be guaranteed.

In sum, in History at least—and I suspect across the humanities more generally—60 per cent of every UOA’s ranking will be dependent on the subjective opinion of just one panel member, who may very well not be an expert in the relevant field.   The criteria the panel intends to use for scoring outputs’ quality are (1) originality, (2) significance, and (3) rigor.  It is beyond me to see how competent judgments on these can be made by anyone who is not an expert in a field.   It is easy, on the other hand, to see why every university in the land was so desperate to get their people on REF committees.

For the stakes are high.  The aggregate REF score for each UOA determines (1) the ranking of that UoA relative to others in the same discipline across the country, and (2) the amount of research funding the university will receive for that UoA from the Higher Education Funding Councils until the next REF.   Both are critical for any school that has aspirations to be a research university, since—as all good sociologists know—what is defined as real is real in its consequences.

2.   What’s new in REF 2014—University-level assessment of outputs

In this context, it is important to note that—unlike in earlier RAEs—outputs scored at below 3* now attract no financial reward.  In previous RAEs it was in the interests of all universities to submit as high a proportion as possible of eligible staff with the intention of benefiting from a multiplier effect, and university websites boasted of the high proportions of their faculty that were “research active” as indexed by submission in successive RAEs.  In the REF, outputs scored at below 3* (defined as “Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence”) now dilute ranking relative to competitor institutions without any compensating gain in income.  The last thing any UoA wants is for the GPA that would otherwise be achieved by its 3* and 4* outputs to be reduced by a long “tail” of 2* and 1* valuations.

From the point of view of maximizing financial and reputational returns from the REF, the rational strategy for every research university is to exclude outputs ranked at 2* (“Quality that is recognised internationally in terms of originality, significance and rigour”) or below from entering a UoA’s submission, even if this means that fewer faculty are entered in the REF.  Sub-panels have also urged universities to be more selective in their submissions than in earlier RAEs, to ease their own workload.  All that material to read, often out of their fields of personal interest and expertise.  Why not sub-contract out the routine stuff?

Individual universities have responded differently to these pressures (and many, including Lancaster, have been understandably cagey about their plans).  But it now seems pretty clear that most, if not all schools with ambitions to be regarded as research universities are going to be far more selective than in previous RAEs in who they submit.  Lancaster warns on its internal staff website:

Lancaster University is aiming to maximise its ranking in each UoA submission so final decisions [on who is submitted] will be based on the most advantageous overall profile for the University. It is anticipated that the number of staff submitted to REF2014 will be below the 92% submitted for RAE2008 and not all contractually REF-eligible staff will be submitted.”

Rumor has it that the target figure for submission may in fact be as low as 65%., but that is, of course, only rumor.

Ironically, one of the reasons first advanced by the funding councils for shifting from the RAE to the REF format was “to reduce significantly the administrative burden on institutions in comparison to the RAE.”  The exact opposite has now happened.  Universities have had little alternative but to devise elaborate internal procedures for judging the quality of outputs before they are submitted for the REF, in order to screen out likely low-scoring items.  Many schools have had full-scale “mock-REF” exercises, in Lancaster’s case a full two years before the real thing, on the basis of which decisions about who will or will not be submitted in 2014 are being made.  The time and resources—which might otherwise have been spent actually doing research—that have been devoted to these perpetual internal evaluation exercises, needless to say, have been huge.

But more important, perhaps, is the fact that the whole character of the periodic UK research audit has significantly changed with the shift from the RAE to the REF, in ways that both jeopardize its (already dubious) claims to rigor and objectivity and could potentially seriously threaten the career prospects of individuals.  For the nationally uniform (at least within disciplines) and relatively transparent processes for evaluating outputs by REF Sub-panels are now supplemented by the highly divergent, frequently ad hoc, and generally anything but transparent internal procedures for prior vetting of outputs at the level of the individual university.   A second major irony of the REF is that if people at Leicester—or elsewhere—are fired or put on teaching-only contracts because they were not entered in the university’s submission, the assessments of quality upon which that decision was taken will have been arrived at entirely outside the REF system.

In previous RAEs all decisions on the quality of outputs were taken by national panels comprised of established scholars in the relevant discipline and constituted with some regard to considerations of representativeness and diversity, even if (as I have argued above) their evaluative procedures still left much to be desired.  In the REF, key decisions on quality of outputs are now taken within the individual university, with no significant external scrutiny, before these outputs can enter the national evaluative process at all.

3.  Down on the ground—enter Kafka

As I said earlier, most universities are playing their cards very close to their chests on what proportion of faculty they intend to enter into the REF and how they will be chosen.  I therefore cannot say how typical my own university’s procedures are.  I have no reason to think they are especially egregious in comparison to others’, but they still, in my view, leave considerably cause for concern.

The funding councils require every university to have an approved Code of Practice that ensures “transparency and fairness in the decision making process within the University over the selection of eligible staff for submission into the REF.”   Lancaster’s Code of Practice is published on the university website.  It is long on promises and short on detail, with all the wriggle-room one expects of such documents.  It says: “Decisions regarding the University submission to the REF will lie with the Vice-Chancellor on the advice of the REF Steering Group.  No other group will be formally involved in the selection of staff to be returned.”[2]  The Steering Group (whose composition is specified in an appendix) is tasked inter alia with adopting “open and transparent selection criteria,” ensuring “that selection for REF submissions do not discriminate on the grounds of age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex and sexual orientation,” and detailing “an appeal process that can be used by all members of eligible staff in order to seek further consideration for submission.”  All good stuff, though it is not discrimination on these grounds that most of my colleagues are worried about so much as the university’s ability to come up with procedures that will deliver informed and fair evaluations of their diverse work.  But what actually happens?

History department members were asked to identify four outputs for potential submission to the REF together with one or more “reserves.”  These items were then all read by a single external reader, who was originally hired by the Department in an advisory capacity as a “critical friend”—<removed>—but who has since been appointed by the university as an external assessor for REF submissions.   This assessor ranks every potential History UoA output on the four-point REF scale.  As far as I understand the procedure,[3] his ranking will be accepted by a Faculty-level Steering Group as definitive, unless he himself indicates that he does not feel competent to evaluate a specific output.  In that case—and in that case only—the output will be sent for a second reading by an independent specialist.   We are not told who chooses second readers or on what criteria; and colleagues have not been consulted on who might appropriately be approached for informed and objective assessments of their work.  All specialist readers remain anonymous.[4]

History is an enormously diverse discipline, in terms of chronology (at Lancaster we have historians of all periods from classical antiquity to the 20th century), geography (we have historians of Britain, several parts of Europe, the Middle East, India, South-East Asia, the Caribbean, and the United States), subject matter (economic, political, social, cultural, etc.), and methodology.  To expect any one historian, no matter how eminent, to be able to judge the quality of research across all these fields is absurd.   At least when outputs are submitted to REF Sub-panels there is a reasonable chance that the person who winds up grading them might have some expertise in or at least knowledge of the field.  The History Sub-panel has 24 members and a further 14 assessors.  Lancaster’s procedure, by contrast, guarantees that decisions as to whether individuals are submitted in the REF at all depend, in most cases, on the recommendation of the same individual, who is very likely not to be technically qualified to evaluate the quality of the work in question.  This is not a criticism of this particular assessor; no one individual could possibly be qualified to assess more than a minority of outputs, given the diversity of the historical research produced within the UoA.  Again, the relevant contrast is with peer reviewers for journals, who are chosen by the editors precisely for their specialist competence to assess a specific manuscript—not for their general eminence in the profession, or their experience of sitting on REF panels.[5]

I find it poignant that so cavalier an attitude toward evaluating the research of colleagues should be adopted in a university that requires external examiners for PhDs to be “an experienced member of another university qualified … to assess the thesis within its own field” and—unlike in North America—also requires all undergraduate work to be both second-marked internally and open to inspection by an external examiner before it can count toward a degree.   Why are those whose very livelihood depends on their research—and its reputation for quality—not given at least equivalent consideration as the students they teach?

A particular casualty of this approach is likely to be research that crosses disciplines—something, in other contexts, both Lancaster University and the Research Councils have been keen to pay lip-service to—or that is otherwise not mainstream (and as such may be cutting-edge).  Given the stakes in the REF, it always pays universities to be risk-averse.   Outputs may be graded below 3* not because their quality is in doubt but because they are thought to be marginal or unconventional with regard to the disciplinary norms of the REF Sub-panel, and what may be the most adventurous researchers excluded from the submission.

Though there is a right of appeal for individuals who feel they have been unfairly treated in terms of application of the procedure, Lancaster’s Code of Practice is crystal clear that: “The decision on the inclusion of staff to the REF is a strategic and qualitative process in which judgements are made about the quality of research of individual members of staff.  The judgements are subjective, based on factual information. Hence, disagreement with the decision alone would not be appropriate grounds for an appeal” (my emphasis).

This is the ultimate Kafkan twist.  The subjectivity of the process of evaluation is admitted, but only as a reason for denying any right of appeal against its decisions on substantive grounds.  These are exactly the grounds, of course, on which most people would want to appeal—not those of sexual or racial discrimination, though it might be argued that the secrecy (aka “confidentiality”) attached to these evaluations—we are not allowed to see external assessors’ comments—would make it impossible to prove discrimination in individual cases anyway.

4.   What comes next?

For whatever reasons, the funding councils for British universities have chosen to allocate that portion of their budget earmarked for research in ways that (at least in the humanities) systematically ignore the normal and well-established international benchmarks for judging the quality of research and publications.  Instead, they have chosen to set up their own panopticon of hand-picked disciplinary experts, whose eminence is not in doubt, but whose ability to provide informed assessments of the vast range of outputs submitted to them may well be questioned.  The vagaries of this approach have been exacerbated in the 2014 REF by putting in place a financial reward regime that incentivizes universities to exclude potential 2* and 1* outputs from submission altogether.  The resulting internal university REF procedures are not only enormously wasteful of time and money that could otherwise be spent on research.  More importantly, they compound the elements of subjectivity and arbitrariness already inherent in the RAE/REF system, ensuring that evaluations of quality on whose basis individuals are excluded from the REF are often not made by subject-matter experts.   Research that crosses disciplinary boundaries or challenges norms may be especially vulnerable in this context, because it is seen as especially “risky.”

Whatever this charade is, it is not a framework for research excellence.  If anything, it is likely to encourage conventional, “safe” research, while actively penalizing risk-taking innovation—above all, where that innovation crosses the disciplinary boundaries entrenched in the REF Sub-panels.  The REF is not even any longer a research assessment exercise, in any meaningful definition of that term, because so much of the assessment is now done within individual universities, in anything but rigorous ways, before it enters the REF proper at all.

Having made clear its intention to exclude from submission in the 2014 REF some of those faculty who would uncontentiously have qualified as “research-active” in previous RAEs, Lancaster University informs us that: “Career progression of staff will not be affected and there will not be any contractual changes or instigation of formal performance management procedures solely on the basis of not being submitted for REF2014.”  I hope the university means what it says.  But it is difficult to ignore the fact that Leicester University’s Pro-VC (as reported in THE) also reiterated that: “the university stands by its previously agreed ‘general principle’ that non-submission to the REF ‘will not, of itself, mean that there will be negative career repercussions for that person,’” even while spelling out his university’s intention to review the contractual positions of all non-submitted staff with a view to putting some on teaching-only contracts and firing others.

The emphasis on the weasel words is mine in both cases.  If universities truly intend that non-submission in the 2014 REF should not in any way negatively affect individuals’ career prospects, then what is to stop them saying so categorically and unambiguously?


[1] This is in part the result of a successful campaign by leading figures and organizations in humanities in the UK against “bibliometrics.”  While I accept that the expectations of journal ranking and citation patterns applicable to the natural sciences cannot simply be transferred wholesale to the humanities, I would also argue that an evaluative procedure that ignores all consideration of whether an output has gone through a prior process of peer review (and if so how rigorous), where it has been published, how it has been received, and how often and by whom it has been cited, throws the baby out with the bathwater.  It also gives an extraordinary intellectual gatekeeping power to those who constitute the REF disciplinary—in all senses—sub-panels, but that is an issue beyond the scope of this post.

[2] REF 2014 Code of Practice Lancaster V4 24 July 2013.  Quoted from LU website, accessed 9 August 2013.  My emphasis on formally.  So far as I can see, what this does is limit legal liability to the VC and a senior advisory committee, while immunizing those who are heavily involved in making the actual assessments of outputs, including, notably, departmental research directors and paid external assessors, against any potential litigation.

[3] I may be wrong on this point, but Faculty-level procedures have never been published—presumably because only the VC and university REF Steering Committee are formally involved in making decisions on who is submitted.

[4] This is the procedure for the History UoA.  Other UoAs at Lancaster may vary in points of detail, for instance in using internal as well as external readers.  I cannot discuss these differences, because these procedures have not been published either.  I doubt the variance would be such as to escape the general criticisms I am advancing here.

[5] Indeed, Lancaster’s Associate Dean (Research) for the Faculty of Science and Technology—where evaluating outputs is arguably easier than in the humanities anyway—has admitted that “some weaknesses in the mock REF exercise are apparent, for example in many cases there was only one external reviewer per department, no doubt with expert knowledge but not in all the relevant areas, who was engaged for a limited time only.”  Michael Koch, SciTech Bulletin #125.

I received this unsolicited email today, the latest of several of the same ilk.  The transformation of British academia into a used-car lot proceeds apace.

Dear Colleague,
Horizon Research Publishing,USA (HRPUB) is a worldwide open access publisher serving the academic research and scientific communities by launching peer reviewed journals covering a wide range of academic disciplines.
We invite you to submit your papers to our journals.


1. Free of Charge – Sept 30 2013 
All papers submitted before Sept 30, 2013 will be published free of charge.

2. High Quality, Rapid Production 
The time between initial screening and online publication is normally around 2 months.

3. Peer Review 
All new articles submitted must undergo a rigorous review. The review results will be notified within 30 days.

4. How to Submit 
Please submit your manuscript through our 
Online Manuscript Tracking System. Also, you can send your manuscript to submission@hrpub.org.
5. Register
Customer support is free for 
registered users .
6. Join Us as Reviewers
Please complete the 
membership application form  and send it to joinus@hrpub.org.


Horizon Research Publishing, USA

 

Following the publication of my Op Ed piece “A scandal in Bohemia” in the New York Times on July 10, I received the following email:

For your information, I have posted the following letter to my blog: labendz.wordpress.com

 

July 16, 2013

 

Dear Professor Seyer,

 

I read your recent opinion piece in the New York Times with great interest and you did not disappoint. Nonetheless, I must take issue with your tone and encourage you to rethink your position on the current political crisis facing the Czech Republic.

 

You write, “In light of Czech history, the latest scandal should be celebrated for its banality, its absurdity. It might just as well be happening in Silvio Berlusconi’s Italy.” And you rejoice in the fact that the Czechs have weathered a serious political scandal – the fall of a government – without resorting to regime change and the willful transformation of the nation-state’s ideological foundation. (I will leave aside the fact that this may be the first fully domestic event of this sort since the foundation of Czechoslovakia.)

 

Somehow, though, I find nothing at all banal about the high tolerance for political and economic corruption that has plagued the Czech Lands and Slovakia since before 1989 – and certainly after Czechoslovakia’s hyper-liberal transition to capitalist-democracy. Indeed, as far as I can tell from the Czech press, even this scandal seems not to have provided the necessary impetus for civil society to demand (in any effective manner) a drastic change in the country’s political culture. What should be celebrated about the failure to do so, when change is so dearly needed?!

 

To that end, why should any country – and its well-wishers like you and I – celebrate its supposed political maturity into something akin to the little sibling of Berlusconi’s Italy? If I believe that this truly represented Europe’s potential, I would be forced to join former President Klaus in his opposition to that Union. Fortunately for me, I do not need to keep such ODiouS company.

 

In his reappraisal of the Czechoslovak revolution of 1989, Timothy Garnton Ash penned a moving, if nostalgic, elegy for the lost political third way, between capitalism and socialism, that so many Czech and Slovak dissidents had hoped to achieve in 1989. According to Ash, they envisioned a moral society that deployed the economic and cultural power of a Western-style capitalist-democracy to fulfill the utopian visions and social imperatives championed – miserably – by political Marxism. Instead, the Czech Republic has become a bastion of political and economic corruption and – at least under Klaus – a poster child for the worst kind radical liberalism.

 

To my mind, the fact that one can characterize this crisis, with salience and integrity, as a banal event, should give us pause and force us to reappraise our praise. Evil banal must not remain evil unchecked. Or is this the terrible “end of history” – the point from which we can go no further? As an aspiring historian, I hope that this not to be the case. What an awful way to conclude one’s book!

 

 

Sincerely,

 

Jacob Ari Labendz

Doctoral Candidate

Department of History

Washington University in St. Louis

 

I welcome debate, and think Mr Labendz deserves an equally public response.  His concerns are serious ones.  But so were mine.

In the Preface to my book Capitalism and Modernity, dated 31 December 1989, I argued that Czechoslovakia’s Velvet Revolution “was not the sort of revolution to which the modern world is used.  It was, for once, a revolution against those ersatz gods of modernity who have stolen, by divine right of ideology, decades of people’s lives, hopes and dreams; a refusal of the reduction of the personal to the political.”  I went on to suggest that “we may be witnessing, in Eastern Europe, not the return of the prodigal to the fold of ‘the West’ hailed by politicians from Thatcher to Bush, but something quite new: a ‘post-modern’ revolution…”

Three months later I visited Czechoslovakia for the first time.  One episode in particular from that trip stuck in my mind.  I wrote about it later in Going Down for Air (2004):

It is early in 1990, that uncertain time between the Velvet Revolution and the first free elections in forty-two years.  At a loose end one evening I decide to take myself to the local opera house, where [Verdi’s] Nabucco happens to be playing.  Before the performance begins a man steps through the curtain to the front of the stage.  I expect to learn that one of the principals is indisposed.  Instead he makes a passionate plea on behalf of his fellow-artists for the audience to vote for Civic Forum, the opposition coalition formed in November 1989 around Václav Havel.

            If the communists steal the elections they’ll make Cambodia look like a dinner party, somebody remarks during the interval.  The tension is palpable, catching in the throat, acrid as the acid rain that hangs in the Ostrava air.

            Va, pensiero is heard in absolute silence—something I have never experienced at an operatic performance anywhere in the world.  No coughing, no shuffling, no whispering.  Nor, I think, have I ever heard a demand for a chorus to be encored.  When the last echo of the last note dies, but not before, the theater explodes.  People are on their feet, yelling for it to be played again.  Va pensiero was encored three times that night before the opera could go on.  I was later told that the same thing had happened in the National Theater in Prague after the Soviet invasion of 1968, leading to Nabucco being banned from Czech stages for the next twenty years.

Verdi’s Nabucco is set in the time of the Jews’ exile in Babylon, and “Va pensiero” expresses the Hebrew slaves’ longing for their lost homeland.   At the opera’s premiere in La Scala, Milan, on March 9, 1842, “Va pensiero” was repeatedly encored.  The performance ended in a riot.  Henceforth, the Austrian authorities decreed, no encores would be allowed in Italian opera houses.

Not for the first or the last time, my evening at the opera in Ostrava led me to ponder the myth of eternal return: the heresy, from a modernist point of view, that history might be better understood in terms of repetition-compulsion than onward and upward progress, or, as a Czechoslovak communist slogan of the 1950s once put it, “Kupředu, zpátky ni krok” (Forward, backward not a step).

I think I know where Mr Labendz—and Timothy Garton Ash—are coming from.  Faced with the sordidness of current politics in Bohemia, it is difficult not to feel nostalgia for those fearful, hopeful days of 1989-90.  But I wonder how much of the nostalgia is for the hopes that we invested in the Czechs, on which they have declined to deliver.   Not for the first time.  Friedrich Engels made a similar complaint when, after a promising start, the Czechs failed to play the role allotted them by the materialist conception of history during the revolutions of 1848-9.  The Czechs, he fumed, “are an absolutely historically non-existent nation.”

Almost twenty-four years have passed—as I said in my Op Ed piece, longer than the entire existence of Tomáš Masaryk’s First Republic—since the events of November 1989.  The democracy established by the Velvet Revolution has survived, even if it lamentably fails to live up to the hopes once invested in it by western “well-wishers.” Whatever we may think of present-day Czech political culture, the fact that a government can fall as a result of the implication of senior political figures, up to and including the Prime Minister, in graft and corruption—and indeed, that the police had sufficient independence from that government to be able to investigate and expose that corruption—is (perverse) proof of just how much has been achieved since 1989.  My point of comparison here is not the Utopian might-have-been of the unrealized dreams of that annus mirabilis, but what once was, the nightmares of the previous half-century.

So yes, I believe there is much to celebrate in the current scandal in Bohemia.  I do not celebrate the corruption per se—though I have come to expect it in liberal democracies.  (I prefer Berlusconi’s Italy, repugnant as it might be, to Mussolini’s.)  I celebrate the fact that the corruption can be exposed.  I celebrate the fact that the fall of a government does not entail a revolution in the whole political system: that politics no longer swallows up civil society.  Above all, I celebrate the fact that today’s Czech public is entertained by the sexual shenanigans of Mr Nečas and Ms Nagyová, not the grotesque political show trials of Milada Horáková and Rudolf Slánský.

As regards the “end of history,” history can be seen as having an end—whether understood as a terminus or a goal—only when it is equated with progress toward some ideal end-state.  I do not make that equation, and I believe modern Czech history gives us every reason not to.  As I wrote in the introduction to Prague, Capital of the Twentieth Century (available here), “The city’s modern history is an object-lesson in humour noir.  Where better to acquire an appreciation of irony and absurdity, an enduring suspicion of sense-making grand theories and totalizing ideologies, and a Rabelaisian relish for the capacity of the erotic to rudely puncture all social and intellectual pretentions toward rationality?”

My book, to which the Times article is a very small footnote, attempts to treat twentieth-century Prague as Walter Benjamin did nineteenth-century Paris—as a repository of the dreamworlds of an era.  I see Mr Labendz’s “lost political third way” (“a moral society that deployed the economic and cultural power of a Western-style capitalist-democracy to fulfill the utopian visions and social imperatives championed—miserably—by political Marxism”) as belonging to those dreamworlds.  It is time to wake up.

Now that Prague, Capital of the Twentieth Century has been out for three months, the reviews are starting to appear.  I’m really gratified by the range of coverage as well as the welcome the book has received.  I did not expect to be listed in the Financial Times “Books of the Year (so far)” (thank you Tony Barber), or for Nicolas Rothwell, one of Australia’s most distinguished writers, to honor me with a 2000-word review in the Australian.  Thanks to all the reviewers, as well as to my publicists at Princeton, Jessica Pellien and Katie Lewis, for doing such a great job of getting the book noticed.

Here are some extracts from reviews.  Click on the links for full text.

“[A] captivating portrait of 20th-century Prague. . . . The breadth of Sayer’s knowledge is encyclopedic, and those willing to stay the course will be rewarded.”–Publishers Weekly

Prague, Capital of the Twentieth Century is an erudite, comprehensive, well-illustrated and witty account of Czech art, design, architecture, literature and music in an era–stretching roughly from Czechoslovakia’s creation in 1918 to the end of the second world war–when few in Paris, Berlin, London or even New York would have thought of the Czechs as not being part of western civilisation. . . . [I]n this book [Sayer] has succeeded in bringing back to life a golden avant-garde era that not long ago was in danger of being written out of history altogether.”–Tony Barber, Financial Times

“Can a city define a century?  Sayer seems to think so and he foregoes a conventional retelling to concentrate on themes, mixing literary and political, the harshly realistic and the absurd.  This is the city of Kafka, Havel and communism with a human face.” —Michael Conaghan, “7 books you should own,” Belfast Telegraph

“Sayer has written a cultural history chockablock with artists, modernist architecture, manifestos, dark comedies, and broken alliances. . . . the scholarship is impressive, the illustrations fascinating … [The book] will be valued by those interested in European cultural history during the twentieth century and how modern art was colored by the horrors of the political landscape.”–Karen Ackland, ForeWord Reviews

“[C]ompelling tales of a city’s artistic, intellectual and political cultures … [T]he book . . . offers an insight into often quite extraordinary life stories connected with Prague as well as their international context.”–Marta Filipova,  Times Higher Education

“[T]he reader of this hypnotic, mazy “surrealist history” turns from its cascade of interlinking chapters quite caught up in words and their shadows, almost swept away … Prague is the stage set for a relentless examination of the hopes of modernism and their eclipse: the capital where irony and absurdity come to shape time’s patterns … Sayer is a master of his sources: he looks back on a past still within reach, receding from us; he tracks down its threads, from liaison to liaison, from city to city. Can a research professor ever have written a book quite so triumphantly eccentric and persuaded a major academic press to publish it so splendidly?”–Nicolas Rothwell, Australian

“Through both the breadth and depth of his knowledge, Sayer will reward the patient reader; in the surrealist fashion, he focuses on the seemingly mundane details to provide a true biography of Prague.”–Kelsey Berry Philpot, Library Journal

“[A] surrealist document in its own right, revealing its truths in a big, messy knot of jarring juxtapositions, playful obscenities, and found objects of profound beauty … [Readers] will likely find themselves delighted by Sayer’s erudition as he reintroduces dozens of figures, many long forgotten or scarcely known to non-Czechs, into our understanding of twentieth-century cultural history.”–Brendan Driscoll, Booklist

“This is a brilliantly written and fascinating book that combines elements of the literary guide, biography, history and essay. Authoritative and subversive, Sayer’s narrative is intellectually dense and brilliantly accessible.”–Dublin Review of Books (Information and extracts)

[T]his is a broad cultural history . . . with Sayer ranging easily across the arts. . . . [C]ontinually illuminating.”–Andrew Mead, Architectural Review (feature review)

“In this erudite, witty and well-illustrated book, Sayer restores Czech avant-garde art between the two world wars to its rightful position at the heart of European culture. A worthy successor to Sayer’s much-praised The Coasts of Bohemia.”–Financial Times, “Books of the Year So Far,” Summer Books Guide

“A real page-turner that leads the reader through all possible facets of Modernism in Prague, starting with Breton’s and Eluard visit to the city in 1935 and ending with the crashing of all modern and Surrealist legacy by the Communist regime in the 1940s and 50s. At the same time, Sayer’s book pays also great attention to previous periods while putting also a strong emphasis on the many efforts, from the Prague Spring till today’s resistance to Prague’s Macdonalization, to recover the revolutionary power and intuitions of the past, in the field of art but as well as in that of daily life. . . . [A] fabulously good read. . . . Derek Sayer stands already out as one of the most convincing representatives of how to rethink our cultural past today.”–Jan Baetens, Leonardo

“A thoroughly engrossing book.”–Jim Burns, Northern Review of Books

There was also substantial coverage of Prague, drawing on Princeton’s publicity material and earlier reviews, in Art Daily.   I was interviewed on the book for BBC Radio 3’s “flagship arts and ideas program” Night Waves (full podcast available here) and more recently for Radio Prague (text and podcast available here).  And I was invited to write an Op-Ed for the New York Times, “A Scandal in Bohemia,” discussing the current Czech political crisis against the background explored in Prague, Capital of the Twentieth Century.