Most academic papers today are published only after some academic peers have had a chance to review the merits and limitations of the work. This seems like a good idea, but there is a growing movement that wants to retort as Albert Einstein did to such a review process.
Academic review process was different in Einstein’s time. In his brilliant career, the only time his work was subjected to blind peer review – the authors don’t know the reviewers and vice versa – he showed contempt for what is now the gold standard of science. Was Einstein right to be so suspicious of the peer-review process? Should we learn from him and begin to question the widespread use of peer review in academic science?
The first part of Einstein’s career was in the German-speaking world. The German physics journals, in which Einstein published his breakthrough work, didn’t have the same peer-review system we use today.
For instance, the Annalen der Physik, in which Einstein published his four famous papers in 1905, did not subject those papers to the same review process. The journal had a remarkably high acceptance rate (of about 90-95%). The identifiable editors were making the final decisions about what to publish. It is the storied editor Max Planck who described his editorial philosophy as:
To shun much more the reproach of having suppressed strange opinions than that of having been too gentle in evaluating them.
Many of the core scientific discoveries were not peer reviewed to modern standards. For example, the publication of the foundational paper describing the double helical structure of DNA by James Watson and Francis Crick in 1953 would have been jeopardised in the context of the classic review system as we know it, because of its speculative nature.
At the prestigious journal Nature, the peer-review system was only formally introduced in 1967. More recently, the discovery of distortion in gravitational waves by a telescope at Harvard – which has crucial consequences for our understanding of the formation of the universe – was presented as preliminary and treated with extreme caution and even sometimes with denigration, because it had not been peer-reviewed.
American adventure
It was only after Einstein came to the US in 1935 that he came face to face with the peer review process. He and his younger colleague, Nathan Rosen, sent a paper on gravitational waves to Physical Review, a journal which had established its reputation as the premier physics journal in the US. The paper had the potential to be highly controversial as it challenged the idea that gravitation was a wave.
John Tate, the editor of the journal, hesitated over Einstein’s paper for a month. He then send it to a reviewer for comments – his selected reviewer was probably the famously gossipy Howard Percy Robertson, one of Einstein’s colleagues at Princeton. The reviewer returned ten pages of comments which cast doubt on many of the central claims in the paper. The editor returned these comments to Einstein, asked him to consider the issues, and make any changes he saw necessary. Here is how Einstein reacted:
We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorised you to show it to specialists before it is printed. I see no reason to address the – in any case erroneous – comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.
Although he withdrew the paper from Physical Review, Einstein went on to publish it in a much more low key outlet, the Journal of the Franklin Institute. However, the published version contains substantial revisions. It appears these revisions were largely on the basis of a discussion he had with Robertson at Princeton. The revised version toned down many of his original huge claims. These revisions may have saved him from public embarrassment.
What would Einstein say today?
Some might see this as an amusing historical incident. But we think it contains some important lessons for scientists of all kinds today. This is because it reflects the current tension regarding the peer-review system.
The story reminds us that double-blind peer review is only a relatively recent invention. For most of history of science, scientific advances were judged in a much more open and public fashion. It also shows us that the peer-review process can provoke displeasure among even the greatest. It can mean scientists not listening to criticism. Sometimes the result is that many ideas don’t see the light of day.
These anecdotal lessons point to wider issues with the peer-review process, which itself hasn’t been studied in much detail. The review process was meant to save scientists from mistakes and public embarrassment. The idea was that peers help to improve our work, and the review process of high-status journals can serve as “stamps of approval” or simply signal of quality.
But sometimes a collegial discussion rather than formalised peer review can be a better way of getting the message across. So far the peer-review process has been largely an item of faith – something that probably produces better science. However, there is a growing body of evidence which is challenging this notion.
An extensive review of the literature on peer review in 1998 identified problems. They found that there is a low level of reliability and agreement over the quality of submitted papers, largely because of a lack of objective evaluation criteria. Even worse, reviewers make mistake in their evaluation and often accept papers they should have rejected. As a direct consequence, established journals are usually biased against innovative work.
In our own field of management science, some have claimed the peer-review system means academic work can simply end up losing its integrity during the review process, and can result in trivial and boring research.
On a more positive note, when reviews are perceived of quality by authors, they tend to generate more citations, which is a measure of the number of times a research paper is mentioned in other journals and is considered a mark of quality. Also, reliability is not necessary for an efficient review process – often it is the process of peer review itself that contributes to improving the paper. Reviewers play a developmental role in the construction of knowledge, and the energy they deploy in this process is primarily driven by moral motives rather than any material interest.
Bad review for peer review
Perhaps the most gentle solution would be to improve peer review. There are clearly disagreements about how this might be done. Some claim the peer-review system needs to become more objective through the introduction of clearer criteria and better trained reviewers who are able to systematically apply these criteria. Others claim that some subjectivity is important because it can stop reviewers herding to established ideas, thereby crowding out alternative and often more innovative approaches.
The frustration regarding the peer-review system has lead to new hybrid systems to emerge. For instance, some scientific communities have experimented with making reviewing process public.
In the hard sciences, there are those who post papers online and other scientists decide whether they are worth being cited. PLOS ONE publishes any paper that has been considered as “technically sound” after a round of editorial review, and readers then judge the relevance of the research. Another alternative system would be to have a set of reviewers rating all the papers submitted online, and revising their judgement in case of resubmission. The growing number of open access journals has raised concerns that peer review would be progressively abandoned and search engines and metrics will replace editors and peer reviewers.
Let’s try something new
Some, like Einstein before them, think that the peer-review system should be abandoned in favour of a “market of ideas” where the best research would naturally be identified by the crowd, hence reducing the cost of the review process. There are many potential dangers of these alternatives to peer review, the most obvious being expanded opportunities for “bad science” to masquerade as legitimate work. However, given the immense cost and frustrations associated with the peer-review process, we think it may be worth considering alternatives.
Peer reviewing is an important scientific institution. But there might need to be a range of forums in which scientific results and discussion takes place – peer-reviewed journals only being one among a number of options. Such options would then compete for both the attention of the readers and the best papers. We think this mixed scientific landscape would have pleased Einstein.
Next, read this: Scientists falter as much as bankers in pursuit of answers