Peer review has constantly been under the spotlight on the social media, in the press and among academics. Among the most popular Google entries, there is a long list of pieces with titles such as, “The end of the peer review show”, “peer review is broken”, “a flawed process” etc.. When we talk on this subject and want to break the ice, an opening gambit is “I suspect that many of you would have a lot of comic or dramatic personal experiences to share about peer review”. Only to add “Ah?” then instantly, sly smiles all around. In a minute, we are overwhelmed by embellished stories about the law of ‘referee #2’, impolite sentences copied from reports echo through the room and people start complaining about receiving reports matured like a vintage bottle of wine. It is easy to captivate your audience by stimulating academic frustration. However, you could even win the jackpot if you showed findings of bias by referees and editors, - sympathy and empathy all around. Probably, a long time ago, this frustration would only have been a confidential subject of conversation to build friendships during coffee breaks at academic meetings. Now, in the age of Twitter and blogs, this frustration has reached a wider audience.
We began reflecting about how to move from anecdotes to evidence-based arguments on peer review in 2011. Flaminio was at a conference on open science in Oxford, where he convinced some professionals from the publishing industry to endorse an EU proposal on peer review he was thinking to put together. A few years later, the project was eventually funded by COST under the title of “PEERE-New Frontiers of Peer Review”. This is when the two of us met for the first time.
A physicist by training with academic experience as a post-doc, Bahar started her job as managing editor only to realise that not only peer review but also the entire publishing process required urgent improvement. She started experimenting with recognition and transparency in peer review and was involved in a pilot of publishing peer review reports at Elsevier, which was the origin of our article.
We realised from the start that we shared the same ambition and goal, though from different sides of the publishing system. PEERE was the right context, as it opened new possibilities to study the impact of the Elsevier pilot on peer review. I remember that Bahar noted that the journal editors involved in the pilot had the impression that referee behaviour changed during the pilot. We decided to perform a study on Elsevier journal data. Our three co-authors, Giangiacomo Bravo, Francisco Grimaldo and Emilia López-Iñesta, were so enthusiastic to have good data to handle, not to mention the enjoyable time they later spent cleaning up the data during their weekends!
However, it took a long time from conception to the final article. Finally, it was not established whether an independent group of academics would have tested Elsevier’s internal trial data! PEERE was instrumental in this. We used a protocol for data sharing, which we developed together with other publishers and academic societies (i.e., Wiley, Springer-Nature and The Royal Society) as part of the PEERE project (see Squazzoni et al. 2017). Note that developing the data sharing protocol required about two years of work by many PEERE members, with intensive discussions about privacy, anonymization and ethical issues.
While some scholarly journals have started publishing peer review reports under increasing pressures for transparency in science, it was still unclear how this practice could affect the peer review process. Editors seemed to find differences. However, our sample was sufficiently large and complete to investigate all the most important issues: Does open peer review compromise referees’ willingness to review? Does it influence referee recommendations? Does it increase turn-around times? Does it change the way referees write their report? Is this for good or bad?
By measuring changes both before and during the Elsevier trial, we reconstructed the effect of peer review manipulations (from confidential to open) in a ‘quasi’ experimental scenario and examined referee behaviour under these two different conditions. We did not find any negative effects on the main factors under investigation. We found that younger and non-academic scholars were more willing to accept to review under open peer review. Male referees tended to write more constructive reports during the trial. In general, we did not find any serious criticalities. However, only 8.1% of referees agreed to reveal their identity in the published report and those who agreed had previously recommended acceptance of the manuscript or minor revisions.
These findings would suggest that open peer review does not compromise the process, but indicated that referee anonymity must be protected. Those who did not want to have their names in the report were perhaps motivated by avoiding retaliation by authors. Our study is positive on open peer review but it suggests that transparency can transform peer review into a signalling game.
This brings us back to the fact that science is a hierarchical, status-based system and any innovation in peer review must be coherent or at least informed by what is the reality around peer review. The most important lesson we have learnt from running PEERE has been that peer review is only part of a larger and more complex system. It is not something isolated from its context; reviewers are not post human creatures from Mars but are of the same nature as those submitting manuscripts to journals. If so, we must remember that scientist behaviour reflects the set of incentives, rewards and rules that characterises the whole scientific community and its institutions. The hyper-competition that dominates the academic world could influence peer review, and we must do our best to preserve the autonomy of referees while at the same time we should create conditions to stimulate constructive and more transparent peer review.
Squazzoni, F., Grimaldo, F., Marušić, M. 2017. Journals could share peer-review data. Nature, 546, 352.
Flaminio Squazzoni is professor of sociology at the University of Milan, Italy, where he leads BEHAVE, a new centre for research and training in behavioural sociology. He has chaired PEERE “New Frontiers of Peer Review” (2014-2018). Follow him on Twitter @squazzoni
Bahar Mehmani is Reviewer Experience Lead in the Global Publishing Development department at Elsevier. She works on several projects related to peer review, all of which are designed to recognize reviewers' contribution to the progress of science. She received her PhD in Theoretical Physics from the University of Amsterdam (UvA) in 2010 and has been postdoc researcher at Max Planck Institute for the Science of Light (MPL). Follow her on Twitter @mehmanib