Learning from Registered Reports

An exciting day for Nature Human Behaviour! We published our first two Stage 2 Registered Reports. I seize that occasion to share a few thoughts on the process of handling Registered Reports at the journal.

Go to the profile of Marike Schiffer
Jul 22, 2019
2
0

The From the Editors section is a channel for us to share our personal views and experiences, so I'll start with a personal disclosure. I became a fan of the Registered Report format, even before I knew about Nature Human Behaviour. I had drafted my first RR protocol and looked for a place to submit it when I came across an NHB announcement inviting RR submissions. So, my first interaction with NHB, before I quit academia to become an Editor, was the submission of a Stage 1 RR protocol (it was desk-rejected).

Registered Reports are an article format where peer-review comes before data collection, and journals commit to publication based on the planned research and analysis, before results are known. Editorially evaluating Registered Reports means evaluating a question and a plan. It means you base your decision on two things only: how important is the question for the discipline? Can the report, following constructive peer-review, provide robust answers? I really like this way of approaching a manuscript and it has certainly influenced the way I read other papers, too. 

Peer review: positive, but sometimes lengthy

A highlight of RRs is that peer-review feels more constructive. Many people have described standard peer-review as the post-mortem of research, which in the worst cases uncovers methodological flaws of a project that render it unpublishable. It’s too late to help. Peer-review of an RR protocol is different. Reviewers seem to approach the task with a more constructive mindset. It reminds me more of a critical lab-meeting where one colleague presents a new project and others provide feedback, so that no time and resources are wasted on a project that falls short of its potential. Everyone is looking for potential pitfalls, but with the aim of making suggestions for how these can be avoided.

So, handling Registered Reports differs from standard formats, and some of the differences present challenges. Most of the RRs that I’ve handled so far have often had an extra round of review compared to standard submissions. Other editors and authors have argued that the time lost is later recouped by swift publication of the Stage 2 RRs. I think this is true; I also believe the pace is a side effect of dealing with a new type of publication that require us to rethink. Abandon "What do the results say?" in favour of "What can we learn, and how can we make  sure we know how to interpret the results?".  Reflecting on my learning curve and the way the community of authors and reviewers has developed even over just the last two years makes me hopeful that lengthy review is just a teething problem.

Experience helps authors, reviewers, & editors

Why is that? For many submissions, not a single author on the RR protocol has authored any RRs before. With standard publications, the first author may be new to the game, but at least one senior author would usually have had published a paper before; they will also have read a lot of papers. This is different for Registered Reports, where some authors learn what is expected of them after having submitted their protocol. I do see more and more high quality first-submissions though, which I attribute to the availability of more Stage 1 RRs that new authors can use as a template, the increase in excellent guidance in the form of publications, recorded talks, and material on OSF. Finally, of course, some RR authors are now repeat offenders.

For each RR we send out, we have one reviewer in addition to the subject experts who is an RR expert. The list of people we can call on to be that expert is growing, which helps. Moreover, these day the reviewers who act as subject experts are also more familiar with the format, which means they find it easier to provide us with helpful feedback.

Of course, it’s not only authors and reviewers who have learned a lot. It’s us, too. These days, when I send an RR out to review, I often know what the pinch points will be. 

If there is a simple, clear-cut problem, I can ask authors to address it prior to peer-review. To provide one example, one of the most widely used tools for a priori power estimates which works a spell for simple designs, cannot provide power estimates for repeated measures ANOVA designs with 2+ factors. Yet, it is regularly used for that purpose. Catching these things early helps avoiding additional rounds of review.

Stats aside, handling quite a few of these manuscripts by now, I have much better understanding of common misconceptions, or of how our feedback can be misinterpreted – which also causes unnecessary delays. For example, many RR protocols lack detail. The way that hypotheses are often described in standard research papers does not contain any clear commitments. I believe this way of speaking about hypotheses has become a (bad) habit. Rising awareness of HARKing (hypothesizing after the results are known) and p-hacking have taught us (the scientific community) that a hypothesis which predicts “behavioural differences between conditions”, without defining the direction of the difference (A > B or B > A?), or the measure (error rates? / mean RTs?) is too vague for confirmatory research. For RRs, which are meant to prevent p-hacking and HARKing, a hypothesis that does not specify the pattern of results that would support it at the level of the analysis measure is often useless. If a protocol does contain that level of detail, that’s a great start. It is not enough; what we often do not think about is that to protect us from HARKing and hindsight bias it’s pivotal to commit to how alternative patterns of results would be interpreted. Many protocols omit that information. I have learned to explain both aspects in decision letters, rather than asking for “detailed hypotheses”, and thereby implicitly and unfairly relying on authors to just guess what they need to do. As I said, handling Registered Reports has changed how I view all types of papers. I wonder whether the way we speak about hypotheses and predictions in the context of RRs will change habits for 'standard' research as well. 

So, after two years of handling RRs at NHB, I remain a huge fan of the format and hope to see it grow across disciplines and journals. I have no doubt that the RRs we got to publish today (https://www.nature.com/articles/s41562-019-0641-3https://www.nature.com/articles/s41562-019-0644-0) and the Stage 1 RRs currently in the works (most are on figshare) are testament to their huge potential.  

 

 


A footnote: what happened to *my* RR? The reason for desk-rejection explained to me in the decision letter was the protocol's narrow scope, i.e. that while it would certainly generate new insights, these did not hold the promise to address a research question of broad importance to the field. I spent 20 seconds disagreeing with the decision & feeling misunderstood, before realizing that that's an appropriate description of the project. Important to a small group of researchers working on specific questions of reinforcement learning and episodic memory, and with the potential to turn into an informative study, but not of groundbreaking importance to the field. I make decision such as this (based on scope) myself today, and they’re tough and sometimes painful to make. The report was successfully submitted elsewhere. I withdrew it after 2 rounds of review when I quit academia, but it’s in my former colleagues’ hands to revive it any time. The report is written, the experiments are coded, and the analysis scripts produce figures and stats. In contrast to everything else I planned but never finished, this work can continue. It is more likely to be resurrected than any other project I've left behind.

Go to the profile of Marike Schiffer

Marike Schiffer

Associate Editor, Springer Nature

Expertise: Cognitive Psychology, Neuroscience, Registered Reports. Bio: Before I saw the light and left Academia to become an Editor at Nature Human Behaviour I oscillated between neuroscience and psychology, with a BSc degree in Psychology (major in Cognitive Neuroscience) and an MSc in Neuropsychology, as well as a PhD in Cognitive Neuroscience and 4 yrs as a post-doc in Experimental Psychology plus a short stint as a Lecturer in Psychology. At Nature Human Behaviour I handle submissions in Cognitive Psychology and Neuroscience.

No comments yet.