The collaboration behind the replication

We attempted a 19-country replication of the original Prospect Theory study from 1979. The story of how our collaboration of 30 went from creating opportunities for early career researchers to comprehensively revisiting Kahneman and Tversky is just as important as our findings.

Like 2 Comment
Read the paper

Every year since 2011, I have had the extreme honor of getting to lead a team of brilliant early career researchers from the Junior Researcher Programme. Through the JRP, on top of six annual team research projects we support, all participants that successfully complete the year are offered an opportunity to contribute to a large-scale research collaboration. The focus of these collaborations has changed over the years, with the original approach of applying the annual JRP theme (such as cognitive biases, well-being, and organizational behavior) to an ongoing study on global health, which we finished in 2015.

Beginning in 2016, we implemented the first of three major transitions to that approach. Recognizing the growing interest in applying psychological research to public policy, we produced a comprehensive policy report covering five major themes in psychological science, with Junior Researchers contributing to the section of their research interest. The report produced by that brilliant group of young professionals generated so much interest and encouraging feedback that we were asked to up the ante by producing a full textbook in 2017.

Just as the first edition of the book was going to press and work on the second edition was being sketched out, a controversial article came out calling into question the validity of loss aversion (simply: the preference for certainty with positive values but for risk with negative ones). While the role of psychological applications to policy go far beyond choice, rationality, and behavioral paradoxes, there is no question that a great deal of trust is placed in loss aversion and those studies most closely associated with it. The most visible such study is without question the 1979 paper on Prospect Theory by Daniel Kahneman and Amos Tversky.

Regardless of where you stand on replication being a crisis or simply a consideration, there is little room to debate the importance of ensuring that assumptions based on scientific evidence are reliable. Along with that, behavioral themes used often to justify applications in policy tend to rely on core concepts such as loss aversion, temporal discounting, and bounded rationality as well as various biases and heuristics. Our own writing was no different, so we of course felt a sense of obligation to check if our reliance on something as pivotal as Prospect Theory was appropriate. No evidence, irrespective of its place in the canon or the accolades of its forebears, is beyond reproach. So after completing work on the second version of the textbook, we implemented the third transition: focusing on a singular, large-scale replication of the 1979 paper.

Carrying out this scale of work involves a gargantuan administrative effort as well as mobilization of human resources en masse. In our case, we set out without project funding or a central research bases. Volunteer coordinators within the JRP as well as a small number of senior advisors that I regularly turn to helped build guidance documents and data collection tools. This also involved multiple rounds of translation, seeking obscure details from the original study, calling in favors for anything from finding income inflation data from old acquaintances working in government to asking mutual friends to make introductions to editors and advisors associated with the original publication.

We started with a group meeting of 25 researchers in Madrid in February 2019 to hash out the formal instructions. After several months of ensuring instruments were sufficiently in place, we began testing in July 2019. The format of the JRP has a requirement that anyone wanting to participate in the collaboration must clear entirely a five-week window late in the summer – this approach maximizes efficiency in the process, especially in terms of facilitating the level of focus and routine goal-sharing that can make (when present) or break (when absent) a collaboration of this scale.

From late July through all of August, we did just that – with all team members spending five weeks living together and spending every day focused on completing this project. While highly intense and at times stressful, the sense of camaraderie and fulfillment from working on this with such a motivated group of early career professionals is made even greater knowing that it resulted in such a meaningful contribution to science.

Frankly, I am looking forward to the next one. While we do not anticipate “just replicating” studies year in and year out, we certainly plan to keep this new model in place for the foreseeable future. How we approach that will be the topic of the second part in this series. If you are interested in becoming involved with the JRP, or you know an early career researcher interested in psychological science, be sure to check out the links in this article - applications to the 2020-21 cohort are open through May.


Special thanks to Corpus Christi College and the Centre for Business Research at the University of Cambridge for hosting the five-week window (again). More on that story here. Photo collage credit to Marlene Hecht and Bojana Većkalov. Images are of all authors in alphabetical order (Wang missing).

Go to the profile of Kai Ruggeri

Kai Ruggeri

Assistant Professor, Columbia University

No comments yet.