March 3, 2024

Nearly 2/3 of psychology papers cannot be trusted

Out of 100 papers, only 31 could be replicated.

Nearly two-thirds of all psychology research should be distrusted because it cannot be replicated, according to a troubling paper in Science.

This study reports work by the Reproducibility Project, an initiative at the University of Virginia in Charlottesville which was launched in 2011 after a succession of scandals in psychology research. Brian Nosek and a huge team of co-authors tried to reproduce the results of 100 articles in three leading journals. They found that only 39 could be replicated.

Although it is impossible to say whether the conclusions of a paper are true or false only because its results cannot be reproduced, it does suggest that it might not stand up to scrutiny. An expert in medical statistics, John Ioannidis, of Stanford University, told Nature that he fears that the true level of non-reproducible research may be as high as 80%.

Reproducibility is at the  heart of the scientific method. As Nosek writes in Science, “Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence. Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error.”

Part of the problem is that promotion and prestige depend mainly on innovation and new research; there are few kudos for confirming another scientist’s results. Nonetheless reproducibility is vital: “Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both. Replication can increase certainty when findings are reproduced and promote innovation when they are not.”

“We see this is a call to action, both to the research community to do more replication, and to funders and journals to address the dysfunctional incentives,” Nosek told the New York Times. He says that research in other disciplines needs to be evaluated as well.
Creative commons
scientific method