How retractions hurt scientists’ credibility

Life scientists who have published papers that are retracted by journals subsequently suffer a 10 percent drop in citations of their remaining work, compared to similar but unaffected scientists, according to a new study by MIT researchers.

Examining hundreds of cases over a 30-year period, the research quantifies the extent that one discredited study — whether an act of malfeasance or a sloppy piece of research — has on the overall reputation of academic scientists.

“The question we’re asking is: Do retractions trigger, at an individual level, something like an infection mechanism, where the retracted author is being punished and discredited for being dishonest or just incompetent?” says Alessandro Bonatti, an associate professor at the MIT Sloan School of Mangement and a co-author of a new paper detailing the study. “We find that yes, there is such a mechanism in place, and it operates through citations.”

The study also finds that, in cases of clear misconduct, high-profile scientists who have a paper retracted experience an even larger drop — 20 percent — in the citations of their additional work.

The study adds to a growing literature on retractions and related problems in science, and suggests that the system of peer-review, while not perfect, does give people in the scientific community room to change their preferences about the quality of work presented to them.

As the authors write in the paper, the academic process of peer review may not provide “the optimal incentive system” in every regard, but the results do run against the “narrative that regards peer review as fundamentally undermined by … forms of misconduct.”

The paper, “The career effects of scandal: Evidence from scientific retractions,” has just been published online by the journal Research Policy, and will appear in a print edition as well. The authors are Bonatti; Pierre Azoulay, the International Programs Professor of Management at MIT Sloan, who is the corresponding author; and Joshua L. Krieger PhD ’17, an assistant professor at Harvard Business School, who worked on the project while completing his doctorate at MIT.

First retraction, then decline

The study looks at a large group of scientists who published papers, from 1977 until 2007, that were later retracted, and compares the citation rates of all their published works to the citation rates of a related group of scientists who never had papers retracted.

There are 376 scientists in the study who have had papers retracted and have collectively authored a total of 23,630 published papers in their careers. The control group of scientists without retractions includes 759 authors with a total of 46,538 published papers to their names.

There are two large methodological keys to the study. First, the control group of scientists was built by identifying researchers who had published papers in the same journal issues, and on similar topics, as the papers that were later retracted. This allowed the MIT researchers to be confident that they were studying two groups of scientists who were largely similar and could be expected to generate similar citation rates for their work.

Second, the 10 percent decline in citations is specifically measured against the “normal” trajectory of citations over time experienced by authors who have never had retractions. That is, academic papers generally do have a typical decline in citations over time. But the citation rates of still-valid papers published by researchers who subsequently had retractions dropped by 10 percent when compared to the expected long-term decline in citations rates that already exists.

“If you look at the time trend of citations to these papers, nothing [unusual] is happening until there is a retraction,” Bonatti explains. “That’s exactly what we’re picking up. It’s not that these are old papers that are getting obsolete, and people are citing newer stuff.”

Instead, Bonatti adds, “Our data is consistent with a learning story.” That is, the scientific community is reacting to new information, in the form of retractions, and re-adjusts its view about the value of the body of work of certain scholars, based on that.

“The mighty fall further”

The pattern the researchers discovered doubles when there is clear scientific misconduct, which is often announced by the journal making the retraction. As the researchers found, misconduct produces a subset of cases where the citation rates of other papers drop 20 percent, among scientists who had been among the top quartile of their peers in terms of citations.

“Once you’re looking at retractions that involve misconduct, those are pretty good signs that something bad happened,” Bonatti observes. “So when the signal is very clear, it doesn’t matter how famous you were to begin with, you’re going to be discredited. … The mighty fall further, because they were standing taller to begin with.”

The current study builds on prior work by Azoulay and Krieger, who in 2014 published a paper, along with two other co-authors, showing that retracted papers produced a 6 percent decline in citations for nonretracted papers in the same subfield of research.

As the scholars acknowledge, the new paper leaves open a number of related questions about retraction policy and the effects of retractions. Many journals publish retractions that present ambiguity about the reasons for their action, leaving outside observers unclear about the precise nature of the problem.

Moreover, as the authors write, they still “cannot say anything definitive” about the theoretical gains researchers receive from publishing dubious work; the study of retractions depends on suspect work being identified. It thus remains unclear how frequently scientists may be getting away with work that deserves retraction.

The research was supported, in part, by the National Science Foundation and the Sloan Foundation.