Even Scientists Are Fooled by a Really Good Story

If you want to know what the current thinking is in a scientific field, the best place to start is not a textbook but a literature review.

As its name implies, a literature review is an attempt by an author (or several authors) to summarize the state of research, often with the purpose of making a particular argument or arriving at something resembling a "consensus." The authors look over dozens or even hundreds of papers and then write a summary based on their findings. For example, back in graduate school, I co-authored a review that examined the role of the microbiome in causing gum disease and tooth loss (periodontitis).

The type of review I wrote is known as a narrative review. But these can be problematic. A reviewer may have an unconscious bias, or he may "cherry-pick" papers that support a particular conclusion. In an effort to combat this, the systematic review is becoming more common. In a systematic review, a protocol that defines in advance which papers to include and exclude is applied to the scientific literature. Then, all the papers that meet the criteria are included in the review. The following table by Rossella Ferrari in the journal Medical Writing highlights the differences between narrative and systematic reviews.

Source: R. Ferrari, Medical Writing, 2015.

Recently, a team of researchers wanted to determine if narrative and systematic reviews produced a difference in the minds of the scientists and doctors who read them. So, they performed a randomized trial on 111 experts in the field of colorectal cancer who were asked to assess the risks and benefits of using a drug (called tinzaparin) on colon cancer patients. Fifty-five read the narrative review, and 56 read the systematic review.

Just as the authors predicted, there was a significant difference between how these experts assessed the drug. Specifically, those who read the narrative review were much more optimistic about the potential benefits of the drug and were likelier to favor clinical trials with the drug than those who read the systematic review.

This paper has two clear implications: (1) Narrative reviews may be intentionally or unintentionally biased; and (2) The audience may be biased by narrative reviews. As it turns out, even scientists can be fooled by a really good story.

Source: Yu M, Montroy J, Fergusson D, Lalu M, Kimmelman J. "Systematic and narrative review lead experts to different cancer trial predictions: a randomized trial." Journal of Clinical Epidemiology (2021). DOI: 10.1016/j.jclinepi.2020.12.006.