ACSH has a long history of balancing hype from various sources with evidence-based science, especially when it comes to countering sensationalized studies that rely on questionable methodologies. Though prominent news outlets constantly report on observational studies as though their conclusions were irrefutable the same type of studies that claim to link diet soda to stroke or phthalates to obesity we were pleasantly surprised to encounter an article on the front page of last week s Wall Street Journal that finally shed light on this scientifically dubious method of research.
Observational studies use sophisticated computer software to analyze data previously collected by others. Unlike randomized controlled interventional studies the gold standard of medical research observational studies lack any human intervention; scientists merely draw inferences about the effects of treatments based on numbers obtained from a database. It s not uncommon that such analyses bend the rules of statistics in order to achieve the desired result, or what we at ACSH familiarly refer to as data-dredging allowing researchers to come up with statistically significant results by reanalyzing data from different perspectives.
To demonstrate how unreliable observational studies can be, take, for instance, the results from two U.K. research teams that both extracted data from the same database to determine whether a popular class of osteoporosis drugs called bisphosphonates increase the risk of esophageal cancer. While one group of researchers found only a small effect and concluded that the medicine was not significantly associated with the disease, the other group found a whopping 30 percent increase in esophageal cancer risk. Both studies were done by reputable authors and published in first-rank journals.
How, then, can two studies that rely on the same data arrive at such contradictory results? Well, according to ACSH advisor Dr. S. Stanley Young of the U.S. National Institute of Statistical Sciences, who was also quoted in the WSJ piece, observational studies are highly susceptible to statistical biases which can be unintentional, but which often produce unreliable findings. On the other hand, such data manipulations can be a short-cut to publication and headlines for researchers and institutions alike: You can troll the data, slicing and dicing it any way you want, Dr. Young says. A great deal of irresponsible reporting of results is going on.
And, of course, the media just love to report on sensational studies when the findings are bound to cause an uproar, says ACSH s Dr. Gilbert Ross. Just because a study is published in a peer-reviewed journal and reported on in the media doesn t mean it s exempt from suspicion. Unfortunately, a good proportion of these will be found to not be replicable; however, the report of a study s reversal will often not make the news.
Indeed observational studies can be replicated only about 20 percent of the time (although Dr. Young believes this number may actually be even lower only 5 to 10 percent). Yet the popularity of observational studies among the scientific community is ever-increasing: Between 2001 and 2011, the number of these studies more than tripled to 263,557, according to an analysis by Thomson Reuters. So why continue to conduct such research if it s laden with so many confounding variables and inaccuracies? Well, for one, observational studies are less expensive to execute, and they take much less time to complete. And though they will never achieve the same statistical accuracy as randomized controlled trials, they can sometimes offer clues about certain health outcomes.
In the meantime, Dr. Ross reminds us that just because a study reports that two events are statistically associated, remember that these links are often the result of much data manipulation. At best, he says, such conclusions are merely suggestive; at worst, they are blatant fallacies.
The bottom line, says Dr. Young, is that you can get just about any answer you want with large observational data sets and this type of statistical modeling. The claims you worry about, when they are tested in randomized clinical trials, essentially never replicate.
