How Accurate are Medical Research Findings?

Most people have set pragmatism as their default position on bioethics. If it works, why not use it? If human embryonic stem cells are reported to be effective, for instance, what harm can there possibly be in using them? In fact, it may be immoral not to use them after the incredible progress reported in this week’s issue of Nature (or Time or the National Inquirer)!

But in an era of science by press release, pragmatists should know how reliable such reports are. And respected studies into the credibility of all medical research – not just on stem cells – suggest that claims of incredible advances are precisely that: incredible. In fact, according to a leading medical statistician, Greek academic John Ioannides, “most claimed research findings are false”.

Dr Ioannides is not a crank or an enemy of science. On the contrary, his work has been published in leading journals and his claims are widely accepted among his colleagues. He has worked at Harvard University, Tufts University and Johns Hopkins University. His ground-breaking 2005 paper in the journal PLoS Medicine has become the most downloaded in its history. Every year he receives hundreds of invitations to speak at conferences. “You can question some of the details of John’s calculations, but it’s hard to argue that the essential ideas aren’t absolutely correct,” Doug Altman, the director of Oxford’s Centre for Statistics in Medicine, told Atlantic Monthly.

Ioannides’s claims are largely statistical and thus require much brain cudgelling for laymen. But his conclusions ought to rattle anyone: that “most research findings are false for most research designs and for most fields” and “claimed research findings may often be simply accurate measures of the prevailing bias”.

Why is this?

There are a number of interlocking reasons. Many studies are too small to be reliable. The best ones involve several thousand subjects, but many studies, especially in genetics, are based on fewer than a hundred. Many studies are badly designed or are hard to compare to other studies of similar data.

Prejudice plays a role as well. It’s not necessarily ideological or financial; old-fashioned chest-beating, turf-protecting arrogance is just as effective. Scientists who are committed to a theory are less likely to find contradictory evidence. “Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure… Prestigious investigators may suppress via the peer review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma,” wrote Ioannides in his 2005 PLoS article.

And finally, “The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.” Ioannides attributes this counter-intuitive effect to cutthroat competition among scientists to publish exciting research first. “This may explain why we occasionally see major excitement followed rapidly by severe disappointments in fields that draw wide attention,” he says. Isn’t this relevant to far-reaching claims made for embryonic stem cells?

Even more discouraging for medical researchers is that the gold-standard of medical research, double-blind randomised trials, are not altogether reliable either. In another 2005 paper published in the Journal of the American Medical Association, Ioannides examined 49 of the top science papers of the previous 13 years. They had appeared in the best journals and had been cited extensively. Yet between one-third and one-half of them were unreliable because they were later found to be either outright wrong or exaggerated.

None of this means that all scientists do shoddy work or that science itself is fatally flawed. Science is a slow slog towards the truth whose milestones are false intuitions and failed experiments.

But it does mean that politicians and voters ought to be wary of early findings until they are repeatedly confirmed by other researchers. Unfortunately, this is a process that may take years to work itself out – far too slow for journalists who are searching for sound-bites. But unless they convey the ever-tentative nature of progress in science, they are deceiving their readers. As science journalist Joann Rodgers, of Johns Hopkins University, says: “Part of the responsibility for publicly communicating science is to help the public understand that scientific truth is a journey.”

Although Ioannides’s analysis is widely accepted, some researchers fear that they might be misinterpreted and used to debunk science or to promote shonky alternative therapies. But he responds that the truth is the best medicine: “The scientific enterprise is probably the most fantastic achievement in human history, but that doesn’t mean we have a right to overstate what we’re accomplishing.” Sound advice for every pragmatist!
Michael Cook is editor of MercatorNet. From here.