Medical Studies: For Entertainment Purposes Only?
The Atlantic Monthly has published an interesting article in it's November issue: Lies, Damned Lies, and Medical Science. It's part of a series called Brave Thinkers, 2010 and it features an interview with Dr. John Ioannidis, who leads a team that specializes in critiquing medical studies, both qualitatively and quantitatively.
“Maybe sometimes it’s the questions that are biased, not the answers,” he said, flashing a friendly smile...
That question has been central to Ioannidis’s career. He’s what’s known as a meta-researcher, and he’s become one of the world’s foremost experts on the credibility of medical research. He and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed.
We've discussed poorly designed studies before on Big Fat Blog. Studies and surveys in the social sciences are especially sensitive to design bias and data manipulation, and medical studies aren't far behind. They may have biases built into their design, and their data may be manipulated and interpreted to support the conclusion that will make continued funding most likely. Glenn Gaesser's Big Fat Lies and Paul Campos's The Obesity Myth (or the Diet Myth) do an excellent job of exploring how this has happened in the study of obesity, in particular. The dodginess of obesity research is referred to somewhat obliquely in the Atlantic article as well:
On the relatively rare occasions when a study does go on long enough to track mortality, the findings frequently upend those of the shorter studies. (For example, though the vast majority of studies of overweight individuals link excess weight to ill health, the longest of them haven’t convincingly shown that overweight people are likely to die sooner, and a few of them have seemingly demonstrated that moderately overweight people are likely to live longer.) And these problems are aside from ubiquitous measurement errors (for example, people habitually misreport their diets in studies), routine misanalysis (researchers rely on complex software capable of juggling results in ways they don’t always understand), and the less common, but serious, problem of outright fraud (which has been revealed, in confidential surveys, to be much more widespread than scientists like to acknowledge).
Now, Dr. Ioannidis's assertion that up to 90% of medical studies produce flawed conclusions seems very dire indeed, and I'm not sure what to think about it. I wonder about the details of his methods and about the scope of the analysis. Is he talking about a particularly error-prone subset of studies? Where does he draw the line between medical research and hard science?
I like data and believe that collecting and analyzing it is useful. I believe that the scientific method - hypothesis to experimentation and observation to theory- is the best way we have of understanding the universe, our world, and our biology. But Dr. Ioannidis's work really illuminates the way science can be warped by economics, by biased assumptions and methods, and by individual egos.
When journalists cover scientific studies they usually serve them up uncritically, based on press releases, often with a dash of sensationalism. And, they bring their biases to the table as well. Qualified scientific and medical journalists that look deeper, like Sandy Szwarc of Junkfood Science (currently on hiatus, but still well worth exploring) are rare and valuable. Science, especially medical science as reflected in the press, is far from infallible. It is always wise to look closely at the studies that concern you most, and at their context.
...and here's a great post on how to do the best you can with nothing but a news article to work with. Thanks, MichMurphy!