Don’t believe the hype, or the research

I spend a lot of time telling people to look for the rational, scientific evidence and to try to avoid the hype from marketers and doctors. But it seems that you can’t always believe the scientific evidence either, judging from quotes like this one from Marcia Angell, the former editor of the most respected and highly ranked medical journal in the world – the New England Journal of Medicine.

“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine”

Scientific articles about how scientific articles are wrong make it even more confusing (see Why most published research findings are false).

Interestingly,  false findings are not usually due to out-and-out fraud (although there are numerous examples of this) but simply due to the usual biases referred to in this blog. We start with the bias of assuming that the treatment will work (based on lab tests, biological plausibility, and our perception of other treatments), we then add our desire (enthusiasm) for it to work, mix it with the biases of the people working on different aspects of the project, then make multiple minor and major, slightly biased decisions along the course of the research. Finally, we submit it to journal editors with their own biases and presto: great results get published. Later publications from less enthusiastic authors are usually less glowing. This is known as the Decline Effect, examples of which are described in Jonah Lehrer’s New Yorker article, The Truth Wears Off. Unfortunately, he is not a scientist and his title is wrong, even though what he is describing (the decline effect) is real. The truth doesn’t wear off: it is the one constant that we are trying to measure in scientific studies. Our initial estimate of it is often biased towards showing the treatment to be better than it is, and better studies get us closer to it, but in general, the truth doesn't change. A more detailed critique of the article can be found here.

So what should we believe? We should believe the most scientifically valid evidence: the most logical and unbiased evidence, because that is the evidence that will provide the best estimate of to the truth. That however, requires an understanding of the scientific method, logical principles, the techniques of critical appraisal, and the time that is needed to apply them. Not something that most patients or busy doctors have. Just because there are bad examples of science out there, it doesn’t mean that the scientific principles are bad. We must remain skeptical (scientific) without being dismissive (cynical).

I think Marica Angell is an example of what George Bernard Shaw was talking about when he said:

“Every person who has mastered a profession is a skeptic concerning it.”