Reasons to operate: the ‘wobbly tripod’ of evidence

"When good evidence is lacking, the best and most dedicated of us do wrong in the utter conviction of being right"

(L. Eisenberg 1977, NEJM, 297, 1230-2)

When surgical practice varies between doctors, hospitals, regions and countries (and it does), there are many possible reasons, apart from variations in the disease or the population. Some reasons that have been given for practice variation in surgery are:

•       Differing rates of insurance (example)

•       Increased uptake of imaging (MRI and back fusion)

•       Financial incentives (see earlier blog post)

•       Newer devices and techniques lowering risk and threshold

•       New indications

•       Supply (of surgeons) driving demand (here and here)

•       Surgeon preference (like the enthusiasm hypothesis, and here)

•       Surgeon experience (here)

•       Patient demands

These reasons may play a varying role in different circumstances, but they miss the big picture. Even if the reasons stated are true, I don’t think that individual surgeons are necessarily conscious of them. There may be some rationalisation involved, but I believe the main reason why surgeons perform unproven surgery is because they think it works.

There are always going to be unscrupulous practitioners in any field, but surgeons probably operate because the prevailing belief is that it probably works. This is particularly the case where the evidence may not be all that clear (like the topics covered in other blog posts). However, this leads to the next question: why do they think it works when it has not been proven – why is the default position one that assumes effectiveness?

Firstly, there may be a lack of awareness of, or poor understanding of, the scientific evidence. Either the surgeon does not actively seek the best evidence, or does not have the skills to properly appraise the evidence. Worse, occasionally a surgeon will have an unscientific or cynical and dismissive approach to the available evidence.

More likely, the surgeon is relying on weaker evidence. There is considerable low level evidence (and therefore a high risk of bias) available in surgery: industry sponsored meetings and advertising; conversations with other surgeons; published reports of case series of patients; personal observation of their own patients (seeing them get better and imputing cause and effect), and finally, and probably worst of all, tradition.

For “real” evidence, surgeons often rely on (what I call) the “wobbly tripod” of surgical evidence.

Many of my posts on procedures (example) refer to the supporting evidence, and how that evidence can dissolve away when more robust, conflicting evidence is produced. Because this is a recurring theme, it needs a name: the ‘wobbly tripod’ supporting surgical practice.

The three legs of the tripod are:

1. Biological plausibility

2. Related evidence from lab / animal studies

3. Personal observation (case series)

Why are these so unstable (wobbly)?

1. The biological plausible mechanism

The problem with this is that plausibility is a requirement; it does not supply its own proof of causality; it merely provides a possible explanation (which may direct future research). Therefore, it is more important in its absence, than its presence.

2. Related evidence from laboratory and animal studies

Part of the problem with this is the presumed accuracy of such data. There is an assumption that experiments from a laboratory are free from the bias often associated with clinical studies; that they are somehow more precise. Anyone who has done these sorts of studies knows that this is not the case. Also, they are a surrogate outcome for what we are really interested in, and the problem with surrogate outcomes is legendary (here). In short, the gap between bench and bedside is wide.

3. Personal observation

Assigning cause-and-effect to an observed association has driven human decision-making since there were humans. It is a very strong driver of behaviour and is probably the main reason why we have difficulty changing a practice that already fits our worldview, on account of someone else’s numbers (see previous post). The problem with personal observation is that it is unreliable: it is very susceptible to bias. We tend to see what we want to see. The reasons are many, but to keep it short: there are many other reasons why patients might improve, and why we might perceive them as improved, and our brains are hard-wired to attribute cause and effect to any association we see, and to not look any further for alternative explanations.

Obviously this doesn’t apply to all surgical practice, otherwise I would be out of a job. But if the wobbly tripod is the only thing supporting a procedure that you are performing or having performed on you, then you might want to look into that evidence with a critical eye; much of the evidence breaks down under questioning.