This story is about a procedure that made sense and had supporting evidence, became common practice, but was later discontinued because a high quality study showed it to be ineffective. The story of extra-cranial intra-cranial bypass surgery ticks all the boxes: overestimation of benefit, seduction by the theory, unrecognised bias in studies, and just plain ineffectiveness despite our best effort and beliefs.
The idea is appealing: if an intracranial (inside the skull) artery to the brain is clogged and causing strokes, then bringing in blood from a good blood vessel (extracranial – outside the skull) to the brain artery to bypass the narrowing will prevent future problems. This extracranial to intracranial (ECIC) bypass was first done in the 1960s and became an established procedure, but not based on any randomised trials comparing the procedure to non-operative treatment.
The randomised trail (RCT) published in 1985 (here) showed that surgery did a little worse than non-operative treatment. The trial started in 1977 and was of a surprisingly high standard, particularly for that time. For example, it used concealed, central randomisation, a valid sample size calculation, clinically important outcomes, a large number of participants (for a surgical study), 100% follow up, only about 2% crossover each way (again, good for a surgical trial), and long term follow up (average 5 years). And importantly, the surgical procedure worked – the surgical complication rate was low and they had a 96% graft patency rate, so the poor results were not due to a failure to achieve the goal of bypass surgery.
Fatal and non-fatal strokes occurred more frequently and earlier in the surgical group. The results were enough to reject the hypothesis that surgery improved the outcomes. The difference can be seen in the graph below.
They also played around with the analyses, adding and subtracting ineligible patients, looking at severe strokes only, type of strokes, strokes per person, size of the treating centre, the artery involved, etc.. In every analysis, there was no benefit to surgery. They even did an analysis excluding patients who had strokes prior to surgery (a best case scenario, biased to make surgery look good), and there was still no benefit.
So how do we explain the previous “successes”?
Simply, the previous studies were biased. They used surrogate endpoints, like cerebral blood flow, cerebral metabolism and EEG tracings. The RCT used clinically important endpoints (stroke and death). The previous studies did not have an adequate (concurrent, randomised) control group, they used “historical” controls. They also relied on variations in the natural history, like transient ischaemic attacks (TIAs). For example, one previous (uncontrolled) study showed an 86% benefit from surgery. The RCT showed a similar high rate of reduction in TIAs of 77%, but the reduction was 80% in the non-operative group, which highlights the need for a control group.
Basically, the randomised trial was a more objective, less biased test, and therefore more likely to provide an estimate of the effect that was closer to the true effect. The previous studies were more likely to show a positive effect, even when there wasn’t one. This is partly because that is what the researchers wanted to and expected to find, and this influenced their interpretation of the data.
The bottom line
The ECIC bypass story is a great example of how much bias is out there in medical studies; bias that leads us to overestimate the effectiveness of the treatment. But it is also a good example of how a more rigorous application of scientific principles leads us to a better estimate of the truth and consequently better treatment, while simultaneously revealing biases that were previously unrecognised.