The parachute analogy

“You don’t need a randomised trial to prove that parachutes work” is a common refrain from doctors who believe that their operation works, despite a lack of good scientific evidence from controlled trials. They see it as an argument-stopper, but it is actually an example of argument by analogy, and a poor one at that. And at least one surgeon has been burned by putting the parachute analogy in writing.

 Arguing by analogy is common, but it can be overdone. The strength of the argument lies in the similarities between the two situations, and there are very few similarities between jumping out of a plane with or without a parachute, and surgery. Unless the risk of death without surgery is immediate and high (about 100%), and the risk of death with surgery is very low (less than 1%), then yes, you don’t need a randomised trial to convince me, just prove those numbers to me and you have won the argument.

A better analogy would involve pointing out that every surgical procedure that has been compared to sham surgery has been shown to be ineffective, and that treatments subjected to randomised trails usually show much weaker benefits than previously thought from less rigorous studies (see these, previous, posts, here, and here).

The example I will use comes from Dr J Cooper, a thoracic surgeon who in 1995 published good results in a series of patients who had undergone a new operation for emphysema, which involved removing 20-30% of each lung (LVRS: lung volume reduction surgery).

The operation became popular and was being done fairly widely, but due to doubts (from others) about the effectiveness a large randomised trial to compare LVRS to medical therapy (the NETT trial) was proposed, and funded by Medicare, the National Institute of Health and the Agency for Healthcare Research and Quality.

Dr Cooper then pulled out the parachute analogy in an editorial from 2001 in which he sees the evidence of benefit from LVRS as overwhelming, and rejects the need for the NETT trial.

In fact, Dr Cooper was still publishing great results from (uncontrolled) case series in 2003 when the results of the NETT trial were published in the New England Journal of Medicine. There was no difference in the rate of death between the two groups at two years. The topic has been covered in a Cochrane review from 2009 where they also note that there is no difference in deaths at 2 years. There is an improvement in lung function and quality of life over that period, but this can be partly explained by the fact that surgery tends to weed out the sicker patients early: the death rate by two years might be the same, but your odds of dying within 90 days is more than 6 times higher if you have surgery.

The arguments persist, mainly looking at what patients might benefit most and what patients do worse, in order to refine the indications for surgery. It sounds a bit like Desperately Seeking Subgroups (an upcoming post on this site) but to be fair, that is the discussion we should be having – we should not be attempting to shut down the debate by using inappropriate arguments by analogy.

Addit: 1 July 2012: See the Healthcare etc. blog post on the parachute analogy here