Trends, forests and trees

How many times have you seen the results of a new way of doing things, where the results after the introduction were shown to be better than they were before? These ‘before-and-after’ studies almost always show an improvement, but does that mean that the improvement was caused by the intervention? Given the example I will show you below, you should conclude that our highly evolved tendency to read cause-and-effect into any association often runs counter to reality.

Mandatory seat belts and motorcycle helmets were introduced in my country in 1973. The law was strongly advertised and enforced, and there was very little protest; I guess we saw it as a no brainer. Later, we were presented with some compelling before-and-after data on road mortality, with a 34% reduction in road deaths between 1970 and 1977 (here). No one argued; this was due to seat belt legislation.

Post hoc ergo propter hoc

Before-and-after studies can lead us to a logical fallacy, that of ‘post hoc ergo propter hoc’: it follows, therefore it is because of. Most of you are aware of confounding, whereby other factors may be responsible for the improvement. Things like traffic lights, divided roads, better barriers, road markings and signage, public driver education campaigns, stricter licencing requirements, higher penalties for drink-driving, random breath testing, introduction of the demerit points system, improved vehicles (brakes, tyres, lights, indicators, glazing, head restraints and impact resistance), speed cameras, changes to buses and heavy vehicles (speed limiting, roll-over resistance and occupant protection) and about a dozen other things.

It is possible that we are not seeing the forest for the trees. Presented below is the decline in road deaths in Australia (from the Australian Bureau of Statistics), but over a much longer time period. Look at 1973, when seat belts and motorcycle helmets were made mandatory.

Do you see a dip? Maybe.

Other interesting data is available. The introduction of seat belts has been associated with an increase in pedestrian and bicyclist deaths. The lack of evidence for seat belts is also mentioned in this Lancet article where it is discussed as part of the risk compensation effect. More on that in a future blog post.

I am not saying that seat belts don’t’ save lives – there are probably many examples of when they have. But are seat belt laws responsible for our current low mortality rates? My opinion is that they play a very minor role, and have probably done more to prevent facial fractures and save windscreens than save lives. I say that because the road fatalities fell at a similar or greater rate over the same time period in countries that did not introduce seat belt laws (same Lancet article), because the rate was falling at the same rate before they were introduced, and because of all the other possible contributing factors mentioned above.

Put it this way: I am very confident that seat belt laws were NOT responsible for the 34% decline between 1970 and 1977, reported above. Bigger drops have been reported over shorter time periods since, and previously.

The bottom line

There are many reasons why things actually get better, and why they might be perceived as getting better, so don’t jump to conclusions about causation when you see a before-and-after study. And the next time you are presented with data showing how things have improved since the introduction of [INSERT INTERVENTION HERE], ask for the long term trends, look for other explanations, and just for fun, suggest that things might continue to improve, even if we stop the intervention.