BBC News reports this morning about a study that indicates that Tamiflu has marginal effects on flu symptoms only cutting them by half a day. Very unlike the BBC was to also briefly mention that this contradicts a previously published review, funded by the pharma company, that found it cut hospital admission rates.
There is a possibility that the two studies have used different measures and interpretations of “clinically relevant flu” symptoms.
But it did make me think of something which is rife in studies on psychoactive medications.
Clinical vs statistical significance.
As researchers, we tend to focus on whether a difference is statistically significant or not. But clinicians, and service users, are interested in whether there is a clinically significant difference. In terms of application and well-being, it is the latter that is important.
Nearly all studies on psychoactive medications that claim that Pill A has had an effect on Disorder B have found statistically significant differences. However, when you look at the details of the measures, you find (1) the measures were there is a difference are actually on physical complaints, like sleep, and eating behaviour, measures of “proper” mood or anxiety or hallucinations show no differences, and can even go in the opposite direction; (2) the difference on global measures of functioning is usually around 1.5 points on a scale, such as “depression”, which is statistically significant, but not clinically significant; (3) non-significant differences tend not to be reported, so you don’t know whether 20 t-tests were conducted, 19 were non-significant and only the 1 significant one is being reported.
This is just expert sleight of hand, an illusion. It is not a genuine effect that actually affects an individual’s level of distress or functioning or well-being beyond getting a good night sleep (not something to sniff at, see my post on insomnia), and a decent meal.
So, the next time you hear about some magic bullet, ask yourself whether its effect is statistically or clinically relevant.