StATS: Where is the confidence interval? (created 2005-03-31).
A recent letter to the editor,
complains about an article claiming that a drug, citalopram, can reduce depressive symptoms
The letter writers dispute (among other things) the claim of a statistically and clinically significant reduction. In the original paper, the authors show several results, and the one that is perhaps the most important is the proportion of patients who score 28 or less on the Children's Depression Rating Scale. By this criteria, 36% of the treated patients and 24% of the control patients showed improvement.
One way to see if the results of a study are clinically significant is to present a number needed to treat plus confidence limits. The difference in percentages is 12% and when you invert it, you get a value of NNT=8.1. This means that you have to treat 8 patients with the drug in order to see one additional improvement over placebo.
Is an NNT of 8 clinically significant? I can't answer this question because I am not a psychiatrist. I will point out that you have to balance the costs of the treatment (economic costs, the inconveniences caused by having to take the medication, and the side effects that these patients will have to endure) against the benefits.
The thing to keep in mind is that all eight patients pay the costs of the drugs to accrue benefits for a single patient. If the benefit of pushing one additional patient below 28 on the Children's Depression Scale is more than 8 times as valuable as the costs that each patient endures, then a change from 24% to 36% is worthwhile from a clinical viewpoint.
Another interesting thing to consider though is the confidence interval. The original paper presents no confidence intervals, and none are mentioned the reply to the letter.
The CONSORT guidelines for reporting of randomized clinical trials recommends the use of confidence intervals for presenting research results. There are 175 journals that have endorsed the CONSORT guidelines, but not the American Journal of Psychiatry.
A crude calculation for the 95% CI gives a NNT that could be as small as 4 or as large as infinity. An infinite NNT is equivalent to no effect whatsoever.
Perhaps the authors used a more sophisticated method to determine statistical significance, but even then, I suspect that the upper limit for the confidence interval for NNT would be in the hundreds or thousands.
I shouldn't pick on these particular authors. There is a general problem with reporting confidence intervals
and discussing clinical importance.
This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Clinical importance or Category: Confidence intervals.