Statistical Evidence. Chapter 3.  Mountain or Molehill? The Clinical Importance of the Results.

3.0 Introduction

Do the research results add up to something important or are the results trivial?  For the results to be important, the study needs to have a narrow focus, it has to measure the right outcomes, and the change in the outcome has to be large from a clinical perspective.

Case Study: Side Effects of Vaccination

A pair of articles on vaccination that appeared next to each other in a 1999 issue of BMJ (Karvonen 1999; Henderson 1999) offer an interesting contrast in reporting styles. I commented about this on the BMJ webpages (Simon 1999).

Both studies used a cohort design to examine side effects of vaccination. In the first article, the authors compared the rate of Type I diabetes among children vaccinated with Haemophilus influenzae type b at the age of three months to children vaccinated at the age of 24 months. They reported the relative risk as 1.06 (p=0.54). In the second study, the authors compared the risk of intermittent wheezing between children with and without pertussis vaccine. They also reported the relative risk as 1.06 (95% CI: 0.81 to 1.37).

Both studies are negative, but the second study tells you something extra. In that study, you know that even after allowing for sampling error, there is no justification for believing that the risk of side effects could be increased by 50%. You know this because the relative risk of 1.5 lies outside the confidence interval. With the first study, you are left wondering. That looks like a small relative risk, but is it possible that sampling error would allow for a 50% or 100% increase in risk? You'd have to calculate the confidence interval for yourself to be sure.

Since you've been such a good reader, I'll save you the trouble. The 95% confidence interval for the relative risk in the first paper is 0.88 to 1.28. So you can rule out a large change in risk in this paper as well.

Unfortunately, neither paper reported a measure of absolute risk. With a bit of effort, you can calculate these values yourself. In the first paper, the number needed to harm for Type I diabetes is 4,500 (95% CI: 1,100 to infinity). You would expect one case of diabetes for every 4,500 children vaccinated. In the second paper, the number needed to harm for intermittent wheezing is 109 (95% CI: 37 to infinity). You would expect one case of intermittent wheezing for every 109 children vaccinated.

Why is this important? Because you need to know what the best course of action is with respect to these vaccinations. If there is a large risk that outweighs the benefit of the vaccination, you should stop vaccinating your patients. Even if the risk does not outweigh the benefit, if it is large enough, you should warn people about the side effect.

Notice that I did not define "large" here. How much of an increase in side effect risk is large? It's an easier question to ask rather than answer, but in the case of vaccines, the answer is especially difficult. What disease is the vaccine trying to prevent? How much more prevalent would that disease become if people stopped using the vaccine? Is the disease life threatening? How serious is the side effect?

These are complicated questions, but they are questions that you have to ask if you want to assess whether the research findings add up to a mountain or if they are just an unimportant molehill.

Mountain or Molehill? What to Look For.

Make sure that any research study measures something of practical importance.

Did they measure the right thing?  Researchers should focus on outcomes of interest to the patient and long term rather than short term outcomes. Examining multiple outcome measures or multiple subgroups will dilute the quality and strength of the evidence.

Did they measure it well?  Certain types of measurements have a lower strength of evidence. Be cautious about measurements that are retrospective because memory is imperfect. Unblinded measurements can allow your patients' expectations to influence the outcome.  Don't trust unvalidated/unreliable measurements or post hoc changes in the protocol.

Were the changes clinically important?  With a large enough sample size, a difference between two groups that is statistically significant might represent a change so small as to be clinically trivial. Specify a clinically important change for a study by asking how much of a change would be needed to convince you to adopt a new treatment or therapy. For negative trials, look for a precise confidence interval or a justification of the sample size that was conducted prior to data collection.

[Remainder of this material deleted out of respect for the publisher's copyright.]

This work is licensed under a Creative Commons Attribution 3.0 United States License. It was written by Steve Simon on 2005-06-10, edited by Steve Simon, and was last modified on 2017-06-15. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Statistical evidence