StATS: No tolerance for ambiguity (created 2006-05-10).
I was at a meeting tonight and put in a plug for my book, Statistical Evidence in Medical Trials, by mentioning that it was intended to help people understand the controversies and the seemingly contradictory research that appears in the medical journals. I went on to give an example: hormone replacement therapy for post-menopausal women. It's a good example, because half of the people in the audience have either had to or will have to make a decision about whether they should take estrogen supplements. The other example which I like to cite is whether men should consider taking a test to look at their PSA levels to try to detect prostate cancer.
Anyway, a woman talked to me afterwards and wanted to know what I thought about a particular author who had written about hormone replacement therapy. I had to defer any comments because I was unfamiliar with this particular author. She then informed me that she had taken hormone replacement therapy and it gave her breast cancer. Thankfully, the cancer has responded well to treatment, but I was struck by the certainty of her comment about how the estrogen supplements caused her cancer.
The lack of uncertainty bothered me, and it highlights an important thing to remember about research in general and Statistics in particular. Statisticians describe the behavior of groups and often are unable to make specific and precise statements about a particular individual. That is perhaps a bit of an oversimplification, but this is a general concept worth remembering. There is a great Sherlock Holmes quote that speaks to this issue:
You can, for example, never foretell what any one man will do, but you can say with precision what an average number will be up to. Sir Arthur Conan Doyle The Sign of Four (1890), as quoted at www.ewartshaw.co.uk/data/jehsquot.pdf.
What does the research about hormone replacement therapy say? Here's the abstract from the 2002 JAMA study that is considered by many to be the definitive result:
CONTEXT: Despite decades of accumulated observational evidence, the balance of risks and benefits for hormone use in healthy postmenopausal women remains uncertain. OBJECTIVE: To assess the major health benefits and risks of the most commonly used combined hormone preparation in the United States. DESIGN: Estrogen plus progestin component of the Women's Health Initiative, a randomized controlled primary prevention trial (planned duration, 8.5 years) in which 16608 postmenopausal women aged 50-79 years with an intact uterus at baseline were recruited by 40 US clinical centers in 1993-1998. INTERVENTIONS: Participants received conjugated equine estrogens, 0.625 mg/d, plus medroxyprogesterone acetate, 2.5 mg/d, in 1 tablet (n = 8506) or placebo (n = 8102). MAIN OUTCOMES MEASURES: The primary outcome was coronary heart disease (CHD) (nonfatal myocardial infarction and CHD death), with invasive breast cancer as the primary adverse outcome. A global index summarizing the balance of risks and benefits included the 2 primary outcomes plus stroke, pulmonary embolism (PE), endometrial cancer, colorectal cancer, hip fracture, and death due to other causes. RESULTS: On May 31, 2002, after a mean of 5.2 years of follow-up, the data and safety monitoring board recommended stopping the trial of estrogen plus progestin vs placebo because the test statistic for invasive breast cancer exceeded the stopping boundary for this adverse effect and the global index statistic supported risks exceeding benefits. This report includes data on the major clinical outcomes through April 30, 2002. Estimated hazard ratios (HRs) (nominal 95% confidence intervals [CIs]) were as follows: CHD, 1.29 (1.02-1.63) with 286 cases; breast cancer, 1.26 (1.00-1.59) with 290 cases; stroke, 1.41 (1.07-1.85) with 212 cases; PE, 2.13 (1.39-3.25) with 101 cases; colorectal cancer, 0.63 (0.43-0.92) with 112 cases; endometrial cancer, 0.83 (0.47-1.47) with 47 cases; hip fracture, 0.66 (0.45-0.98) with 106 cases; and death due to other causes, 0.92 (0.74-1.14) with 331 cases. Corresponding HRs (nominal 95% CIs) for composite outcomes were 1.22 (1.09-1.36) for total cardiovascular disease (arterial and venous disease), 1.03 (0.90-1.17) for total cancer, 0.76 (0.69-0.85) for combined fractures, 0.98 (0.82-1.18) for total mortality, and 1.15 (1.03-1.28) for the global index. Absolute excess risks per 10 000 person-years attributable to estrogen plus progestin were 7 more CHD events, 8 more strokes, 8 more PEs, and 8 more invasive breast cancers, while absolute risk reductions per 10 000 person-years were 6 fewer colorectal cancers and 5 fewer hip fractures. The absolute excess risk of events included in the global index was 19 per 10 000 person-years. CONCLUSIONS: Overall health risks exceeded benefits from use of combined estrogen plus progestin for an average 5.2-year follow-up among healthy postmenopausal US women. All-cause mortality was not affected during the trial. The risk-benefit profile found in this trial is not consistent with the requirements for a viable intervention for primary prevention of chronic diseases, and the results indicate that this regimen should not be initiated or continued for primary prevention of CHD. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial. JE Rossouw et al. Jama 2002: 288(3); 321-33. [Medline] [Abstract] [Full text] [PDF]
This research did not show that if you take estrogen supplements, you will get breast cancer. If you read the full article, you will find that of the 8,506 women on hormone replacement therapy, 166 developed invasive breast cancer. Of the 8,102 women who took placebo, 124 developed invasive breast cancer. So for the vast majority of women, nothing happened, even after five years of follow-up on average.
Among the women where something bad happened, it happened more often in the active treatment group. Now you can't just take the ratio of 166 / 8506 or 124 / 8102 to get a risk of breast cancer, since the women were followed for a variable amount of time, but the more complex statistics tell pretty much the same story.
There are multiple outcomes in this study and when you look at the big picture, you find out that for the average woman, the probability of all the beneficial effects was more than offset by the probability of all the detrimental effects. But in all honestly, the average woman in the study did not have anything bad happen to her, so the average women is healthy with or without hormone supplements.
If a smoker who dies at age 50 from lung cancer can say that smoking caused his/her early death, then can the rare smoker who lives to age 90 claim that smoking was responsible for that longevity?
Suppose in a randomized trial, the first nine patients to get Drug A died and the last one survived. Among those who got Drug B, the first nine patients lived, and the last one died. It sounds like Drug B is a lot better, but if you were the last patient recruited to the study, you were better off with Drug A. In other words, you're probably better off with Drug B, but we can't guarantee that everyone who takes Drug B will have a better outcome than if they took Drug A.
It is human nature, perhaps, but anytime a tragedy befalls us, we have a strong need to find an absolute cause or an explanation. But in the world of statistics there is no absolute cause, only odds and probabilities. It is a cold comfort, perhaps, to speak about chances and tendencies, but it is also more intellectually honest. There's a t-shirt that they sell every year at the Joint Statistics Meetings that proclaims in bold letters on the front "Being a statistician means never having to say you're certain."
By the way, the website where I found the Arthur Conan Doyle quote from is a treasure trove of good quotes. Here is another one:
Bayesian statistics is difficult in the sense that thinking is difficult. Donald A. Berry Teaching Elementary Bayesian Statistics with Real Applications in Science, American Statistician 51:241–246 (1997), as quoted at www.ewartshaw.co.uk/data/jehsquot.pdf.
This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Critical appraisal.