StATS: Top ten studies in EBM, Part 4 (September 14, 2006)

About a year ago, I wanted to give a talk on the ten studies that anyone who teaches EBM needs to know. These studies should be

• well known in the research community;
• actual research studies (as opposed to editorials); and
• illustrative of important issues in EBM.

My previous writings on the topic were a bit rambling,

• Stats: Ten research studies that anyone teaching EBM should be familiar with (January 17, 2005)
• Stats: Another top ten study in EBM (February 14, 2005)
• Stats: More on the top ten studies in EBM (February 28, 2005)

so I am going to try to organize things a bit better.

Study #1. Historical firsts in clinical trials. You can go way back if you want to, and the study of limes for British sailors (citation needed) is an interesting tale. I want to highlight the first published trial that formally used randomization:

Streptomycin treatment for pulmonary tuberculosis. Medical Research Council Streptomycin in Tuberculosis Trials Committee. BMJ 1948; ii: 769-782.

which showed that streptomycin was superior to bed rest. Actually, a randomized trial testing the efficacy of immunization against whooping cough was started first, but did not get reported until 1951. Two historical perspectives on the streptomycin trial appear in a special 1998 theme issue of the BMJ celebrating 50 years of progress in randomized trials:

• Controlled trials: the 1948 watershed. Doll R. British Medical Journal 1998: 317(7167); 1217-20.
• Use of randomisation in the Medical Research Council's clinical trial of streptomycin in pulmonary tuberculosis in the 1940s. Yoshioka A. British Medical Journal 1998: 317(7167); 1220-1223. [Medline] [Full text] [PDF]

Study #2. Spectrum bias. When you are evaluating a diagnostic test, you need a broad range of disease severity. If you only test people who have obvious manifestations of a disease versus patients who are obviously healthy, you are omitting a large number of patients in the middle of the spectrum. Such a situation occurs commonly in a case-control design, for example. This leads to estimates of sensitivity and specificity that are too good to be true, and this problem is called spectrum bias. This first paper that identified spectrum bias is

Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. Ransohoff DF, Feinstein AR. N Engl J Med 1978: 299(17); 926-30. [Medline] [Abstract]

Another early paper that discusses spectrum bias is

• Spectrum bias in the evaluation of diagnostic tests: lessons from the rapid dipstick test for urinary tract infection. Lachs MS, Nachamkin I, Edelstein PH, Goldman J, Feinstein AR, Schwartz JS. Ann Intern Med 1992: 117(2); 135-40. [Medline]

Spectrum bias is just one of several issues that you face when you make a critical evaluation of a research article on diagnosis. The STARD initiative promotes good quality reporting for these type of articles and will help you evaluate diagnostic tests properly.

• The STARD Initiative -- Towards Complete and Accurate Reporting of Studies on Diagnostic Accuracy. STARD Group. Accessed on 2003-07-29. www.consort-statement.org/stardstatement.htm

Study #3. Evaluating alternative medicine. There is a huge controversy over how to apply evidence based medicine to alternative and complementary medicine. A science fair project by a fourth grader, Emily Rosa, evaluated Therapeutic Touch (TT) and eventually end up being published in a major medical journal.

A close look at therapeutic touch. Rosa L, Rosa E, Sarner L, Barrett S. Jama 1998: 279(13); 1005-10. [Medline] [Abstract] [PDF]

Emily Rosa's experiment was very simple, perhaps too simple. If practitioners of Therapeutic Touch are able to manipulate energy fields, they must first be able to detect energy fields. She would hold her hand above either the left or right hand of the practitioner and ask him/her to tell which hand. The choice of hand was randomly determined by a coin flip. A screen with two holes in it prevented the practitioner from seeing what was going on.

Emily Rosa got 21 experienced practitioners to agree to the test. They were right only 44% of the time. Did this simple experiment disprove the healing power of TT? Perhaps not. TT is a complex intervention and this experiment only looked at a single aspect of it.

But this experiment gets at the heart of the claim by many in the alternative medicine community that medical research, especially randomized trials, are reductionist and fail to represent how alternative medicine is practiced in the real world. On the other hand, a simple experiment like this one raises the question: how can proponents of TT manipulate an energy field that they can't even detect at a 50% success rate?

A good balanced view of the controversy of the reductionist nature of clinical trials is

• Evaluating complementary medicine: methodological challenges of randomised controlled trials. S. Mason, P. Tovey, A. F. Long. Bmj 2002: 325(7368); 832-4. [Medline] [Full text] [PDF]

A more sharply skeptical view appears in

• Alternative medicine--the risks of untested and unregulated remedies. M. Angell, J. P. Kassirer. New England Journal of Medicine 1998: 339(12); 839-41.

where the authors write

It is time for the scientific community to stop giving alternative medicine a free ride. There cannot be two kinds of medicine - conventional and alternative. There is only medicine that has been adequately tested and medicine that has not, medicine that works and medicine that may or may not work.

Studies #4A and 4b. Placebo surgery trials. Two widely cited early randomized double blind trials for surgery are

• An evaluation of internal-mammary-artery ligation by a double-blind technic. Cobb LA, Thomas GI, Dillard DH, Merendino KA, Bruce RA. N Engl J Med. 1959 May 28;260(22):1115-8. [Medline]
• Comparison of internal mammary artery ligation and sham operation for angina pectoris. Dimond EG, Kittle CF, Crockett JE. Am J Cardiol. 1960 Apr;5:483-6. [Medline]

These trials examined a treatment for angina that involved cutting an artery that leads away from the heart to encourage more blood flow to the heart itself through the narrowed coronary arteries. While anecdotal evidence for this procedure accumulated in the 1950s, these two randomized double blind trials showed that the surgery was ineffective.

These two studies are frequently cited as rationale for use of placebos and blinding. See, for example,

• Deconstructing the placebo effect and finding the meaning response. Moerman DE, Jonas WB. Ann Intern Med 2002: 136(6); 471-6. [Medline] [Abstract] [Full text] [PDF]
• Removing bias in surgical trials. Johnson AG, Dixon JM. British Medical Journal 1997: 314(7085); 916-7. [Medline] [Full text]
• Problems of Randomized Controlled Trails (RCT) in Surgery. Lefering R, Neugebauer E, Published in the Proceedings of the International Conference on Nonrandomized Comparative Clinical Studies in Heidelberg, April 10 -11,1997. Accessed on 2003-06-30. www.symposion.com/nrccs/lefering.htm

Some additional placebo surgery trials include

• Placebo effect in surgery for Meniere's disease: three-year follow-up. Thomsen J, Bretlau P, Tos M, Johnsen NJ. Otolaryngol Head Neck Surg 1983: 91(2); 183-6. [Medline]
• Transplantation of embryonic dopamine neurons for severe Parkinson's disease. Freed CR, Greene PE, Breeze RE, Tsai WY, DuMouchel W, Kao R, Dillon S, Winfield H, Culver S, Trojanowski JQ, Eidelberg D, Fahn S. N Engl J Med 2001: 344(10); 710-9. [Medline]
• A controlled trial of arthroscopic surgery for osteoarthritis of the knee. Moseley JB, O'Malley K, Petersen NJ, Menke TJ, Brody BA, Kuykendall DH, Hollingsworth JC, Ashton CM, Wray NP. N Engl J Med 2002: 347(2); 81-8. [Medline]

Again, these trials showed no additional effectiveness of the therapies over a placebo surgery. A placebo surgery trial is highly controversial, because patients in the placebo arm undergo all of the risks of surgery, (side effects of anesthesia, increased chance for infection, side effects associated with subsequent antibiotic use, etc.).

Study #5. The danger in using a surrogate endpoint. Three drugs, encainide, flecainide, and moricizine, were thought to prevent heart attacks by suppressing asymptomatic ventricular arrhythmias (actually, ventricular premature depolarization or VPD), a condition associated with arrhythmic death. A large scale randomized trial, however, showed that rather than reducing your risk of arrhythmic death, these drugs actually increased the risk of death.

Preliminary report: effect of encainide and flecainide on mortality in a randomized trial of arrhythmia suppression after myocardial infarction. The Cardiac Arrhythmia Suppression Trial (CAST) Investigators. N Engl J Med 1989: 321(6); 406-12. [Medline]

In 1989, the researchers ended the two arms of the study associated with encainide and flecainide early because of a higher rate of arrhythmia deaths, of nonfatal cardiac arrests, and a higher rate of overall mortality. The study was then redesigned to compare only the third drug, moricizine, to placebo, with some changes in the entry criteria to enroll only the more seriously ill patients. This redesigned study, CAST-II, was also ended early, because of excessive cardiac deaths during the first two weeks of drug exposure, and a futility analysis. The futility analysis showed that the current data held out little hope that additional data would accumulate to the point where there would be evidence of long term survival.

Why did this happen. One possible explanation is that only a few people with VPD will die from the condition, limiting the effectiveness of these drugs, but all those who take these drugs are exposed to potential side effects. The CAST trials illustrate the important principle that a change in a surrogate outcome (reduction in VPD) does not always translate into a reduction in mortality. Here are some additional references for the CAST trials.

• Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. Echt DS, Liebson PR, Mitchell LB, Peters RW, Obias-Manno D, Barker AH, Arensberg D, Baker A, Friedman L, Greene HL, et al. N Engl J Med 1991: 324(12); 781-8. [Medline]
• Mortality following ventricular arrhythmia suppression by encainide, flecainide, and moricizine after myocardial infarction. The original design concept of the Cardiac Arrhythmia Suppression Trial (CAST). Epstein AE, Hallstrom AP, Rogers WJ, Liebson PR, Seals AA, Anderson JL, Cohen JD, Capone RJ, Wyse DG. Jama 1993: 270(20); 2451-5. [Medline]
• The Cardiac Arrhythmia Suppression Trial: first CAST. then CAST-II. Greene HL, Roden DM, Katz RJ, Woosley RL, Salerno DM, Henthorn RW. J Am Coll Cardiol 1992: 19(5); 894-8. [Medline]

Study #6. Meta-analysis skewers a long held assumption about placebos. It has been common knowledge that patients placed on a placebo will get better at an amazing rate. This common knowledge, if you trace it back, all goes back to a study done in 1955. Hrobjartsson and colleagues undertook a careful review of 130 studies that included both a placebo arm and a no treatment arm.

Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment. Hrobjartsson A, Gotzsche PC. N Engl J Med 2001: 344(21); 1594-602. [Medline] [Abstract] [Full text] [PDF]

This study showed that the placebo effect was absent or equivocal to a large extent. One possible explanation is that the natural course of the disease, regression to the mean, and other factors might produce effects that are confused with the placebo effect.

Study #7A and 7B. Randomized studies trump observational studies. Another commonly held belief was that if you gave estrogen supplements to women who are going through menopause, they would have better health outcomes. There was some observational evidence to support this contention as well as a plausible biological mechanism. But a carefully conducted randomized trial in 1998

Randomized Trial of Estrogen Plus Progestin for Secondary Prevention of Coronary Heart Disease in Postmenopausal Women. Hulley S, Grady D, Bush T, Furberg C, Herrington D, Riggs B, Vittinghoff E. JAMA 1998: 280(7); 605-613.

and another randomized study in 2002

Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial. Jacques E. Rossouw, Garnet L. Anderson, Ross L. Prentice, Andrea Z. LaCroix, Charles Kooperberg, Marcia L. Stefanick, Rebecca D. Jackson, Shirley A. A. Beresford, Barbara V. Howard, Karen C. Johnson, Jane Morley Kotchen, Judith Ockene. Jama 2002: 288(3); 321-33. [Medline] [Abstract] [Full text] [PDF]

showed that women receiving hormone supplements in a randomized double blind trial showed slightly worse health outcomes. A meta-analysis of randomized trials that excluded observational studies

• Hemminki E, McPherson K. Impact of postmenopausal hormone therapy on
  cardiovascular events and cancer: pooled data from clinical trials.
  BMJ. 1997 Jul 19;315(7101):149-53.

also gave evidence that HRT was not effective.

 There is still active debate about why the randomized trial showed different results. Some commentaries on the HRT controversy are

• The discrepancy between observational studies and randomized trials of menopausal hormone therapy: did expectations shape experience? N. F. Col, S. G. Pauker. Ann Intern Med 2003: 139(11); 923-9. [Medline] [Abstract] [PDF]
• Rethinking Hormones, Again. Roni Rabin. The New York Times 2006 (January 31).

and the impact of these studies on prescribing patterns is shown at

• Blumel JE, Castelo-Branco C, Chedraui PA, Patients' and clinicians'
  attitudes after the Women's Health Initiative study. Menopause. 2004
  Jan-Feb;11(1):57-61.

Study #8. Where does meta-analysis stand on the hierarchy of evidence.

Issues in Comparisons between Meta-analyses and Large Trials. Ioannidis J, Cappelleri J, Lau J. Jama 2002: 279(14); 1089-93. [Medline]

Study #9. Financial conflicts of interest.

Conflict of interest in the debate over calcium-channel antagonists. Stelfox HT, Chua G, O. Rourke K, Detsky AS. N Engl J Med 1998: 338(2); 101-6. [Medline]

Study #10. Nonrandomized trials are evidence also.

The disappearance of Reye's syndrome--a public health triumph. Monto AS. N Engl J Med 1999: 340(18); 1423-4. [Medline]

Other possible candidates.

[Prevention of the first occurrence of anencephaly and spina bifida with periconceptional multivitamin supplementation (conclusion)] [Article in Hungarian]. Czeizel E, Dudas I. Orv Hetil. 1994 Oct 16;135(42):2313-7. [Medline]

The existence of publication bias and risk factors for its occurrence. Dickersin K. Jama 1990: 263(10); 1385-9.

A Randomized, Controlled Trial of the Effects of Remote, Intercessory Prayer on Outcomes in Patients Admitted to the Coronary Care Unit. Harris WS, Gowda M, Kolb JW, Strychacz CP, Vacek JL, Jones PG, Forker A, O’Keefe JH, McCallister BD. Archives of Internal Medicine 1999: 159(19); 2273-2278. [Medline]

Final report on the aspirin component of the ongoing Physicians' Health Study. Physicians' Health Study Research Group. N Engl J Med 1989: 321(3); 129-35. [Medline]

Randomised trials, human nature, and reporting guidelines. Schulz KF. Lancet 1996: 348(9027); 596-8. [Medline]

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Teaching resources.