StATS: What alternative medicine can teach us about evidence-based medicine (August 23, 2005)

This is a rough outline of a seminar I will present in a couple of days. It incorporates material from another talk,

The title of this talk seems to be backwards. The question most people are interested in is, "What can evidence-based medicine teach us about alternative medicine?" It turns out that we can learn quite a bit. A quick search of PubMed for the term "alternative medicine" restricted (for my convenience) to article in PubMed Central and limited to meta-analyses and systematic overviews produced the following reference.

A systematic review of the quality of homeopathic clinical trials. Jonas WB, Anderson RL, Crawford CC, Lyons JS. BMC Complement Altern Med 2001: 1; 12. [Medline] [Abstract] [Full text] [PDF] BACKGROUND: While a number of reviews of homeopathic clinical trials have been done, all have used methods dependent on allopathic diagnostic classifications foreign to homeopathic practice. In addition, no review has used established and validated quality criteria allowing direct comparison of the allopathic and homeopathic literature. METHODS: In a systematic review, we compared the quality of clinical-trial research in homeopathy to a sample of research on conventional therapies using a validated and system-neutral approach. All clinical trials on homeopathic treatments with parallel treatment groups published between 1945-1995 in English were selected. All were evaluated with an established set of 33 validity criteria previously validated on a broad range of health interventions across differing medical systems. Criteria covered statistical conclusion, internal, construct and external validity. Reliability of criteria application is greater than 0.95. RESULTS: 59 studies met the inclusion criteria. Of these, 79% were from peer-reviewed journals, 29% used a placebo control, 51% used random assignment, and 86% failed to consider potentially confounding variables. The main validity problems were in measurement where 96% did not report the proportion of subjects screened, and 64% did not report attrition rate. 17% of subjects dropped out in studies where this was reported. There was practically no replication of or overlap in the conditions studied and most studies were relatively small and done at a single-site. Compared to research on conventional therapies the overall quality of studies in homeopathy was worse and only slightly improved in more recent years. CONCLUSIONS: Clinical homeopathic research is clearly in its infancy with most studies using poor sampling and measurement techniques, few subjects, single sites and no replication. Many of these problems are correctable even within a "holistic" paradigm given sufficient research expertise, support and methods.

This is an open source journal, so I can include as much of it as I like on my web pages without worrying about copyright restrictions. If you are curious, you can read the full free text of this article on the web. There's even a journal called evidence-based Complementary and Alternative Medicine, and you can find the full free text of articles like the following:

Complementary and Alternative Medicine Approaches for Pediatric Pain: A Review of the State-of-the-science. Tsao JC, Zeltzer LK. Evid Based Complement Alternat Med 2005: 2(2); 149-159. [Medline] [Abstract] [Full text] [PDF] In recent years, the use of complementary and alternative medicine (CAM) in pediatric populations has increased considerably, especially for chronic conditions such as cancer, rheumatoid arthritis and cystic fibrosis in which pain may be a significant problem. Despite the growing popularity of CAM approaches for pediatric pain, questions regarding the efficacy of these interventions remain. This review critically evaluates the existing empirical evidence for the efficacy of CAM interventions for pain symptoms in children. CAM modalities that possess a published literature, including controlled trials and/or multiple baseline studies, that focused on either chronic or acute, procedural pain were included in this review. The efficacy of the CAM interventions was evaluated according to the framework developed by the American Psychological Association (APA) Division 12 Task Force on Promotion and Dissemination of Psychological Procedures. According to these criteria, only one CAM approach reviewed herein (self-hypnosis/guided imagery/relaxation for recurrent pediatric headache) qualified as an empirically supported therapy (EST), although many may be considered possibly efficacious or promising treatments for pediatric pain. Several methodological limitations of the existing literature on CAM interventions for pain problems in children are highlighted and future avenues for research are outlined.

This is also an open access article, but the notice at the top of the article reminds us that

if an article is subsequently reproduced or disseminated not in its entirety but only in part or as a derivative work this must be clearly indicated.

That's an important reminder and I apologize for only including the abstracts of these two articles. It's a bad habit to read only the abstract. The abstract often leaves out important details. Often the important limitations appear only in the paper itself. Sometimes, the outcome measures highlighted in the abstract are the ones that are statistically significant, rather than the ones that are clinically important.

There is a lot more that can be said about the empirical evidence for or against various alternative medicine approaches, but I want to turn the question around. I am interested in what alternative medicine can teach us about evidence-based medicine.

First it might help to define exactly what alternative medicine is. I don't want to dwell on this point too much, but you can get a good understanding of what a person thinks about alternative medicine by how they define it.

You can define it by exclusion, (everything that they didn't teach you in medical school), but that is a very squishy definition:

Alternative medicine describes health products, practices, and approaches that are not part of conventional medicine. But in the same way that an alternative band can be outside the establishment one year and part of the mainstream the next, the list of once alternative healing practices accepted by conventional medicine changes frequently, as new therapies and treatments are found to be effective. This makes alternative medicine an expanding, changing field of health care, as well as a booming business.

You can also define alternative medicine as anything that has not yet been proven scientifically.

What most sets alternative medicine apart, in our view, is that it has not been scientifically tested and its advocates largely deny the need for such testing. By testing, we mean the marshaling of rigorous evidence of safety and efficacy, as required by the Food and Drug Administration (FDA) for the approval of drugs and by the best peer-reviewed medical journals for the publication of research reports. Of course, many treatments used in conventional medicine have not been rigorously tested, either, but the scientific community generally acknowledges that this is a failing that needs to be remedied. Many advocates of alternative medicine, in contrast, believe the scientific method is simply not applicable to their remedies. They rely instead on anecdotes and theories. Alternative medicine--the risks of untested and unregulated remedies. Angell M, Kassirer JP. New England Journal of Medicine 1998: 339(12); 839-41.

That is also a squishy definition (what is scientific proof?) but also a recipe that allows conventional medicine to preempt any approaches which have been proven effective and to leave alternative medicine with only the failures and unproven approaches. Seems a bit unfair to me.

The Institute of Medicine has published a book:

and they offer a nice definition.

The National Center for Complementary and Alternative Medicine provides a taxonomy as a way of defining alternative medicine.

NCCAM classifies CAM therapies into five categories, or domains:

1. Alternative Medical Systems Alternative medical systems are built upon complete systems of theory and practice. Often, these systems have evolved apart from and earlier than the conventional medical approach used in the United States. Examples of alternative medical systems that have developed in Western cultures include homeopathic medicine and naturopathic medicine. Examples of systems that have developed in non-Western cultures include traditional Chinese medicine and Ayurveda.

2. Mind-Body Interventions Mind-body medicine uses a variety of techniques designed to enhance the mind's capacity to affect bodily function and symptoms. Some techniques that were considered CAM in the past have become mainstream (for example, patient support groups and cognitive-behavioral therapy). Other mind-body techniques are still considered CAM, including meditation, prayer, mental healing, and therapies that use creative outlets such as art, music, or dance.

3. Biologically Based Therapies Biologically based therapies in CAM use substances found in nature, such as herbs, foods, and vitamins. Some examples include dietary supplements,3 herbal products, and the use of other so-called natural but as yet scientifically unproven therapies (for example, using shark cartilage to treat cancer).

4. Manipulative and Body-Based Methods Manipulative and body-based methods in CAM are based on manipulation and/or movement of one or more parts of the body. Some examples include chiropractic or osteopathic manipulation, and massage.

5. Energy Therapies Energy therapies involve the use of energy fields. They are of two types:

 * Biofield therapies are intended to affect energy fields that purportedly surround and penetrate the human body. The existence of such fields has not yet been scientifically proven. Some forms of energy therapy manipulate biofields by applying pressure and/or manipulating the body by placing the hands in, or through, these fields. Examples include qi gong, Reiki, and Therapeutic Touch.

* Bioelectromagnetic-based therapies involve the unconventional use of electromagnetic fields, such as pulsed fields, magnetic fields, or alternating-current or direct-current fields.

The comments in the Angell et al editorial, though, are interesting because they highlight some of the controversies about alternative medicine. They argue that

It is time for the scientific community to stop giving alternative medicine a free ride. There cannot be two kinds of medicine - conventional and alternative. There is only medicine that has been adequately tested and medicine that has not, medicine that works and medicine that may or may not work. Once a treatment has been tested rigorously, it no longer matters whether it was considered alternative at the outset. If it is found to be reasonably safe and effective, it will be accepted. But assertions, speculation, and testimonials do not substitute for evidence. Alternative treatments should be subjected to scientific testing no less rigorous than that required for conventional treatments.

There is similar antagonism in the alternative medicine community to the standards of medical research.

evidence-based medicine is characterised by various shortcomings, the most important of which I have listed below.
1. Outcome or therapy based - not cause based
2. Excessive reliance upon clinical trials favours a symptomatic reductionist perspective.
3. Favours therapies supported by commercial interests prepared to fund research and clinical trials.
4. Does not emphasise nutrition.
5. Disease oriented - not focused upon optimum health.

The term used most often by critics of evidence-based medicine is "reductionism."

We should probably stop and define evidence-based medicine here. There are a variety of definitions, but my favorite is from an article by Sackett et al in the British Medical Journal.

evidence-based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.

It's important to remember to include the values of the individual patient in any EBM decision. Suppose you are considering a treatment that has as a side effect impairs the production of sperm and reduces your fertility. Some men, would not even consider such a treatment. They have a strong desire to father their own children now or in the future, and they would sacrifice their own health in order to maintain their ability to produce children. Other people would be totally indifferent to this side effect. A man with a vasectomy does not worry to much about drugs that alter his sperm production. Others might actually perceive reduced fertility as a benefit rather than a side effect.

So who is right in all of this debate. It turns out that the critics are half right. Some of their complaints are just "sour grapes." but they also highlight some areas for improvement, especially in the conduct of randomized clinical trials.

A balanced perspective on this controversy appears in (Mason 2002). They point out that:

"..many practitioners argue that research methods dissect their practice in a reductionist manner and fail to take into account complementary medicine's holistic nature."

They argue that randomized trials have to be adapted to the special features of CAM. In particular, they point out that the tendencies of randomized trials and CAM are often in conflict. Randomized trials:

In contrast, CAM

Note that these are tendencies. Some randomized trials focus on more than one disease, but the tendency is to focus on a single disease. Some types of CAM are standardized, but the tendency is to offer individualized therapies.

It's not just CAM that exhibits these conflicts, though. The Medical Research Council wrote a report in April 2000 ([pdf]) that discusses the evaluation of complex interventions where it is difficult to isolate the individual components of the intervention. They mention several examples.

Does a physiotherapist contribute significantly to the management of knee injuries? This role goes beyond a simple sequence of exercises.

The package of care to treat a knee injury may be quite straightforward and easily definable - and therefore reproducible: “This series of exercise in this order with this frequency for this long, with the following changes at the following stages”. However, the physiotherapist may have, in addition to the exercises, a psychotherapy role in rebuilding the patient's confidence, a training role teaching their spouse how to help with care or rehabilitation, and potentially significant influence via advice on the future health behaviour of the patient. Each of these elements may be an important contribution to the effectiveness of a physiotherapy intervention.

How does a stroke unit improve the quality of care for stroke patients? The concept of a stroke unit is difficult to standardize.

For example, although research suggests that stroke units work, what, exactly, is a stroke unit? What are the active ingredients that make it work? The physical set-up? The mix of care providers? The skills of the providers? The technologies available? The organisational arrangements?

How cognitive behavioral therapy works? This approach is highly individualistic.

Does success depend on the personality of the therapist? The personality, health status, social status, or other characteristic of the patient? The content of the therapy? The way it is delivered? The frequency of contact? The location of contact? The duration and the timing? What other components count?

Rather than arguing that randomized trials need to be adapted to the special needs of CAM, perhaps randomized trials should be adapted to meet the special needs of many types of medical interventions.

Furthermore, the claim that a practice is holistic should not be used as a blithely disregard evidence from an overly simplistic randomized trial. Perhaps the randomized trial can get to the heart of the issue by focusing on a single key dimension to the problem. A fourth grade student evaluated Therapeutic Touch (TT) for a science fair project. This project was highlighted on a Public Broadcasting Service show "Scientific American Frontiers" and published in the April 1, 1998 issue of JAMA (Rosa 1998) and received a lot of press coverage (CNN has a very nice story).

Therapeutic Touch is a therapy to improve health through the manipulation of the human energy field. There apparently is no physical touching. The official website on therapeutic touch describes it as:

" intentionally directed process of energy exchange during which the practitioner uses the hands as a focus to facilitate the healing process. It is a contemporary interpretation of several ancient healing practices. Therapeutic Touch is a scientifically-based practice founded on the premise that the human body, mind, emotions and intuition form a complex, dynamic energy field. The human energy field is governed by pattern and order. In health, the field is balanced, however in disease, the energy is characterized by imbalance and disorder."

Emily Rosa's experiment was very simple, perhaps too simple. If practitioners of Therapeutic Touch are able to manipulate energy fields, they must first be able to detect energy fields. She would hold her hand above either the left or right hand of the practitioner and ask him/her to tell which hand. The choice of hand was randomly determined by a coin flip. A screen with two holes in it prevented the practitioner from seeing what was going on.

Emily Rosa got 21 experienced practitioners to agree to the test. They were right only 44% of the time. Did this simple experiment disprove the healing power of TT? Perhaps not. TT is a complex intervention and this experiment only looked at a single aspect of it.

The experiment does shift the burden of proof, however. Detection of energy fields is a fundamental aspect of TT that all other aspects of this therapy rely on. How can practitioners of TT manipulate energy fields that they cannot even detect? Any further research should be discontinued until practitioners of TT can demonstrate the ability to detect energy fields in a rigorous blinded study.

Larry Sarner (Emily Rosa's step-father) makes much the same point in an article on the Quackwatch web site that responds to criticisms of the Rosa study. In particular, he responds to the criticism of reductionism:

[Critical comment #5] This was not a test of TT, but a parlor game. What the practitioners were required to do during the experiment invalidated its applicability to TT, especially since TT is a holistic process and can't be validly analyzed in parts. Emily's test was not of efficacy or technique (or "healing"), but I of raw ability. It's very much like testing a surgeon to see if he can l tell, without looking, in which hand the scalpel is being held. In any event, there was some movement. Emily presented her hand after each coin flip, which required relative movement between her hands and the subject's. Both subjects and Emily had at least small I movements of their hands during the trials, and some practitioners even wiggled their fingers or hands. Previous descriptions of the sensations of feeling an HEF state that the field itself is constantly in motion, and the literature states that such motion can be easily felt. Significantly, all of Emily's subjects agreed to the protocol and none voiced any concern that the test setup would pose a problem in demonstrating their ability. The argument about TT being "holistic" is a thinly disguised attempt to get back to "outcome" (i.e., clinical) testing, where it is easier to obfuscate, ignore negative results, or explain away nonconforming data. There have been numerous clinical trials on outcomes using TT. The results are highly mixed. Some tests do not have statistically significant results, others revealed slight positive effects (though statistically significant), and several actually reported statistically significant effects, but negative (i.e., the control group did better than the TT group). Holistic practitioners' prejudice against what they call "reductionism" (analyzing things in parts) is not shared by others in scientific medicine.

There is, by the way, a huge financial incentive to demonstrate the ability to detect energy fields. The James Randi Education Foundation offers a one million dollar prize to anyone who can show, under carefully controlled conditions, evidence of any paranormal, supernatural, or occult power or event. James Randi himself says that TT as well as several other alternative medicine therapies (Iridology, Reiki, Homeopathy and Applied Kinesiology) would qualify for the challenge.

Do commercial ties influence research findings? There are many documented cases where money does alter the research. Perhaps the best understood conflict of interest involves the tobacco companies. Financial support from tobacco companies has a large and quantifiable impact on the findings of a study. Articles on passive smoking written by authors affiliated with the tobacco industry were far more likely to conclude that passive smoking was not harmful (Barnes 1998). A review of studies on the economic effects of laws restricting smoking (Scollo 2003) showed that tobacco affiliations were associated with greater use of subjective outcomes, a lower rate of peer review, and a greater tendency to report negative economic impacts.

Support or commercial ties with pharmaceutical companies can also be troublesome. At least thirty studies have examined whether authors with commercial ties come up with more favorable conclusions about the drugs they are studying. A review of these studies, (Lexchin 2003) showed that industry-financed studies were four times more likely to reach conclusions favorable to the company's product when the researchers were supported by the drug company. The authors offered five possible explanations:

  1. drug companies might preferentially support and test only those drugs that have especially good prospects;
  2. the drug company sponsored trials could be of poorer quality and therefore more likely to draw contradictory conclusions;
  3. researchers might deliberately chose the "wrong " dose of the standard drug offered in the control group, leading to a higher rate of efficacy for the new drug, fewer side effects noted for the new drug, or both;
  4. drug companies might preferentially publish only the studies that support the use of the new drug; and
  5. drug companies might deliberately target symposiums, since the lack of peer review might allow them to make stronger statements about their drugs than the data itself would support.

Another problem is that authors rarely disclose possible conflicts. A review of disclosure of conflicts of interest (Hussain 2001) calculated the rate of disclosure at 1.4% (52 out of 3,642), a number that is far too low to be credible. If authors fail to report potential conflicts of interest, it may be out of the stubborn beliefs that commercial ties only influence other people (Boyd 2003).

Charges of financial conflict of interest are sometimes a "red herring" that is intended to distract from a discussion of the merits of the research. Stephen Senn tells an interesting story about himself (Senn 2001) where such a charge was leveled. Stephen Senn is a famous statistician with over 190 publications. Because of his stellar reputation, he is widely sought out as a statistical consultant to the pharmaceutical industry. In a discussion with an academic researcher, though, Dr. Senn was informed that his "source of employment" meant that his recommendations about the proper analysis of crossover trials were worthless. It didn't matter that Dr. Senn had written the definitive textbook on that very subject (Senn 1993).

So how should you approach a research article where the authors have declared a conflict of interest? You should be cautious, but not cynical. If the research is objective, well documented, and subject to external review, then you should not let financial conflict of interest exert a veto power over the findings. On the other hand, an editorial article or opinion piece written by an author with commercial ties to a product being discussed in the editorial is very troublesome (Angel 1996).

Is there an explicit assurance from the author that the industry support still allowed the author to independently assess the data and to publish the results without first getting approval from the sponsor? A reasonable review period by the sponsor is acceptable as long the final decision to publish rests with the author and not the sponsor. A 2001 revision to the statement on publication ethics from the International Committee of Medical Journal Editors (Davidoff 2001) highlights how important this assurance is.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Teaching resources.