StATS: The post-modern assault on evidence-based medicine (December 18 2007)

As mentioned in a December 7, 2007 weblog entry, I sent a brief email to Michael Shermer, editor of Skeptic Magazine. In it, I included a brief pitch for an article, "The post-modern assault on evidence-based medicine" and he gave me some encouragement to work on this. Guidelines for contributions are at http://www.skeptic.com/the_magazine/contribute.html and they accept articles of varying lengths (500 to 5000 words) though they prefer shorter articles.

Here are some of the talking points in the article. I'm including heading in this draft, but they are just to help me organize my thoughts.

Introduction. Evidence Based Medicine (EBM) has radically changed the way that doctors, nurses, and other health care professionals practice medicine. There are many competing definitions for EBM. A classic definition from one of the most popular books on the topic (Sackett 2000) defined EBM as "the integration of best research evidence with clinical expertise and patient values." The three pillars of EBM (best evidence, clinical expertise, and patient values) are all important, but some definitions will fail to emphasize all three elements eqaully. Interestingly, the first edition of this popular reference (Sackett 1997) defined EBM as "integrating individual clinical expertise with the best available clinical evidence from systematic research," a definition that fails to mention patient values at all.

The five steps in applying EBM in a clinical situation are:

  1. CONVERT the need for information into answerable questions.
  2. TRACK DOWN the best evidence with which to answer the questions.
  3. CRITICALLY APPRAISE the evidence for its validity, impact, and applicability.
  4. INTEGRATE the critical appraisal with our clinical expertise and with our patient's unique biology, values, and circumstances.
  5. EVALUATE our effectiveness and efficiency in executing steps 1-4 and seek ways to improve them both for next time. (Source: www.oucom.ohiou.edu/ebm/def.htm).

Most of the attention the EBM receives, however, focuses on the third step, critical appraisal. [[Elaborate]]

The Cochrane Collaboration. At the forefront of the EBM movement is the Cochrane Collaboration. The Cochrane Collaboration is organization of a large number of medical professionals who donate their services to produce evidence based systematic reviews on a variety of health topics. Some of the readers may be more familiar with the term meta-analysis. Meta-analysis constitutes the statistical tools used to quantitatively combine the results of multiple research studies. A systematic overview is a careful and reproducible method for gathering all available research on a particular topic, which may or may not include a quantitative pooling (meta-analysis) as part of the process. Thus, systematic overview is a more general term.

The Cochrane Collaboration uses a fairly rigid set of guidelines, again developed by medical professionals, to insure uniformity and a high level of quality for these systematic reviews.

The Cochrane Collaboration was named in honor of a famous physician, Archie Cochrane, who argued that "because resources would always be limited, they should be used to provide equitably those forms of health care which had been shown in properly designed evaluations to be effective. In particular, he stressed the importance of using evidence from randomised controlled trials (RCT's) because these were likely to provide much more reliable information than other sources of evidence." (www.cochrane.org/docs/archieco.htm)

Post modern critique. Any movement that makes major changes to the way that health care is provided is going to have its share of supporters and critics. But recently, EBM has come under a vigorous criticism assault from writers who apply post-modern techniques in their criticism. I'm a big fan of both EBM and post-modern philosophy, though both groups, particularly the latter, will sometimes take their arguments to ridiculous extremes. I also see EBM as a logical tool for investigating and validating some of the social, political, and historical influences on the process of combing best available evidence with clinician knowledge and patient values.

The most vicious of these assaults comes from an article (Holmes 2006) that calls for a wholesale rejection of EBM. I will use the name of the lead author (Holmes) to refer to this article. Holmes misses the mark and totally mischaracterizes the EBM movement. This criticism does, however, provide an opportunity to understand how EBM fits into a post-modern view of the world.

The heart of the critique in Holmes is that EBM is "dangerously exclusionary." According to Holmes, EBM relies on rules developed by the Cochrane Collaboration. And the Cochrane Collaboration, according to Holmes, will accept only randomized clinical trials and therefore rejects 98% of the available evidence.

No conspiracy theories, please. First it's worth noting that the Cochrane Collaboration is not a shadowy organization that secretly pulls the strings of the entire research community. Cochrane Collaboration reviews are not immune from criticism (Alderson 2003, Eysenbach 2005). To the extent that the research community has adopted models developed by the Cochrane Collaboration, it is out of respect for the quality of work that they produce (Oleson 1998).

Where does 98% come from? Holmes also misreads one of their sources here. The 98% comes from a quote by David Sackett, in his famous book about EBM (Sackett 2000). Sackett claimed claimed that only 2% of the published research is sound with valid conclusions. If you read the whole book, however, you will see that nowhere does Sackett equate randomized with sound/valid nor non-randomized with unsound/invalid. In fact, a careful reading of this book will show you that there are criteria both for randomized and non-randomized studies that you can use to gauge the validity of either type of study. Sackett does suggest that randomized studies are more appropriate for evaluating efficacy and non-randomized studies for evaluating harm, but even here, it is obviously more a convention to simplify the presentation. Sackett recognizes that randomized trials are sometimes more appropriate for some studies of harm and non-randomized studies are sometimes more appropriate for some studies of efficacy.

What evidence does the Cochrane Collaboration accept? Does the Cochrane Collaboration only accept randomized trials? Well yes and no. The Cochrane review relies on the "best available" evidence. Although there are exceptions to this rule (see below), the Cochrane Collaboration will generally exclude non-randomized studies from the systematic overview if good quality randomized trials are available. If randomized trials are not available, or if they are uniformly flawed, then they use non-randomized evidence.

This is consistent with the research hierarchy that underlies evidence based medicine. At the top of the hierarchy are systematic overviews, the kinds produced by the Cochrane group and others. If you need guidance from the medical literature, look at systematic overviews first. If a systematic overview is not available, then look for a high quality randomized study. If a randomized study is not available, look for a high quality non-randomized study. If that does not exist, look for a case series or for an expert opinion. This is not unlike the coffee drinker who always chooses cream over milk for their coffee if both are available, but will tolerate milk and only as a last resort rely on that awful non-dairy creamer.

The research hierarchy is indeed exclusionary in that it excludes one level of evidence if a higher form of evidence is available. Perhaps what is being exercised here is not exclusion but discretion. The belief of many in the EBM movement is that a single well-conducted randomized trial will trump any number of non-randomized trials. Is this belief true? The most common argument for the research hierarchy is (irony of ironies) anecdotal evidence: mammary artery ligation, post-menopausal hormone replacement therapy, [[cite other examples]]. As a statistician, I'm supposed to sneer at anecdotal evidence, but I can't. I'm not entirely convinced (see below), but I do find these anecdotes to be fairly persuasive.

Cochrane does not exclude observational data. [[Talk about observational data. http://www.cochrane.org/reviews/en/ab003517.html "We selected all internally-controlled clinical trials and observational studies comparing child or maternal health outcomes with exclusive breastfeeding for six or more months versus exclusive breastfeeding for at least three to four months with continued mixed breastfeeding until at least six months. Studies were stratified according to study design (controlled trials versus observational studies), provenance (developing versus developed countries), and timing of compared feeding groups (three to seven months versus later)."]]

Does Cochrane exclude qualitative data? [[Talk about the Cochrane Qualitative Research Methods Group. http://www.joannabriggs.edu.au/cqrmg/ "The Cochrane Qualitative Research Methods Group develops and supports methodological work on the inclusion in systematic reviews of findings from studies using qualitative methods and disseminates this work within and beyond the Collaboration's Review Groups. The Cochrane Qualitative Methods Group focuses on methodological matters arising from the inclusion of findings from qualitative studies into systematic reviews."]]

[[Example of qualitative http://medicine.plosjournals.org/perlserv/?request=cite-builder&doi=10.1371/journal.pmed.0040238]]

The authors have conducted many interesting qualitative research studies. [[Cite some examples.]] Would the authors be able to publish more and have their research taken more seriously if there was no evidence based medicine? Let's be very clear about this. The qualitative research that the authors are fond of was derided much more strongly prior to EBM.

"Thus, the authority of the clinician must be understood as a discursive power that shapes the realm of the possible and, in doing so, often ignores certain symptoms that would allow a more appropriate diagnosis."

Another factor at work here is the authors apparent mistrust that statistical knowledge will be capable of addressing all of the important in nursing in particular and in medicine in general. One of the sources cited in Holmes explains this well. "Codes for the production of nursing knowledge have been skewed towards knowledge that is statistically verifiable, rupturing the methodological pluralism that the nursing community had previously accepted as suitable for the production of nursing knowledge."

The role of individual patient preference. Allowing patients to make choices was almost unheard of prior to EBM. [[Talk about value based medicine and the concept of particularizing or individualizing. Note that value based medicine is not a rejection of EBM but rather incorporates it under a broader umbrella. www.mja.com.au/public/issues/177_06_160902/lit10253_fm.html ]]

The dreaded I word, Intuition. Unmentioned by Holmes in this list of other ways of knowing is the dreaded I word, Intuition. What role should intuition play in the practice of medicine? My hunch is that intuition is overrated as a way of knowing. I could cite a whole bunch of empirical studies that criticize our process of intuition, but that research would not be all that persuasive to someone who views intuition as an important way of knowing. Think back to the last time that you were able to convince a skeptical audience that your viewpoint was correct. Were they persuaded by your powers of intuition? Or did it take something else?

Also, I know of several ways to reconcile two conflicting empirical studies. I can split the difference, I can search for a methodological  explanation of the disparity, I can use a third replication to break the tie, or I can select the results from the empirical study that is higher on the research hierarchy.

How do I reconcile two people offering conflicting intuitions? I can choose the one who agrees with my current intuition (which makes you ask, why did you bother searching for those other intuitive insights) or I could choose the intuitive insight from the person I respect and trust more. This brings us back to the days of eminence based medicine.

But the point to remember is the EBM does not force you to abandon your intuition. Your intuition is part of your clinical judgment, and clinical judgment is explicitly acknowledged in the definition of EBM. I wouldn't want it any other way. If all of the research said that a certain therapy was good for you, but your doctor or nurse had a nagging suspicion that you were an exception, I'd want to hear about it, wouldn't you?

Deconstructing Holmes. You can turn the deconstruction argument around and ask what are the historical, linguistic, social, and political discourses that influence Holmes's wholesale rejection of EBM. All of the authors are in the nursing profession. Their are some veiled hints that these authors chafe at the unequal power dynamic between doctors and nurses.

What exactly is deconstruction? Holmes defines it as a process that "works to demonstrate how concepts or ideas are contingent upon historical, linguistic, social and political discourses, to name but a few." This would actually be an excellent exercise, but one that Holmes fails at miserably.

So what are the historical, linguistic, social, and political discourses? It all boils down to scientists who don't want to "risk being alienated from their colleagues." That's the only historical, linguistic, social, or political discourse that Holmes presents and it's a pretty lousy one to be honest. Scientists are a rather quarrelsome lot, and can sometimes be downright rude. [[Include a few examples]] If they don't want to risk alienation from their colleagues, why are they using all these nasty names?

Is EBM a surrogate for cost containment at the expense of quality of care?

Deconstructing EBM using EBM. EBM does a better job of deconstructing itself than Holmes does. For example, what influence does money have on EBM. Quite a bit. [[Summarize the research]]

[[EBM does not pretend to be objective. Cite mammography study.]]

[[Talk about POEMs, and self-reported outcomes.]]What influence does the sex and race of the patient have on a clinician's decisions? [[Summarize the research.]]

What influence does a doctors preconceived notions have on EBM? [[Talk about confirmation bias.]]

What influence does the dominance of the English language have on EBM? [[Talk about language bias.]] Ignoring non-English language publications is a dangerously exclusionary practice that was discovered through EBM. [[Elaborate]]

What impact does the availability of information on the Internet have on EBM? [[Talk about FUTON bias.]]

EBM criticisms of the hierarchy of evidence. Some proponents of EBM have researched the very issue that bothers Holmes, the rejection of some of the 98% of the research. [[Talk about research on observational versus randomized studies.]]

[[* Incorporating quality of evidence into decision analytic modeling. R. S. Braithwaite, M. S. Roberts, A. C. Justice. Ann Intern Med 2007: 146(2); 133-41. [Medline] [Abstract] I'm reproducing the abstract here because I find the results intriguing. Our objective was to illustrate the effects of using stricter standards for the quality of evidence used in decision analytic modeling. We created a simple 10-parameter probabilistic Markov model to estimate the cost-effectiveness of directly observed therapy (DOT) for individuals with newly diagnosed HIV infection. We evaluated quality of evidence on the basis of U.S. Preventive Services Task Force methods, which specified 3 separate domains: study design, internal validity, and external validity. We varied the evidence criteria for each of these domains individually and collectively. We used published research as a source of data only if the quality of the research met specified criteria; otherwise, we specified the parameter by randomly choosing a number from a range within which every number has the same probability of being selected (a uniform distribution). When we did not eliminate poor-quality evidence, DOT improved health 99% of the time and cost less than $100 000 per additional quality-adjusted life-year (QALY) 85% of the time. The confidence ellipse was extremely narrow, suggesting high precision. When we used the most rigorous standards of evidence, we could use fewer than one fifth of the data sources, and DOT improved health only 49% of the time and cost less than $100 000 per additional QALY only 4% of the time. The confidence ellipse became much larger, showing that the results were less precise. We conclude that the results of decision modeling may vary dramatically depending on the stringency of the criteria for selecting evidence to use in the model.]]

A lot of post-modern writing is frustrating to read because it offers only complaints and no solutions. The message seems to be "objective reality is an illusion--deal with it!" Holmes is actually refreshing in that they cite what a world without EBM would be like. "an honest plurality of voices will open up a space of freedom for the radical singularity of individual and disparate knowledge(s)." [[Extract other quotes from here and other articles by the authors.]]

What was life like before EBM? Before EBM became prominent, changes in medicine occurred when a small group of respected experts opined that changes were needed. In a humorous article, Isaacs and Fitgerald (BMJ 1999;319:1618-1618) call this eminence-based medicine. "Eminence based medicine---The more senior the colleague, the less importance he or she placed on the need for anything as mundane as evidence. Experience, it seems, is worth any amount of evidence. These colleagues have a touching faith in clinical experience, which has been defined as 'making the same mistakes with increasing confidence over an impressive number of years.' The eminent physician's white hair and balding pate are called the "halo" effect."

It's ironic that Holmes uses words like "hegemonic" to describe EBM. EBM actually disbanded the prior hegenomy of respected experts, a hegemony, it is worth noting, that excluded for the most part the voices of women, the voices of people from developing countries, and the voices of non-physicians.

Today, thanks to EBM, the process of impacting clinical practice is much more democratic.  If you don't like the status quo, conduct your own research and publish it. Unless you conduct this research sloppily, it will get into the next systematic overview. If you don't trust the people conducting systematic overviews, conduct your own systematic overview. I won't pretend that it's easy to conduct your own systematic overview, but for the record, a systematic overview is scientific in that its methods are transparent and open to replication. The recipe is out there for anyone who has the stamina to follow it. If you believe that a systematic overview is dangerously exclusionary, replicate it but with a broader set of inclusion criteria. And finally, if you don't like the rules used by the Cochrane Collaboration, study the empirical impact of using a different set of rules. If EBM is as dangerous as claimed in Holmes, then there ought to be plenty of empirical studies out there that could demonstrate this. I hope that the post-modern critics don't think that only non-quantitative tools can be used to critique EBM, because if they did, they would be dangerously exclusionary.

[[Other commentaries.  http://bjp.rcpsych.org/cgi/content/full/180/1/1  Am J Med. 1997 Dec;103(6):529-35. Problems in the "evidence" of "evidence-based medicine". Feinstein AR, Horwitz RI. Psychiatric Bulletin (2004) 28: 317-318 Psychiatry in the future The next 15 years: postmodern challenges and opportunities for psychiatry Richard Laugharne.]]

Is it fair to use a flawed tool, EBM, to evaluate the flaws of EBM? Holmes claims that EBM proponents "seldom question the authority of their own discourses, but deploy them unknowingly." This statement is demonstrably false. EBM is a system with built-in self-correction. If EBM produces bad conclusions, one can study this and propose remedies using EBM itself.

What can EBM learn from post-modern thought? [[Cite the Sheldrake example of blinding. Abandon dichotomies of good and bad. Incorporate non-traditional sources of information into systematic reviews. Don't focus on just the quantitative meta-analytic summary. Think of systematic reviews as a way to highlight the deficiencies in a particular health care area, and to highlight the deficiencies of particular research methodologies.]]

The charge of fascism. The article by Holmes et al uses the strong term "fascism" to describe the evidence based movement. You can tell that the authors are nervous about using such a strong term. They immediately qualify the term by pointing out that the "fascism of the masses" (as practiced by Hitler and Mussolini) has been replaced by microfascisms, "polymorphous intolerances that are revealed in more subtle ways." But micro doesn't mean less serious. In fact Holmes point out that micro fascism is less brutal but more pernicious. That's the problem with analogies to fascism; they rapidly lose any sense of perspective. The prospect of an insurance company using an EBM argument to deny women under the age of 50 a free mammogram is indeed bad, but can it compare to life under a brutal dictator like Saddam Hussein?

[[Cite Archie Cochrane's battle against Fascists during the Spanish Civil War. It's found either at http://www.plos.org/cms/node/99 or at http://hcrenewal.blogspot.com/2006/12/another-post-modernist-attack-on-ebm.html]]

Personal notes. I need to review the definitions of economic rationalism (Economic rationalists tend to favour Deregulation, Privatisation, a free market economy, privatisation of state-owned industries, lower direct taxation and higher indirect taxation, and a reduction of the size of the Welfare State.), empiricism (n philosophy generally, empiricism is a theory of knowledge emphasizing the role of experience, especially sensory perception, in the formation of ideas, while discounting the notion of innate ideas. In the philosophy of science, empiricism is a theory of knowledge which emphasizes those aspects of scientific knowledge that are closely related to experience, especially as formed through deliberate experimental arrangements. It is a fundamental requirement of scientific method that all hypotheses and theories must be tested against observations of the natural world, rather than resting solely on a priori reasoning, intuition, or revelation. Hence, science is considered to be methodologically empirical in nature.), positivism (Positivism is a philosophy that states that the only authentic knowledge is scientific knowledge, and that such knowledge can only come from positive affirmation of theories through strict scientific method.) and pragmatism (Pragmatism is a philosophic school that originated in the late nineteenth century with Charles Sanders Peirce, who first stated the pragmatic maxim. It came to fruition in the early twentieth-century philosophies of William James and John Dewey. Most of the thinkers who describe themselves as pragmatists consider practical consequences or real effects to be vital components of both meaning and truth. Other important aspects of pragmatism include anti-Cartesianism, radical empiricism, instrumentalism, anti-realism, verificationism, conceptual relativity, a denial of the fact-value distinction, a high regard for science and evolution, and fallibilism. Pragmatism began enjoying renewed attention from the 1950s on, because of a new school of philosophers who put forth a revised pragmatism that criticized the logical positivism that had dominated philosophy in the United States and Britain since the 1930s, notably in the work of analytic philosophers like W.V.O. Quine and Wilfrid Sellars. Their naturalized epistemology was further developed and widely publicized by Richard Rorty, whose later work grew closer to continental philosophy and is often considered relativistic. Contemporary pragmatism is still divided between those thinkers who work strictly within the analytic tradition, a more relativistic strand in the wake of Rorty and lastly neoclassical pragmatists like Susan Haack who stay closer to the work of Peirce, James and Dewey.).

What's a good definition of postmodernism?

A general and wide-ranging term which is applied to literature, art, philosophy, architecture, fiction, and cultural and literary criticism, among others. Postmodernism is largely a reaction to the assumed certainty of scientific, or objective, efforts to explain reality. In essence, it stems from a recognition that reality is not simply mirrored in human understanding of it, but rather, is constructed as the mind tries to understand its own particular and personal reality. For this reason, postmodernism is highly skeptical of explanations which claim to be valid for all groups, cultures, traditions, or races, and instead focuses on the relative truths of each person. In the postmodern understanding, interpretation is everything; reality only comes into being through our interpretations of what the world means to us individually. Postmodernism relies on concrete experience over abstract principles, knowing always that the outcome of one's own experience will necessarily be fallible and relative, rather than certain and universal. Postmodernism is "post" because it is denies the existence of any ultimate principles, and it lacks the optimism of there being a scientific, philosophical, or religious truth which will explain everything for everybody - a characteristic of the so-called "modern" mind. The paradox of the postmodern position is that, in placing all principles under the scrutiny of its skepticism, it must realize that even its own principles are not beyond questioning. As the philosopher Richard Tarnas states, postmodernism "cannot on its own principles ultimately justify itself any more than can the various metaphysical overviews against which the postmodern mind has defined itself." http://www.pbs.org/faithandreason/gengloss/postm-body.html

Note that postmodernism is not the same as relativism.

belief that there is no grand narrative.

Deconstruction is "A term that, for all practical purposes, was introduced in the literature by Derrida. It means to undermine the conceptual order imposed by a concept that has captivated our imaginations and ways of seeing things." http://www.california.com/~rathbone/local3.htm

Perhaps the most prominant definition of postmodernism comes from Jean-Francois Lyotard. According to Lyotard, the "postmodern" (see his book The Postmodern Condition: A Report on Knowledge, p.xxv) is an incredulity towards meta-narratives. This means the postmodern is one who is skeptical of theories that speak in grand generalities and that universalize their conclusions by pretending there are no exceptions. Translated into therapy theory this often means that the postmodern therapist works to avoid dogmatic posturing and claiming to state the "Truth" of the client's situation. In postmodern discussion forums this means that the common quest is not for consensus to emerge around some grand statement but for paralogical conversation to emerge. For many people, especially in postmodern therapy, "postmodern" means disillusionment with the standard way of understanding things. In this case, the therapist tries to offer a less "pathologizing" way of thinking about the client's issuesSome disillusioned postmoderns, however, are nostalgic, and see no path forwards, whereas other postmodern therapists are visionary . Even the most visionary, however, are likely to be tolerant of alternative and multiple points of view on an issue, all a consequence of learning to live without faith in metanarratives, to live with uncertainty and not-knowing. http://www.california.com/~rathbone/local5.htm

modern - Also called "modernist." In the context of a postmodern vocabulary, the "modern" does not mean "contemporary." In fact, the "modern" or "modernism" is seen as out-of-date. The "modern" is understood to have emerged during the 18th century Enlightenmentwhen philosphers were challenging superstitions (which often included religion) of premodern beliefs. They replaced faith in superstition with faith that science and objectivity could build us a better world. Moderns prefer objective and factual language. "Modern" therapies (as postmoderns use the term) are therapies that pretend to be scientific when they are not by using scientific sounding terms are methods. http://www.california.com/~rathbone/local4.htm

Bibliography

Alderson P, Chalmers I. Survey of claims of no effect in abstracts of Cochrane reviews. BMJ 2003: 326(7387); 475. [Medline] [Full text] [PDF]

Eysenbach G, Kummervold PE. "Is Cybermedicine Killing You?"--The story of a Cochrane disaster. J Med Internet Res 2005: 7(2); e21. [Medline] [Full text] http://www.jmir.org/2005/2/e21/.

Olsen O, Ezzo J, Gøtzsche PC, Hadhazy V, Herxheimer A, Kleijnen J, McIntosh H. Quality of Cochrane reviews: assessment of sample from 1998. BMJ 2001 (13 October); 323: 829-832.

Strauss SE, McAlister FA. CMAJ 2000 (October 3), 2000; 163 (7)

Winch S, Creedy D, Chaboyer W. Governing nursing conduct: the rise of evidence-based practice. Nurs Inq 2002; 9: 156–61.

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 United States License. It was written by Steve Simon on 2007-12-07, edited by Steve Simon, and was last modified on 2010-04-01. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Critical appraisal

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at