More on the weaknesses of randomized trials (created 2005-02-14)

I've written about the randomized trial and how it can sometimes oversimplify the research question and how it often raises difficult ethical questions. Complementary and Alternative Medicine (CAM) practices raise interesting questions about the randomized trial. Here are a few weblog entries on the topic:

and I am writing about this in the first chapter of my book on Statistical Evidence.

Two new web resources seem to confirm this. The first:

discusses the problems with applying EBM to surgery outcomes. I have several citations about this in my bibliography, but this page raises an additional issue. Surgery is a highly individualized intervention with wide practices in variation. Unless you can quantify this practice variation, randomized trials will always have an unacceptable amount of noise in them. Also, since there is a difficult learning curve, it is unrealistic to expect a surgeon to offer both a new and a standard surgery with equal skill in a randomized trial.

The second resource discusses the application of Evidence Based Medicine to CAM.

This article also mentions the large practice variation in CAM and cites the variety of ways that accupuncture is performed as an example. The author also suggests that a formal decision analysis might work to the benefit of CAM.

CM therapies have long been prey to the view that no decision is worth taking until the evidence crosses a certain threshold (the level of which is never made explicit) and at which point the evidence becomes "strong." Decision analysis, conversely, allows an explicit estimate of how much evidence is enough evidence.

In the example that he cites, if the cost of untreated disease is 20 times worse than the cost of the treatment, then a threshold for using the treatment is 5% or greater. This might be an excellent example of how to apply the NNT in a practical sense.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Randomization in research.