Small relative risks (January 13, 2005)
This page is moving to a new website.
I found a quote on the Skeptic's Dictionary web site that is worth commenting on. The author, Robert Todd Carroll was describing the Vioxx controversy and the lawyers who are now aggressively recruiting people for a lawsuit against Merck, the manufacturer of Vioxx. In it, he repeats a common misconception about relative risk (RR):
According to mathematician John Brignell, "In observational studies, [scientists] will not normally accept an RR of less than 3 as significant and never an RR of less than 2." -- skepdic.com/refuge/funk43.html
This is a widely quoted rule, but it vastly oversimplifies the issue. A small RR is a weaker form of evidence, because it is possible to be swamped by even small biases and flaws in the research study. A large RR is a stronger form of evidence, because only a huge bias or flaw could produce an alternate explanation for these findings.
Weak evidence, however, is not the same thing as no evidence. Weak evidence that is replicated and backed up by a credible biological mechanism can become sufficiently persuasive.
The original source of the rule cited above is a paper by Austin Bradford Hill published in 1965 [Medline]. It listed nine criteria that you should consider when evaluating a research finding from observational data. It was developed during a time when Epidemiologists had just successfully built a case linking smoking and lung cancer on the foundation of numerous research studies, none of which was perfect. The lessons learned in that situation became guidelines for applying Epidemiologic principles to other toxic exposures.
The first criterion that Hill cited, strength of association, has unfortunately morphed into a dichotomy that all RRs smaller than 2 are bad and all RRs larger than 2 are good, in spite of numerous cautions, both by Hill himself in that article and by many Epidemiologists since then.
In the classic textbook, Modern Epidemiology (Second Edition), Rothmann and Greenland [BookFinder4U link] point out (page 24) that the relationship of cigarette smoking and cardiovascular disease is a weak association, because it is pretty hard to double the risk of a disease that is already responsible for about 40% of all deaths, according to the American Heart Association. Nevertheless, pretty much everyone accepts the link between smoking and cardiovascular disease. If you relied on John Brignell's advice about relative risks less than 2 or 3, then your standard of proof would require that 80% to 120% of all smokers would have to die from cardiovascular disease. Nothing, not even cigarettes, can produce 120% mortality, and even 80% mortality from a single cause is probably out of reach.
A good recent commentary on this issue is "The missed lessons of Sir Austin Bradford Hill" by Carl Phillips and Karne Goodman.
That's my major quibble. A minor quibble is that this rule is usually restricted, even by John Brignell, to observational studies. But the study that showed an increase in risk for Vioxx was a randomized study.
This doesn't mean that the rest of Dr. Carroll's comments are incorrect. It is indeed possible that we are all overreacting to weak data. I don't think the final chapter has been written yet on Vioxx and other drugs in its class.
A chapter of the book I am writing, Statistical Evidence, talks about Corroborating Evidence. This chapter mentions strength of association, among other factors that can influence the quality of evidence in research. I also have a weblog entry on side effects which mentions Vioxx and other COX-2 inhibitors.
I sent an email along the same lines as this weblog entry to Dr. Carroll and he published it on his web site at
References