P.Mean >> Category >> Measuring benefit/risk (created 2007-06-26).

There are many measures of risk or benefit. I describe some of these (the odds ratio, the relative risk, the number needed to treat) and explain the advantages and disadvantages of particular measures. Articles are arranged by date with the most recent entries at the top. You can find outside resources at the bottom of this page.

2011

21. P.Mean: There's more than one way to calculate a Fisher's exact p-value (created 2011-07-21). I was trying to check the calculations associated with a two by two table and I noticed an inconsistency in the reporting of results. One program reported a p-value of 0.4588 for the two-tailed Fisher's exact test, and the other package reported a p-value of 0.308088. The packages otherwise agreed with one another. So which package is right? Well it turns out that both of them are correct because there is more than one way to calculate a Fisher's exact p-value. To understand this, you need to recall the computational details of Fisher's exact test.

2010

20. P.Mean: The odds ratio in logistic regression is the opposite of what it should be (created 2010-11-22). I have data in the following a table that clearly shows a positive association, but when I run a logistic regression model, the odds ratio is reported as less than 1. How can this be?

19. What is the number needed to treat? (September/October 2010)

2009

18. P.Mean: Calculating NNT for indirect comparisons (created 2009-04-20). To calculate the Numbers Needed to Treat (NNT) statistic for response rates when the effect size is shown as an odds ratio I carry out the following calculation: NNT = (1-(CER*(1-OR))) / ((1-CER)*(CER)*(1-OR)) [1] CER = Control Event Rate OR = Odds Ratio My query occurs when I am calculating this for an indirect comparison. So for example if I am comparing A and B vs a common comparator C I have the following set up: Trial 1 - A vs C: Response rate A = 0.8 Response rate C = 0.6. Trial 2 - B vs C: Response rate B = 0.7 Response rate C = 0.55. Indirect comparison gives (for example) A vs B odds ratio of 0.85 (0.6, 1.2). Is it valid to calculate the NNT by substituting CER = 0.8 and OR = 0.85 into the first equation [1]?

Other resources:

Julian D. Absolute and relative truth in clinical trials. Lancet. 2002;359(9321):1945-6; author reply 1946. Description: This article criticizes a recent publication for failing to present risk in absolute rather than relative terms. Available at: http://www.ncbi.nlm.nih.gov/pubmed/12057577 [Accessed January 30, 2009].

Straus SE, Sackett DL. Applying evidence to the individual patient. Ann Oncol. 1999;10(1):29-32. Description: This paper provides practical guidance on the NNT/NNH tradeoffs. Available at: http://annonc.oxfordjournals.org/cgi/reprint/10/1/29 [Accessed January 30, 2009].

Douglas G Altman. Confidence intervals for the number needed to treat. BMJ. 1998;317(7168):1309-1312. Excerpt: "The number needed to treat is a useful way of reporting results of randomised clinical trials. When the difference between the two treatments is not statistically significant, the confidence interval for the number needed to treat is difficult to describe. Sensible confidence intervals can always be constructed for the number needed to treat. Confidence intervals should be quoted whenever a number needed to treat value is given" [Accessed February 8, 2010]. Available at: http://www.bmj.com/cgi/content/full/317/7168/1309.

Interesting quote: "Drugs are prescribed because of their potential benefits, but in every case there are risks of harms; before prescribing, the former should be weighed against the latter. This is commonly called assessing the "benefit to risk ratio." But benefit and risk are non-comparable: one is an actual outcome, the other a chance of one. Benefits are properly balanced by harms. However, the two are incommensurate and cannot be combined into a ratio. One should therefore talk about the benefit to harm balance, which is a complex function of the seriousness of the problem to be treated, the efficacy and safety of the drug to be used, and the efficacy and safety of other available drugs." Jeff Aronson. www.bmj.com/cgi/content/full/329/7456/30

Robert M. Pruzek. Introduction to the Special Issue on Propensity Score Methods in Behavioral Research. Abstract: This issue includes six articles that present logic, method s, and models for causal analyses of observational data, in particular those based o n propensity score (PS) methods. The articles include a general introduction to pro pensity score analysis (PSA), uses of PSA in mediation studies, issues involved in c hoosing covariates, challenges that often arise in PSA applications, hierarchi cal data issues and models, and an application in an educational testing context. In thi s editorial I briefly summarize each article and make a few recommendations that r elate to future applications in this field: the first pertains to how propensi ty score (PS) work could profit by connecting it with stronger forms of randomized exp eriments, not just simple randomization; the second to how and why graphical me thods could be used to greater advantage in PSA studies; then why it might be help ful to reconsider the meaning of the term "treatments" in observational studies a nd why conventional usage might be modified; and finally, to the distinction betwe en retrospective and prospective approaches to observational study design, not ing the advantages, when feasible, of the latter approach. Available at http://rmpruzek.com/wp-content/uploads/2011/07/rPruzek.Editorial.SpecialIssue.MBR_.PSA11.pdf

3M APR-DRG Severity-of-Illness Software Excerpt: Grouping, validating, and pricing claims data will never be the same. The State of Maryland is leading the move toward a more equitable and accurate way to pay for health care: A severity-based payment system. Supported by years of research, Maryland is acting on the fact that patients with a higher level of severity require more healthcare resources and that there are significant variances in patient severity in the existing payment system. As the State of Maryland moves toward its implementation of having hospitals set payment rates based on the 3Mô APR-DRGô Classification Systemís severity scoring, hospitals might feel overwhelmed with the requirements for change. But, this change doesnít have to be complex. In fact, it can be easy and simple, thanks to 3M APR-DRG Severity-of-Illness Software.

The ďArms RaceĒ on American Roads: The Effect of SUVís and Pickup Trucks on Traffic Safety [pdf] Description: This web publication paper calculates expected deaths if a million drivers switched from light trucks to cars. It could easily be adapted to NNT and NNH calculations.

Creative Commons License All of the material above this paragraph is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-11-23. The material below this paragraph links to my old website, StATS. Although I wrote all of the material listed below, my ex-employer, Children's Mercy Hospital, has claimed copyright ownership of this material. The brief excerpts shown here are included under the fair use provisions of U.S. Copyright laws.

2008

17. Stats: Calculating NNT for observational studies (March 3, 2008). Recent discussion at the Evidence Based health list centered on the calculation of NNT in a case-control study. While it is indeed possible to do so, I have always been a bit curious why NNT and NNH are computed almost exclusively for randomized studies and why they are rarely used for observational studies. No one says this explicitly, but I suspect that the reason is that the NNT and NNH lead to problematic interpretations in observational studies.

2007 

16. Stats: Analyzing data under an Intention to Treat model (December 19, 2007). Dear Professor Mean, I need to know how to analyze a data set using the intention to treat principle.

2006 

15. Stats: Handling dropouts in NNT/NNH calculations (January 16, 2006). Someone asked a question on the Evidence-Based Health email discussion group about how to handle dropouts in an NNT/NNH calculation. There is no standard way of handling this, but a little bit of common sense goes a long way.

2005 

14. Stats: Fractions are funny (December 13, 2005). On my web page about odds ratios, I point out the fractions are funny and cite some well known examples of how fractions behave in a counter intuitive fashion.

13. Stats: Converting an odds ratio to a relative risk (August 3, 2005). The odds ratio and the relative risk do not always agree, especially when the baseline risk level in the control group is large. I have a simple Excel spreadsheet that will calculate the relative risk from the odds ratio and the control risk, ConvertORtoRR.xls.

12. Stats: The difference between absolute risks and relative risks (July 15, 2005). There are two general ways to compare a treatment and a control group, relative comparisons and absolute comparisons. For a relative comparison, the basic computation is division. When the ratio A/B is larger than one, that implies that A is superior to B. For an absolute comparison, the basic computation is subtraction. When the difference A-B is greater than zero, that implies that A is superior to B.

11. Stats: What does a 60% drop mean? (June 20, 2005). A friend sent me an email quoting the following statistic: "A woman's chances of getting married dropped 60% for every rise of 15 points in her IQ, according to an English study reported in The Atlantic." reported in June 2005 Touchstone. I do not have easy access to either The Atlantic or Touchstone magazine. But the number seemed too large to be credible. so I ran a few simple calculations.

10. Stats: Incidence density ratio (April 19, 2005). Someone asked me about a technical term, incidence density ratio, that was used in an article: Comprehensive discharge planning and home follow-up of hospitalized elders: a randomized clinical trial. Naylor MD, Brooten D, Campbell R, Jacobsen BS, Mezey MD, Pauly MV, Schwartz JS. Jama 1999: 281(7); 613-20.. Looking at the article, they only mention the term once and in context with another term, rate ratio, that you are probably familiar with.

9. Stats: Side effects of Cox-2 inhibitors (February 15, 2005). There has been so much published about side effects of Cox-2 inhibitors that it is hard to keep up with the evolving story. Here are a few recently released articles.

8. Stats: Small relative risks (January 13, 2005). I found a quote on the Skeptic's Dictionary web site that is worth commenting on. The author, Robert Todd Carroll was describing the Vioxx controversy and the lawyers who are now aggressively recruiting people for a lawsuit against Merck, the manufacturer of Vioxx. In it, he repeats a common misconception about relative risk (RR): According to mathematician John Brignell, "In observational studies, [scientists] will not normally accept an RR of less than 3 as significant and never an RR of less than 2."

7. Stats: Odds ratios less than one (January 6, 2005). Someone sent me an email asking how to interpret an odds ratio less than 1.

2004 

6. Stats: Testing for side effects (December 29, 2004). Today's USA Today has a nice summary of the recent controversies over drugs that are being removed from the market because of an unacceptable increased risk of side effects. Under special scrutiny these days are a class of drugs known as COX-2 inhibitors.

5. Stats: Neyman bias (December 15, 2004). Selection of controls in a case-control study is difficult enough, but you also have to worry about the selection of the cases. Do you select incident cases (for example all breast cancer patients newly diagnosed during a given time frame) or prevalent cases (for example, all breast cancer patients who are alive during a given time frame). These can lead to very different answers, because the probability of finding a case in a given time frame is related to mortality risk.

4. Stats: Number Needed to Harm Examples (February 18, 2004). I was trying to track down a reference on the NNH (number needed to harm) for antibiotic use, but noticed instead the large number of good examples of NNH calculations in journal articles with full text on the web.

2001 

3. Stats: Odds ratio versus relative risk (January 9, 2001). Dear Professor Mean:  There is some confusion about the use of the odds ratio versus the relative risk. Can you explain the difference between these two numbers

2000

2. Stats: Intention to treat (January 27, 2000). Dear Professor Mean: I'm confused by medical journal articles that talk about the use of "Intention to Treat" analysis. What does this term mean?

1. Stats: Number needed to treat (January 27, 2000). Dear Professor Mean: How are patients and their doctors supposed to decide whether a research finding has practical significance? Why don't the medical journals make things clearer?

What now?

Browse other categories at this site

Browse through the most recent entries

Get help

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-11-23.