P.Mean: Errors in statistical methodology (created 2008-10-19).
From those of you who review/appraise articles regularly, I would like to hear what kinds or errors you find most often in the statistical methodology. I will be training nurses to critically appraise the statistical methodology sections of articles, and since their time and knowledge of statistics are limited, I hope to focus mainly on errors they are likely to find in real articles.
With all due respect, I would suggest that you discourage the critical appraisal of statistical analysis choices. There are several reasons for this:
- It takes a lot of experience to spot problems with a statistical analysis. Even things that look simple superficially, such as when to use a mean and when to use a median, can have a lot of subtle distinctions.
- There's a lot of people out there who know one approach to data analysis very well, and they tend to apply that approach to everything they see. One person, for example, was an expert in structural equations modeling (SEM), and he told me (quite sincerely) that he didn't understand why every paper didn't use this approach. SEM can account for measurement error in your independent variables and didn't every research problem suffer from measurement error? It's like the old saying, when your only tool is a hammer, everything looks like a nail.
- Nine times out of ten (I apologize that this is a subjective estimate, but it is one that I believe in, and I think that I've seen others cite this rate), if there is a problem with a paper, it is with how the data was collected and not with how it was analyzed.
That being said, I would encourage your students to look at things like,
- was there a good control group,
- how were dropouts, exclusions, and non-compliers handled, and
- did they measure the right outcome variable.
These are aspects of the statistical DESIGN and not the statistical ANALYSIS. But if you want to talk about problems with statistical analysis, the two big problems are
Inadequate (sometimes grossly inadequate) sample sizes: There are lots of references about this, here are a few in my files:
- Why Have Recent Trials of Neuroprotective Agents in Head Injury Failed to Show Convincing Efficacy? A Pragmatic Analysis and Theoretical Considerations. Andrew I.R. Maas, Ewout W. Steyerberg, Gordon D. Murray, Ross Bullock, Alexander Baethmann, Lawrence F. Marshall, Graham M. Teasdale. Neurosurgery 1999: 44(6); 1286-1298.
- Ethics and sample size P. Bacchetti, L. E. Wolf, M. R. Segal, C. E. McCulloch. Am J Epidemiol 2005: 161(2); 105-10. aje.oxfordjournals.org/cgi/content/full/161/2/105
Failure to include confidence intervals and a discussion of clinical importance. Here are a few references:
- How well is the clinical importance of study results reported? An assessment of randomized controlled trials. Chan KB, Man-Son-Hing M, Molnar FJ, Laupacis A. Cmaj 2001: 165(9); 1197-202.
- Is 3-mm Less Drowsiness Important? Portnoy JM, Simon SD. Annals of Allergy, Asthma and Immunology 2003: 91(4); 324-325.
- Clinical vs Statistical Significance. Hopkins WG, Sportscience. www.sportsci.org/jour/0103/inbrief.htm
This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-04-01. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Critical appraisal.