StATS: Flaws in a research paper (created 2000-01-27).

Dear Professor Mean, A small group I've been teaching has gotten extremely interested in how to decide when there are too many flaws in a paper which would completely invalidate (and circular file) it.

Gee, how many bugs do you have to find in a software program before you decide to uninstall it?

There is no obvious answer to either question, of course, but here are a few things to think about.

First, you need to respect the fact that there is more than one possible use for a paper. So a paper that might be useless for giving someone guidance on how to treat their patients, might still be very useful for someone who is trying to design a similar (and hopefully better) research study.

Second, context is critical. You might be more tolerant of flaws, depending on the problem at hand. The original paper that showed the Mozart effect on IQ, for example, probably wasn't all that convincing, but if it encouraged a few more of us to listen to classical music, what's the harm? I'd be a lot more cautious about changing my lifestyle if a paper came out that said chocolate was a carcinogen.

These are silly examples, of course, but the point is that you need to examine the context of the problem when you are deciding how strict a standard to apply to research. At a minimum, I would think you would have to look at the severity of the illness, the cost and side effects of the treatment, and the availability of alternate treatments.

Third, there is some pretty solid evidence (MacCoun 1998) that indicates that we tend to look harder for flaws in research that defies our personal intuition and beliefs. A meta-analysis would use blinding to minimize this bias, but that's not possible in a journal club. So you need to take a step back and ask yourself if you have been equally critical of studies in different areas.

Fourth, I have a tendency, and I think others do also, to slip sometimes into a mindset of research nihilism. A lot of research papers have a lot of flaws, so you start to think all research is garbage. You don't want your skepticism to turn into cynicism.

Finally, try to avoid false or unproductive dichotomies. Some people like to classify everything they see and slap on labels like "good research" and "bad research". There are shades of gray to everything, of course, and it takes a lot of skill to appreciate these shades of gray. Deciding how dark the gray has to be before you call it black may not be a good use of your time. Rather than deciding whether to junk a study, you might be better off asking what type of corroborating evidence would help you make a better decision about this problem.

Further reading

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Critical appraisal.