StATS: Selective reporting of research findings (March 14, 2005)

I have talked extensively about publication bias in my weblog

and address this issue in detail on my book on Statistical Evidence which I hope to finish sometime soon.

A related problem is when researchers decide to report or not report particular data analyses based on how impressive the results appear. This is called publication bias in situ (PBIS) by Phillips 2004.

Examples of this problem occur when researchers have discretion in

(1) Which exposures and outcomes to consider in datasets with many variables.
(2) Which functional forms to use to represent variables (e.g., how to divide continuous variables into categories).
(3) Whether to conduct separate analyses by subgroup, and which subgroup results to emphasize.

and they then focus their attention on that analyses which produce the more interesting results. The more interesting results, of course, are those results which tend to show that a new drug or therapy is better and the less interesting results are those that tend to show that the new drug or therapy is about the same as the standard drug or therapy. I make a joke about this sort of thing when I tell people how wonderful statistical software is. It allows you to run ten separate analyses and then choose the one that gives you the smallest p-value.

Evaluating PBIS is difficult because you normally don't have access to the original research protocol to see what the researchers had originally intended.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Publication bias.