StATS: Confusion about p-values (January 18, 2005)

Someone wrote to me with a statement that represents a commonly held, but false belief. He stated, in effect, that a p-value of 0.06 means that there is only a 6% probability that the null hypothesis is true.

While the p-value is indeed a probability, it represents effectively a probability about the data, given that the null hypothesis is true. The statement above is effective a statement about the probability that the null hypothesis is true, given the data. It's a rather subtle reversal of a conditional probability, but one with important ramifications.

The only way that you can compute a probability for the null hypothesis is to use Bayesian statistics. This involve specifying a prior probability for the null hypothesis, something that is quite controversial.

Bayesians delight in computing things like the probability that the sun will rise tomorrow. It takes a different way of looking at things, but I have found that the Bayesian approach becomes more attractive for some of the more complex problems in Statistics.

I have a bibliography of references on p-values (and confidence intervals), and have made a feeble attempt to accumulate a few references about Bayesian statistics. When I get the time, I would like to write a page on some simple Bayesian models. There is an entire software program devoted to Bayesian analysis, BUGS (Bayesian inference Using Gibbs Sampling):

The Edstat-l listserv has had an active debate about the weaknesses of p-values and null hypothesis significance testing starting in late December.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Pvalues.