StATS: Choosing between two conflicting analyses (May 16, 2007)

Someone wrote in and asked about an analysis where there was only a limited amount of data. The simple analysis using an odds ratio produced a significant result (p=0.048). A referee suggested that they run a logistic regression model adjusting for two covariates. These covariates were not imbalanced between the two groups. With the logistic regression model, the p-value changed from 0.048 to 0.06. The researcher wanted to know if it was appropriate to adjust for covariates in such a small data set. Here's what I suggested.

If I were you, I would write the paper to include both analyses. Point out that the effect seen in your study is marginal (both 0.048 and 0.06 represent marginal p-values) and that further research is needed.

Whether one analysis or the other is correct is not really relevant. They both are saying the same thing, so that makes your case stronger in the sense that two different analyses provide comparable conclusions.

The only reason you dislike presenting the competing analysis is that some critics will use the p-values to argue (incorrectly) that there is no difference between these two groups. There is certainly data that is strongly suggestive of a difference, however, and most readers of your article will recognize this whether the p-value is 0.048 or 0.06 or anything in between.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Pvalues.