StATS: What is an adequate sample size for establishing validity and reliability? (April 9, 2007)

Someone from Mumbai, India wrote in asking whether a sample of 163 was sufficiently large for a study of reliability and validity. This was for a project that was already done, and this person was worried that someone would complain that 163 is too small.

To demonstrate whether a sample size is adequate after the sample has already been collected, you need only to produce confidence intervals for your statistical estimates. There are a large number of reliability and validity analyses at your disposal, but many of these involve the computation of a correlation coefficient of one type or another. If your confidence interval for those correlation coefficients is reasonably narrow, then your sample size is adequate. If it is so wide that you cannot make any definitive conclusions, then your sample size is inadequate.

If, for example, you have a confidence interval for a correlation that ranges from 0.2 to 0.9, that interval is so wide as to be meaningless. You can eliminate the possibility of a zero correlation, of course, but the goal of most reliability and validity studies is to establish a strong correlation. Here you have so little data that you can not distinguish between a very weak correlation 0.2, a moderately strong correlation (0.5) or a very impressive correlation (0.9). Without the actual data, I cannot compute any confidence intervals, but with 163 patients, I suspect that they are more than narrow enough to establish an adequate sample size.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Measuring agreement or Category: Sample size justification.