Problems with a very small sample

Steve Simon

2001/10/11

[StATS]: Problems with a very small sample (October 11, 2001)

Dear Professor Mean, Are there problems with a very small sample? Can the t-test be used with a sample of just three subjects? – Anxious Abdelwahab

Dear Anxious,

It sounds like you got your funding from the Jack Benny Research Foundation.

You can use the t-test, but the question is should you? If you want your power to be 80%, a sample size of 3 is only capable of detecting a 2.3 standard deviation difference (paired test). The situation is even worse for the two-sample t-test. To give some perspective, most effect sizes seen in published research range from 0.2 to 0.8 standard deviations.

Unless you know for a fact that the differences you are likely to see are huge (and I mean Godzilla sized differences), then you should not run the experiment. You will almost certainly accept the null hypothesis, and you won’t know if it is because of your small sample size or the fact that there really is no difference. In other words, the experiment will not provide any useful information.

This page was written by Steve Simon while working at Children’s Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children’s Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Small sample size issues.

issues](../category/SmallSampleSizeIssues.html). this one at [Category: Small sample size with general help resources. You can also browse for pages similar to Children’s Mercy Hospital website. Need more information? I have a page reproducing it here as a service, as it is no longer available on the Hospital. Although I do not hold the copyright for this material, I am This page was written by Steve Simon while working at Children’s Mercy