This page has moved to my new website.
I've corrected a broken link on this article, which was originally published at my old website,
A colleague sent me a very nice paper,
Design and analysis of pilot studies: recommendations for good practice. G. A. Lancaster, S. Dodd, P. R. Williamson. J Eval Clin Pract 2004: 10(2); 307-12. [Medline] [Abstract]
that covers some of the same ideas in my web page, Stats: Designing a pilot study.
This is a very well researched article and has some excellent recommendations. Right in the abstract, the authors warn against just slapping on the label of pilot study when your sample size is too small. The authors also state in the abstract that
A well-conducted pilot study, giving a clear list of aims and objectives within a formal framework will encourage methodological rigour, ensure that the work is scientifically valid and publishable, and will lead to higher quality RCTs.
They mention seven major objectives of a pilot study
Sample size calculation. Before you can calculate power for a randomized control trial, you need to know the variability of your outcome measure if it is continuous, or the proportion(s) you expect to see in the control population if the outcome measure is categorical. The authors cite Browne 1995 to support the use of a sample size of at least 30 patients in this scenario and suggest that the 80% upper one side confidence limit be used rather than the estimate itself.
Integrity of study protocol. If you are planning a large trial, especially a trial that will recruit from several different centers, a small scale run of the full protocol will help you assess the logistics of the study. The authors explain that the pilot can help you evaluate things like the storage and testing of equipment and materials.
Testing of data collection forms or questionnaires. I heartily endorse this use for a pilot study. The authors point out that testing is especially important for forms that the patients themselves have to complete, or data collection forms used by several different people. The authors say you should evaluate comprehensibility and consistency. They also point out that pilot testing of a survey is not a substitute for assessing validity and reliability for that survey.
Randomization procedure. This is a bit surprising, and I would have lumped it in with integrity of the study protocol. I do have to admit, though, that randomization has become more complicated than just the simple flipping of a coin. The need to conceal the randomization list from those who recruit patients into the study leads to the use of sealed envelopes and 24 hour randomization hot lines.
Recruitment and consent. It is often harder to recruit subjects into a research study than you think. The authors point out that a pilot study will give you a good estimate of how long it will take to recruit the subjects you need for your research. They cite Ross 1999, which indicates that slower than expected recruitment of subjects is one of the most common reasons why studies are abandoned early.
Acceptability of intervention. The intervention you have planned for your research subjects might not be palatable. The authors point out that this may especially be a problem with drugs given to children, who often cannot or will not tolerate medications that adults take with no difficulty.
Selection of most appropriate outcome measure. This is also a bit surprising. The authors claim that when you need to choose between two or three different outcome measures, a pilot study can help you make that choice. I suspect, however, that most if not all of the considerations for selecting an outcome measure would come from external information and not from a pilot study.
The authors then warn against including pilot data in the full study. I would tend to disagree. It's hard enough to get good data, why toss away the extra data that you get in a pilot study. You do need to be careful about this, especially if you sharply curtail the planned study based on findings in the pilot study.
This paper is a very good effort to outline the scientific rationale for pilot studies and should provide helpful guidance for committees that have to review these types of research.