Testing baseline imbalance in a randomized study (January 19, 2005).
This page is moving to a new website.
Randomization will roughly balance out the covariates between the treatment group and the control group because of the law of large numbers. Once in a while, though, an important amount of covariate imbalance will creep into a randomized study. Just as a flip of 100 coins will not always yield exactly 50 heads and 50 tails, a randomized study will not always yield perfect covariate imbalance.
When such an imbalance occurs, it is called a chance bias or accidental bias. It can seriously affect the quality of your analysis.
One reference,
argues that the likelihood of covariate imbalance is small if the total sample size is 200 or more, while another,
argues that a total sample size of 20 provides inadequate protection against even a single covariate imbalance, while a total sample size of 80 protects even against imbalance across multiple covariates.
It is always good to examine the covariates in a randomized study to insure that you have reasonable balance. But most statisticians recommend that you avoid a statistical test of significance for this situation.
One reference,
makes the point that not only should you avoid a hypothesis test, but that you should also avoid visual inspection of differences between groups.
As the trial size increases the absolute size of imbalance in baseline characteristics will reduce owing to reduction in sampling error. Hence the absolute magnitude of any chance bias in outcome will tend to decrease with sample size. Nevertheless, the possible chance bias on a statistical test of an outcome measure does not change with sample size,5 so chance bias is as much of a possibility for large trials as for small. An imbalance of a given absolute size will have a greater effect on the statistical tests for larger sample sizes than for small. This means that inspection of the distribution of baseline variables between groups is also an inappropriate method on which to base the decision to adjust or not adjust a statistical test of a trial outcome.
These authors recommend that you select baseline covariates prior to data collection and adjust for them in an analysis of covariance model.
The reason to avoid a test of significance for baseline values in a randomized trial is that such a test addresses only whether there was a flaw in the randomization process. Perhaps the numbers that you thought were random really weren't so random after all.
A more serious concern is subversion of randomization, where some of the physicians recruiting patients for the study will try to steer certain types of patients towards one treatment or away from the other treatment. They could do this subconciously by applying the inclusion and exclusion criteria more rigidly for certain types of patients if they know that a certain treatment will be assigned. They could also deliberately delay a patient's entry into a trial for a day or two so that a different patient gets the less preferred treatment.
Physicians who deliberately tries to subvert the randomization process are behaving unethically. They recognize that equipoise is not applicable for this particular patient, so they should offer the patient the preferred treatment outside of the randomized trial.
For this reason, the randomization list should be concealed from the recruiting physicians, so they will not know what treatment is coming up next. Concealing the randomization list is always possible, even when the study is not blinded.
So a test of significance is only appropriate if you think there was a flaw with the randomization process, such as a physician's attempt to subvert the randomization of the study.