Statistical Evidence. Chapter 2. Who Was Left Out?

2.0 Introduction

Research studies often have a narrow focus, but sometimes it can be too narrow. When too many patients are left out, those who remain may not be not representative of the types of patients you will encounter.

Case study:  Nicotine patches

In a study of teenage smokers (Smith 1996), researchers recruited 22 volunteers from five public high schools in the Rochester, MN area for participation in a smoking cessation program, involving behavioral counseling, group therapy, and nicotine patches. Researchers measured the number of cigarettes smoked, side effects, and blood levels of nicotine.

The purpose of the research was to evaluate "the safety, tolerance, and efficacy of 22 mg/d nicotine patch therapy in smokers younger than 18 years who were trying to stop smoking." The authors also listed a secondary goal, "to compare blood cotinine levels, nicotine withdrawal scores, and adverse experiences with those of adults obtained in previous patch studies." Cotinine is a metabolite of nicotine and provides a useful objective measure of cigarette smoking. It also allowed the authors to examine whether nicotine toxicity was an issue.

This study did not include major segments of the teenage smoking population. The study included only white subjects because there were too few minority students in the Rochester area. Subjects had to get parental permission, excluding smokers who wished to keep their habit secret from their parents. Subjects were also volunteers, and thus could be considered more motivated to quit than the typical teenage smoker.

The study also had a serious drop out rate. Of the presumably thousands of teenage smokers in the Rochester, Minnesota area, only 71 volunteers responded to the initial call for subjects. Of the 71 volunteers, 55% met inclusion criteria. Of the remaining 39, 44% declined to attend the initial meeting. Of the remaining 22, 14% were non-compliant. Of the remaining 18, 39% failed to respond to the one year survey. Only 11 completed the entire study (50% of those who started the study; 28% of those meeting inclusion criteria; 15% of the initial volunteers.)

This study had a serious problem with who was left out. The large number of subjects who did not get into the study or who did not complete the study makes it hard to generalize the findings of this research.

Who was left out? What to look for.

When you are trying to figure out who was left out and what impact this has, ask the following questions:

Who was excluded at the start of the study? In a desire to create a nice clean homogenous research study, researchers may apply rigid and unrealistic entry criteria. The patients excluded can often have a different prognosis than those who make it into the study. This exclusion can make it difficult to extrapolate to the types of patients that you normally see.

Who refused to join the study? Almost all research involves the informed consent of volunteers. Many potential patients can and do refuse to participate in research studies. This can dramatically affect the results of the research.

Who dropped out during the study? Not everyone who starts out in a research study will finish it. Volunteers always have the option of withdrawing their consent to participate at anytime and some patients will miss their follow-up appointments because they moved or they just plain forgot. If these dropouts have a different prognosis, then?

Who stopped or switched therapies? If there are compliance issues, handle the non-compliant patients carefully. Patients who have problems with compliance will also often have trouble with other self-care habits and thus be at greater risk for adverse outcomes. Excluding non-compliant patients can lead to some serious biases.

[Remainder of this material deleted out of respect for the publisher's copyright.]

This work is licensed under a Creative Commons Attribution 3.0 United States License. It was written by Steve Simon on 2005-06-03, edited by Steve Simon, and was last modified on 2008-11-25. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Statistical evidence