StATS: Designing a questionnaire (December 24, 2004)

I'm behind in my reading of the British Medical Journal, and the first issue I looked at today has a gem of an article.

Questionnaire development is something that many researchers do, but few researchers do well. Here's a quick summary of the questions this paper raises.

What information are you trying to collect? A clear plan at the outset can prevent a lot of grief later on.

Is a questionnaire appropriate? The authors don't really explore the alternatives, especially personal interviews and focus groups. Instead, they list several examples where a questionnaire is inappropriate.

Could you use an existing instrument? This is an excellent point. We spend way to much time re-inventing the wheel rather than applying or adapting existing questionnaires.

Is the questionnaire valid and reliable? Whole books can (and have) been written on this subject. The authors do little beyond defining the terms. The time spent on establishing validity and reliability should represent at least 10% of your research effort, and in some cases might warrant an entirely separate project.

How should you present your questions? The authors contrast open and close ended questions in a nice table. They also mention two words, "frequently" and "regularly" that often lead to ambiguous and confusing situations. The authors encourage the use of reverse scaling, which means that not all questions are worded in a positive manner. This is possibly good advice, but it could possibly backfire as well.

Apart from questions, what else should you include? The authors point out the importance of an introductory letter or information sheet.

What should the questionnaire look like? The physical layout of the questionnaire is very important. A haphazard design will decrease your response rate and/or harm the quality of your responses. The authors suggests a rule of thumb that questions should be twelve words or less unless they are of a sensitive or personal nature.

What approvals do you need before you start? In many situations, you should get IRB approval if you are in the United States, or approval from a similar research ethics committee. A poorly designed questionnaire is at a minimum a breach of courtesy since it wastes the time of your research subjects. Questionnaires also have the potential of causing undue offense if they handle sensitive health care matters in an impersonal way. The is also a confidentiality risk with pretty much all questionnaires, but some with certain topics (such as sexually transmitted diseases, or drug/alcohol abuse), this concern is especially acute. The authors don't mention another important risk. If a questionnaire identifies a patient as having a risk of committing suicide, the researcher needs to have a plan in place  to provide appropriate referrals.

In short, this is a very useful list of questions. Some of the details are difficult to include in a journal article because of space limits. I had been planning to teach a course on questionnaire design next year, and I think that this paper might be the only handout I would need for the class.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Survey design.