Open-ended questions on a survey (March 25, 2005)
This page is moving to a new website.
No one seems to talk about how to handle those pesky open-ended questions you see on a survey. I usually hold my breath and hope that the researcher doesn't think to mention it. Alicia O'Cathain and Kate Thomas address this important issue in a recently published article,
and they gently scold us for ignoring an important source of information.
Researchers may use general open questions without giving much thought to why they are doing so, simply including the question because it is usual practice. Or they may be driven by a desire to offer respondents an opportunity to voice their opinion. Closed questions represent the researchers' agenda, even if they have been developed through listening to people's views in focus groups and depth interviews. The use of 'any other comments' may redress the power balance between researchers and research participants. Respondents may take this opportunity to ask for clarification or information about a health issue or health service, or voice concern about the research. If researchers include a general open question for this reason then they will need to consider how best to respond to individuals about such queries and concerns.
What should you do with this data?
General open questions may produce little more than the closed questions on the questionnaire and rather than considering it unethical to analyse these responses, a more appropriate strategy might be preliminary analysis involving reading the responses so that the researcher can consider the contribution they make to the study overall. If the comments merely corroborate or slightly elaborate upon the answers to closed questions, then formal analysis may not be worthwhile [23]. It may be good practice to report within publications that the responses to the general open question did not provide additional information to the closed questions. It is where they offer insights or issues not available in the closed questions that formal analysis could be considered good practice, even if the role of this analysis is to identify hypotheses or questions for further study. Formal analysis may be prompted by either the strength of numbers making particular comments, or the strength of feeling within a small number of the comments.
A first step is to see if those who responded to the open ended question differ in their demographics from those who did not. O'Cathain and Kate Thomas cite an example of a nursing survey where the proportion of open ended respondents increased among those who were dissatisfied with their jobs.
The authors also suggest content analysis.
Content analysis may be undertaken where the researcher takes the following steps:
1. Reads a sub-set of the comments.
2. Devises a coding frame to describe the thematic content of the comments.
3. Assigns the codes to all the comments. The coding frame can be applied using software designed for this purpose or manually. Two coders may be needed to test the reliability of assigning codes.
4. The codes can be entered into a statistical package alongside the data from the closed questions and treated as variables in a quantitative analysis.
This is not exactly a formal qualitative analysis, but it is similar enough that you may want to consult with an expert in this area.
You can and should use quote of particular comments that give your reader an idea of what the general tenor of the comments was like. Be sure, though, that you don't include any information that might violate patient confidentiality.