StATS: Real-life examples of survey mistakes (January 31, 2006)
Tzippy Shocat was nice enough to forward a link to an article that she wrote for the iSixSigma website (www.isixsigma.com), titled "Tips for Getting the Most from Six Sigma Surveys." There were some amusing examples of bad survey practices that she cites.
A very simple rule is to only ask questions that people have the ability to answer.
A market research firm asked cell phone owners which carrier had the best customer service. Thirty-nine percent of those asked responded "I don't know." The reason for this is simple. Most people use only one cell phone and one carrier, and therefore cannot compare between the carriers. Information gleaned from these types of questions is likely to be worthless.
This sort of problem also pops up when you ask parents questions about their children. Can they respond accurately about events that may have occurred during day care hours or school hours, for example?
Also, don't stack the deck by assuming that responses will only be positive.
One service group asked, "How much did we improve relative to last year?" The available responses ranged from "somewhat" to "very much." There was no way for anyone to respond that the service quality had not improved or had worsened. Such optimistic questioning not only leads to incomplete information but also damages customer faith.
Carol Tavris cites an opposite example in "Mismeasure of Woman" [BookFinder4U link], an excellent book which someone borrowed and has not returned. As I remember it, Dr. Tavris mentions a survey of PMS symptoms that only included negative items. Is it possible that some women actual feel a sense of elation during certain stages of their menstrual cycle? If you don't ask, you'll never know.
Another important issue is changing scales from what people normally expect. A five point Likert scale, for example, will often include a not applicable or don't know response. This is traditionally coded as a 9.
One researcher used this scale: 1=non-relevant, 2=very low, 3=low, 4=medium, 5=high, 6=very high. The relatively high number of the "1=non-relevant" responses signaled to this author the problem with the scale, and the survey was redone.
Perhaps the best comment, though, was along the lines of "don't ask questions if you're not ready to hear the answer.
Surveys are often used by the service sector to ascertain customers' needs and wants. Asking customers what they would like naturally raises customer expectations. Thus you should not ask about services unless you are willing to listen and provide what customers say they want. A company unable or unwilling to eventually provide those services faces a significant drop in customer satisfaction.
This is excellent advice. The article also comments on the needs for clear and measurable goals, as well as the importance of getting a representative sample. You can read the full article at
This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Survey design.