In this article I discuss a couple of basic principles to increase the quality of survey data from surveys. Although there are many factors involved with maximising survey data quality, understanding how long a survey should be, and the order in which questions should be asked are two easily controllable factors that can have a remarkable impact on the quality of data.

In a previous post I commented on a phenomenon known as "survey satisficing", which I believe is the biggest source of error leading to misleading data in online data collection. To review, satisficing occurs when a respondent becomes tired or unmotivated to answer questions thoughtfully and accurately. It is well known amongst behavioural scientists that human beings are prone to take mental shortcuts when performing tasks. Essentially it boils down to the fact that thinking requires considerable effort, which leads to a people taking mental shortcuts. Nobel prizes have been won understanding this phenomenon (e.g., Kahneman and Tversky), so it is fairly robust. The bad news is that survey satisficing leads to unreliable data. The good news is that it can be reduced, or even eliminated, by adhering to a few principles of effective survey design.

The first principle is knowing how long a survey should be. Perhaps the biggest contributor to satisficing is survey length. This is a no brainer; have you ever tried to complete a 40 minute survey, concentrating on every question? It is mentally exhausting, which works well for spy interrogations but not trying to understand customer attitudes! So, how long should a survey be to strike a balance between obtaining the right amount of data, while not being too long? Well, I would like to give a straight answer, but the truth is it depends. For an intervening survey where browsers are invited to complete a survey on entering a website, a survey should only consist of a few questions. Respondents are visiting the website for a reason (perhaps to shop), so do not have sufficient motivation to complete more than around five questions. At the other extreme, a survey sent out to employees may be much longer. They are motivated to complete the survey because of their affiliation. A forty minute survey in this situation may not be unreasonable. As a rule of thumb (suitable for general surveys emailed to opt-in panellists), a survey should be between 20-30 questions. After this, respondents tend to start responding inaccurately to "get it over with".

The second principle is knowing which order questions should be asked. Again, this principle is linked to survey satisficing. My own research, and the research of others, has shown that affective responses to stimuli generally decrease over time. What this means is that people's motivations to answer survey questions thoughtfully and accurately decrease steadily over time. However, this downward trend can be reduced by introducing unobtrusive interruptions, or countering the slide in mental alertness with easier questions. It is beyond the scope of this article to discuss how unobtrusive interruptions can be used to increase survey accuracy, but I would like to offer advice on the ordering of questions. To counter the steady decline in mental alertness, questions should be asked from the order of most difficult (i.e., requires most thinking), through to most easy. Open ended questions for example require considerable mental effort -they should be asked first. Demographic information (name, job title, etc.) require relatively little mental effort, and should be put at the end of the survey. Put specific questions somewhere near the beginning, and general questions somewhere near the end.

Implementing these two principles in your survey design can have a remarkable impact on the quality of data collected from online hosted surveys. People are not machines, and are often not capable of concentrating for extended periods of time. This steady decline in concentration can be countered by knowing how long a survey should be, and in which order questions should be asked. Knowing this will ensure higher quality responding, enabling more accurate and honest data analysis.

An interesting report was released by Michael Conklin recently, with evidence speaking to the damaging effects of poor quality data collected online. The report suggests that any data collected from an online panel which DOESN’T use a technology to control for systemic error or bias in the data is considered poor quality. They use a technology called “MarketTools”, which is essentially the same as Deloosh’s Red Shield.

The report is available here: http://www.markettools.com/pdfs/resources/WP_TSQuality_0409.pdf

His results are not surprising to us, though will be perhaps surprising to many who collect their data from firms who make no effort to ensure data integrity (interestingly, we are not aware of any data collection firms in Australia who use technologies such as MarketTools or red Shield). Strikingly, his results found:

• Even a small proportion of bad respondents caused risk of making a bad decision from the data to increase exponentially.
• As sample size increased, risk of making a bad decision increased even more.
• Eliminating only one type of bad respondent actually
• compounded the risk. They key appears to be to remove bad respondents using a wide criteria

Fundamentally, to increase data quality, survey company needs to know:

• Who actually participated in the study
• Whether each survey taker for this study unique
• How engaged each respondent was throughout the survey?

The key finding seems to be: If your sample has 30% invalidated people, you have 2.03 times the risk of making the wrong decision—your risk is 100% higher. Sobering stuff given the increase in online data collection in recent times.

Gone are the days of cold calling a random selection of consumers during dinner time—telephone research is not only costly, its nowadays near impossible as intolerance of unsolicited communication in society has become the norm. Increasingly, marketers are using Internet surveys to conduct their market research. In comparison to more traditional methods, collecting data from online research panels is cheap and fast.Online data collection for many Marketers seems like a silver bullet—but is it really?

Although the benefits of collecting data online are clear, there is concern over the quality of data. There is some evidence to suggest that in certain conditions, data collected online may be extremely poor. The culprit responsible for introducing unacceptable levels of error into data collected online is the motivation given to panellists to participate in research. Inviting a consumer to answer a few questions about laundry powder over the phone evokes a very different motivation to respond than inviting consumers to participate in an online survey. Many, if not all market research panellists who participate in online surveys are motivated to participate by offers to win cash prizes or earn points redeemable for ipods and movie tickets. Given that Internet surveys are also convenient for the panellists, these forms of enticement invite thoughtless responses, with motivations to participate driven not by an altruistic desire to help firms improve their products and services, but simply by a selfish desire to get rewarded.

Since the recent popularity of online data collection, those in the Internet survey business have coined new terminologies to describe what we term rogue research participants. “Straight-liners” is used to describe those who click in a straight line down a survey page of radio buttons. “Speeders” are those who complete a ten?minute survey in under a minute, and “survey-pros” are those who have multiple identities and responses to increase their chances of winning a prize. Although some online research firms have attempted to reduce this type of noise in the data, existing methods are crude and inaccurate. If online market research companies make any efforts at all to reduce rogue respondents, it is often impossible to separate those who are motivated by a genuine desire participate in the research, from those who thoughtlessly click through the Internet survey driven simply by the desire to earn more points for cash or prizes.

The aims of our research in this area are twofold. First, we aim to determine the extend to which data is distorted. Studies in this area have found mixed results. We suggest contextual factors moderate the degree of systematic error leading to unacceptable levels of noise in the data. Second, we aim to develop a robust rubric of technologies and procedures to decrease the amount of systematic error born from the method of collection.

More about our research is available here: http://www.managementmarketing.unimelb.edu.au/redshield/

Jon Krosnick, professor of communication, political science and psychology at Stanford University, proposed a phenomenon of statistical survey satisficing to explain how mental effort can lead to response bias (2002). The theory presupposes optimal question answering by a survey respondent involves a great deal of cognitive effort. To ease the effort involved, people resort to heuristic processing (cutting corners) to answer questions. With weak satisficing, a respondent tends to execute all the cognitive steps required to optimize their responses, but in an incomplete way biased through heuristic processing. With strong satisficing, the respondent offers responses that on the surface may seem reasonable, but may actually have occurred with minimal memory search or information integration.Antecedents influencing survey response satisficing include the ability and motivation of the respondent, and to the difficulty of the task (questionnaire/instrument evaluation). The consequences of satisficing include: choosing “no-opinion” response options when one of the other options are true, indifferent responding to multiple objects on a response scale, and acquiescent responding. Indeed, our results in Project Red Shield suggest abnormal responding consistent with these effects. We believe response satisficing is an important consideration in the design and collection of data from online survey.

Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. T., Hanemann, W. M., Kopp, R. J., Mitchell, R. C., Presser, S., Ruud, P. A., Smith, V. K., Moody, W. R., Green, M. C., & Conaway, M. (2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, 371-403.