Why Red Shield is Important
Project Red Shield has important implications for increasing the quality of
online survey data for academics and research practitioners alike. In these times of hyper-competitiveness and mass consumerism, market research is critical to a company’s success. Since the emergence and rapid diffusion of the Internet, methods of conducting
market research have evolved to increasingly rely on online data collection. In comparison to more traditional methods of data collection, online research surveys are relatively cheap, quick to implement, and capable of harvesting large amounts of data in short periods of time. For these reasons, market researchers and academics alike have embraced the Internet as a channel for conducting research. Accordingly, Market Research Agencies have redesigned their operations to fulfill the demand.
Advances in communications technologies, coupled with an increased reliance on
consumer data to guide strategic decision making, consumers have become more protective of their privacy and therefore more intolerant of unsolicited invitations to participate in voluntary surveys. As a result, traditional methods of data collection such as telephone research have become not only prohibitively expensive, but victim to the same systematic response bias present in online survey research data. Given the resistance to collecting data through unsolicited telephone calls, the types of people willing to respond to telephony nowadays likely possess unusual traits and behavior not representative of the population.
Preliminary research suggests collecting data online introduces significant error into the data, resulting in potentially misleading results. The reason for this error is thought to emanate from motivations to participate. Motivation is an important antecedent to data integrity. For example, the motivation a consumer has for answering questions about laundry powder over the phone may evoke a different response than the motivation to answer the same questions in an online survey. Many, if not all market research panelists who participate in online surveys are motivated to participate by offering cash prizes or points redeemable for cash or prizes. In contrast, the motivation to respond to telephone or paper and pencil type surveys in altruistic in nature. The motivation to participate for online survey data is reward based, which invites thoughtless responses, thereby introducing unacceptable amounts of error into the data. Given the amount of noise introduced into the data from online data collection, it is reasonable to expect the results of data collected online are not reliable or valid. Worse, but not much worse, conclusions drawn from the analysis of online survey data may be inaccurate and strategic decision may be misguided. The Red Shield system corrects for collection method error bias. Online data collected using a Red Shield system is invariant to traditionally response-bias robust methods such as paper-and-pencil surveys, while retaining the benefits of fast collection speed, low cost, and large data set acquisition.
History
Red Shield began as an information system to increase the quality of online survey responses at the school of information management, Victoria University Wellington. The developer of Red Shield, Brent Coker, was concerned that the quality of data he was collecting for an online scale development project was not sufficient to ensure valid and reliable results. Although standard techniques such as reverse coding appeared to reduce some systematic error, dimensional analysis of a semantic differential scale suggested significant error was confounding the results. Most notably, the results of exploratory and confirmatory factor analysis did not seem consistent with earlier studies that had used traditional paper and pencil techniques. In order to ensure proper calibration of the scale, Brent developed a set of procedures to determine if the source of error emanated from the method of collection—a public survey posted on the Internet. After implementing the system, analysis of new data collected suggested significant systematic error from the method of collection. The system was successful in reducing the error to acceptable levels. After refinement the information system was named Red Shield after a popularly used programming icon used to denote data integrity.
Red Shield System Overview
The Red Shield System consists of three separate but dependent processes.
-
Process 1: Pre-survey Quality
Ensuring accurate results from quality data begins with the sample. The first stage of Red Shield includes a set of systems and processes to ensure respondents are representative of the population to be sampled, and genuinely interested in participating in market research. Broadly, the systems includes four main controls including: 1) outlier response history quarantine; 2) subject management control; 3) participation history regression; and 4) segmentation variable replenishment.
-
Process 2: Concurrent Quality
Recent research suggests one of the biggest determinants of high quality responding is the look and feel of the survey. Plain or poorly designed web surveys not only dramatically increase the likelihood of non-completion, but also the likelihood of indifferent or dishonest responses. Process two uses human-computer interaction theory to ensure the design of the online survey is optimal in terms of eliciting accurate and reliable responses.
-
Process 3: Post-survey Quality
Process three incorporates a system of raw data quality analysis. Regression type techniques are also used to detect the presence of predefined sources of error incorporated in Process 2. Respondent history analysis is conducted, and statistical checks such as outlier analysis are compared against level 1 and 2 checks. Depending on the system used, it is possible to incorporate data transformation in the process.