Back in April 2009, I released the results of my Freedom to Surf study, finding a positive correlation between Workplace Internet Leisure Browsing, and employee productivity. Since then I have been inundated with questions from reporters and fellow academics wanting to know more. Below I have summarized some FAQs.

Q.
Can you briefly summarize the relationship you have found between WILB and productivity?

A.
My main prediction was that WILB had a positive effect on worker productivity. In my study, I found support for this prediction –those office workers who WILBed were approximately 12% more productive than those who didn’t (or couldn’t) WILB.

Q.
Why did you want to study the effects of WILB on productivity?

A.
Increasingly I noticed press reports of employees getting punished or even losing their jobs for surfing the internet while at work. I had been studying internet consumer behavior for many years, and it somehow didn't seem right that employers would assume that WILBing is akin to goofing off. In fact, based on some exploratory work I had strong reason to suspect that WILBing should have a positive effect on employee performance. My main aim is to educate employers on how they can maximise the performance of their employees.

Q.
How did you measure productivity?

A.
Productivity was measured using the Endicott Worker Productivity Scale.

Q.
Why do you think that surfing has this positive effect on productivity?

A.
There are two reasons:

1) We need breaks in our workday (much like we need to take breaks when driving a long distance). Concentration levels in the tasks we perform begin to wane fairly quickly after about an hour or so. But, not all breaks restore concentration to the same extent. For example, a walk in the forest will refresh your mind much faster and to a greater extent than simply sitting in a lunch room. WILB is similarly an enjoyable activity, which enables efficient restoration of concentration. WILB tasks such as organising financial matters online actually don’t affect productivity, but enjoyable WILB tasks such as watching uTube or updating Facebook do.

2) Overwhelmingly, management research suggests that workers who feel they have a degree of freedom in the workplace perform better than those who feel they a stifled and not in control (E.g., Google is well known for giving space to workers so they can innovate). Health benefits are also evident: Those who feel they are not in control at work have a higher chance of heart attack. The internet is now so firmly ingrained in our lives, that if you block or restrict access to it, it makes us feel like we are not in control. The result is less motivation to give 100% in the workplace, and less loyalty to employer.

Q.
In the following graph, it appears that productivity was boosted the most when people read non-work blogs (including Twitter and Facebook). Is that accurate?

A.
I believe the graphs show evidence that enjoyable (and possibly effortless) WILB activity (watching uTube, Facebooking), has a more positive effect on productivity than relatively mundane internet tasks such as internet banking and even shopping (for men at least!). My current line of study will find direct evidence of this prediction.

Who coined the term 'Workplace Internet Leisure Browsing' (WILB)?
A.
I coined this term because I did not know what else to call it. I have since learned of other academics who have studied this phenomenon who use the alternative term "Personal Web Usage".

Q.
According to your research, what are the most popular WILB activities?

A.
The following graph suggests which activities are more common than others:

Q.
How much people who use the Internet at work engage in WILB?

A.
71%

Q.
What is the limit of WILBing?

The data show up to 12% of workday WILBing could be considered OK. Any more time than that however and productivity starts to drop.

Q.

What are you working on now?

A.
My current research is looking at how to foster employee creativity and output efficiency.

Q.
Where can I get a copy of the official Freedom to Surf report?

A.
I will post the report on this blog as soon as it is published.

In this article I discuss a couple of basic principles to increase the quality of survey data from surveys. Although there are many factors involved with maximising survey data quality, understanding how long a survey should be, and the order in which questions should be asked are two easily controllable factors that can have a remarkable impact on the quality of data.

In a previous post I commented on a phenomenon known as "survey satisficing", which I believe is the biggest source of error leading to misleading data in online data collection. To review, satisficing occurs when a respondent becomes tired or unmotivated to answer questions thoughtfully and accurately. It is well known amongst behavioural scientists that human beings are prone to take mental shortcuts when performing tasks. Essentially it boils down to the fact that thinking requires considerable effort, which leads to a people taking mental shortcuts. Nobel prizes have been won understanding this phenomenon (e.g., Kahneman and Tversky), so it is fairly robust. The bad news is that survey satisficing leads to unreliable data. The good news is that it can be reduced, or even eliminated, by adhering to a few principles of effective survey design.

The first principle is knowing how long a survey should be. Perhaps the biggest contributor to satisficing is survey length. This is a no brainer; have you ever tried to complete a 40 minute survey, concentrating on every question? It is mentally exhausting, which works well for spy interrogations but not trying to understand customer attitudes! So, how long should a survey be to strike a balance between obtaining the right amount of data, while not being too long? Well, I would like to give a straight answer, but the truth is it depends. For an intervening survey where browsers are invited to complete a survey on entering a website, a survey should only consist of a few questions. Respondents are visiting the website for a reason (perhaps to shop), so do not have sufficient motivation to complete more than around five questions. At the other extreme, a survey sent out to employees may be much longer. They are motivated to complete the survey because of their affiliation. A forty minute survey in this situation may not be unreasonable. As a rule of thumb (suitable for general surveys emailed to opt-in panellists), a survey should be between 20-30 questions. After this, respondents tend to start responding inaccurately to "get it over with".

The second principle is knowing which order questions should be asked. Again, this principle is linked to survey satisficing. My own research, and the research of others, has shown that affective responses to stimuli generally decrease over time. What this means is that people's motivations to answer survey questions thoughtfully and accurately decrease steadily over time. However, this downward trend can be reduced by introducing unobtrusive interruptions, or countering the slide in mental alertness with easier questions. It is beyond the scope of this article to discuss how unobtrusive interruptions can be used to increase survey accuracy, but I would like to offer advice on the ordering of questions. To counter the steady decline in mental alertness, questions should be asked from the order of most difficult (i.e., requires most thinking), through to most easy. Open ended questions for example require considerable mental effort -they should be asked first. Demographic information (name, job title, etc.) require relatively little mental effort, and should be put at the end of the survey. Put specific questions somewhere near the beginning, and general questions somewhere near the end.

Implementing these two principles in your survey design can have a remarkable impact on the quality of data collected from online hosted surveys. People are not machines, and are often not capable of concentrating for extended periods of time. This steady decline in concentration can be countered by knowing how long a survey should be, and in which order questions should be asked. Knowing this will ensure higher quality responding, enabling more accurate and honest data analysis.

An interesting report was released by Michael Conklin recently, with evidence speaking to the damaging effects of poor quality data collected online. The report suggests that any data collected from an online panel which DOESN’T use a technology to control for systemic error or bias in the data is considered poor quality. They use a technology called “MarketTools”, which is essentially the same as Deloosh’s Red Shield.

 

The report is available here: http://www.markettools.com/pdfs/resources/WP_TSQuality_0409.pdf

 

His results are not surprising to us, though will be perhaps surprising to many who collect their data from firms who make no effort to ensure data integrity (interestingly, we are not aware of any data collection firms in Australia who use technologies such as MarketTools or red Shield). Strikingly, his results found:

 

• Even a small proportion of bad respondents caused risk of making a bad decision from the data to increase exponentially.
• As sample size increased, risk of making a bad decision increased even more.
• Eliminating only one type of bad respondent actually
• compounded the risk. They key appears to be to remove bad respondents using a wide criteria

Fundamentally, to increase data quality, survey company needs to know:

• Who actually participated in the study
• Whether each survey taker for this study unique
• How engaged each respondent was throughout the survey?

The key finding seems to be: If your sample has 30% invalidated people, you have 2.03 times the risk of making the wrong decision—your risk is 100% higher. Sobering stuff given the increase in online data collection in recent times.

 

 

Gone are the days of cold calling a random selection of consumers during dinner time—telephone research is not only costly, its nowadays near impossible as intolerance of unsolicited communication in society has become the norm. Increasingly, marketers are using Internet surveys to conduct their market research. In comparison to more traditional methods, collecting data from online research panels is cheap and fast.Online data collection for many Marketers seems like a silver bullet—but is it really?

Although the benefits of collecting data online are clear, there is concern over the quality of data. There is some evidence to suggest that in certain conditions, data collected online may be extremely poor. The culprit responsible for introducing unacceptable levels of error into data collected online is the motivation given to panellists to participate in research. Inviting a consumer to answer a few questions about laundry powder over the phone evokes a very different motivation to respond than inviting consumers to participate in an online survey. Many, if not all market research panellists who participate in online surveys are motivated to participate by offers to win cash prizes or earn points redeemable for ipods and movie tickets. Given that Internet surveys are also convenient for the panellists, these forms of enticement invite thoughtless responses, with motivations to participate driven not by an altruistic desire to help firms improve their products and services, but simply by a selfish desire to get rewarded.

Since the recent popularity of online data collection, those in the Internet survey business have coined new terminologies to describe what we term rogue research participants. “Straight-liners” is used to describe those who click in a straight line down a survey page of radio buttons. “Speeders” are those who complete a ten?minute survey in under a minute, and “survey-pros” are those who have multiple identities and responses to increase their chances of winning a prize. Although some online research firms have attempted to reduce this type of noise in the data, existing methods are crude and inaccurate. If online market research companies make any efforts at all to reduce rogue respondents, it is often impossible to separate those who are motivated by a genuine desire participate in the research, from those who thoughtlessly click through the Internet survey driven simply by the desire to earn more points for cash or prizes.

The aims of our research in this area are twofold. First, we aim to determine the extend to which data is distorted. Studies in this area have found mixed results. We suggest contextual factors moderate the degree of systematic error leading to unacceptable levels of noise in the data. Second, we aim to develop a robust rubric of technologies and procedures to decrease the amount of systematic error born from the method of collection.

More about our research is available here: http://www.managementmarketing.unimelb.edu.au/redshield/

Jon Krosnick, professor of communication, political science and psychology at Stanford University, proposed a phenomenon of statistical survey satisficing to explain how mental effort can lead to response bias (2002). The theory presupposes optimal question answering by a survey respondent involves a great deal of cognitive effort. To ease the effort involved, people resort to heuristic processing (cutting corners) to answer questions. With weak satisficing, a respondent tends to execute all the cognitive steps required to optimize their responses, but in an incomplete way biased through heuristic processing. With strong satisficing, the respondent offers responses that on the surface may seem reasonable, but may actually have occurred with minimal memory search or information integration.Antecedents influencing survey response satisficing include the ability and motivation of the respondent, and to the difficulty of the task (questionnaire/instrument evaluation). The consequences of satisficing include: choosing “no-opinion” response options when one of the other options are true, indifferent responding to multiple objects on a response scale, and acquiescent responding. Indeed, our results in Project Red Shield suggest abnormal responding consistent with these effects. We believe response satisficing is an important consideration in the design and collection of data from online survey.

Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. T., Hanemann, W. M., Kopp, R. J., Mitchell, R. C., Presser, S., Ruud, P. A., Smith, V. K., Moody, W. R., Green, M. C., & Conaway, M. (2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, 371-403.