Big delusion

This article printed by VentureBeat is too much. The title claims "The Internet is killing off marketing surveys & it's for the best". This article is tagged as "Big Data". Big delusion is what it is.

This is a great example of the kind of revisionist history that is practised in the name of Big Data. You'd also notice that there is no data or evidence presented to support any of its many far-reaching claims.

First comes the howler:

about eight years ago, people started raising concerns about respondent quality [of traditional marketing surveys]; and as social media took off, some dared to wonder aloud whether online ratings and reviews were eliminating the need for surveys altogether.

Eight years ago would be 20062008. What happened in 20062008 that made people question instruments that have been used for decades? What kinds of concerns? How do "social media" and online ratings and reviews solve those problems? We will not learn.

Apparently, those eight-year-old concerns were not sufficient to sink the sorry enterprise until "recently". We're told -- again with no data -- that "we’re witnessing the demise of the lengthy, grid-question littered, rating-scale driven survey as we know it."

Raise your hand if you have done an online survey for a market research company. Is the survey "lengthy"? Does it contain a "grid"? Does it ask you to use a "rating scale"? Well, I thought so. Just in the last week, I reviewed an online survey design submitted by an outside agency which contains thirty questions, replete with multiple grids and multiple rating scales.

***

Later on, the author attacks blinded designs. I'm not kidding. This is the charge: '[Consumers] ask themselves, “Why should I invest time and candor in responding to questions posed by some person or entity that won’t even reveal their identity, let alone respond?”'

Apparently, tweets and online reviews are the new way. Don't worry if the tweets are unrelated to your research question or that most of those online reviews were purchased by your social media marketing agency and written by people who have not ever used your product.

And may we ask how they propose to measure "unaided brand awareness"? It is well known that the following two questions lead to vastly different responses:

(A) Name three services which can be used to create an online survey

(B) Have you heard of the following companies that provide online survey tools? SurveyMonkey, Zoomerang,...

***

Next comes the obligatory Big Data moment: "Big Data is also reducing the need for quantitatively rigorous, predictive surveys." What could that even mean? We now prefer quantitatively weak, unpredictive surveys?

The author explains, today we can just "harvest and analyze the masses of behavioral data already available." From where? Do log data or tweets inform us about attitudes, motivation, and psychology? As usual, we are asked to assume Big Data solves some problems, therefore Big Data solves those problems.

***

Now I have no doubt that "Big Data" will impact the market research field. This does not excuse poor arguments presented without evidence.

Making Big Data add value is not as simple as "harvesting". Tweets and reviews have all the characteristics of the OCCAM framework: they are observational in nature, lack controls, adapted from their original purpose, merged with other datasets, and to the deluded, they are complete (N=All).

 

Comments

[Jack]

2014-8 = 2006 not 2008?

[Ken]

It seems like no one has learnt the lessons of the early days of polling for voting intentions, where they managed to obtain a sample of only people who owned a telephone resulting in bias toward Republican voters. Looking at Twitter we probably have people who are more middle-class, more technologically aware and more opinionated than the average. Nice if that is what you want to sample, but I suspect it isn't what most companies are interested in.