Big Data, Plainly Spoken (aka Numbers Rule Your World)

An expanded version of chart 3

This is an expanded version of Chart 3 that appeared in my FiveThirtyEight article:

These were the five routes in which following Kayak's recommendations would get me to a lower price than the 14-day-out price for each flight. The purple dot was when Kayak first suggested buying. The gray dotted line was the price on my first day of search.

A few key observations here:

• The high variability of prices along many routes
• The variability of the price variations: Los Angeles to Honolulu was rather flat throughout; Los Angeles to Chicago experienced two stages of price hikes; New York to Boston showed a parabola.
• The algorithm does not always find the lowest possible price but did pretty well. As I explained before, using  ex-post data to complain about decisions that must be made without such data is bad form.

Notes on evaluating predictions, or Background to my airfare predictor article

My article on whether we can trust airfare prediction models is published today at FiveThirtyEight, the new data journalism venture launched by Nate Silver after he moved to ESPN.

This topic was originally conceived as a chapter of Numbersense (link) but I dropped it. As I have noted in my review of Nate Silver's book, he has a keen interest in evaluating predictions, and not surprisingly, he encouraged me to get this piece done.

Putting Big Data to the Test

Just like Google Flu Trends (link), Oren Etzioni's Farecast has been held up as a Big Data success story. I have been a Farecast user for years, and though I use the tool, I've always wondered how accurate are those predictions. If you're a user, you've probably wondered as well. I have also complained that Big Data practitioners are too lax in offering quantitative evidence for their Big Data projects--it's a bit ironic when we tell others to use data and throw away their gut feelings.

One of the reasons for this oversight is that it is hard work to evaluate predictions properly. In this post, I will cover how I designed the evaluation strategy.

Humility First

The first rule of evaluation is to check your ego at the door. The goal of evaluating a predictive model is to measure how well it performs. It is tempting for the evaluator to reinvent the wheel, devise a new way of predicting, and prove its superiority--but that is not evaluation. The evaluator is like a quality-control analyst, or a code reviewer.

Assumptions, Assumptions, Assumptions

One of the core messages of Numbersense (link) is that every analysis has assumptions, often called "theory". People who think their analyses contain no assumptions are usually ones who haven't thought carefully about their models. Making "no assumptions" is itself an assumption. In the same way, evaluating models require assumptions, and lots of them! Bear this in mind as you keep reading.

What to Compare to

In my article, I explained why the right comparable is the most realistic alternative strategy for purchasing air tickets if one were to not use Kayak/Farecast. This is one of my most important assumptions and it took me a while to figure this out.

At first glance, you might think a "natural" comparable would be the actual price trajectory for a given route for a given travel period. In other words, consider when the algorithm recommended buying and when it recommended waiting and judge it based on whether the algorithm led you to the lowest price during your search. 

Using such a metric is to commit a form of hindsight bias. You have to remember that any algorithm (or human) must make the decision to buy or wait when the future prices are not known yet. In addition, we expect that there will be substantial price volatility in the future. If we were able to re-run history many times, the price paths would be different and the algorithm's performance would also vary. When we are staring at the realized price path (ex-post), it is easy to forget about the underlying volatility.

A worse problem with judging the algorithm against the lowest possible prices is that there may be no way to get to that lowest price! Remember to check your ego, you are the evaluator, not the modeler. The question is are you able to find an existing alternative solution that would lead you to those lowest prices without cheating and using future price data?

Don't Compare to Imaginary Toys

Instead of the theoretical maximum (i.e. the model that always finds the best possible price), let's consider the existence of a Best Realizable Model (BRM). Let's suppose its performance will be 70% of the theoretical maximum. Then, in theory, we can compare Kayak/Farecast to this BRM.

The catch is we don't know anything about BRM. In particular, we don't know if it performs at 70% or 30% of the theoretical maximum. If Kayak/Farecast gets to 25% of ideal and BRM 70%, then Kayak/Farecast is pretty poor. But at the same 25%, if BRM performs at 30%, then Kayak/Farecast is impressive.

Neither the theoretical maximum nor the Best Realizable Model should be used in this evaluation, simply because they are not real strategies, and just imaginary toys.

Don't Bow to Random 

At the other extreme, modelers like to compare their algorithms to the "random" strategy. In the case of airfare prediction, one such strategy might be picking a random number of days before departure and taking the lowest fare on that day. The random strategy is throwing a die, and using no skill at all. This is unsatisfactory because it creates too low a bar. 

Compare to Next Best Alternative

In my view, a far better approach is to figure out what you'd have done in the absence of the predictive model. In my case, I typically wait till two weeks before departure, and so that is the comparable.

Some readers have commented that they tend to buy three or four weeks before departure, and one pointed to an analysis claiming that 54 days is exactly the right moment to get the cheapest fare. This takes us back to the point raised earlier, that every evaluation strategy makes assumptions. If we do an analysis starting 30 days out, a different reader will object, saying he or she typically purchases 21 days out. (Remember, though, the earlier you start this exercise, the longer you have to track day-by-day wait or buy recommendations.)

***

Another "obvious" (until you think about it) evaluation criterion is to focus on the probability estimates themselves, rather than the outcomes they produce. The probability estimates are given in the form "there is 79% chance that the fare would go up by $20 or more in the next seven days." These forecasts are, of course, given for every route, for every departure date, and for every date of search. 

It may take more than a moment's thought but such probability estimates are essentially impossible to verify. The forecast statement basically asserts that if the same forecasting situation arises many many times, they would be right about 79% of the time. But we can't in real life replicate the same forecasting situation many many times. 

One way around this is to forget about real life, and check the probability estimates by simulating many future worlds. A major problem of this approach is that even if you can show that the probability estimates are good relative to those simulations, the travelers who use these forecasts still may not save money. Again, it lacks the "what would you otherwise have done" dimension.

As you dig deeper, you'll find more tricky issues. One is the continuous time (24/7) nature of online travel search. If you need to measure whether "the fare went up $20 or more in the next seven days," you'd have to be monitoring fares continuously over those seven days. To make things even more complicated, in each of those seven days, for the same itinerary, Kayak/Farecast is updating its prediction in a rolling 7-day window.

***

One other consideration I'd like to cover. Nate Silver is famous for predicting all 50 states correctly. Remember though that anyone who has the slightest knowledge of US politics can predict 40 out of 50 states--where he demonstrated his skill was in those swing states.

Now in the context of airfare prediction, if it were true that prices are much cheaper two or three months prior to departure, then people who would be purchasing in those time frames do not really need an algorithm to help them. It is important to test predictive models under situations in which they are most likely to demonstrate their skills. I would therefore recommend that you evaluate airfare predictions closer to the departure date when you'd need it most.

***

As you can see, evaluating predictive analytics is filled with challenges. But as I demonstrate here, it can be done. It should be done.

The reality is most A/B tests fail, and Facebook is here to help

Two years ago, Wired breathlessly extolled the virtues of A/B testing (link). A lot of Web companies are in the forefront of running hundreds or thousands of tests daily. The reality is that most A/B tests fail.

A/B tests fail for many reasons. Typically, business leaders consider a test to have failed when the analysis fails to support their hypothesis. "We ran all these tests varying the color of the buttons, and nothing significant ever surfaced, and it was all a waste of time!" For smaller websites, it may take weeks or even months to collect enough samples to read a test, and so business managers are understandably upset when no action can be taken at its conclusion. It feels like waiting for the train which is running behind schedule.

Bad outcome isn't the primary reason for A/B test failure. The main ways in which A/B tests fail are:

1. Bad design (or no design);

2. Bad execution;

3. Bad measurement.

These issues are often ignored or dismissed. They may not even be noticed if the engineers running the tests have not taken a proper design of experiments class. However, even though I earned an A at school, it wasn't until I started running real-world experiments that I really learned the subject. This is an area in which theory and practice are both necessary.

The Facebook Data Science team just launched an open platform for running online experiments, called PlanOut. This looks like a helpful tool to avoid design and execution problems. I highly recommend looking into how to integrate it with your website. An overview is here, and a more technical paper (PDF) is also available. There is a github page.

The rest of this post gets into some technical, sausage-factory stuff, so be warned.

***

Bad design is when the experiment is set up in such a way that it does not provide data to answer the research question. I will just give one example here, which is one of many, many ways to fail.

Let's say you want to test changing the text on your registration button from "Sign up Now" to "Join Free". You run a standard A/B test randomizing your visitors into two paths, everything else being the same except the different button text. After you present the test result, the business owner asks to look at geographical segments. That is when you realized that your visitors are split 40% English and 60% non-English. Your site doesn't yet have translated pages so everyone sees the English version. What's the problem?

The problem is noise in your data. Since you don't have translated pages, it is very likely that your registrations would be English-speakers-biased even though your traffic is more international. A better test design should have restricted the test to only English-speaking countries. Because the change in the button text is not likely to influence non-English speaking visitors, when you average those results into the average registration rate, they dilute the signal coming from English speakers. You are at risk of making a false negative error.

The examples of failed designs are endless. I will discuss other examples in future posts. Worse than failed designs is no design at all. The Facebook team quotes Ronald Fisher: "To consult the statistician after an experiment is finished is often merely to ask him to conduct a post-mortem examination. He can perhaps say what the experiment died of."

***

Worse than bad or no design is bad execution of a good design. The essence of designed experiments is controlling the system--making sure everything else is the same so that we can isolate the effect of the treatment being investigated. Let's just say web sites are complex beasts and bad execution is the norm rather than the exception.

Start with the experimental unit. How do you identify individuals? The simplest way is to assign user ids. But user ids are only available when the user logs in. If the same user performs activities in both logged-in and logged-out states, you're out of luck. The next technology available is the cookie. But cookies can be cleaned out, and cookies do not persist across different devices or browsers. There are also caching and other technical complications--solutions for other engineering problems that adversely affect A/B test execution.

At other times, your experiment may not involve individuals. It may involve impressions (page views). You might want to randomize the design of a particular web page regardless of who loads it. There are no ready made identifiers for impressions.

Next is randomization, which is at the core of the A/B testing methodology. Through the years, I have met many formulas for computing random numbers on the fly, half of which are probably not truly random. We should have a  standard for this.

Another issue relates to understanding the structure of the website, and its pathways. This is one of the toughest things to do for a large website. It's easy to unintentionally move some element of a page, not realizing that some visitors pass by that element on the way to the conversion page several steps later. Frequently, it's not that people are negligent--people really didn't know such a pathway existed when the test was designed.

Further, if you run many simultaneous tests, it is extremely important to understand how one test might affect another. Needless to say, this issue is often shoved under the rug. The typical reasoning is if we randomize everything, we will be fine. This is a fallacy: it's only fine if none of your tests produce any significant effects, which I hope is not the case; if the inputs are random, the outputs are random only if the treatment made no difference!

***

The third problem is bad measurement, which is often hidden from view. The most tricky problem is the data you are analyzing are not what you think they are.

For example, the log of the experiment only tells you what treatment the unit was supposed to receive, but not what it actually received. It is considered acceptable to assume that the actual treatment is the design treatment. Or, you may have the opposite problem, that you have a log of the actual treatment but you don't know what the design calls for, which means you are making the assumption that the random assignment of treatment has been properly administered. Needless to say, these assumptions are quite bold. Too bold for my taste.

***

So, Facebook comes to the rescue last week when they introduced PlanOut, which is an internally grown platform for managing and executing online experiments. I have previously commended the Facebook Data Science team on the meticulous way they ran their experiments (link). I'm glad that they are sharing their system with the rest of the world.

While I have not tried the system (we put together something similar at Vimeo, though less fully featured), I noted the following key components:

- abstracting the experimental parameters as a separate layer

- ability to target specific segments for testing

- ability to work with different levels of experimental units (user_ids, cookie_ids, ...)

- ability to set experimental fractions other than 50/50

- inclusion of standard random number generators

- integrated logging

- management interface for multiple simultaneous experiments to different segments

What I hope they will release in the future:

- establishing the experimental units

- quality-control charts

- monitoring reports

 

Talking shop about probability

It strikes me that the media loves to talk about probability, a subject about which journalists are ill-trained to write. The latest example of this is Forbes' attempt to draw a lesson out of the Warren Buffett's gimmicky $1 billion NCAA pool. As we all learned, by the time the 25th match drew to a close, all 8.7 million entrants have gotten at least one winner wrong, thus there would be no payout. (There are 32 matches in the first round of the tournament.)

The author calls this "human folly, bots' wisdom." Specifically, he said "A simple, randomized computer program probably would have done a better job of sizing up the upsets than we did."

It's not clear what he meant by a "randomized computer program". I'd assume he means the computer that selects at random one of the 4.3 billion possible first-round brackets. If so, the computer's chance of getting the right one is also remote.

We all know friends who spent hours or days slicing and dicing data to fill out their brackets. Nate Silver posted his methodology here, for example. It's unclear how many of the 8.7 million entries were purely "human".

***

When the columnist moves to the numbers, confusion reigns:

If people had filled out their brackets totally randomly, it would have taken 2 to the 25th, or 34 million people, to cover every single possibility. So with 8.7 million entrants, a simple computer program (call it MoronBot) would have enjoyed about a 26% chance of coming up with at least one surviving contender.

If we take this at face value, then on average, three out of four years, there would be no one left after the first 25 matches, even if every entry is generated by MoronBot. So it isn't surprising that no one got past the first 25 this year in particular.

What is lost in the math is the fact that the entire setup is a gimmick. Buffett has already determined that the chance of payout is zero. And it doesn't matter whether humans or bots are entering the stakes. This is precisely why the entry fee to this pool is, well, zero. Ever heard of no free lunch?

***

What is driving the odds is the exponential multiplier. Let's re-do the Forbes calculation but considering all 32 teams in the first round. Now, there are 4.3 billion unique first-round brackets (up from 34 million)! Even if all 8.7 million entrants were distinct brackets, they would constitute only 0.2 percent of possible brackets. The odds have dropped over 100-fold just by observing seven more first-round matches. It would be hundreds of years before we expect to find one perfect first-round bracket.

Worse than that, if this hundreds-year event happened, 8.7 million people will share the $1 billion prize since they are all now part of Moronbot. Yes, that means each entrant gets $115, but it's paid in 40 annual installments. Well, that's $3 per year, before taxes.

My advice: just have fun picking your bracket, and focus on your office pool.

***

The columnist went on to praise Big Data and artificial intelligence. However, the only analysis provided in the piece relates to a Moronbot making random picks, which requires zero intelligence. This is followed by a handwaving argument saying if a computer achieved X by making random picks, then the same computer will do X+Y by using some more intelligent algorithm, where Y is a positive number. In the meantime, if a human uses his or her brain, then Y is a negative number. What is X? What is Y? and what are these algorithms? and how are these different from models created by the Nate Silvers of the world?

Some past talks

For those who weren't able to attend my recent talks, a few have surfaced online.

***

JMP put up the video of the webcast from last Friday with Alberto Cairo, a data visualization expert and author of The Functional Art. You can access it from here. This event is part of their Analytically Speaking series with recent guests such as David Hand and Michael Schrage. I also appear on this recording of the panel celebrating the International Year of Statistics.

***

Agilone, an emerging vendor of self-service marketing analytics software, hosted me at their recent user conference, as well as a webcast. Here is a clip, in which I explain the structure of analytics teams that I have assembled.

***

Last year, I gave a fun, lightning talk at the Leaders in Software & Art conference. The recording is here.

***

Joe Dager did several long interviews with me that is well worth listening to. Here's Part 1, and then there's Part 2.

Round-up of coverage of the Big Miss of Big Data

There is now some serious soul-searching in the mainstream media about their (previously) breath-taking coverage of the Big Data revolution. I am collecting some useful links here for those interested in learning more.

Here's my Harvard Business Review article in which I discussed the Science paper disclosing that Google Flu Trends, that key exhibit of the Big Data lobby, has systematically over-estimated flu activity for 100 out of the last 108 weeks. I also wrote about the OCCAM framework, which I find useful to think about the "Big Data" datasets we analyze today versus more traditional datasets from the past.

The HBR article caught the attention of various outlets, including New York Times and ABC News.

Slate was probably the earliest to react, and noticed a post on this blog that was the precursor to the HBR article.

Readers who are specifically interested in GFT should read the source materials themselves, which are quite accessible. Start with the Science paper. After that, you can read the original research article by the Google team, hosted at google.org (click on the PDF link in the blue box at the bottom of the page). There are some bold claims in this paper, as well as caveats. They seemed to be concerned about "false alerts" at the time, such as news events rather than illness that drive certain searches. (For those statistically inclined, the underlying model involves only 1,152 points of data--128 weekly aggregates in nine regions--but a search through 450 million simple logistic models to not only define which search terms are important but also determine how many search terms to include in the final regression.)

Then read the article by Cook, et. al., which covers an update to the model made after the 2009 season when GFT totally missed the pH1N1 swine flu epidemic. Notice that this Big Miss is the opposite error to the "false alert" problem. (See Chapter 4 of Numbers Rule Your World for a thorough discussion of different types of prediction errors, and how to think about them.) From the charts in the Cook article, you can see that in the runup to the Big Miss, GFT systematically under-estimated flu activities for as many weeks as you can count.

The overhaul was drastic. The search term topics that accounted for 70% of the original model were reduced in importance to 6% while two other topics that counted for 8% originally were inflated to 69% in the updated model. This dramatically improved the "fit" statistic (RMSE) for the "first phase" of the Big Miss from 0.008 to 0.001.

Next, there is Butler's article for Nature, (Feb 2013) which precedes the Science article, but first pointed out the over-estimation problem for the 2012 flu season. One possibility is that the model update described above over-compensated for the Big Miss, making it more susceptible to the False Alert.

Other media coverage of Google Flu Trends include Guardian (which focuses on the need to understand causality), CRN, and the Economist (which talks mostly about Twitter data which is much more problematic than Google search data).

***

Tim Harford has probably the most educational of the pieces revisiting Big Data for the Financial Times. (When I wrote this line, the FT link wasn't working. The title of the article is "Big Data: Are we making a big mistake?" if you need to find a different link.) His is the longest and covers a lot of ground, and has great examples, including one of my own. Highly recommended.

***

One of the slogans of the Big Data industry (of which I'm a part) is the push toward "evidence-based" decision-making in place of "gut feelings" or "instincts". Until now, I'm afraid there has been plenty little "evidence" presented to support the assertions of universal, revolutionary goodness of Big Data (try searching for quantitative assessment of Big Data projects). I hope we are witnessing the birth of evidence-based decision-making inside the industry of dishing out evidence-based advice.

The reason I put out the OCCAM framework is to steer our community toward a more constructive approach to tackling "Big Data" problems. It requires a fundamental shift in how we define the problem. I have a moderately more technical take on some of the statistical challenges in an article published in Significance, earlier this year. This article discusses six technical challenges where we need substantial progress.

Statisticians sometimes dismiss these as "old news," claiming that the same problems exist in smaller datasets and the problems are well known. A recent example is Jon Schwabish's tweet saying that this discussion induces a "yawn". This reaction feels a bit like Fermat writing in the margin claiming he has a proof. The rest of the world don't want to wait 358 years to figure out what the goods are which these statisticians are hiding.

In my view, there has been some interesting work but nothing to settle debates. If we have great solutions, we won't be discussing these same problems today.

***

Back to flu prediction. It's really something that is well worth pursuing!

It's a nice-defined, self-contained problem that has social benefits and whose results can be easily measured. We should be grateful that the Googlers spent time working on it. It's a problem I'd love to work on if I have time and resources on my hands.

The researchers also have pioneered this type of research using search term data. This is highly significant, and the data represent a perfect example of what I call OCCAM data: the data is purely Observational (related to what Harford calls "found data" or what Dan Eckles calls "data exhaust"); it has no Controls; it is seemingly Complete; it wasn't collected for the purpose of predicting flu trends, that is, it is Adapted from other uses; and the search data was Merged with the CDC data (the matching of states and regions, and of weeks were not exact as you can tell from the original research article).

The several published versions of the predictive models are clearly failures but anyone who is in this business knows model building is an iteractive process. One can learn from these mistakes. I happen to think they need to wipe the slate clean and use an entirely different approach. It's a small price to pay if there is reward down the road.

I sincerely hope that this coverage will lead to improved modeling and analytical techniques rather than a retrenchment.

If you find other related links, please place them in the comments.

 

 

Numbersense Pros: Cathy O'Neil talks about trust in data analysis

Cathy O'Neil may need no introduction to blog readers. She's the author of the hard-hitting MathBabe blog, and she shares my passion for explaining how data analysis really works. She is co-author of the recent book Doing Data Science (link), with Rachel Schutt. Cathy has a varied career spanning academia and industry, as she explains below.

***

KF: How did you pick up your impressive statistical reasoning skills?

CO:

Thanks for the flattery, but I wouldn't call my skills impressive. I've always done my best thinking by assuming I understand nothing and starting from scratch. The best I can say about myself is that I have learned how to think abstractly and a few cool methods, or better yet rules of thumb, that help me get at very basic information.

 

What I know about thinking abstractly happened mostly during my mathematical training, first at math camp in high school, then as an undergrad in a highly welcoming and mathematically vibrant community at UC Berkeley in the early 1990s, and then during grad school at Harvard and to some extent my post-doc at MIT and my two-year Assistant Professor stint at Barnard College. 

 

To be honest most of the last few years of being a "grownup" academic was spent learning non-math stuff like how to teach and write letters of recommendation. 

 

Then I learned a bunch explicitly in the realm of statistical reasoning when I first got to D.E. Shaw, from my boss Steve Strong. Since then I feel like I've just been corroborating what Steve explained to me early on, which is that people fool themselves into thinking they understand stuff they don't.

 

KF: How would you rate the relative importance of academia and real-world experience in training your data interpretation skills?

 

CO:

 

I'd say that, on the whole, learning to think abstractly has been at least as important to me as rules of thumbs, and certainly more important than a given algorithm or technique. For example, from my experience working in industry, the most common mistake is answering the wrong question, not using the wrong technique. 

 

I routinely tell people that, as a mathematician, you are a professional learner, with an added advantage of getting used to being wrong and feeling stupid. I'm sure the same can be said about other disciplines, but I'll stick to what I think I know on that score. 

 

KF: What advice would you give to a young graduate with a BS in a quantitative field: get an advanced degree in Statistics, or go find a job in analytics?

 

CO:

 

I don't think it's a waste of time to get a Ph.D. and then an industry job, because although you're not honing specific skills in your future line of work, you are honing brain paths and habits of mind which don't come easy under time pressure and/or with money on the line. And of course, there are some people who love the feeling of getting things to work so much that they don't have patience for the thesis thing, and that makes sense for those people, as long as they don't give short shrift to the high-level perspective.

***

 

KF: What is your pet peeve with published data interpretations?

 

CO:

 

Better question: what isn't my pet peeve with published data interpretations? 

 

I'm a huge complainer about everyone and everything, in spite of the fact that I think data and data analysis techniques are powerful and can and should be used for good. 

 

I guess if I had to pinpoint my single most massive peeve, which really cannot be termed "pet," it would have to be hiding perverse incentives (and almost all incentives are perverse in some way) behind what people present at "objective truth". In my experience, outside of the world of sports where everything is transparent (except steroid use), there is always some opacity and gaming going on and someone's either making money off of it, gaining status from its publication, or wielding power through it. 

 

And come to think of it, you've asked me the wrong question altogether. My biggest peeve with data interpretations is how many aren't published at all. For example, the Value-Added Model for teachers is being used to evaluate teachers but I can't seem to get my hands on the source code to save my life. Not to mention the NSA models.

 

[Edit: David Spiegelhalter also complained about what studies don't get published but for a different kind of concern. See this interview. In the recent furor over Google Flu Trends, the researchers expressed dissatisfaction that the underlying algorithm isn't properly documented in the public domain.]

 

***

 

KF: Which source(s) do you turn to for reliable data analysis?

 

CO:

 

I don't trust anything or anyone, including my own analysis. Everything comes with caveats. Having said that I usually trust people more when they are open about their caveats. On the other hand, even admitting that opens me up to being fooled by people who write up fake caveats to seem trustworthy. It's really an endless loop.

 

So for example, I like raw data, especially when I know how the data was gathered. For example, look at this gif, which shows a map of death penalty executions. In some sense that's as good as it gets, but of course it is also misleading in a sense since there are way more people in, say, California (38M in 2013) than in Nevada (3M in 2013), so even though they look similar on the map it's not so.

 

Bottomline is, never trust anything until you've checked it, and even then only trust your own memory of it for about 20 minutes.

 

KF: What advice do you have for the average reader of this blog? Surely, checking everything they read is not too realistic.

 

CO:

 

Of course, we don't have time to check everything. My suggestion is to remain skeptical of anything that you haven't checked through. And of course, don't confuse skepticism with cynicism, but also don't confuse skepticism with evangelism.

 

KF: Thank you so much. I've really enjoyed our conversation.

 

 

PS. I subsequently wrote about the chart that Cathy referenced in this interview. See here.

 

Big delusion

This article printed by VentureBeat is too much. The title claims "The Internet is killing off marketing surveys & it's for the best". This article is tagged as "Big Data". Big delusion is what it is.

This is a great example of the kind of revisionist history that is practised in the name of Big Data. You'd also notice that there is no data or evidence presented to support any of its many far-reaching claims.

First comes the howler:

about eight years ago, people started raising concerns about respondent quality [of traditional marketing surveys]; and as social media took off, some dared to wonder aloud whether online ratings and reviews were eliminating the need for surveys altogether.

Eight years ago would be 20062008. What happened in 20062008 that made people question instruments that have been used for decades? What kinds of concerns? How do "social media" and online ratings and reviews solve those problems? We will not learn.

Apparently, those eight-year-old concerns were not sufficient to sink the sorry enterprise until "recently". We're told -- again with no data -- that "we’re witnessing the demise of the lengthy, grid-question littered, rating-scale driven survey as we know it."

Raise your hand if you have done an online survey for a market research company. Is the survey "lengthy"? Does it contain a "grid"? Does it ask you to use a "rating scale"? Well, I thought so. Just in the last week, I reviewed an online survey design submitted by an outside agency which contains thirty questions, replete with multiple grids and multiple rating scales.

***

Later on, the author attacks blinded designs. I'm not kidding. This is the charge: '[Consumers] ask themselves, “Why should I invest time and candor in responding to questions posed by some person or entity that won’t even reveal their identity, let alone respond?”'

Apparently, tweets and online reviews are the new way. Don't worry if the tweets are unrelated to your research question or that most of those online reviews were purchased by your social media marketing agency and written by people who have not ever used your product.

And may we ask how they propose to measure "unaided brand awareness"? It is well known that the following two questions lead to vastly different responses:

(A) Name three services which can be used to create an online survey

(B) Have you heard of the following companies that provide online survey tools? SurveyMonkey, Zoomerang,...

***

Next comes the obligatory Big Data moment: "Big Data is also reducing the need for quantitatively rigorous, predictive surveys." What could that even mean? We now prefer quantitatively weak, unpredictive surveys?

The author explains, today we can just "harvest and analyze the masses of behavioral data already available." From where? Do log data or tweets inform us about attitudes, motivation, and psychology? As usual, we are asked to assume Big Data solves some problems, therefore Big Data solves those problems.

***

Now I have no doubt that "Big Data" will impact the market research field. This does not excuse poor arguments presented without evidence.

Making Big Data add value is not as simple as "harvesting". Tweets and reviews have all the characteristics of the OCCAM framework: they are observational in nature, lack controls, adapted from their original purpose, merged with other datasets, and to the deluded, they are complete (N=All).

 

Numbersense Pros: An interview with David Spiegelhalter

I am excited to chat with Professor David Spiegelhalter, who is no strangers to our UK audience, and our statistics colleagues. Perhaps his most well-known contribution is the DIC criterion for model selection, introduced by a paper by him and collaborators. He holds the impressive title of Winton Professor for the Public Understanding of Risk at the University of Cambridge (link). He also writes a blog called Understanding Uncertainty (link), and as the accompanying photos show, is someone who knows how to enjoy life.

I mean, a statistician who appeared on Winter Wipeout (link to Youtube for spectacular splashes at 15:12 and 16:10), who'd have thought? Yes, Wipeout is that obstacle course show held over a pool of water. He also made this rather more educational Youtube (link).

***

KF: How did you pick up your impressive statistical reasoning skills?

DJS:

Well that's your label and not mine.  I started off doing pure maths, but that got too hard, and then I did mathematical statistics, and then got both too hard and too boring, and so for years now I have preferred getting involved in real problems that people are trying to handle using data.  

But generally the data are messy, incomplete, and not as relevant as desired.  While some technical insights are vital, I think any skill comes mainly from an apprenticeship of dealing with many problems, making many mistakes, trying to explain things, and far too much time spent critiquing studies.

 

KF: What is your pet peeve with published data interpretations?

DJS:

That's easy to identify - it's a non-scientific approach to science reporting.  I have a naive view that scientists should do investigations to answer a question, and they should be pleased whatever the answer.  But it seems clear from many publications that some researchers set out to prove a point, and do everything they can to do so: in the worst instances they write an inflated abstract, the journal puts out a press release, and the media lap it up.  

I feel the public get fed a diet of highly selected and biased studies (often, ironically, on diet) that have gone through so many filters that they become very unrepresentative of the bulk of research conducted. In my more cynical old-man moments, I would say that the very fact that a study is reported in the media is a reason to ignore it - almost certainly you would not have heard about it if the results had different.

 

KF: That last point sounds counterintuitive. Let's take a diet example. The media has been telling us new research suggests that four or more cups of coffee each day is great for you. If the research result were null, surely it wouldn't get picked up by the media. Why would that be a bad thing?

DS:

Say the media tells us four cups or more of coffee every day is great for you, and I judge that, if the study had shown no effect of coffee, it would not have been press-released and the media would not have picked it up.  This probably means there is an unknown number of studies out there that showed the opposite to what I am being told by the media, but I am not hearing about them because they are not newsworthy enough.  Therefore ignore the media.  It also saves a lot of time.

 

KF: That's rather sobering. Which source(s) do you turn to for reliable data analysis?

DJS:

These would tend to be individuals and teams that I know and trust: Andrew Gelman (link to my interview) comes to mind, and there are other great scientists whose opinions I value.  I also respect people who are trying to produce good odds for future events, without pushing for one side or another.  A purely financial interest produces objectivity, and so sports-betting sites are good examples - it will be interesting to see how 5-38 develop.

 

KF: Thank you very much for sharing your insights.

***

David and Michael Blastland just published a new book called "The Norm Chronicles", which I had a chance to preview. It's an idiosyncratic look at the idiosyncratic risks of modern living.

***

You can find prior interviews here: Larry Cahoon, Len Testa, Andrew Gelman, Felix Salmon, Avinash Kaushik, Dean Baker.