Big Data, Plainly Spoken (aka Numbers Rule Your World)

A gathering of artists and technologists this Friday

On Friday, I'm attending and speaking at the Leaders in Software and Art Conference, organized by Isabel Draves. LISA is an amazing gathering of artists interested in technology and software. For example, there is a panel on 3D printing and hardware hacking, and one on "creative coding, art and advertising". Check out videos from past years, and click here to register. My talk is at around 3:30 in a tightly packed day of activities.

Q&A on Big Data

There was a lively, fun discussion after my talk yesterday night in New York. For those who couldn't attend, let me review some of the conversation. Here you go:

Q: Tell us more about the chapter in Numbersense titled "Are They New Jobs When No One Can Apply?" Related to economic data, can you talk about the idea that we still need to import foreign workers because there aren't enough skilled labor available domestically?

A: That chapter is really about the use of statistical adjustments. Some people say they are skeptical of the official unemployment numbers, the seasonally adjusted numbers, because it includes jobs that are made up--there isn't a job posting that you can apply to. These people think that raw data is always better than adjusted data, because statisticians are doing naughty things to the data. In reality, it's the opposite. Raw data is bad and adjusted data is better.

In terms of the skilled labor issue, this is related to the argument about structural unemployment, the claim that the unemployment problem cannot be solved because people need retraining.The proposed solution is to send more people to college. This is usually supported by data showing that the unemployment rate of college graduates is much lower than that of non college graduates. However, there is a cohort problem here. If we drill down to recent college graduates, there are many who aren't finding jobs. So if we make more college graduates, there will be even more competition for those jobs, and it will suppress the income of those jobs.

Unemployment rate and economic indicators are complex things. The chapter [Chapters 6-7 in Numbersense] goes into quite a bit of detail to address the question of why they seem out of touch with reality.

***

Q: How is Big Data going to impact the movie industry? Can you predict what people will watch?

A: To some extent. But remember the example of the Target model predicting pregnancy I just described [Chapter 5 in Numbersense]. If you use a statistical criterion, like the predictive lift, you can congratulate yourself on having built a great model. But then if you apply a more common-sense metric, like the hit rate, you notice the really high proportion of errors even when the lift is high.

In social-science problems like this, I always advocate the combination of quantitative and qualitative data. It is not sufficient to just use frequencies. They don't tell you why someone watched something. [Then I went on a rant about why correlation is still not causation in the Big Data era.]

***

Q: What can you say about how the Democratic and Republican parties used Big Data in the recent presidential elections? Can you talk about the likely voter models?

A: Somewhere in the book (Prologue), I discussed the likely voter models. I used this to illustrate the point I was making earlier about understanding what part of the argument is data-driven and which part is theory-driven. In the book, I compared the work of Nate Silver and the guy who created the UnSkewedPolls.com website. They both used the same data set of the poll results but came out with really different projections.

This points to the important idea that any data analysis involves theory--you can't avoid it. There is this myth out there that says when you have loads of data, the data itself is objective and stops all debate. That's so far from the truth. Anyone who has worked with data knows that you have to make assumptions. In the case of likely voters, the Republican guy made assumptions about how their party members would be motivated to come out to vote. This is a necessary assumption because if you just use the data, you will always predict that the future is the past, and you will never be able to predict a surge in interest. In this case, the theory part of the analysis turned out to be spectacularly wrong.

***

Q: Big Data is impacting the education sector in many ways, such as Value Added Models being applied to evaluate teachers. What do you think of these models?

Education is a good example of Big Data. It fits the five criteria that I just laid out for what is a Big Data study. The value-added models for instance are silly, partly because the data is co-opted--test scores are originally intended to measure student performance but then they're being used to evaluate teachers. Also, the whole pay-for-performance concept doesn't work when you can't measure performance well; it backfires and causes rampant cheating. The first chapter of Numbersense was originally going to be about how teachers and principals all over the country were turned into cheaters; but then I ended up writing about a different kind of fraud--how schools game the school rankings.

***

Q: On one of your slides, you mentioned eHarmony trying to port its algorithm from matchmaking to hooking up employees and employers. That sounds like a promising thing. What do you think of it?

A: Just like all predictive models, you have to be careful in understanding how accurate the predictions are. The media do a poor job of reporting the accuracy. The eHarmony model can be evaluated in the same way that I evaluated the Target model for predicting pregnancy. You have to think about both false positives and false negatives, and the fact that those two trade off each other. In the media, you often read about one of those two metrics, and they hide the other one.

***

Q: Did the Gates Foundation stop funding the "small schools movement" because of [Howard Wainer's] statistical analysis? [See Prologue of Numbersense.]

A: Not really. However, they did use statistics to come to that conclusion. To their credit, the people in the Gates Foundation ran some rigorous analyses comparing the small schools they funded with larger schools. And they learned that the small schools were not better than the larger schools, and in some cases, even worse. So they started to spend the money on other things like curriculum development.

***

Q: Do you have an opinion on the recent Malcolm Gladwell piece about doping in sports?

A: I do have reactions to the Gladwell piece. Those of you who came to this talk three years ago for my other book remember that anti-doping testing was one of the main examples I spoke about. In fact, other people have requested that I write about Gladwell's assertion that doping should be made legal. I will be putting up my response hopefully in the next few days. Look for it on my blog.

***

I'm sure there were more questions. So, please accept my apologies to any attendee whose question I could not recall. Thanks for attending.

Find out more about the book here.

 

An interview, and a talk

For those in New York, I'll give a talk tonight at NYU's main library. Details are here. If you're not affiliated with NYU, please make sure you RSVP to put yourself on the guest list.

The talk covers what is meant by Big Data, why you need numbersense, and several examples of using numbersense to interpret data analyses. Click here for more details.

***

For those who aren't in New York, Steve Lundin posted a video of his interview with me after my talk in Chicago this summer. Here is the link to his site. The topics include how businesses can get started in analytics, and data privacy.

Limiting scope paradoxically makes a problem harder

I left a comment on one of Andrew Gelman's recent posts about Malcolm Gladwell (link). This post discusses briefly a review of Gladwell's recent book. 

A commenter ("Haile") made the following defense of Gladwell:

My point is that many of these criticisms are based on Gladwell’s failure to present rigorous statistical evidence of arguments that are not statistical in nature in the first place.

when (if) reading Gladwell, it’s time to put down the statistical hammer, because virtually nothing that he talks about has anything to do with phenomena that are statistical in nature.

The argument is simply that if the author is merely describing one person, a sample size of one, and not attempting a generalization, then statistics has no standing in this conversation.

In my response, I will gloss over the important issue of whether Gladwell is making general comments about human psychology or sociology in his books. I will consider what it means to study an individual sample by itself.

It turns out that reducing the scope of the data makes the analysis harder, not easier. This may feel counterintuitive.

In my response to the comment, I used one of the examples from the new book. The greatness of David Boies, the famous lawyer, is argued to be a result of his dyslexia, which is billed as a "desirable difficulty". Here is what I said: 

Let’s examine the theory that Boies is a great lawyer because he is dyslexic. How would you prove/disprove such a statement? All we observed are two separate things: Boies is a great lawyer; and Boies has dyslexia. Well, maybe a third thing, which is that Boies said he became a great lawyer because he is dyslexic. A correlation does not a causation make. First-person speculation is not science either.

If Gladwell is attempting a generalization, he could expand his data set to multiple Boies, and hope to find stronger evidence for the theory. If he is forced to use N=1 samples, his task is more formidable, not less.

Note that saying Boies benefited from his dyslexia is very different from saying Boies is X feet tall. The latter is purely descriptive and can be verified easily. The former is not descriptive, and a hard problem.

Andrew's other post on the Gladwell genre is really good. It captures many of my own sentiments.

Unsampling: how a meaningless word invaded Big Data

If you know any statistics, you know "sampling". It's the idea of measuring some subset of the population. Using the Law of Large Numbers, you are able to learn from the sample and generalize to the population.

Your Stats professor never told you what "unsampling" is. You're not going to find this word in a statistics textbook either.

What does it mean? The "un" implies that you can recover the population from a sample. That makes no sense at all. If you have a sample of 5,000 Americans, how do you "unsample" to the 300 million plus population?

Yet, "unsampling" is a Big Data term. It appears that Google Analytics people invented this word. In their software, they talk about "sampled reports" and "unsampled reports". "Unsampled" means "not sampled," at least that's what the marketing folks want us to believe.

This would imply "unsampled" means the reports use all the data instead of some subset of the data.

Except that's not what the word means either.

I learned this the hard way. If you are a power user of Google Analytics, you might have seen this error message too:

This restriction to standard table reports is extremely severe. Almost all standard reports are univariate cuts of the data which means that if you are analyzing correlations, forget it.

In addition, an "unsampled" report cannot use more than say 100 million rows of data. Even if I'm needing a report of summary statistics that take up two rows of output, if I have more than 100 million rows of input (for the Premium version), they will not run it. I'm not sure how something that has an upper limit in size can be "not sampled".

I wish they would stop using that word. It's inaccurate, and nonsensical.

 

Numbersense Pros: Interview with Larry Cahoon Part 2

Yesterday, Larry Cahoon, a 29-year veteran at the Census Bureau, answered some questions. The rest of the interview is printed below.

***

KF: How can a data analyst improve his or her skills?

LC:

I have to say my best training has been many, many hours of just playing with statistics, playing with graphics, and reading the analysis others have done. The more data I see, the more analysis I do, the more graphics I look at and produce, the more I learn about how to look at data and how to see the pitfalls in the data analysis. This is getting down in the mud and dealing with real data with all of its warts. My wife tells people I’m a statistician through and through as every time she looks at my computer, there is a graphic of some type on the screen.

Finally, I have always been an avid reader of Science Fiction. No one can read Science Fiction without being forced to consider any problem from different perspectives and take into considerations differing assumptions. This in turn has helped me develop the ability to question the assumptions being made in any data analysis.

***

KF: What are your pet peeves with published data interpretations?

LC:

I seem to return again and again to the same issues with the analyses I see in the newspapers, online, and just about anywhere. The most basic problem is one of incomplete analysis.

We see so many papers and news reports where a data difference is observed and then based on no data whatsoever, the author goes off with an entire line of speculation without any data to justify that speculation. This line of thinking then frequently ends up with the claim that these two things are correlated, and therefore we have cause and effect.

The media then fans these reports by writing a story without asking basic questions, such as is the data itself any good or have they any evidence for the claims that are being made. The media acts as if the claims have been proven – especially in how they headline the story.

My second pet peeve is what I call an emphasis on a one dimensional world. This is usually reflected in simple statements like: A causes B. The world is much more complex than that. Those who investigate airline accidents have been telling us for some time that there is seldom just one cause for each accident. Rather there are a number of causes. We need to carry that knowledge over to our statistical analysis and reporting.

***

KF: Which source(s) do you turn to for reliable data analysis?

LC:

I can’t say that I have any favorite source for data analysis. If forced to name one, I would say that I tend to like the work of the Pew Research Center (link) Their surveys seem to be well designed, the questions they ask well thought out, and the analysis something I can trust.

I like the data that is available from the Federal Government. But the government agencies rightly avoid most detailed data analysis in an effort to remain nonpartisan.

***

KF: Thank you so much for your time. We're lucky that you continue to blog in your retirement.

You can view the earlier installments of the interview series here.

Numbersense Pros: Interview with Larry Cahoon Part 1

A short while ago, I introduced Larry Cahoon's blog, GoodStatsBadStats. He started the blog almost two years ago after retiring from the US Census Bureau (site offline due to government shutdown), where he spent 29 years working on the statistical design of most of the household surveys conducted by the Bureau. Cahoon received his PhD from Carnegie Mellon University.

Cahoon spent the final seven years of his career working on the 2010 decennial census. He was involved in almost all areas of census design and operations. His work included focusing on the many issues of both over coverage and under coverage in the census, the effectiveness of the publicity campaign, and the use of the American Community Survey as a replacement for the long form of the census.

Larry was very gracious and offered great insights and long answers. I have divided the interview into two parts. Part 2 will appear tomorrow.

***

KF: What are the key skills of statistical reasoning?

LC:

There are three things on the top of my list when I think about statistical reasoning skills. A solid knowledge of statistical methods and principles is a necessary starting point. I extend this to include the ability to think in terms of probability. This is a way of thinking or a way of viewing the world at the most basic level. 

Equally important is a solid foundation in logical reasoning. The third piece is recognizing and dealing with the assumptions that must be made in any statistical analysis. One needs to be able to look at the problem from a multitude of perspectives and ask if the assumptions being made are good ones. Can I make a logical argument why they are true and why a contrary set of assumptions is false? The ability to consider alternate assumptions and confounding factors that may not be obvious is very important to any statistical analysis.

KF: Was graduate school useful training for your career?

LC:

In my own background, my first year of graduate school just about drove me crazy as I did nothing but statistics. Most of it was extremely theoretical work. I have always said that the masters’ degree they gave me after that year was in some sense worthless as I got very little exposure to the practical side of the statistical profession.

But what I did learn from the full immersion was to think in terms of probability. I came to see statistics not just as theory but as a way of thinking. Thinking in probability terms became second nature. It helped that I was trained in a Bayesian environment long before it reached the levels of use that we now see. 

I saw statistical testing as a sometimes useful but limited tool. It became clear to me that much more important questions are what is the best estimate and what is the decision rule that would come out of the analysis. It was just as important to know how that data is going to be used as it is to know how the analysis is to be done.

KF: How did your work at the Census Bureau influence you?

LC:

To do good statistics, knowledge of the subject matter it is being applied to is critical. I also learned early on that issues of variance and bias in any estimate are actually more important than the estimate itself. If I don’t know things like the variability inherent in an estimate and the bias issues in that estimate, then I really don’t know very much.

A favorite saying among the statisticians at the Census Bureau where I worked is that the biases are almost always greater than the sampling error. So my first goal is always to understand the data source, the data quality and what it actually measures.

But, I also still have to make decisions based on the data I have. The real question then becomes given the estimate on hand, what I know about the variance of that estimate, and the biases in that estimate, what decision am I going to make.

KF: I want to reiterate this point to my readers who are not statisticians. In data analysis, we are using available data (the sample) to make a general statement, say use the response of subjects enrolled in a clinical trial to describe the effect of a new drug on all potential patients. Imagine you are trying to hit the bull's eye. Shotgun #1 produces a wide scatter around the target while Shotgun #2 produces a narrow scatter but the average shot lands wide of the target. We say that #1 has high variance and low bias while #2 has high bias but low variance. Both types of errors contribute to the shot being off the bull's eye.

Because the Census Bureau typically uses large samples, the sampling error (variance) is very manageable. What is hard to control is bias, meaning the entire sample is not representative of the population under study. This is Shotgun #2.

LC continues:

As I matured in my career, I learned that diversity matters. The greater the exposure and the more diverse the exposure to real world statistics, the better the practitioner would become. So while I worked for many years in the area of survey design, when I went to the annual statistical meetings, I always made the effort to maximize my exposure to other areas of statistics. I would always return home with a few textbooks on areas of statistics outside of what I was actually working on at the job.

A curious nature with a desire to continue learning is essential. Today a good training route is to read as many statistical blogs as you can find the time for. Especially important is to seek out and read the work of those who disagree with me. This forces me to think much more critically of the work I am doing.

***
Part 2 will follow tomorrow.

Read all the previous interviews here. 

 

Know your data 15: How to get robbed by Google

You walk into Walmart, and there is a big notice to customers, informing you that by walking into the store, you consent to being continuously photographed, and being used as a spokesperson for any brands that are carried in the store. You will not be compensated for endorsing these brands because Walmart opens its doors to you without an entrance fee. You should feel blessed that Walmart won’t charge you an entrance fee—in fact, Walmart promises never to charge that. It's none of your business that the brands are paying Walmart for your endorsements.

You will not be notified which brands you are endorsing, when you are endorsing them, how you are endorsing them, or to whom your endorsement will be peddled. If you find the preceding terms distasteful, you can exercise your all-American right to shop elsewhere. Before you turn away, give a little thought to those everyday low prices.

Next week, the big notice will be removed because by shopping here once, you have issued a blanket consent. Every time you walk into the store, you are renewing your consent.

You won't see this notice at Walmart (just yet), but last week, Google announced that this is how they will treat you from now on (link). To Google’s credit, they are simply copying what Facebook has been doing for some time. Now that Google’s doing it, it must not be evil.

***

Next, pity the unsuspecting marketer who spends real money on Google's "shared endorsements" (or "sponsored stories" to use Facebook's terminology.) According to the New York Times reporter:

Such product endorsements, especially from friends and acquaintances, are a powerful lure to brands, replicating word-of-mouth marketing on a broad scale. Social advertising — which includes a wide range of ads, including endorsements — is a $9.5 billion business, according to eMarketer, accounting for 8 percent of digital ad spending.

Most of that money is being poured down the drain. These marketers are getting robbed by not thinking about the counterfactual! (I explain this in detail in the marketing chapters of Numbersense. Learn more about the book here.)

"Word of mouth" operates in the shadows but marketers know it's happening. They even attempt to quantify it through something called the "Net Promoter Score," (link) computed based on answers to the question of whether one would recommend the brand to a friend.

But word of mouth has always been free marketing. This was happening before Google or Facebook even existed. Why should marketers pay social media companies for something that is free? Are they just getting robbed?

***

Here is an example of the kind of ads that would show up featuring your friend or acquaintance.

If Roger Willis is someone I know and trust, such as my Dad or a close friend, I don't need this ad to tell me the obvious. If he is an acquaintance, not a friend, I'm unlikely to buy something based on an ad like this. If he is a friend of a friend, I'm even less likely to care.

Saying the effect is small is not saying no one will ever be influenced. In my view, marketers should not take the bait.

 

 

 

 

Numbersense Pros: Interview with Len Testa

In the first chapter of my first book, Numbers Rule Your World (link), I explored the concept of variability using a pair of examples, one of which was Disney's FastPass virtual reservation system. Truly grasping the ins and outs of variability is one of the most important objectives for a budding statistician (or data scientist). In the discussion, I highlighted the work of Len Testa, whose website, TouringPlans.com, provides custom, computer-optimized itineraries for saving time in Disney's theme parks. Testa's team does exemplary work in applying mathematical models to solve a practical problem. I'm glad to present an interview with Testa today.

Len Testa holds a Masters degree in Computer Science from North Carolina A&T State University. He is the co-author of the Unofficial Guides to Walt Disney World, Disneyland, and Britain's Best Days Out, as well as a contributor to other books. His grad school research on the Time-Dependent Traveling Salesman Problem forms the basis of the Unofficial Guide's itinerary software.

***

KF: The problem of hitting a sequence of destinations in the fewest steps has a long history. Lots of people have worked on it. The most famous of these problems, the Traveling Salesman, even gets onto the mainstream press. However, most of this work is highly theoretical. Your touring plans to me are a shining example of amazing applied work that makes a lot of people happier. How does your work differ from the others? 

LT:

A lot of times when you're trying to analyze data to solve a particular problem, you can approach it either from the perspective of "management" - the people controlling the process - or from the side of the "consumer." 

The thing we try to model is optimal movement through a theme park. That is, if you're a customer and you want to ride 10 attractions, in what order should you visit them to minimize your wait in line?

The first time we approached this problem, we tried to figure out all of the things you need to know if you're running a theme park: ride speed, how many vehicles to have on the track, when to schedule entertainment to draw people to other parts of the park, and so on.  It was complicated.  

Then we looked at it from the point of view of the consumer.  Consumers, it turns out, have a lot less information about how a theme park is run.  About the only thing they really have is the posted wait time at every attraction.  But it turns out that the posted wait time is really a synthesis of all the small decisions a theme park manager makes, so that's all you need.  It's also a lot simpler to model.

KF: That is a really great answer. I hope all the budding data analysts out there are listening. Simplicity is a beauty.

***

KF: What is your pet peeve with published data interpretations?

LT:

Lack of context, especially around economic or political analyses. Yeah, an $860 billion stimulus package sounded like a lot of money in 2008. But in relation to a $15 trillion economy, it's what – 6%? But all of the discourse was on the raw number, not its size relative to the economy.

***

KF: Do you have other tips for doing great applied data work?

LT:

I find reasoning by analogy to be a powerful way to understand and explain things. For example, if you're putting off a flight to Europe because you're afraid of a plane crash, which is a 1-in-500,000 chance, then why would you ever drive to work, where the odds of dying are an order of magnitude worse? So a lot of it is "If I'm willing to accept X then I should be willing to accept Y” type of thing.

Another helpful thing is being able to apply Bayes Theorem, especially when you're trying to make a business case for something. I remember one time we were trying to get funding to re-do some computer system (at another job), and we calculated a probability of 80% that if we made these changes, we'd succeed in reducing customer problems and lower future operating costs. Some people looked at that as a 1-in-5 chance of failure. I pointed out that we were making decisions every day with a lot less than a documented 80% chance of success.

***

KF: Which sources do you turn to for reliable data analysis?

LT:

For economic analyses, Paul Krugman and the Financial Times. Nate Silver.

Closer to home, I like the mix of statisticians we have now at TouringPlans.com. They have different styles they use to approach problems, so it's useful to hear two different views. And when they agree, you know you've got a decent shot of being right.

KF: Statisticians disagree? They don't know the truth. Readers: you heard it on this blog first! Len, thank you so much for your time.

Mark your calendar

This is a cross-post on both my blogs.

***

I'll be speaking at the NYU Bookstore on Oct 8 (next Tuesday), 6-7:30 pm. See here.

On Oct 9 (Wed), I'll be speaking at the Princeton Tech Meetup. The meeting starts at 7; my talk starts at 8. Details here.