Big Data, Plainly Spoken (aka Numbers Rule Your World)

Sense and nonsense about Big Data and surveys

The American Association for Public Opinion Research (AAPOR) has put out its Big Data report last month (link). This one is worth reading. It has some of the most current citations, and readers of this blog will be very receptive to its core messages. The team who wrote the report is a mix of academics and practitioners.

***

In Big Data, there are many self-evident truths, according to the people who talk about Big Data. One of these is the idea that Big Data will make surveys obsolete. How could it not since Big Data means you have hundreds if not thousands of times more "respondents", the ability to track trends in "real time", and the ability to evolve your survey questions? This AAPOR report, I suppose, is a response to such claims.

Then there are those who say surveys are merely a suboptimal stopgap in the old days of "small data". They explain surveys measure "stated preferences" while Big Data (i.e. found data, observational data) measure "revealed preferences", and aiming for the latter is self-evidently better. Revealed preferences are closer to "the truth". Surveys merely represent an approximation, and with Big Data, we no longer need to run surveys.

***

This topic nicely ties in with Chapter 1 of Numbers Rule Your World (link). In that chapter, I explain the success of Disney in keeping customers happy despite having to wait two hours for rides that last two minutes. The "imagineers" realized that managing perception is even more important than optimizing reality. Customer happiness improves even though measured waiting times (i.e. revealed information) worsen or stay the same.

If Disney relies solely on revealed metrics, the data would say customers are waiting longer, or just as long, as before. When Disney conducts surveys and ask people how they feel, they say they are waiting less and so are happier. Feelings are crucial data that are not revealed by any observations. To the extent that feelings are "revealed", it requires an assumption on the part of the observer; since the measured waiting time is reduced, the customers must be feeling happier.

***

On a website, the web log has measured data that reveal the paths of users through the website. One can observe where traffic drops off. But that is not enough. In order to reduce attrition, the designer needs to understand why users exit. Surveys provide the answer here.

What if you learn that users exit by clicking on the home page icon? So you test a version of the design in which the home page icon is not clickable. You observe that the exit rate has significantly fallen off in the new version. Problem is the users now find a different way to exit. Relying only on revealed preferences frequently leads to superficial actions that cure symptoms but not root causes.

Revealed preferences and stated preferences are two different dimensions and they both have strengths and weaknesses. Logs are bigger and faster but researchers have no control over the composition of the responders. Neither is a substitute for the other. I am interested in seeing work on integrating the two approaches. The AAPOR report has a good discussion of this subject plus a few references to new work on integration.

 

 

 

 

Reading some wild traffic statistics from New Zealand

Only 6% of crashes in New Zealand involve foreign drivers, according to the latest figures provided by the Ministry of Transport.

But in some remote regions of the South Island particularly popular with tourists for their scenery... foreign drivers are involved in about a quarter of all crashes.

These sentences come from a CNN article about a vigilante movement in those regions popular with tourists. The vigilantes snatch car keys from tourists who annoy them by holding up traffic along the scenic routes. 

My friend Tonny saw this article and thought about Numbers Rule Your World. I love to hear stories about how you're able to relate the stories in my book to other real-world situations.

***

The 6% aggregate number hides the effect of tourists on the rate of accidents. The effect of tourists is different depending on which region one is looking at. For this particular article, only the 25% number is relevant--and even this point is not clearcut. I'd like to know whether the vigilante incidents are exclusively occurring in those "remote regions of the South Island", as implied.

The 25% figure does not address the more important question of whether locals or tourists are more likely to get into accidents in those regions. While the tourists accounted for one out of four accidents, they also comprise a large proportion of the traffic. If, say, 50% of the cars on those roads are driven by tourists, then they are disproportionately less likely to be involved in accidents. The base rate of tourist traffic is the missing data.

Further, a margin of error is useful, especially if few accidents occur in those remote areas.

Chapter 1 of Numbers Rule Your World deals with the notion of the statistical average and Chapter 3 investigates when it is appropriate to aggregate data and when it's not. Learn more about the book here. 

 

Some data science principles from Gelman, Rosling and me

I discovered Hans Rosling's Gapminder work when I first started Junk Charts almost ten year ago, with this series of posts. So I was very excited to meet Hans yesterday at the Data, Children and Post-2015 Agenda Event hosted by the UNICEF Data and Analytics Section. And he gave a marvellous talk. I came away touched in equal parts by his humanity, the animated passion for his subject, and the insatiable desire to communicate.

Before getting to Hans, the event's host also made an impression. The UN has put a lot of effort into the Open Data movement. They revamped the website that hosts data from MICS (Multiple Indicator Cluster Surveys), which can be a good source of data for classes and projects. An older resource called DevInfo also appears to be very useful for data about the plight of children (link). The home page for UNICEF data is here.

Hans is a straight talker. And here are a few zingers from his talk.

It's a personality disorder for someone to be interested only in the data. Data is not enough.

He came back to this point at the end of the talk, pointing out that great work comes from people who understand the statistical reasoning and how the data is collected.

We don't need Big Data. We need Basic Data.

Here is an example of Basic Data, presented in the simplest possible way:

You have all the granular data and yet the majority of people continue to harbor myths about world social statistics. The above chart, for example, makes the point that in the last thirty years, Asia (which holds more than half of the world's population) has dramatically reduced fertility rates to reach the same level of the Americas. And yet, when Rosling quizzes his audience about world population growth, 80 to 90 percent still hold the impression that global population will continue to grow at a either linear or sub-linear rate.

Throughout the presentation, I noticed a further cleansing of his visual palette. This leads to another provocation:

The passion of the people plus Excel were all you need. You don't need fancy software.

He was talking about the Ebola crisis in Liberia, where he worked with locals to help measure and staunch the emerging epidemic. Many Western news outlets did not do enough homework and reported vastly inflated numbers during the course of the epidemic. As of yesterday, there are no known cases in Liberia. Hurray!

Saving the best for last. My favorite quote of the evening:

Big Data is a big bag of numerators without denominators.

This gets at the heart of the first C in OCCAM datasets: we are in desperate need of controls.

***

Meanwhile, Gelman found an elegant way to describe the mentality of statisticians:

***

I talk about Big Data, statistics education, and business analytics as the second part of the interview with KDnuggets came on line. See here.

I argue that introductory statistics should be taught as a liberal arts course. Reflecting on Rosling's disappointment that the majority of highly-educated people are so ignorant of basic world facts, I also wonder whether the education sector will find a way to teach students these facts. Thinking back to the my own college days, the introductory  courses in statistics, economics, psychology, etc., were great at training me how to think theoretically but none bothered to connect the theories with any real-world statistics! Here is a past post about the dearth of a Census 101 class.

One of the questions I pose most frequently to my team members is: Do you think or do you know? In the spirit of stacking this post with quotes, I offer:

    Thinking comes before knowing but knowing doesn't come from thinking.

The first part of the KDnuggets interview is here.

 

 

 

 

New interview

I answered some questions from KDnuggets. Here is Part 1 of the interview.

They picked this image.

Cluster analysis in the classroom

I am a guest at the New School's Journalism + Design program this semester.

The students conducted interviews about the question of what makes someone famous. Their interviewees were asked to name five famous people. We had images of these people up on the wall.

Then, we put the pictures into clusters. We tried two different ways of doing it.

At the end, we compared our result to what a computer program generated.

Here are some interesting applications of cluster analysis in the press: FiveThirtyEight made clusters of the intra-season performance profiles of NFL quarterbacks; Pew Research Center made clusters of political values and attitudes.

 

Reflecting on Alamar's reflection on sports data

Ben Alamar reflects on the rise of data analytics in the NBA (link).

I like this passage very much, which really nails home the point that good analytics requires intuition:

The hours of waiting [during draft meetings] were often filled with watching film of prospects. It helped me refine my analysis, as I soaked up details from scouts that I never would have seen on my own. ("Rewind that. ... Did you see his foot placement there, getting ready for the rebound? That's NBA ready.") During one of these sessions, we were watching film of Syracuse point guard Jonny Flynn. I mentioned that, based on the rate at which he collected steals, he was likely a good defender. But one of the scouts explained that Flynn's steal total was likely higher than other point guards' because Syracuse played mostly zone defense, which allowed guards to attack the ball more. I checked that insight against the data and it seemed true, so I adjusted my defensive statistics to account for the dominant style of defense used by a player's team.

I'm glad to hear that the style of play is included in the models. I cringe every time I hear a (usually English) football (i.e. soccer) commentator claiming that a team "deserves" to be in the lead because it is dominating the time of possession when in fact, the other team is using a counter-attack strategy. When the other team ekes out a 1-0 victory on a sneak attack, the commentator loses his wit.

Alamar sees the next big challenge in NBA analytics as deriving value from the SportsVU data. What is SportsVU? Alamar tells us they installed cameras everywhere that "capture the coordinates of 10 players plus the ball 25 times every second." This is the typical "Big Data" scenario--data is collected without any design or any research question in mind. It raises a few intriguing questions:

• The granularity of such data (here it is 25 times a second, that is, to say, four-hundredth of a second apart) can be arbitrarily small. When have we reached the point of picking up just background noise?
• The very act of relating such data as "predictors" to outcomes such as scoring statistics presupposes the model in which the precise movements of the players or balls are correlated with those outcomes. Whether we like it or not, any resulting analysis will take on a causal interpretation--this is what separates trivia from an actionable insight. Is this type of predictor the most relevant to explaining outcomes? If not careful, we may just believe this story because that's the one we start with.

Careers in Data Science and Business Analytics: a new course

Data Science and Business Analytics are red-hot in the business world -- there are definitely more jobs than there are qualified people. Is this the right field for you? How do you find a job in this field? More importantly, how do you build a lasting career in analytics? I'll be addressing these questions in a new course I'm developing to be held in late March.

Here is the tentative outline of the one-day seminar. Please let me know if you have any suggestions. There is still time to change the contents.

Also, on my sister blog, I'm announcing the next installment of my data visualization workshop.

***

 

NEW YORK UNIVERSITY

SCHOOL OF PROFESSIONAL STUDIES

CAREERS IN DATA SCIENCE AND BUSINESS ANALYTICS

A One-day Seminar, Mar 28, 2015 (Saturday), New York City

By Kaiser Fung

 

Seminar Objectives

Data Science and Business Analytics have seen tremendous growth in the last few years, and there is fierce competition for the best talent. Numerous academic programs have sprung up, and many workers are making a career transition. The job market, while vibrant, is confusing because the field is fast evolving, and because the field encompasses different job types and career paths. The goal of this seminar is to help students develop a plan for finding a job in Data Science and/or Business Analytics, and building a lasting career in this field.

Specific objectives include: understanding the nature of the job, and the state of the job market; determining whether the job is right for you; creating a networking strategy; improving your resume; developing your pitch; making the decision to invest in further education, and building a lasting career.

The seminar is divided into four sessions. Interaction is encouraged throughout the day. Students complete hands-on exercises, especially in the third session; and receive handouts on tips and resources.

Course Outline