Big Data, Plainly Spoken (aka Numbers Rule Your World)

More poll fodder

In the previous post, we discussed xyopia and causal inference relating to poll results.

In this post, I want to discuss the administration of the poll. As a reminder, here is the snapshot I'm reacting to.

How many people were polled? How did these people learn about the poll? Were there any pre-screening?

I found the USA Today article related to this poll, in which they claim they had "more than 1,000" respondents. So let's assume they had 1,000 respondents.

That is not enough disclosure! The statement that employees are happier because of free food is a statement about the average employee at the average company. There are two levels of sampling involved here. It is unlikely that they first selected companies and then found employees within those companies. It's more likely that they found respondents and asked them where they work.

We don't know how many companies were represented by the 1,000 respondents. That depends on how they collected the data.

Let's consider two extreme cases.

If I told you that all 1,000 respondents came from two companies, one serving free food and one not, you'd be very skeptical of the result. You should be confident that company A's employees are happier than company B's but any conclusion beyond those two companies is dubious.

If I told you that 1,000 respondents came from 1,000 companies, you should be skeptical also. This means that the happiness measure for each company is based on a sample size of one employee. (Averaging across companies mitigates this issue a little.)

In practice, you get different numbers of respondents by company, which introduces yet another wrinkle in the analysis of the data. Some companies are given more weight than others.

***

The article discloses that only 16% of poll respondents work at companies offering free food. That would be 160 respondents, of which 107 rated their happiness in the top 2 boxes. How many companies did those 100 odd people represent? 

Of note, n=160 is very close to the minimum sample size needed to establish the conventional 95% confidence level for statistical significance - if both groups had about 160 respondents.

This means that the large sample size (~840) is crucial to the statistical conclusion. That size of sample makes the estimate of the no-free-food sample very "precise", plus/minus 3%. The estimate for the free-food sample is not precise, about plus/minus 7%.

One hopes that poll reporting comes with an appendix containing the missing information. It's useful for establishing trust.

 

Poll fodder inducing xyopia in elevators

Inside elevators in commercial buildings in New York, there are frequently screens that pump out commercials or infomercials all day long. I took a snapshot of this one that showed up last week:

These poll results annoy me in many ways. Let me explain.

The conclusion "free snacks is the key to happiness" is clearly wrong, even if the data were trustworthy (I'll talk about the data in a future post). I call this "xy myopia" or xyopia, in short. The writer of the poll has narrowly framed the problem: in this xyopic world, happiness hinges on just one factor - whether one's employer offers free food. In the real world, the happiness of an employee depends on a host of factors, of which free food is likely to be a minor one.

Xyopia arises from the reasonable concept of ceteris paribus ("all others equal"). Ceteris paribus is of little value in data analysis - it is often implicitly assumed by the analyst, but the assumption is false, which leads to bad conclusions.

In this case, the analyst just assumes that the two groups - employees getting free food, and those not getting free food - are equal on all other dimensions. So not true! The employer who offers free food is more likely to treat employees better across the board, such as providing more vacation days, better benefits, allowing work from home, etc. Besides, free food is only relevant in office settings (think Starbucks baristas). Certain demographic groups may value free food more than others. 

Unless it is established that the respondents who got free food are identical to the respondents who didn't in all other ways, the conclusion that free food is key results because it is the only factor you analyzed. This is xypoia.

(Not to mention, "free snacks" is not the same thing as "free food.")

***

Now, think about what question the poll writer asked that led to the headline "Free snacks is the key to happiness". It appears that they asked a pair of questions: "Does your employer offer free food?" and "Rate your happiness at work". Asking the first question frames your answer to the second question.

I'm going to make up my own poll:

A) Does your employer provide financial support for your personal development?

B) Rate your happiness at work (1 = extremely unhappy, 5 = extremely happy).

Based on this poll, I intend to conclude that paying for personal development is the key to employee happiness.

***

If you extend your vision to the basket of factors that affect employee happiness, such as their pay, benefits, treatment, work-life balance, family life, etc., you'll recognize more problems with the data analysis.

The poll result of 67% versus 56% is a strong result only if the gap is wider than for other factors, such as the poll I created above. I might find that 75% of the employees who get reimbursed for professional development answer extremely or very happy.

Also, consider a poll with three questions:

A) Does your employer provide free food?

B) If answer to A) is yes, do you consume the free food provided by your employer?

C) Rate your happiness at work (1 = extremely unhappy, 5 = extremely happy).

This creates three subsets of respondents (No - NA; Yes - No; and Yes - Yes) and leads to a better understanding of the causal mechanism.

***

Polls are easy to do but hard to do well.

 

 

Don't miss our networking mixer with star speakers on Tuesday

We are delighted to feature two great speakers at our Data Science Networking Night on Tuesday, June 26, 2018. The talks cover two aspects of analytics and data science that combine art and science. This is a founding principle of Principal Analytics Prep, the training program I founded to produce well-rounded individuals who bring critical thinking and technical prowess to data analysis.

Professor Andrew Gelman's talk will be on story-telling. He poses a paradox: in statistics, we aim for representative samples and averages but stories emphasize the extreme and unusual. Is it possible to use stories to narrate statistics?

Shira Mitchell, from Mathematica Policy Research, discusses her work with the Millennium Villages Project, aimed at improving well-being in Africa. Many real-world programs like the MVP generate observational data which present major challenges to analysts. How do we know if these projects achieve their targets?

After the talks, there will be an opportunity to meet other data scientists and analysts.

Food and beverages will be served.

The event is held in downtown Manhattan.

Due to overwhelming demand, we just opened some more spots. Register here before they run out!

Dead man tells no tales but dead statistician had the story down

On March 23, 2018, just after 9:30 am, a Tesla SUV plowed straight into a road divider as it was passing by the exit ramp to Route 85 from Route 101, in Mountain View, California. The impact caused the battery to burst into flames, and the driver was trapped in his seat by a seat belt. He died in the hospital.

Last week, the National Transportation Safety Bureau (NTSB) issued a preliminary report providing further details. Here are the key findings:

• The "AutoPilot" feature was switched on at the time of the crash. It was on continuously for the last 19 minutes of the drive (out of 32 minutes total).
• The driver's hands were not on the wheel in the last 6 seconds before the crash. In the last 60 seconds, hands were detected on three separate occasions, totaling 34 seconds.
• In the final 3 seconds, still with no hands on the wheel, the car accelerated from 62 mph to 71 mph, with "no pre-crash braking or evasive steering movement detected."
• The autopilot system was set to a cruise speed of 75 mph, which is above the speed limit of 65 mph.
• During the 19-minute stretch before the crash, the Autopilot system warned three times about putting hands on wheel, the last time was 15 minutes prior to the crash.
• The car with the single driver was on the HOV lane, which should be in effect at that time in the morning.

These are all telling observations. What did the reporter doing the Tesla beat at the Wall Street Journal think is the most salient finding? Well - the title of their article says it all: "Tesla Autopilot System Warned Driver to Put Hands on Wheel, U.S. Investigators Say." (link)

It is hard to imagine how that particular finding is the most important... unless you work for Tesla, or someone who wants to pin the blame on the driver who could not defend himself.

Problem #1

The Autopilot was continuously turned on for almost 20 minutes prior to the crash. The last warning to the driver happened "more than 15 minutes" before the crash. When driving at 60 mph and above, 15 minutes is 15 miles or more from the crash site. For example, the last warning might have been issued when passing Palo Alto, but the crash happened 15 minutes later in Mountain View. Rather than supporting the view that the driver recklessly ignored repeated warnings, this finding raises the question of why the car did not detect imminent collision, and take evasive action.

Problem #2

Hands not on wheel is taken to be a finding of major importance but here is the problem: every accident investigation will discover hands not on wheel. Why? Because our sample consists of only car-casses (pun intended).

If the driver had taken evasive steering action, it would have compensated for computer error, and the accident would have been avoided!

This is a reverse of the classical Wald paradox that I covered some years ago. In that example, statistician Abraham Wald warned that you couldn't inspect the damage on warplanes that returned home to determine which parts of the planes were most subject to damage - because your sample is missing the planes which were shot down!

In our example, we get to see the dead but not the survivors. In order to understand whether the computer made errors, we'd need to also include cases in which the driver prevented the Autopilot from crashing the car.

Problem #3

Given that the driver did not steer the wheel in the last seconds, why did the Tesla vehicle accelerate? Why was it allowed to go above the speed limit? This suggests technical challenges.