Big Data, Plainly Spoken (aka Numbers Rule Your World)

Statistical significance explained, using a recent political polling experiment

Our dataviz blog neighbor Alberto reacted to this chart in the New York Times:

The title of his post makes two important points: "Don't just visualize uncertainty; explain it and don't let captions contradict it". The New York Times article is here.

***

Don't let your visual contradict your message.

This is a piece of advice embedded in the Trifecta Checkup (see the Guide here). Notice the green arrow between the Q and V corners. This arrow means to make sure that the answer to our Question and the Visual design are in sync with each other. The chart should convey our message, not contradict it.

 

Alberto pointed to the wide interval for the Independents in the top section of the graphic:

The chart assumes readers understand the concept of statistical significance (a form of uncertainty). The wider the interval, the less reliable is the result.

In the experiment that was run, Independents who read articles about the Democrats shifting leftward were 6 percent less likely to vote for a Democrat in 2020, compared to Independents who read articles of "unrelated" topics. However, the size of the experiment was small, and so the uncertainty of that 6 percent is high.

If a different random sample of participants were used, then we expect to see a different result: not necessarily 6 percent. The bar tells us that the percent could reasonably range from -12% to roughly zero %. In a sense, the story here is the effect is likely to be negative but we need more data (more participants) to know the extent of the harm.

When the right edge of the bar is so close to zero, this is not a strong result. This is what concerns Alberto.

Further, the other three bars cross the zero line. This means that the results for Democrats, Republicans and the average voter are not conclusive. The experiment does not show that reading the left-shift articles would affect how they would vote.

I've just made a video explaining statistical significance. Check back here in the next few days. Here is the video. 

***

Looking at the entire set of results, every group crosses the zero line, and the other result worth noting - in the bottom chart - is that Democrats are marginally more likely to vote for their party's nominee in 2020 if they had read the articles about the left shift, compared to reading about "unrelated" topics.

These bars are relatively wide, and most results are not distinguishable from zero, which suggests that they need to expand the sample size.

***

There are other issues with the design of this experiment. It needs a "negative control". There should be a third test group that reads articles about the Republican party moving right. Surely, Independents also take into consideration what is going on in the Republican party.

Also, as stated in the first sentence of the New York Times piece, their real question is "the potential electoral penalty of repelling persuadable voters". The experiment however does not address persuadables in a direct way. This design flaw can be stated as such: not all persuadables are independents, and not all independents are persuadables. Beware of taking the results about Independents and substituting with persuadables.

I also have a small edit for the claim that "because of the random assignment [Ed: who reads which set of articles] ... the difference in responses between the groups can be attributed to the effect of reading about the leftward shift." This statement works only if we add "compared to reading about other topics labeled 'unrelated' in the experiment." We can't say anything about topics not used in the experiment, and as I pointed out above, we specifically don't know what the differential response would be against reading about the Republican party shifting rightward. Nor is it a given that all likely voters read the types of articles used in this experiment. 

 

[PS. The video explaining "not statistically significant" is now up and running. It gives another example - using the recent story about Instagram running an experiment to hide the Likes counter as fodder. Here's the link. ]

Vaping saves lives. Maybe.

My new Inside the Black Box video walks through how one evaluates the claim by Juul and e-cigarette companies that e-cigarettes will save lives.

This claim is substantively similar to Tesla's claim that its "autopilot" technology saves lives. See my previous video for a discussion of Tesla's claim.

Although stated absolutely, these technology vendors are in fact making a relative claim. Tesla argues that human drivers are more prone to accident fatalities while e-cigarette companies use smokers of tobacco as the strawman. In both cases, the question is whether they are making the right comparisons.

For the vaping analysis, I look at eight subgroups of people to see how they are affected. See the video for details. The following slide is a summary:

 

The credibility of the claim depends on (a) do smokers completely switch out of tobacco? (b) how many non-smokers pick up vaping? (c) are there harms from vaping as yet unknown? [pending results of the ongoing investigation]

***

Bonus for those wanting to dig in deeper:

The analysis sounds like a pre-post analysis of behavorial change across time surrounding an event of interest (the launch of vaping). While this interpretation is fine, it is limited because it's a point in time.

There is a better way of thinking about what we've done.

Given that we are now living in the "post-vaping" world, the "pre-vaping" world is a counterfactual. We are imagining how people would have behaved as if e-cigarettes did not exist. We observe the post-vaping world in which we live, and imagine the pre-vaping world we don't.

***

If you or your friend is looking to learn core data science skills, hands-on from senior industry professionals, check out Principal Analytics Prep's Part-Time Immersive Program for evening courses covering SQL databases, Python, Advanced Excel, Web Tech & Analytics, Analytics Strategy and Statistical Reasoning. See here for more information.

***

 

 

 

 

Tesla's slick stats

Tesla likes to advertise that its cars save lives. In the latest video, I examine this claim. The problem is that Tesla owners should not be compared to the average car buyer. Tesla is a luxury car and a status symbol. Tesla should be compared to other luxury cars. How does Tesla fare? Check out the video to find out.

***
One additional issue: what's the better metric for automotive fatality rate? Tesla uses deaths per million vehicle miles driven while others in industry use deaths per million vehicle-years.

This reminds me of the same issue in measuring air safety, a topic I addressed in my first book, Numbers Rule Your World (Chap 5) (link). Should airlines be measured based on deaths per million flight miles or deaths per million flights? MIT Professor Arnold Barnett looked into this issue and came out advocating for deaths per million flights. The main insight is that most air crashes happen at the start or end of a flight; the probability of a crash is not uniformly distributed over the miles of each journey. Thus, the death per million flights metric gets closer to the risk experience.

I'd imagine something similar at work here. The chance of getting killed in a car accident is not spread evenly throughout a trip. The additional risk is borne by taking an additional trip - once you're on a trip, the increase in risk over the distance of the trip isn't material.

***

To see the short video on Tesla, click here.

To see more of my videos, follow the channel.

If you are looking for evening classes to learn data science from industry experts in a high-touch, hands-on setting, check out the Fall Programs at Principal Analytics Prep.

 

 

 

The list of worst college majors, and why you should ignore it

Business Insider reports on the bottom 20 college majors, with the warning "what not to study". The list looks extremely random including everything from visual and performing arts to international relations to math and computer science (!).

The author explains these majors have the highest unemployment rates.

Here are a few reasons why you can and should ignore this study.

1. Presumably the journalist is advising college students, who will soon become new college graduates. But the analysis references college graduates which include anyone with a college degree, ranging from a new college graduate to someone who's worked for 40 years. New college graduates have a much higher unemployment rate than all college graduates. (See this recent Huffington Post article.)
   
   
2. Students do not randomly select a college major. Thus, students who choose to major in international relations are not the same type of people who decide to major in engineering. If an international relations student were to switch major to engineering, it would not follow that this person's employment prospect had suddenly brightened. It might even worsen because (a) the student could be relatively less competitive against other engineering majors; and (b) the student and the major might now be mismatched.
   
   
3. If everyone followed this journalist's advice, then each college would need a very small number of majors. Unless the economy suddenly produced a bonanza of jobs, where would the unemployed go? You got it... the unemployment rate would accrue to the several remaining majors, instead of being spread out over hundreds of majors. The unemployment rate of these "good" majors would rise to the level of the average unemployment rate. Oops.
   
   
4. Not everyone needs or wants a job. Changing majors doesn't change that reality.

***

There is an even bigger howler in the article. The analyst at Bankrate equated "job stability" with low unemployment rate, using language such as "Having a high-paying job doesn't necessarily mean you'll have job stability, and vice versa." So, majors with high unemployment rates are bad because people job-hop.

But the unemployment rate is not a measure of job tenure, and thus not an indicator of job stability. In fact, if people job-hop a lot, the unemployment rate will be relatively low, because the same job can be held by multiple people in the course of a year.

Let's imagine a country of 10 people with 5 jobs. The unemployment rate is 50%. If no one changes jobs, the same five people are employed always, and the rate is stable at 50%. The government stipulates that no person can hold on to the same job for longer than 6 months. We put the 10 people in a circle, and every other person is given a chair and seated. Every 6 months, each person moves clockwise by one slot: the five previously seated no longer have a seat, the five seats now occupied by the five previously standing. Everyone is now employed 6 months out of the year. The employment rate is now 100% (counting part-timers).

Thus, unemployment rate of zero can coincide with high job instability.