Big Data, Plainly Spoken (aka Numbers Rule Your World)

Statistical evidence is both powerful and limited: fraud in baseball and lotteries

In Chapter 5 of Numbers Rule Your World (link), I told the story of how Professor Rosenthal used statistical evidence to uncover a cheating scandal in a Canadian lottery. Statistics can be very powerful in identifying suspicious data - in this case, the winning probability of store owners who sell lottery tickets was found to be way higher than normal.

In a textbook presentation, you can simply do the math and compute the relevant probabilities and invoke statistical "significance" arguments. Hey, we've found evidence of cheating. End of story.

In real life, that would just signal the beginning of a long story. The statistical conclusion is merely that "someone cheated" or at best, that "someone cheated almost surely". But you can't catch the cheater(s) based on that evidence! The question left unanswered is: who cheated?

In the example from the book, we know that as a group, store owners cheated. That doesn't mean every store owner cheated. Who cheated? For that, you need old-school shoe leather. You need to find the receipts.

***

I was inspired to write this post after a friend asked me whether the Astros were the only baseball team that stole signs. Here is a summary of the cheating scandal engulfing professional baseball right now.

The statistical argument goes like this: the kind of sign-stealing using video can only happen at home stadiums. Teams who cheated in this way would have a performance advantage in their home games compared to the away games. But of course, every team enjoys a home-field advantage so we're talking about an abnormally high home-field advantage.

Where statistics demonstrates its power is here. We have guidelines for what level of home-field advantage should be considered abnormally high.

The question my friend asked is: If there was a second team who also did remarkably better at home games than away games - similar to the Astros, would that be sufficient evidence to convict that team as well?

Sadly, the power of statistics is also limited. We think cheating might be behind the abnormally high home-field advantage - but we can't be sure! We are not unsure about the observed outcome; the uncertainty concerns the cause of such an outcome.

Similar to the lottery fraud scheme described in Numbers Rule Your World (link). We are very confident that too many winners were store owners but nothing in the data tells us that the cause is cheating! The case didn't get solved until specific store owners were caught. You need to find receipts. 

Unless there is a confession by the cheaters, or a reporter or detective discovering incriminating evidence, we don't know what caused the anomaly in the data. Suspicion is not proof of wrongdoing.

In baseball's case, it was a former Astros pitcher who confessed to the scheme. Without receipts, the cheaters can always offer other explanations for the performance advantage. The most powerful alternative explanation is coincidence.

***

Here's an example of an alternative story. Someone could claim that their hitters started making "superman" poses before they get on the field, and that explained the advantage.

But spinning such stories is to run "story time". We have some data - the performance advantage - that show something. Then, we seamlessly move to a story that has nothing to do with the data just presented. Just because there is a presentation of data doesn't mean the data support the story!

 

 

Quick hits on data issues in the Democratic primaries: cancelled poll, and cost of complexity

The Democratic primaries are generating - unexpectedly - stories with good data angles.

First, there was the "cancelled" Selzer poll leading up to Monday's contest (link). While we still don't have a proper explanation, what was reported in the press is that there was a single instance in which Pete Buttigieg's name might have not been read, possibly due to the long list of candidates, some of whom will be "off screen". In this case, the alleged instance is due to someone increasing the font size on the screen, thus moving names off the viewable area.

I find this explanation inadequate. In any survey involving thousands, there will be a few issues - kind of like dropping bad ballots. What if, for example, a phone connection got momentarily mangled just as Buttigieg's name was read? If the Buttigieg's camp found one bad case, they surely went around looking for more examples - and as far as we're told, they didn't find any more.

Further, any competent survey designer - and Selzer is a good one, according to Nate Silver - knows that you always randomize a list of responses. In other words, the order in which the candidates are listed for the same question will be randomly assigned for every phone call. So if Buttigieg's name got missed because it's at the bottom of the list for some calls, his name would be at the top of the list for other calls. If x percent of the calls are flawed because of omitting names, each candidate will be affected by the same amount. The randomization equalizes the impact of such unpredictable errors.

Cancelling such an important poll is a decision that changes the dynamics of the race, and a much more thorough accounting is needed.

Selzer's explanation is an "abundance of caution". If true, this instance opens up a flood gate for delegitimizing polls. If a complaint based on a single anomaly is allowed to topple a poll, then any candidate can conspire to preempt poll results that s/he doesn't expect to like.

The decision to cancel the poll is not fair to all parties. Turns out someone leaked partial results of that poll on the day of the Iowa primary (I saw this on FiveThirtyEight, which was not the source of the leak). It clearly shows that some candidates would be happier with the poll result than other candidates.

***

In the turmoil around the Iowa caususes, we must differentiate between what the data show and what might explain the data. There is no doubt that this screwup hurts the winner, and any other candidate that did unexpectedly better while it helps any candidate who did unexpectedly worse. This outcome is also not fair to all parties.

The problem is when we impute a cause, or motivation to explain the effect. We don't know if someone conspired to help or hurt any particular candidate - unless specific corroborating evidence is subsequently found.

Yet, we don't need a motivation to agree on the positive/negative effects on specific candidates.

***

Further, people are underestimating the cost of complexity. Having two candidates in 2016 and having 10 candidates now makes things much more complicated. (I'm using two and ten for illustration. The list is longer if we include every fringe candidate.)

This increase in candidates is magnified by having multiple rounds of voting, and realignments. With two candidates, the realignment decision is either to join the other side or go home. With 10 candidates, there are so many possibilities!

Then, apparently, they also promised to provide vast amounts of data. This policy is itself commendable. With two candidates, validating the numbers is easy. There is really only one number per precinct to check. With ten candidates, there are nine numbers per precinct to check. There are over 1,000 precincts. The resulting tabulation has lots of missing values (holes) because by definition, most candidates become non-viable by the end of the night for each precinct but the set of candidates that remain viable vary by precinct.

A simple way to think about the cost of complexity is this: each number has some unknown error rate, say 1 percent. If we need an accurate result, we need the probability of no errors. That is, 1 minus the probability of one error minus the probability of two errors ... minus the probability of all errors. The 1-percent error gets magnified very fast as the number of entities grow!

 

 

In times of paranoia, seek out data and stick to facts

All around the world, people are getting very afraid of the coronavirus. What's happening now is a live version of a disease outbreak investigation, which is discussed in Chapter 2 of Numbers Rule Your World (link).

To understand what's going on, you should see this as the exact opposite of "big data". We have a new flu virus. There is almost no direct data on this virus. The data are dribbling in as cases are being reported. We have a couple hundred deaths.... in a country with almost 1.5 billion people. Public health officials must make key decisions now before more data become available. But we have already passed the most dangerous phase, because they already found what causes the disease.

I saw a graphic last night, shown during an interview with an expert who investigated the SARS epidemic. Supposedly the mortality rate for the coronavirus is about 3 percent. This number has a large margin of error at this stage because of the shortage of data. This shortage is due to the newness of the phenonemon.

The closest relative to the coronavirus is the SARS virus which had a mortality rate of about 10 percent, which is three times higher than what we currently know about the coronavirus.

We also know that the coronavirus spreads faster than SARS. Most of the news reports I read mention the number of infected but rarely the number of deaths. This is a major oversight.

What's fearful is a deadly disease. A disease that infects lots of people but acts like the common cold is not a public-health crisis. So as data roll in, we should keep an eye on the mortality rate.

The mortality rate is the number of deaths divided by the number of infected. Both counts are growing each day. If the growth of infected is faster than the growth of deaths, the mortality rate will go down (not up!). The mortality rate will go up only if the growth rate of deaths exceeds the growth rate of infected.

In fact, by the time I finish this post, the latest report said the coronavirus death rate is 2 percent, so it has already been adjusted downwards because the flu is spreading fast. (The death rate of regular flu is about 1 percent, according to the graphic that was shown last night. This level acts as a control for interpreting the mortality rate.)

That report contains a typically alarmist quote by a medical professional: "There are likely to be many times more cases in Wuhan than officially confirmed". Okay, but if that is true, the real mortality rate is many times smaller than what is official! In terms of missing data, it's the unreported deaths that are more significant than unreported infections.

Also note the network effect. It's very unlikely that the growth rate of deaths can be higher than the growth rate of infected because infection grows exponentially due to transmissions. Deaths don't transmit.

It would also be useful to know what the cure rate is - because some of the infected will be in the treatment stage. During the report last night, the journalist wrongly asserted that 97 out of 100 infected will survive. This interpretation is correct only if we exclude those infected that are currently in treatment. (The technical term is right-censored data. We don't yet know how many in this subgroup will survive or not.)

While one should be cautious because this coronavirus is new, in times of paranoia, one should seek out the data. Ignore the unfounded rumors, innuendo and folklore. Stick to the facts. So far, the data aren't that alarming.

***

I n Chapter 2 of Numbers Rule Your World (link), I describe two modes of statistical modeling. Disease outbreak investigation illustrates one mode, in which the chief concern is establishing cause-effect relationships. This is always hard if you have observational data - without controls. But it's even more challenging in these crisis situations, with small drips of data. If you understand what statistical tools are available, and what the limitations are, you have a better grasp of what's going on right now.

 

Image credit: By https://www.scientificanimations.com - https://www.scientificanimations.com/wiki-images/, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=86358105

 

P.S. [2/3/20] See also my post on Junk Charts about a graphic used by the New York Times in a report on the virus.

The absurdity of predictive policing

Big (surveillance) data is changing everything - that is undeniable. But is Big Data changing everything for the better? That is what you have to think about.

As someone who's done data analytics for a long time, I can say confidently that the potential for harm is at least as large, but probably larger, than the potential for good.

A case in point is data as related to crime and policing. This post picks up from the perceptive article in the New York Times, "How Reporting Practices can Skew Crime Statistics" (link).

The article points out something really important - data is as data is defined. Different definitions result in different analytical results, frequently contradictory. No one should interpret analytical results without first learning the data definitions.

The author describes two extremely popular styles of "big data" analytics: top N ranking, and trend analysis. In crime statistics, sample insights are (1) "South Bend, Indiana is the 27th worst city for violent crime rate among all cities with over 100,000 people." and (2) "Violent crime has worsened in South Bend since mayor Pete got elected in 2012."

With the volume of data available today, those two simple analyses still account for the vast majority of "analytics" out there.

South Bend, Indiana is the 27th worst city for violent crime rate among all cities with over 100,000 people.

Here is a list of definitions that affect the interpretation of this statement, most of which are described in the NYT article:

• whether the suburbs are included as part of the "city", or more broadly, how the boundaries of the city are drawn. This matters because the crime rates in the suburbs are different from the crime rates in the city centers.
• whether the error in the count of population is comparable across all cities under consideration
• what is considered a crime
• what proportion of crime is reported
• what crime is classified as "violent"
• are there any exceptions

You might think the population of a city is an objective fact but it isn't. Definitions matter, and they matter a lot when you're comparing different cities.

Violent crime has worsened in South Bend since mayor Pete got elected in 2012.

In addition to the points listed above, we also need to know if any of those definitions changed between 2012 and 2019. Changing how something is defined almost always breaks the continuity of any analysis, thus undermining any trend analysis. I raised this point in my book Numbersense (link) in the chapter on obesity - if the medical community changes how we measure obesity, then all historical data are rendered useless so such proposals must be considered using great caution.

Further, we should find out whether violent crime was increasing or decreasing prior to 2012 in South Bend, whether violent crime was increasing or decreasing in other cities comparable to South Bend since 2012, whether violent crime was increasing or decreasing in the nation as a whole since 2012, etc.

***

Some readers of my book Numbersense (link) might be mystified why I spent so much energy discussing definitions - of grad school admissions statistics, of U.S. News ranking, of obesity measures, of unemployment rates, of inflation rates, of fantasy football metrics, etc., etc.

A key to numbersense - to judging the statistics and data thrown at you - is to understand clearly how the data are collected, and in particular, how any metric is defined.

The discussion of crime statistics by the New York Times makes this case cogently.

***

Now let me get to the absurdity of predictive policing.

A particularly popular and cited application of Big Data "for social good" is predictive policing. The goal of predictive policing is to prevent crime from happening.

The ideal of "Minority Report" is often hoisted on us. Wouldn't it be great if some AI or machine learning or predictive model can figure out who will be burglarizing your home next year, and have the police lock the would-be burglar up before s/he could commit the crime?

I have done a good amount of predictive modeling in my career so I am not saying what I'm saying because I'm a luddite. It's because of that experience that I am leading you down the following rabbit hole:

There are not many burglaries. According to Statista, the worst state (New Mexico) suffered 770 burglaries per 100,000 residents in 2018. This means that if the analyst is given a list of 100,000 New Mexicans chosen at random, a perfect model should label 770 of those as potential burglars. That is to say, we want the model to pick out 0.77 percent of the list.

A perfect model does not exist. Any model generates false positives. Because burglars are rare, any reasonably predictive model would need to red-flag many more than 0.77 percent of the list to have any chance of capturing all 770 potential burglars. Let's say the model points the finger at 1 percent of the list. That would mean 1,000 potential burglars. Since there should be only 770 burglars, the first thing we know is that at least 230 innocent people would be flagged by this model.

Next, think about what happens to the 1,000 flagged people. In the ideal of predictive policing, police would knock on their doors and arrest them for their future crimes. These people who haven't burglarized anyone would be duly jailed by courts who buy into predictive policing. (How would they defend themselves against an allegation of future crime?) 

We could say with certainty that those 1,000 did not subsequently commit burglary - given that they were incarcerated.

So humor me, and try to answer the following question: Of the 1,000 people flagged by this model, how many of these were accurately predicted?

 

 

 

A student skipping class might be grievously depressed. Claims like these discredit data science

This is Part 3 of a multi-part post on student surveillance technologies. You many want to read Part 1 and Part 2 first.

In the Washington Post article on student surveillance, one of the featured use cases is predictive analytics. In short, the location data are merged with other data to “score” students. Scoring is a form of rating by means of a formula.

The most famous example of predictive scoring is the ubiquitous and consequential credit scoring system. The credit score is a prediction of whether an individual is likely to repay the credit card company which is lending him or her money to purchase something today (without any collateral). I consider it a fundamental example of big data analytics, and have an extensive discussion of credit scoring in Chapter 2 of Numbers Rule Your World (link).

[Image credit: Frankieleon on Flickr]

***

Misunderstanding predictive analytics is flooding the marketplace with lots of defective products that harm people. I'm quite concerned by the language used by the business folks in the Post article to sing the praises of predicting student well-being. These statements reveal unfounded optimism about the accuracy of predictive analytics, and a shallow grasp of cause-effect relationships.

One vendor presented how their predictive model works: “a student avoiding the cafeteria might suffer from food insecurity or an eating disorder”. This kind of overly simplistic analysis is giving data science a bad name.

In reality, an extreme minority of people who avoid the cafeteria will be suffering from an eating disorder. Most scoring algorithms work with not one but multiple predictive factors, and even then, such an app will trigger many false alarms.

Even more shocking is the deduction that “a student skipping class might be grievously depressed”. It’s not the lack of modeling skill that concerns me here; it is the lack of common sense. For each student who skips class because of “grievous” depression, there are tons of non-depressed students who skip class. The insertion of “grievous” makes it even worse.

Behind any such deduction is a cause-effect model. It should be clear that Skip Class -> Grievously Depressed does not fly.

This may be a fallacy of transitivity. Grievously depressed -> Skip class is likely to hold. But reversing the causal order does not work.

***

Flawed data science products have consequences. Welfare checks are wasted on large numbers of healthy students. Students rightfully demand to be treated as adults. The truly grievously depressed students meanwhile may be missed.  

***

In Chapter 4 of Numbers Rule Your World (link), and Chapter 5 of Numbersense (link), I discuss ways to examine predictive models, and dealing with their inaccuracy.

New data on vaping

In my video about vaping (link to video, link to blog post), I explained why the e-cigarette industry’s claim that vaping saves lives is a half-truth. Their argument assumes people who vape would have died from smoking traditional cigarettes if they hadn't chosen to vape instead.

That’s true only when all of the following conditions apply:

1. The person would have smoked cigarettes if e-cigarettes were not available
2. The person switched completely to e-cigarettes
3. The person would have died from smoking-related illnesses had s/he smoked cigarettes
4. The person would not have died from vaping-related or other illnesses had s/he vaped
5. The person would not have undertaken any other measure, available now or in the future, to mitigate the risk of death

(1) fails for every vaper who would never have smoked cigarettes.

We expect this category to be sizeable – precisely because powerful marketing asserts that vaping is safe. The potential harm of a new product is hard to quantify at the start but we now know that people have died from it.

What’s easier to establish is how big this problem is. A recent Wall Street Journal article gives us the data: according to a federal survey, 20% of all middle- and high-school students in the U.S. “recently used e-cigarettes”, compared to 4% who used cigarettes. If the prevalence of cigarettes is stable, then four out of five vapers in school would never have shouldered the risk of smoking-related death but are newly exposed to possible harm through e-cigarettes.

***

In thinking about point (3), keep in mind that not all smokers die from smoking-related illnesses. You should only count the preventable deaths, that is to say, the person would have lived longer had s/he not smoked. What proportion of smokers actually die from smoking-related illnesses?

This question is harder to answer than one thinks. The basic analysis would be identifying who's the smoker, and whether the person died from a disease associated with smoking such as lung cancer. However, there are few diseases with a single cause so even if a smoker died from lung cancer, it's not assured that smoking is the cause of death. The counterfactual question we're interested in is whether this same person would have died from that disease had s/he not smoked. To complicate this further, research showed that if someone quit smoking early enough, the risk factor goes down by 90 percent. CDC has this page that shows some relevant numbers.

***

My video takes a deeper look into the different groups of people that may be impacted by vaping.

 

The research on SAT bias is great data science

I just put up a follow-up video on the SAT controversy, including a speculation about what might happen if the SAT were to be toppled from college admission. Link to video.

***

One of the key arguments against the SAT is its alleged bias against certain subgroups of test-takers. There has been a lot of research into ways to measure and combat such biases. Chapter 5 of Numbers Rule Your World (link) gets into the science behind this.

In the video, I discuss the following example featured also in the book

Is this test item biased against some subgroup? If so, which subgroup, by how much, and for what reason? It turns out that it is basically impossible to answer these questions purely by intuition. Test designers run experiments to measure the bias, and are often surprised by what they find.

This whole area of research is a great example of a practical application of data science. The issues range from how to collect the right data, and how to analyze such data to make causal conclusions. You can read about it in my book (link).

Or try the video first.

 

The Harden-Miami link: thinking about indirect metrics in an analysis plan

A co-worker pointed me to a Reddit thread that someone put up to showcase a small data-science side project - to establish whether NBA star James Harden's performance is affected by his "affinity for ... strip clubs" (thereafter, SC).

The analytical plan used in this project is typical of many data science projects. In this post (and possibly a few more), I raise issues that analysts should ponder when conceptualizing this type of analysis. These comments are designed to help those who are interested in advancing the sophistication of their analytical work. I am not re-analyzing the data, or debunking what's been done. I am impressed by what "AngryCentrist" has put together - starting with how AC gathered the data to how AC executed an analytical plan resulting in some conclusions.

***

Let me summarize the analytical game plan as follows: (you can read the gory details in this google doc)

1. The outcome variable is John Harden's on-court performance, expressed in an inverse scale (i.e. the larger the metric, the worse his performance). The explanatory variable is the quality rating of SCs in the cities where Harden played away games. The analysis looks at the correlation between these two variables.
2. Only the last four seasons of away games in which Harden played are used. In each season, there are usually one or two away games against any given team. Team and city are synonymous except for Los Angeles which fields two teams. (Brooklyn and New York are treated as separate cities.)
3. On-court performance is measured as the number of metrics on which Harden's in-city performance is worse than his season average. Only six metrics (e.g turnovers) are considered over four seasons so this performance metric is a number between 0 and 24. Zero means Harden out-performed his season average for each of the 6 metrics over each of the 4 seasons. By contrast, if Harden did worse than his season average for every metric over those four seasons, then the performance metric attains the maximum value of 24. The redditor, AngryCentrist, called this the "number of sub-par performances".
4. SC ratings are based on the first 10 results with customer ratings when doing a search on Google of the type "[city] strip clubs".
5. The analyst concluded that the two variables were positively correlated based on fitting a linear regression model. Higher SC quality rating is associated with worse on-court performance. The r-squared was 20 percent (which is not impressive).

***

For this post, I focus on the explanatory variable, the SC average rating, and the concept of "noise" in our data.

The SC average rating is an example of an indirect metric, a "noisy" proxy for what we want to measure. Notice that the outcome variable is about a particular person (Harden) while the explanatory variable measures a large set of people, of which Harden is allegedly a member.

The "noise" comes from several sources. If Harden was not a member of the set, then the explanatory variable has no direct connection with the outcome. So we hope that Harden not only visited SCs in all those cities, but also reviewed his experiences on Google, thus contributing to the average ratings.

If Harden did enjoy himself at the SCs and submitted reviews to Google, his ratings would still have been immaterial, just one vote in the average. Since the analysis only extracted "top 10" SCs based on a Google search, one expects the selected SCs are the more popular ones, which means that Harden's ratings would be even more insignificant.

If we don't require Harden's ratings to be included in these SC averages, we would at least require that his ratings - if revealed - to be highly correlated with the average ratings among Google reviewers. This assumption has two components - Harden would have to patronize the popular SCs, rather than ones that are more exclusive, or discreet; and Harden's opinion of his experiences there would have to be similar to that of the average patron.

Let's pretend that Harden visited one club prior to playing an away game in Miami. The average rating of the top 10 clubs in Miami is then not quite up to the task. That specific club might not even be part of the ten. This gap is also part of the "noise" of indirect measurement.

Timing is yet another source of noise. What if Harden engaged in the adult activity after the game rather than before the game?

Timing enters the picture in another way. Adult entertainment venues come and go, just like any other businesses. If Harden had visited a club before a match early in the NBA season, and that club subsequently shut down, then this club would not feature in the Google search, and would be absent from the average rating. Besides, new clubs that opened after Harden's visit would enter the average rating, if they become popular enough. Either of these situations causes the indirect metric to stray away from its proper value.

***

Here's the big picture. The indirect measure of average quality rating of SCs is a good proxy only under a large set of presumptions. None of the above conditions is directly observable, so each is presumed.

In general, the further away the measurement is from what is supposed to be measured, the more "noisy" is the measurement.

A workaround is to change our theory of the link between SC rating and Harden's performance. Maybe the link does not require physical visits to these establishments and subsequent reviews. Perhaps it is Harden's knowledge of a vibrant SC scene, regardless of active participation, that is doing the trick. This link though cannot be proven or disproven by the data, and thus is also presumed.

Assumptions are not avoidable. Verifiable assumptions are better than unverifiable ones. A good analysis plan should make the fewest, and weakest assumptions.

 

 

 

Statistical significance explained, using a recent political polling experiment

Our dataviz blog neighbor Alberto reacted to this chart in the New York Times:

The title of his post makes two important points: "Don't just visualize uncertainty; explain it and don't let captions contradict it". The New York Times article is here.

***

Don't let your visual contradict your message.

This is a piece of advice embedded in the Trifecta Checkup (see the Guide here). Notice the green arrow between the Q and V corners. This arrow means to make sure that the answer to our Question and the Visual design are in sync with each other. The chart should convey our message, not contradict it.

 

Alberto pointed to the wide interval for the Independents in the top section of the graphic:

The chart assumes readers understand the concept of statistical significance (a form of uncertainty). The wider the interval, the less reliable is the result.

In the experiment that was run, Independents who read articles about the Democrats shifting leftward were 6 percent less likely to vote for a Democrat in 2020, compared to Independents who read articles of "unrelated" topics. However, the size of the experiment was small, and so the uncertainty of that 6 percent is high.

If a different random sample of participants were used, then we expect to see a different result: not necessarily 6 percent. The bar tells us that the percent could reasonably range from -12% to roughly zero %. In a sense, the story here is the effect is likely to be negative but we need more data (more participants) to know the extent of the harm.

When the right edge of the bar is so close to zero, this is not a strong result. This is what concerns Alberto.

Further, the other three bars cross the zero line. This means that the results for Democrats, Republicans and the average voter are not conclusive. The experiment does not show that reading the left-shift articles would affect how they would vote.

I've just made a video explaining statistical significance. Check back here in the next few days. Here is the video. 

***

Looking at the entire set of results, every group crosses the zero line, and the other result worth noting - in the bottom chart - is that Democrats are marginally more likely to vote for their party's nominee in 2020 if they had read the articles about the left shift, compared to reading about "unrelated" topics.

These bars are relatively wide, and most results are not distinguishable from zero, which suggests that they need to expand the sample size.

***

There are other issues with the design of this experiment. It needs a "negative control". There should be a third test group that reads articles about the Republican party moving right. Surely, Independents also take into consideration what is going on in the Republican party.

Also, as stated in the first sentence of the New York Times piece, their real question is "the potential electoral penalty of repelling persuadable voters". The experiment however does not address persuadables in a direct way. This design flaw can be stated as such: not all persuadables are independents, and not all independents are persuadables. Beware of taking the results about Independents and substituting with persuadables.

I also have a small edit for the claim that "because of the random assignment [Ed: who reads which set of articles] ... the difference in responses between the groups can be attributed to the effect of reading about the leftward shift." This statement works only if we add "compared to reading about other topics labeled 'unrelated' in the experiment." We can't say anything about topics not used in the experiment, and as I pointed out above, we specifically don't know what the differential response would be against reading about the Republican party shifting rightward. Nor is it a given that all likely voters read the types of articles used in this experiment. 

 

[PS. The video explaining "not statistically significant" is now up and running. It gives another example - using the recent story about Instagram running an experiment to hide the Likes counter as fodder. Here's the link. ]

Vaping saves lives. Maybe.

My new Inside the Black Box video walks through how one evaluates the claim by Juul and e-cigarette companies that e-cigarettes will save lives.

This claim is substantively similar to Tesla's claim that its "autopilot" technology saves lives. See my previous video for a discussion of Tesla's claim.

Although stated absolutely, these technology vendors are in fact making a relative claim. Tesla argues that human drivers are more prone to accident fatalities while e-cigarette companies use smokers of tobacco as the strawman. In both cases, the question is whether they are making the right comparisons.

For the vaping analysis, I look at eight subgroups of people to see how they are affected. See the video for details. The following slide is a summary:

 

The credibility of the claim depends on (a) do smokers completely switch out of tobacco? (b) how many non-smokers pick up vaping? (c) are there harms from vaping as yet unknown? [pending results of the ongoing investigation]

***

Bonus for those wanting to dig in deeper:

The analysis sounds like a pre-post analysis of behavorial change across time surrounding an event of interest (the launch of vaping). While this interpretation is fine, it is limited because it's a point in time.

There is a better way of thinking about what we've done.

Given that we are now living in the "post-vaping" world, the "pre-vaping" world is a counterfactual. We are imagining how people would have behaved as if e-cigarettes did not exist. We observe the post-vaping world in which we live, and imagine the pre-vaping world we don't.

***

If you or your friend is looking to learn core data science skills, hands-on from senior industry professionals, check out Principal Analytics Prep's Part-Time Immersive Program for evening courses covering SQL databases, Python, Advanced Excel, Web Tech & Analytics, Analytics Strategy and Statistical Reasoning. See here for more information.

***