Big Data, Plainly Spoken (aka Numbers Rule Your World)

The research on SAT bias is great data science

I just put up a follow-up video on the SAT controversy, including a speculation about what might happen if the SAT were to be toppled from college admission. Link to video.

***

One of the key arguments against the SAT is its alleged bias against certain subgroups of test-takers. There has been a lot of research into ways to measure and combat such biases. Chapter 5 of Numbers Rule Your World (link) gets into the science behind this.

In the video, I discuss the following example featured also in the book

Is this test item biased against some subgroup? If so, which subgroup, by how much, and for what reason? It turns out that it is basically impossible to answer these questions purely by intuition. Test designers run experiments to measure the bias, and are often surprised by what they find.

This whole area of research is a great example of a practical application of data science. The issues range from how to collect the right data, and how to analyze such data to make causal conclusions. You can read about it in my book (link).

Or try the video first.

 

The Harden-Miami link: thinking about indirect metrics in an analysis plan

A co-worker pointed me to a Reddit thread that someone put up to showcase a small data-science side project - to establish whether NBA star James Harden's performance is affected by his "affinity for ... strip clubs" (thereafter, SC).

The analytical plan used in this project is typical of many data science projects. In this post (and possibly a few more), I raise issues that analysts should ponder when conceptualizing this type of analysis. These comments are designed to help those who are interested in advancing the sophistication of their analytical work. I am not re-analyzing the data, or debunking what's been done. I am impressed by what "AngryCentrist" has put together - starting with how AC gathered the data to how AC executed an analytical plan resulting in some conclusions.

***

Let me summarize the analytical game plan as follows: (you can read the gory details in this google doc)

1. The outcome variable is John Harden's on-court performance, expressed in an inverse scale (i.e. the larger the metric, the worse his performance). The explanatory variable is the quality rating of SCs in the cities where Harden played away games. The analysis looks at the correlation between these two variables.
2. Only the last four seasons of away games in which Harden played are used. In each season, there are usually one or two away games against any given team. Team and city are synonymous except for Los Angeles which fields two teams. (Brooklyn and New York are treated as separate cities.)
3. On-court performance is measured as the number of metrics on which Harden's in-city performance is worse than his season average. Only six metrics (e.g turnovers) are considered over four seasons so this performance metric is a number between 0 and 24. Zero means Harden out-performed his season average for each of the 6 metrics over each of the 4 seasons. By contrast, if Harden did worse than his season average for every metric over those four seasons, then the performance metric attains the maximum value of 24. The redditor, AngryCentrist, called this the "number of sub-par performances".
4. SC ratings are based on the first 10 results with customer ratings when doing a search on Google of the type "[city] strip clubs".
5. The analyst concluded that the two variables were positively correlated based on fitting a linear regression model. Higher SC quality rating is associated with worse on-court performance. The r-squared was 20 percent (which is not impressive).

***

For this post, I focus on the explanatory variable, the SC average rating, and the concept of "noise" in our data.

The SC average rating is an example of an indirect metric, a "noisy" proxy for what we want to measure. Notice that the outcome variable is about a particular person (Harden) while the explanatory variable measures a large set of people, of which Harden is allegedly a member.

The "noise" comes from several sources. If Harden was not a member of the set, then the explanatory variable has no direct connection with the outcome. So we hope that Harden not only visited SCs in all those cities, but also reviewed his experiences on Google, thus contributing to the average ratings.

If Harden did enjoy himself at the SCs and submitted reviews to Google, his ratings would still have been immaterial, just one vote in the average. Since the analysis only extracted "top 10" SCs based on a Google search, one expects the selected SCs are the more popular ones, which means that Harden's ratings would be even more insignificant.

If we don't require Harden's ratings to be included in these SC averages, we would at least require that his ratings - if revealed - to be highly correlated with the average ratings among Google reviewers. This assumption has two components - Harden would have to patronize the popular SCs, rather than ones that are more exclusive, or discreet; and Harden's opinion of his experiences there would have to be similar to that of the average patron.

Let's pretend that Harden visited one club prior to playing an away game in Miami. The average rating of the top 10 clubs in Miami is then not quite up to the task. That specific club might not even be part of the ten. This gap is also part of the "noise" of indirect measurement.

Timing is yet another source of noise. What if Harden engaged in the adult activity after the game rather than before the game?

Timing enters the picture in another way. Adult entertainment venues come and go, just like any other businesses. If Harden had visited a club before a match early in the NBA season, and that club subsequently shut down, then this club would not feature in the Google search, and would be absent from the average rating. Besides, new clubs that opened after Harden's visit would enter the average rating, if they become popular enough. Either of these situations causes the indirect metric to stray away from its proper value.

***

Here's the big picture. The indirect measure of average quality rating of SCs is a good proxy only under a large set of presumptions. None of the above conditions is directly observable, so each is presumed.

In general, the further away the measurement is from what is supposed to be measured, the more "noisy" is the measurement.

A workaround is to change our theory of the link between SC rating and Harden's performance. Maybe the link does not require physical visits to these establishments and subsequent reviews. Perhaps it is Harden's knowledge of a vibrant SC scene, regardless of active participation, that is doing the trick. This link though cannot be proven or disproven by the data, and thus is also presumed.

Assumptions are not avoidable. Verifiable assumptions are better than unverifiable ones. A good analysis plan should make the fewest, and weakest assumptions.

 

 

 

Statistical significance explained, using a recent political polling experiment

Our dataviz blog neighbor Alberto reacted to this chart in the New York Times:

The title of his post makes two important points: "Don't just visualize uncertainty; explain it and don't let captions contradict it". The New York Times article is here.

***

Don't let your visual contradict your message.

This is a piece of advice embedded in the Trifecta Checkup (see the Guide here). Notice the green arrow between the Q and V corners. This arrow means to make sure that the answer to our Question and the Visual design are in sync with each other. The chart should convey our message, not contradict it.

 

Alberto pointed to the wide interval for the Independents in the top section of the graphic:

The chart assumes readers understand the concept of statistical significance (a form of uncertainty). The wider the interval, the less reliable is the result.

In the experiment that was run, Independents who read articles about the Democrats shifting leftward were 6 percent less likely to vote for a Democrat in 2020, compared to Independents who read articles of "unrelated" topics. However, the size of the experiment was small, and so the uncertainty of that 6 percent is high.

If a different random sample of participants were used, then we expect to see a different result: not necessarily 6 percent. The bar tells us that the percent could reasonably range from -12% to roughly zero %. In a sense, the story here is the effect is likely to be negative but we need more data (more participants) to know the extent of the harm.

When the right edge of the bar is so close to zero, this is not a strong result. This is what concerns Alberto.

Further, the other three bars cross the zero line. This means that the results for Democrats, Republicans and the average voter are not conclusive. The experiment does not show that reading the left-shift articles would affect how they would vote.

I've just made a video explaining statistical significance. Check back here in the next few days. Here is the video. 

***

Looking at the entire set of results, every group crosses the zero line, and the other result worth noting - in the bottom chart - is that Democrats are marginally more likely to vote for their party's nominee in 2020 if they had read the articles about the left shift, compared to reading about "unrelated" topics.

These bars are relatively wide, and most results are not distinguishable from zero, which suggests that they need to expand the sample size.

***

There are other issues with the design of this experiment. It needs a "negative control". There should be a third test group that reads articles about the Republican party moving right. Surely, Independents also take into consideration what is going on in the Republican party.

Also, as stated in the first sentence of the New York Times piece, their real question is "the potential electoral penalty of repelling persuadable voters". The experiment however does not address persuadables in a direct way. This design flaw can be stated as such: not all persuadables are independents, and not all independents are persuadables. Beware of taking the results about Independents and substituting with persuadables.

I also have a small edit for the claim that "because of the random assignment [Ed: who reads which set of articles] ... the difference in responses between the groups can be attributed to the effect of reading about the leftward shift." This statement works only if we add "compared to reading about other topics labeled 'unrelated' in the experiment." We can't say anything about topics not used in the experiment, and as I pointed out above, we specifically don't know what the differential response would be against reading about the Republican party shifting rightward. Nor is it a given that all likely voters read the types of articles used in this experiment. 

 

[PS. The video explaining "not statistically significant" is now up and running. It gives another example - using the recent story about Instagram running an experiment to hide the Likes counter as fodder. Here's the link. ]

Vaping saves lives. Maybe.

My new Inside the Black Box video walks through how one evaluates the claim by Juul and e-cigarette companies that e-cigarettes will save lives.

This claim is substantively similar to Tesla's claim that its "autopilot" technology saves lives. See my previous video for a discussion of Tesla's claim.

Although stated absolutely, these technology vendors are in fact making a relative claim. Tesla argues that human drivers are more prone to accident fatalities while e-cigarette companies use smokers of tobacco as the strawman. In both cases, the question is whether they are making the right comparisons.

For the vaping analysis, I look at eight subgroups of people to see how they are affected. See the video for details. The following slide is a summary:

 

The credibility of the claim depends on (a) do smokers completely switch out of tobacco? (b) how many non-smokers pick up vaping? (c) are there harms from vaping as yet unknown? [pending results of the ongoing investigation]

***

Bonus for those wanting to dig in deeper:

The analysis sounds like a pre-post analysis of behavorial change across time surrounding an event of interest (the launch of vaping). While this interpretation is fine, it is limited because it's a point in time.

There is a better way of thinking about what we've done.

Given that we are now living in the "post-vaping" world, the "pre-vaping" world is a counterfactual. We are imagining how people would have behaved as if e-cigarettes did not exist. We observe the post-vaping world in which we live, and imagine the pre-vaping world we don't.

***

If you or your friend is looking to learn core data science skills, hands-on from senior industry professionals, check out Principal Analytics Prep's Part-Time Immersive Program for evening courses covering SQL databases, Python, Advanced Excel, Web Tech & Analytics, Analytics Strategy and Statistical Reasoning. See here for more information.

***

 

 

 

 

Tesla's slick stats

Tesla likes to advertise that its cars save lives. In the latest video, I examine this claim. The problem is that Tesla owners should not be compared to the average car buyer. Tesla is a luxury car and a status symbol. Tesla should be compared to other luxury cars. How does Tesla fare? Check out the video to find out.

***
One additional issue: what's the better metric for automotive fatality rate? Tesla uses deaths per million vehicle miles driven while others in industry use deaths per million vehicle-years.

This reminds me of the same issue in measuring air safety, a topic I addressed in my first book, Numbers Rule Your World (Chap 5) (link). Should airlines be measured based on deaths per million flight miles or deaths per million flights? MIT Professor Arnold Barnett looked into this issue and came out advocating for deaths per million flights. The main insight is that most air crashes happen at the start or end of a flight; the probability of a crash is not uniformly distributed over the miles of each journey. Thus, the death per million flights metric gets closer to the risk experience.

I'd imagine something similar at work here. The chance of getting killed in a car accident is not spread evenly throughout a trip. The additional risk is borne by taking an additional trip - once you're on a trip, the increase in risk over the distance of the trip isn't material.

***

To see the short video on Tesla, click here.

To see more of my videos, follow the channel.

If you are looking for evening classes to learn data science from industry experts in a high-touch, hands-on setting, check out the Fall Programs at Principal Analytics Prep.

 

 

 

Wage inequality and its sources

In the sister blog, I featured a data graphic showing the difference in pay levels between white male workers in the U.S. and women workers of various races. That post discusses purely visual matters, in effect accepting the analysis as given. In this post, I take a deeper look at the underlying data analysis.

Let us review the analysis strategy behind the chart, and then discuss why this simple strategy is not particularly insightful.

The starting point of the analyst is income data collected by the Census Bureau. Even here, income is measured in several ways. It appears that using personal income (as opposed to household or family income) is the most appropriate here. There is an additional complication of how to handle "missing values", which may arise in this context because someone is not employed and thus earn zero income. When one says "median income", does one include or exclude the zero earners? What about those who only worked part of the year?

After those questions are addressed, one can work with median incomes, computed at the right level of aggregation. The report linked to by Business Insider only contains aggregation by race, and aggregation by gender, each calculated separately. What is needed is a cross-tabulation. It is not possible to obtain the median income of Asian women from the median income of Asians and the median income of women - unless the analyst makes unwarranted assumptions.

So the analyst learns that white men made $46,000 in 2017 while Asian women made $27,000 and Hispanic women made $21,000. This data can be plotted directly, or after computing the race-gender discount (off the white male wage level).

***

The analyst wants to make this data come alive by using a different unit, the number of days worked.

One way to achieve this is by converting the annual salary to daily salary, then computing how many days the median Asian woman must work in order to earn $46,000, the median for the white man. This is roughly 611 days, which suggests that the Asian woman must work 246 extra days.

In the above description, I have seamlessly papered over several annoying details. I have assumed - without checking with reality - that everyone works 365 days a year. In fact, no one worked 365 days a year. And even if I obtained the correct value of the average number of days worked, call it X, it will also be the case that X will vary between race and gender. Thus, I made a further implicit assumption that such variance is not large enough to bother about.

I justify this lack of care because I rounded the median salaries to the nearest thousand. The questions I raised above ought to concern those analysts who insist on printing estimates in decimals. Is it possible to attain that level of precision while making simplifying assumptions?

***

Also notice that the analysis strategy is counterfactual in nature - it requires conjuring up a hypothetical scenario. It's a comparison of a white man who works exactly one year, and a woman (of any race) who works till she earns $46,000 or whatever is the current median wage for white men.

The notion of "extra days" is an invention since there are only 365 days in a year, and the women can never catch up unless the white guys stop working.

***

When comparing white men and Asian women, both gender and race are shifted. The Business Insider story leads readers to attribute the wage gap to primarily gender but unless we see the median salaries for Asian men, Hispanic men and black men, one can't be sure.

Most likely, there is a gender "effect" as well as a race "effect". The gender effect may even be differently sized for each race. This is known as an "interaction" effect.

***

Finally, there are even more factors to be considered. It is well known that at least some of the wage gap is explained by the difference in the mix of jobs and industries that men and women tend to be employed in. So one can't conclude discrimination without further investigation.

Unequal pay for equal work is discrimination but unequal pay for unequal work need not be.

***

Now, check out my comments about the calendar visualization of the wage gaps by Business Insider.

 

More Grubhub theatrics, with Yelp as supporting actor

Just recently, I made a short clip about how Grubhub is extracting lead-generation fees from unsuspecting restaurant owners by setting up a network of shadow websites (and phone numbers). Diners who thought they were ordering directly from restaurants were instead shuffled through the Grubhub toll booth. Click here to see how this works.

 

 

Now, Vice discovered yet another of Grubhub's toll booths. This time, the toll booth is set up on the Yelp superhighway. When a Yelp user clicks on the restaurant's phone number on its Yelp page, a pop-up shows up, with two options, one of which redirects the user to a shadow number owned by Grubhub. These numbers are not labelled "direct" and "Grubhub" but "general questions" and "deliveries and orders". So now both Yelp and Grubhub are making lead-generation money off the restaurant owner.

***

This dispute is about causality! And because causality is tough to establish, it creates a gray zone of disagreement.

Ideally, the restaurant owner pays for orders caused by Grubhub's marketing activities. By cause, we mean the orders would not have materialized without Grubhub's marketing. In the examples we saw, it's highly unlikely that Grubhub did anything to cause those orders. That's because those diners thought they were ordering directly from the restaurants - they are quietly re-routed to the shadow websites and phone numbers set up by Grubhub, sometimes without even the restaurants' knowledge.

It is the secrecy that gives the game away. If Grubhub's causal value is clear, it can form open partnerships with the restaurants without resorting to trickery.

As I pointed out in the video, the majority of the digital marketing industry relies on similar tactics. Search engines do not typically send you directly to the webpage you clicked on - you are often rerouted through the search engine's server, so that the search engine can "track" the click and use the paper trail to receive lead-generation money. Anyone passing through this toll booth is counted as "causal" but in reality, many of these users would have found their way to the webpage even if the search engine didn't exist. (Consider going directly to Macys.com instead of typing Macys into a search engine.)

***

This is a great example of how data, algorithms and software are silently running our lives, and often to the detriment of those who don't understand what's going on. Our video series is a small effort to help you stay in front of these data-driven technologies.

The more you know, the more you can leverage its power, and avoid its harms.

Grubhub's lead generation tactics leave bitter taste with restaurant owners

In my new 3-minute video, I discuss the controversy about the revelation that Grubhub has been playing a dirty trick on its restaurant customers.

As I explain in the video, what Grubhub did is similar to what Google has been doing with its search engine.

Most of these online businesses make money off "lead generation" - they bring prospective customers to brands. Restaurant owners hope Grubhub will bring them additional diners. These have to be incremental diners to justify the expense, which is a sizable 20-percent fee Grubhub assesses for each lead.

Grubhub has every incentive to count each order as an incremental lead. The dirty trick involves counting orders who should not be charged the lead-generation fee. Grubhub sets up shadow restaurant websites with similar URLs that pretend to be official websites. The diner usually does not know s/he is ordering from Grubhub's website and not the official restaurant website. This bit of deception qualifies the order for the lead generation fee - if the same diner had ordered from the restaurant's own website, Grubhub would have collected a much lower fee.

Similarly, lots of us search for brand names on Google, instead of going directly to the brands' websites. This leads some brand managers to buy their own brand names from Google. Google over the years has made it harder and harder to see which result is a paid ad. If we click on those ads, the brands pay Google a lead generation fee. Nevertheless, if we are searching for a brand's name, we will be visiting that brand's website, whether or not we pass through the Google toll booth!

In Numbersense (link), I talk about the importance of counterfactual thinking, and measuring incremental rather than absolute metrics.

Click here to see the video about the Grubhub controversy.

 

Know your data 25: Roundup of the fast-and-furious news

The genie is out of the bottle. The press is now reporting, almost daily, new discoveries of all the unspoken things tech companies have been doing with personal data collected from our usage of their products and services.

First up, the reporter from Washington Post wondered how much surveillance is happening on the Chrome browser. He learned that the browser loaded 11 thousand trackers in just one week. Horrified, he dumped Chrome (a Google product) for Firefox.

Traditionally, the website you visit installs a cookie (a type of tracker) to remember you: when you visit a page, and it automatically remembers who you are, that's the classic use of a cookie. Almost all of the trackers nowadays are invited guests of the website you visit, third parties who just want to follow you around.

I have more to say about the surveillance industry in this video. It's not just Chrome that you have to worry about.

***

When multiple websites like Facebook and Instagram temporarilly went down last week, news came out that Facebook automatically tags any photo you upload with "labels". During the downtime, the images did not load but the labels did. People saw things like "Photo may contain: two people, smiling, a beach". These are almost surely AI-created tags.

This is nothing new actually. In the past, you can see these labels if you are on a slow internet connection.

The real concern is the disclosure that Facebook automatically tags photos with people's names, using facial recognition technology. It appears that telling Facebook you don't want to be tagged means the tagging is moved underground, behind the scenes, out of your reach. I've been calling this the "illusion of control."

***

Another instance of illusion of control came by way of Amazon, which admitted that audio recordings of Alexa users are retained even after these users delete their accounts and data.

***

Next, a new email tool called Superhuman lets users to track whether the emails they sent have been opened by their recipients. This feature was on by default. This type of tracking uses "pixels" rather than cookies. A pixel is a tiny transparent "image" that is designed to hide from human eyes. When your browser pulls the pixel from its owner's server, you have made yourself known to that server.

This tracking feature has always been part of commercial email software. Whenever Amazon or your favorite brand sends you an email, analysts can tell when you open it, or interact with it. This data allow brands to assess the effectiveness of their emails.

Remember this the next time you pretend to your friend or colleague that you haven't seen his/her email.

 

Some things of interest

My review of Jim Albert's book Visualizing Baseball is on the sister blog. As I mentioned in the review, I thought a more appropriate title should be "Fast Intro to Baseball Analytics". Thus, the book may be of interest to some of the readers here.

***

Over the past year, I've been experimenting with the video medium. Here are some clips on various topics:

Analyzing observational data, containing a good example of using critical thinking

Alarming college admissions statistic, a nice example of data driving behavior in a feedback loop

Did you betray your friend today?, which explores how data flow through businesses

Two-part exploration of route optimizing apps like Waze: why they might not give you the fastest route, and why these apps are still great

To keep track of future releases, subscribe to our channel.