Big Data, Plainly Spoken (aka Numbers Rule Your World)

Buzzfeed notices the fake data menace

I first wrote about the fake data menace here (that was 2017) and again (2018). Earlier this year, I discussed an article in the Hustle about fake Amazon reviews, and made a video about it. 

This holiday season, Buzzfeed finally noticed. Here's a nice article detailing the scams that abuse online review platforms. Amazon (and other online retailers) prefer to look sideways for obvious reasons.

The reviewer featured in the Buzzfeed article has spent over $15,000 on "verified purchases" on Amazon - all of which were stuff that she bought in order to earn commissions after posting 5-star reviews. She wrote hundreds of reviews, not a single one true. She actually doesn't think she did anything wrong: "I'm just a pawn in their marketing scheme."

It's unfair to say Amazon and other retailers have no protections against fake reviews. The typical response to this problem is to set up community rules, and look sideways. If a case is raised to their attention, they will enforce the rule - but only on that case. For example, Amazon bans sellers from giving away free products in exchange for reviews but the ecommerce giant is not proactively seeking out offenders. (For example, there are many Facebook groups in which sellers and fake reviewers find each other. The article even mentions a rebate website where people can buy stuff from Amazon, wait for the return period to expire, and then get a full "rebate".) With a platform which is so vast, this mechanism allows the vast majority of fake reviews to persist.

Now that the reviewer has seen the sausage factory, her attitude towards the data is different. She confessed that she does not look at reviews when buying for herself: "When I see an off-brand product that’s Amazon’s Choice, [that label] doesn’t mean anything to me anymore."

There is zero financial incentive to fight fake reviews, for the simple reason that they work. These sham reviews are amplified by data science products - such as recommendation engines, which consider reviews as "data", the heralded unstructured "big data" - which push products that have large number of positive reviews (dubbed "relevant") to the top of search results, which inevitably are displayed in order of "relevance" by default. The top search results get clicked on the most, which creates a positive feedback loop keeping those products at the top. This is the "marketing scheme" mentioned above.

***

It's hard to come up with a solution to this fake data menace. For ecommerce, the harm done by the fake reviews is that other shoppers are duped into buying things that don't need, or things that are not what they appear to be. Also, honest merchants and brands lose market share. So you have this "little white lies" justification. Now, if these reviews are for medical services, then someone could get truly harmed by poor quality of care.

On the other side, lots of people benefit from the scams: the online retailers and the sellers all earn more revenues; the people who write fake reviews get free products and side gigs; friends of the fake-review writers get free handouts; suppliers of the sellers and retailers get more business; etc. Even data scientists arguably benefit as they have "more data" to work with.

Comment below if you have solutions!

 

The movement to topple the SAT is a fight over who can be subjective

(Photo credit: melystu, Flickr)

Overheard on the news: the University of California system may drop the SAT requirement for college admissions, which could be a body blow to the testing industry because UC is gigantic and the crown jewel of U.S. public universities. (There is a LA Times article about this news although it's behind a paywall.)

I'll discuss both sides of this argument.

To the supporters of the SAT, test scores are "objective" measures of ability. That perked up my ears because I believe there is no such thing as objective statistics. What they mean when they say "objective" is standardized: test scores provide a standard of comparison, a systematic way of comparing applications. As someone who have evaluated applications, I heartily appreciate the value of having such standards.

You might think the transcripts that list what courses were taken, and what grades were achieved would constitute objective data. Alas! Two classes with the same titles "Introduction to Algebra" can be completely different. And if we are talking humanities, classes may have unique titles (and contents). The same class with the same title taught by different instructors can also be different. To properly interpret transcripts would require a lot of contextual information not found on the applications.

Is high school GPA better than standardized tests? Because of grade compression, it has become practically impossible to use grades for evaluation. The issue here is less subjectivity but lack of variance: how can you differentiate between one A and another A?

The best argument supporting standardized testing is that it offers a benchmark for comparisons - over time, and between students.

***

Now turning to the opponents of the SAT. They like to argue that the SAT is not objective enough. They point to studies that show that test items may be biased against certain subgroups, such as women and African Americans. (Chapter 3 of my book Numbers Rule Your World (link) covers the fascinating world of how the ETS, the group that runs the SAT, uses statistical techniques to reduce subgroup bias in standardized testing.)

One problem with the opposition is that they have nothing to offer as a substitute. What are they suggesting as the replacement for the SAT?

Perhaps the substitutes are teacher recommendations, high school GPAs, etc. The Achilles heel is that those instruments are no more objective, and in my mind, clearly more subjective than the SAT. So there is an inconsistency in this line of reasoning: they complain that the SAT is not objective enough but they don't have an alternative that is more objective.

(On a related note, I'm eagerly waiting for the climate skeptics to deliver their own alternative set of models that issue predictions about the future, which can be verified by other scientists.)

For readers who know the materials in Chapter 3 of Numbers Rule Your World (link), you learn how hard it is to remove all subgroup bias from test items. I showed some examples of test items that look fine but were found to be wildly biased against certain groups when measured in an experiment. If the primary concern is about subgroup bias, one should also be worried about such bias emanating from human reviews of transcripts, teacher recommendations, application essays, etc.

***

If and when the SAT gets knocked out of the college admissions process, it will just clear the way for another testing company to rebuild this industry.

The data science profession will likely be leading this charge. Many data scientists are already promising predictive models that find best applicants, best customers, best employees, etc. These models are data-hungry. They require huge mounds of data, structured data, preferably numeric. Subjective data like teacher recommendations do not lend themselves to such models. But standardized tests, especially multiple-choice questions, effortlessly generate huge mounds of such data. So, what do you think is hot in this startup arena?

Many data science startups administer all kinds of tests, the goal of which is to generate training data for their predictive algorithms. That's why they may kill the SAT but they won't kill standardized testing.

The Harden-Miami link: thinking about indirect metrics in an analysis plan

A co-worker pointed me to a Reddit thread that someone put up to showcase a small data-science side project - to establish whether NBA star James Harden's performance is affected by his "affinity for ... strip clubs" (thereafter, SC).

The analytical plan used in this project is typical of many data science projects. In this post (and possibly a few more), I raise issues that analysts should ponder when conceptualizing this type of analysis. These comments are designed to help those who are interested in advancing the sophistication of their analytical work. I am not re-analyzing the data, or debunking what's been done. I am impressed by what "AngryCentrist" has put together - starting with how AC gathered the data to how AC executed an analytical plan resulting in some conclusions.

***

Let me summarize the analytical game plan as follows: (you can read the gory details in this google doc)

1. The outcome variable is John Harden's on-court performance, expressed in an inverse scale (i.e. the larger the metric, the worse his performance). The explanatory variable is the quality rating of SCs in the cities where Harden played away games. The analysis looks at the correlation between these two variables.
2. Only the last four seasons of away games in which Harden played are used. In each season, there are usually one or two away games against any given team. Team and city are synonymous except for Los Angeles which fields two teams. (Brooklyn and New York are treated as separate cities.)
3. On-court performance is measured as the number of metrics on which Harden's in-city performance is worse than his season average. Only six metrics (e.g turnovers) are considered over four seasons so this performance metric is a number between 0 and 24. Zero means Harden out-performed his season average for each of the 6 metrics over each of the 4 seasons. By contrast, if Harden did worse than his season average for every metric over those four seasons, then the performance metric attains the maximum value of 24. The redditor, AngryCentrist, called this the "number of sub-par performances".
4. SC ratings are based on the first 10 results with customer ratings when doing a search on Google of the type "[city] strip clubs".
5. The analyst concluded that the two variables were positively correlated based on fitting a linear regression model. Higher SC quality rating is associated with worse on-court performance. The r-squared was 20 percent (which is not impressive).

***

For this post, I focus on the explanatory variable, the SC average rating, and the concept of "noise" in our data.

The SC average rating is an example of an indirect metric, a "noisy" proxy for what we want to measure. Notice that the outcome variable is about a particular person (Harden) while the explanatory variable measures a large set of people, of which Harden is allegedly a member.

The "noise" comes from several sources. If Harden was not a member of the set, then the explanatory variable has no direct connection with the outcome. So we hope that Harden not only visited SCs in all those cities, but also reviewed his experiences on Google, thus contributing to the average ratings.

If Harden did enjoy himself at the SCs and submitted reviews to Google, his ratings would still have been immaterial, just one vote in the average. Since the analysis only extracted "top 10" SCs based on a Google search, one expects the selected SCs are the more popular ones, which means that Harden's ratings would be even more insignificant.

If we don't require Harden's ratings to be included in these SC averages, we would at least require that his ratings - if revealed - to be highly correlated with the average ratings among Google reviewers. This assumption has two components - Harden would have to patronize the popular SCs, rather than ones that are more exclusive, or discreet; and Harden's opinion of his experiences there would have to be similar to that of the average patron.

Let's pretend that Harden visited one club prior to playing an away game in Miami. The average rating of the top 10 clubs in Miami is then not quite up to the task. That specific club might not even be part of the ten. This gap is also part of the "noise" of indirect measurement.

Timing is yet another source of noise. What if Harden engaged in the adult activity after the game rather than before the game?

Timing enters the picture in another way. Adult entertainment venues come and go, just like any other businesses. If Harden had visited a club before a match early in the NBA season, and that club subsequently shut down, then this club would not feature in the Google search, and would be absent from the average rating. Besides, new clubs that opened after Harden's visit would enter the average rating, if they become popular enough. Either of these situations causes the indirect metric to stray away from its proper value.

***

Here's the big picture. The indirect measure of average quality rating of SCs is a good proxy only under a large set of presumptions. None of the above conditions is directly observable, so each is presumed.

In general, the further away the measurement is from what is supposed to be measured, the more "noisy" is the measurement.

A workaround is to change our theory of the link between SC rating and Harden's performance. Maybe the link does not require physical visits to these establishments and subsequent reviews. Perhaps it is Harden's knowledge of a vibrant SC scene, regardless of active participation, that is doing the trick. This link though cannot be proven or disproven by the data, and thus is also presumed.

Assumptions are not avoidable. Verifiable assumptions are better than unverifiable ones. A good analysis plan should make the fewest, and weakest assumptions.

 

 

 

Myth: standardizing variables makes them normal

There are lots of myths in data science which are repeated endlessly on social media and the internet. One popular myth says that standardizing variables is to make them normal. This is not true!

This myth is found explicitly in the Python documentation on the StandardScaler method in the popular Scikitlearn package (link):

Standardization of a dataset is a common requirement for many machine learning estimators: they might behave badly if the individual features do not more or less look like standard normally distributed data (e.g. Gaussian with 0 mean and unit variance).

Let me first show you a quick counter-example, then provide some intuition around the standardization procedure. (Warning: this is a rather long read that is more technical than usual.)

***

(a) I generate 10,000 random numbers between -5 and 7 from a uniform distribution. A uniform distribution means that each value between -5 and 7 has equal chance of appearing. This is clearly NOT a normal distribution (also known as the Bell curve), in which values cluster around the average - in a uniform distribution, the average value appears just as frequently as any other value in the range.

The following histogram shows the count of values for our 10,000 uniform random numbers, and as expected, we see an almost equal representation of every value in the range.

(b) The average of the 10,000 numbers is 1.0 (rounded to 1 decimal place). This falls unsurprisingly on the midpoint of the range -5 to 7. The standard deviation, which measures the spread of the data, is 3.5 (rounded to 1 decimal place).

(c) Now, I standardize the data, which means subtracting the average value and dividing by the standard deviation. This is what the histogram looks like after standardizing the data. You can immediately see that the standardized data do not look anything like a Bell curve.

(d) The average of the 10,000 standardized numbers is 0, and the standard deviation is 1. So the standardization formula did change the values of the average and standard deviation but it did not change the shape of the distribution. To see this more clearly, I made the horizontal axis range the same on both charts.

You can see that the middle of the distribution is shifted from 1 to 0. Also, the standardized numbers are squeezed into a smaller range than the original numbers. This reflects dividing each number by the standard deviation (which is dividing by 3.5).

***

Now, let's develop some intuition about this business of standardization, normal distributions, normal probability plots, and so on.

Here is a generic normal distribution, or a Bell curve:

One beautiful thing about normal distributions is that they are completely determined by its average value and its standard deviation. The average pins the midpoint of the distribution, where the familiar bulge of the distribution sits. The standard deviation controls how widely the data are spread out along the range of values shown on the horizontal axis.

The above normal distribution has average 5.1 and standard deviation 3.0.

We can standardize this distribution, which means subtracting the average and dividing by the standard deviation. As we learned earlier, the effect is to recenter the midpoint to zero, and to fix the standard deviation to 1. Here are the histograms of the original normal variables, and the standardized normal variables.

In this case, the standardized variables do look like a Bell curve. Well, that is because the original data have the shape of a Bell curve to begin with.

***

Let's introduce a tool that tests whether a given set of data has the shape of the Bell curve. This is called the normal probability plot. (Technically, it's a type of qqplot in which the reference "q" is the normal distribution.)

This plot exhibits a straight line if the test distribution is a normal distribution. Deviation from the normal distribution is revealed by bending on the left and right edges of the sequence of dots. In the following chart, I test the set of random uniform numbers from before. You can see the bending which tells us that those numbers are not shaped like a normal distribution.

The next normal probability plot asks whether the standardized uniform numbers we generated above is a normal distribution, and it is a resounding no! You can see the bending on both ends of the curve.

Can we salvage this situation by making more data? No! Here is what the plot looks like when I generate 1 million numbers instead of 10,000. The bending does not go away.

As before, the average value of the 1 million standardized numbers is 0 and the standard deviation is 1 but the shape after standardization is not normal. Standardizing variables does not make them normal.

***

The standard normal distribution has mean 0 and standard deviation 1. A normal distribution is completely determined by those two statistics. However, the reverse is not generally true! If a distribution has mean 0 and standard deviation 1, it does not mean we have a standard normal distribution, or any normal distribution.

The following uniform distribution in the range [-sqrt(3), sqrt(3)] has mean 0 and standard deviation 1, by construction. Here is what the histogram looks like:

This does not look anything like "standard normally distributed data", even though it has mean 0 and sd 1. Every value, including the average value, has similar representation.

(Standardizing this distribution does nothing at all since we'd be subtracting the mean of zero, and dividing by 1.)

***

Why then do we standardize data? First, note that the average of the standardized data is fixed to zero. Next, the standard deviation is fixed to 1 - standard deviation is the average of the squared distances of the data from the average: thus by fixing sd to 1 (relative to the average of zero), the dispersion of the standardized data is being constrained. If a particular value strays far away from zero, it causes the standard deviation to rise but the total is capped under 1, and so the range of possible values in the standardized data is restricted.

Standardization places different data sets on the same scale so that they can be compared systematically. It does not turn non-normal data into normal data.