Big Data, Plainly Spoken (aka Numbers Rule Your World)

The research on SAT bias is great data science

I just put up a follow-up video on the SAT controversy, including a speculation about what might happen if the SAT were to be toppled from college admission. Link to video.

***

One of the key arguments against the SAT is its alleged bias against certain subgroups of test-takers. There has been a lot of research into ways to measure and combat such biases. Chapter 5 of Numbers Rule Your World (link) gets into the science behind this.

In the video, I discuss the following example featured also in the book

Is this test item biased against some subgroup? If so, which subgroup, by how much, and for what reason? It turns out that it is basically impossible to answer these questions purely by intuition. Test designers run experiments to measure the bias, and are often surprised by what they find.

This whole area of research is a great example of a practical application of data science. The issues range from how to collect the right data, and how to analyze such data to make causal conclusions. You can read about it in my book (link).

Or try the video first.

 

Bowled over by T-mobile

It's rare these days to call a customer-service line and end up feeling ... delighted, impressed, surprised, amazed.

Here's my story with T-mobile (US), and why you should consider them if you are thinking about a mobile operator right now. [Disclosure: no one paid or contacted me to do this post.]

The beginning of the call set the perfect tone. A computer voice chimed in and said: "If you'd like to use our automated system, please press 1 now."

Wow, wow, wow. This is the first company I dealt with (in the last 5-10 years) that does not presume that I want to speak to their chatbot. Their default assumption is I want to speak with a human being, but also it's a quick press of "1" to reach the chatbot. Just brilliant.

Contrast this with every other call - you are desperately pressing "0" or some random numbers, or yelling "operator", "representative", "human", hoping that somehow the system will let you switch to an operator.

I was then put on hold. The wait was probably about 5 minutes. During this time, the computer voice reminded me of the option to switch to the chatbot, with helpful commentary, such as "if you want to do something simple, such as checking your balance, you can press 1 to switch to the automated system."

The problem I had was most likely a computer glitch. In one week, the system claimed I have used up my entire month's of broadband data. That is highly unusual. Either there was a bug in measurement, or somehow I visited a "bad" website that loaded a lot of junk I was not aware of.

Here is where I encountered the second grace. The service rep actually was given the power to think and solve problems! No mere reading from the script. After we spent maybe a few minutes going back and forth, she said let's solve the problem, there is no point in explaining where it went wrong (we didn't have the relevant data anyway), and she offered to reset the usage, which is exactly the solution I had in mind.

Finally, she hit the home run. She explained that the reset would take a couple of hours, and she would call me back to confirm that it's done. We agreed on a time. She called me, and confirmed, which took less than a minute.

So, T-Mobile, job well done! Please keep it up. I'm a fan.

Buzzfeed notices the fake data menace

I first wrote about the fake data menace here (that was 2017) and again (2018). Earlier this year, I discussed an article in the Hustle about fake Amazon reviews, and made a video about it. 

This holiday season, Buzzfeed finally noticed. Here's a nice article detailing the scams that abuse online review platforms. Amazon (and other online retailers) prefer to look sideways for obvious reasons.

The reviewer featured in the Buzzfeed article has spent over $15,000 on "verified purchases" on Amazon - all of which were stuff that she bought in order to earn commissions after posting 5-star reviews. She wrote hundreds of reviews, not a single one true. She actually doesn't think she did anything wrong: "I'm just a pawn in their marketing scheme."

It's unfair to say Amazon and other retailers have no protections against fake reviews. The typical response to this problem is to set up community rules, and look sideways. If a case is raised to their attention, they will enforce the rule - but only on that case. For example, Amazon bans sellers from giving away free products in exchange for reviews but the ecommerce giant is not proactively seeking out offenders. (For example, there are many Facebook groups in which sellers and fake reviewers find each other. The article even mentions a rebate website where people can buy stuff from Amazon, wait for the return period to expire, and then get a full "rebate".) With a platform which is so vast, this mechanism allows the vast majority of fake reviews to persist.

Now that the reviewer has seen the sausage factory, her attitude towards the data is different. She confessed that she does not look at reviews when buying for herself: "When I see an off-brand product that’s Amazon’s Choice, [that label] doesn’t mean anything to me anymore."

There is zero financial incentive to fight fake reviews, for the simple reason that they work. These sham reviews are amplified by data science products - such as recommendation engines, which consider reviews as "data", the heralded unstructured "big data" - which push products that have large number of positive reviews (dubbed "relevant") to the top of search results, which inevitably are displayed in order of "relevance" by default. The top search results get clicked on the most, which creates a positive feedback loop keeping those products at the top. This is the "marketing scheme" mentioned above.

***

It's hard to come up with a solution to this fake data menace. For ecommerce, the harm done by the fake reviews is that other shoppers are duped into buying things that don't need, or things that are not what they appear to be. Also, honest merchants and brands lose market share. So you have this "little white lies" justification. Now, if these reviews are for medical services, then someone could get truly harmed by poor quality of care.

On the other side, lots of people benefit from the scams: the online retailers and the sellers all earn more revenues; the people who write fake reviews get free products and side gigs; friends of the fake-review writers get free handouts; suppliers of the sellers and retailers get more business; etc. Even data scientists arguably benefit as they have "more data" to work with.

Comment below if you have solutions!

 

The movement to topple the SAT is a fight over who can be subjective

(Photo credit: melystu, Flickr)

Overheard on the news: the University of California system may drop the SAT requirement for college admissions, which could be a body blow to the testing industry because UC is gigantic and the crown jewel of U.S. public universities. (There is a LA Times article about this news although it's behind a paywall.)

I'll discuss both sides of this argument.

To the supporters of the SAT, test scores are "objective" measures of ability. That perked up my ears because I believe there is no such thing as objective statistics. What they mean when they say "objective" is standardized: test scores provide a standard of comparison, a systematic way of comparing applications. As someone who have evaluated applications, I heartily appreciate the value of having such standards.

You might think the transcripts that list what courses were taken, and what grades were achieved would constitute objective data. Alas! Two classes with the same titles "Introduction to Algebra" can be completely different. And if we are talking humanities, classes may have unique titles (and contents). The same class with the same title taught by different instructors can also be different. To properly interpret transcripts would require a lot of contextual information not found on the applications.

Is high school GPA better than standardized tests? Because of grade compression, it has become practically impossible to use grades for evaluation. The issue here is less subjectivity but lack of variance: how can you differentiate between one A and another A?

The best argument supporting standardized testing is that it offers a benchmark for comparisons - over time, and between students.

***

Now turning to the opponents of the SAT. They like to argue that the SAT is not objective enough. They point to studies that show that test items may be biased against certain subgroups, such as women and African Americans. (Chapter 3 of my book Numbers Rule Your World (link) covers the fascinating world of how the ETS, the group that runs the SAT, uses statistical techniques to reduce subgroup bias in standardized testing.)

One problem with the opposition is that they have nothing to offer as a substitute. What are they suggesting as the replacement for the SAT?

Perhaps the substitutes are teacher recommendations, high school GPAs, etc. The Achilles heel is that those instruments are no more objective, and in my mind, clearly more subjective than the SAT. So there is an inconsistency in this line of reasoning: they complain that the SAT is not objective enough but they don't have an alternative that is more objective.

(On a related note, I'm eagerly waiting for the climate skeptics to deliver their own alternative set of models that issue predictions about the future, which can be verified by other scientists.)

For readers who know the materials in Chapter 3 of Numbers Rule Your World (link), you learn how hard it is to remove all subgroup bias from test items. I showed some examples of test items that look fine but were found to be wildly biased against certain groups when measured in an experiment. If the primary concern is about subgroup bias, one should also be worried about such bias emanating from human reviews of transcripts, teacher recommendations, application essays, etc.

***

If and when the SAT gets knocked out of the college admissions process, it will just clear the way for another testing company to rebuild this industry.

The data science profession will likely be leading this charge. Many data scientists are already promising predictive models that find best applicants, best customers, best employees, etc. These models are data-hungry. They require huge mounds of data, structured data, preferably numeric. Subjective data like teacher recommendations do not lend themselves to such models. But standardized tests, especially multiple-choice questions, effortlessly generate huge mounds of such data. So, what do you think is hot in this startup arena?

Many data science startups administer all kinds of tests, the goal of which is to generate training data for their predictive algorithms. That's why they may kill the SAT but they won't kill standardized testing.