Big Data, Plainly Spoken (aka Numbers Rule Your World)

People ask me what coding language should they learn to become a data scientist

Lots of people who want to become a data scientists are obsessed with finding out the one coding language they should learn to ensure their successful transition. And they are all disappointed - visibly so - when I offer them the counter-intuitive answer - that what differentiates data scientists is not what language they use, but whether they can think critically about the data, and outputs from data analyses.

I wrote a blog post about this earlier in the year, and published it with KDNuggets, one of the leading data science newsletters. They have just told me I got a "Gold Badge" for one of the top blog posts of 2019.

So let me repeat this unpopular piece of advice. (They do say the best medicine is the bitterest medicine.) If you want to get into data science, and you're spending all of your energy trying to learn detailed syntax of a new coding language, stop!

Think about what you're doing, why you're doing it, where's the evidence other than hearsay that something works, how do you know that it works, what alternatives are there, and what are the trade-offs.

***

At Principal Analytics Prep, we hire instructors with at least 10 years of industry experience who can bring practical tips to the classroom. Critical thinking accrues with practical experience, and it's easiest to learn from people who've picked it up the hard way. As I mentioned in the KDnuggets blog, math/engineering textbooks are the worst at training critical thinking - the fact that there are answer keys at the end of the books is damning.

 

The research on SAT bias is great data science

I just put up a follow-up video on the SAT controversy, including a speculation about what might happen if the SAT were to be toppled from college admission. Link to video.

***

One of the key arguments against the SAT is its alleged bias against certain subgroups of test-takers. There has been a lot of research into ways to measure and combat such biases. Chapter 5 of Numbers Rule Your World (link) gets into the science behind this.

In the video, I discuss the following example featured also in the book

Is this test item biased against some subgroup? If so, which subgroup, by how much, and for what reason? It turns out that it is basically impossible to answer these questions purely by intuition. Test designers run experiments to measure the bias, and are often surprised by what they find.

This whole area of research is a great example of a practical application of data science. The issues range from how to collect the right data, and how to analyze such data to make causal conclusions. You can read about it in my book (link).

Or try the video first.

 

Bowled over by T-mobile

It's rare these days to call a customer-service line and end up feeling ... delighted, impressed, surprised, amazed.

Here's my story with T-mobile (US), and why you should consider them if you are thinking about a mobile operator right now. [Disclosure: no one paid or contacted me to do this post.]

The beginning of the call set the perfect tone. A computer voice chimed in and said: "If you'd like to use our automated system, please press 1 now."

Wow, wow, wow. This is the first company I dealt with (in the last 5-10 years) that does not presume that I want to speak to their chatbot. Their default assumption is I want to speak with a human being, but also it's a quick press of "1" to reach the chatbot. Just brilliant.

Contrast this with every other call - you are desperately pressing "0" or some random numbers, or yelling "operator", "representative", "human", hoping that somehow the system will let you switch to an operator.

I was then put on hold. The wait was probably about 5 minutes. During this time, the computer voice reminded me of the option to switch to the chatbot, with helpful commentary, such as "if you want to do something simple, such as checking your balance, you can press 1 to switch to the automated system."

The problem I had was most likely a computer glitch. In one week, the system claimed I have used up my entire month's of broadband data. That is highly unusual. Either there was a bug in measurement, or somehow I visited a "bad" website that loaded a lot of junk I was not aware of.

Here is where I encountered the second grace. The service rep actually was given the power to think and solve problems! No mere reading from the script. After we spent maybe a few minutes going back and forth, she said let's solve the problem, there is no point in explaining where it went wrong (we didn't have the relevant data anyway), and she offered to reset the usage, which is exactly the solution I had in mind.

Finally, she hit the home run. She explained that the reset would take a couple of hours, and she would call me back to confirm that it's done. We agreed on a time. She called me, and confirmed, which took less than a minute.

So, T-Mobile, job well done! Please keep it up. I'm a fan.

Buzzfeed notices the fake data menace

I first wrote about the fake data menace here (that was 2017) and again (2018). Earlier this year, I discussed an article in the Hustle about fake Amazon reviews, and made a video about it. 

This holiday season, Buzzfeed finally noticed. Here's a nice article detailing the scams that abuse online review platforms. Amazon (and other online retailers) prefer to look sideways for obvious reasons.

The reviewer featured in the Buzzfeed article has spent over $15,000 on "verified purchases" on Amazon - all of which were stuff that she bought in order to earn commissions after posting 5-star reviews. She wrote hundreds of reviews, not a single one true. She actually doesn't think she did anything wrong: "I'm just a pawn in their marketing scheme."

It's unfair to say Amazon and other retailers have no protections against fake reviews. The typical response to this problem is to set up community rules, and look sideways. If a case is raised to their attention, they will enforce the rule - but only on that case. For example, Amazon bans sellers from giving away free products in exchange for reviews but the ecommerce giant is not proactively seeking out offenders. (For example, there are many Facebook groups in which sellers and fake reviewers find each other. The article even mentions a rebate website where people can buy stuff from Amazon, wait for the return period to expire, and then get a full "rebate".) With a platform which is so vast, this mechanism allows the vast majority of fake reviews to persist.

Now that the reviewer has seen the sausage factory, her attitude towards the data is different. She confessed that she does not look at reviews when buying for herself: "When I see an off-brand product that’s Amazon’s Choice, [that label] doesn’t mean anything to me anymore."

There is zero financial incentive to fight fake reviews, for the simple reason that they work. These sham reviews are amplified by data science products - such as recommendation engines, which consider reviews as "data", the heralded unstructured "big data" - which push products that have large number of positive reviews (dubbed "relevant") to the top of search results, which inevitably are displayed in order of "relevance" by default. The top search results get clicked on the most, which creates a positive feedback loop keeping those products at the top. This is the "marketing scheme" mentioned above.

***

It's hard to come up with a solution to this fake data menace. For ecommerce, the harm done by the fake reviews is that other shoppers are duped into buying things that don't need, or things that are not what they appear to be. Also, honest merchants and brands lose market share. So you have this "little white lies" justification. Now, if these reviews are for medical services, then someone could get truly harmed by poor quality of care.

On the other side, lots of people benefit from the scams: the online retailers and the sellers all earn more revenues; the people who write fake reviews get free products and side gigs; friends of the fake-review writers get free handouts; suppliers of the sellers and retailers get more business; etc. Even data scientists arguably benefit as they have "more data" to work with.

Comment below if you have solutions!