Big Data, Plainly Spoken (aka Numbers Rule Your World)

Behind the proposed strengthening of FDA vaccine approval rules

News just broke that the FDA may announce more "rigorous" standards for a coronavirus vaccine to get approved. This indicates a willingness to listen to some of the critics of the perceived rushed job.

The specific rules mentioned in this Washington Post article are:

1) Participants must be followed for a median of at least two months, starting from when they receive the second dose

2) The analysis must be conducted after at least five severe cases of Covid19 have been observed in the placebo arm.

3) The analysis must be conducted after some minimum cases of Covid19 in older people has been recorded.

***

Rules 2 and 3 are fine in spirit. It's important that the vaccine works for severe cases and older people. These rules ensure that all trial analyses include the two groups that are of great concern to the medical community.

Recall that the case definition uses mild symptoms instead of severe symptoms. Since a lot more people develop mild symptoms, it's possible that the interim analysis contains only mild cases. It's also possible because of self-reporting that the cases used in the interim analysis involve only younger people.

I suspect those scenarios are unlikely anyway. Because testing is triggered by self-reporting, severe cases are more likely to surface than mild cases. So long as they have sufficient older participants in the trial, they should show up in the interim analysis because older people tend to have more severe illnesses.

Nevertheless, if the vaccine is approved by the end of 2020, it will be on the strength of an interim analysis. Such analyses are never designed to make statistically reliable statements about subgroups, such as older people or severe cases. This is another reason why the trial should run to completion even if the FDA grants interim approval.

***

Revised rule 1 is misleading and potentially worrying. 

Let's talk about its good intention first. Rule 1 sets up a second criterion for triggering the interim analysis. Previously, the only criterion is to exceed a prespecified total number of cases. This creates the potentially undesirable scenario in which enrollment is supercharged (either by pushing marketing, or relaxing admission standards), or the true case rate is markedly higher than assumed, such that the first interim analysis is triggered when most participants have barely taken the second dose.

By setting observation time as a second criterion, the new rule provides more confidence that the protection afforded by an approved vaccine lasts at least 2 months.

As I explained in this previous post, Moderna does not count cases until 2 weeks after the second dose. So if they track people for two months, it's really only 1.5 months of observation time. That isn't a lesser worry. What perturbs me is the use of the word "median".

Median is the middle person so the statement actually allows up to half of the participants to be observed for fewer than 2 months. Further, it is highly unusual and in fact statistically fallacious to perform analysis in which different participants are observed for different amounts of time. I hope this is a communications faux pas, a misunderstanding on the part of the journalist. The sensible rule is to require every participant (not the median participant) to be observed for at least 2 months (preferably longer) after the second dose.

 

 

P.S. [10/6/2020: A group of scientists and medical experts agree! STAT News writes about their pushback on rule #1. They want "a minimum of two months' observation for all participants".]

Buzzfeed notices the fake data menace

I first wrote about the fake data menace here (that was 2017) and again (2018). Earlier this year, I discussed an article in the Hustle about fake Amazon reviews, and made a video about it. 

This holiday season, Buzzfeed finally noticed. Here's a nice article detailing the scams that abuse online review platforms. Amazon (and other online retailers) prefer to look sideways for obvious reasons.

The reviewer featured in the Buzzfeed article has spent over $15,000 on "verified purchases" on Amazon - all of which were stuff that she bought in order to earn commissions after posting 5-star reviews. She wrote hundreds of reviews, not a single one true. She actually doesn't think she did anything wrong: "I'm just a pawn in their marketing scheme."

It's unfair to say Amazon and other retailers have no protections against fake reviews. The typical response to this problem is to set up community rules, and look sideways. If a case is raised to their attention, they will enforce the rule - but only on that case. For example, Amazon bans sellers from giving away free products in exchange for reviews but the ecommerce giant is not proactively seeking out offenders. (For example, there are many Facebook groups in which sellers and fake reviewers find each other. The article even mentions a rebate website where people can buy stuff from Amazon, wait for the return period to expire, and then get a full "rebate".) With a platform which is so vast, this mechanism allows the vast majority of fake reviews to persist.

Now that the reviewer has seen the sausage factory, her attitude towards the data is different. She confessed that she does not look at reviews when buying for herself: "When I see an off-brand product that’s Amazon’s Choice, [that label] doesn’t mean anything to me anymore."

There is zero financial incentive to fight fake reviews, for the simple reason that they work. These sham reviews are amplified by data science products - such as recommendation engines, which consider reviews as "data", the heralded unstructured "big data" - which push products that have large number of positive reviews (dubbed "relevant") to the top of search results, which inevitably are displayed in order of "relevance" by default. The top search results get clicked on the most, which creates a positive feedback loop keeping those products at the top. This is the "marketing scheme" mentioned above.

***

It's hard to come up with a solution to this fake data menace. For ecommerce, the harm done by the fake reviews is that other shoppers are duped into buying things that don't need, or things that are not what they appear to be. Also, honest merchants and brands lose market share. So you have this "little white lies" justification. Now, if these reviews are for medical services, then someone could get truly harmed by poor quality of care.

On the other side, lots of people benefit from the scams: the online retailers and the sellers all earn more revenues; the people who write fake reviews get free products and side gigs; friends of the fake-review writers get free handouts; suppliers of the sellers and retailers get more business; etc. Even data scientists arguably benefit as they have "more data" to work with.

Comment below if you have solutions!

 

TSA can now count its workers as paid

Last week, I pointed out that the U.S. payroll survey uses the following accounting rule: 

The Labor Department... determines the number of people on nonfarm payrolls by asking employers for a head count during the pay period including the 12th of the month. For many federal employees, that would be the pay period from Jan. 6 to Jan. 19.

Further, one only needs to have worked for one hour during that period to be counted as employed.

This weekend, we heard word that the government shutdown has no end in sight. 

On Sunday (Jan 13), we heard more news - that the TSA has decided to issue each TSA worker a one-time one-day pay plus $500 bonus. This means TSA can now report all workers as having been paid during the pay period including the 12th of January.

 

 

The importance of knowing how things are counted

Yesterday, on the sister blog, I posted a pair of charts that digs into how the unemployment rate is computed. This number has been in the news a lot because we supposedly have a super-tight labor market, the lowest unemployment rate since the peak of the tech boom in 2000! But the story is much more complex if one understands how unemployment levels are counted.

Another reason why counting unemployment is in the news is because of the government shutdown. And the Wall Street Journal has a nice (but rather convoluted) article about what that means for the unemployment number. This is where you can see how politics enter into the statistics - it's via rules that determine what gets counted and what doesn't.

***

As I pointed out in Numbersense (link), the Labor Department uses a specific definition of employment, and here is how the WSJ describes it:

The Labor Department... determines the number of people on nonfarm payrolls by asking employers for a head count during the pay period including the 12th of the month. For many federal employees, that would be the pay period from Jan. 6 to Jan. 19.

In other words, someone just needs to have worked at least one hour during that pay period to be considered "employed".

Based on that definition, if the government shutdown extends beyond Jan 19, then there should be a big negative impact on the unemployment rate.

***

But wait - politicians of neither party would want that to happen (in case they are the ones in power at the moment).

So what do we learn from the WSJ reporter? This:

even if the shutdown extends beyond the pay period, workers would be counted if legislation is enacted that requires agencies to pay employees for what they would have earned had there been no shutdown, a spokesman for the Labor Department’s Bureau of Labor Statistics said. Such practice would be consistent with how the department accounted for workers during previously lengthy shutdowns.

The reason given for this is precedence. But that actually means that the unemployment rate in that scenario would not accurately portray the state of employment in the U.S.

An economist cited in the article further explained that the following could also happen:

some federal employees initially furloughed have been called back to work and some are being paid with supplemental funds.

Since it only takes one hour of work during that specific pay period to be counted as employed, it's not hard to come up with ideas about how to turn these not-working people into employed people.

As a data analyst, there is no getting around learning all these nuances if you want to interpret data properly. It's the nature of our work.

When should you leave the baseball stadium when the game is just too damn long

My new article, joint work with Pravin and Sriram, has appeared in FiveThirtyEight this week. It's motivated by this chart, showing the lengths of baseball games:

Since 2000, the average Major League baseball game lasts over 3 hours; one out of 10 games takes over 3.5 hours. The added time is almost all non-action time. In fact, there was an article in the Wall Street Journal in which they logged a bunch of games and discovered that less than 10 percent of the game time can be classified as "action." For the uninitiated, the "non-action" time includes pitchers staring at the batter, batters taking practice swings, managers slow-walking to the mound to chit-chat with the pitcher, etc.

To their credit, the League office has started to pay attention to this problem although it seems like an agreement with the players' union is hard to come by. Some fans are taking matters into their own hands - why fight with all others to get out of the stadium when you can leave in the middle of the game, with one team "comfortably" ahead?

In our article, we model this decision as a stay-or-leave decision, and use Wald's Sequential Probability Ratio Test to scientifically determine the optimal time to leave. Here's the bottom line:

A back-of-the-envelope estimate pegs the total amount of time saved at 1,612 hours for the season. The price to pay was the 61 mistakes — dramatic comebacks that were heard on the radio from the parking lot by the fans who left early.

 

A Half-day in the Life of a Data Scientist

An organization wanted to understand its base of members so the first order of business was constructing a database of all people who can be considered members.  We decided to define membership broadly. Members included those who join the Facebook group, and those who subscribed to the newsletter.

 

The organization kept two separate lists which I would merge to create a master list. For simplicity, I’ll call them the FB list, and the Email list. In merging, the key is the key. Let me explain. The simplest key is an email address. If someone’s email address shows up on both lists, then I infer that those entries concern the same person, and combine them. My goal is to remove double counting of anyone who appears on both lists.

 

Sounds simple enough.

 

But never that simple, right? First, the Facebook group is the graveyard of data. Facebook provides zero statistics on group members and activities. Yes, the company that makes a business out of data does not hear the data-deprived group owners who have been pleading for years.

 

What is a data scientist to do? Scrape, that’s what. Members can find out who else is in the group by the scroll-wait-reset-scroll routine. You know that feeling. I know you do. You scroll to the bottom of a web page. Your browser gets the hint. It loads a few more items, while the slider floats away, usually to the wrong spot. You re-set the position, and scroll some more. After much scrolling, I scraped that page to compile the Facebook list. It’s got the name of the person, their Facebook username, and their location (when available).

 

Notice I didn’t say email address. So the FB list did not contain the all-important key. Another possible key is first and last names. Reviewing the email list, I realized that newsletter subscribers are not required to provide names so matching names to the FB list will yield few hits. The third candidate is not as accurate; I tried matching the Facebook username to the email username.

 

The client furnished an Excel file, which I’ve been calling the Email list. Upon opening the list, I turned the email address into all uppercase letters. I have matched enough text data to know that people are hardly in control of their fingers when they type text into web forms. “John”, “JOHN”, “joHN”, “JOhn”, and so on typically mean the same thing, regardless of case. (The occasional sadist offers “J0hn,” or “Jhon,” or “Jo hn.”)

 

Meanwhile, the client wondered if email addresses are really case-insensitive. I suggested asking Google. The search engine gave an ambiguous answer. The part after the @ sign is case-insensitive whereas the part before @ is case-sensitive, but then most email providers treat both parts as case-insensitive.

 

It’s rare when Google complicates your life. I fished out the UPPERCASE(email_address) formula, deleted it, broke up the email address into the user name and domain name parts, upper-cased the domain name, and reconnected the two parts, re-inserting the @ sign. The machine must follow these steps but a human being instinctively knows where to apply the cut. Some researchers believe the brain executes those steps at warp speed but I don’t buy it.  

 

Next, I dropped the domain names from the split-and-spliced email addresses to get ready to match to Facebook usernames. Sheesh, the client did not ask if Facebook usernames are case sensitive or not. (They aren’t.) I proceeded to merge the two lists.

 

I executed a “full outer join.” With this procedure, any username that appears in one or both of the lists will find its way to the output dataset. On this first attempt, nothing merged. Even though username “davidcolumbus,” say, lived on both lists, the computer did not combine the data; the two matches sat one on top of the other.

 

I took a deep breath, for I had reached a point where I must be honest with myself. This project was sure to bust the two hours I originally allotted. The merge could easily take another hour, maybe two, if no new issues emerged.

 

The matching rows did not combine because the computer only joins eponymous columns. Since the Facebook and email usernames are different entities, those columns carry different labels.

 

But syncing those labels solves one problem while creating another! Members who appear on only one list have only one of the usernames. Besides, Facebook usernames are unique while email usernames, when detached from their domains, are not. A better solution is to set up a third username column in both lists, whose purpose in life is to be the matching key.

 

What about the other columns? Did I want them combined or not? Take as an example first and last names which show up on both lists. If I standardized the labels of these columns, the computer would attempt to merge them. What if David Columbus appeared as Dave Columbus on the other list with matching usernames? Forcibly combining the name columns would cause one of these variations to be dropped. If I wanted to keep both spellings, I must retain all name columns, which happens should I assign distinct labels, which is exactly the opposite of what I did with the username columns.

 

If that isn’t confusing enough, I stumbled upon another issue. In the Email list, while most names appeared as “First <space> Last,” there were examples of “Last <space> First”, and “Last <comma> First”, and “First Initial <space> Last,” and so on.  As an analyst, your first thought is “What’s wrong with our designers? Why didn’t they create separate text boxes for first and last names?” Then, you accept that blame gets you nowhere; you still have to fix what’s broken.

 

A soft voice enters your head. You wish you hadn’t seen the problem. You hope it was just a bad dream. But you wake up.

 

In front of me I had two paths. I could follow path A, and that meant developing code to automatically detect the various anomalies and fixing them. This path would take hours. Which is the first name in “Scott Lewis”? How would a computer figure this out? What rule could apply generally?

 

And then, there was path B, better known as handcrafting. If I had 1,000 rows of data, and if it took two seconds to scan a name and determine the type of anomaly, I would have completed the exercise in 30 minutes or so.

 

I chose path B. It was ugly and unsexy but more of a sure thing.

 

I wish I could tell you I stopped looking. But I couldn’t help it. Some cultures embrace double surnames, like “De” something or “Von” something. My code was  parsing “Chris De Jong” as first name Chris, and last name Jong. I needed a more complex rule. Something like “If the name has three words, take the first as first name, and the last two as the surname.” This rule runs afoul of someone like “Mary Anne Rutherford.” At a crossroad again. I could teach a computer how to lump the middle name, or I could exercise my brain some more.

 

By this time, I was exhausted. If you have followed me to this point, you have my admiration. In the next installment, I shall finish the assignment.

The mystery of the official inflation rate

In case you are not subscribed to my dataviz feed, I put up a post yesterday that is highly relevant to readers here interested in statistical topics. The post discusses a graphic of a New York Times article that interprets the official inflation rate (known as the CPI). I devoted an entire chapter of Numbersense (link)to the question of why the official inflation rate diverges from our everyday experience.

In a larger context, inflation rate is an invented metric, invented to measure some quantity that has no objective reality. This is true of a lot of statistics. Revenues and profits are also invented concepts, for example, and only attain meaning through generally accepted accounting rules. Obesity, which is discussed in Chapter 2 of Numbersense (link) is another example of a quantity that has meaning only because of a convention of measuring.

The article in NYT brings up one of the points I raised in the book, which is that price increases are magnified in our imagination while price decreases are taken for granted.

The other larger point of the chapter on inflation is that anyone wishing to comment on whether CPI reflects real experience ought to understand how CPI is constructed. A superficial understanding such as that it is the average price of a basket of goods is useless because there are so many little details that affect the statistic. Because inflation has no objective basis, it is pointless to argue if it reflects reality: all we are left with is discussing the rules and you can't discuss the rules without knowing them well.

Details matter a lot in statistics. This is one of the reasons why I keep asking my Big Data colleagues to talk specifics. A statistician who only talks in generality is like the Manhattan realtor who can't tell you the size of the listed apartment.

 

 

The backlash against Big Data has started

It is inevitable that all the hype around "Big Data" leads to a backlash. As someone who's been working in "data science" before the term existed, I am happy to see widespread validation of the field but also concerned about over-promise and under-deliver. Several recent articles went overboard in criticizing data science -- while their points are sometimes valid, the tone of these pieces misses the mark. I'll discuss one of these articles in this post, and some others in the next few days.

***

Andrew Gelman has a beef with David Brooks over his New York Times column called "What Data Can't Do". (link) I will get to Brooks's critique soon--my overall feeling is, he created a bunch of sound bites, and could have benefited from interviewing people like Andrew and myself, who are skeptical of Big Data claims but not maniacally dismissive.

The biggest issue with Brooks's column is the incessant use of the flawed man versus machine dichotomy. He warns: "It's foolish to swap the amazing machine in your skull for the crude machine on your desk." The machine he has in his mind is the science-fictional, self-sufficient, intelligent computer, as opposed to the algorithmic, dumb-and-dumber computer as it exists today and for the last many decades. A more appropriate analogy of today's computer (and of the foreseeable future) is a machine that the human brain creates to automate mechanical, repetitious tasks at scale. This machine cannot function without human piloting so it's man versus man-plus-machine, not man versus machine.

I use such an analogy in Chapter 2 of Numbers Rule Your World, to compare and contrast the credit-scoring algorithmic paradigm with the manual underwriting paradigm of the past. The point is that there is more similarity than difference between the automated and the manual methods; the automated methods are faster, better able to handle multiple threads, and unfazed by individual bias.

***

A major blind spot is ignoring the work of Kahneman and Tversky, and other behavioral psychologists, who have shown convincingly that the human brain is subject to all kinds of biases, and uses heuristics that lead to incorrect judgements.

A large body of work, for instance, points to the "priming" effect. Someone may walk into the supermarket and buy detergents just because he or she heard a story about cheating on the radio. Of course, people would deny such influences but experiments prove the effects exist. There's also the experiment that shows that subjects who are asked to hold a pencil in their mouth to activate "grin" muscles feel happier than those made to activate "growl" muscles.

It is comedic when Brooks tells us that "people are really good at telling stoires that weave together multiple causes and multiple contexts... data... cannot match the explanatory suppleness of even a mediocre novel". I mean, does he care if the "stories" and "novels" lead to correct decisions? Or is he just in it for entertainment?

***

While I agree with some of Brooks's diagnosis of the problems with data-driven analyses, it is often the case that the alternative of not using data or using the brain as he calls it does not create a demonstrably better outcome.

Under "Big Data has trouble with big problems," he complained: "We've had huge debates over the best economic stimulus, with mountains of data, and as far as I know not a single major player in this debate has been persuaded by data to switch sides."

Macroeconomics has always been a field that suffers from lack of data (in fact, a sample path of one). I don't know what "mountains of data" he's talking about, especially since the things we're doing like quantitative easing has not been done ever before. Nor do I understand why the proof of utility is side-switching. He may be right that the economists have not switched sides but this says more about the people--mind you, who have embarrassing records when it comes to managing the economy--than about the data. Given that Brooks doesn't think the economic decisions were based on data, then who should we blame the many economic failures on? The human brain?

"Data creates bigger haystack... and the needle we are looking for is still buried deep inside." This is definitely true. But why is data analysis the problem here? What's the alternative he has in mind? Most data analysts sooner or later realize that "data science" is as much art as science. We don't have to pick one or the other; we can have the best of both worlds.

***

Brooks made a really great point at the end of the piece, which I will paraphrase: any useful data is cooked. "The end result looks disinterested, but, in reality, there are value choices all the way through, from construction to interpretation." Instead of thinking about this as cause for concern, we should celebrate these "value choices" because they make the data more useful.

This brings me back to Gelman's reaction in which he differentiates between good analysis and bad analysis. Except for the simplest problems, any good analysis uses cooked data but an analysis using cooked data could be good or bad.

 

Typepad's complacency towards spam comments

It appears that the IT world today considers spam detection to be a solved problem. One doesn't hear many complaints about spam. Spam used to only infiltrate our emails; they now appear in many formats, including spam websites that adulterate search engine results, and spam comments that pollute blogging communities.

Many types of methods have been deployed to fight spam. Fighting spam is a statistical problem: algorithms try to predict whether each incoming email or comment is spam or not.

Bayesian spam filters, which determines what features of a piece of text are predictive of spam, are popular - the buttons that allow users to "report spam" or "report not spam" are integral to such algorithms, enabling them to learn by experience. Blacklists that block IP addresses are another method. The blacklist is a type of heuristic, essentially fixed rules.

***

I have been bothered by Typepad's seeming ineptitude in spam filtering for a long time. Certain comments that are obviously spam do not get filtered automatically. These are comments that appear to be extremely easy to identify via some simple heuristics. Namely, if the name of the person writing the comment is the name of a pharmaceutical drug or a brand of shoes or fashion, classify the comment as spam.

Imposing such rules will get rid of the following types of spam comments that recently have been "false negatives" under Typepad's so-called spam filtering product:

Cheap Jordans said:

If a thing is worth doing it is worth doing well.

Tiffany & Co Outlet Jewellery said:

There is someone that is coming or passing away in your life around the clock, so you may lose sight of those seen, and forget those remembered. There is gain and loss in your life, so you may catch sight of those unseen, and remember those forgotten. Nevertheless, doesn‘t the unseen exist for sure? Will the remembered remain for ever?

bike clothing said:

reaction of officials in Boston who advised residents to stop drinking tap water for fear of possible contamination which has not been verified.

louboutin shoes said:

My readers will note that I disapprove of how most statistics textbooks discuss the correlation/causation issue: you are told what not to do, but left with no idea of what to do.

The blog recently got flooded with 55 spams on a single day, coming from "louboutin", "louboutin shoes", and "christian louboutin sale". I just don't understand how any spam filtering algorithm worth its name can miss such obvious cases. It is extremely unusual to have someone whose first name is louboutin and last name is shoes. If someone has such a name, I'd argue (and sorry to be mean) that your comment deserves being expunged.

***

Every time I provide feedback to Typepad, their IT staff recommends that I manually block those IP addresses (the blacklist method). It appears that they don't see this as an obvious failure. They certainly don't realize that the failure to identify the spam already occurred. So I'm mystified. If you know anything about spam filters, please address these questions:

• Is it operationally difficult to add some heuristics to the algorithm?
• Is there something wrong with rules that assume no one is named "viagra" or "cheap viagra" or "cheap jordans"?
• Are there any new developments in spam filtering worth our attention?