Big Data, Plainly Spoken (aka Numbers Rule Your World)

The confusing pictures in Israel and in the Israel study

Today and tomorrow, I’m taking you through another real-world study which has become the favorite of mainstream media, the study by Israel’s largest health fund (insurer), that confirms the effectiveness of the Pfizer and Moderna vaccines. The report has appeared in the prestigious New England Journal of Medicine (NEJM) here (with supplements).

It has come to my attention that my readership is split between those who see my commentary on scientific papers as an anti-science provocation, and those who value reading my notes and have requested more. The goal of my blogs has always been to promote critical thinking, to encourage chewing before you swallow information. My target readers are people who want to stay for the entire movie – even to visit the sets – after watching the trailer. If you’re here, that probably describes you.

What is a Real-World Study?

The Israel study by Clalit is an example of a real-world observational study. I covered the basics of such studies here, and also described a similar study by the Mayo Clinic – nference team here.

In brief, real-world studies analyze data that are collected by healthcare systems, which are much more challenging than experimental data tracked during randomized clinical trials (RCTs). In real-world studies, the difference between the vaccinated and unvaccinated individuals is not just caused by vaccination, but it may be explained by a myriad of other factors, such as age bias, geographical bias, and socio-economic bias. In an RCT, because we use coin tosses to determine someone’s vaccine eligibility, all other factors are equalized, avoiding this mess.

The Clalit study adopts the same high-level strategy as the Mayo Clinic’s, manufacturing a synthetic control group that mimics the placebo group in a RCT. All vaccinated individuals during the study period are considered. Each vaccinated customer (who survives the exclusions) meets a matching algorithm that attempts to locate an unvaccinated individual who is similar on a list of factors, such as age and sex. Unmatched people are dropped from the analysis set. The vaccinated and unvaccinated subgroups can now be treated as if randomized. The researchers then compute vaccine effectiveness (VE) using various case-counting windows, and outcome metrics. They select one of these analyses to highlight on the front page.

For Clalit, they conclude that the real-world vaccine effectiveness was 57% during the case-counting window of 14 to 20 days after the first dose (1D+14 to 1D+20), under the criterion of “symptomatic Covid-19” which is closest to what the vaccine trials were measuring. Surprisingly, they did not mention until later the effectiveness from 7 days after the second dose, which was 94%. This number looks like the VE reported by Pfizer during the vaccine trial, even though the study populations are different.

Footnote: Yet another real-world study contradicts the lobby who’s been pushing for single doses or delaying second doses after “re-analyzing” the Pfizer case curve to argue that its vaccine yielded 90% efficacy after a single dose. For what it’s worth, the Mayo Clinic study reported 70% effectiveness in the comparable time frame as Clalit’s; neither number is close to 90%.

In both studies, the day of the first shot determines Day 0 for a vaccinated person, and sets the day of “study enrollment” (“recruitment”) for the matched unvaccinated individual. As I explained before, fixing a “Day 0” for the unvaccinated is a necessary hassle in these observational studies, a problem that does not arise in a randomized clinical trial in which people in the control group are given two placebo injections. As with the Mayo Clinic study, the date of infection of a matched control depends on the identity (and thus Day 0) of the vaccinated individual, when the algorithm selects one out of multiple possible pairings.

Matching Mechanics

The details of the matching process vary between the Clalit and Mayo Clinic studies. The Israeli insurer requires a stricter form of matching (“exact”) while the Mayo Clinic uses the propensity scoring approach. The propensity score distills a set of matching variables into a single dimension, so that the matches are approximate. For example, all matched pairs in the Clalit study must be the same sex. In propensity scoring models, it’s permissible to pair a male with a female – so long as the matched pair is sufficiently similar on other factors.

Stricter matching offers better assurance that the matched pairs are sufficiently similar to each other. However, statistical adjustments come with trade-offs. The yield of Clalit’s stricter matching process is lower than for the Mayo Clinic. As far as I can tell, every vaccinated individual at Mayo (excepting exclusions) is successfully matched. By contrast, only four out of ten vaccinated individuals remain in Clalit’s analysis set after the exclusion and matching process. The high drop-off rate presents formidable challenges to generalizing the study’s findings.

“All persons who were newly vaccinated during the period from December 20, 2020 to February 1, 2021, were matched to unvaccinated controls in a 1:1 ratio according to demographic and clinical characteristics.”

I sighed when I read this sentence which initiated the second paragraph of the NEJM paper.

If I am harsh, I’d say the quoted sentence is false. But such careless language routinely appears in journals in this field, including the Lancet, NEJM, BMJ, etc. The sentence implies (wrongly) that the findings of the real-world study applies to all Clalit clients who were vaccinated before February 1, 2021. The media have taken the liberty of extending this result to all vaccinated Israeli since half of Israel are members of Clalit.

The key data are to be found in the Supplement (Table S2), the top part of which is shown below.

'

The first column concerns the “full” vaccinated population while the third column describes the “matched” vaccinated population. In the table header, we learn of a drastic drop-off from 1.5 million to 597K. This 60% deletion is in large part due to the incurable imbalance problem I mentioned before (see here). Only two out of every five vaccinated individuals remain in the analysis set.

When reading real-world studies, we must always guard against our natural instinct to assume random (see this prior post). Many of the assumptions we make about randomized clinical trials (RCTs) cannot apply. If the analysis set were a random sample of 40 percent of the “full” data, the study’s findings can be generalized to the full dataset. But in an observational study, the analysis subset is decidedly not a random sample of the full population – remember, the point of matching cases to controls is to get rid of unwanted biases that exist in the full population! Therefore, this study’s finding concerns the matchable sub-population (the third column), and not all vaccinated Israelis.

Eventful Exclusions

The research team behind the Clalit study faces an insurmountable problem. The pace of vaccination in Israel is too fast for this type of studies. They quickly run out of unvaccinated individuals for matching, especially people in groups prioritized for vaccinations. These include health care workers, adults over 60 years old, and adults with coexisting conditions.

Look back at Table S2. Nine percent of the vaccinated people are 80 years or older, and this age group got trimmed down to only 4% in the analysis set. Meanwhile, a quarter of those vaccinated were under 40 (which is surprising given the priority rules), and this age group was over-weighted to 36% after the matching process. The matching process has introduced a strong youth bias into the real-world study. This must inform our understanding of the VE estimates.

Exclusions also cause the analysis set to deviate from the full set of vaccinated customers. Figure 1 in the NEJM paper provides partial information. Some highly consequential exclusions include people living in nursing homes, people confined to their homes, and health care workers. Because these high-risk segments have been omitted, the estimates of vaccine effectiveness are exaggerated if extrapolated to the general population. These exclusions also make any findings on hospitalizations, severe cases and deaths hard to stomach.

Of note, the original study protocol did not contain those specific exclusions. I’m not convinced by the justification of “high internal variability in the probability of exposure or the outcomes.” Another influential exclusion that was subsequently appended removes people who “had a health care interaction within 3 days before the vaccination date”. This one exclusion rule knocks out 13% of the vaccinated individuals (202K), or one-fifth of all exclusions. The justification for this rule is that those interactions “may indicate the start of symptomatic disease and may preclude vaccination.” I don’t know what to make of this since the rule was applied to the vaccinated group (Figure 1), and it appears that the excluded individuals have a high chance of getting infected after vaccination. We don’t know how they define a “health care interaction” or decide on 3 days, which was not pre-specified.

The full list of exclusions are: under 16, not a customer of Clalit, not a “continuous” customer of Clalit, a positive PCR test prior to December 20, 2020, residents of nursing homes, people confined to the home, health care workers, missing home address, missing BMI, missing smoking status, people who had a health care interaction within 3 days before vaccination.

The matching factors include: age, sex, sector (ethnicity), neighborhood, number of influenza vaccines in past 5 years, pregnancy, and number of coexisting conditions.

I’m curious about matching coexisting conditions on the count, which ignores severity or type of condition. This is an "exact" matching criterion that is inexact. An obese individual with healthy lungs can be paired with an individual with normal weight and chronic pulmonary disease, if both have just one coexisting condition (actually one or two but let’s not dwell on that here). I can’t comment further without access to the data; I just want to point this out for any methodology nerds out there.

***

So far, we've learned that the exclusion and matching process used by the Clalit team generated an analysis set that appears to bias younger, lower-risk individuals, thus we expect the vaccine effectiveness estimates to be over-stated. The precise impact of such bias is hard to gauge in real-world studies, which is why researchers prefer randomized trials. The team in Israel faces the unusual challenge of the fast pace of vaccination, leaving them with incurable imbalance. In the next post, I address how they attempted to work around this problem, and the limits to what scientists can do about it. (It's a look behind the movie sets that I promised you.)

These studies demonstrate that the vaccines are effective enough (and also safe enough). As with all real-world observational studies, we must be careful about generalizing the results to larger populations. The point of matching is to define two comparable groups by deleting individuals without appropriate comparators but in doing so, neither group represents the universe from which it is drawn.

Finally, it's important for anyone reading real-world studies to pull up the latest real-world Covid-19 statistics on Israel. Here are the cases and deaths charts from OurWorldinData.org.

 

The decline in cases has stalled for two weeks, and the daily total is currently 6 times higher than at the start of the current wave. Deaths may also be stalling, and are currently almost three times higher than at the start of the current wave.

Of course, one must not jump to conclusions. The curves are not purely driven by vaccinations.

 

Behind the proposed strengthening of FDA vaccine approval rules

News just broke that the FDA may announce more "rigorous" standards for a coronavirus vaccine to get approved. This indicates a willingness to listen to some of the critics of the perceived rushed job.

The specific rules mentioned in this Washington Post article are:

1) Participants must be followed for a median of at least two months, starting from when they receive the second dose

2) The analysis must be conducted after at least five severe cases of Covid19 have been observed in the placebo arm.

3) The analysis must be conducted after some minimum cases of Covid19 in older people has been recorded.

***

Rules 2 and 3 are fine in spirit. It's important that the vaccine works for severe cases and older people. These rules ensure that all trial analyses include the two groups that are of great concern to the medical community.

Recall that the case definition uses mild symptoms instead of severe symptoms. Since a lot more people develop mild symptoms, it's possible that the interim analysis contains only mild cases. It's also possible because of self-reporting that the cases used in the interim analysis involve only younger people.

I suspect those scenarios are unlikely anyway. Because testing is triggered by self-reporting, severe cases are more likely to surface than mild cases. So long as they have sufficient older participants in the trial, they should show up in the interim analysis because older people tend to have more severe illnesses.

Nevertheless, if the vaccine is approved by the end of 2020, it will be on the strength of an interim analysis. Such analyses are never designed to make statistically reliable statements about subgroups, such as older people or severe cases. This is another reason why the trial should run to completion even if the FDA grants interim approval.

***

Revised rule 1 is misleading and potentially worrying. 

Let's talk about its good intention first. Rule 1 sets up a second criterion for triggering the interim analysis. Previously, the only criterion is to exceed a prespecified total number of cases. This creates the potentially undesirable scenario in which enrollment is supercharged (either by pushing marketing, or relaxing admission standards), or the true case rate is markedly higher than assumed, such that the first interim analysis is triggered when most participants have barely taken the second dose.

By setting observation time as a second criterion, the new rule provides more confidence that the protection afforded by an approved vaccine lasts at least 2 months.

As I explained in this previous post, Moderna does not count cases until 2 weeks after the second dose. So if they track people for two months, it's really only 1.5 months of observation time. That isn't a lesser worry. What perturbs me is the use of the word "median".

Median is the middle person so the statement actually allows up to half of the participants to be observed for fewer than 2 months. Further, it is highly unusual and in fact statistically fallacious to perform analysis in which different participants are observed for different amounts of time. I hope this is a communications faux pas, a misunderstanding on the part of the journalist. The sensible rule is to require every participant (not the median participant) to be observed for at least 2 months (preferably longer) after the second dose.

 

 

P.S. [10/6/2020: A group of scientists and medical experts agree! STAT News writes about their pushback on rule #1. They want "a minimum of two months' observation for all participants".]

Buzzfeed notices the fake data menace

I first wrote about the fake data menace here (that was 2017) and again (2018). Earlier this year, I discussed an article in the Hustle about fake Amazon reviews, and made a video about it. 

This holiday season, Buzzfeed finally noticed. Here's a nice article detailing the scams that abuse online review platforms. Amazon (and other online retailers) prefer to look sideways for obvious reasons.

The reviewer featured in the Buzzfeed article has spent over $15,000 on "verified purchases" on Amazon - all of which were stuff that she bought in order to earn commissions after posting 5-star reviews. She wrote hundreds of reviews, not a single one true. She actually doesn't think she did anything wrong: "I'm just a pawn in their marketing scheme."

It's unfair to say Amazon and other retailers have no protections against fake reviews. The typical response to this problem is to set up community rules, and look sideways. If a case is raised to their attention, they will enforce the rule - but only on that case. For example, Amazon bans sellers from giving away free products in exchange for reviews but the ecommerce giant is not proactively seeking out offenders. (For example, there are many Facebook groups in which sellers and fake reviewers find each other. The article even mentions a rebate website where people can buy stuff from Amazon, wait for the return period to expire, and then get a full "rebate".) With a platform which is so vast, this mechanism allows the vast majority of fake reviews to persist.

Now that the reviewer has seen the sausage factory, her attitude towards the data is different. She confessed that she does not look at reviews when buying for herself: "When I see an off-brand product that’s Amazon’s Choice, [that label] doesn’t mean anything to me anymore."

There is zero financial incentive to fight fake reviews, for the simple reason that they work. These sham reviews are amplified by data science products - such as recommendation engines, which consider reviews as "data", the heralded unstructured "big data" - which push products that have large number of positive reviews (dubbed "relevant") to the top of search results, which inevitably are displayed in order of "relevance" by default. The top search results get clicked on the most, which creates a positive feedback loop keeping those products at the top. This is the "marketing scheme" mentioned above.

***

It's hard to come up with a solution to this fake data menace. For ecommerce, the harm done by the fake reviews is that other shoppers are duped into buying things that don't need, or things that are not what they appear to be. Also, honest merchants and brands lose market share. So you have this "little white lies" justification. Now, if these reviews are for medical services, then someone could get truly harmed by poor quality of care.

On the other side, lots of people benefit from the scams: the online retailers and the sellers all earn more revenues; the people who write fake reviews get free products and side gigs; friends of the fake-review writers get free handouts; suppliers of the sellers and retailers get more business; etc. Even data scientists arguably benefit as they have "more data" to work with.

Comment below if you have solutions!

 

TSA can now count its workers as paid

Last week, I pointed out that the U.S. payroll survey uses the following accounting rule: 

The Labor Department... determines the number of people on nonfarm payrolls by asking employers for a head count during the pay period including the 12th of the month. For many federal employees, that would be the pay period from Jan. 6 to Jan. 19.

Further, one only needs to have worked for one hour during that period to be counted as employed.

This weekend, we heard word that the government shutdown has no end in sight. 

On Sunday (Jan 13), we heard more news - that the TSA has decided to issue each TSA worker a one-time one-day pay plus $500 bonus. This means TSA can now report all workers as having been paid during the pay period including the 12th of January.

 

 

The importance of knowing how things are counted

Yesterday, on the sister blog, I posted a pair of charts that digs into how the unemployment rate is computed. This number has been in the news a lot because we supposedly have a super-tight labor market, the lowest unemployment rate since the peak of the tech boom in 2000! But the story is much more complex if one understands how unemployment levels are counted.

Another reason why counting unemployment is in the news is because of the government shutdown. And the Wall Street Journal has a nice (but rather convoluted) article about what that means for the unemployment number. This is where you can see how politics enter into the statistics - it's via rules that determine what gets counted and what doesn't.

***

As I pointed out in Numbersense (link), the Labor Department uses a specific definition of employment, and here is how the WSJ describes it:

The Labor Department... determines the number of people on nonfarm payrolls by asking employers for a head count during the pay period including the 12th of the month. For many federal employees, that would be the pay period from Jan. 6 to Jan. 19.

In other words, someone just needs to have worked at least one hour during that pay period to be considered "employed".

Based on that definition, if the government shutdown extends beyond Jan 19, then there should be a big negative impact on the unemployment rate.

***

But wait - politicians of neither party would want that to happen (in case they are the ones in power at the moment).

So what do we learn from the WSJ reporter? This:

even if the shutdown extends beyond the pay period, workers would be counted if legislation is enacted that requires agencies to pay employees for what they would have earned had there been no shutdown, a spokesman for the Labor Department’s Bureau of Labor Statistics said. Such practice would be consistent with how the department accounted for workers during previously lengthy shutdowns.

The reason given for this is precedence. But that actually means that the unemployment rate in that scenario would not accurately portray the state of employment in the U.S.

An economist cited in the article further explained that the following could also happen:

some federal employees initially furloughed have been called back to work and some are being paid with supplemental funds.

Since it only takes one hour of work during that specific pay period to be counted as employed, it's not hard to come up with ideas about how to turn these not-working people into employed people.

As a data analyst, there is no getting around learning all these nuances if you want to interpret data properly. It's the nature of our work.

When should you leave the baseball stadium when the game is just too damn long

My new article, joint work with Pravin and Sriram, has appeared in FiveThirtyEight this week. It's motivated by this chart, showing the lengths of baseball games:

Since 2000, the average Major League baseball game lasts over 3 hours; one out of 10 games takes over 3.5 hours. The added time is almost all non-action time. In fact, there was an article in the Wall Street Journal in which they logged a bunch of games and discovered that less than 10 percent of the game time can be classified as "action." For the uninitiated, the "non-action" time includes pitchers staring at the batter, batters taking practice swings, managers slow-walking to the mound to chit-chat with the pitcher, etc.

To their credit, the League office has started to pay attention to this problem although it seems like an agreement with the players' union is hard to come by. Some fans are taking matters into their own hands - why fight with all others to get out of the stadium when you can leave in the middle of the game, with one team "comfortably" ahead?

In our article, we model this decision as a stay-or-leave decision, and use Wald's Sequential Probability Ratio Test to scientifically determine the optimal time to leave. Here's the bottom line:

A back-of-the-envelope estimate pegs the total amount of time saved at 1,612 hours for the season. The price to pay was the 61 mistakes — dramatic comebacks that were heard on the radio from the parking lot by the fans who left early.

 

A Half-day in the Life of a Data Scientist

An organization wanted to understand its base of members so the first order of business was constructing a database of all people who can be considered members.  We decided to define membership broadly. Members included those who join the Facebook group, and those who subscribed to the newsletter.

 

The organization kept two separate lists which I would merge to create a master list. For simplicity, I’ll call them the FB list, and the Email list. In merging, the key is the key. Let me explain. The simplest key is an email address. If someone’s email address shows up on both lists, then I infer that those entries concern the same person, and combine them. My goal is to remove double counting of anyone who appears on both lists.

 

Sounds simple enough.

 

But never that simple, right? First, the Facebook group is the graveyard of data. Facebook provides zero statistics on group members and activities. Yes, the company that makes a business out of data does not hear the data-deprived group owners who have been pleading for years.

 

What is a data scientist to do? Scrape, that’s what. Members can find out who else is in the group by the scroll-wait-reset-scroll routine. You know that feeling. I know you do. You scroll to the bottom of a web page. Your browser gets the hint. It loads a few more items, while the slider floats away, usually to the wrong spot. You re-set the position, and scroll some more. After much scrolling, I scraped that page to compile the Facebook list. It’s got the name of the person, their Facebook username, and their location (when available).

 

Notice I didn’t say email address. So the FB list did not contain the all-important key. Another possible key is first and last names. Reviewing the email list, I realized that newsletter subscribers are not required to provide names so matching names to the FB list will yield few hits. The third candidate is not as accurate; I tried matching the Facebook username to the email username.

 

The client furnished an Excel file, which I’ve been calling the Email list. Upon opening the list, I turned the email address into all uppercase letters. I have matched enough text data to know that people are hardly in control of their fingers when they type text into web forms. “John”, “JOHN”, “joHN”, “JOhn”, and so on typically mean the same thing, regardless of case. (The occasional sadist offers “J0hn,” or “Jhon,” or “Jo hn.”)

 

Meanwhile, the client wondered if email addresses are really case-insensitive. I suggested asking Google. The search engine gave an ambiguous answer. The part after the @ sign is case-insensitive whereas the part before @ is case-sensitive, but then most email providers treat both parts as case-insensitive.

 

It’s rare when Google complicates your life. I fished out the UPPERCASE(email_address) formula, deleted it, broke up the email address into the user name and domain name parts, upper-cased the domain name, and reconnected the two parts, re-inserting the @ sign. The machine must follow these steps but a human being instinctively knows where to apply the cut. Some researchers believe the brain executes those steps at warp speed but I don’t buy it.  

 

Next, I dropped the domain names from the split-and-spliced email addresses to get ready to match to Facebook usernames. Sheesh, the client did not ask if Facebook usernames are case sensitive or not. (They aren’t.) I proceeded to merge the two lists.

 

I executed a “full outer join.” With this procedure, any username that appears in one or both of the lists will find its way to the output dataset. On this first attempt, nothing merged. Even though username “davidcolumbus,” say, lived on both lists, the computer did not combine the data; the two matches sat one on top of the other.

 

I took a deep breath, for I had reached a point where I must be honest with myself. This project was sure to bust the two hours I originally allotted. The merge could easily take another hour, maybe two, if no new issues emerged.

 

The matching rows did not combine because the computer only joins eponymous columns. Since the Facebook and email usernames are different entities, those columns carry different labels.

 

But syncing those labels solves one problem while creating another! Members who appear on only one list have only one of the usernames. Besides, Facebook usernames are unique while email usernames, when detached from their domains, are not. A better solution is to set up a third username column in both lists, whose purpose in life is to be the matching key.

 

What about the other columns? Did I want them combined or not? Take as an example first and last names which show up on both lists. If I standardized the labels of these columns, the computer would attempt to merge them. What if David Columbus appeared as Dave Columbus on the other list with matching usernames? Forcibly combining the name columns would cause one of these variations to be dropped. If I wanted to keep both spellings, I must retain all name columns, which happens should I assign distinct labels, which is exactly the opposite of what I did with the username columns.

 

If that isn’t confusing enough, I stumbled upon another issue. In the Email list, while most names appeared as “First <space> Last,” there were examples of “Last <space> First”, and “Last <comma> First”, and “First Initial <space> Last,” and so on.  As an analyst, your first thought is “What’s wrong with our designers? Why didn’t they create separate text boxes for first and last names?” Then, you accept that blame gets you nowhere; you still have to fix what’s broken.

 

A soft voice enters your head. You wish you hadn’t seen the problem. You hope it was just a bad dream. But you wake up.

 

In front of me I had two paths. I could follow path A, and that meant developing code to automatically detect the various anomalies and fixing them. This path would take hours. Which is the first name in “Scott Lewis”? How would a computer figure this out? What rule could apply generally?

 

And then, there was path B, better known as handcrafting. If I had 1,000 rows of data, and if it took two seconds to scan a name and determine the type of anomaly, I would have completed the exercise in 30 minutes or so.

 

I chose path B. It was ugly and unsexy but more of a sure thing.

 

I wish I could tell you I stopped looking. But I couldn’t help it. Some cultures embrace double surnames, like “De” something or “Von” something. My code was  parsing “Chris De Jong” as first name Chris, and last name Jong. I needed a more complex rule. Something like “If the name has three words, take the first as first name, and the last two as the surname.” This rule runs afoul of someone like “Mary Anne Rutherford.” At a crossroad again. I could teach a computer how to lump the middle name, or I could exercise my brain some more.

 

By this time, I was exhausted. If you have followed me to this point, you have my admiration. In the next installment, I shall finish the assignment.

The mystery of the official inflation rate

In case you are not subscribed to my dataviz feed, I put up a post yesterday that is highly relevant to readers here interested in statistical topics. The post discusses a graphic of a New York Times article that interprets the official inflation rate (known as the CPI). I devoted an entire chapter of Numbersense (link)to the question of why the official inflation rate diverges from our everyday experience.

In a larger context, inflation rate is an invented metric, invented to measure some quantity that has no objective reality. This is true of a lot of statistics. Revenues and profits are also invented concepts, for example, and only attain meaning through generally accepted accounting rules. Obesity, which is discussed in Chapter 2 of Numbersense (link) is another example of a quantity that has meaning only because of a convention of measuring.

The article in NYT brings up one of the points I raised in the book, which is that price increases are magnified in our imagination while price decreases are taken for granted.

The other larger point of the chapter on inflation is that anyone wishing to comment on whether CPI reflects real experience ought to understand how CPI is constructed. A superficial understanding such as that it is the average price of a basket of goods is useless because there are so many little details that affect the statistic. Because inflation has no objective basis, it is pointless to argue if it reflects reality: all we are left with is discussing the rules and you can't discuss the rules without knowing them well.

Details matter a lot in statistics. This is one of the reasons why I keep asking my Big Data colleagues to talk specifics. A statistician who only talks in generality is like the Manhattan realtor who can't tell you the size of the listed apartment.

 

 

The backlash against Big Data has started

It is inevitable that all the hype around "Big Data" leads to a backlash. As someone who's been working in "data science" before the term existed, I am happy to see widespread validation of the field but also concerned about over-promise and under-deliver. Several recent articles went overboard in criticizing data science -- while their points are sometimes valid, the tone of these pieces misses the mark. I'll discuss one of these articles in this post, and some others in the next few days.

***

Andrew Gelman has a beef with David Brooks over his New York Times column called "What Data Can't Do". (link) I will get to Brooks's critique soon--my overall feeling is, he created a bunch of sound bites, and could have benefited from interviewing people like Andrew and myself, who are skeptical of Big Data claims but not maniacally dismissive.

The biggest issue with Brooks's column is the incessant use of the flawed man versus machine dichotomy. He warns: "It's foolish to swap the amazing machine in your skull for the crude machine on your desk." The machine he has in his mind is the science-fictional, self-sufficient, intelligent computer, as opposed to the algorithmic, dumb-and-dumber computer as it exists today and for the last many decades. A more appropriate analogy of today's computer (and of the foreseeable future) is a machine that the human brain creates to automate mechanical, repetitious tasks at scale. This machine cannot function without human piloting so it's man versus man-plus-machine, not man versus machine.

I use such an analogy in Chapter 2 of Numbers Rule Your World, to compare and contrast the credit-scoring algorithmic paradigm with the manual underwriting paradigm of the past. The point is that there is more similarity than difference between the automated and the manual methods; the automated methods are faster, better able to handle multiple threads, and unfazed by individual bias.

***

A major blind spot is ignoring the work of Kahneman and Tversky, and other behavioral psychologists, who have shown convincingly that the human brain is subject to all kinds of biases, and uses heuristics that lead to incorrect judgements.

A large body of work, for instance, points to the "priming" effect. Someone may walk into the supermarket and buy detergents just because he or she heard a story about cheating on the radio. Of course, people would deny such influences but experiments prove the effects exist. There's also the experiment that shows that subjects who are asked to hold a pencil in their mouth to activate "grin" muscles feel happier than those made to activate "growl" muscles.

It is comedic when Brooks tells us that "people are really good at telling stoires that weave together multiple causes and multiple contexts... data... cannot match the explanatory suppleness of even a mediocre novel". I mean, does he care if the "stories" and "novels" lead to correct decisions? Or is he just in it for entertainment?

***

While I agree with some of Brooks's diagnosis of the problems with data-driven analyses, it is often the case that the alternative of not using data or using the brain as he calls it does not create a demonstrably better outcome.

Under "Big Data has trouble with big problems," he complained: "We've had huge debates over the best economic stimulus, with mountains of data, and as far as I know not a single major player in this debate has been persuaded by data to switch sides."

Macroeconomics has always been a field that suffers from lack of data (in fact, a sample path of one). I don't know what "mountains of data" he's talking about, especially since the things we're doing like quantitative easing has not been done ever before. Nor do I understand why the proof of utility is side-switching. He may be right that the economists have not switched sides but this says more about the people--mind you, who have embarrassing records when it comes to managing the economy--than about the data. Given that Brooks doesn't think the economic decisions were based on data, then who should we blame the many economic failures on? The human brain?

"Data creates bigger haystack... and the needle we are looking for is still buried deep inside." This is definitely true. But why is data analysis the problem here? What's the alternative he has in mind? Most data analysts sooner or later realize that "data science" is as much art as science. We don't have to pick one or the other; we can have the best of both worlds.

***

Brooks made a really great point at the end of the piece, which I will paraphrase: any useful data is cooked. "The end result looks disinterested, but, in reality, there are value choices all the way through, from construction to interpretation." Instead of thinking about this as cause for concern, we should celebrate these "value choices" because they make the data more useful.

This brings me back to Gelman's reaction in which he differentiates between good analysis and bad analysis. Except for the simplest problems, any good analysis uses cooked data but an analysis using cooked data could be good or bad.

 

Typepad's complacency towards spam comments

It appears that the IT world today considers spam detection to be a solved problem. One doesn't hear many complaints about spam. Spam used to only infiltrate our emails; they now appear in many formats, including spam websites that adulterate search engine results, and spam comments that pollute blogging communities.

Many types of methods have been deployed to fight spam. Fighting spam is a statistical problem: algorithms try to predict whether each incoming email or comment is spam or not.

Bayesian spam filters, which determines what features of a piece of text are predictive of spam, are popular - the buttons that allow users to "report spam" or "report not spam" are integral to such algorithms, enabling them to learn by experience. Blacklists that block IP addresses are another method. The blacklist is a type of heuristic, essentially fixed rules.

***

I have been bothered by Typepad's seeming ineptitude in spam filtering for a long time. Certain comments that are obviously spam do not get filtered automatically. These are comments that appear to be extremely easy to identify via some simple heuristics. Namely, if the name of the person writing the comment is the name of a pharmaceutical drug or a brand of shoes or fashion, classify the comment as spam.

Imposing such rules will get rid of the following types of spam comments that recently have been "false negatives" under Typepad's so-called spam filtering product:

Cheap Jordans said:

If a thing is worth doing it is worth doing well.

Tiffany & Co Outlet Jewellery said:

There is someone that is coming or passing away in your life around the clock, so you may lose sight of those seen, and forget those remembered. There is gain and loss in your life, so you may catch sight of those unseen, and remember those forgotten. Nevertheless, doesn‘t the unseen exist for sure? Will the remembered remain for ever?

bike clothing said:

reaction of officials in Boston who advised residents to stop drinking tap water for fear of possible contamination which has not been verified.

louboutin shoes said:

My readers will note that I disapprove of how most statistics textbooks discuss the correlation/causation issue: you are told what not to do, but left with no idea of what to do.

The blog recently got flooded with 55 spams on a single day, coming from "louboutin", "louboutin shoes", and "christian louboutin sale". I just don't understand how any spam filtering algorithm worth its name can miss such obvious cases. It is extremely unusual to have someone whose first name is louboutin and last name is shoes. If someone has such a name, I'd argue (and sorry to be mean) that your comment deserves being expunged.

***

Every time I provide feedback to Typepad, their IT staff recommends that I manually block those IP addresses (the blacklist method). It appears that they don't see this as an obvious failure. They certainly don't realize that the failure to identify the spam already occurred. So I'm mystified. If you know anything about spam filters, please address these questions:

• Is it operationally difficult to add some heuristics to the algorithm?
• Is there something wrong with rules that assume no one is named "viagra" or "cheap viagra" or "cheap jordans"?
• Are there any new developments in spam filtering worth our attention?