Big Data, Plainly Spoken (aka Numbers Rule Your World)

Do wearable healthcare devices work?

A report came out from Stanford School of Medicine about a study of Apple Watch's health monitoring features. Some headline writers are proclaiming that "finally, there is proof that these watches benefit our health!" For example, Apple Watch Stanford Study Shows How It Can Save Lives (link).

When you read the official story, you will learn the following facts about the study:

• The research is funded by Apple
• It was a purely observational study in which they follow (400,000) people who wear Apple Watches
• Participants must own both an Apple Watch and an iPhone to be eligible (plus meeting other criteria)
• There was no "control" group - they did not follow anyone who did not use Apple Watch or use any other health monitoring wearables
• Every participant is self-selected
• The device issued warnings to only 0.5 percent of the participants (~ 2,160)
• Those who received a warning were directed to a video consultation; and the doctor decided whether or not to send the participant an ECG patch, which is used to establish the "ground truth". Only about 30 percent were sent patches, and of those, 70 percent (450) returned the patches for analysis.
• Only those who had ECG data were analyzed. One third of these were shown to have experienced "atrial fibrillation" (irregular heartbeat). This means that two-thirds got false alarms. But if we include the 70% who were not sent patches after the video consultation as false alarms as well, then out of every 100 warnings, only 7 were validated.
• There is no discussion of false negatives: did any of the 99.5 percent who did not receive warnings experienced irregular heartbeats?
• We do know that if there were significant false negatives, then more warnings would have to be sent, which pushes up false alarms.
• Despite the headlines, any lives saved were extrapolated.

There are some major methodological limitations about this study.

Firstly, the study design prevents drawing conclusions of the type "People wearing Apple Watches .... compared to those who did not wear Apple Watches." It does not include anyone not wearing Apple Watches.

Secondly, it's difficult to interpret the accuracy metrics. Is 20 percent false alarms a good or bad number? Is 0.5 percent receiving warnings a reasonable proportion given the demographic and health characteristics of the study population?

Hopefully, this study is just the beginning, and more rigorous studies are being planned.

 

Is A/B testing that scary?

Reader AR pointed me to this Fast Company article that examines the ethics of A/B testing.

The only way to comprehend this point of view is to think of A/B testing not as a scientific experiment but as a decision-making process that involves running an experiment. The researchers are unhappy that A/B tests could lend support to decisions that have undesirable impact on society.

Two such examples are described:

• Two images are tested for a job ad. During the test, site visitors were shown one of the two images, selected at random. The winner of the test is an image that disproportionately drives male applicants.
• Separate pricing tests are run in different zip codes. The "winning" prices at the conclusion of these tests are different for different zip codes. Because racial profiles differ by zip code, prices are in effect different for different races. Therefore, the test result leads to race-based discrimination.

There are two important questions to discuss here. First, what is the alternative to A/B testing? Is that method of decision-making better? Second, is the harm produced by the experiment itself, or by the decision made as a result?

 

Alternatives to A/B Testing

Consider the image test described above. Presumably, the test is run because someone believes that one of those two images might perform better at driving applicants. At most companies, a test sees the light of day after teams of people debate and prioritize testing ideas. If the test including a sexist image is run, then the team in charge of testing has approved it for some (possibly bad) reason.

If they didn't have A/B testing, how would they have decided which image to run? And if the image is not explicitly sexist - in other words, if the analyst had to analyze the data to learn that one image drove more male applicants - how would that insight be surfaced without running the other image? The alternative decision process may be even worse.

It is certainly true that automated A/B testing is risky - because no human beings are involved in turning test results into actions. The absence of humans is usually touted as a benefit by vendors of such testing tools. In this example, a human analyst reports on the test result, and includes the analysis by gender showing that while total applications increased, the winning image disproportionately attracted male applicants. The decision-makers can and should decide not to adopt the winning image based on that analysis. The A/B test revealed the bias but did not cause it.

Even without the gender issue, such analysis and discussion of results is necessary. For example, ad clicks can be generated by placing ads near scroll bars to stimulate accidental clicking. Human analysts can report that clicks increased but only through accidental clicking. The decision-makers can and should decide not to implement the winning design.

From where does the harm come?

The other example is more far-fetched. I am reverse-engineering the pricing test as described. Given that the test led to different prices for different zip codes, they would be running separate A/B tests stratified by zip code. Given the law of supply and demand, it might be the case that the winning price would be lower in poorer zip codes and higher in richer zip codes. This definitely results in price discrimination by zip code. If the design team did not want price discrimination by zip code, then such a test design would not have been approved so the test itself isn't creating harm.

Further, race-based price discrimination is accused because zip codes are correlated with race. Almost all variables are correlated with race. Age is correlated with race, so are income, education, what websites one visits, etc. So this standard leads to a banning of all segmentation and targeting policies. The only possible pricing policy would be one price for all.

***

In short, human supervision of A/B testing from design to interpretation is definitely needed. A/B tests provide a wealth of data to support decision-making. The biases highlighted by the Fast Company article are merely revealed by the testing - they are not caused by it.

 

How scary are the anti-vaxxers?

I don’t agree with Daniel’s conclusions in his article in Slate about the measles “crisis” but he did his research and there are lots to chew on. You don’t have to agree with him to find this article thought-provoking.

There is one paragraph which everyone should read. It’s a celebration of science, and how it saved lives. (Daniel used this story for a different purpose: he argued that we never “eradicated” measles, and therefore, the anti-vaxxers could never have reversed some mythical victory.)

During the most recent, major wave of measles infection in the U.S., between 1989 and 1991, close to 56,000 people fell ill and more than 100 people died...The 1989–91 epidemic was large enough and deadly enough to cast light on two pressing problems: First, that a single vaccine dose was not sufficient to protect children, and second, that black and Latino children, especially those living in urban areas, were less likely to be vaccinated, and thus more vulnerable to the disease.

Efforts were made to address both issues in the years that followed. A second measles shot was recommended for all children, while the federal government ramped up efforts to provide free vaccination to high-risk groups. The plan worked. By 1994, vaccine coverage for measles was closing in on 90 percent. The number of cases reported every year soon dropped from the thousands into the hundreds, and then into the tens. It was in response to this decline that experts from the CDC announced that measles had been “eliminated” from the U.S.

It's a great example of finding the drivers behind the data, and executing actions that successfully changed the numbers.

***

From the rest of the article, here are some useful tidbits:

• Vaccines work because of a phenomenon called “herd immunity,” which is a type of wisdom of the crowd. Diseases spread when people interact with each other. If both sides are vaccinated, then the risk of spreading is much, much lower. Thus, the higher the proportion of the vaccinated, the lower risk for everybody. The threshold desired by health authorities is 93 to 95 percent.
• In the past several decades, at a national level, the vaccination rate has stayed around 91 percent. So it is below the desired level but seems close enough not to cause alarm.
• Discredited research by a Dr. Wakefield ignited the anti-vaxxer movement. The publication of such research allows people to confirm their prior beliefs, and it often is hard to dislodge such beliefs, even when the research has been invalidated.
• In some localities, e.g. Somalis in Minnesota, the rate of vaccination has dropped drastically to below 50%. Those are isolated communities, and in aggregate, the level of vaccination has not changed.
• The number of measles cases while small has shown signs of increase. There have been triple-digit cases 7 out of the last 10 years, but only twice in the previous decade.
• The fatality rate of measles is extremely low, 11 deaths in 18 years, which is said to be similar to being killed by scorpions. Those who read Chapter 2 or 4 of Numbers Rule Your World (link) will recognize the need to think about the cost of errors. 

 

Grad school admissions, small data, grade inflation, and when metrics become anti-informative

Andrew Gelman nails it again with this post titled "combining apparently contradictory evidence." He uses the example of repeated tests given to the same student, such as the scores from multiple assignments within the same course. One student might get 80,80,80 for three equally-weighted assignments while another student might get 80,100,60. The issue is that the sample size of three is too small to judge not only the average score but also the variability in score.

I made a comment about exactly the same problem I encountered when reading applications for the MS program at Columbia. Most of the applicants have good STEM undergrad degrees and no meaningful work experience. At first, I thought the three reference letters would be useful to differentiate the applicants.

It turns out that most applicants get 3 good references, almost always from professors who taught them. Occasionally, an applicant would get a poor reference, i.e. the professor is not recommending the student. However, in all such cases, the one poor reference is contradicted by two good references. So who do I believe? I typically don't know the authors of these references, and therefore have no external information about their reliability.

I am very aware that the sample of three is too small. One is tempted to think that because this applicant got inconsistent references while most other students did not is a "signal" that this applicant must be worse than the average, but drawing that conclusion is to ignore the small sample size - and the small-sample problem is even worse when drawing conclusions using the observed inconsistency of grades!

***

Thinking back to the grad school admissions process also makes me more sympathetic about the practical rationale for Princeton's decision to walk back its grade-deflation policy (see critical post here).

You might think undergrad GPAs are useful for making admissions decisions. The decades of grade inflation have vanquished this metric, as almost every recent graduate has a GPA say in the 3.5 to 3.8 range.

In fact, when metrics are gamed, it is usually not just uninformative - it can be anti-informative. Such metrics can lead to very bad conclusions.

The difference in GPAs no longer reflects a difference in ability between students. They are more likely to be influenced by other factors such as (a) when the student graduated and (b) whether the school or department uses grade deflation (or a grading curve).

Take date of graduation. If someone has a GPA lower than 3.0, it is almost always the case that this student graduated in the 1990s or before. But the GPA numbers are typically not presented together with year of graduation - so if the analyst does not recognize and adjust for this long-term grade-inflation trend, then the older candidates face a systematic discrimination.

This line of thinking takes me back to Princeton's decision to end grade deflation. Same problem here - when the admissions officer reads the GPA, it's not typically presented next to the college that grants it. There are hundreds of colleges the officer might come across during the admissions process, so it's impossible to hold in one's head the grading policies of so many colleges. In fact, even though I know a lot about Princeton's grading policies over the years, it still requires unreasonable effort to bring this contextual information into the decision-making. For one thing, I'd have to be aware of the different periods of grade inflation, then deflation, then inflation, etc. Therefore, I believe that the grade-deflation policy put Princeton graduates at a disadvantage when competing for scholarships, grad-school spots, etc.

***

If schools are required to release data on grades, then it would be possible to overcome the interpretation problems with the GPAs. Knowing the grade distributions by major, by school, by year is a good start.

 

 

Exercising fails to stop obesity, doh

Axios has an informative article about obesity, and the various remedies such as exercising, diets, and so on. Their headline is: "Health and wellness are booming, but we're fatter than ever." They have compiled some data, shown in a triplet of graphs:

The problem of obesity is complex, and fascinating from a data perspective. I devoted an entire chapter of Numbersense (link) to issues around measuring obesity.

There is much more underneath the surface than what is presented here. Let me unpack the layers of complexity.

Correlation is not Causation

The simplest issue to explain - just because statisticians have been screaming about it forever. If you look at the obesity chart and the gym chart, it is entirely accurate to say that gym membership has been rising in lock step with obesity rate during this decade. Both metrics rose by roughly 20%; and so it is very tempting to argue that going to gyms makes you fatter.

Of course, if you draw that conclusion, you've just been disinvited from the party of statisticians.

Ecological Fallacy

Here's the disturbing bit: the charts are also compatible with the opposite conclusion - that gym membership reduces obesity. This is an example of why it's so hard to interpret observational data.

Note that the data analyst collapsed a 2x2 matrix into two aggregate rates. Imagine four types of people: those with or without gym membership, crossed with those who are obese or not obese. When you're aware of the four types, you should realize that the rate of obesity, aggregated across gym membership, is not a great metric. It's pretty obvious that the obesity rate of those who are gym members is lower than that of those who do not have membership. The average rate paints them with the same brush.

In the same way, gym membership, aggregated across obese and not obese people, is not a great metric.

You can reasonably assume that obesity rate for the gym members should be lower than the average obesity rate, for example, if the average is 25%, then perhaps the obesity rate for non-gym members is 15%.

It's possible that the 15% rate has not changed over time but if the obesity rate of the non-gym-members increases, the overall obesity rate will increase (note that there are five times as many non-gym-members as there are gym members). The 15% rate for gym members could even have improved, and the overall obesity rate could still decline to 30% - it just requires the non-gym-members to get even more obese.

When aggregating the rates, some information is lost, and that weakens our ability to draw conclusions about individuals.

Indirect Metrics

Gym membership is not the same as gym usage. The gym's ability to influence obesity would require usage, not just membership.

 

CDC Diet Recommendation

The bit about the CDC complaining that people don't consume the recommended levels of fruits and vegetables makes me wonder if their problem formulation is overly simplistic. The dietary guidelines appear to be an optimization of nutritional benefits. But the real problem is to maximize nutritional benefits under a budget constraint. Each item in the basket of recommended foods delivers an amount of benefits at a level of cost. The total cost can't exceed the household budget.

For anyone taking a traditional class on optimization, "the diet problem" is often the first problem discussed. Here is one exposition of the diet problem.

 

Why stock X is falling today

In this tumultous market, there is a flash flood of journalists and pseudo-journalists explaining "why stock X is falling today". On a simple web search, the first bunch of results have X equaling Atlassian, Hanesbrand, WWE, Overstock.com, Southwest.

These authors all follow the same script. They look at each company's financial statements, or recently announced earnings report summaries, pull out some negative bits, and claim that those items are the causes of the sharp stock declines.

It's disturbing how little science there is in these pieces. Even the problem statement is completely warped. For example, the Motley Fool article takes the recently-popular format of a synopsis, starting with a one-liner on "What Happened." Eventually, the reader expects to hear an explanation of why it happened, and what does it mean for you.

So what happened according to the Motley Fool reporter?

"After the company reported third-quarter financial results that beat industry watchers' top- and bottom-line forecasts, shares in Southwest Airlines Co. (NYSE:LUV) are trading 8.2% lower at 2:15 p.m. EDT today."

This is one loaded statement. The causal link between the Southwest's 3rd quarter results and the drop in its stock price is already baked in. The reporter isn't asking if the 8.2% price drop is caused by the financial results, or how much of the drop can be explained by the results; the entire problem is framed such that the only possible conclusion is that the stock drop is caused by and only by those results!

PS. There is one scenario under which that problem statement is justified - if the stock price chart shows a steep plunge right after the earnings were announced. From the chart below, it appears that the drop started around noon, in fact, following a few hours in which the stock was trending up. (I'm not seeing where that 8.2% came from.)

***

How would a statistician approach this question?

One starts with the overall market decline. Then estimate any excess decline of airline stocks versus the overall market. And if the decline hasn't been fully explained, then look for factors directly related to Southwest.

 

 

New Facebook data breach, and the importance of Business Intelligence

 

On Friday, Facebook announced that hackers have gained access to personal data of at least 50 million users (see here for example). Analysts immediately connected this incident to the Cambridge Analytica scandal. How is this data breach different from the Cambridge scandal?

How the data was breached

One should take any announcement of how a data breach occurred with a grain of salt: it's obvious that companies will not publish anything that attracts lawsuits; plus, it's unclear that there is a penalty for lying about the real reasons for a data breach. There is no external auditing, and the companies control the narratives and statistics surrounding these events.

Based on what Facebook told us, the Cambridge Analytica scandal involves Facebook partners gaming the system to obtain data about Facebook users. At worst, it is a violation of community standards, i.e. claiming to be doing academic research. The research firm uses tools provided by Facebook to obtain the data. (The firm maintains that it disclosed to users that it was using the data for non-research purposes.)

In the present case, Facebook claims that a combination of coding bugs (i.e. unintended features) enabled unknown partners to gain access to the user's entire account. Not even that, but any accounts on third-party apps or websites that the user signs on to using Facebook.

 

How the breach was discovered

The current scandal demonstrates the value of the business intelligence/business analytics functions within companies. We were told that Facebook first realized that certain metrics were showing unusual trends, and upon investigation, they discovered the bugs.

This is entirely believable. That's what happens when you have good data reports. They surface anomalies. These then have to be investigated. These investigations are extremely tricky because all you know is the trends are different. There are a thousand reasons for the shift. The analyst's job is to establish a cause-effect. Especially since the development community adopted "agile" practices, all kinds of self-imposed changes are occurring all the time with no warnings. For a site as large and complex as Facebook, it takes a huge effort just to get a list of all site changes within some specified time window!

What complicates this situation more is that the vulnerability was traced to multiple bugs, not just one. I could imagine the twists and turns and the false alarms that were generated during the investigation.

The nature of this problem is no different from an investigator trying to chase down an e-coli outbreak, which is detailed in Chapter 2 of Numbers Rule Your World (link).

The data science community is guilty of talking down on the business intelligence function. There is a misperception that BI is for less skilled people doing boring things. The reality is there is more science in BI than in so-called data science (defined here as software engineering). Science, after all, is about figuring out why things are as they are. Engineers, by contrast, use our understanding of science to change the way things are.

 

You've been warned

In my debut Youtube video, released a few weeks ago, I explain how Facebook collects data about you. My biggest tip about protecting yourself is to not use Facebook to log onto other websites. Convenience is the drug that lures you into the trap.

Think about those one-password services. Instead of hundreds of password, you have one. So the thief needs only one password to assess hundreds of sites. Using Facebook to log onto other websites makes Facebook the centralized point of attack by the bad players.

Note that this is not limited to Facebook. Using Google, Yahoo, Amazon, etc. to log onto other services is no different!

If it's more convenient for you, it's also more convenient for your enemies. Remember that.

Also in the video, you will learn

• why did Facebook invent the Like button
• how do Facebook know what you are doing outside Facebook
• why not clicking doesn't mean you're not being tracked
• why you shouldn't use Facebook to log into other services

See the video here, and send me ideas for future episodes!

A purely mathematical theory of risk is not enough 2

,

This is the continuation of the last post on Chapter 5 of John Adams's excellent book on Risk. In the previous post, I discussed the problems of measurement that undermines the mathematical theory of risk. This post focuses on the particularly compelling example of traffic black spots, which reveals some risks in the statistical interpretation of risk data.

***

What Adams said:

Parts of the [road] network that have experienced bad spell are defined as accident black spots. When they are "treated", the numbers of accidents usually go down - but they probably would have gone down anyway.

He goes on to show a table of data used to identify traffic black spots. By convention, you list all the sites by the rate of accidents in descending order. The top sites with the most accidents are grouped together as black spots. In that table, 14 sites were identified as the worst sites with on average 3.53 accidents in a "before" period. In the "after" period, the accident rate dropped to 1.50.

Here's the catch: there was no treatment separating the "before" and "after" period. The analysis is an "A/A test," which I have frequently advocated as a way for data analysts (or our clients) to get a feel for "uncertainty". So the halving of the accident rate has zilch to do with any safety measure. One plausible explanation for the sudden drop in accident rate is the phenomenon of regression to the mean: if you select the worst sites for any given time, then in the next window of time, it is likely that those sites will experience below-average accident rates. (Vice versa, if you focus on the best sites for any given time, it is likely that those sites will suffer above-average accident rates in the after period.)

If you read the table of data carefully, you will notice that the 2000+ sites that were perfectly safe (zero accidents) in the "before" period subsequently suffered an increase in the accident rate in the "after" period. The average accident rate in the "after" period was 0.19, still low as an average number, but when multiplied by 2,000 sites, it can't be ignored. To see this even more clearly, in the "after' period, the black spots contributed 1.50x14 = 21 total accidents while the "no accident" spots contributed 2000x0.19 = 380 total accidents, 18 times more than the black spots!

The disease of the "Top X" analysis is an epidemic in our Big Data era; the OCCAM nature of these datasets worsens the disease (link). Pouring money on the "top" prospects is folly when one has not established causation of treatment and effect.

***

"Top X" type analysis is regrettably common in practice. You take a set of data and divide up the data into groups (often called deciles, quartiles, etc. depending on how many groups). Then you can compute some metric of the "lift". For example, in the "before" period, in the aggregate (> 2,600 sites), the average accident rate was 0.26. So if we predict that accidents will happen in the "black spots" (i.e. the worst class of sites), then the lift is 3.53/0.26 = 13.5 times. That is to say, the black spots are predicted to see 13.5 times more accidents than the average site.

Unexpectedly, in the "after" period, this lift number plunged to below 6 times. In other words, the predictive power is poor (when computed on data not even collected at the time of model-building). Given that no treatment was applied, if the predictive model captured the underlying dynamics, we should expect to see the lift number to be close to 13 times again. (We're assuming no strong known other factor that could cause the accident rates to collapse during the "after" period - in fact, in aggregate, the total number of accidents slightly increased.)

Here is the more subtle error: usually, the "after" period is after some safety measure is applied to the black spots (and not the other sites). The expected result is a drop in accident rate at the black spots, providing evidence that the safety measure actually caused the reduction in accidents. It is therefore very convenient for analysts (and policymakers who hire them) to conclude that the drop from 13.5 times to 6 times is evidence of the effectiveness of the safety measure. What the above analysis shows is that even in the absence of a safety measure, the accident rate could have dropped by half! So if you did apply a safety measure, it would need to produce a massive effect beyond the regression to the mean.

There are two reasons why analysts may fall prey to this type of bad analysis: one is confirmation bias - seeing in the data what you want or expect to see; the other is being what Nassim Taleb calls "fooled by randomness".

***

Adams goes on to discuss "accident migration". This is a second-order phenomenon: sites near to "treated black spots" often see an increase of accidents - the hypothesis is that since drivers take more caution when passing through black spots, they end up being more careless after the black spots are treated and so accidents spike up in nearby areas. (Adams does point out that it is possible that accident migration is not a separate thing from regression to the mean.)

Some related reading is the material on highway congestion in my book, Numbers Rule Your World (link). The problem of road congestion is slippery. If you successfully reduce the amount of traffic along one route, drivers will then change their habits to return that route to congestion - for example, they may have chosen to go along local roads before.

A purely mathematical theory of risk is not enough 1

I have been reading the excellent book by John Adams titled Risk (link). This is a geographer's treatment of a subject that is a staple of mathematics, particularly probability math. A mathematical treatment creates objects called probability distributions, which are then taken as complete representations of risk. Adams challenges that construct, bringing a social scientist's sensitivity to the table. In particular, he points out how the mathematics of risk is undermined by measurement issues (i.e. data issues) and statistical issues. He is not invalidating math, just pointing out large cracks that are often ignored.

I will provide a more comprehensive review of the book eventually. I'm very excited by Chapter 5, titled "Measuring Risk", and specifically the example of "traffic black spots". This example is very instructive for anyone who is interested in the practical implications and interpretation of risk measures.

The post got long so I have split it into two parts. The second part will be posted on Monday, and it concerns a delicious bit of analysis related to traffic black spots.

 

 

 

***

Here are some key sentences related to the quality of the collected data:

1. The connection between potentially fatal events [accidents] and actual fatal events is rendered extremely tenuous by the phenomonen of risk compensation.... avoiding action is taken, with the result that there are insufficient fatal accidents to produce a pattern that can serve as a reliable guide to the effect of specific safety interventions. As a consequence, safety planners seek out other measures of risk, usually - in ascending order of numbers but in decreasing order of severity - they are injury, morbidity, property damage and near misses.
2. It is much easier to achieve "statistical significance" in studies of accident causation if one uses large numbers rather than small. Safety researchers therefore have an understandable preference for non-fatal accident or incident data over fatality data... in exercising this preference, they usually assume that fatalities and non-fatal incidents will be consistent proportions of total casualities.
3. [Evoking some data from London] The correlation between fatality rates and injury rates is very weak. Is the weak correlation real or simply a recording phenomenon? How many injuries equal one life?
4. Uncertainty in the data increases as the severity of the injury decreases. The fatality statistics are almost certainly the most accurate and reliable of the road accident statistics... the categorization and recording of injuries is generally not informed by any evidence from a medical examination.... [A British Medical Asssociation report said that] only one in four casualties classified as seriously injured are, in fact, seriously injured and many of those classified as slightly injured are in fact seriously injured..... some 30 percent of traffic accident casualties seen in hospital are not reported to the police, and that at least 70 percent of cyclist casualties go unreported.

This last point is widely applicable in the data science/analytics world. We often have large amounts of unreliable data, and small amounts of more reliable data. What we hope to have - and we don't - is large amounts of reliable data. A lot of bad analyses results from assuming we have large amounts of reliable data.

 

Good job Bloomberg, and some comments on ad measurement

I'm continuing to see important articles as our journalists shine light on the tech companies. This Bloomberg article reveals a previously secret relationship between Google and Mastercard. It is important also as a reminder that the intense focus on Facebook is misguided - as other big tech companies engage in similar practices.

I'd recommend reading the article in full. Here I'd like to bring out a few key observations for you to chew on.

1) The payment processors have become part of the surveillance network that has been constructed by the tech industry. In the past (when I was working in that industry almost two decades ago), payment processors played the role of toll booths that happened to be a very lucrative business collecting cash as the transactions stream through. Users' purchase data were safeguarded and remained private.

Sometime in the last 5 to 10 years, the payment processors decided that the purchase data is a goldmine. They now monetize this data through mostly secret deals with all kinds of companies. Most card-users are not aware that their purchases are being disclosed to third parties.

An analogy is Gmail and other web-based email systems. Gmail in particular pioneered the concept of reading users' emails to extract information. It is now known that Gmail allows selected partners to extract all kinds of information from emails, such as shopping receipts, boarding passes, etc. We knew this because of the scandal about the Unroll.me service in which the service provider extracted shopping information from emails under the pretense of finding email lists from which users can be unsubscribed from.

 

2) Next time reporters hear the tech companies talk about "anonymized" data or hiding of "personally identifiable information", they really need to press the companies for details. Look out for clever word play. It is simply not possible to get value from these datasets without access to information about each person.

In this instance, Google is merging two datasets, the clickstream (what ads people clicked on) and Mastercard purchases, with the goal of claiming that person A purchased item X after clicking on some Google ad. The Mastercard data contain your card number, and/or your name. The clickstream data contain one or more of the following: your Google "cookie", your Google user name, or your email. You can't merge the datasets unless there is a common field. In this case, it will most likely be your name or email address. How could that be "anonymous"? Good question if you're asking it. And the reporters should be pushing on this point.

Further, if the data are fully anonymized, the potential utility of the data is much lower, and the price marketers are willing to pay for them is much lower. If I don't have your name and address, how can I send you a piece of junk mail?

 

3) While the article points out the violation of privacy of this partnership, it inadvertently legitimizes the highly dubious science behind matching the clickstream data and the Mastercard purchase data. The reporter described the program as "powerful" and "potent" but should have talked to third-party experts to understand this science better. I'll just list several major problems with this approach:

• As far as I know, Mastercard does not have line-item level details on our purchases. It may know that you spent $500 at Best Buy, but it doesn't know that that total includes an iPad and a TV, for example. So if the ad shows a TV, how does Google know that Best Buy purchase included a TV? The reporter claims that Google knows that someone bought "red lipstick" with her Mastercard. I find that dubious.
• Do you remember what ads you clicked on 30 days ago? 15 days ago? 5 days ago? This system counts any purchases made on Mastercard as long as 30 days ago as "caused" by an ad click.
• Does Google know what these users are doing on non-Google websites? For example, does Google know that the user who saw a Google ad 10 days ago also saw a Facebook ad for the same product 3 days ago?
• Does Google know anything about the users' interactions offline? For example, the user who clicked on the Google ad 10 days ago may have heard a radio ad, seen a TV ad, talked to a friend about the product, been handed a flyer at the mall, saw a piece of direct mail, etc.

The reporter suggested that Facebook, the other digital advertising behemoth, is also pursuing similar deals with payment processors. If Facebook were to adopt this same method for measuring ad performance, then both Facebook and Google would have claimed credit for the same purchase, as each would have matched that purchase to its own clickstream data. One solution to this particular issue is to merge the Facebook and Google data, which simply merges two surveillance networks into one extra-large surveillance network.