Big Data, Plainly Spoken (aka Numbers Rule Your World)

How scary are the anti-vaxxers?

I don’t agree with Daniel’s conclusions in his article in Slate about the measles “crisis” but he did his research and there are lots to chew on. You don’t have to agree with him to find this article thought-provoking.

There is one paragraph which everyone should read. It’s a celebration of science, and how it saved lives. (Daniel used this story for a different purpose: he argued that we never “eradicated” measles, and therefore, the anti-vaxxers could never have reversed some mythical victory.)

During the most recent, major wave of measles infection in the U.S., between 1989 and 1991, close to 56,000 people fell ill and more than 100 people died...The 1989–91 epidemic was large enough and deadly enough to cast light on two pressing problems: First, that a single vaccine dose was not sufficient to protect children, and second, that black and Latino children, especially those living in urban areas, were less likely to be vaccinated, and thus more vulnerable to the disease.

Efforts were made to address both issues in the years that followed. A second measles shot was recommended for all children, while the federal government ramped up efforts to provide free vaccination to high-risk groups. The plan worked. By 1994, vaccine coverage for measles was closing in on 90 percent. The number of cases reported every year soon dropped from the thousands into the hundreds, and then into the tens. It was in response to this decline that experts from the CDC announced that measles had been “eliminated” from the U.S.

It's a great example of finding the drivers behind the data, and executing actions that successfully changed the numbers.

***

From the rest of the article, here are some useful tidbits:

• Vaccines work because of a phenomenon called “herd immunity,” which is a type of wisdom of the crowd. Diseases spread when people interact with each other. If both sides are vaccinated, then the risk of spreading is much, much lower. Thus, the higher the proportion of the vaccinated, the lower risk for everybody. The threshold desired by health authorities is 93 to 95 percent.
• In the past several decades, at a national level, the vaccination rate has stayed around 91 percent. So it is below the desired level but seems close enough not to cause alarm.
• Discredited research by a Dr. Wakefield ignited the anti-vaxxer movement. The publication of such research allows people to confirm their prior beliefs, and it often is hard to dislodge such beliefs, even when the research has been invalidated.
• In some localities, e.g. Somalis in Minnesota, the rate of vaccination has dropped drastically to below 50%. Those are isolated communities, and in aggregate, the level of vaccination has not changed.
• The number of measles cases while small has shown signs of increase. There have been triple-digit cases 7 out of the last 10 years, but only twice in the previous decade.
• The fatality rate of measles is extremely low, 11 deaths in 18 years, which is said to be similar to being killed by scorpions. Those who read Chapter 2 or 4 of Numbers Rule Your World (link) will recognize the need to think about the cost of errors. 

 

Grad school admissions, small data, grade inflation, and when metrics become anti-informative

Andrew Gelman nails it again with this post titled "combining apparently contradictory evidence." He uses the example of repeated tests given to the same student, such as the scores from multiple assignments within the same course. One student might get 80,80,80 for three equally-weighted assignments while another student might get 80,100,60. The issue is that the sample size of three is too small to judge not only the average score but also the variability in score.

I made a comment about exactly the same problem I encountered when reading applications for the MS program at Columbia. Most of the applicants have good STEM undergrad degrees and no meaningful work experience. At first, I thought the three reference letters would be useful to differentiate the applicants.

It turns out that most applicants get 3 good references, almost always from professors who taught them. Occasionally, an applicant would get a poor reference, i.e. the professor is not recommending the student. However, in all such cases, the one poor reference is contradicted by two good references. So who do I believe? I typically don't know the authors of these references, and therefore have no external information about their reliability.

I am very aware that the sample of three is too small. One is tempted to think that because this applicant got inconsistent references while most other students did not is a "signal" that this applicant must be worse than the average, but drawing that conclusion is to ignore the small sample size - and the small-sample problem is even worse when drawing conclusions using the observed inconsistency of grades!

***

Thinking back to the grad school admissions process also makes me more sympathetic about the practical rationale for Princeton's decision to walk back its grade-deflation policy (see critical post here).

You might think undergrad GPAs are useful for making admissions decisions. The decades of grade inflation have vanquished this metric, as almost every recent graduate has a GPA say in the 3.5 to 3.8 range.

In fact, when metrics are gamed, it is usually not just uninformative - it can be anti-informative. Such metrics can lead to very bad conclusions.

The difference in GPAs no longer reflects a difference in ability between students. They are more likely to be influenced by other factors such as (a) when the student graduated and (b) whether the school or department uses grade deflation (or a grading curve).

Take date of graduation. If someone has a GPA lower than 3.0, it is almost always the case that this student graduated in the 1990s or before. But the GPA numbers are typically not presented together with year of graduation - so if the analyst does not recognize and adjust for this long-term grade-inflation trend, then the older candidates face a systematic discrimination.

This line of thinking takes me back to Princeton's decision to end grade deflation. Same problem here - when the admissions officer reads the GPA, it's not typically presented next to the college that grants it. There are hundreds of colleges the officer might come across during the admissions process, so it's impossible to hold in one's head the grading policies of so many colleges. In fact, even though I know a lot about Princeton's grading policies over the years, it still requires unreasonable effort to bring this contextual information into the decision-making. For one thing, I'd have to be aware of the different periods of grade inflation, then deflation, then inflation, etc. Therefore, I believe that the grade-deflation policy put Princeton graduates at a disadvantage when competing for scholarships, grad-school spots, etc.

***

If schools are required to release data on grades, then it would be possible to overcome the interpretation problems with the GPAs. Knowing the grade distributions by major, by school, by year is a good start.

 

 

Exercising fails to stop obesity, doh

Axios has an informative article about obesity, and the various remedies such as exercising, diets, and so on. Their headline is: "Health and wellness are booming, but we're fatter than ever." They have compiled some data, shown in a triplet of graphs:

The problem of obesity is complex, and fascinating from a data perspective. I devoted an entire chapter of Numbersense (link) to issues around measuring obesity.

There is much more underneath the surface than what is presented here. Let me unpack the layers of complexity.

Correlation is not Causation

The simplest issue to explain - just because statisticians have been screaming about it forever. If you look at the obesity chart and the gym chart, it is entirely accurate to say that gym membership has been rising in lock step with obesity rate during this decade. Both metrics rose by roughly 20%; and so it is very tempting to argue that going to gyms makes you fatter.

Of course, if you draw that conclusion, you've just been disinvited from the party of statisticians.

Ecological Fallacy

Here's the disturbing bit: the charts are also compatible with the opposite conclusion - that gym membership reduces obesity. This is an example of why it's so hard to interpret observational data.

Note that the data analyst collapsed a 2x2 matrix into two aggregate rates. Imagine four types of people: those with or without gym membership, crossed with those who are obese or not obese. When you're aware of the four types, you should realize that the rate of obesity, aggregated across gym membership, is not a great metric. It's pretty obvious that the obesity rate of those who are gym members is lower than that of those who do not have membership. The average rate paints them with the same brush.

In the same way, gym membership, aggregated across obese and not obese people, is not a great metric.

You can reasonably assume that obesity rate for the gym members should be lower than the average obesity rate, for example, if the average is 25%, then perhaps the obesity rate for non-gym members is 15%.

It's possible that the 15% rate has not changed over time but if the obesity rate of the non-gym-members increases, the overall obesity rate will increase (note that there are five times as many non-gym-members as there are gym members). The 15% rate for gym members could even have improved, and the overall obesity rate could still decline to 30% - it just requires the non-gym-members to get even more obese.

When aggregating the rates, some information is lost, and that weakens our ability to draw conclusions about individuals.

Indirect Metrics

Gym membership is not the same as gym usage. The gym's ability to influence obesity would require usage, not just membership.

 

CDC Diet Recommendation

The bit about the CDC complaining that people don't consume the recommended levels of fruits and vegetables makes me wonder if their problem formulation is overly simplistic. The dietary guidelines appear to be an optimization of nutritional benefits. But the real problem is to maximize nutritional benefits under a budget constraint. Each item in the basket of recommended foods delivers an amount of benefits at a level of cost. The total cost can't exceed the household budget.

For anyone taking a traditional class on optimization, "the diet problem" is often the first problem discussed. Here is one exposition of the diet problem.

 

Why stock X is falling today

In this tumultous market, there is a flash flood of journalists and pseudo-journalists explaining "why stock X is falling today". On a simple web search, the first bunch of results have X equaling Atlassian, Hanesbrand, WWE, Overstock.com, Southwest.

These authors all follow the same script. They look at each company's financial statements, or recently announced earnings report summaries, pull out some negative bits, and claim that those items are the causes of the sharp stock declines.

It's disturbing how little science there is in these pieces. Even the problem statement is completely warped. For example, the Motley Fool article takes the recently-popular format of a synopsis, starting with a one-liner on "What Happened." Eventually, the reader expects to hear an explanation of why it happened, and what does it mean for you.

So what happened according to the Motley Fool reporter?

"After the company reported third-quarter financial results that beat industry watchers' top- and bottom-line forecasts, shares in Southwest Airlines Co. (NYSE:LUV) are trading 8.2% lower at 2:15 p.m. EDT today."

This is one loaded statement. The causal link between the Southwest's 3rd quarter results and the drop in its stock price is already baked in. The reporter isn't asking if the 8.2% price drop is caused by the financial results, or how much of the drop can be explained by the results; the entire problem is framed such that the only possible conclusion is that the stock drop is caused by and only by those results!

PS. There is one scenario under which that problem statement is justified - if the stock price chart shows a steep plunge right after the earnings were announced. From the chart below, it appears that the drop started around noon, in fact, following a few hours in which the stock was trending up. (I'm not seeing where that 8.2% came from.)

***

How would a statistician approach this question?

One starts with the overall market decline. Then estimate any excess decline of airline stocks versus the overall market. And if the decline hasn't been fully explained, then look for factors directly related to Southwest.

 

 

New Facebook data breach, and the importance of Business Intelligence

 

On Friday, Facebook announced that hackers have gained access to personal data of at least 50 million users (see here for example). Analysts immediately connected this incident to the Cambridge Analytica scandal. How is this data breach different from the Cambridge scandal?

How the data was breached

One should take any announcement of how a data breach occurred with a grain of salt: it's obvious that companies will not publish anything that attracts lawsuits; plus, it's unclear that there is a penalty for lying about the real reasons for a data breach. There is no external auditing, and the companies control the narratives and statistics surrounding these events.

Based on what Facebook told us, the Cambridge Analytica scandal involves Facebook partners gaming the system to obtain data about Facebook users. At worst, it is a violation of community standards, i.e. claiming to be doing academic research. The research firm uses tools provided by Facebook to obtain the data. (The firm maintains that it disclosed to users that it was using the data for non-research purposes.)

In the present case, Facebook claims that a combination of coding bugs (i.e. unintended features) enabled unknown partners to gain access to the user's entire account. Not even that, but any accounts on third-party apps or websites that the user signs on to using Facebook.

 

How the breach was discovered

The current scandal demonstrates the value of the business intelligence/business analytics functions within companies. We were told that Facebook first realized that certain metrics were showing unusual trends, and upon investigation, they discovered the bugs.

This is entirely believable. That's what happens when you have good data reports. They surface anomalies. These then have to be investigated. These investigations are extremely tricky because all you know is the trends are different. There are a thousand reasons for the shift. The analyst's job is to establish a cause-effect. Especially since the development community adopted "agile" practices, all kinds of self-imposed changes are occurring all the time with no warnings. For a site as large and complex as Facebook, it takes a huge effort just to get a list of all site changes within some specified time window!

What complicates this situation more is that the vulnerability was traced to multiple bugs, not just one. I could imagine the twists and turns and the false alarms that were generated during the investigation.

The nature of this problem is no different from an investigator trying to chase down an e-coli outbreak, which is detailed in Chapter 2 of Numbers Rule Your World (link).

The data science community is guilty of talking down on the business intelligence function. There is a misperception that BI is for less skilled people doing boring things. The reality is there is more science in BI than in so-called data science (defined here as software engineering). Science, after all, is about figuring out why things are as they are. Engineers, by contrast, use our understanding of science to change the way things are.

 

You've been warned

In my debut Youtube video, released a few weeks ago, I explain how Facebook collects data about you. My biggest tip about protecting yourself is to not use Facebook to log onto other websites. Convenience is the drug that lures you into the trap.

Think about those one-password services. Instead of hundreds of password, you have one. So the thief needs only one password to assess hundreds of sites. Using Facebook to log onto other websites makes Facebook the centralized point of attack by the bad players.

Note that this is not limited to Facebook. Using Google, Yahoo, Amazon, etc. to log onto other services is no different!

If it's more convenient for you, it's also more convenient for your enemies. Remember that.

Also in the video, you will learn

• why did Facebook invent the Like button
• how do Facebook know what you are doing outside Facebook
• why not clicking doesn't mean you're not being tracked
• why you shouldn't use Facebook to log into other services

See the video here, and send me ideas for future episodes!

A purely mathematical theory of risk is not enough 2

,

This is the continuation of the last post on Chapter 5 of John Adams's excellent book on Risk. In the previous post, I discussed the problems of measurement that undermines the mathematical theory of risk. This post focuses on the particularly compelling example of traffic black spots, which reveals some risks in the statistical interpretation of risk data.

***

What Adams said:

Parts of the [road] network that have experienced bad spell are defined as accident black spots. When they are "treated", the numbers of accidents usually go down - but they probably would have gone down anyway.

He goes on to show a table of data used to identify traffic black spots. By convention, you list all the sites by the rate of accidents in descending order. The top sites with the most accidents are grouped together as black spots. In that table, 14 sites were identified as the worst sites with on average 3.53 accidents in a "before" period. In the "after" period, the accident rate dropped to 1.50.

Here's the catch: there was no treatment separating the "before" and "after" period. The analysis is an "A/A test," which I have frequently advocated as a way for data analysts (or our clients) to get a feel for "uncertainty". So the halving of the accident rate has zilch to do with any safety measure. One plausible explanation for the sudden drop in accident rate is the phenomenon of regression to the mean: if you select the worst sites for any given time, then in the next window of time, it is likely that those sites will experience below-average accident rates. (Vice versa, if you focus on the best sites for any given time, it is likely that those sites will suffer above-average accident rates in the after period.)

If you read the table of data carefully, you will notice that the 2000+ sites that were perfectly safe (zero accidents) in the "before" period subsequently suffered an increase in the accident rate in the "after" period. The average accident rate in the "after" period was 0.19, still low as an average number, but when multiplied by 2,000 sites, it can't be ignored. To see this even more clearly, in the "after' period, the black spots contributed 1.50x14 = 21 total accidents while the "no accident" spots contributed 2000x0.19 = 380 total accidents, 18 times more than the black spots!

The disease of the "Top X" analysis is an epidemic in our Big Data era; the OCCAM nature of these datasets worsens the disease (link). Pouring money on the "top" prospects is folly when one has not established causation of treatment and effect.

***

"Top X" type analysis is regrettably common in practice. You take a set of data and divide up the data into groups (often called deciles, quartiles, etc. depending on how many groups). Then you can compute some metric of the "lift". For example, in the "before" period, in the aggregate (> 2,600 sites), the average accident rate was 0.26. So if we predict that accidents will happen in the "black spots" (i.e. the worst class of sites), then the lift is 3.53/0.26 = 13.5 times. That is to say, the black spots are predicted to see 13.5 times more accidents than the average site.

Unexpectedly, in the "after" period, this lift number plunged to below 6 times. In other words, the predictive power is poor (when computed on data not even collected at the time of model-building). Given that no treatment was applied, if the predictive model captured the underlying dynamics, we should expect to see the lift number to be close to 13 times again. (We're assuming no strong known other factor that could cause the accident rates to collapse during the "after" period - in fact, in aggregate, the total number of accidents slightly increased.)

Here is the more subtle error: usually, the "after" period is after some safety measure is applied to the black spots (and not the other sites). The expected result is a drop in accident rate at the black spots, providing evidence that the safety measure actually caused the reduction in accidents. It is therefore very convenient for analysts (and policymakers who hire them) to conclude that the drop from 13.5 times to 6 times is evidence of the effectiveness of the safety measure. What the above analysis shows is that even in the absence of a safety measure, the accident rate could have dropped by half! So if you did apply a safety measure, it would need to produce a massive effect beyond the regression to the mean.

There are two reasons why analysts may fall prey to this type of bad analysis: one is confirmation bias - seeing in the data what you want or expect to see; the other is being what Nassim Taleb calls "fooled by randomness".

***

Adams goes on to discuss "accident migration". This is a second-order phenomenon: sites near to "treated black spots" often see an increase of accidents - the hypothesis is that since drivers take more caution when passing through black spots, they end up being more careless after the black spots are treated and so accidents spike up in nearby areas. (Adams does point out that it is possible that accident migration is not a separate thing from regression to the mean.)

Some related reading is the material on highway congestion in my book, Numbers Rule Your World (link). The problem of road congestion is slippery. If you successfully reduce the amount of traffic along one route, drivers will then change their habits to return that route to congestion - for example, they may have chosen to go along local roads before.

A purely mathematical theory of risk is not enough 1

I have been reading the excellent book by John Adams titled Risk (link). This is a geographer's treatment of a subject that is a staple of mathematics, particularly probability math. A mathematical treatment creates objects called probability distributions, which are then taken as complete representations of risk. Adams challenges that construct, bringing a social scientist's sensitivity to the table. In particular, he points out how the mathematics of risk is undermined by measurement issues (i.e. data issues) and statistical issues. He is not invalidating math, just pointing out large cracks that are often ignored.

I will provide a more comprehensive review of the book eventually. I'm very excited by Chapter 5, titled "Measuring Risk", and specifically the example of "traffic black spots". This example is very instructive for anyone who is interested in the practical implications and interpretation of risk measures.

The post got long so I have split it into two parts. The second part will be posted on Monday, and it concerns a delicious bit of analysis related to traffic black spots.

 

 

 

***

Here are some key sentences related to the quality of the collected data:

1. The connection between potentially fatal events [accidents] and actual fatal events is rendered extremely tenuous by the phenomonen of risk compensation.... avoiding action is taken, with the result that there are insufficient fatal accidents to produce a pattern that can serve as a reliable guide to the effect of specific safety interventions. As a consequence, safety planners seek out other measures of risk, usually - in ascending order of numbers but in decreasing order of severity - they are injury, morbidity, property damage and near misses.
2. It is much easier to achieve "statistical significance" in studies of accident causation if one uses large numbers rather than small. Safety researchers therefore have an understandable preference for non-fatal accident or incident data over fatality data... in exercising this preference, they usually assume that fatalities and non-fatal incidents will be consistent proportions of total casualities.
3. [Evoking some data from London] The correlation between fatality rates and injury rates is very weak. Is the weak correlation real or simply a recording phenomenon? How many injuries equal one life?
4. Uncertainty in the data increases as the severity of the injury decreases. The fatality statistics are almost certainly the most accurate and reliable of the road accident statistics... the categorization and recording of injuries is generally not informed by any evidence from a medical examination.... [A British Medical Asssociation report said that] only one in four casualties classified as seriously injured are, in fact, seriously injured and many of those classified as slightly injured are in fact seriously injured..... some 30 percent of traffic accident casualties seen in hospital are not reported to the police, and that at least 70 percent of cyclist casualties go unreported.

This last point is widely applicable in the data science/analytics world. We often have large amounts of unreliable data, and small amounts of more reliable data. What we hope to have - and we don't - is large amounts of reliable data. A lot of bad analyses results from assuming we have large amounts of reliable data.

 

Good job Bloomberg, and some comments on ad measurement

I'm continuing to see important articles as our journalists shine light on the tech companies. This Bloomberg article reveals a previously secret relationship between Google and Mastercard. It is important also as a reminder that the intense focus on Facebook is misguided - as other big tech companies engage in similar practices.

I'd recommend reading the article in full. Here I'd like to bring out a few key observations for you to chew on.

1) The payment processors have become part of the surveillance network that has been constructed by the tech industry. In the past (when I was working in that industry almost two decades ago), payment processors played the role of toll booths that happened to be a very lucrative business collecting cash as the transactions stream through. Users' purchase data were safeguarded and remained private.

Sometime in the last 5 to 10 years, the payment processors decided that the purchase data is a goldmine. They now monetize this data through mostly secret deals with all kinds of companies. Most card-users are not aware that their purchases are being disclosed to third parties.

An analogy is Gmail and other web-based email systems. Gmail in particular pioneered the concept of reading users' emails to extract information. It is now known that Gmail allows selected partners to extract all kinds of information from emails, such as shopping receipts, boarding passes, etc. We knew this because of the scandal about the Unroll.me service in which the service provider extracted shopping information from emails under the pretense of finding email lists from which users can be unsubscribed from.

 

2) Next time reporters hear the tech companies talk about "anonymized" data or hiding of "personally identifiable information", they really need to press the companies for details. Look out for clever word play. It is simply not possible to get value from these datasets without access to information about each person.

In this instance, Google is merging two datasets, the clickstream (what ads people clicked on) and Mastercard purchases, with the goal of claiming that person A purchased item X after clicking on some Google ad. The Mastercard data contain your card number, and/or your name. The clickstream data contain one or more of the following: your Google "cookie", your Google user name, or your email. You can't merge the datasets unless there is a common field. In this case, it will most likely be your name or email address. How could that be "anonymous"? Good question if you're asking it. And the reporters should be pushing on this point.

Further, if the data are fully anonymized, the potential utility of the data is much lower, and the price marketers are willing to pay for them is much lower. If I don't have your name and address, how can I send you a piece of junk mail?

 

3) While the article points out the violation of privacy of this partnership, it inadvertently legitimizes the highly dubious science behind matching the clickstream data and the Mastercard purchase data. The reporter described the program as "powerful" and "potent" but should have talked to third-party experts to understand this science better. I'll just list several major problems with this approach:

• As far as I know, Mastercard does not have line-item level details on our purchases. It may know that you spent $500 at Best Buy, but it doesn't know that that total includes an iPad and a TV, for example. So if the ad shows a TV, how does Google know that Best Buy purchase included a TV? The reporter claims that Google knows that someone bought "red lipstick" with her Mastercard. I find that dubious.
• Do you remember what ads you clicked on 30 days ago? 15 days ago? 5 days ago? This system counts any purchases made on Mastercard as long as 30 days ago as "caused" by an ad click.
• Does Google know what these users are doing on non-Google websites? For example, does Google know that the user who saw a Google ad 10 days ago also saw a Facebook ad for the same product 3 days ago?
• Does Google know anything about the users' interactions offline? For example, the user who clicked on the Google ad 10 days ago may have heard a radio ad, seen a TV ad, talked to a friend about the product, been handed a flyer at the mall, saw a piece of direct mail, etc.

The reporter suggested that Facebook, the other digital advertising behemoth, is also pursuing similar deals with payment processors. If Facebook were to adopt this same method for measuring ad performance, then both Facebook and Google would have claimed credit for the same purchase, as each would have matched that purchase to its own clickstream data. One solution to this particular issue is to merge the Facebook and Google data, which simply merges two surveillance networks into one extra-large surveillance network.

 

 

What's wrong with the Dropbox research?

In a mailing list I subscribe to, some users were not happy about the academic research using Dropbox data collected on academics, as written up by the co-authors in Harvard Business Review (link).

In a nutshell, the researchers obtained "anonymized" "project-folder-related" data from Dropbox on university-affiliated accounts, and did some simplistic bivariate correlations, and proceeded to draw several conclusions about "best practices" on "successful team collaborations." This type of research is very common in this "Big Data" age, and I have already written extensively about its many challenges.

OCCAM Data

This Dropbox dataset has all five descriptors of the OCCAM characteristics of "BIg Data". It is Observational, seemingly Complete, with no Controls, Adapted from its original use, and Merged (with data from Web of Science). These characteristics cause many problems with the analysis, which I describe below. For more on OCCAM data, check out this post, and my other posts on OCCAM data.

Found Data

Implicit in their analysis - and most other uses of "found data" - is the assumption of complete data. The authors believe that because their data consist of tens of thousands of researchers and 500,000 projects, they must have all the data. In this case, the authors knew that there are other platforms out there but they waved away the inconvenience. This implies they believe they have all of the "informative" data.

The authors also assumed that (a) all relevant collaborative research involves putting all relevant files on one of the major online platforms (e.g. nothing on Slack or emails) and (b) all project collaborators use one and only one platform. Further, they assume that everyone has highly organized and structured folder directories within Dropbox for which an external person who knows nothing about the projects, or a machine, can infer its contents. These problems arise because the researchers did not start with a research question and design the data collection. They chose to adapt "found data" to their own objectives.

Ecological Fallacy, and Story Time

A typical conclusion is "People at higher-performing universities seemed to share work more equally, based on the frequency of instances that collaborators accessed project folders." Top universities are ranked by the aggregate performance of all teams (that use Dropbox, and are identified correctly). It does not follow that every team at a top university is a top team.

This is an instance of "story time." A piece of data is offered about something related, then while the reader is dozing off, it is linked to a conclusion that is not directly supported by the data. That conclusion is elaborated by a lot of words hypothesizing why it must be true. In this case, they say "It’s likely that more frequent collaborations led to positive spillover of information, insights, and team dynamics from one project to another." But they provide no evidence at all for this last statement. That's just a story.

Proxy Unmasking

Here's another one of the conclusions: "People at higher-performing universities seemed to share work more equally, based on the frequency of instances that collaborators accessed project folders." They make a conclusion about work allocation between different collaborators but what they actually measured was the relative frequency at which collaborators accessed the project folders in Dropbox. That's a proxy measure, and it's convenient based on "found data", but not a good proxy.

Xyopia

It does not appear that a multiple regression model was run. The presentation walks through apparently a series of bivariate analyses. The word control does not appear in the entire article. So this work suffers from xyopia - in each analysis, the one explanatory variable being analyzed is presumed to be the chief and only variable that influences the outcome.

Causation Creep, and More Story Time

The authors made no attempt to establish causality at all. They just interpreted every correlation as causal. So every conclusion is "story time". They print one analysis of the data, then they draw a causal conclusion that one would believe only when half-asleep.

***

People are also upset about data privacy.

1. It does not appear that the academic users understood that using Dropbox means they are part of research studies.
2. People don't believe the data are truly anonymized. It's pretty clear that the anonymization can be easily reversed. Just take the HBR article for example. If they removed the names of the authors but retained information about the authors as: one junior faculty at Northwestern Univ. Business School, one senior faculty at Northwestern Univ. Business School, and one employee of Dropbox - I don't think you can find another article that fits those criteria. So is that anonoymous?
3. It's unclear if how they "anonymized folders" or analyzed them. There are folders with highly descriptive names, there are folders with partially descriptive names that only the collaborators may be able to decipher, and there are folders with names that do not identify the project (e.g. old_work). If they converted all folder names to alphanumeric strings, then all information about the contents of the folders is lost. If they don't convert those names, then there are clearly privacy concerns.

It's clear that some kind of IRB review is necessary to approve Big Data research projects to make sure privacy is protected.

Poll fodder inducing xyopia in elevators

Inside elevators in commercial buildings in New York, there are frequently screens that pump out commercials or infomercials all day long. I took a snapshot of this one that showed up last week:

These poll results annoy me in many ways. Let me explain.

The conclusion "free snacks is the key to happiness" is clearly wrong, even if the data were trustworthy (I'll talk about the data in a future post). I call this "xy myopia" or xyopia, in short. The writer of the poll has narrowly framed the problem: in this xyopic world, happiness hinges on just one factor - whether one's employer offers free food. In the real world, the happiness of an employee depends on a host of factors, of which free food is likely to be a minor one.

Xyopia arises from the reasonable concept of ceteris paribus ("all others equal"). Ceteris paribus is of little value in data analysis - it is often implicitly assumed by the analyst, but the assumption is false, which leads to bad conclusions.

In this case, the analyst just assumes that the two groups - employees getting free food, and those not getting free food - are equal on all other dimensions. So not true! The employer who offers free food is more likely to treat employees better across the board, such as providing more vacation days, better benefits, allowing work from home, etc. Besides, free food is only relevant in office settings (think Starbucks baristas). Certain demographic groups may value free food more than others. 

Unless it is established that the respondents who got free food are identical to the respondents who didn't in all other ways, the conclusion that free food is key results because it is the only factor you analyzed. This is xypoia.

(Not to mention, "free snacks" is not the same thing as "free food.")

***

Now, think about what question the poll writer asked that led to the headline "Free snacks is the key to happiness". It appears that they asked a pair of questions: "Does your employer offer free food?" and "Rate your happiness at work". Asking the first question frames your answer to the second question.

I'm going to make up my own poll:

A) Does your employer provide financial support for your personal development?

B) Rate your happiness at work (1 = extremely unhappy, 5 = extremely happy).

Based on this poll, I intend to conclude that paying for personal development is the key to employee happiness.

***

If you extend your vision to the basket of factors that affect employee happiness, such as their pay, benefits, treatment, work-life balance, family life, etc., you'll recognize more problems with the data analysis.

The poll result of 67% versus 56% is a strong result only if the gap is wider than for other factors, such as the poll I created above. I might find that 75% of the employees who get reimbursed for professional development answer extremely or very happy.

Also, consider a poll with three questions:

A) Does your employer provide free food?

B) If answer to A) is yes, do you consume the free food provided by your employer?

C) Rate your happiness at work (1 = extremely unhappy, 5 = extremely happy).

This creates three subsets of respondents (No - NA; Yes - No; and Yes - Yes) and leads to a better understanding of the causal mechanism.

***

Polls are easy to do but hard to do well.