Big Data, Plainly Spoken (aka Numbers Rule Your World)

Some statistics about nutrition statistics

I only read nutrition studies in the service of this blog but otherwise, I don't trust them or care. Nevertheless, the health beat of most media outlets is obsessed with printing the latest research on coffee or eggs or fats or alcohol or what have you.

Now, the estimable John Ioannidis has published an editorial in BMJ titled "Implausible Results in Human Nutrition Research". John previously told us about the crisis of false positives in medical research.

Oops, here are some statistics on nuitrition "science":

• In 52 attempts at using randomized experiments to validate findings from observational studies, the number of times the findings were replicated: 0
• In the NHANES questionnaire (the basis of all those findings), two-thirds of the participants provided answers that imply an energy intake that is "incompatible with life". I haven't read this paper; seems like worthwhile reading.
• There are at least 34,000 papers on PubMed with keywords "coffee OR caffeine" which means this one nutrient has been associated with almost any interesting outcome.
• Almost every single nutrient imaginable has peer reviewed publications associating it with almost any outcome. A statistician should never give the advice "If at first you don't succeed,..."
• Many findings are entirely implausible (and still get published in top journals)... for example, the idea that a couple of servings a day of a single nutrient will halve the burden of cancer is clearly "too good to be true," even more so for anyone who is familiar with this literature
• "Big datasets just confer spurious precision status to noise"
• Randomized experiments offer hope but are woefully undersized (like requiring 10 times the current sample).

Just to nail home the point, John concludes: "Definitive solutions will not come from another million observational papers or a few small randomized trials."

 

Should I tell students that the maximum score in the class is 137?

This op-ed by Richard Thaler caught my attention because I have a similar experience.

In my statistics classes, I have noticed a pattern: if the mid-term exam is hard, with a lower average score (say 75-80%), the students look crestfallen and feel that they did not learn; eventually, when it comes to evaluating the instructor, I receive lower grades, with comments indicating that I have not taught them properly to do well in exams.

When the mid-term exam is easier, I get more positive feedback. (This happens even if the two cohorts get similar inflated number of As.)

To solve this problem, Thaler tells his students that in his class, the top score is not 100 but 137. Now, the average student gets around 90 points or so, and he reports that the student feels happier even though on a relative scale, he or she didn't do better than others in prior classes!

***

I am considering trying this out next term although I'm not sure it works.

In the case of my students, the source of anxiety, I think, is self-doubt. Many of my exam questions are open-ended. I emphasize this early and often because real-life data problems do not have simple solutions that involve applying some textbook formula. I also tell students that there are many acceptable solutions to these questions.

For example, instead of "Use this formula to deal with the miscoded data in column X", I'd ask: "There is a problem in column X. Identify it and show me how you'd solve the issue."

I think the trouble is the students (despite being in a statistics class) want the certainty of knowing that they got the points. They may not yet have developed the conviction in their own solution strategies. They feel anxious because of that uncertainty. They want to know what the answer key is, even after I emphasize that different answers, even contradictionary, ones may be accepted (not if they come from the same student though).

***

Another challenge of implementing Thaler's idea is what should be the maximum score. The average score of the class depends on who the students are and how hard the exam is, and I don't have the exams set before the syllabus is drawn up.

***

A third point, which merits a separate post, is the discomfort with "manipulating emotions". Recall that some Facebook researchers were accused of doing so for running a randomized experiment (link). Here, Thaler is presenting a method to manipulate emotions as a celebration of behavioral economics.

Should student self-worth that is tied to getting As and 90% scores (no matter the actual worthiness) be a paramount objective for educators? Right now, as Thaler points out, this is the case because administrators use course evaluations as the sole source of information about an instructor's ability. This loops back to the previous post: metrics cause perverse behavior; also, the need to collect small data, in this case, classroom observations, and interviews with instructors.

***
Happy Memorial Day weekend for those in the States.

 

 

Five Not-to-be-Missed Ideas about Big Data

If I believed in upvoting, I would be clicking my fingers bent on this op-ed article that appeared in the New York Times at the beginning of May. It's written by "data scientists" at Facebook and Google. (Thanks Dean Eckles for drawing my attention to it!)

Here are a list of important ideas not to be missed:

• The things we can measure are never exactly what we care about
• When everything hinges on metrics, people will game the metrics to the point of losing any meaning
• Big data in the form of behaviors and small data in the form of surveys complement each other
• Data teams should hire social psychologists, anthropologists and sociologists
• Human experts help data experts figure out what they are missing

 

Here are some old posts on the blog that dig into those topics, in chronological order:

Numbersense in education (link): gaming statistics, subjectivity in metrics, accountability and metrics

Toward a more useful definition of Big Data (link): limitations of Big Data

Big Delusion (link): the idea that survey research is killed by Big Data

An Anthropological study shows why complete data designs are counterproductive (link): data is incomplete, shoe-leather research

Two unsolved problems of Big Data studies: confirmation and controls (link): limits of data, need for human investigation, incomplete answers

The false promise of data correction (link): errors in data, gaming of data

Pondering OCCAM data in medicine (link): thinking about what is missing in the data

Count data are less useful than you think (link): limits of metrics

Reflecting on Alamar's reflection on sports data (link): too much data, subjective metrics

KD Nuggets interviews me (link, link): data integrity, human intelligence, statistical reasoning

Some data science principles from Gelman, Rosling and me (link): data not enough, there is still uncertainty

Sense and Nonsense about Big Data and Surveys (link): surveys still important

 

PS. [5/19/2015] oops, I forgot to link to Dean's list of related links! See here for even more articles about these ideas.

Talking past the consumer on customer data

I mentioned the Harvard Business Review article on business use of customer data in the "Big Data" era. In the previous post, I looked at the nature of the evidence used by the authors. In this post, ignoring my discomfort with some of the evidence, I examine the conclusions of the article.

***

The report has a three-part structure: the first section describes the issues; the second section communicates results from a few surveys conducted by frog - a global strategy and design agency - on various issues related to data privacy; and the third section presents examples of their recommendations for clients, which they offer generally to businesses involved in collecting and monetizing customer data.

The survey results are revealing (although the sample size of 900 in five countries is tiny so I'm not sure you should believe them). The agency found that 97% of the people surveyed are concerned about businesses and governments mis-using their data. Seventy-two percent of Americans are reluctant to share information with businesses because they "just want to maintain their privacy".

The authors also learned that consumers have grossly under-estimated the extent of data collection. Only 25% of the respondents said they knew businesses tracked their location, and only 14% said they knew businesses shared their web-surfing history. Finally, their analysts attributed dollar value to the privacy of different types of data.

I follow them up to this point. In fact, the authors summed it up very nicely at the beginning of the article: most [companies] prefer to keep consumers in the dark, choose control over sharing, and ask for forgiveness rather than permission.

Unfortunately, I am let down by the list of recommendations that follow. They feel to me like tweaks on failed ideas, rather than paradigm shifts.

***

The first recommendation is "educate the consumers". The authors gave an example of one of their own consulting clients who required "customers" to watch a video and give preliminary consent before sharing their own (genomic) data. And the personal data is withheld until the "customer" returns a hard-copy agreement. 

We don't need to be reminded that every day, we "voluntarily" sign Terms and Conditions which no ordinary person actually reads. Frequently, we are told not to use a website if we don't agree with any part of a lengthy agreement written in one-sided language favoring the business.

The "new" solution doesn't change the status quo. In fact, it gives businesses a stronger case for arguing that their users have voluntarily given up the right to their own data. In my view, until businesses confront the issue of properly disclosing how they collect data, what information is being collected, and how such data are being sold or traded, consumers will continue to find such practices creepy.

***

The second recommendation looks good on paper but is impractical. Another one of frog's client is featured here. This client allows customers to specify which pieces of data can go to whom.

Assume there are 100 variables (only!) being collected and five levels of access control. That amounts to 500 yes/no questions each user is required to answer in order to gain full control of the data. In practice, most users will decide not to bother because it is too complex and time-consuming. The solution is a form of suffocation by paperwork.

For the data analysts, such a solution creates headaches. It generates self-selected data of the worst kind. Each variable has its own source of bias as different subsets of users decide to withhold their data for their own reasons.

To implement such a system properly requires a herculean effort. Say I reviewed the list of 100 variables and divided them into five groups of 20 variables using the five levels of control (from allowing anyone to see my gender to hiding my age from everyone). Two months later, I changed my mind. I removed access to 80 of the 100 variables from everyone. Now, the database administrator should find all instances of those 60 variables and delete them. Some of the data may already have been sold to other entities, and what if those other entities re-sell my data after I asked for the data to be deleted by the original source?

***

The last recommendation is an argument that businesses should not need to pay users for their data. Given the finding in the second section that users assign meaningful dollar values to their data, this seems to be a solution for businesses rather than for consumers. 

Pandora's free advertising-supported service is used as an example of customers' willingness to exchange their privacy for "in-kind value". The article failed to mention just how much money Pandora has been paying for such data! As this other HBR article tells us, Pandora is "13 years, 175 million users, little profit". It has never been able to establish a profitable business model because while 80% of its revenues come from advertising to those "free" accounts, 60% of its revenues immediately goes out the door as royalty payments for the "free" music! It's not surprising that many consumers are willingly engaging in this lop-sided exchange with Pandora.

***

I often wonder if consumers realize that over-sharing their data works to their disadvantage, would they become more interested in how businesses use their data?

For instance, insurance companies will be very interested in acquiring data from personal analytics devices, like Fitbit. They will use the data to predict whether you have health risks, and they will charge you more for insurance. Everyone is at risk for something.

The Uber app gives its users the ability to track their drivers -- in Manhattan, it's like watching a horse-race when your driver tries to negotiate the city gridlock. The same data is used by Uber to get an accurate picture of supply and demand, which drives their surge-pricing algorithms. That's how you end up paying five to ten times the normal cab rate.

Businesses use personal data to reduce information asymmetry, which in the past prevented them from extracting maximum value from consumers.

Today, the data privacy question is phrased as "Company X would like to collect information about your heart rate and in exchange, you will get notified if any irregularity is detected. Are you willing to share such data with Company X?"

Imagine you are asked a different question: "Company X would like to collect information about your heart rate and in exchange, you will get notified if any irregularity is detected. Being notified of heart-rate irregularity may help you but 80% of the warnings will be false alarms. Also, your heart rate data will be used by our insurance arm to adjust your insurance premiums. There is a 50% chance that your premium will increase after sharing your data. Are you willing to share such data with Company X?"

Deflategate 3: nature of evidence

Last time we heard about Deflategate on this blog, Warren Sharp compiled some statistics on fumble rates, showing that the Patriots were unusually good at avoiding fumbles. (link, link) I thought the level of analysis was "above average" and remarked that statistical evidence of this type can only get you so far. The metric is indirect, and it does not speak to causation.

The official investigators have now issued their report. New York Times has its coverage here. As one reader commented, this article, currently nearing 800 comments, has more comments than most articles with more serious subject matter. The NYT article is one of the better ones out there on this subject.

Two set of new evidence has emerged.

The first, which is getting most of the headlines and attention, are text messages involving two Patriots employees who discussed their deflating operation. These text messages are highly incriminating for the two involved and for me, also incriminating for Tom Brady, the team's superstar quarterback (who refused to release his own text messages or other correspondence to the investigators). The text messaging evidence shores up the causal evidence in a way that numbers by themselves could never accomplish.

The takeaway from the text evidence is the power of "metadata". Metadata is data about the text messages (sender, recipient, date and time of sending, length, etc.), as distinct from the content of the texts. Metadata went mainstream when the U.S. government was revealed to have been massively scooping up metadata on domestic phone calls, but denied collecting contents of said phone calls (See these coverage, for example). The investigators can use metadata to learn who else is in the circle of insiders, how often they communciate, when they communicate, etc. Notice that these pertinent questions do not require knowing the contents of the text themselves. (This is not to say knowing the contents of at least some of the text messages is important--at the minimum, to zoom in on the relevant texts.)

But these investigators could not determine when the deflator operation started, how often it occurred or the full scope of the operation. This is likely to do with selective disclosure of the text messages by selected parties (e.g. none from Brady).

Another takeaway is the inherent bias in surveillance data. Simply put, you only know what you can measure, and there is much that are not being measured. To get the "full scope", the investigators would need phone records, emails, and even wiretap evidence following the key players around (just kdding).

***

The second set of evidence is also extremely important to the story but it has received far less attention. One reason I like the NYT coverage is that the reporter gets to this evidence before talking about the text messages. For the first time, I see direct evidence of football tampering. The NFL rule requires footballs to be inflated to between 12.5 and 13.5 pounds per square inch. According to the NYT report, after the Colts raised suspicion at half-time of the Patriots-Colts matchup, all of the footballs were found to be underinflated (below 12.5 pounds), with the minimum vaule of 10.5.

This is the first time I see a clear admission that all of the footballs were underinflated. This is much more convincing evidence that someone tampered with the footballs than any of the fumble analysis.

Further, the referee had already weighed the balls before the game, and at the time, found all of the Colts-supplied footballs to be about 13 pounds, and only two of the Patriots-supplied footballs to be under-inflated.

Once tampering is established, the investigators can move on to finding the cause. Here, they are helped by videotapes from surveillance cameras, and also the texts.

***

One nitpick about the sentence: 'The report uses the nebulous phrase “more probable than not” several times in making its conclusions.' To a statistician, this is a very precise statement, not nebulous at all! I interpret the investigators to mean there is more than 50% chance. That is the standard of "preponderance of evidence."

***

FiveThirtyEight has a lengthy discussion of the report. They helpfully showed a screenshot of the measured ball weights:

 

 

 

 

 

Story time, known unknowns and the endowment effect in an HBR article on customer data

Harvard Business Review devotes a long article to customer data privacy in the May issue (link). The article raises important issues, such as the low degree of knowledge about what data are being collected and traded, the value people place on their data privacy, and so on. In a separate post, I will discuss why I don't think the recommendations issued by the authors will resolve the issues they raised. In this post, I focus my comments on an instance of "story time", some questions about the underlying survey, and thoughts about the endowment effect.

***

Much of the power of this article come from its reliance on survey data. The main survey used here is one conducted in 2014 by frog, the "global product strategy and design agency" that employs the authors. They "surveyed 900 people in five countries -- the United States, the United Kingdom, Germany, China, and India -- whose demographic mix represented the general online population". (At other points in the article, the authors reference different surveys although no other survey was explicitly described other than this one.)

Story time is the moment in a report on data analysis when the author deftly moves from reporting a finding of data to the telling of stories based on assumptions that do not come from the data. Some degree of story-telling is required in any data analysis so readers must be alert to when "story time" begins. Conclusions based on data carry different weight from stories based on assumptions. In the HBR article, story time is called below the large graphic titled "Putting a Price on Data".

The graphic presented the authors' computation of how much people in the five nations value their privacy. They remarked that the valuations have very high variance. Then they said:

We don't believe this spectrum represents a "maturity model," in which attitudes in a country predictably shift in a given direction over time (say, from less privacy conscious to more). Rather, our findings reflect fundamental dissimilarities among cultures. The cultures of India and China, for example, are considered more hierarchical and collectivist, while Germany, the United States and the United Kingdom are more individualistic, which may account for their citizens' stronger feelings about personal information.

Their theory that there are cultural causes for differential valuation may or may not be right. The maturity model may or may not be right. Their survey data do not suggest that there is a cultural basis for the observed gap. This is classic "story time."

***

I wonder if the HBR editors reviewed the full survey results. As a statistician, I think the authors did not disclose enough details about how their survey was conducted. There are lots of known unknowns: we don't know the margins of error on anything, we don't know the statistical significance on anything, we don't know whether the survey was online or not, we don't know how most of the questions were phrased, and we don't know how respondents were selected.

What we do know about the survey raises questions. Nine hundred respondents spread out over five countries is a tiny poll. Gallup surveys 1,000 people in the U.S. alone. If the 900 were spread evenly across the five countries, their survey has fewer than 200 respondents per country. A rough calculation gives a margin of error of at least plus/minus 7 percent. If the sample is proportional to population size, then the margin of error for a smaller country like the U.K. will be even wider.

The authors also claim that their sample is representative of the "demographic mix" of the "general online population." This is hard to believe since they have no one from South America, Africa, Middle East, Australia, etc.

The graphic referenced above, "Putting a Price on Data," supposedly gives a dollar amount for the value of different types of data. Here is the top of the chart to give you an idea.

The article said "To see how much consumers valued their data, we did conjoint analysis to determine what amount survey participants would be willing to pay to protect different types of information." Maybe my readers can help me understand how conjoint analysis is utilized for this problem.

A typical usage of conjoint is for pricing new products. The product is decomposed into attributes so for example, the Apple Watch may be thought of as a bundle of fashion, thickness, accuracy of reported time, etc. Different watch prototypes are created based on bundling different amounts of those attributes. Then people are asked how much they are willing to pay for different prototypes. The goal is to put a value on the composite product, not the individual attributes.

***

Also interesting is the possibility of an "endowment effect" in the analysis of the value of privacy. We'd really need to know the exact questions that the survey respondents were asked to be sure. It seems like people were asked how much they would pay to protect their data, i.e. to acquire privacy. In this setting, you don't have privacy and you have to buy it. A different way of assessing the same issue is to ask how much money would you accept to sell your data. That is, you own your privacy to start with. The behavioral psychologist Dan Kahneman and his associates pioneered research that shows the value obtained by those two methods are frequently wide apart!

In a classic paper (1990), Kahneman et. al. told one group of people that they have been gifted a mug, and asked them how much money they would accept in exchange for it (the median was about $7.) Another group of people were asked how much they were willing to pay to acquire a mug; the median was below $3.

Is this the reason why businesses keep telling the press we don't have privacy and we have to buy it? As opposed to we have privacy and we can sell it at the right price?

***

Despite my reservations, the HBR piece is well worth your time. It raises many issues about data collection that you should be paying attention to. Read the whole article here.