Big Data, Plainly Spoken (aka Numbers Rule Your World)

Title IX, causal claims, and plausibility in statistics

Andrew Gelman discusses a paper and blog post by Ian Ayres on the Freakonomics blog. Their main result is summarized as:

We find that a ten percentage-point increase in state-level female sports participation generates a five to six percentage-point rise in the rate of female secularism, a five percentage-point increase in the proportion of women who are mothers, and a six percentage-point rise in the proportion of mothers who, at the time that they are interviewed, are single mothers.

Andrew finds these claims implausible, so do I.

Ayres uses the econometrics methodology called instrumental variables regression to support these claims. Since the data is observational, and as Andrew pointed out, there wasn't even a period of time in which one could find exposed and unexposed populations (since the TItle IX regulation was federal), one must treat such regression results with a heavy dose of skepticism.

It is useful to understand that causal claims are possible here only if we accept all the assumptions of the instrumental variables method.

Besides, plausibility is assisted by the ability to outline the causal pathways. It should be obvious that more females competing in college sports does not directly cause more females to become secular. The data on sports competition and on secularism come from different sources and this presents a hairy problem. The analysis would have been more convincing if it found that among the women who participated in college sports, more became secular; what the analysis linked was higher participation rate and higher secularism among all women in the state.

What is it about sports participation that would cause people to become secular? (The visual evidence from professional American sports would lead me to hypothesize the opposite--that sports participation may be associated with higher religosity!) Is this specific to the female gender? Do we find male secularism increase as sports participation by men went up?

As Andrew pointed out, the magnitude of the estimated effect seems too large to believe. I'd prefer to see these effects reported at more realistic increments. A jump of 10% participation is very drastic. For example, according to the chart here (the one titled "a dramatic, 40-year rise"), the percent of women participating in high school sports has moved just 2 percent from 1995 to 2011.

***

Andrew is right that this is an instance of "story time". And we are not saying that statisticians should not tell stories. Story-telling is one of our responsibilities. What we want to see is a clear delineation of what is data-driven and what is theory (i.e., assumptions). The plausibility of a claim depends on the strength of the data, plus whether we believe the parts of the theory that are assumed.

***
See how Andrew articulates these--and other--concerns on his post.

Upcoming Meetings

(This post is cross-posted on my two blogs.)

Posting will be light this week, as I prepare for a number of meetings. Please come find me if you are in the neighborhood.

On May 20 (this Tuesday), I am the Banquet Speaker at the Midwest Biopharmaceutical Statistics Workshop (MBSW), to be held in Munice, Indiana on the Ball State campus. More information on this event here. I will be talking about how Big Data is affecting us, and the need for Numbersense. The key lessons will come from Google Flu Trends and digital marketing attribution models.

Next week, on May 28, I will be in Toronto at the Statistical Society of Canada's Annual Meeting. I am a participant in an invited session on "Business Analytics: Where does the statistician fit?" My article in Significance is a good starting point. More information about SSC here.

On June 18, I am giving a Keynote Speech at Predictive Analytics World, in Chicago. You can register here. The agenda looks great, and you will learn a lot about all kinds of applications across many industries at this conference.

 

The trouble with some journal-published papers

In the popular science genre, one often comes across "published in a peer-reviewed journal" as a certificate of authenticity. Given that the authors of such reports or books typically do not have the technical chops to understand the materials deeply, it's not a surprise that they require third-party validation. However, "published in a peer-reviewed journal" is pretty weak.

I just read a paper published in a peer reviewed journal that made me cringe. This is not any journal but the California Management Review, which according to Wikipedia, is "along with other publications such as Harvard Business Review and MIT Sloan Management Review, among the most influential and viable sources of contemporary business research". It has an impact factor of 1.667.

The paper is called "Organizational Blueprints for Success in High-Tech Startups: Lessons from the Stanford Project on Emerging Companies" by James Baron and Michael Hannan (pdf here). This paper was referenced in the New York Times Upshot article titled "Yes, Silicon Valley, Sometimes You need More Bureauracy", and my friend Xan G. was unhappy with the way the reporter presented the findings of the paper. I am not convinced by the original paper either.

***

First, let's talk about Xan's complaint. This sentence neatly summarizes the reporter's point of view:

Yet a human resource department is essential. The two [researchers] found that companies with bureaucratic personnel departments were nearly 40 percent less likely to fail than the norm, and nearly 40 percent more likely to go public — data that would strike many Silicon Valley entrepreneurs as heresy.

Going back to the source, we find the following (type of) chart from which the reporter extracted the evidence:

This chart shows the six "organizational blueprints" of which the Engineering blueprint is treated as the reference level. Only the Autocratic blueprint has higher likelihood of failure than Engineering. Four out of five non-Engineering blueprints did better than Engineering, and among those four, Bureaucratic was the worst! So how can the reporter conclude that this chart supports more bureaucracy (which is defined as having a human resource department) for Silicon Valley? Why not go for Commitment blueprint and have a 100% lower rate of failure instead of 40%?

Despite the name "Bureaucratic", the researchers never said that having an HR department is a defining characteristic of this blueprint. Having HR departments is compatible with several of the other blueprints. In fact, on page 14, the researchers stated: "Commitment and Star firms tended to be the fastest to bring in HR expertise."

What's more, the reporter described the Engineering blueprint as "the norm", which is not a term that the researchers used. The researchers used the word "modal" (page 11) which means the most frequently used but this usage is contradicted by Figure 3, in which the largest piece of the pie chart is the "Aberrant" type (which probably maps to "Non-type") 33% against 31% Engineering.

***

Next, the original paper in CRM is an instance of "story time". The data they have collected is nice but very limited; they really stretched in making their stories way beyond what their analysis could support. Most of the conclusions I'd consider based on theory rather than evidence. Besides, in simplifying the technical content to suit CRM's target audience, so much is lost that what remains is impossible to interpret.

Just look at Figure 6 above. The reader might guess that there is some kind of regression model being run with likelihood of failure as the response variable and the organizational blueprint as a predictor. The Engineering blueprint is selected as the reference level and given a value of 0%. What does it mean by the Commitment blueprint being 100% less likely to fail? Does this mean no companies using Commitment have ever failed?

If you'd like to know what the actual failure rates are, you will not find them within the 26-page article. If Engineering failed at a one-percent rate, then Bureaucratic failed at 0.6 percent, hardly a concern. You'd note that there is no indication of standard errors, error bars, or sample size. (They referenced a more technical paper in the footnote but I didn't see actual failure rates there either.)

***

There are other problems with the CRM article. The authors said they have a sample of "nearly 200 technology start-ups" but made no mention of how these start-ups were selected. In the entire paper, not a single company's name was mentioned.

Knowing how the sample is selected matters a lot here. There may be survivorship bias, for example, in that companies which failed fast are not in the sample.

Much is made about firms that shifted their organizational blueprint as it aged. A Founder's blueprint is contrasted with a CEO's blueprint. On page 15, we are told that "only 18 of the 165 firms in Figure 4 changed from one pure model type to another; of these, 14 moved between Engineering and Bureaucracy, the two closest pure type models". So somewhere along the way, we lost 35 firms in the sample without explanation.

Then on page 16, the researchers said: "One obvious question to ask is: Do changes in HR blueprints accompany changes in senior management within startups? The answer is yes." Later, on page 21, they asserted "we found compelling evidence that changing the HR model is destabilizing to high-tech start-ups".

The level of confidence in those statements is at odds with the sample of 18 firms that changed the blueprints, of which 14 moved between the two closest types.

Let's do a quick calculation. With 6 basic types, there are 30 possible A->B shifts, in which B->A is regarded as different from A->B. There were only 18 observed shifts, 14 of which ended up in the same A->B pair so there can at most be five unique shifts in the data. And yet they are able to draw conclusions about changes in HR blueprints in general?

***

The subject of the research is worth investigating. I can't help but think that the research would have been better without the pretense of being data-driven. Most if not all the conclusions are mostly supported by interviews, and not so much by the data anyway.

I am annoyed with the presentation of a series of charts that have no meaning--all relative values without disclosing the reference level; the presentation of statistical results without mentioning sample sizes or error bars; and the presentation of results from manually gathered data without even naming one company and describing the sampling methodology.

 

 

 

 

 

 

 

 

The mystery of the official inflation rate

In case you are not subscribed to my dataviz feed, I put up a post yesterday that is highly relevant to readers here interested in statistical topics. The post discusses a graphic of a New York Times article that interprets the official inflation rate (known as the CPI). I devoted an entire chapter of Numbersense (link)to the question of why the official inflation rate diverges from our everyday experience.

In a larger context, inflation rate is an invented metric, invented to measure some quantity that has no objective reality. This is true of a lot of statistics. Revenues and profits are also invented concepts, for example, and only attain meaning through generally accepted accounting rules. Obesity, which is discussed in Chapter 2 of Numbersense (link) is another example of a quantity that has meaning only because of a convention of measuring.

The article in NYT brings up one of the points I raised in the book, which is that price increases are magnified in our imagination while price decreases are taken for granted.

The other larger point of the chapter on inflation is that anyone wishing to comment on whether CPI reflects real experience ought to understand how CPI is constructed. A superficial understanding such as that it is the average price of a basket of goods is useless because there are so many little details that affect the statistic. Because inflation has no objective basis, it is pointless to argue if it reflects reality: all we are left with is discussing the rules and you can't discuss the rules without knowing them well.

Details matter a lot in statistics. This is one of the reasons why I keep asking my Big Data colleagues to talk specifics. A statistician who only talks in generality is like the Manhattan realtor who can't tell you the size of the listed apartment.

 

 

Navel-gazing at airfare data

It seems like Seth Kugel's article in the New York Times about "Crunching the Numbers to find the Best Airfare" is quite popular. In this article, he said things like this:

The overall take on the best day to book tickets turns out to be somewhat underwhelming, if you look at the country as a whole. Hopper’s data shows it’s actually Thursday, but don’t expect that fact to save you much money. Reserve a domestic flight on Thursday and you’ll spend, on average, $10 less than if you reserve on Saturday, the worst day to book domestic flights. With international flights, you’ll save, on average, $25 over Sunday, the worst day to book flights abroad. (Those are “maximum averages” that assume you would have booked on the worst day and are now booking on the best.)

This is meaningless navel-gazing.

As I explained in my notes to my Kayak article on 538, talking about best or worst fares is meaningless unless one can describe a strategy with which the traveler could attain those fares. This strategy must work in real time, before it is known that a particular fare would be the best or the worst.

Without such a strategy, we are talking about paper gains and losses.

Analysts who follow Kugel's logic though rarely realize that they are talking about paper money. So, later in the article, Kugel said this:

For the vast majority of routes,... avoid booking on weekends and try midweek; for the average American flier, those savings will add up in the long run.

What savings? Those would appear to be the "maximum averages" defined above, the difference between the best and the worst days for given routes based on a lot of historical data. But there is no strategy to reliably attain the best fare; in fact, there is no strategy to reliably buy the worst-priced tickets either.

As I said before, if the goal is to gain provable savings, you need to write down how you are making the purchase decision today, then you need to define what your new strategy is--whether it is using Kayak, or using Google (which doesn't do predictions)--and then you should compare the two methods.

 

 

Beyond the surface of poll numbers

This Upshot piece about increasing prevalence of tattoos in the U.S. linked to a Fox News Poll, summarized here by Fox News reporters.

The statistics say that 20% of voters haves at least one tattoo, up from 13% in 2007. Fourteen percent haves two or more tattoos, up from 8%.

The problem with poll result reporting are many. Let me count them:

How should we interpret those numbers? One thing not discussed is the increasing level of acceptance of what used to be a social taboo. This is happening at the same time that more people are getting more tattoos. A serious understanding has to start with how much more willing is the tattooed person to disclose their inked status today compared to years past.

A second related question is whether people with tattoos are less likely to be voters. If you compare the three sources, you'll notice an increasing imprecision in the reporting. The pollsters used the term "registered voters"; the Fox News writer turned that into "voters"; and finally, the Upshot writer replaced it with "Americans". This is probably what most readers would do anyway. For this poll question, there is reason to doubt whether people with tattoos are registered to vote in equal proportion to the population at large.

Then, as with any poll, you wonder whether a subgroup of the respondents are paying attention or just making things up. Fox News told us "Ten percent of tattooed Americans say they don’t like tattoos." This could be buyer's remorse, or it could be people not taking the poll seriously.

It is often pointless to ask questions like these: "By a 73-16 percent margin, most voters say they would hire someone who has a visible tattoo." In essence, the pollster is asking someone if they would discriminate against people with visible tattoos. It's like asking someone "would you not hire someone if they are black?" You're not likely to get an accurate answer.

The sample size is roughly 1000 people, and this was designed to achieve a margin of error of plus/minus 3 percent according to the press release. Blink and you may miss the qualifier that said "results based on the full sample". When you start to look at subgroups within the full sample, you get into trouble.

Credit to the pollsters, they disclosed this information. For example, the Fox News report on the Fox News poll said "A third of those under age 30 have a tattoo (34 percent)". The margin of error on this number is about 7.5 percent which means that there was only sufficient samples to nail the estimate down to a range of 15 percent.

***

Polls are everywhere but good reporting on polls is rare.

 

An anthropological study shows why complete data designs are counterproductive

I saw Joe N.'s tweet asking me about a study of how professors spend their time, reported by Lisa Wade at Sociological Images. This is an anthropological study, something that I am not at all familiar with although the people in the field seem to believe that they can make statistically valid observations.

I'm glad the author of the study, John Ziker, wrote a (really) long article describing what he was trying to accomplish. The key point is that the study is a preliminary exploration, with important limitations; a follow-up study is planned which may give generalizable conclusions.

Here are some issues with the first study that makes a statistician nervous:

- the sample was between 14 and 30 professors (tiny): Wade reported it to be 16. Ziker definitely started with 30.

- the selection was non-random, based on the first 30 people who responded to a school-wide announcement

- about half the initial respondents did not complete the study, and provided only partial data (one to six days)

- despite the tiny sample, some analysis required slicing the data further into four segments by grade level! I wonder how many department chairs were in that sample. (See chart on right)

- each professor is followed for a two-week period but only every other day, thus each professor at most contributed one observation per day of week

- the interviews were every other day "so the time taken for the interview did not appear on the previous day’s report." This is a horrible problem to deal with! Because time allocation is the subject of the study, the measurement method (in-depth interviewing) interferes with the measured outcome. It seems to me impossible to believe that the time spent answering questions every other day did not affect time allocation on the non-interview days.

- Ziker reasoned: "While we cannot make a claim that all faculty have the same work patterns as our initial subject pool — they do not comprise a random sample — the results are highly suggestive because of the consistency across our subjects who did represent.". In order not to fall prey to the law of small numbers, a better way to say this is: we make the assumption that the small sample is representative on both mean value and dispersion, which then leads to the assumption that all faculty have consistent work patterns similar to the observed.

- "With our initial 30 Homo academicus subjects, we ended up with a 166-day sample with each day of the week well represented." I am assuming that Ziker did not drop the 16 professors with partial data and made charts like the one on the right by ignoring the identity of the professor and aggregating over days of the week. Let's review what lies behind this chart. Each respondent contributed at most one observation per day of week; about half of the respondents did not even contribute data for all seven days. So the time allocation on any particular day is averaged over anywhere from 14 to 30 professors. These professors span a variety of ranks, departments, tenure, backgrounds, etc. and were not randomly selected. It's hard for me to trust this chart at all.

***

In general, I am a big fan of shoe leather research in which the researcher goes out there and gather the relevant data they need to address their specific research question, rather than picking up what data they could find, and then tailoring the research question to avoid the imperfection in the data. So I don't want to sound too negative. It's a difficult research problem they are dealing with. What they learned from this first study is useful to inform future explorations but drawing conclusions at this stage is premature.

At the end of his article, Ziker described the "experience sampling" method that will form the next phase of this study. I am very excited about this methodology.

Roughly speaking, they will ask participants to install a mobile app, which pops questions from time to time asking them what they are doing at that moment. Instead of exhaustively tracking a small number of participants over the course of time, they will get little bits of data, incomplete schedules, for a large number of professors. If the sample is big enough and randomized appropriately, they can analyze the data ignoring the professor identity, and report results for the "average professor". This method also retains the other benefit of the original design, which is that the respondents report their activities close to the time in which they occurred.

Data scientists pay attention! You don't have to collect complete data at the user level to do proper research. Designs like this "experience sampling" approach produce statistically valid findings without the need for complete data. In fact, trying to collect complete data is counterproductive, leading to shaky conclusions as shown above.

 

Take back the GPA

Princeton - apparently due to pressure from parents - is on the verge of losing its nerve in the battle against grade inflation (link). It is the most prestigious college to have instituted an A-grade quota. By contrast, 62 percent of all grades given at Yale are As and at Harvard, the most common grade given is A. At Stanford, 93 percent of grades were As or Bs, and that was 20 years ago(!) and then it decided to hide its grade distribution.

Presented without comment the nation-wide grade trends:

More info here. In a generation or two, colleges will stop awarding grades since they don't mean anything. This is another instance of the inevitable perversion of measurement.