Big Data, Plainly Spoken (aka Numbers Rule Your World)

Numbersense in education: gaming the statistics, cheating scandals, and more

My friend Kate alerted me to the notable New York Times story on academic fraud at the University of North Carolina (Chapel Hill). Phantom courses have been created to provide students with A grades, in some cases, for the benefit of athletes. This story fits the larger pattern of fraudulent practices across the education sector, which is the subject of Chapter 1 of Numbersense (link).

The story about law school admissions in Chapter 1 has many parallels with this story. Falsifying grades to make them look better is a common theme. Law schools do it for their US News ranking; universities with big-time sports programs do it for their NCAA eligibility and on-the-field competitiveness. For those who don't follow US college sports, it is helpful to know that the NCAA (governing body) sets minimum average GPA for "scholar"-athletes (see this story for example). In addition, the NCAA created the "Academic Progress Rate" (Wikipedia) which tracks graduation rates and uses this metric to control the number of scholarships that can be given to athletes. Scholarships is a crucial tool for recruiting the best athletes from high schools. In other words, the NCAA practices a form of performance-based management.

Another parallel with the law schools story is that the college administrators do not see the problem of fraud as cultural or ethical, instead turning to their PR staff for damage control. It is laughable to think that the entire fraud is known only to two already-retired employees, as the reporter writes with amusement, just as it was laughable to think that the head of admissions single-handedly faked all kinds of admissions statistics at those law schools.

Those who read Chapter 1 (and 2) of Numbersense (link) know that all metrics have subjective components and that one should never manage by metrics alone. Once rules are set to compute a metric, the metric will be immediately gamed, and the gap between the measured and real outcomes will continue to grow. The mistake isn't in using metrics but in treating them as truth, and not being vigilant about gaming. If severe punishment were to meet the cheaters, one would suspect the amount of cheating would be reduced.

One of the great laments in the education sector in the last several decades is the adoption of the "corporation" model by many great universities in the U.S., and at the schools level, through the ill-conceived No Child Left Behind legislation. This means rule by quantitative metrics, "performance"-based management, the acceptance of systemic inequity, plausible deniability by the executive class, treating education as an "investment" and students as "customers", etc.

A fundamental problem with this policy is the difficulty of measuring "performance" in education. When you have a half-accurate metric, and you treat it as truth, you are in a world of trouble.

***

Let's talk about accountability and punishment. In each of these cheating scandals, administrators and/or professors were at fault for manipulating grades and other statistics. Eventually, one or two "bad apples" were identified and dismissed while the "executives" expressed shock at being duped by previously loyal staff members. According to the NYT reporter, for example, the school believes that the administrators and other professors in the same department were not aware that their department offers courses that do not meet even once.

Even if I grant them their naivete, there is a deafening silence in the official story. What about the students? What about the beneficiaries of this fraud? Here, the plausible deniability vanishes. It is simply impossible that the students were duped. If you signed up for a course, and you couldn't figure out where or when the class meets, do you just go through a whole semester and not ask a question? When at the end of the term, you are given an A in the course for which you did not do any work, does no student ever ask a question?

What I really want to know is what UNC will do about those fraudulent grades. In my opinion, all these grades should be invalidated, and graduates who benefited from such courses should be given a choice: forfeit their degrees or earn those credits properly. As my friend pointed out to me, this course of action is lenient--because there is nothing punitive about it.

This is a moment in which we can tell if UNC is a business or an educational institution. If it is a business first, it will crater to its customers (and investors). If it is an institution that cares about academic integrity, it must act bravely.

***

The record on such matters unfortunately is dismal, as Kate reminded me. The students who copied final exams at Harvard last year were "forced to withdraw" for a year (not sure whether this will be noted on the transcripts). In spite of the investigation, many students and alumni remain convinced that the students were not wrong to copy their exams.

I also found this interesting article on how the withdrawal of athletes associated with the cheating scandal affected Harvard's APR (link). The administrators were talking openly about how this student leaving and so on affects the APR. The APR is probably the most gamed statistic yet to make the news.

Numbersense Pros: Interview with Len Testa

In the first chapter of my first book, Numbers Rule Your World (link), I explored the concept of variability using a pair of examples, one of which was Disney's FastPass virtual reservation system. Truly grasping the ins and outs of variability is one of the most important objectives for a budding statistician (or data scientist). In the discussion, I highlighted the work of Len Testa, whose website, TouringPlans.com, provides custom, computer-optimized itineraries for saving time in Disney's theme parks. Testa's team does exemplary work in applying mathematical models to solve a practical problem. I'm glad to present an interview with Testa today.

Len Testa holds a Masters degree in Computer Science from North Carolina A&T State University. He is the co-author of the Unofficial Guides to Walt Disney World, Disneyland, and Britain's Best Days Out, as well as a contributor to other books. His grad school research on the Time-Dependent Traveling Salesman Problem forms the basis of the Unofficial Guide's itinerary software.

***

KF: The problem of hitting a sequence of destinations in the fewest steps has a long history. Lots of people have worked on it. The most famous of these problems, the Traveling Salesman, even gets onto the mainstream press. However, most of this work is highly theoretical. Your touring plans to me are a shining example of amazing applied work that makes a lot of people happier. How does your work differ from the others? 

LT:

A lot of times when you're trying to analyze data to solve a particular problem, you can approach it either from the perspective of "management" - the people controlling the process - or from the side of the "consumer." 

The thing we try to model is optimal movement through a theme park. That is, if you're a customer and you want to ride 10 attractions, in what order should you visit them to minimize your wait in line?

The first time we approached this problem, we tried to figure out all of the things you need to know if you're running a theme park: ride speed, how many vehicles to have on the track, when to schedule entertainment to draw people to other parts of the park, and so on.  It was complicated.  

Then we looked at it from the point of view of the consumer.  Consumers, it turns out, have a lot less information about how a theme park is run.  About the only thing they really have is the posted wait time at every attraction.  But it turns out that the posted wait time is really a synthesis of all the small decisions a theme park manager makes, so that's all you need.  It's also a lot simpler to model.

KF: That is a really great answer. I hope all the budding data analysts out there are listening. Simplicity is a beauty.

***

KF: What is your pet peeve with published data interpretations?

LT:

Lack of context, especially around economic or political analyses. Yeah, an $860 billion stimulus package sounded like a lot of money in 2008. But in relation to a $15 trillion economy, it's what – 6%? But all of the discourse was on the raw number, not its size relative to the economy.

***

KF: Do you have other tips for doing great applied data work?

LT:

I find reasoning by analogy to be a powerful way to understand and explain things. For example, if you're putting off a flight to Europe because you're afraid of a plane crash, which is a 1-in-500,000 chance, then why would you ever drive to work, where the odds of dying are an order of magnitude worse? So a lot of it is "If I'm willing to accept X then I should be willing to accept Y” type of thing.

Another helpful thing is being able to apply Bayes Theorem, especially when you're trying to make a business case for something. I remember one time we were trying to get funding to re-do some computer system (at another job), and we calculated a probability of 80% that if we made these changes, we'd succeed in reducing customer problems and lower future operating costs. Some people looked at that as a 1-in-5 chance of failure. I pointed out that we were making decisions every day with a lot less than a documented 80% chance of success.

***

KF: Which sources do you turn to for reliable data analysis?

LT:

For economic analyses, Paul Krugman and the Financial Times. Nate Silver.

Closer to home, I like the mix of statisticians we have now at TouringPlans.com. They have different styles they use to approach problems, so it's useful to hear two different views. And when they agree, you know you've got a decent shot of being right.

KF: Statisticians disagree? They don't know the truth. Readers: you heard it on this blog first! Len, thank you so much for your time.

Innovation and the New York City subway

Innovation and NYC subway don't usually come together but something changed in the past year or so. One of the greatest life-changers has been the installation of countdown clocks in many of the stations, telling riders how long till the next (several) trains arrive. Now there is a smartphone app for this. (link)

***

Readers of Chapter 1 of Numbers Rule Your World learn the concept behind these countdown clocks as a "congestion management" tool. It turns out that the psychology of waiting is as important as the mathematics of queuing. A minute of waiting is not the same as a minute of waiting. For instance, a minute of waiting when one is uncertain as to how long the total wait would be is much more irritating than a minute of waiting when one knows how long the total wait would be. By installing such clocks, the MTA removes the unknown which causes anxiety.

My life has absolutely improved as a result of these clocks. I feel much calmer knowing when the next train will arrive. The most amazing part of this congestion management tool is that the actual waiting time has not changed at all but the riders perceive their waiting time to have shortened.

Chapter 1 looks at the problem of congestion and how statistical concepts help us understand and tackle the problem.

Reading guides for instructors, book clubs, etc.

I want to point your attention to recent coverage of Numbers Rule Your World in Chance News. If you are using my book for class or a book club, their writeups act well as reading guides for several chapters.

Notes on Chapter 1 (ramp meters)

Notes on Chapter 3 (hurricanes, test bias) and Chapter 4 (drug testing)

***

Chance News has been an extremely valuable resource for statistics teachers for years. They pick up news stories and interesting items, ask intriguing questions but don't offer answer keys. Most of the questions are open-ended, which is as most statistics questions are. We owe much gratitude to the volunteers who have kept the site alive all these years.

 

A conspicuous absence in storm reporting

The NYT has a nice article about the challenges of predicting hurricane intensity. A researcher pointed out it's difficult to get inside a storm to measure wind speeds and so:

it is not enough data to plug into a numerical model and yield a forecast that has a high degree of certainty.

I had the TV on most of the weekend, and there was around-the-clock coverage of Irene. I did not hear a single instance in which the forecaster (or broadcaster) provided information on the level of uncertainty of any of the predictions they were giving out. This would typically involve talking about probabilities or margins of error.

In other words, the experts admit that their forecasts are highly uncertain but the news report them as if they are 100% certain. We keep hearing comments such as "Irene will remain a Category 1 hurricane when it arrives in New York around Sunday morning and the maximum wind speed is expected to be around 70 mph."

***

The experts tell us the wind speed number is highly uncertain, and they quantify this uncertainty. Look at this Friday evening release from the National Hurricane Center, for example.

In the section labeled "wind speed probability table for selected locations", I looked up New York City. There was only a 5 percent chance of "sustained wind speeds (1 minute average)" of over 74 mph while there was a 56 percent chance that the wind speeds would be below 39 mph!

The press release is difficult to read, requiring readers to translate time, speeds, etc. to normal scales. However, if you are on the hurricane beat, there is no excuse for not spending the time to understand these data and conveying this information to citizens.

***

As discussed in Chapter 1 of Numbers Rule Your World, how much things vary around the average value is very important information not to be missed.

 

[Update, Sep 2]

Larry Cahoon, a reader of the blog and the book, emailed me about this post, and he makes several good points. In my original post, I didn't make these clear. It is in the hurricane wind speeds that the reporting failed; when it comes to the projected track of the storm, the reports typically include information on uncertainty. In addition, I'm complaining about the reporting of the science, not the science itself. Here, in Larry's words:

I read you piece on the error reporting for Hurricane for Irene and would say you have missed much of the reporting or the quality for the forecasts for the storm.

Perhaps I see more of the quality stuff as my main source for hurricane watching is the National Weather Service through http://www.hwn.org/ .
The discussion piece put out with each forecast is particularly interesting and reveals the weather services concerns with their forecasts in considerable detail. The error in forecast track is plotted at http://hwn.org/stormpulse_atlantic.html and is easy to see. This is picked up by many in the media and is sometimes referred to has the "Hurricane Cone."

The other area where the error in the forecast track shows up in the plots of the "spaghetti models" which seem to have been pickup up by a number of media outlets and even the major networks. A Google search of the phrase will give you many hit and some go into greater detail and discuss the impact of some of the forecast tracks.

There were areas where the media here in MD failed - mainly on wind forecasts that seemed to assume that the winds were of a uniform speed at a given distance from the storm regardless of which direction from the center of the storm someone was located.  That was clearly contrary to what the weather service tell everyone.

What I have failed to see are good discussions of the accuracy of the wind speed forecasts.  Although I did see a piece in the last two days discussing the National Hurricane Center's concerns with the quality for forecasts they are able to give for wind speeds.

 

New developments in breast cancer screening

Mammogram screening for breast cancer has become controversial. It's an emotional topic and the science is persuasive but not definitive. The danger of over-screening is ever present not just for breast cancer but for all types of diseases. False positives do have victims although the medical establishment doesn't often talk about it.

Here are two recent articles of new studies:

Screening has little impact on breast cancer deaths: study (link)

It's very hard, and often impossible, to measure directly the impact of a health policy because we cannot randomly assign people into treatment and control groups for comparison. Once a policy is in place, everyone is subject to the policy. If, as in this case, there are women who elect not to have mammograms, it may seem like we have two comparison groups: those who did and those who did not have mammograms. The challenge is that the women who choose not to do them are different from those who choose to do them in many ways. For instance, uninsured women are probably less likely to do them, and women who don't have insurance are probably poorer, less educated, etc. Such factors could explain any change in breast cancer deaths, as opposed to the use of mammograms.

This new study does a sort of pre-post analysis. In different countries, the mammogram guidelines went into effect at different times. If such screening truly reduces breast cancer deaths, then one should observe a marked drop in deaths when comparing the period before to the period after adoption of those guidelines. Thus, we can look at the timing of the policy and the timing of drop in cancer deaths to test this hypothesis.

The analysis is tricky. Many other factors may be contributing to the decline in breast cancer deaths. So the researchers find pairs of countries that have similar health care systems but different timing of mammogram policies. Of course, the analysis would break down if the matching procedure is found to be flawed.

Their conclusion is that the timing of the drop in breast cancer deaths is unrelated to the timing of regular use of mammograms.

Mammograms: computer-aided detection doesn't help (link)

This study confirms earlier findings that having computers help radiologists read results does not lead to higher accuracy. Sadly, it leads to over-diagnosis. This reminds me of the issue in Chapter 1 of Numbers Rule Your World: this technology may be worthless but what if patients perceive them to be useful?

The Economist implicates historians for bad statistics

This piece by the normally sane Economist was nominated to me via Twitter for a place in the Junk Charts hall of fame (This HOF was also a suggestion, it doesn't exist yet). While the chart is indeed quite shocking, the quantitative reasoning behind this piece, ironically titled "Quantifying history", is what deserves ignomity. So, I am posting it on Numbers Rule Your World instead.

The entire chart can be summarized in one sentence: human population has exploded in the 20th century. Any other commentary can be interpreted as a different way of stating this fact.

I said I wouldn't comment on the graph but I can't resist: the "21st century" is represented, with proper seriousness as befits a footnote, by the decade 2000-2010. You could be let off on probation in ordinary situations but when the chart presents the data (economic output and years lived) as proportions of the total sum, then the failure to either project to 2100 or to omit the 21st century is an offense that shocks the conscience.

***

The accompanying text posits a "democratic view" that "one person's life is just as much a part of mankind's story as another's."

In statistical terms, this is stamping out diversity, and treating every individual as the average. As I pointed out in Chapter 1, ignoring variability is a horrible thing. Lots of individuals have disporportionately large influence on the course of history. To name a few that comes to mind, George Washington, Nobel, Newton, Einstein, Shakespeare, Beethoven, Lenin, Hitler, Mao, etc. etc. cannot be replaced by any randomly selected person. Today, in the US, we have a special day in celebration of selected few individuals who have changed the course of history.

History is the stories of outliers, and certainly not the story of the average. So, the entire concept behind the chart is rotten.

I also reckon that while the population has exploded in the 20th century, what has also exploded is the number of poor people. Again, one must pay attention to the distribution of people, not just the average person.

Finally, what is "economic output (1990 $)"? This metric is a recent human invention, and it measures specific conditions that are deemed important to measure by the current consensus opinion of economists and politicians. I think at least some explanation is on order before using this metric to compare people across centuries.

E-coli in Europe, and risk perception

David Ropeik wrote in the Guardian about our irrational fears and misperception of risks. Worth reading (link). He asks: Why, if the actual risk for any given person is so low, does it feel so scary to so many?

Russia tried to ban all vegetable imports from the EU, which is a horrible idea. It is almost for sure that by this time, the contaminated batch of greens has been completely deprecated so any such measures are no more than PR stunts. Apparently, the EU convinced Russia to lift the ban (link).

Because of fear and, I must say, the lack of leadership to tackle the fear, quite a bit of unnecessary economic losses have been suffered. The EU estimates that farmers will lose $300 million.

Also, don't forget the amount of uncontaminated produce that has been laid to waste.

***

One other point: the level of risk is not the same for everyone. Most E-coli fatalities in past outbreaks have been elderly women or children with already compromised immune systems. In this case, 13 of 19 deaths were adult women, a little unusual but still a concentration of risk among a subset of the population. (link)

Just like a lot of situations, the "average" risk is not useful here. It's important to know if you are in the high-risk subgroup or not.

 

Some work till they die, some retire after they die: the discontent of being averaged

Flowing Data has a post about early retirement which provides an opportunity to highlight, yet again, the discontent of being averaged (Chapter 1 of Numbers Rule Your World).

The post comes with this chart. For each country and each gender, he compares the life expectancy and the average retirment age, the difference between which yielding the "number of years in retirement".

Unfortunately, one should be careful when taking the difference between two averages. This is because the population that underlies one average may not be the same one that underlies the other average.

The life expectancy is clearly an average over the entire population. Average retirement age, however, is almost certainly an average over people who live to at least retirement age. Thus, taking their difference doesn't make sense. What we need is the life expectancy of those people who at least live to see retirement.

The (somewhat) funny result of taking the difference of those averages can be seen in the case of Mexican men (the bottom blue bar). By this metric, the average years in retirement for Mexican males is essentially zero: they work till they die.

Another way to see that this metric can't work: if the average years in retirement is zero, then some will die after they retire while others will retire after they die! Retiring after death is quite literally captured when you substract the average retirement age from the average life expectancy!

***

By locating the right data, one can improve this analysis. What is needed is a life table. (one source is here). From this table, we can find for every age, the expected remaining life.

Take all the age groups above the retirement age, compute weights for each age group as a proportion of all remaining age groups, use these weights to compute the weighted average remaining life.

For Mexican males, this yields about 7 years, assuming the retirement age is indeed 75 as shown on the chart above.

 

P.S. See also the comments from the Flowing Data readers.

 

 

The statistics of anti-doping: the entire book in one real-life example

A reader of my blog, Joran E., pointed me to this great article (by Ross Tucker) that covers one of the newer anti-doping measures (the biological passport), which links to this recent NYT article on two Italian cyclists found guilty of doping. So while I was researching this latest development, I came across the latest legal maneuvres in the case of Alberto Contador, the Spanish cyclist and multiple winner of Tour de France who tested positive after last year's victory, and subsequently blamed a contaminated steak (I mentioned his case here last year).

Anti-doping provides a perfect back drop to revisit all five statistical concepts that form the spine of my book, Numbers Rule Your World.

***

The most potent form of doping these days is human growth hormones (HGH), EPO and similar compounds that have the characteristic of occurring naturally so that labs must seek to separate dopers from people who have "natural highs". By contrast, for compounds that don't occur naturally, such as clenbuterol that ensnared Contador, even minute amounts can be proof of wrongdoing.

In order to know what level of a compound is "unnatural", statisticians need to establish what is natural. This is the concept behind Chapter 1: we calculate the "average" (natural) value, but focus on examining variations around the average.

Admitting that the natural value is not uniform across all people, statisticians determine different averages for different "types" of people; the simplest such subgroups would be male/female, and age groups. The biological passport takes this idea to the extreme: each individual athlete is tracked over time to establish his or her own average. We just put into practice the concept behind Chapter 3, which is to avoid lumping together things that are different.

The cases of the two Italian cyclists represent the first two in which athletes have been punished based on evidence from the biological passport. Previously, the enforcers need a failed drug test or a police bust to convict dopers.

Readers who like the materials in the Conclusion chapter (as related to Chapter 4) on the hematocrit test should definitely read Tucker's article for details on how the biological passport works.

***

The developments in the Contador case are very discouraging: the Spanish cycling federation showed an unwillingness to expose the biggest star of the sport, first by assessing a one-year ban (when the norm is two years), and most recently, overturning that shortened punishment. USADA, the US anti-doping body, expressed its concern here, and the reversal of the ban is under appeal.

The Spanish authority accepted the Contador camp's explanation of unintentional consumption of tainted beef as the reason for testing positive. Statisticians who believe in the logic of hypothesis testing will find such a conclusion absurd.

Let's walk through how we apply the logic as described in Chapter 5 of Numbers Rule Your World to this situation. Assuming that Contador did not dope, what is the chance that minute amounts of clenbuterol would be found in his body? Unfortunately for Contador and other athletes failing this drug test, the chance is vanishingly small.

Like most accused dopers, his camp did not challenge the presence of clenbuterol; they merely offered an alternative theory for why it was there. A large number of coincidences had to occur in order for their theory to be believed: beef had to be taken from Spain into France to serve Contador, and only Contador (not any of his teammates); a different source of beef must have been used on other days during the Tour on which Contador ate beef (since he tested negative on most other days of the tour); he was one unlucky fellow since anti-doping tests have high false negative rates in general, and he managed to test positive on that one time he ate the contaminated beef; he was also extremely unlucky since Europe banned the use of clenbuterol to raise cows in the 1990s, and the beef he ate on that one occasion had to have come from an unscrupulous farmer violating the ban.

Statisticans would politely listen to all that, and declare "rare is impossible". It's much easier to believe that he was doping. (We would admit that there is a miniscule chance that the conclusion is incorrect -- the chance is precisely that of those coincidences occurring.)

***

Why does the scientific process disintegrate into this sort of he-said-she-said argument?

The concept behind Chapter 2 proves useful here. The statistical model that links the biological passport and/or the drug test to doping is one based on correlation, not causation. The passport or drug test does not provide direct evidence of doping (unlike a police bust). But as I point out in the book, correlational evidence can be powerful, and has been profitably used in all kinds of decisions. Because clenbuterol is not produced naturally in the human body, this test result is very close to causal evidence; it's less secure for things like EPO and HGH.

It's just more complicated when causal evidence is unavailable because people can now advance all sorts of hypotheses to explain the correlation. We then get story time, a phenomenon I frequently discuss on my blog. I'm happy to hear the stories but one must seek evidence to support these stories.

In the Contador case, for example, I'd like to see evidence that the steak was eaten, receipts from the vendor who imported the beef, documentation of which farm raised the cow, inspection of the farm to confirm that it used clenbuterol, traceback of beef from that farm to find the presence of clenbuterol, etc. In none of the reports on this case have I seen any of this evidence, and more disturbingly, the supporters of Contador don't appear to be asking any such questions. (See, for instance, Christian Josi on Huffington Post.)

***

Why would statisticians accept the chance of falsely accusing a clean athlete, however small that chance is? This is because we know that there is no such thing as a perfect test. The only test that will never yield a false positive is the test that never issues any positive results!

We already accept this type of situation in the Western legal system. A criterion of "beyond reasonable doubt" in the courts does not guarantee no wrongful convictions. In fact, thanks to the work of groups such as the Innocence Project, we know that some unfortunate people are wrongfully convicted, sometimes for long sentences for grievous crimes they did not commit.

As explained in my book (which I won't repeat here), the real issue in anti-doping is not about false positives but about false negatives. I fear that the entire system is so lenient toward dopers that they would take the (small) risk of detection.  I'll make a case for this in a future post.