Big Data, Plainly Spoken (aka Numbers Rule Your World)

Where do the numbers come from?

Note to readers: There will be few updates to the blog in the next few days until southern Manhattan gets its power back.

Note to AT&T: If you listen to your customers, you won't need to "investigate". Cell phone service has been unavailable since Monday.

***

Lots of numbers are thrown around in the media all the time. Do you know how they are derived? This is a really important question to ask every time you see a number.

Sometimes, the source is pretty obvious. Polls and surveys provide a lot of data. Websites report a lot of numbers because every click is registered in a database. Some data are painstakingly collected by human beings, like the situational statistics collected in a baseball or football match.

What you may not realize is how unreliable some of the data sources can be. The recent New York Times article discusses a change to how much sugar Americans are consuming (link). It discloses how inexact this science is.

Some of the steps involved in deriving that number:

• An average American diet is estimated
• For each food item, the sugar content is estimated
• For each food item, the "food loss" is estimated. The idea is that if food is bought but not consumed, the sugar is not consumed.
• A Nielsen survey of food purchases is used
• Interviews conducted by CDC are used
• Businesses and lobbyists for the sugar industry are consulted to provide "expertise"
• Academics are consulted to provide "expertise"; however, none of these professors are willing to talk about what "advise" they provided. They claimed that they could not recall. Apparently, no notes were taken by anyone at these meetings, and also they could not remember something that happened in 2008.

This enterprise sounds impressive because it is so complex. But complexity is only a good thing if we are able to measure the finer components more precisely; in this case, even at the finer level, each step involves guesswork. The error of each step then compounds. When you add in lobbyists who have an agenda, it's a grand mess.

The news here is that the Agricultural Department decided that the "food loss" estimates in the past are flawed, and by using new "methods", their estimate of sugar consumption fell by a whooping 25 percent. How can we believe this new research? To me, it is impossible to create "food loss" estimates for hundreds of food items. If you look at the report, they have food loss estimates for each of spinach, squash, sweet potatoes, snap beans, tomatoes, broccoli, etc. and the tables go on for pages.

The researchers are not conservative either. They make huge revisions to the prior estimates. The article mentioned pumpkins, which went from 20 percent lost to 69 percent lost. Yes, they claim that 70 percent of all pumpkins sold are never eaten. I mean, it's Halloween but are we really growing pumpkins just for the fun of one night?

Now these researchers told the reporter that comparing the old estimates to the new estimates is "improper because of changes in methodology". This is absurd; new methods must be compared to existing methods.

We sorely need an estimate of the margin of error of this sugar consumption number. Given that it has so many components, each of which has a wide margin of error, my suspicion is that the entire enterprise collapses by its own weight.

***

What are some ways to create a much more credible estimate? Look at the diet, and focus on the big items. Recruit a panel of people (like Nielsen does) and ask them to record their own behavior for a period of time... how much they purchased? how much they trashed? At least, we have a set of real data to work with. The sample size may be smaller but the quality of the data is much higher.

I'm sure there are many other superior methods. It's an interesting problem for statisticians to think about. For others, be careful when you're fed numbers.

 

Other doping myths busted

The Lance Armstrong story continues to regale.

The biggest myth that is busted is that a series of negative tests prove anything. (This shouldn't be controversial to anyone who follows doping. Marion Jones, Tyler Hamilton, etc. etc. all passed hundreds of tests before getting caught.)

Another myth in the doping circle is the "victimless crime". Of course there are losers. Apart from Armstrong, who now stands to lose both reputationally and financially, there is a long list of losers.

1. The international cycling union (UCI)

Basically been exposed as a fraud and shown to be unable to manage its own sport. It is highly embarrassing to have outsiders (US Anti-Doping Agency) do the work it should have been doing to keep the playing field level. The chairman apparently even blasted the cyclists for ratting out Armstrong (link).

2. Nike

It doesn't matter what Nike is now saying about being deceived. The fact remains that they have sullied their brand over so many years with what is now known to be one of the greatest frauds in sporting history. Given Nike's deep roots in sports, it is really hard to believe that they could have been so thoroughly deceived. Is it being deceived or is it self-deception?

3. Former teammates

If Armstrong loses his prize money, does this mean there will be clawbacks from his heavily doped teammates?

4. Former rivals

Interesting story on Bobby Julich, who used to be a bitter rival of Armstrong's, here. He and several others have now confessed to doping also, and have been forced to resign from coaching their current team. The confessed doper, Riis, is also a part of this story. Riis, if you've read Chapter 4, is another well-known cyclist who passed a large number of tests but later confessed to doping. More on Riis here.

5. Still clueless athletes

Not just the UCI head but other former athletes believed that Armstrong is the victim and those who confessed now are criminals. Their argument is that a doper who doesn't get caught is as good as not doping. Given the statistics of anti-doping, and especially if we are not willing to accept some false convictions, such an attitude makes any anti-doping program a joke. Contador, for example (link), who got caught cheating himself. Here's Joachim "regretting" that he "did not dope", and hailing Armstrong as "the winner". (link)

6. Fans of the sport

First, there were the hoards of people who believed Landis for a year or so until he finally confessed. If honesty is something of value, it's really hard to continue to watch this sport. More so than cheating, what is sadder about the situation is that every doper is a liar who lives the lie till the day he confesses, or he takes the lie to the grave.

***

Another myth also busted is "doping does not help". This excuse has always been suspect since there won't be any doping if dopers cannot enhance their performance. Now, people like Bobby Julich are confessing that the period in which they performed the best was precisely when they succumbed to doping.

Frankie Andreu also did his best work when he doped. (link) He quit the sport early, renounced doping, and was sidelined for many years while the sport lived in denial.

***

A whole series of interesting articles at VeloNews.

Predictive modeling

Love this xkcd comic (link). Would have been even better cut to the final third. Great food for thought for predictive modelers.

A few years ago, some economists wanted to change their last name because economists with last names beginning close to the front of the alphabet have greater chance of winning the Nobel (according to their research). (link to PDF)

Lance Armstrong beat statistical odds, but it's not enough

Readers of Chapter 4 of Numbers Rule Your World would not be surprised about the fall of Lance Armstrong. The big theme of the chapter is that any screening tests has to balance between false positives and false negatives. The anti-doping agencies are so concerned about not falsely accusing anyone that they leave a gigantic hole for dopers to walk through. Strings of negative tests, therefore, do not prove one innocent; however, even a single positive test could be quite telling.

While we think about Armstrong's plight, let's not forget about this fact: every one of those who now confessed  passed hundreds of tests in their careers, just like Armstrong did. In fact, fallen stars like Tyler Hamilton and Floyd Landis also passed lots of tests before they got caught. In effect, dopers face a lottery with high odds of winning and low odds of losing.

Sadly, the media covering this issue continues to ignore false negative errors in testing. I have yet to find one article covering the current situation that acknowledges that dopers pass tests all the time! This over-confidence in scientific models is a disease, not just in drug testing.

***

In the book, I also discussed why we need a lie detector test in athletics. So, we arrive now at a day when a good handful of big-name cyclists decided to shed their lies and open up to us.

***

Another myth shattered by this scandal is the idea that stars don't need to cheat. It is most likely the opposite. At the very top of any sport, especially a sport that pays, the difference between the number 1 and the number 2 is vast in terms of financial reward but infinitestimal in terms of physics. Every little advantage counts. Placebos count.

It's hard to imagine why someone who has no chance of winning anything would take drugs that might kill them. So, when they say everyone was cheating, I wonder if they meant everyone who was competitive was cheating.

***

Here're my previous articles calling for "more false positives" in drug testing (link, link).

There are many stories because so many cyclists are named. Some are already selling their stories. Here are some links:

New York Times on the big report (link)

Tyler Hamilton already has a book (link)

George Hincapie testified (link)

Levi Leipheimer statement (link)

Glossary of secret codewords (link)

Former Irish massage therapist told her story (link)

Former coach continues to deny, deny (link)

Tour de France organizers silent. Imagine not having anything to say about a 7-time champion. (link)

Also silent are his sponsors. Nike? (link)

 

 

 

Predictive hiring models, and Chapter 2

This Wall Street Journal article about predictive modeling in the workplace is well worth reading. Your reaction may vary from excitement to stomach churning. (link)

As with most reporting on analytics, the author never addresses the issue of model errors. For example, this article started out with:

When looking for workers to staff its call centers, Xerox Corp. used to pay lots of attention to applicants who had done the job before. Then, a computer program told the printer and outsourcing company that experience doesn't matter.

The fun may be squeezed out if we want a more balanced story. I get that. But surely, someone should ask if this "computer program" makes no errors. Because if you read this style of reporting, you would think every prediction is correct.

The reporter does get into the creepiness and the legal murkiness of using such models but those discussions hover over the unspoken assumption that the predictions issued by these computer algorithms using "big data" are infallible. It's one thing if the model excludes, say, women from being hired because well, every woman fails in that particular job; it's another thing if the model excludes women erroneously! (I can already see the "I took steroids but look, it didn't help." excuse.) Later in the article, we are told that a startup company has created the model of "an ideal call-center worker". The fact that there could be an "ideal" reflects a level of over-confidence in models that bothers on insanity. (And I'm someone who builds models for a living.)

As you read the article, bear these points in mind:

• In Chapter 2 of Numbers Rule Your World, I talk about models of correlations versus models of causation. In my view, when people's livelihood is at stake, a correlational model is not good enough. All these models described in the article are correlation based. The factors they use are things like commuting distance, and social networks. There is a serious danger of "causation creep" here. 
• An analogy: if you look at crime statistics, it is certainly true that African Americans are more likely to be criminals. If you build a crime prediction model, race will be a strong predictor even if it's not the only predictor. A lot of us are uncomfortable with this kind of racial profiling.
• A technical note: this type of models suffers from "rejection inference" (terminology from the credit scoring world). I didn't find space to talk about rejection inference in Chapter 2 but I might as well say something here. The output of such predictive models is used to make a hiring decision. The candidate can either be hired or rejected. If the candidate is hired, the company can track his or her performance and make a determination as to whether, ex-post, the hiring decision was good or bad. If the candidate is rejected, the company has no way of knowing whether the rejection was just or not. So when any company collects data to build a predictive hiring model, the dataset is already biased, as it does not contain candidates who would have been rejected.

 

The New Zealand sun alibi

BBC News, reporting on the results of a study of Vitamin D supplements, inserted a curious sentence: "The study, carried out in New Zealand, which gets more sunshine annually than the UK, found the vitamin D supplements increased blood levels of the vitamin. But this had no significant impact on the rate or severity of colds." (my italics)

This is yet another study that shows supplements/drugs that succeed in making some metric look better but fail to deliver any true health benefits.

The New Zealand sun is the alibi used by advocates of Vitamin D supplements to discredit this randomized trial. They argue that because the sun delivers Vitamin D, someone who lives under the sun would be less likely to be deficient in Vitamin D. And the supplements only help those who are deficient.

Unfortuately, this elaborate story has no collaborative evidence. Instead, we have an expert who said this is so ("a leading UK cold expert said vitamin D is useful"), bringing his own anecdote ("He said he takes [vitamin D supplement] every year as a precaution").

Test result from randomized controlled experiment versus expert opinion based on anecdotes. That's not a fair match, and yet the BBC seems to prefer the latter, and thus serves us the NZ Sun alibi.