Big Data, Plainly Spoken (aka Numbers Rule Your World)

A conspicuous absence in storm reporting

The NYT has a nice article about the challenges of predicting hurricane intensity. A researcher pointed out it's difficult to get inside a storm to measure wind speeds and so:

it is not enough data to plug into a numerical model and yield a forecast that has a high degree of certainty.

I had the TV on most of the weekend, and there was around-the-clock coverage of Irene. I did not hear a single instance in which the forecaster (or broadcaster) provided information on the level of uncertainty of any of the predictions they were giving out. This would typically involve talking about probabilities or margins of error.

In other words, the experts admit that their forecasts are highly uncertain but the news report them as if they are 100% certain. We keep hearing comments such as "Irene will remain a Category 1 hurricane when it arrives in New York around Sunday morning and the maximum wind speed is expected to be around 70 mph."

***

The experts tell us the wind speed number is highly uncertain, and they quantify this uncertainty. Look at this Friday evening release from the National Hurricane Center, for example.

In the section labeled "wind speed probability table for selected locations", I looked up New York City. There was only a 5 percent chance of "sustained wind speeds (1 minute average)" of over 74 mph while there was a 56 percent chance that the wind speeds would be below 39 mph!

The press release is difficult to read, requiring readers to translate time, speeds, etc. to normal scales. However, if you are on the hurricane beat, there is no excuse for not spending the time to understand these data and conveying this information to citizens.

***

As discussed in Chapter 1 of Numbers Rule Your World, how much things vary around the average value is very important information not to be missed.

 

[Update, Sep 2]

Larry Cahoon, a reader of the blog and the book, emailed me about this post, and he makes several good points. In my original post, I didn't make these clear. It is in the hurricane wind speeds that the reporting failed; when it comes to the projected track of the storm, the reports typically include information on uncertainty. In addition, I'm complaining about the reporting of the science, not the science itself. Here, in Larry's words:

I read you piece on the error reporting for Hurricane for Irene and would say you have missed much of the reporting or the quality for the forecasts for the storm.

Perhaps I see more of the quality stuff as my main source for hurricane watching is the National Weather Service through http://www.hwn.org/ .
The discussion piece put out with each forecast is particularly interesting and reveals the weather services concerns with their forecasts in considerable detail. The error in forecast track is plotted at http://hwn.org/stormpulse_atlantic.html and is easy to see. This is picked up by many in the media and is sometimes referred to has the "Hurricane Cone."

The other area where the error in the forecast track shows up in the plots of the "spaghetti models" which seem to have been pickup up by a number of media outlets and even the major networks. A Google search of the phrase will give you many hit and some go into greater detail and discuss the impact of some of the forecast tracks.

There were areas where the media here in MD failed - mainly on wind forecasts that seemed to assume that the winds were of a uniform speed at a given distance from the storm regardless of which direction from the center of the storm someone was located.  That was clearly contrary to what the weather service tell everyone.

What I have failed to see are good discussions of the accuracy of the wind speed forecasts.  Although I did see a piece in the last two days discussing the National Hurricane Center's concerns with the quality for forecasts they are able to give for wind speeds.

 

Save the Statistical Abstract

Krugman points to this plea from Robert Samuelson to save the U.S. Statistical Abstract. Under pressure from Congress to "save money", the Census Bureau will disband the small team that assembles publications on the statistics of the United States. Apparently, this move cuts 24 jobs and saves $2.9 million annually.

This development is disturbing in many ways:

• The Abstract will simply vanish. They are not killing just the paper version. There will be no online version either.
• Samuelson claims someone at the Bureau made this statement: "[The Bureau] has to choose between its basic job of devising surveys and collecting statistics about economic, social and governmental conditions and the less-important task of publicizing the results." I hope this is a mis-quotation. Presenting and publicizing data is "less important" than collecting data? The priority is totally backwards.
• Terminating this statistical publication is an assault on our democracy. By not publicizing these statistics, the effect is to ring-fence this data so that only specialists will access it. By not organizing them into a publishable state, only specialists will have the time and the expertise to sift through the data and make sense of it. 
• Statistics is an annoying profession to those who find themselves being measured by the numbers. When things are not measured, they will be "guesstimated" with any number of assumptions. Policies will be put in place, and their effects will also be "guesstimated". Accountability becomes a vague concept, infinitely debatable. This does not bode well for effective governance.

Headline writers, health-beat journalists, editors embarrass themselves en masse

You'd think reading the health sections of our mass media would go some way towards prolonging your life. But you should think twice. A horrible disease is being spread by health reporters. Their crime is spreading false information in an attempt to attract eyeballs.

Note the most read article on Yahoo! News the other day:

A lot of people clicked on the link that proclaimed: "Too much TV may take years off your life."

The article dutifully reported the findings of an Australian study: that "people who averaged six hours a day of TV lived, on average, nearly five years less than people who watched no TV." Also, that "For every hour of television watched after age 25, lifespan fell by 22 minutes." These were almost direct quotations from the scholarly paper.

By the fourth sentence, the reporter informed us: But other experts cautioned that the study did not show that TV watching caused people to die sooner, only that there was an association between watching lots of TV and a shorter lifespan. For the next few sentences, the reporter allowed these experts to tell us why correlation is not causation.

And yet, the reporter, the headline writer and the editor persisted in telling readers "too much TV may take years off your life".

***

Such mischief is not limited to Yahoo!  Here's Time magazine with an even more ridiculous headline:

And taking the prize was AOL/HuffingtonPost:

This above headline is frankly embarrassing. According to the study, each hour of TV after age 25 is associated with 22 minutes lower lifespan. Each additional hour of TV is associated with a higher risk of death. But how did this finding morph into (just) one hour of TV could shorten one's life? Turn this around. Would doing just 1 hour of exercise in one's lifetime prolong one's life?

And the ailment has already crossed the Atlantic. Here is the Independent (UK):

If the writer has read the paper, he will note that in the sample, only 1% of the people reported watching 6 hours a day or more. In fact, across all age groups (25 and over), the average number of hours of TV viewing was 1.5-2.5 hours per day. So the statement about 6 hours a day is useless for 99% of the population.

***

Here are some other pertinent information from the study ignored by the media:

• Every finding in the study just marginally passed the minimum standard of statistical evidence required for publication in a scholarly journal. For example, the reported 22 minutes decline in lifespan had a margin of error of 21.5 minutes.
• Also, that 22 minutes difference come from an extreme-to-extreme comparison. It's the heaviest 1% of TV watchers versus those who don't watch any TV at all.
• What is a plausible link between TV watching and "all-cause mortality" (death by any reason)"? Is the hypothesis that electromagnetic waves are harmful? that excessive violence is harmful? that high-decibel screaming makes us sick? The researchers did not speculate at all. They did bring up smoking as a comparable risk factor, which I find puzzling since the biochemical link between smoking and death is pretty clear by contrast.

If I have the chance, I'd do a survey of those journalists who published these articles to find out what proportion of them stopped watching TV to gain those extra minutes of life.

 

What to think of massive food recalls?

There's a lot I can say about the ongoing salmonella outbreak investigation and the massive recall of 36 million pounds of turkey, much of which you'll find in Chapter 2 of Numbers Rule Your World.

One question you might want to ask is: how big a risk is it for you?

It should be clear that the level of risk is different for different people. For example, I don't eat ground turkey so my risk is the risk of CDC identifying the wrong culprit (a false positive), meaning that I might eat something else that would get me sick. If you do consume ground turkey, your risk is that of buying a pack of ground turkey that happens to be contaminated -- plus the risk of a false positive.

According to the official press release, about 10% of Americans consume ground turkey in any given week. This means about 1/3 of Americans consume ground turkey in any given 4 weeks (1 - (0.9)^4 assuming independence from week to week), 6/10 in any 8 weeks.

***

Because fresh food is perishable, the risk depends on time. Assuming a temporary source of contamination, the batches of bad food would make its way through the food cycle in a fixed amount of time. (Permanent sources would be much easier to identify because you can just test samples from each machine to find the culprit.)

The following time-line chart from CDC is instructive:

Notice a few features of this chart:

• the vertical scale has some very tiny numbers, we are talking about a few cases per week across the entire country
• there is a "background" level of so-called sporadic infections (maybe about 2 a week) that would be visible whether or not there is a specific outbreak
• the horizontal axis plots the time of reported illness which means that the actual times of infection would be shifted to the left
• it was on July 27 when CDC issued a public health warning: note that this is to the far right of the curve. Could it be that the contaminated food has mostly made it through the distribution system? That the outbreak is well past its prime?
• Cargill recalled batches dated all the way back to Feb 20. Could it be true that such meat is somewhere still on sale? I hope not.

In the book, I discuss the perils of statistical modeling with so few cases, the chance of false positives, the difficulty of establishing a cause-effect relationship, the incentives of health agencies, the logic of food recalls, and so on.