Numbers Rule Your World

Getting inside my head

Some interviews with me or snippets of such have surfaced recently. Here is a list:

Kate Meersschaert interviewed me for New Learning Times (link; registration required). I talked about my teaching philosophy, and why I write books.

Jay Ulfelder, a political scientist who keeps an interesting blog, recommends Numbers Rule Your World, and a few other books for political scientists (link).

If you haven't heard already, 2013 is the International Year of Statistics. I was one of the talking heads here.

Here, I talk about the history of Junk Charts, and the new paradigm of interactively building graphs, as opposed to the template paradigm popularized by Excel.

 

Book review: The Signal and The Noise

Nate Silver first attracted attention two election cycles ago with the launch of his fivethirtyeight.com website (538 is the number of electoral votes in the United States.) He makes clean charts, which I like a lot. Since that time, he has earned a platform on the New York Times website, which goes some way to explaining the vitriol hurled at him during the just-concluded election season by the right wing. Being in the national conversation is convenient when one has a book coming off the press – and I write this admiringly, as I believe Silver’s popularity and influence further the cause of anyone in favor of the data-driven mindset. Surely, the predictive success of his model, as well as those of a number of copycats, has resoundingly humbled the pundits and tealeaf readers, who talked themselves into ignoring the polling data.

The book is titled The Signal and the Noise (link). As explained by Silver, these terms originated in the electrical engineering realm, and have long served as a metaphor for the statistician’s vocation, that is, separating the signal from the noise. Imagine making a long-distance call from California to Tokyo. Your voice, the signal, is encoded and sent along miles of cables and wires from one handset to the other, picking up interference, the noise, along the way. The job of electrical engineers is to decipher the garbled audio at the other end, by sizing and removing the noise. When the technology fails, you have a “bad connection”, and you can literally hear the noise.

***

It is in the subtitle—“why so many predictions fail – but some don’t”—that one learns the core philosophy of Silver: he is most concerned with the honest evaluation of the performance of predictive models. The failure to look into one’s mirror is what I often describe as the elephant in the data analyst’s room. Science reporters and authors keep bombarding us with stories of success in data mining, when in fact most statistical models in the social sciences have high rates of error. As Silver’s many case studies demonstrate, these models are still useful but they are far from infallible; or, as Silver would prefer to put it, the models have a quantifiable chance of failing.

In 450 briskly-moving pages, Silver takes readers through case studies on polling, baseball, the weather, earthquakes, GDP, pandemic flu, chess, poker, stock market, global warming, and terrorism. I appreciate the refreshing modesty in discussing the limitation of various successful prediction systems. For example, one of the subheads in the chapter about a baseball player performance forecasting system he developed prior to entering the world of political polls reads: “PECOTA Versus Scouts: Scouts Win” (p. 88). Unlike many popular science authors, Silver does not portray his protagonists as uncomplicated heroes, he does not draw overly general conclusions, and he does not flip from one anecdote to another but instead provides details for readers to gain a fuller understanding of each case study. In other words, we can trust his conclusions, even if his book contains little Freakonomics-style counter-intuition.

***

Performance measurement is a complex undertaking. To illustrate this point, I list the evaluation methods deployed in the key case studies of the book:

1. McLaughlin Group panel predictions (p. 49): proportion of predictions that become “completely true” or “mostly true,” ignoring predictions that cannot be or are not yet verifiable
2. Election forecasts (p. 70): proportion of Republican wins among those districts predicted to be “leaning Republican” (underlying this type of evaluation is some criterion for calling a race to be “leaning”)
3. Baseball prospect forecasting (p. 90): number of major-league wins generated by players on the Top 100 prospect list in specified window of time; the wins attributed to individual players are computed via a formula known as “wins above replacement player”
4. Daily high-temperature forecasts (p. 132): average difference between predicted temperature (x days in advance) and actual temperature relative to “naïve” methods of prediction, such as always predicting the average temperature, or predicting tomorrow’s temperature to equal today’s
5. Rainfall forecast (p. 135): how close to, say, 20%, is the proportion of days on which it actually rained, among those days when the weather service forecasts 20% chance of rain
6. Earthquake forecast (p. 160): whether an earthquake in the predicted range of magnitude occurred in the predicted range of time at a predicted region of the world, or not (this is a binary outcome)
7. GDP growth forecast (p. 182): the proportion of times in which the economist’s prediction intervals contain the actual GDP growth
8. Chess (Ch. 9): winning games
9. Poker (p. 311): amount of earnings
10. Long-range global temperature forecast (pp. 398, 402): actual trend against predicted trend. (Note that this is the same method as #7 but with only one prediction interval.)

If you are thinking the evalution methods listed above seem numerous and arbitrary, you’d be right. After reading Silver’s book, you should be thinking critically about how predictions are evaluated (and in some cases, how they may be impossible to verify). Probabilistic forecasts that Silver advocates are even harder to validate. Silver tells it like it is: this is difficult but crucial work; and one must look out for forecasters who don’t report their errors, as well as those who hide their errors by using inappropriate measurement.

***

Throughout the book, Silver makes many practical recommendations that reveal his practitioner’s perspective on forecasting. As an applied statistician, I endorse without hesitation specific pieces of advice, such as use probability models, more data could make predictions worse, mix art and science, try hard to find the right data, don’t just use readily available data, and avoid too much precision.

The only exaggeration in the book is his elevation of “Bayesian” statistics as the solution to predictive inaccuracy. What he packages as Bayesian has been part of statistical science even before the recent rise of modern Bayesian statistics. (The disagreement between Bayesians and non-Bayesians is over how these concepts are utilized.) Silver’s exposition focuses on probability updating in sequential decision-making, which is understandable given his expertise in sequential settings with a rich tradition of data collection, such as baseball and polling. (At one point, he makes an astute comment about data analysts selecting more promising settings in which to work.) The modern Bayesian movement is much broader than probability updating, and I’d point you to Professor Andrew Gelman’s blog and/or books as a place to explore what I mean by that. It must be said, though, that the technicalities of Bayesian statistics are tough to convey in a mass-market book.

***

In spite of the minor semantic issue, I am confident my readers will enjoy reading Silver’s book (link). It is one of the more balanced, practical books on statistical thinking on the market today by a prominent public advocate of the data-driven mindset.

Predictive modeling

Love this xkcd comic (link). Would have been even better cut to the final third. Great food for thought for predictive modelers.

A few years ago, some economists wanted to change their last name because economists with last names beginning close to the front of the alphabet have greater chance of winning the Nobel (according to their research). (link to PDF)

Significance of early voting

Andrew Sullivan highlights the significance of early voting in U.S. Presidential Elections (link). His second quote (from The Fix) requires a visit to your statistical adviser.

Andrew considers this an "important point":

Democrats generally vote early more than Republicans. In the five 2012 swing states where a 2008 early voting breakdown is available, Democrats voted early more than Republicans in all five. Even in a very good GOP year in 2010, Democrats voted early more often than Republicans in North Carolina, Iowa and Nevada.

I'm afraid many will misinterpret this statement. What really matters is which party succeed in pushing more wavering voters to vote for them early, and thus nail down those votes. It would give Democrats no advantage if those early voters are hardcore party-line voters. The hardcore Democrat can cast his/her vote on Election Day or any day prior but it's still one sure vote.

In other words, not every early voter is alike. Early voters who are independents, previously undecided, wavering, etc. are more useful than early voters who are party-line voters.

There may be an adverse selection effect, in the sense that undecided voters are less likely to vote early since, well, they haven't made up their minds. So the people who are voting early are really the ones you have in your bag.

***

For me, the significance of early voting are these two points :

• Early votes skew later voting propensity (which is why I don't think it's a good idea). This is similar to the East Coast/West Coast issue: by the time people in the West Coast wake up, there would already be results from the East Coast, or at the minimum, results from exit polls. This can have the effect of causing some people to stay home or other people to rush out to vote. It also allows campaign managers to take remedial action before Election Day. 
• Early votes is an insurance policy. It guards against gaffes leading up to Election Day.

How averages get usurped by policymakers

This WSJ reporter doesn't much like statisticians, growth charts, and CDC (link). Her headline is "Is Baby Too Small? Growth Charts Make It Hard to Tell." This is her lede:

Elias Thorsteinsson weighed 6 pounds, 11 ounces, when he was born in January 2010. That put him in the 25th percentile of U.S. babies, meaning 25% of babies were smaller than he was at birth. But he didn't gain weight as quickly as other infants and dropped to the 1st percentile on the pediatric growth chart when he was 6 months old...Elias gained weight rapidly once he started eating solid foods. His head grew even faster—to the 99th percentile on the head circumference chart. His pediatrician wanted to do an MRI to rule out brain abnormalities. But Elias was developing normally, so Mrs. Stebbins refused. "We'd been through the wringer with these percentiles," she says. "I said, 'I have a big head. His father has a big head. He'll grow into it.' "

Is this damned lies and statistics or is this calling everything that has four legs a pig?

She goes on to complain, among other things, the existence of CDC and WHO standards which are not identical, and the existence of pediatricians who don't "always take the time to fully explain what the growth charts mean". (my italics)

***

The history of growth charts (height and weight tables) is fascinating, and not widely known.

They were pioneered by actuaries rather than doctors. In other words, they were first created for commercial reasons -- to enhance corporate profits of insurers -- rather than for health reasons.

The first charts were descriptive. It merely gave the body shape for the "average" person of a certain age and gender. Eventually, these charts became prescriptive - people who are not average are rated as "overweight", "obese", "underweight", etc. It's a big difference between knowing you're 50% above the average and being told you have to slim down, which is to say, you should be average.

Now, it appears that the growth charts may become detached from reality. The reporter includes an unattributed quote from the CDC: "The CDC says it doesn't plan to adjust its charts because it doesn't want the ever-more-obese population to become the new norm." This sounds like the standards will now be detached from the data, and simply be whatever the CDC wants us to weigh.

 

PS. This is not an anti-CDC post. There is an interesting problem here relating to using indices. If we keep using average BMI as the "target", and we are failing to revert to the mean, then the "target" will keep moving.

 

A statistical dead heat is not a virtual tie

Every election year, journalists have a feast over the hundreds of poll results that never seem to end. They frequently abuse language as they try to explain what the polls mean. Because polls are small samples of people, poll results can only say so much. Specifically, when races are tight, they don't tell us much. This lack of clarity creates a certain nervousness among the prognosticators.

It was refreshing to see this headline: "Among Republicans, Santorum in statistical dead heat with Romney". (link) When was the last time the news tell us two candidates are neck and neck? The actual result was 30% Santorum to 28% Romney. You'd expect a headline like "Santorum slightly in front of Romney", even though the difference between them is statistically meaningless.

Some other media called this a "virtual tie". I hate this term. A "virtue tie" is a tie that really isn't. Alternatively, a "virtual tie" is nearly a tie. Neither sense of the word is applicable here. It is a tie, nothing "virtual" about it. In fact, when the Washington Post prints "Polls show Rick Santorum virtually tied with Mitt Romney nationally" (link), it gives the impression that Mitt Romney is slightly ahead... that story certainly did not come from the Pew poll. By the way, the Post manages to print this article and mention two recent polls without actually citing any numbers!

***

I just have to re-print the following table from Pew's press release here. Reporters frequently ignore the margin of error (again, this Yahoo reporter did well to feature the margin of error in the 2nd paragraph). Because these are polls with very few respondents, the margin of error is plus/minus 5 percentage points or more (for any subgroup being analyzed).

The 30% to 28% comparison was made for respondents who are "Republican or leaning Republican registered voters". This means the margin of error is given by the fourth line up from the bottom, which is plus/minus 5 percentage points. This means that the poll can only tease apart differences of larger than 10 percentage points. Notice that for any result concerning Republican voters (excluding those who said they are leaning Republican), the margin of error is even larger, at plus/minus 6 percent.

Yes, if you think these polls are useless to measure tight races, you'd be right.

 

Statistical fun at the expense of a President

On Dec 28, Cristina Fernández de Kirchner, the President of Argentina, received some bad news. She has been diagnosed with thyroid cancer, and must take a leave of 20 days to undergo surgery (on Jan 4). The discovery of the cancer was a result of routine screening.

The Economist blog reported this unhappy occasion, telling readers that "Ms Fernández is the latest of a series of South American presidents to fight cancer." Perhaps we are supposed to infer that being leaders of South American countries is a hazard to one's life (otherwise, what's the point of writing about it?). The Council of Foreign Relations even picked as a "top trend of 2011" in Latin America "the Region's Presidents' Battle Cancer".

The next day, Reuter's reporter Daniel Wallis told us that Venezuela's President Chavez has taken the correlation as evidence that "the United States might have developed a way to give Latin American leaders cancer."

***

Back in October, Kirchner won re-election by a landslide. Her supporters within Argentina are numerous. On Jan 3, the day before she went under the knife, they held an overnight vigil. According to the Washington Post (link), "At 1 a.m. on the morning of her operation, supporters assembled in various plazas and squares around the country, and remained there until it was finished."

Also on Oct 3, on a Financial Times blog, Jude Webber suggested that President Kirchner has benefited and will continue to benefit in public opinion from her illness:

According to this poll published last month, four days before the routine exam which revealed she had a cancerous tumour, Fernández’s popularity was 67 per cent – well above the 54 per cent she secured in the October elections. After the op, the only way is up, at least initially, pollsters say.

The Washington Post piece mentioned above has another angle. The reporter describes "what’s interesting is the panic that her illness has inspired".

***

It was announced that the three-and-a-half-hour surgery to remove her thyroid gland went successfully. CNN found a supporter who opined: "The truth is I was praying a lot, with all my strength, and now I feel very happy. I think the Virgin has granted a miracle."

Good news arrived, confirming that the cancer has not spread, and the President could get back to work soon.

***

On Jan 7, the doctors conceded that the initial diagnosis was a "false alarm"; Kirchner did not have cancer after all. According to the FT, the false positive only occurs in 2% of cases. So Kirchner was very unlucky.

How does a false alarm occur? According to Dr. Bianco, cited in this ABC News blog, typically, a lump is identified, and if it is large enough, a biopsy is taken, which means cells from the thyroid gland are taken and examined for signs of cancerous growth.

Next, Dr. Braunstein specifically tells us that doctors deliberately cause "false positives":

If there’s about a 20 percent to 30 percent chance those cells are cancerous, many doctors — including Braunstein — recommend removing the thyroid gland.

In other words, there is no simple black-or-white test of whether cancer is present in the cells. This, I suspect, is also due to the fact that only a tiny sample of cells are examined -- and the diagnosis is a general statement not just of the examined cells but of the entire gland. So there is a margin of error.

Based on what Dr. Braunstein is telling us, about 70 to 80 percent of the removed thyroid glands would be cancer-free. The chance of a "false positive' is actually really high (if you have been asked to have the operation)!

How does one reconcile this with the other report that the false positive rate is 2%. I'm not so sure.

***

As a side effect of this surgery, Kirchner would need to take hormone pills for the rest of her life.

 

 

The p word

P stands for pandemic. And this article nicely describes the predicament of policymakers as they grapple with the early stages of a possible outbreak in a new strain of influenza. At best, they are basing their policies on educated guesses, with the emphasis on guessing since data is in short supply.

This situation is akin to the one described in Chapter 2 of Numbers Rule Your World. At this point of the investigation, they only identified one cluster of cases, in the U.S. that can be traced back to pigs. "We don't want to overplay or underplay... we're trying to get that right." according to an official at WHO. Nice goal but unfortunately, unrealistic. The reality is that there are winners and losers on either side of this zero-sum decision. Public health advocates have little to lose from false alarms - they have the "better safe than sorry" mentality. Powerful business lobbies have much to lose from false alarms - and their voices are being heard: WHO has been warned not to call this "swine flu", according to the journalist.

How to know when polls are jokes

Andrew Gelman has two posts (link 1, link 2) about a pollster Doug Schoen who has been making news about his survey of Occupy Wall Street protestors. Doug claimed his firm interviewed 200 people in Zuccotti Park.

Gelman cites the work of Azi Paybarah, who complained that (1) Doug drew ridiculous conclusions that contradict his data; and (2) he seemed to have altered the wording a survey question after the fact. Of course, both offenses make a mockery of polling. If you've read Charles Seife's Proofiness (my review here and here), you'd already have lost all respect for political polls like these.

***

We should examine why Doug Schoen even conducted this poll. The headline of the results was:

On Oct. 10 and 11, Arielle Alter Confino, a senior researcher at my polling firm, interviewed nearly 200 protesters in New York’s Zuccotti Park. Our findings probably represent the first systematic random sample of Occupy Wall Street opinion.

Our research shows clearly that the movement doesn’t represent unemployed America and is not ideologically diverse. Rather, it comprises an unrepresentative segment of the electorate that believes in radical redistribution of wealth, civil disobedience and, in some instances, violence.

Judging from this, I would say the primary purpose of this poll is to compare protestors in Zuccotti Park to "unemployed America".

In other words, the concept is flawed beyond repair.

Firstly, comparison is only possible if you have A vs. B. There is no indication that Doug's firm reached out to the cross-section of "unemployed America" to assess what they believe in. Unless Doug is unemployed America, I'm not sure how they could come to such a conclusion.

Secondly, Occupy Wall Street can never represent "unemployed America". One doesn't need a poll to know that. New York City is not America. People who live in New York City are definitely richer and more Democratic than most parts of America. New York City, as the beneficiary of bailout money, also does not have the unemployment rate of say Detroit. So, a more interesting question is whether Occupy Wall Street represents "unemployed New York City".

This overarching issue is an extension of Andrew's complaint that most of the country are in favor of taxing the very rich, which makes Doug's point that these protestors - who also support taxing the very rich - are radicals look silly. Basically, this poll has no reference point, without which one can not draw any conclusions.

Thirdly - and this is a commentary on the general media coverage rather than Doug specifically, there is this strange idea that if you're not one of the "unemployed America", you have no standing to advocate for "unemployed America". That's like saying a white person cannot possibly support civil rights because you're not a colored person.

Save the Statistical Abstract

Krugman points to this plea from Robert Samuelson to save the U.S. Statistical Abstract. Under pressure from Congress to "save money", the Census Bureau will disband the small team that assembles publications on the statistics of the United States. Apparently, this move cuts 24 jobs and saves $2.9 million annually.

This development is disturbing in many ways:

• The Abstract will simply vanish. They are not killing just the paper version. There will be no online version either.
• Samuelson claims someone at the Bureau made this statement: "[The Bureau] has to choose between its basic job of devising surveys and collecting statistics about economic, social and governmental conditions and the less-important task of publicizing the results." I hope this is a mis-quotation. Presenting and publicizing data is "less important" than collecting data? The priority is totally backwards.
• Terminating this statistical publication is an assault on our democracy. By not publicizing these statistics, the effect is to ring-fence this data so that only specialists will access it. By not organizing them into a publishable state, only specialists will have the time and the expertise to sift through the data and make sense of it. 
• Statistics is an annoying profession to those who find themselves being measured by the numbers. When things are not measured, they will be "guesstimated" with any number of assumptions. Policies will be put in place, and their effects will also be "guesstimated". Accountability becomes a vague concept, infinitely debatable. This does not bode well for effective governance.