Big Data, Plainly Spoken (aka Numbers Rule Your World)

Prestigious college desperately plays down shameful manipulation of statistics

The LA Times (link) made the following comment as it describes the shameful situation in which the Dean of Admissions at the prestigious Claremont McKenna College (#9 on US News ranking of Liberal Arts colleges) inflated the average SAT scores of incoming students in order to manipulate national college rankings:

The collective score averages often were hyped by about 10 to 20 points in sections of the SAT tests, Gann said. That is not a large increase, considering that the maximum score for each section is 800 points.

Not a large increase? Are they wilfully ignorant, or just ignorant? I hope it's not a quote from CMC President Pamela Gann but an embellishment by the reporter. When one interprets whether 10 to 20 points is a "large" increase or not, one must find the right reference distribution of scores.

The maximum score is 800 but that is for individual scores. The 10 to 20 points manipulation is of the average scores of the freshmen class (about 300 students). The distribution of individual scores is much, much more variable than the distribution of average scores. So while 10 or 20 points for an individual may not be material, shifting the average score by 10 or 20 points is fraud of a massive scale.

Let's take a rough guess. According to the College Board, the standard deviation of individual scores is about 110 points (See the footnote on "Recentering" on this page). This means the standard deviation of the average scores of samples of 300 is 6.4 (this is known as the standard error). A 10-point fraud is about 1.6 standard errors. A 20-point fraud is just over 3 standard errors.

It's easier to visualize the scale of this:

Imagine the college's true SAT score average to be at "Z Score" = 0. Think of that as the median value (50th percentile). A 10-point fraud moves the average to 1.6 on the Z-Score scale, and as the diagram shows, that is moving from 50th to 95th percentile! And according to LA Times, that is the lower bound of the alleged manipulation.

Another way to see the size of this manipulation is to look at the average SAT scores for the top colleges (I found some data here but it's from 2004.) For instance, there is only about a 10-point spread between Columbia, Penn, Duke and Rice. Even a few points will shift the SAT score rankings.

***

So, after failing ethics, maybe the College is failing statistics too.

 

PS. [2/1/2012] @rags and I have been discussing what value of standard deviation should be used in the standard error formula. The proper value should be the standard deviation of the SAT scores of typical freshmen at CMC (or similar schools). The number I found is the standard deviation for the entire SAT test taking population so it is an over-estimate. If you find a school-level standard deviation number, please let me know and I'll adjust the computation. I don't think the conclusion would change much though given what we see in the table of average scores by college shown above.

PS. [2/4/2012] If the standard error was over-estimated, then the distribution of average scores would be even tighter than stated. This would make a 10- or 20-point manipulation even more egregious.

Does being underrated have statistical meaning? (I don't have an answer.)

While I was reading Eater's article on "NYC's Most Underrated Restaurants" (link), a thought keeps troubling me. Here, the editors conducted a poll of the readers to discover which restaurants are underrated.

That sounds innocuous enough but something is amiss. A poll measures the average opinion. An underrated restaurant is one in which the average opinion is purportedly wrong. Something about using an instrument for discovering the average opinion to disprove the average opinion doesn't sit right.

***

So I set about thinking how one might mathematically describe the idea of being underrated.

The most obvious choice seems to be a gap between an intrinsic rating and the average opinion of the rating. The intrinsic rating can never be measured, and we use polls to estimate its value (really, a set of values). The underrated restaurant receives a poll rating that is below its intrinsic rating.

Something doesn't sound right about that too. The intrinsic rating simply doesn't exist. There is no higher being to tell us which restaurant is better than which other restaurant.

So, perhaps we can compare a poll rating with a "population" rating. The population rating is defined as what the rating would have been if everyone who cares to rate restaurants is polled. But now the gap between the two ratings is just the error in the sample. If the sample size of the poll is large enough, the concept of underrated (or overrated) would dissolve.

(One could argue that the population rating is the intrinsic rating because if one must find a value for the intrinsic rating, a good choice would be the population rating.)

***

Maybe underrated is a function of a subset of the population. An object is considered underrated by a subset of a population if the rating by that subset is substantially lower than the population rating. Put simply, it's just a cliche of people, like readers of Eater, voicing their dissent against the average opinion.

But one can find such a subset in any sample since there must be people who rate a given restaurant lower than average. Can there ever be a consensus of an underrated something?

Let me know if you have any thoughts on this.

 

Not taking numbers at face value, even if they come from Google

Thanks to @jeanniecrowley for pointing me to Rocky Agrawal's wonderful piece on how Google - and by extension, many hyped-up tech companies - abuses statistics to deceive the public. (link here).

The post is worth reading in full. The highlight for me is this bit:

In July, [Google CEO] Page claimed that the service had 10 million users who shared 1 billion items a day. That sounds incredibly impressive. But let’s do the math. That would mean that the average user was sharing 100 items a day...

So how did we get to that number? Well, it turns out Google was counting every potential recipient of that message. A single message from Scoble today would count 240,000 times toward that number. That’s preposterous.

There are many cases similar to this one where one can easily spot mischief just by taking a skeptical position. It's a simple division to get to an average of 100 items per user per day. If one knows a little bit of statistical thinking (say, stuff from Chapter 3 of Numbers Rule Your World), one quickly realizes that not all 10 million users can be like the "average user"... in fact, with so many users, there would be lots of inactive users, who would share zero items per day. For every non-sharer, there must be some active user who shares 200 items per day to keep the average at 100.

What Agrawal is arguing is that even the maximum sharer ("Scoble") probably didn't share 100 items a day, and the maximum can't be smaller than the average.

***

I like to call these "true lies". Under certain assumptions, exclusions and definitions, one can certainly justify that these statistics are "true" but the effect on consumers of such statistics is to mislead. Frequently, one needs to examine a set of statistics side-by-side to fully understand the data.

Most importantly, one should start with why we are computing the statistic.Take the Facebook "Like" button, which is touted in a lot of places as a measure of marketing success. I just went over to the McDonald's U.S. Facebook page. It shows about 14 million Likes. What does that statistic mean? Since Facebook reports about 150 million U.S. users, does this mean only less than 10% favorability? Who are these people who "like" McDonald's? Does this reflect the success of social-media marketing in gaining more fans? Or are these 14 million hard-core fans who have always loved McDonald's and now they take the opportunity to advertise their love to the world? Do we expect the act of "Liking" to generate additional revenues for McDonald's? If so, how so? What a surprise - the number of "Likes" is insufficient to answer any of these questions.

 

 

The hysteria machine

Our mass media is a hysteria machine. In Chapter 5 of Numbers Rule Your World, I looked at the news coverage in the aftermath of jet crashes. What's happening in the wake of the cruise ship grounding in Italy is following the same pattern.

***

Here is an article talking about "10 other horrifying cruise ship disasters". (link) This is just like tables of the 10 worst airline disasters in history. The #1 on the list is unsurprisingly the Titanic. That's really a laughable comparison: 1500 dead and 700 survivors versus 4000 survivors and 10-30 dead. Unless the point of this article is to tell us how much safer travel is today compared to a century ago.

***

Here's an industry insider advising readers "how to pick a cruise line for safety". (link) Given the scarcity of accidents (see previous post), it would seem pointless to spend a lot of time choosing between cruise lines specifically for safety.

 

Is the media reporting irrational behavior or giving bad advice?

This article written by BusinessWeek (link) about the financial impact of the recent cruise ship accident in Italy is hard to decipher. The title says "Carnival wreck timing may exacerbate loss in cruise business".

I can see two possible causes of such financial losses, based on my reading of the somewhat incoherent piece:

1) This gigantic ship (that carries more than 4,000 passengers) is super-expensive (costs hundreds of millions of dollars), and somehow it is damaged beyond repair, and thus the cruise company would have to refund all of bookings that have been made for the upcoming summer season. This is plausible but is it really true that the ship can't be repaired, or that the company has no backup plans whatsoever, or cannot split the tours into multiple smaller boats, etc.?

2) This accident will cause potential customers to run away to other cruise ship operators, or cancel their cruises altogether because of fearing future accidents.

The second point seems to be implied although it wasn't directly addressed. If people are really cancelling their vacations because of this accident, it is an irrational behavior, similar to avoiding air travel after a crash occurs (something I discussed in Chapter 5 of Numbers Rule Your World.)

I found this website that collects statistics on cruise ship incidents. According to this table, about 10-20 people are "thrown overboard" each year in the last 10 years. So far, 5 people have been found dead and another 15 are missing in this one accident, quite clearly the worst in many years. Each year, almost 20 million passengers worldwide go on cruises, according to this Cruise Market website. The chance of dying from a cruise is really small, and this accident doesn't change this fact.

When the media publishes these articles, its impact is more than reporting irrational behavior. It may also encourage irrational behavior as some readers may take this as advice to cancel their vacations.

 

 

PS. Reuters (link) has more complete quotes, and there is no doubt that point 2 above was the intended message, that the cruise industry is thrown into turmoil because of this accident. The bank analyst claimed this is a "PR nightmare", that this may have long-term impact on the "Carnival brand", etc. It can only be a PR nightmare if readers believe these analysts.

Statistical fun at the expense of a President

On Dec 28, Cristina Fernández de Kirchner, the President of Argentina, received some bad news. She has been diagnosed with thyroid cancer, and must take a leave of 20 days to undergo surgery (on Jan 4). The discovery of the cancer was a result of routine screening.

The Economist blog reported this unhappy occasion, telling readers that "Ms Fernández is the latest of a series of South American presidents to fight cancer." Perhaps we are supposed to infer that being leaders of South American countries is a hazard to one's life (otherwise, what's the point of writing about it?). The Council of Foreign Relations even picked as a "top trend of 2011" in Latin America "the Region's Presidents' Battle Cancer".

The next day, Reuter's reporter Daniel Wallis told us that Venezuela's President Chavez has taken the correlation as evidence that "the United States might have developed a way to give Latin American leaders cancer."

***

Back in October, Kirchner won re-election by a landslide. Her supporters within Argentina are numerous. On Jan 3, the day before she went under the knife, they held an overnight vigil. According to the Washington Post (link), "At 1 a.m. on the morning of her operation, supporters assembled in various plazas and squares around the country, and remained there until it was finished."

Also on Oct 3, on a Financial Times blog, Jude Webber suggested that President Kirchner has benefited and will continue to benefit in public opinion from her illness:

According to this poll published last month, four days before the routine exam which revealed she had a cancerous tumour, Fernández’s popularity was 67 per cent – well above the 54 per cent she secured in the October elections. After the op, the only way is up, at least initially, pollsters say.

The Washington Post piece mentioned above has another angle. The reporter describes "what’s interesting is the panic that her illness has inspired".

***

It was announced that the three-and-a-half-hour surgery to remove her thyroid gland went successfully. CNN found a supporter who opined: "The truth is I was praying a lot, with all my strength, and now I feel very happy. I think the Virgin has granted a miracle."

Good news arrived, confirming that the cancer has not spread, and the President could get back to work soon.

***

On Jan 7, the doctors conceded that the initial diagnosis was a "false alarm"; Kirchner did not have cancer after all. According to the FT, the false positive only occurs in 2% of cases. So Kirchner was very unlucky.

How does a false alarm occur? According to Dr. Bianco, cited in this ABC News blog, typically, a lump is identified, and if it is large enough, a biopsy is taken, which means cells from the thyroid gland are taken and examined for signs of cancerous growth.

Next, Dr. Braunstein specifically tells us that doctors deliberately cause "false positives":

If there’s about a 20 percent to 30 percent chance those cells are cancerous, many doctors — including Braunstein — recommend removing the thyroid gland.

In other words, there is no simple black-or-white test of whether cancer is present in the cells. This, I suspect, is also due to the fact that only a tiny sample of cells are examined -- and the diagnosis is a general statement not just of the examined cells but of the entire gland. So there is a margin of error.

Based on what Dr. Braunstein is telling us, about 70 to 80 percent of the removed thyroid glands would be cancer-free. The chance of a "false positive' is actually really high (if you have been asked to have the operation)!

How does one reconcile this with the other report that the false positive rate is 2%. I'm not so sure.

***

As a side effect of this surgery, Kirchner would need to take hormone pills for the rest of her life.