Big Data, Plainly Spoken (aka Numbers Rule Your World)

One averages or two?

While averages are a great way of summarizing data, in practice the key concern for statisticians is how many averages? Is one average too simplistic? Would one average for each subgroup be more informative? This is the principal concept discussed in Chapter 3 of Numbers Rule Your World.

Consider this paragraph that summarizes the finding of a recent British study linking the consumption of red meat to the risk of bowel cancer (my italics):

People who consume more than about 90 grams (3.2 ounces) a day are at greater risk of getting bowel cancer, which kills 16,500 people in Britain every year, the Department of Health said, urging them to reduce their intake to 70 grams. Forty-two percent of men eat more than 90 grams every day on average, compared with 12 percent of women.

Notice that men as a group differs significantly from women in terms of consuming a lot more red meat. And also note that the guideline prescribed one level of consumption (70 grams per day) for both genders.

It is also the case that men are much more likely to get bowel (colon or rectal) cancer than women in the UK, as seen here: 

So, you wonder if in fact there should be two guidelines, a stricter rule for men than for women.

By issuing one optimal level of consumption for both genders, that is, one average instead of two, the health authority in the UK has made a statistical assumption of "additivity". They are stating that the effect of red-meat consumption is independent of gender: given the base rate of cancer for each gender (which is far from equal as shown above), an equal increase in consumption of red meat causes an equal increase in risk of bowel cancer. This condition is also known as "no interaction effect".

Is this plausible? Medical experts can tell us. The point for consumers is to recognize that an underlying assumption has been used when issuing this guideline.

***

It is very instructive to examine the possible "impact" of this change in guidelines.

From the Cancer Research UK website linked above, we learn that there are about 30 million men in the UK, so about 12-13 million consume more than 90 grams of red meat a day, the level of consumption which the Bloomberg article identified as posing significantly higher bowel-cancer risk.

However, only 20,000 cases of such cancer is reported per year among men (with about 16,500 deaths across both genders). So, it is a certainty that almost all of the 12 million men who consume a risky amount of red meat will not develop bowel cancer even if they don't change their eating habit. (It's a kind of gamble who among these millions will be the unlucky ones.)

So, how many people might benefit from this new guideline?

Among men, we know the upper limit is 20,000 per year. The reality is much smaller than that. Firstly, red-meat consumption is only one of many risk factors for bowel cancer and so a portion of the 20,000 are surely men who do not currently consume more than 90 grams per day; for these men, the new guideline is irrelevant. Secondly, some of those who reduce their consumption now will still get bowel cancer because the statistics is only promising reduced odds, not certainty.

Thirdly, the maximum benefit assumes perfect compliance. In practice, not all men will choose to follow the guideline. In fact, the most at-risk people are the most likely to be eating more than the optimal amount of red meat, and they are precisely the ones who are the least likely to comply with the guideline. The overall risk of bowel cancer is very low which also deters compliance. So, a big haircut on the potential effect for non-compliance.

That may not leave us with many saved lives. And the possible effect on women is still smaller because the number of cases for women is smaller.

Nobody says curing people is easy!

 

 

Why statisticians ignore rankings

Outsourcing this to Matthew Handy, who wrote an excellent piece on how to think about ranking or league tables. From the Significance blog.

Matthew makes really good points. I'd add the following: the clear sign that this type of rankings is flawed is the inexplicably large year-to-year variability. Things like the quality of a school do not change abruptly. Schools may climb up the rankings slowly over decades but when the rankings show a school going up and down a few ranks every year, what we are watching is the imperfection of the ranking model, not any evidence of real changes in quality. (Technically, we say these changes are not statistically significant.)

So, the only use of these tables is to read them in (very) large chunks, say in quartiles. Imagine if we all do that, the sponsors of such ranking studies would have to find a different line of work!

The Google Bing showdown

I was going to write about the showdown between Google and Bing that shook up the tech world a few weeks ago but that would be superfluous as John Langford has already written an excellent post, which coincides with my point of view.

Let me provide the background to the dispute, and then leave you in John's capable hands.

***

Around 1997, Google emerged from the labs of Stanford University, with a revolutionary new algorithm to rank Web pages according to their relationship to other Web pages. In brief, a page is given a higher rank if it has more "friends" and if its friends are more "influential". The technology is considered a scintillating example of creating a business on top of some probability/statistics theory. It has since become the dominant player in the search engine business.

In 2009, Microsoft launched a "new" search engine branded Bing. Microsoft has suffered in the shadow of Google for many years in the search engine sector, but despite the long odds, Bing has taken some market share (although not very much) from Google (see link). Bing, of course, also has an algorithm underlying it, and has the advantage of learning from what all others have done in the intervening years.

Recently, Google engineers publicly mocked Microsoft engineers for "plagiarizing" their search engine. They charged that Microsoft collects data on what users are searching for and where they landed after the searches (maybe via Internet Explorer, maybe via Bing toolbar), data then used by the Bing algorithm to decide which search results to display. The Google engineers described a sort of sting operation to demonstrate the effect (on made-up queries and web pages).

From the perspective of someone using a search engine, one might be excited by Bing's incorporation of this new source of data. There is no doubt that such relevant information will improve the ability the search engines to rank web pages. Why wouldn't one use this data? (Whether Microsoft should collect this information from users quietly is a different matter, as John pointed out.)

From the perspective of an engineer developing algorithms, one might feel disturbed that one's work can be so easily "reverse-engineered". One could see how the engineers felt "robbed in daylight". This issue is larger than Google or Bing.

An analogous though not exact situation is the attempt to "repair" or "trick" credit-scoring algorithms, as described in Chapter 2 of Numbers Rule Your World, where I explain why it is often better to keep these algorithms locked up. Public disclosure may not be in the public's best interest. Pretty much any algorithm can be gamed if it is not a "black box".

***

Read John's take, and tell me what you think.

Facebook's search for a business model

It was obvious from the start that Facebook, and pretty much any of those Web 2.0 / Social Media businesses, must pursue a bait-and-switch strategy: first, build a big community by providing a well-designed service for free; next, invite the marketers and advertisers into the party. The challenge is how to do this without killing the goose. Twitter is next in line to cross this line while Facebook has been experimenting with mixed success in the past year or two.

The media has been selling us a good story about how great Facebook's business is. We have been looking for evidence to assess the credibility of this assertion. (See previous posts here and here.)

A guest post at ReadWriteWeb offers Peter Yared's view on what sort of marketing campaigns works and what doesn't on Facebook. It's one person's opinion but interesting nonetheless. The list of what doesn't work is much longer than what works.

In short, the post confirms that Facebook users want marketers to stay mostly in the shadows, and engage in interactions that last seconds, not even minutes. They also want to get something in return for any "engagement", which makes you wonder how marketers intend to get payback for the investment in these campaigns.

***

You may want to poke fun at the spectrum of marketing speak used in the article: "only one engagement feature per tab", "sweepstakes have ... no viral uptake", "the campaign being graded on the number of engagements", "the apps... got to be on brand", etc.

The trap of missing labels

As I have explained before, there are solid reasons to apply statistical adjustments to data. But what should worry us is the practice of not labelling adjusted data as such. For example, the unemployment rate commonly referenced does not count so-called "discouraged" workers; the inflation rate excludes so-called volatile components, such as gas prices; a lot of economic statistics are seasonally adjusted.

Missing labels presents an easy trap of drawing conclusions as if the data were unadjusted. For instance, we frequently hear people explain away bad unemployment numbers in the fall by seasonal reduction in summer jobs. This makes sense if the unemployment numbers were not seasonally adjusted. The seasonal adjustment already accounts for the effect of summer jobs so that explanation doesn't fly.

***

So, today I saw this piece by Gregory White (on Business Insider) in which he makes this claim:

The San Francisco Federal Reserve just released a report that suggests the natural rate of unemployment in the U.S. (also sometimes called NAIRU) is going up and may actually be 6.7%. That means, relying on last month's 9% unemployment rate, we're only 2.3% away from the natural rate.

This interpertation has always bothered me -- and if there are economists reading this blog, maybe you can tell me if I make sense here.

The natural rate of unemployment seems to account for those "unemployable" people, people who don't have the right skills, cannot find a job, are not willing to work at the prevailing wage level, etc. To me, the unemployment rate of 9% as typically reported (not the U-6 number) already excludes these types of workers. So, you can't do 9% - 6.7% and claim that our gap is 2.3%. In fact, our gap is almost all of the 9% because the adjustment already takes care of the "natural" rate.

Update: podcast, reviews, Japanese edition, etc.

Since my last update, a few items have been published related to the book.

***
I had a delightful conversation with Joe Dager of Business 901. The podcast can be heard here (just over a half hour). Joe's audience are business people interested in Six Sigma and analytics.

***

The American Statistician came out with a positive review. For those who have a subscription, the review is here.

David Weinberger reported that Brian Kernighan -- yes, the K in K&R, the C bible -- mentioned Junk Charts in his discussion of "why (in)numeracy matters".

***

I happened upon a tweet that said the Japanese edition of Numbers Rule Your World is coming out on Feb 20, 2011. I hope that's correct.

PS. It's been confirmed. Here's the Amazon link to the Japanese edition. And the cover.

***

You can meet me live at various events, which I keep updated on the right column of the blog.

 

Why should we care about these studies?

Reader Joe M. points us to this article from Slate (via Kottke).

Supposedly we are to believe that our success/happiness/fate/etc. is predetermined by the alphabetical rank of our name. This was something that the Freakonomics team has already pitched to us years ago, when they recited the claim that you have a "higher chance" of winning the Nobel Prize if your surname begins with an "A" than a "Z".

I really don't care enough about these studies to give them a serious look. Just on the surface, I have many doubts about this line of research:

• In places such as China, names are not made up of alphabets. What is the relevant hypothesis for these countries? If you can specify the hypothesis without first looking at the data, and your hypothesis proves correct, then maybe I'll pay some attention.
• Lots of things make people happy or successful. The magnitude of the effect matters. It is very likely that if it exists, this "name effect" is extremely small - you'd only be able to see it if you control for every other factor. It is very likely that many other factors have a larger impact on your happiness and success than this "name effect". So, if you want happiness or success, you should spend your time and energy on those other things, rather than on naming babies. This is the classic fallacy in statistics of confusing "statistical significance" with "practical significance".
• I haven't seen the following type of test of intervention: have randomly selected "Z" named kids change their names to "A" names, and track to see if they end up being more successful than those who did not change their names.
• The above test is unlikely to yield the result we want. If the "name effect" is strong, it should be noticeable and if noticed, it should lead to a large shift in naming conventions; we should see parents preferring "A", "B", "C" names. We don't see this. Why?

***

For those who want to dig deeper, the following excerpt on how the researchers analyzed various studies makes for fascinating reading: (italics are theirs, bolding by me)

1.) Business school students were invited by e-mail to receive up to four free tickets to attend a basketball game. The students were told there was a limited supply and that tickets would be given away on a first-come, first-serve basis. Seventy-six students put in requests before all the tickets were spoken for. Average response time was 23 minutes (students love getting free stuff!). Response time correlated negatively with alphabetic rank. In other words, students at the end of the alphabet put their bids in, on average, earliest.

2.) E-mails were sent out to adults offering them $500 to participate in a survey. Average response time was between six and seven hours. The same negative correlation between response time and alphabetic rank was observed, but only when the researchers looked at the names the respondents were born with. When Carlson and Conard looked at married names or names changed for some other reason, the correlation dwindled to insignificance. This, they conclude, demonstrates that the "last name effect" derives from "a childhood response tendency." Only people who grew up with a name at the back of the alphabet demonstrated truly Pavlovian responses to the $500 offer.

3.) Students of drinking age in a wine-appreciation class were told, verbally, that they could receive $5 and a free bottle of wine if they participated in a wine survey. Average response time was about six hours. Again, a negative correlation was found between response time and alphabetic rank. The R-Zs responded, on average, about an hour faster than the A-Is. Carlson and Conard also compared the responders as a whole with the students who didn't respond. The R-Zs were more likely to be responders.

4.) Undergraduate students who were paid to participate in this and other studies were asked to imagine the following situation: You need a new backpack and as you pass a bookstore you see that it's selling brand-name backpacks for 20 percent off "while supplies last." But you don't have your wallet with you! It would take you 15 minutes to go home, get your wallet, and come back to the store. Do you do that right away? Yet again there was a negative correlation between (hypothetical) response time and alphabetic rank. The R-Zs were more likely to say they would trudge home and back to take advantage of the sale.

Notice that in each case, the researchers went into the study with  one hypothesis and came out with a modified conclusion *after* looking at the data. It is almost surely the case that the hypothesis was either left vague ("there is negative correlation") or was abandoned in the face of contradicting data ("A-K names are different from L-Z names"). The result would be more credible if the researchers had proposed that A-P names would be different from R-Z names *without* looking at the data first. Same thing with the birth name versus married name.

By forming and also verifying hypotheses on the same data, these researchers are running a very high risk of false positive findings. In the A-R finding, for example, they have in effect simultaneously checked at least 25 hypotheses (bisecting the group at A, at B, ..., at Y) and possibly more (since they may not have limited themselves to binary partitions). In typical statistical practice, we would need to use a p-value much higher than the conventional 5% to "accept" such findings. Put differently, if the conventional 5% were to be used, the chance of a false positive is not 5% as advertised but some shockingly large number that you'd be ashamed to report. (For those curious, look up the problem of multiple comparisons.)

Also, I worry about publication bias. We have here four examples of this purported effect. What if researchers have looked at a hundred other examples, and those didn't show this effect, and were shoved away?

Contrary to Freakonomics, I do not recommend changing your name to Aaron or Audrey any time soon.

Keeping one's appetite after touring the sausage factory

For those wanting to a glimpse the sausage factory that manufactures unemployment statistics, this New York Times column paints a pretty good picture.

The article even comes with a picture:

If this post is on my sister site Junk Charts, I'll say something about the chart itself. But this isn't that place. (Ignore the darker section of the chart on the right side.)

First, look at the red columns in the bottom of the chart; these represent so-called "benchmark revisions" to the job gain/loss count. "Benchmark revisions" are, to put it politely, adjustments to prior adjustments to the job statistic. These adjustments are used to correct for things like seasonality and sample biases. (See my previous post on seasonal adjustments, and other adjustments.)

It turns out that certain adjustments -- typically these are based on the assumption that past experience approximates the future -- are unreliable, especially while we are experiencing a deep recession the magnitude of which we haven't seen for a long time. Thus, when the statisticians look themselves in the mirror at the end of the year, they see white hair. They find that their prior adjustments don't make sense in light of new data, and thus have to go back to restate the previous numbers.

In some instances, the errors were extreme. In the early part of 2008, for instance, the job losses were understated by roughly 200,000 per month but looking at the top chart, we learn that the (adjusted) job losses reported at the time were below 100,000 so the error rate exceeded 100%. Ouch.

For me, this suggests that such adjustments should be suspended in extraordinary times. Adjusting data poorly is worse than not adjusting at all. Notice that seasonality is much more predictable than sample biases (such as the so-called birth-death model that tries to predict small business growth), and thus does not contribute much to the error. So don't suspend every kind of adjustments.

***

Floyd Norris correctly identifies a key problem with public consumption of economics statistics:

But it is the first number for each month that people remember. The revisions over the next two months will be barely mentioned in news stories. When the final numbers are posted, they will seem like ancient history.

Given that Norris understands this problem, I find it amusing that he does not conclude that the media needs to do a better job of reporting. Perhaps they should warn readers that the first number is highly inaccurate; perhaps they should highlight the revised numbers in each report, rather than the most current, highly inaccurate number.

Presumably he wrote this story because he thinks this is a serious problem. But alas, the tour of the sausage factory has left his appetite for sausages intact.

Further thoughts on the Facebook business model, and the challenge of Web analytics

I have often grumbled about "story time!", the practice of spinning grand stories based on tiny morsels of data. It's not that I disapprove of story-telling per se -- it is that the story-teller has got to find evidence to support his/her stories.

A few days ago, I dissected the Trefis financial model used to support a $100 billion valuation for Facebook. A handful of aggressive assumptions must be believed to make it happen. So it is very pleasant to find in Business Insider some actual data to help us assess the credibility of some of these assumptions.

This chart gives the clickthrough rate (CTR) and cost per click (CPC) of Facebook ads.

For those not in this industry, CTR is the number of ads that get clicked on divided by the number of ads shown to Facebook users; and CPC is the average dollars paid by advertisers to Facebook for each click on their ads.

The chart does not provide per-advertiser data. Instead, advertisers are grouped by the industry they are in (health care, internet, etc.), and the aggregate results are shown.

The one thing that should jump out at us is the range of clickthrough rates: it's mostly in the range of 0.01 to 0.1. (The last one -- Tabloids and Blogs -- seems mislabeled, and the last two rows are sufficiently different from the rest that one would want to check the numbers again.)

Mind you, that is 0.01% to 0.1%. What does 0.01% mean? Yes, that's 100 clicks per 1 million ads shown to Facebook users. (My friend Augustine has long ago pointed out Facebook's abysmal metrics, relative to other advertising platforms. Look here for his perspective.)

***

Now, put yourself in the shoes of say Pfizer showing ads to Facebook users. Clicks don't equal revenues. Only some proportion of clicks would turn into sales. For illustration, say 5% of clicks lead to sales. With 100 clicks, they get 5 sales. They have to show 1 million ads to get 100 clicks. The clicks cost them $130 according to the data in the chart. If the value of each sale is more than $130/5 = $26, then Pfizer just about breaks even on the ads. 

What's on the advertiser's mind? 

One strategy is to flood Facebook with ads. More ads mean more clicks, even if the clickthrough rate is tiny. Perhaps unexpectedly, this sort of tactic works only to a limited extent. It bumps up against the law of diminishing returns. The clickthrough rate typically falls say when you double the number of ad impressions.

Another strategy is to use statistical models to selectively show ads only to Facebook users most likely to click on them. This raises the clickthrough rate. What it doesn't solve is the "quality" of Facebook users, or put differently, their tendency to pay attention to advertising.

***

Now, let's fancy yourself the person trusted to build these statistical models. You are trying to predict who's going to click on a given type of ad and who's not. What you have at your disposal is historical data on who got shown what ad, and whether they clicked or not. (You would typically want to grab any other data you can get your hands on, such as what the user has been doing on Facebook recently. You can get more creative, such as what Facebook has been secretly doing, described here.)

Let's say you are given the data on 1 million ads that were displayed. According to the above, you will find 100 clicks in this data... and 999,8999 999,900 non-clicks. Now, assume that you discover that there are some commonalities among the 100 users who clicked on the ad. Say, 50 out of the 100 signed on to Facebook after midnight, and live in the East Coast. That's a very strong signal.

Now, how many users would you find among the non-clicks who signed on after midnight and live in the East Coast? Oops, there would be multiples more than 50 such cases who did not click on the ad. This is all due to the tiny clickthrough rate.

That, in brief, is the challenge of Web analytics. If this excites you, there are lots of opportunities out there.