Big Data, Plainly Spoken (aka Numbers Rule Your World)

Know your data 4: Facebook is everywhere

The key paragraph in this Wired article is at the end:

users’ off-Facebook activities are basically part of the Facebook ecosystem thanks to “Likes” published all over the Web. If you click the Facebook Like button on any given site, that data is transmitted to your own Facebook profile and can be promoted by marketers in ads to your friends.

So it is that there are (digital) spies following us everywhere, recording our conversations, and sending reports to marketers. Facebook may be the biggest such operation but Google and others have similar ambitions: that's why they encourage us to use the same logon credentials.

***

That's the scary part. Here's why it's happening.

Most Web businesses charge nothing for their services, and their users also appear to value the services at zero -- with very few exceptions (typically businesses that cater to niche, upper-income audiences), experiments to charge more than zero have failed. This could be partly due to low barriers to entry: someone else can always put up a similar website and, at least initially, decide to forego making any revenues.

The problem is quite intractable. If users can get the service for free today, they have no skin in the game. They will move on to a competitor if you start charging. Also, they are trained to expect the service for free.

But businesses need to turn a profit. So they have turned to advertisers. Advertisers want to see return on their investment, and since the Web is data-rich, they want to see direct actions. Gone are the days of TV and radio advertising where they are willing to pay for proverbial eyeballs, estimated using surveys. Nowadays, they can, and want to, see actual people clicking on their ads, and better yet, actual people spending money after clicking their ads. They are willing to pay a lot more money for advertising space if they have confirmation that it is effective.

Facebook is desperate. It is different from Google. A Google user is searching for something, and if an ad is relevant, the user will click on it. The Facebook user is typically chatting with friends; not surprisingly, advertisers have not been impressed with the effectiveness of Facebook ads. That's why they are trying to insert these ads in to our conversations. We'd feel like a waiter standing next to our table at a restaurant, listening to our private conversation, and then inserting himself into the conversation to sell us the special whatever of the day.

It turns out most of Facebook conversations are not information-rich. Most of the chitchat is just that. So, Facebook wants to know our habits and likings. They want to know what we are doing when not chatting online. So they set up this network of feelers around the Web, the Like buttons. On the one hand, these sell convenience to the users and their communities; on the other hand, they compile profiles of users, secretly, that can be sold to marketers.

Will this strategy succeed? It's hard to tell. To what extent will Facebook users tolerate this form of digial sleuthing? Facebook's main product is convenience, and community; inserting ads compromise the product. How much private space are Facebook users willing to give up to be a part of this community (without paying an entry fee)? Should Facebook be required to disclose how it is collecting and using our data? Will another competitor emerge to take the users away?

 

A poorly-told story adorned with data

Reader John M. sent in this link, so we make it three days in a row: another Times article which pretends to provide data insights but in fact just spins a story.

***

The first thing that bothered John was the title of the article. At first, I didn't know what he was talking about because the headline I saw was "Parenting by Gays More Common in the South, Census Shows".  I was a little mystified. How can one object to a Census finding? Later, I discovered that the same article has an alternative title, as shown below:

Story time!

Compare the header to the lede shown below the header. If Jacksonville were the South, and population size were hospitality, then well, the wine came from those grapes. Poring over the entire article, I could not find a single statistic from the Census that provides a direct measurement of "welcoming"-ness. It would be great to know the distribution of gay parents across the country, the proportion of gay households, survey results of gay people, and so on.

***

I'm not saying Ms. Tavernise did not have an engrossing human interest story on her hands. I just don't think she needs to drag Census data into the narrative. Doing so weakens the story because the data are not relevant.

One can summarize the whole story thus: Southern states (and using Florida as an example) have become more welcoming to gay families in recent years. Even some churches now welcome gay worshippers. These families still face barriers, such as economic hardship, lack of legal status, and awkward situations in which they lie about their sexual identities. Many of the gay parents have kids, often from their prior heterosexual marriages.

Whenever "data" appear, the narrative loses focus. For example:

Gay couples in Southern states like Arkansas, Louisiana, Mississippi and Texas are more likely to be raising children than their counterparts on the West Coast, in New York and in New England.

The pattern, identified by Mr. Gates, is also notable because the families in this region defy the stereotype of a mainstream gay America that is white, affluent, urban and living in the Northeast or on the West Coast.

I think mainstream gay America is also stereotypically single so saying gay couples don't fit the stereotype is not telling us much. Besides, maybe Arkansas, Louisiana, Mississippi and Texas are less white, less affluent and less urban than the Northeast or the West Coast, regardless of sexual orientation. None of these speak to Southern hospitality towards gays.

***

One particular statistic ought to be used more carefully:

About 32 percent of gay couples in Jacksonville are raising children, Mr. Gates said, citing the 2009 Census data, second only to San Antonio, where the rate is about 34 percent.

It's hard to know how high these numbers are without context. (For example, this report (PDF link) tells us that overall, 20 percent of gay couples in the U.S. have kids.) Does this show that gay couples in the South are more likely to raise kids? Or does this show that gay couples are less likely to be "out" in the South, and one of the main reasons for coming out is to get benefits for the kids?

The point is that a large proportion can be due to a large numerator or a small denominator.

 

 

 

The New York Times publishes another advertisement as news

The New York Times strikes again. Here is another article which makes you wonder if it is a press release by companies selling personal DNA test kits, or is it serious science/health reporting?

The article claims that there are two new studies which support removing restrictions to the sales of personal DNA test kits to the public. It turns out the studies are far from convincing (the first one is utterly worthless while the second one is hard to judge based on what is reported). Also, the argument has nothing to do with science. Egregiously, the entire article contains not one statistic even though the argument is purpurtedly supported by "new research", "two separate studies of genetic tests".

***

First, let's look at the argument:

Most people say they'll pay for genetic tests even if the predictions are sometimes wrong, and most people don't seem to be traumatized even when they receive bad news.

Later, we are told that:

Most people [surveyed] said they'd be willing to spend money out of their own pocket for the test.

So the Times thinks that it is ok for medical tests to tell lies just because people don't care. And that if people are willing to pay for something, they should get it. You'd have expected to read about science but that's not what we get. For example, the entire article says not one word about how predictive DNA information is for various types of diseases. (I don't have first-hand knowledge but I suspect that for most diseases, genetics is only mildly predictive and thus, this data is likely to be worth little.)

***

When we think about the argument, we have to wonder whether the studies are done right. It's puzzling why people who pay hundreds of dollars for tests would not care that the tests tell lies.

There are two types of lies (Chapter 4 of Numbers Rule Your World): being told one is at a genetic risk for some disease when in fact one does not have elevated risk for it; and being told one is not at risk when in fact one has elevated risk for some disease. From the perspective of the buyer of this test, the first lie is much easier to detect than the second lie: if you're told you have risk of colon cancer, you can take a screening test to confirm whether you have it or not.

Any test, including these DNA test kits, can be tuned to allow more of the false positive errors, or more of the false negative errors. This also affects the perception of customers to the product since the costs of the two errors are asymmetric. So, we can't interpret these studies without knowing how they tuned the tests.

But we don't even need to go there. The first study has a list of glaring problems that should lead anyone to discount the findings immediately:

•  the participants are 100% "self-selected", people who make a choice to buy DNA test kits; recall that these people then tell us they like the DNA test kits. (The Times reporter noted this half way through the article but apparently this flaw did not stop him from using this study as a key source.)
• the study was done by the Scripps Translational Science Institute, whose mission statement says it "aims to replace the status-quo of one-size-fits-all-medicine with individualized health care that is based on the known genetic factors influencing health and disease and that takes advantage of advances in digital technology for real-time health monitoring."
• the test kits used in the study was sold at a "steep discount" to participants by Navigenics, a company that sells such tests. The reporter tells us that such vendors want to get doctors out of the way so that these tests could become less expensive to the patients but again, knowing this does not affect his opinion of the study.
• the study has no control group, or even any group for comparison at all. It looks at 2000 people who bought the product at a steep discount. (How did it get published in the New England Journal of Medicine?)

And those are just glaring flaws based only on the information available in the Times article.

***

What most disturbs me about this article is the language used by the reporter, who drops the pretense of being an objective observer. It is the voice of one who has an agenda; these are exactly the words that one might find in the marketing materials of STSI or Navigenics. Here are some samples:

Members of a national advisory committee concluded that personal DNA testing needed to be carefully supervised by experts like themselves.

What if the would-be guardians of the public overestimated the demand for their supervisory services?

Supervised? Experts like themselves? Would-be guardians? Demand for supervisory services?

In two separate studies of genetic tests, researchers have found that people are not exactly desperate to be protected from information about their own bodies.

Protected from information about their own bodies?

For now residents of New York still can't be trusted to buy these tests directly.

The paternalists argue that it's still unclear how to interpret some of these genetic tests.

Can't be trusted? Paternalists? 

And none of these quotes is presented as a quote of someone. So, pray tell whether the Times is a news organization or a billboard company selling ad space.

Everything is possible, say some

A business proposal disguised as a piece of reporting slipped through the editorial filters at the New York Times recently, which allows us to discuss how one might look at business proposals, or marketing plans.

The article in question "reports" the "projection" by some previously unknown "research" firm that the "value" of Facebook (the company) "could" reach $125 billion. This would be 2.5 times the value paid by the most recent round of investors, led by Goldman Sachs. The price paid by these investors is 25 times the annual revenues (which is not an official number but one that is being whispered about). Further, whispers have it that the price-to-earnings ratio would be close to 100.  (Just for comparison, when Google went public, investors rewarded it a valuation of 40 times earnings.) 

***

Why was this article published by the Times? It's one-sided, not authoritative, uses a sample of one, and contains no insight or analysis. In fact, it reads like a press release by the researcher known as Trefis. Worse, as I show below, the assertion is breathtakingly foolish.

My favorite line is that Trefis thinks that a valuation of $100 billion is "certainly possible". Is it possible to be "possibly possible" or "probably possible"? Or maybe "impossibly possible"? Trefis is "certain" that such a number is "possible" but not certain of that number, only that it is possible. My head hurts thinking about what they might actually mean by "certainly" possible.

***

As for the "math", Trefis claims that there are four developments that would push the valuation to $125 billion:

• ad revenues per page double
• page views up by 50%
• share of search market hits 10%
• game revenues per user double.

The first thing to note when reading this sentence is whether we should attach "AND" or "OR" between each condition. Do they need all four conditions to happen to justify the forecast or just one of four?

Usefully, an interactive graphic is included in the article, a quick check of which shows that changing one condition does not push the value to $125 billion. Here, I doubled the game revenue per user which expands the value to $54 billion, a far cry from $100-125 billion. (The Trefis value of $45 billion is what their base case is without the above four assumptions.)

This means investors would have to buy the "certain possibility" that within the next 5 years or so, a bunch of things will double.

Worse, what does it mean by "game revenues per user double"? In Chapter 1 of Numbers Rule Your World, I call this sort of thing " the discontent of being averaged". Facebook's revenues from games and apps are being averaged here, divided by the number of registered users. In the next 5 years or so, one would expect to see registered users grow, probably very rapidly.

Requiring that the average (per-user) revenue to double is difficult if the base of users stays put; when the base of users is also growing rapidly, it is quite unthinkable that you can double per-user revenue. Let say they have 1 billion users today, each contributing $1 revenue. If the user base grows to 2 billion, and we require that each contributes $2, then the total revenues would be $4 billion which is 4 times those of today!

The exact problem also applies to the "ad revenues per page views". Page views would need to soar dramatically to hit the valuation, which means this is another assumption of super-duper growth.

Averages are very dangerous things. It causes one to think simplistically, and oftentimes, simplistic logic leads to errors of great magnitude.

***

Finally, I did a little experiment. I can increase the registered user base which would increase game revenues per user, which would allow me to expand the value further to hit $125 billion. What would the registered base need to be by 2017 in order to get that valuation?

A picture speaks louder than words:

What's the world population again?

Maybe we have moved from "certainly possible" to "certainly impossible"?

 

 

Misinformation about steroid testing

Steroids in sports have been in the news yet again. There's the Sports Illustrated article on Lance Armstrong, and there's the Yahoo! story on deer antlers.

While doing research for the book, I learned that steroid testing is a very complex science, and there are so many reasons why the test results cannot be treated as "definitive" -- as I show in the book, the errors are not symmetric either: the information value of a "negative" finding is low while a "positive" finding is usually credible.

Then, I came across this Yahoo! article featuring NFL manager Ozzie Newsome defending the league's testing program, repeating a laundry list of nonsense.

***

No. 1, I think we probably, in professional sports, have the best testing of all the four major sports.

Professional sports in general, and US professional sports in particular, have less stringent testing than amateur sports so comparing them against each other isn't saying much.

Our testing for performance enhancers goes year-round [and] they’re random. So, if there’s anything that any of our players are taking, then they will be found out about because they will test positive for it, and there would be a suspension.

Taking the second sentence first: He asserts that a drug test is 100% accurate, there are no false negatives, and everyone who takes something illegal will test positive. That's the exact opposite of what is happening out there. There are a lot of false negatives; most dopers are not caught.

His first sentence is useful as it explains why it is extremely hard to catch dopers. Year-round testing is standard fare for any serious drug-testing program: this is to expose off-season steroid use. It's a myth that atheletes consume steroids -- and I'm using this term loosely to include other performance enhancing drugs -- right before an important competition or only during the competitive season; instead, it is more effective (not to mention less prone to detection) to use them during off-season training. (Off-season steroid testing is controversial and operationally complex because it requires athletes to disclose their whereabouts at all times so that inspectors can show up unannounced. It also has led to hilarious scenes of athletes evading testers, with ingenious reasons for why they were where they said they would be when the testers showed up.)

Random testing can mean one of two things. If he means unannounced timing, then this also is standard for any drug-testing program. In fact, any testing program in which testers do not show up unannounced is not serious. If the athlete knows he/she is being tested tomorrow, there is no chance that the test will come up positive.

If by random, he means random selection of who to be tested, rather than random timing of the test, that's another matter. I'm not sure that random selection is the desirable option; the alternative would be targeted testing, targeting the people who have more incentive to cheat (ironically, these would be the higher-ranked athletes).

This situation reminds me of the so-called "random" bag search in New York subways. If I'm concerned about safety only (not fairness), a targeted bag search will be much more effective than a random bag search. Imagine that the random search is implemented by stopping every 50th passenger walking into a station. This search policy will yield a high rate of false positives, that is, innocent people being searched. In addition, it will also have high false negatives because you won't stop a suspicious looking passenger if he/she doesn't arrive at the 50th, 100th, 150th, etc. slots. (If you do stop such passengers, then the policy cannot be described as "random".)

***

That being said, if there is a product that he’s taking, and he’s passing the tests, then I can’t explain that one.

He's not trying hard enough. Just off the top of my head: he's a false negative (which is a highly probable scenario as explained in the book); he's lucky (how many times is each athlete tested each year anyway?); he's taking a new steroid that doesn't have a test yet; he's taking a masking agent that doesn't have a test yet; he's taking a known steroid that the NFL does not currently test for (e.g. HGH); he's taking a known steroid for which a reliable test hasn't been devised yet; he was on an off day of his steroid regimen when his urine was collected; he's got a legal pass to take the steroids (this is called Therapeutic Use Exemption); he's used someone's urine or other tricks; etc.

 

Insurance relies on subsidies

Felix Salmon explains the bad bargain known as "index annuities" very well indeed. In the beginning, these annuities are pretty good investment products:

You hand over a lump sum, and an insurance company promises to pay you an income so long as you’re alive. If you die early, you lose out (but don’t need the money any more); if you live long, you win, and do need the money.

Then, these products got ruined when the sellers introduced an early cashout provision which allows buyers to "take out up to 100% of your money penalty-free if you are diagnosed with a terminal illness or enter a nursing home."

As Felix explains:

The reason for me to buy an annuity is that I want to be subsidized by the short-lived if I turn out to be one of the long-lived. What I don’t want is for the short-lived to be able to get their money back in full, because then my subsidy goes away, and there’s no point in buying an annuity at all.

***

This is the same point I try to get across in Chapter 3 of Numbers Rule Your World. The term "subsidy" has a terribly negative connotation in the U.S. but insurance only works because of subsidies. The arrangement, in the case of disaster insurance, is for the lucky to subsidize the unlucky, and it only works if there is uncertainty as to who will be the lucky one. The participants have to feel that the subsidy arrangement is "fair" or else they wouldn't join the scheme.

Neat reads 8

The New York Times reported on a bit of controversy about the use of statistical techniques to prove the existence of ESP. Here's a profile of the researchers who claimed to have proven ESP, and here's an overview of the statisticians who disagree. The statistical discussion is rather technical, and Andrew Gelman has a good post on it. My view is similar to Andrew's: the real issue is how to use statistical methods to measure effects that are really small. Indeed, is this a solvable problem? If the "noise" in the data is much stronger than the signal we're trying to detect, is it possible to detect the signal with a reasonable level of accuracy?

***

You are not alone in noticing that Google search results have been invaded by spam. It is now often the case that the first-page results consist of nothing but commercial websites, spam websites, and unreliable content created by several companies set up specifically to game Google's algorithms. Google is either losing the battle with spammers or content to make money from whomever.

See Paul Kedrosky's pieces here and here. Unlike Paul, I don't think curation is a standalone solution; it is not scalable to the multitude of possible searches. Smart algorithms with human input is our only option. This is in its essence the same problem as the email spam problem.

Here is a summary of three articles on this subject. 

In Chapter 2 of Numbers Rule Your World, I briefly mentioned why it is a horrible idea for regulators to force banks to reveal their credit-scoring algorithms. Doing so would lead to the kind of gaming that has harmed Google's search algorithms.    

Small-number scams redux

I gave some examples of how crooks can use "small numbers" to hatch up scams in several contexts here. The essence is to steal a little from a lot of people. Each person has little incentive to report or deal with the theft.

Another example is supermarkets. I have often speculated whether supermarkets can generate extra revenues by conveniently overcharging a random proportion of their bills by a few cents each; few customers would notice or even if they notice, it would not be worth their while to go back to the stores to argue over a few cents. Over thousands or millions of bills, that is a lot of revenues. The telling signs would be that the billing errors are skewed positive (over-charges).

Now, there is evidence that such fraud is widespread. The New York Post reported work done by the New York City Department of Consumer Affairs. They estimated that one in three bills contains at least one over-charged item.

***

I wish I thought of this in college. This sounds like a wonderful class project to learn about data collection, sampling and hypothesis testing that has real-world impact (although you'd have to apply for some funding to buy groceries!)

 

The Washington Post confuses readers about seasonal adjustments

For an article that starts out in a really promising fashion:

If you assume each month's unemployment rate is a fixed number, think again.

it is truly disappointing that the rest of the article makes a mess of the statistics. So much so that it should come with a health warning: the Washington Post will make you confused about seasonal adjustments if you aren't already confused.

***

The Labor Department makes adjustments to the raw unemployment statistics each week or month it presents the data. The 9% or 10% unemployment rate we hear about has been almost always a "seasonally-adjusted" number. Think of it as the "run rate" of unemployment, having controlled for seasonal patterns of unemployment. In the example suggested by the Washington Post article, unemployment decreases each summer when lifeguards are hired and increases after the summer when these jobs disappear. The "run rate" should reflect unusual changes in lifeguard employment but not the annual pattern of increase and decrease.

The adjustment is for unusual changes, which means the statisticians must first establish what the expected seasonal pattern should be. The expectation is derived from what happened in the recent past, and so each year, the adjustment factors will be revised because the "recent past" has shifted. 

The "news" in the Washington Post story concerns an update to the adjustment factors. There is no change to the adjustment method. Nor is the case that certain new patterns were discovered (as far as we know). The reporter makes a complete mess of this when she tells us:

Changes to the rates occur because the government aims to remove seasonal "noise" from the employment numbers. That "noise" can occur when, say, lifeguards are hired in June and let go in September.

The hiring and letting go of lifeguards is a well-known pattern that has been included in the existing seasonal adjustment. Unless the lifeguard employment picture is particularly unusual in the "recent past" in such a way that it isn't captured fully by the existing adjustment factor, the lifeguard scenario has nothing to do with the "news" coming out of the Labor Department.

The adjustment method itself dictates that there will be such a revision each year, and as the reporter points out, the changes are minor (less than 0.1% in any month) and only affect 3 or 4 months. This really isn't news.

***

Buried deep in the article is a different type of adjustment worth paying attention to. The Labor Department makes two types of adjustments to the job creation number: seasonal adjustment, and correction of small-business bias. The latter is done through something called a birth-death model, and in normal economies, it is supposed to capture small-business jobs that were created that are missed in the surveys used to learn about job creation.

Not surprisingly, what is good practice in normal economies doesn't prove to be very practical in abnormal economies. In the recent past, the birth-death model has added hundreds of thousands of "phantom jobs" to the job creation statistic because the small-business jobs just weren't there. Once a year, the Labor Department looks back to see if indeed these small-business jobs exist. So, according to the Washington Post: 

The economy added a net total of 1.1 million jobs in 2010. That total will likely turn out to be lower - by 366,000.

In many lines of work, the magnitude of this error would cost jobs. John Crudele at the New York Post has done an admirable job over the years screaming about this issue. (Here's one of his more recent posts on the topic.)

Click here for my previous post on seasonal adjustments.

 

 

 

Neat reads 7

Playing preset numbers in lotteries is a way to share the wealth

41,000 winners in this lottery. Preset numbers include things like the statistically least popular numbers, the diagonal numbers on the ticket, and so on.

How to make lots of people sick and believe nonsense with fraudulent research published in top journals which stood idly by for 12 years before retracting the junk

Andrew Wakefield (formerly a British doctor, now based in the US) doctored data and published erroneous research linking the MMR vaccine to autism. Result: a surge in measles (56 cases in the year before the paper came out, 1,348 cases ten years later in the UK). Ben Goldacre has been on this case for years.

A data explosion remakes retailing

NYT reports on how retailers are using data to improve their business. I do this kind of work for a living. This covers some basic stuff. The key is how data provide feedback to retailers on what the customers want and don't want as well as their decision-making process.