Big Data, Plainly Spoken (aka Numbers Rule Your World)

Know your data 8: revealing the little choices in your life

When I think about Facebook's latest "innovation", about who might gain from this forced sharing of personal choices, I think it may be that they want to appease academic researchers.

Here is CNN's description of how this so-called "real-time app" behaves:

because I've logged in to Spotify with my Facebook identity, every song I listen to is automatically shared to Facebook.

Spotify is the recently launched online music service. A number of other big-brand Internet companies is apparently on board, such as Netflix, Hulu, and Yahoo! News. I agree with Farhad Manjoo at Slate who said:

it's somehow eluded Zuckerberg that sharing is fundamentally about choosing. You experience a huge number of things every day, but you choose to tell your friends about only a fraction of them, because most of what you do isn't worth mentioning.

This opinion captures a law of data which hasn't been explored enough: that as quantity goes up, quality of data goes down. Before this change, users voluntarily disclose their preferences via Like buttons or similar setups; this means that users have performed a valuable act of filtering the data before exposing them (free public service!) With this new application, there is surely more data but I doubt if there is more useful data.

Perhaps this is a way to force the non-sharers to share their personal choices. In the past, only those who like to share their likes and dislikes do so but now, many users will find their choices exposed regardless of their wishes. This expansion of the base of data collection is useful for researchers and analysts; however, Facebook has the option of collecting the data without exposing them to people's networks. Facebook simply needs to make deals with Spotify, Netflix, Hulu, etc. to share data in the background. In fact, by encouraging users to use the same logon account, they probably already have this behind-the-scenes sharing in place already.

Previously, academic researchers do not have access to such data as businesses either do not collect them or would not make them available publicly due to competitive reasons. So I think at least one group can benefit from this rather alarming situation.

 

Now coming to your local bookstore

Numbers Rule Your World has been translated to several languages and is on sale in several countries. Interestingly, the book is coming out with different titles and book covers.

In order of appearance:

Japan:

Apparently, the title translates to "Dope Statistics", and there is a series of Japanese books called "Dope Economics", etc. The color scheme and cover art are similar to the English edition. Here is Amazon.jp link.

 ***

Korea:

They chose an orange cover, with a wind-up figure marching to a rhythm. Google Translate can't handle the title but the subtitle seems to say "Numbers you should dominate." It is on sale here.

 ***

Chinese:

The title is a mouthful: "Think like a statistician: 10 statistical concepts that allow you to think clearly and improve your lives." You can get it from here or here or here. Commercial Times recommends it.

***

Portuguese (Brazil):

The Brazilian edition just got published by VS Editora Ltda. It's available here and here. The publisher tells me the cover art was selected by readers of their blog. They went for no-nonsense, a striking blue color.

Mirror, mirror

There has been a lot of self-righteous reporting in the business press regarding the "rogue trader" denting a bank's reputation, acting in "shady" ways, etc. The $2 billion lost in one trade would certainly qualify as an extreme event.

I invite readers to think about the entire spread of trade returns that a bank like UBS sees in any given period of time:

(I took the chart from this website, which illustrates the returns of using a particular trading strategy repeatedly. Whatever the strategies are used in aggregate by UBS traders, one should see a similar distribution, with a different average return and a different spread commensurate with the amount of aggregate risk taken by the traders.)

We call the negative outliers "rogue traders". What do we call the positive outliers?

 

Deplorable journalistic traits, and why data analysis is not as simple as it seems

Felix Salmon laments that the standard of statistical literacy among journalists is appalling. He brings us this example about people who download music illegally:

About 95 percent of music downloads in 2010 were unlicensed and illegal, with no money flowing back to artists, songwriters or record producers, according to Alex Jacob, a spokesman for the International Federation of the Phonographic Industry. So riches could await a company that persuades some of these Internet scofflaws to change their ways.

He argues, rightfully, that the source makes the statistic (from an organization fighting against music piracy) hard to take seriously. He also feels that the number fails the "sniff test". Could it really be true that only 5% of the music are paid for? If the music market were to be say $10 billion today, then they are suggesting that the real market could be as high as 20 times that number, so $200 billion.

I left a comment on his blog to point out the craziness of the last sentence in the quote above, the delusion that if Bittorrent and other illegal download methods suddenly vanished, the online music revenues would jump 5-fold, 10-fold, etc. overnight. This is the same delusion that makes politicians/economists claim that we can solve our unemployment problem by giving our workforce more college degrees (in effect, shifting people from one bucket to a different bucket). I debunked that claim here.

***

The other point Felix made is well worth repeating: 95 percent of downloads is not the same as 95 percent of people doing downloads because a small number of people account for an outsized proportion of total downloads. Although Felix didn't state this directly - he assumed it in an example, it is most likely true that illegal downloaders are on average downloading many more songs than legal downloaders. For price is no barrier to the former group.

What this means is that if 95 percent of downloads were illegal, then the proportion of people who are illegal downloaders is likely to be considerably lower than 95 percent.

One final point: it is also foolhardy to bluntly divide the world into Illegals and Legals. In statistics, we like to think there is a continuum with most people having done at least one illegal download, while perhaps most so-called Illegals have paid at least once for music downloads. So, if we want to do a proper analysis of this phenomenon, we should put a probability of downloading illegally on each individual, rather than assuming that each person is either an Illegal or a Legal, and not both.

***

Yes, this makes data analysis sounds complicated. It's easy to fall into the many traps. But there is only a small number of fundamental concepts, and once you understand those, you'll find them popping up everywhere.

Small distractions

Business Insider makes a common error in proclaiming that Belgium's higher GDP growth rate is "bringing up the Eurozone GDP growth rate average". It lists a bunch of countries in order of growth rates, making the case that the big countries like France, Germany, UK and Italy were laggards by comparison.

If Belgium is bringing up the Eurozone growth rate average, then Germany also is bringing up the average, despite its growth rate being 1/7th that of Belgium. In fact, the data show that the contribution of Belgium to GDP growth of Eurozone is essentially identical to the contribution of Germany. The reason is that Germany's economy is 7 times that of Belgium. So, each unit of growth by Germany has 7 times more "power" than each unit of growth in Belgium.

In fact, Europe should thank Italy because it contributed almost as much as Germany and Belgium combined. While Italy's growth rate was half that of Belgium, its economy is 4 times as big.

Comparing the growth rates of individual countries is misleading. It is much better to think in terms of proportion of contribution to the whole (in million Euros). We can start with this table:

Many countries have not reported their latest quarterly numbers so we can't continue from here. However, if all the data were available, we would now sum the column labelled "contribution" and then compute the proportion of contribution by each country. This metric should be used to evaluate which countries are "bringing up the Eurozone average".

***

This example falls into the category of "error in spirit, not necessarily error in arithmetic". An analyst can reasonably protest - and many do in similar situations - that it is a mathematical certainty that all else equal, an increase in a component of the average will bring up the average. This type of thinking is analogous to that which proclaims an effect to be worth talking about because it is "statistically significant" even though the size of the said effect is so small as to render it practically meaningless.

Consider this: the elevator gets stuck with you inside it. The engineers are called in. The first thing they notice is the ugly rust on the door as the paint has come off. Instead of inspecting the motor, which is much more likely to improve your experience, they return to their office to obtain fresh paint. Getting distracted by insignificant features of a data set while missing the big picture is a common pitfall in data analysis.