Big Data, Plainly Spoken (aka Numbers Rule Your World)

Why many published data analyses have so little real value

There is one mode of data analysis that rarely generates anything of real value. Here, the analyst asks the question: what can I do with this dataset? Unfortunately, the data could be very large, but it may not contain the information needed to address a meaningful question.

Let's take the Citibike dataset as an example. It contains transactional data of when a rider checks out or checks in a bicycle.

This leads to a pile of predictable analyses and data graphics.

For example:

This dynamic map shows the popularity of different bike stations throughout the day.

Another example:

This one shows the top 10 bike stations by popularity.

Those are the typical outputs of analyzing Citibike data. The problem: why would Citibike management care?

***
Instead, we should ask Citibike management what problems they are concerned with.

Someone I recently interviewed gave a good answer: maybe the managers want to know whether there are riders who could not get bikes or there are bicycles that are rarely checked out. In other words, excess demand or excess supply. If the analyst comes back with an answer, to tell management whether they are facing excess demand or supply, that would be a useful outcome.

We'll ignore whether we want the answer at the local or global scale. Let's take one particular neighborhood (say, East Village) for argument's sake. Can we say if there is excess demand or excess supply in East Village of Citibikes?

You can now filter the data to only East Village bike stations, and count the number of unique bike identification numbers across the transactional dataset. That gives you a number.

But what does that number mean?

It turns out the count of unique bicycles is measuring neither supply nor demand.

If there were excess demand, meaning some riders are turned away, that information is not in the transactional dataset. It's unlikely to be in any dataset owned by Citibike. They would have to find creative ways to collect the data.

***

This is just one of many examples in which despite the sizes of today's datasets, they still don't contain the right data that hold the key to answering meaningful questions.

Because the Citibike dataset is available, we are bomboarded with analyses that focus on imaginary problems of little value.

 

 

 

They destroyed huge amounts of food again

Every year or so, another huge food recall makes the news, and it makes me sad.

The latest is the recall of 200 million eggs (link) due to salmonella "risk."

The identification of the cause of such disease outbreaks is a great case study of causal investigations, which I cover in detail in my book, Numbers Rule Your World. In that section, I also raise questions about the (in)sanity of these broad-based food recalls.

How does a food recall save a life?

All of the following have to happen:

• Some proportion of the recalled eggs has to be contaminated
• The contaminated eggs have to be purchased by consumers
• The contaminated eggs have not already been consumed or discarded before the recall (while the investigation was happening. From this epi curve, it looks like the greatest concentration of reported cases occurred in Nov/Dec 2017, four months ago.)
• Those who bought the eggs learn of the outbreak and subsequent recall
• Those who knew of the recall take action to return or dispose of their eggs
• Someone would have eaten those tainted eggs if they weren't recalled
• When eating the tainted eggs, the consumers do not cook them thoroughly
• The immune systems of those people who ate the tainted eggs fail to fight off the illness on their own
• The sick person decides to go to a hospital, thus identifying him/herself to CDC
• The sick person does not recover, typically due to having some pre-existing conditions that weakens his/her defence

The short version: the saved life has to be someone who would have died from consuming the tainted eggs were they not subject to this massive recall.

Each one of the above steps happen with some level of probability. For all these steps to happen at the same time is highly unlikely.

The economic fallout of dumping all those eggs is certain. The benefit of the recall in human lives is highly uncertain. So, it's not clear to me that these recalls are reasonable.

 

Laplace on behavioral psychology channeled through Miller and Gelman

I enjoyed reading the recent article by Josh Miller and Andrew Gelman (link), which relates the work of Kahneman, Gigerenzer, etc. to the work of the French mathematician Laplace from the 1800s.

Here are several quotes that particularly interest me.

Laplace offered a definition of the word "illusion." He said a "visual illusion" (optical illusion) is caused by a weakness in the sense of sight but can be corrected by touch while a "cognitive illusion" is caused by a weakness in the mind but can be corrected by thought and calculation.

This reminds the authors (and me) of the distinction between System 1 and System 2 thinking, made popular by Kahneman in his "Thinking Fast and Slow" paradigm. It's saying that slow thinking is a cure for the errors of fast thinking.

But slow thinking has its own weaknesses as well. For example, it can't adapt to a fast-evolving environment.

In many practical problems, there is a tradeoff between speed of decision-making and its accuracy.

This discussion brings up one of the missing pieces of the current AI (artificial intelligence) program. It seems to put 100% faith in slow thinking - through calculations involving huge amounts of data.

***

I like Laplace's illustration of the fallacy of "law of small numbers." Male to female births over a long period of time (large numbers) are expected to be roughly equal. However, people tend to think that the same holds true at shorter time scales, like every month.

This sequence of births is related to a sequence of coin tosses. This is a problem that I have discussed before in this blog post. I use it to clarify the difference between probability problems and statistical problems. While it is possible (but not probable) that one could observe a streak of 20 heads in throwing 20 coins, one must ask when your prior belief of fair coins is shaken as each head materializes!

***

Laplace also wrote "in return for small gains, we have no fear at all in exposing our lives to risks much less unlikely than the drawing of a quine in the French lottery; and yet one would not choose to get the same benefits, with the certainty of losing one's life if a Quine were to occur."

This phenomenon is called the description-experience gap. Experienced frequencies are more influential than computed probabilities.

 

 

 

Facebook Update

The controversy over Facebook user data is simmering. Here is a summary of the latest developments.

***

Buzzfeed published a leaked memo from one of Facebook's top executives, titled "The Ugly," that describes a disturbing philosophy that both reflects an understanding of the power of private data in transforming lives and a reckless intentional disregard for potential harm. The memo ought to be read in full but the most shocking sentences are:

Maybe someone dies in a terrorist attack coordinated on our tools. And still we connect people. The ugly truth is that we believe in connecting people so deeply that anything that allows us to connect more people more often is *de facto* good...That’s why all the work we do in growth is justified. All the questionable contact importing practices. All the subtle language that helps people stay searchable by friends...

Once the memo leaked, the author claimed that it was a "provocation." His boss claimed that he disagreed with this memo.

***

Facebook has suspended two more organizations from getting access to user data. For example, a Canadian company AggregateIQ that is thought to be an arm of SCL, which is involved in the Cambridge Analytica scandal. Also, Cubeyou, a California company, is alleged to have resold data collected by a University of Cambridge academic center.

These cases involve shady practices in which users are coaxed or tricked into providing personal data through "gamification" (e.g. quizzes) and consent language hidden inside terms and conditions that everyone knows few people read. In each case, Facebook accused academic researchers of profiting from data collected under the pretense of academic research. The academics countered that they disclosed to Facebook users in terms and conditions that the data are collected for both research and commercial usage.

So, everyone believes s/he is acting by the rules. Why is anyone apologizing?

***

Facebook supporters say that the social-media giant needs revenues from selling user data to compensate for the core product being free.

Strangely, the current controversy has nothing to do with Facebook selling user data for dollars. All three suspended accounts supposedly "stole" data from Facebook by posing as academics so they took the data or free. In fact, Facebook drew attention to these cases by announcing the suspensions.

The real story, which hasn't yet been told, concerns the commercial deals in which Facebook earns revenues from selling user data to third parties.

As I indicated in my 7 Principles post, there is no need to hide data collection if the collected data benefit users; and businesses can derive value by analyzing user data without selling them to third-parties.

***

Another report confirms what I said here before: that privacy is the civil rights issue of our day. We are quickly heading to the world in which privacy is stripped by default, and purchased by the privileged.

TechCrunch tells us that Facebook CEO deleted messages from user inboxes but Facebook users are not allowed to delete any messages they have ever sent.

Congress may have joined the circus for publicity but it is far from innocent. Congress has passed various laws that stifle our right to privacy. Most recently, when President Trump passed the omnibus spending bill, Congress snuck in the CLOUD Act, which

is a surveillance bill that allows the US and foreign governments to obtain your online data directly from service providers like Facebook, Google, Slack, etc., without a warrant. The EFF has called it “a new backdoor around the Fourth Amendment.”

Previously, Congress gave Internet (pipe) providers free rein to collect and sell user data.

Is Congress on the people's side or not?

 

Know your data 22: Facebook and Google play while the cat's away

Facebook and Google have now been hoisted in front of the public, and rightfully reprimanded for their invasive data collection. (Notice, the deafening silence of our politicians and government officials on this issue.) In reality, the entire industry has been condoning and abetting these practices. You can follow my Know Your Data series of posts all the way to 2010, documenting what some of the big names have been doing in taking away every citizen's privacy.

The Guardian's article is indispensable in its meticulous documentation of everything Facebook and Google collect about you. Facebook is in fact less intrusive than Google, simply because Google has embedded itself everywhere: Chrome (browser), Android (mobile phones), Chromecast (TV), search, Google Phone, Google Analytics (on most websites), Google Apps (used by a lot of universities and businesses), Gmail, Google Hangouts, Google Maps, Google Calendar, you name it. 

For the author of the article, Google has over 5 gigs of data on him while Facebook has 600 megs. (We are making the assumption that we are being shown everything.)

Some of the key takeaways:

• Deleting something merely disconnects you from your data. The data still exist in the corporate servers. (You should have known this when these companies tell you that you can come back and restore your old contacts, data, etc.)
• The era of "anonymity" is long gone. All of the data are traced to you. Some readers may remember the days when we were told that cookies are harmless files that are not personally identifiable.
• Your mistakes, bloopers, etc. are all stored. If you accidentally click on a spam ad, it is stored, and likely used by some algorithm to profile you.
• Google seems to have kept not just metadata (titles of images, sent dates and recipients of emails, etc.) but all the data (photos, emails, etc.)
• No one cares whether the information about you is accurate or not. 

If there is one copy of this data, you can bet there are lots of copies of the data. This is called "redundancy" - you just have to keep multiple copies to recover from inevitable data losses.