Big Data, Plainly Spoken (aka Numbers Rule Your World)

Using data to guess authorship of the Federalist Papers

 

I was recently reminded of how statistics settled a question of authorship of The Federalist Papers. These were 85 arguments published under a pseudonym in support of the U.S. Constitution. It is now known that they were written by Alexander Hamilton, James Madison and John Jay. For a long time, there was uncertainty about the authorship of twelve of these. Around 1960, two statisticians, Frederick Mosteller and David Wallace, published a celebrated paper that solved the riddle.

The key components of the solution include:

• Noticing that people’s writing styles differ in terms of word preference. Certain writers habitually use certain words more often than others.
• “Common words” like prepositions are better differentiators than less common words. For one thing, common words are common, and therefore we have more data to establish the base rate of authors. For example, Madison almost never wrote the word “upon” while Hamilton used the word quite often; thus, a document that contains no “upon”s is much more likely to have been written by Madison.
• Each differentiator word is a signal. A model combines multiple signals coming from multiple differentiator words. The combination is more than the sum of parts.
• The Bayesian model produces a probability that a specific author wrote a specific article.

 

The back story is also interesting:

• The success of 1960 was not assured. Mosteller and others tried other methods before, for example, examining average sentence lengths, and failed.
• Mosteller himself expressed the self-doubt characteristic of most competent statisticians: “the odds can never be greater than the odds against an outrageous event”. In other words, stranger things can happen.
• No matter how high the probability is, we still only have proven a correlation. For example, one respondent argued that Hamilton could have written the first draft, which Madison edited. The data could not exclude such an event. (Nevertheless, anyone forwarding this possibility ought to produce evidence to support it.)

False-positive science

The Atlantic reports on the dynamics of yet another group of scientists coming to grips with having wasted time and resources chasing down a dead end. (link)

It's a good read but long. Here is the gist of it:

Almost 20 years ago, some researchers made a huge splash by claiming to have discovered the "depression gene". The one gene eventually engendered 450 publications, and when counting related genes, over 1,000 publications. A recent large-scale "validation" study is likely to bring down the entire cottage industry - the depression gene is found to have little explanatory power for depression after all.

Gene data is an example of a type of Big Data. Big Data can be big in terms of the number of individuals in the dataset, or the number of measurements per individual. Two decades ago, the scale was attained by virtue of more measurements, not more individuals. The original study looked at about 300 or so individuals but each person's genome is vast.

The basic analysis is to compare the average depressed individual versus the average not-depressed individual in the sample. The data analyst sifts through large numbers of genes to find one or a few that are highly correlated with having depression. This is a classic fishing expedition, because of the large number of candidate genes, and also because of the large number of ways to define depression.

Such an analysis rides on top of a "model" of the world in which a single gene is responsible for depression. Over the years, the scientific community has discovered that this model is wrong. The new model assumes depression is indicated by a large set of genes each contributing a weak effect.

This type of structure is very hard to elicit from the typical datasets of the past - those that have numerous measurements on few individuals. Nowadays, we have data on lots of individuals but the sourcing of the data and other problems pose formidable challenges. It's also not clear how to use a model that spreads the blame thinly around a large number of genes for treatment.

Science is proceeding as it should - weak theories are overturned with more research. The article laments that it took 20 years to turn the tide, earlier warnings were ignored, the publish-or-perish culture in academia creates perverse incentives, retraction of scientific studies, etc.

***

I recently wrote about the challenge of Big Data expanding the variety of measurements here. Also, in writing Numbersense (link), I was concerned that the explosion of data collection causes an avalanche of false-positive science.

 

Losing control of the data

Another day, another story about Facebook data. Ars Technica reports that Facebook is suing a South Korean app developer Rankwave, claiming the company misused data it received from Facebook. Rankwave creates mobile apps through which it obtained Facebook user data for 10 years.

All we know about this situation came from the Facebook press release, and it's not clear what the offense is. The article cited the violation as using the Facebook data to "create and sell advertising and marketing analytics and models". That's how Facebook uses the user data, same as why Facebook partners want access to user data.

One part of the press release rings very true: Facebook admits that it does not control the data once shared with third parties. Facebook lawyers demanded Rankwave do the following:

• Provide a full accounting of Facebook user data in its possession;
• Identify all individuals, organizations, and governmental entities to which it had sold, or otherwise distributed, Facebook user data;
• Provide a full record of the access logs and permissions it had granted third parties to access the data;
• Delete and destroy all Facebook user data after returning it to Facebook;
• Provide Facebook with full access to all storage and related devices so that Facebook could confirm deletion and destruction of the data through an audit.

These all sound great but would any company, even Facebook, be able to deliver the above logs, reports, devices, etc.? Given how data are spread out in big networks of servers ("data clouds", "data lakes", etc.), this wishlist sounds like a fantasy.

In this new video, I talk about the data sharing ecosystem, why it is so hard to delete anything, and how companies lose control of the data. It's the flip side of speed and convenience. What is the price we're willing to pay?

Click here to see all my videos.

Labeling data: a crucial part of machine learning

Several recent news stories cover the topic of “labeling” data. For example, this Bloomberg article says Amazon is sending voice recordings from its Echo speakers to be heard and transcribed by human listeners. This Reuters article discusses Facebook’s contractors in India, Romania, Philippines, etc. hired to “label” status updates, shared links, event posts, Stories feature uploads, videos and photos.

The authors of these articles express genuine shock and awe. They apparently believed that “machine learning” means no humans involved. The tech industry allows this misconception to fester by being opaque about how machine learning works.

(The reporters are also dismayed by the privacy invasion. The Echo speakers are constantly recording in users’ households. Facebook did not have explicit permission from users to it send their data out for labeling.)

***

Humans have always been a part of the machine learning workflow, and will continue to be. Let’s use one of the examples in the Facebook report to illustrate this point.

A machine learning model predicts profane language in a Facebook video. For the time being, I assume that a predictive model already exists. This model is being fed videos, and automatically makes predictions - without human involvement. Video 1 – it predicts profanity; Video 2 – no; 3 – no; 4 – yes; etc.

Computers work fast, and can make tons of predictions quickly. The question is whether these predictions are accurate. It’s one thing to create these models in the laboratory; it’s another thing when they are unleashed to the world, and affecting Facebook users, e.g. by deleting videos that are predicted to contain profanity.

Why should Facebook and its data scientists care about predictive accuracy?

User complaints. When users find their videos deleted due to profanity, they complain if said clips do not in fact contain profane language. Other users are upset to unsuspectingly encounter videos with profanity (that the machine fails to identify and delete).

***

It’s not easy to measure if the machine-learning model is correctly predicting profanity. The machine can’t be both decider and judge at the same time. The judge typically is a human who views the video to determine if it contains profanity. These human judges are the “annotators” described in the news articles. They are hired to look through videos and apply a profanity “label” if they find profanity.

As disclosed in the articles, companies typically hire two or three judges for each item because profanity is a somewhat subjective opinion. They might order more detailed labeling, e.g. label types of profanity as opposed to one overall label of profanity.

***

Now let’s remove the assumption that we already have a machine learning model. Where does this model come from? Such a machine has to know what features of the video are correlated with presence of profanity. To discover this correlation, the machine needs to be told which videos have profane language, and which do not.

This is a chicken-and-egg problem and it is solved by having humans label a big batch of videos at the start, building the “training data”. In the Facebook example, they hired over 200 people to create data labels, later reduced to 30. The first team was building a large training dataset; after the predictive model has been produced, the future labeling by the reduced workforce is used to monitor the accuracy of the predictions.

Any company that claims to use our data to predict our behavior must create training data, i.e. labeled data. In most cases, humans must create the labels – by reading our emails, listening to our conversations, viewing our videos, reviewing our calendars, scanning our receipts, and so on.

How far companies should go and what methods they shoudl use in collecting such data are ethics questions that should be discussed.

Some practice case interview questions for data science

In Part 1 of my KDnuggets article, I explained what hiring managers mean when they look for critical thinking in the arena of data science and analytics. These requirements relate to the nature of data problems found in industry and business settings. The datasets are generally observational, self-selected, non-random, with hidden biases, and increasingly OCCAM (link); the business leaders have high-level objectives ("we want to increase customer loyalty"). The data scientist/analyst is the person in the "middle," trying to figure out how to make the problem precise, and solvable by a systematic analysis of available data.

In Part 2, I offer some practice case interview questions, based on three recent news events

• the college admissions scandal
• IPOs of ride-sharing companies like Lyft and Uber
• the Blue Apron post-IPO doldrums.

Long a staple of the management consulting hiring process, the case interview is a free-flowing dialogue between the interviewer and the interviewee. The interviewer is holding back some data to simulate what is known at the beginning of a data analysis process. The interviewee must be willing to probe, digging out more data, and shaping the structure of the analysis. The end product is an analytical framework. No one knows if the framework would be successful until it is implemented.

Those who do well in case interviews are good at (a) thinking on their feet (b) embracing uncertainty e.g. by making appropriate assumptions (c) listening to the interviewer's hints and (d) persuading.

***

As I mentioned in Part 2, the best way to practice is to form a group of 3-5 people, and interview each other. If there is enough interest, we can start a group in the comments below.