Big Data, Plainly Spoken (aka Numbers Rule Your World)

Birth right, and the Census

Credit: Francesco Ugolini, Flickr

In a nation put together by immigrants, and as an immigrant, it's breathtaking to experience the current climate in which immigrants are demonized. Legal immigration is being targeted in the name of rooting out illegal immigration.

The latest attack is on so-called birthright citizenship. In this article, the author cites Fox News that ran statistics on "births to unauthorized immigrants" used to support the abolishment of birthright. Fox News cites Pew but does not link to the source of the data. I traced it back to this 2015 article.

How does anyone count these births? It turns out Pew did not do a survey on this topic. Pew cites surveys run by the Census Bureau, namely, American Community Survey and Current Population Survey.

***

Fox News will soon discover that those statistics will no longer be available due to actions being pushed by the President and Department of Commerce.

You may be aware of the other controversy: the Census Bureau is being forced to add a question on citizenship to the Census. Our Supreme Court just shielded Commerce Secretary Wilbur Ross from answering questions about this decision.

If a citizenship question exists on the survey, the unauthorized immigrant will simply refuse to take the survey, or lie. Why would anyone acknowledge one's illegal status, and risk being thrown out of the country? Suddenly, the number of unauthorized immigrants reported by these surveys will be severely underestimated, and births to unauthorized will plunge as a percent of total births. Not what the anti-birthright team wants!

Note that this comment is concerned about the accuracy of the data coming out of the Census, and I am not endorsing illegal immigration.

 

 

Why stock X is falling today

In this tumultous market, there is a flash flood of journalists and pseudo-journalists explaining "why stock X is falling today". On a simple web search, the first bunch of results have X equaling Atlassian, Hanesbrand, WWE, Overstock.com, Southwest.

These authors all follow the same script. They look at each company's financial statements, or recently announced earnings report summaries, pull out some negative bits, and claim that those items are the causes of the sharp stock declines.

It's disturbing how little science there is in these pieces. Even the problem statement is completely warped. For example, the Motley Fool article takes the recently-popular format of a synopsis, starting with a one-liner on "What Happened." Eventually, the reader expects to hear an explanation of why it happened, and what does it mean for you.

So what happened according to the Motley Fool reporter?

"After the company reported third-quarter financial results that beat industry watchers' top- and bottom-line forecasts, shares in Southwest Airlines Co. (NYSE:LUV) are trading 8.2% lower at 2:15 p.m. EDT today."

This is one loaded statement. The causal link between the Southwest's 3rd quarter results and the drop in its stock price is already baked in. The reporter isn't asking if the 8.2% price drop is caused by the financial results, or how much of the drop can be explained by the results; the entire problem is framed such that the only possible conclusion is that the stock drop is caused by and only by those results!

PS. There is one scenario under which that problem statement is justified - if the stock price chart shows a steep plunge right after the earnings were announced. From the chart below, it appears that the drop started around noon, in fact, following a few hours in which the stock was trending up. (I'm not seeing where that 8.2% came from.)

***

How would a statistician approach this question?

One starts with the overall market decline. Then estimate any excess decline of airline stocks versus the overall market. And if the decline hasn't been fully explained, then look for factors directly related to Southwest.

 

 

The cost of bad advice

In the last week, I kept being prompted by the recommendation engines to read the article in Slate titled "Don’t Just Look at the Odds: The Math Says You Should Buy a Mega Millions Ticket." It's obviously false but eventually I succumbed and clicked on the link.

I wanted to stop reading after I encountered this sentence, about half way through the article: "One assumption in this model is that you’ll take the annual payment option and receive the full $1.6 billion."

The pertinent assumption that they made isn't whether the winner should take the lump sum or annual payment options. The blockbuster is to assume that there would only be a single winner.

Earlier in the article, they stated a true but irrelevant point: that the odds of a winning ticket and the price of each ticket are still th same no matter how many tickets are sold.

I persisted in reading the entire article for the sake of writing this post. I wanted to know whether they will acknowledge the biggest factor here, which is that the more tickets are purchased, the more likely there will be more than one winners. Even having two winners can be devastating. The $1.6 billion suddenly turned into $0.8 billion. With three winners, it's $0.53 billion. If the ticket is purchased as part of an office pool with, say 10, participants, then $1.6 billion becomes $160 million. Not a bad payday but much, much worse than projected.

 

PS. They updated the post because too many people are pointing out the obvious about multiple winners. They continue to deny its importance.

Their math is wrong. Here is a good webpage that explains the math from a previous situation 10 years ago. The number of winners follows what is called a Poisson distribution. In the example given, there is a 60% chance of a single winner, but 30% chance of double winners and 10% chance of more than two winners. How can this be called "incredibly low odds"?

Another issue is that there are certain sets of numbers that attract much more tickets than usual. If one of those sets is the winner, you're going to have lots of winners.

They continue to completely miss the point.

 

 

 

 

New Facebook data breach, and the importance of Business Intelligence

 

On Friday, Facebook announced that hackers have gained access to personal data of at least 50 million users (see here for example). Analysts immediately connected this incident to the Cambridge Analytica scandal. How is this data breach different from the Cambridge scandal?

How the data was breached

One should take any announcement of how a data breach occurred with a grain of salt: it's obvious that companies will not publish anything that attracts lawsuits; plus, it's unclear that there is a penalty for lying about the real reasons for a data breach. There is no external auditing, and the companies control the narratives and statistics surrounding these events.

Based on what Facebook told us, the Cambridge Analytica scandal involves Facebook partners gaming the system to obtain data about Facebook users. At worst, it is a violation of community standards, i.e. claiming to be doing academic research. The research firm uses tools provided by Facebook to obtain the data. (The firm maintains that it disclosed to users that it was using the data for non-research purposes.)

In the present case, Facebook claims that a combination of coding bugs (i.e. unintended features) enabled unknown partners to gain access to the user's entire account. Not even that, but any accounts on third-party apps or websites that the user signs on to using Facebook.

 

How the breach was discovered

The current scandal demonstrates the value of the business intelligence/business analytics functions within companies. We were told that Facebook first realized that certain metrics were showing unusual trends, and upon investigation, they discovered the bugs.

This is entirely believable. That's what happens when you have good data reports. They surface anomalies. These then have to be investigated. These investigations are extremely tricky because all you know is the trends are different. There are a thousand reasons for the shift. The analyst's job is to establish a cause-effect. Especially since the development community adopted "agile" practices, all kinds of self-imposed changes are occurring all the time with no warnings. For a site as large and complex as Facebook, it takes a huge effort just to get a list of all site changes within some specified time window!

What complicates this situation more is that the vulnerability was traced to multiple bugs, not just one. I could imagine the twists and turns and the false alarms that were generated during the investigation.

The nature of this problem is no different from an investigator trying to chase down an e-coli outbreak, which is detailed in Chapter 2 of Numbers Rule Your World (link).

The data science community is guilty of talking down on the business intelligence function. There is a misperception that BI is for less skilled people doing boring things. The reality is there is more science in BI than in so-called data science (defined here as software engineering). Science, after all, is about figuring out why things are as they are. Engineers, by contrast, use our understanding of science to change the way things are.

 

You've been warned

In my debut Youtube video, released a few weeks ago, I explain how Facebook collects data about you. My biggest tip about protecting yourself is to not use Facebook to log onto other websites. Convenience is the drug that lures you into the trap.

Think about those one-password services. Instead of hundreds of password, you have one. So the thief needs only one password to assess hundreds of sites. Using Facebook to log onto other websites makes Facebook the centralized point of attack by the bad players.

Note that this is not limited to Facebook. Using Google, Yahoo, Amazon, etc. to log onto other services is no different!

If it's more convenient for you, it's also more convenient for your enemies. Remember that.

Also in the video, you will learn

• why did Facebook invent the Like button
• how do Facebook know what you are doing outside Facebook
• why not clicking doesn't mean you're not being tracked
• why you shouldn't use Facebook to log into other services

See the video here, and send me ideas for future episodes!