Big Data, Plainly Spoken (aka Numbers Rule Your World)

Where is NYT's data science team when you need them?

The management at New York Times was smart enough to create a data science team but apparently the newsroom has not yet been exposed to STAT 101.

Today, NYT printed this headline: "Top C.E.O. Pay Fell -- Yes, Fell -- in 2015" (link to article). I immediately thought, not so fast, are they talking about average pay or median pay.

Anyone who has taken STAT 101 knows that when analyzing data with extreme values (CEO pay would clearly fall in this bucket), one should use median values, not averages. Since this data came from a survey of 200 top-paid CEOs, the median would be the CEO whose pay is ranked right in the middle of the pack.

A few paragraphs in, my suspicion was confirmed:

After years of steady increases, the average compensation among the top executives in 2015 was down 15 percent from the 2014 figure of $22.6 million, according to the Equilar 200 Highest-Paid C.E.O. Rankings, conducted for The New York Times.

Are they using average values because the median values were unavailable? This seems strange since the Equilar study should have all 200 values.

Clicking on the link and looking at the very first bullet point under "Key Trends and Takeaways" yields the following:

Median pay for Equilar 200 CEOs was $16.6 million in fiscal year 2015. The CEOs on this list saw a 5% pay increase at the median.

Instead of a 15% drop in pay, the Equilar analyst concluded that there was a 5% increase in pay. We also learned that the year before, the median pay jumped by 21%.

Not only did NYT got the statistics wrong but it loudly screamed the wrong conclusion in the headline.

We demand better.

PS. The median of $16.6 million means that the 100 CEOs at the bottom half of this list earned between $12.2 million (the minimum to get on the list) and $16.6 million, a range of $4.4 million. If one takes the $4.4 million the other direction and look at the range $16.6 to $21.0 million, that includes 54 CEOs. Take another $4.4 million up to $25.4 million, and there are only 25 CEOs above this number. This is how you can see the pay data is heavily skewed.

 

 

Tip of the day: don't be Theranosed

Theranos (v): to spin stories that appeal to data while not presenting any data

To be Theranosed is to fall for scammers who tell stories appealing to data but do not present any actual data. This is worse than story time, in which the storyteller starts out with real data but veers off mid-stream into unsubstantiated froth, hoping you and I got carried away by the narrative flow.

Theranos (n): From 2003 to 2016, a company in Palo Alto, Ca., the epicenter of venture capital, founded by Elizabeth Holmes, a 19-year-old Stanford University dropout, raised over $70 million to develop and market a "revolutionary" technology for blood testing that is said to require only a finger-prick of blood. The company grew its valuation to $9 billion without ever publishing any scientific data in a peer-reviewed medical journal. It turned out that the new technology was used only in 12 out of 200 tests on its menu, meaning that the business has been based on selling old technology at bargain basement prices subsidized by the venture-capital money. Further, it emerged that the new technology was not accurate, that the new technology has been shelved since last year, and in some cases when old technology was used, the lab personnel improperly handled the machines--all of which eventually led to a blanket retraction of two full years worth of test results. The company claimed that these results have been "corrected" in the last few weeks. It is unclear what "correction" means when such blood was taken from patients up to two years ago. The company is still in business, and Walgreens, one of its most prominent partners, continues its commercial relations with the company. For many years, the business and technology press has issued countless glowing reviews of the company (see this epic list just covering 2013-2015.) Until 2014, the company board consists entirely of politicians, former cabinet members and military leaders. All these individuals have been Theranosed.

The Wall Street Journal has done an exemplary job following this case, and deserves a Pulitzer for this effort. The latest revelation relating to the full-scale retraction is here.

 

Law versus ethics: how the new overtime pay rule will play out

In any discussion of ethics , someone is bound to bring up the law: it is okay to do something if something is not against the law. The logical conclusion of this belief is that ethics is irrelevant in the presence of the law.

The recent stories relating to minimum wage and the overtime pay rule changes (for example, link) argue strongly against such a point of view. The law has stipulated that non-managerial (i.e. lower-wage) workers must be paid 1.5 times salary for any overtime work. The Obama administration has now raised the maximum salary to be considered non-managerial from ~$24,000 to ~$48,000. This reportedly will raise wages by $1.2 billion annually.

This rule change is designed to combat gaming of the existing rule. Employers avoid paying overtime by just paying employees above the $24,000 threshold and calling them "managers" even though they aren't managing anyone or anything. If you have to pay someone 10 hours overtime per week at $24,000 base salary (i.e. $12 hourly @ 50 weeks a year, 40 hours a week), that means paying an extra $180 per week, or $9000 annually. So the employer saves up to $8000 by paying this employee $25,000 and calling the position a "manager".

The new rule changes the numbers/thresholds but keep the structure in place. It will just lead to another round of gaming.

According to the article, some employers are already scheduling meetings with employees to change their compensation packages. It's clear that some employers will elect to hire more part-time help, and others may just adjust base pay, and still others will find ways to nullify the revised rule. Fundamentally, the allocation of profits between the employees is a zero-sum game. Whatever is paid to workers is not paid to corporate management.

The other issue is one-size-fits-all. It's surprising that this wage level does not vary with geography or industry. There is a practical problem here. Pay levels are not publicly available so it is hard to create granular rules, much less to monitor them.

This takes us back to ethics. I don't believe that purely legal approaches will solve the problem of fair pay for workers. Fairness is pitted against self-interest/greed. One positive result of the rule revision is to generate discussion around this topic. Any discussion should include ethical considerations.

The same kind of gaming rules affects many predictive models. The Google Pagerank algorithm is a way to estimate the "importance" of any given webpage. It is quite a powerful scoring mechanism because most people reportedly do not scroll past a few pages of results. Thus, there is a "land grab" for the first-page real estate on Google and other search engines. An entire cottage industry known as SEO/SEM has arisen, which promotes a bag of tricks to game the algorithm. Even though Google does not disclose the full details of the algorithm, this doesn't stop gaming because you can reverse-engineer the algorithm to a certain extent. The gaming defeats the original intention of Pagerank which is to surface the most relevant content for users. More rules lead to more gaming but more discussion of ethics will help.

Dreadful analysis shows the importance of numbersense

Last year, Gizmodo capitalized on the fallout from the Ashley Madison hacking scandal and published a sensational article claiming the website that, if you haven't heard, promotes adultery, has "almost no" real women on it. The subtext is that millions of gullible, disloyal males were paying monthly fees to the website to do nothing or, cue the laugh track, to converse with "badly-designed robots." These men, according to Gizmodo, were buying a "fantasy," and "almost no" hookups were ever consummated.

That conclusion was ridiculous on its face. It assumes that men have no common sense. In fact, not one man but over 30 million men with zero common sense. (Ashley Madison has already been in business for over a decade.)

It didn't however stop the journalist from all kinds of emoting, such as:

the more I examined those 5.5 million female profiles, the more obvious it became that none of them had ever talked to men on the site, or even used the site at all after creating a profile [italics from the original]

In case that isn't extreme enough, she elaborated:

Actually, scratch that. As I’ll explain below, there’s a good chance that about 12,000 of the profiles out of millions belonged to actual, real women who were active users of Ashley Madison.

In casual conversations, I keep hearing this story. Except that the story has been debunked within a week of its publication, but as per the state of the media today, the debunking got a fraction of the press lavished on the original, dreadful piece of data journalism. Most of the outlets that helped spread the initial nonsense never bothered to print the retraction.

What the journalist faced was reality. As soon as the piece got published, a number of readers, both male and female, commented on their personal experiences with the website. There were couples who found love and eventually got married. There were female users who refuted the conclusion that Ashley Madison was "a science fictional future where every woman on Earth is dead." Besides, people with inside knowledge pointed out how the data were completely misinterpreted.

For those interested in "numbersense" in data analysis, it is very instructive to read both the original article and the retraction that is thinly disguised as further juicy finding. How can a data analyst avoid falling into the traps that lead them to utterly invalid results?

***

A lot of "numbersense" has to do with how you process the information you have, and the information you don't. Of the data that you have, what do you believe and what you don't. Of the data you don't have, what assumptions you make.

The journalist believed the hackers when they boasted that the data dump, 20 compressed gigabytes and all, contained all of Ashley Madison's customer data. This turned out to be wrong. Belatedly, it was shown that at least 550 tables exist in the data infrastructure, and the journalist analyzed just four tables! Tellingly, this additional knowledge did not stop her from issuing even more "insights" in the second article of the series.

Further, people with inside knowledge gave the reporter the hint that the information she needed was hiding in plain sight. There was a column in the data table called "ishost" (i.e. "is this user a host?"), and a "host" is internal jargon for a "chat bot" (incidentally, these are making the news by way of Facebook, Microsoft. and Microsoft.) According to the ishost column, there were only 70,000 or so bot accounts, far from the millions of accounts claimed by the journalist!

In the original article, the journalist cheerfully related her process of discovering that no real women used the website. The high point was: "three data fields changed everything." These fields supposedly measured the frequency of specific actions on the site, such as sending emails to other users. It turned out that the columns did not measure any human activity at all. They recorded bot activity, thus invalidating her entire analysis.

For example, "mail_last_time" did not mean "a timestamp indicating the last time a member checked the messages in their Ashley Madison inbox," as asserted in the original article. In fact, insiders told the journalist it indicated the last time a bot sent an email to an Ashley Madison member. 

This is amateur hour: to infer the content of a column of data by the name of the column. One can never guess the intention of the developer who names the column, let alone know whether the column has deviated from its initial definition over time.

And this little situation illustrates perfectly why analysts of Big Data owes it to consumers to be extra careful. Much of Big Data is observational, which means the origin of such data is obscure, or obscured by the organizational layering, or washed away by time. The current practice in database development scoffs at data dictionaries or data flow diagrams or any kind of documentation. The spirit of "agile" development devalues stability in the data environment. So it has become even more arduous than ever before to understand one's data.

***

The essence of numbersense is captured here. Are you the analyst who looks at "mail_last_time" and convinces yourself that it measures human activity and thus proves that no female humans exist on the website? Or are you the analyst who ceaselessly asks questions to get to the bottom of what that column measures?