Big Data, Plainly Spoken (aka Numbers Rule Your World)

Sitting still against the myth that sitting kills

The fad of standing while working may die hard but science is catching up to it.

The idea that standing at work will make one healthier has always been a tough one to believe. It requires a series of premises:

• Using a standing desk increases the amount of standing
• Standing longer improves one's health
• The health improvement is measurable using a well-defined metric
• The incremental standing is of sufficient amount to effect an improvement in health
• No other factors are required to attain the said improvement
• No other factors ameliorate the said improvement (e.g. standing more may expense more energy, causing one to snack more)

Now, the Cochrane foundation has looked at the evidence, and found it lacking (link). Cochrane researchers:

The quality of evidence was very low to low for most interventions mainly because studies were very poorly designed and because they had very few participants. We conclude that at present there is very low quality evidence that sit-stand desks can reduce sitting at work at the short term. There is no evidence for other types of interventions. We need research to assess the effectiveness of different types of interventions for decreasing sitting at workplaces in the long term.

Seems like they haven't passed the first hurdle - does using standing desks actually reduce the amount of sitting at work?

Ergo, don't feel guilty about sitting down.

 

The day after the shock: pollsters, forecasters scratch their heads

Pollsters, forecasters, and the likes were embarrassed by the Bernie Sanders upset in Michigan Tuesday night. Nate Silver called it among the greatest polling error in primary history. Now, they are struggling to explain the big miss. Recall the polls conducted close to the contest showed a Clinton lead of about 20 percent points. The actual outcome was a gap of 1.5 points, with a million votes cast.

This type of miss is hard to explain because any plausible explanation must deal with these facts:

• The miss of 20 percent points is a huge gap to explain
• The same methodology used in other states has generally been accurate

Take, for example, the claim that polls missed young voters because of a bias toward landlines. This explanation doesn't fly because pollsters use the same methodology in each state. If they are not polling enough young voters in Michigan, they most likely did not poll enough young voters in other states either. Besides, the math doesn't work.

Here's an excerpt from Huffington Post (link):

The pre-election polls also appear to have underestimated turnout among young voters, who overwhelmingly support Sanders. That same NBC/Wall Street Journal/Marist poll showed 18- to 29-year-olds making up 15 percent of the electorate, whereas the exit poll showed 20 percent of the electorate in that age group. In the pre-election poll, Sanders won the age group 74 percent to 25 percent; in the exit poll, his margin was bigger -- 81 percent to 18 percent.

But a back-of-the-envelope calculation shows that this could account for perhaps 7% of the gap only. Based on the data cited, the other 85% of the people in the pre-election poll had given Clinton a 32% edge in order for the average to be 20%. Assume now that 80% of the primary voters gave Clinton a 32% edge and 20% (i.e. the 18- to 29-years old group) gave Sanders a 63% edge. The result is a Clinton edge of 13%, still far from accurate.

 ***

Many of the other explanations offered by Huffington Post (link) or FiveThirtyEight (link) has a similar tone. They are plausible but the numbers don't quite work. Perhaps it is combination of factors - the perfect storm. Or perhaps voters did make up their minds or change their minds late in the game.

Also note that all these explanations assume the existence of statistical bias. This source of error is very different from statistical variability - the argument that the error made by the pre-election polls is a one-in-a -hundred year storm.

How do you react to a probability forecast?

There is a healthy debate going on about last night's Michigan primary on the Democratic side, in which Bernie Sanders pulled off a major upset. Polls leading up to the primary put Hillary Clinton ahead by about 20 percent points. That would be a huge margin. Fivethirtyeight, which has been doing a stellar job covering the elections, uses polls heavily in generating forecasts, and they had predicted that Clinton would win with > 99% probability.

The trendy thing to do these days is to issue probability forecasts. I have found it frustrating to be on the receiving end of such forecasts.

Some 538 readers took this opportunity to knock Nate Silver (plus almost every other forecaster) for getting Michigan completely wrong. But Nate has his supporters too. Here is a typical reaction:

From a purely technical perspective, this statement is correct. But the same statement would have been correct, had the forecast been 98%, 97%, ... , 1% !

In theory, on sufficient repeated observations, one can measure the accuracy of such forecasts. But presidential elections only occur once every four years. So there isn't enough data to do this.

I am not piling on 538. I do think they are the best in the business and I respect Nate's ability. But there is the issue of evaluating accuracy when it comes to probability forecasts. If someone says there is 60% chance something will happen, and it happens, is that accurate? if it doesn't happen, is that accurate?

The other reason given above for the miss is that one can't blame the model if the input data, in this case, the polling data, were bad. This again is technically correct. It is a version of garbage in, garbage out. But as someone who builds models, I must admit using this reason is a cop-out. (Just make a model that clings to the polling data.)

 

How to get fired, the data scientist edition

Investors of Herbalife (HLF) are owed a full explanation of the metrics disaster that the management disclosed suddenly to the SEC last week. As a regulator, the SEC should be requesting much more information about what happened. (Click on the 8K filed in March 2016. Here is the LA Times article.)

On Feb 25, Herbalife management held an earnings call to discuss 2015 4th quarter and full year results. Investors were told that worldwide active new members (excluding China) grew by 17 percent in the 4th quarter, year over year. A week later, they revised the number down to 3%. (That is an inflation factor of more than 5 times.)

For the full year, Herbalife management reported growth (again excluding China) of 8%. A week later, they revised the number to 3%.

Breaking out US results, Herbalife management reported 4th quarter growth to be 71% and the revised number, published one week later, plunged to 31%.

In the EMEA region, Herbalife management said 4th quarter members grew by 44%. A week later, they revised the number to 18%.

In November 2015, Herbalife management similarly discussed financial results, comparing third-quarter metrics to those of one year ago.

Herbalife management told investors growth of active members in North America grew by 33%. Four months later, they revised the number down to 2%. (That is an inflation factor of over 15 times!)

The worldwide growth (excluding China) in the 3rd quarater was first reported as 21%. Four months later, management revised the number down to 9%.

In the U.S., growth of 33% was reported for the 3rd quarter. Four months later, management revised the number down to 1%. (Yes, inflation factor is over 30 times.)

For EMEA, growth of 34% is now revised down to 10%.

***

After this sudden disclosure, one can have no confidence in any numbers put out by Herbalife management - not now, not in the future, and not in the past. In the embarrassing filing, these fumbling managers blamed the disaster on "database scripting errors."

They then make a breathtaking generalization: "The Company did not discover these errors earlier because it had limited visibility into the likely rate of change in this metric upon its first use." (my bolding)

The managers imply that it is normal for metrics to be inflated by two or three or thirty times when they are first put to use. I hope the SEC regulators are not buying this nonsense. The statement cannot even be taken at face value. While the metric may have been invented just recently, Herbalife has been around since 1980 so there are plenty of historical data to observe the "rate of change."

Even if the business were entirely new, it would be the responsibility of the data scientists to come up with meaningful metrics, and of management to ensure that they are properly defined and computed.

***

Suspiciously, all the revisions are of the same cloth. The positive growth numbers were all revised down drastically. The few negative growth numbers that were mentioned in the SEC filing were only marginally altered. A drop in North American new members in the 2nd quarter of 2015 was revised from -16% to -15%; worldwide new members (excluding China) in the 1st quarter went from -4% to -2%.

The errors were not random. Positive news were massively inflated while negative news were barely touched. Selected countries in selected quarters, such as Mexico in Q2 and Brazil in Q3, were adjusted slightly in the favorable direction while the aggregate statistics in most quarters were adjusted down dramatically.

When errors are not random, and biased in the direction that inflates financial results, it is normal to suspect foul play. Foul play can be outright fraud, or it can be a blindness caused by a desire to see numbers trend in a positive direction. This blindness is very harmful to data scientists: it causes one to ignore warning signs or not ask enough questions of the data. This is a test case of whether one has "numbersense."

The selective disclosure of data prevents us from seeing the big picture. Herbalife should provide the entire breakdown of membership growth in different regions and countries for each quarter.

***

The report goes into "specifics" about the so-called "scripting errors." But again, the words convey almost nothing useful, even to experts.

The managers blame "errant inclusion of additional categories of data in calculating the metric for parts of 2015 which were not included [in prior periods]." If true, this is juvenile. This means someone changed the formula used to compute the metric without disclosing the changes. And then, there was no auditing of the calculation before its publication.

It is very hard for me to believe that an analyst would expand the definition of a new membership metric, knowing full well that the metric exists to measure growth. If true, it is a case of poor judgment.

The second issue is described as "quarterly aggregation issues which created variances from period-to-period depending on when the greatest level of activity occurred during the relevant period".

I don't understand this sentence. Quarterly aggregation is a simple operation. One simply needs to take the calendar and divide it into four quarters, each day belonging to a specific quarter. Are these managers disclosing that they vary the mapping of days to quarters based on when the maximum activity level occurs?

***

The stock price initially went down by 7.5% after this damning filing but within two days, it has recovered most of that loss, now only down by 2%. This stock price movement indicates that investors do not yet grasp how bad this news is.

Maybe investors were tricked by Herbalife's deceptive statement: "The errors do not impact the company's financial statements in any way."

This statement is true because the new members metric is non-GAAP and does not show up on GAAP financial statements. That is not the full story. Think about why managers want to compute an active new members metric.

Herbalife wants to grow revenues. Revenue growth comes in two ways: increase the number of subscribers, or increase the spending per subscriber. Having a members count allows managers to track revenue per member. So now they can make decisions on whether to invest in acquisition of new subscribers, or in expanding the relationship of existing subscribers.

Since total revenues is a function of number of new subscribers, and since total revenues are not being revised but the number of new subscribers is dramatically reduced, the formula stipulates that the revenue per subscriber metrics must have been dramatically cut as well. If previously investors thought each Herbalife subscriber produces say $1000 in revenues, now the management is disclosing that the number is closer to $500. That is a big deal.

There is one other possibility. It is possible that the subscriber counting system is completely separate from the revenue booking system. I have seen this separation in many companies. It becomes the analysts' job to reconcile the two systems because those numbers should be almost perfectly correlated. So, Herbalife's statement could be true if they did not reconcile the revenues and the subscriber counts. But if this is the case, then we can't believe any of their numbers.