Big Data, Plainly Spoken (aka Numbers Rule Your World)

It's difficult for journalists to get statistics right: the real estate edition

A friend pointed me to a good article on the buoyant real-estate market in Boston (link). The journalist tells us the median home sale price in Suffolk County has gone up 31 percent in the five years since the 2010 crash.

The article makes some excellent points. When it comes to talking about median house prices, it fails basic statistics.

Here is a quote (maybe a misquote) from Barry Bluestone, whom the reporter describes as a "go-to mind" on Boston housing. He complains that "median home sale prices--the metric by which many people judge the market--can be misleading." How so?

He explains: "If a bunch of expensive homes sell all at once, it will appear like median prices have risen rapidly. And if a bunch of lower-priced homes have hit the market recently, the opposite will be true."

That is a strange thing to say. The whole point of using median prices, as opposed to average prices, is that the median--as it depends only on the ranking of prices, not the actual prices themselves--is not easily affected by a few extreme values.

Perhaps he is computing the median prices on a very small sample, such as the sales of homes in a local district within a short period of time. If that is the case, if he only has a handful of prices, then the problem is not which statistic but the sample size.

In addition, the median price is a one-number summary of a list of individual prices. So I must quibble with the use of the plural in this sentence: "median home sales prices in Suffolk County are up more than 31 percent..." If the reporter gets how the median is computed, he would have chosen the singular verb.

 

 

Statbusters: standing may or may not stand a chance

In our latest Statbusters column for the Daily Beast, we read the research behind the claim that "standing reduces odds of obesity". Especially at younger companies, it is trendy to work at standing desks because of findings like this. We find a variety of statistical issues calling for better studies.

For example, the observational dataset used provides no clue as to whether sitting causes obesity or obesity leads to more sitting. Further, as explained in the column, what you measure, and even more importantly, what you don't measure makes and breaks the analysis.

These lessons are highly relevant to anyone working with "big data" studies.

Read the full column here.

What you learned and not learned about machine learning

My friend Liliane pointed me to this HBR article about "machine learning." (link)

As a basic introduction, the article is pretty good. Here are 3 things you learned and 2 things you didn't learn from this article.

3 Things You Learned

1. Machine learning is not the same as learning as you know it. At the end of step 5 ("Iterate"), the author proudly states "It has learned." But what does "learning" mean?
2. Machine learning is an application of statistics. The author defines it this way: "machine learning applies statistical models to the data you have in order to make smart predictions about data you don’t have." The decision tree model predicts by estimating the probabilities of the target behavior within successively smaller subsets of the population. The probability is the proportion of members of a given subset who exhibit some specified behavior, which means it is computed by having replicates of members of the subset. Now, we can't have clones of Karl but we can have lookalikes of Karl. This is where "big data" comes into play.
3. Machines are taught to learn by human beings. The modelers frame the problem, determine the variables used in the model, express the variables in the scale suitable for the model, interpret the probabilities computed by the model, and make decisions based on those probabilities.

2 Things You Didn't Learn

1. Like Nate Silver likes to say, the key to understanding data is probabilistic thinking, something completely absent from the decision tree description. Astute readers might wonder how the "machine" is able to make such precise decisions. In reality, the "machine" make mistakes. Did Karl really remain a subscriber and did Cathy quit? Attached to all those subsets in the bottom layer of the decision tree are estimated probabilities of defection. Cathy may belong to a subset that has an estimated 30 percent chance of quitting - that might be three times the overall defection rate of 10 percent, and thus places Cathy in the high-likelihood-to-defect category. But you know what, 7 out of 10 customers in that subset are projected not to defect. And we haven't even gotten to talking uncertainty about this uncertainty.
2. There is one key human decision that went unmentioned. The machine can go as far as placing any customer in one of those subsets at the bottom layer of the decision tree; each subset has an estimated probability of defection. So, from left to right, the probabilities might run 2%, 3%, 5%, 6%, 10%, etc. No machine can tell the business manager what the cutoff probability is for inclusion in the marketing campaign.

I cover most of these points in my discussion of marketing data in Numbersense (link). In the short HBR article, there is perhaps not enough room to include these deeper issues but not appreciating these points leads to the failure to create value using these models.