Big Data, Plainly Spoken (aka Numbers Rule Your World)

ASA President meets OCCAM data

Just leaving this quote from ASA President Jessica Utts here (Source: Amstat News Dec 2016):

A few days ago, I was in Vietnam and took a four-hour bus ride from Ha Long Bay to Hanoi. When I arrived, my fitness tracker had given me credit for taking 9,124 steps and climbing 81 flights of stairs during those four hours, even though I only left my seat once during a short rest stop. ...

In the opposite extreme, I once walked the full length of the Atlanta airport with my hand on my four-wheeled suitcase and got no credit for any steps. I've noticed a similar lack of credit when wheeling a grocery cart, and pushing a baby stroller allegedly has the same effect.

Great example of how (seemingly) complete data con the analyst. Imagine the data analysts and "scientific" researchers mining and squeezing every ounce of information out of such data with their algorithmic bags of tricks.

And this is not just fun and games, either.

The health plan where [her friend] works sets rates based on data acquired from employees' personal fitness devices!

What causes trouble is the nature of the data. Much of the data we analyze nowadays are "adapted," collected originally for some other purpose. Here, the fitness trackers were conceived as toys that have a potential health benefit, an objective in which the devices need only be marginally accurate. The data then get packaged up and eventually end up in some insurance company's database. An analyst now pulls the data out and is having a field day revamping the statistical models by adding a source of new data. The models may even improve a little in the aggregate because the data are somewhat accurate on average.

But at the individual level at which the data get utilized, there are many inaccuracies that bias the models in a discriminatory way. For example, people who walk around pushing baby strollers (i.e. people of a certain age and more likely women) are more likely to have underestimates, which in the insurer's new model are regarded as signals of lower enthusiasm for fitness.

Worse than that, if one knows that the health plan sets rates based on the number of steps taken, one can easily hang the device off one's dog, or design any number of tactics to fool the machine.

Much of the "smarts" in data analyses occur prior to the analyses. Being relentless in understanding how data were collected, especially when they are collected by third parties with different priorities and incentives, goes a long way. Business managers who buy the end products without inquiring about the data sources do so at their own peril. Lots of money can be lost by investing in counterproductive, Big Data-driven smart-playthings.

***

The Fitbit-type data is a great example of OCCAM: observational, no controls, seemingly complete, adapted and merged datasets that are the norm in the Big Data age - and such data should not be analyzed without a ton of thinking!

More on OCCAM data here, here.

What do these items have in common?

What do the following items have in common?

A. Motion-detection light switch in your office that shuts off while you're typing at your computer

B. Automatic flush that turns on while you're seated

C. Voice-recognition system that picks up ambient noise and asks you to repeat something you didn't say

D. Auto-correct software that flips a correct spelling to the wrong one you didn't anti-auto-correct the last time

E. Fuzzy-logic rice-cooker that won't let you extend the cooking time unless it has completely cooled down

F. Online search box that gives you useless results if you don't select one of the matched items in the drop down menu

G. Banner ads that follow you everywhere advertising the item you just purchased an hour ago

H. Online retailer recommending that you buy another big-screen TV a few days after you bought one from the same store

Explain this list or add to it!

Inspired by water leaks

For me, 2016 is a year of water leaks.

I was forced to move apartments during the summer. (Blame my old landlord for the lower frequency of posts this year!) That old apartment was overrun by water issues. In the past four years, there were two big leaks in addition to annual visible "seepage" in the ceiling. The first big leak ruined my first night back from Hurricane Sandy-induced evacuation. It was an absolute gusher as a pipe broke in the apartment right above me - people were knocking on my door and I woke up to find a downpour through the light fixture in my kitchen.

By contrast, the second big one was a slow-moving leak behind the closet in my living room.The water coursed through ten apartments from the top of the building downwards, eventually reaching the ground-floor tenants who raised the alarm. By the time I returned home from work, everything in the closet was soaked, and water started dripping from the living-room ceiling.

Imagine one is building a statistical model to detect water leaks in apartment buildings. (You know that's where I was headed...)

***

The first key issue to consider is the speed of the leak - think of it as the strength of the signal. The first downpour was quickly detected, and well-contained because it was so strong. The downstairs tenant pounded on my door, and the superintendant immediately turned off the water supply. The second incident was drip-drip as well as hidden behind the closet, and therefore it took a while to get detected. When you have a weak signal, or if the signal is hidden or mixed in with noise, you have a much harder problem on your hands.

Traditionally, we build models that presume some uncertainty, and rely on discovering causal relationships or persistent correlations. Recently, as I shall discuss below, some models can be rendered effective just by collecting more of the right data.

As a statistician, I like to find auxiliary data that are correlated with the signal. So I'd study the pattern of piping in the building, past incidents of leakage, causes of water leaks, and so on. In a different incident in a relative's building, also in 2016, the water leak was caused by a defective valve installed on all the dishwashers in the building, courtesy of a recent building-wide renovation project. Several other apartments already had leaked before for the same reason so in principle, subsequent leakage events could well have been predicted by the insurer. In this paradigm, I use past data to establish causal pathways (or at least persistent correlations), which allows me to issue probabilities of leakage for each apartment in the building.

A new paradigm of predictive modeling is taking shape but poorly understood. These "Big Data" modelers would call for installation of sensors everywhere. Imagine a sensor had been added to the dishwater, and every five seconds, it would transmit a message, proclaiming "I'm not leaking", "I'm not leaking." In this way, any leak would be found within five seconds of occurring. Surely, this model could outperform traditional models if predictive accuracy were measured.

***

While the predictive accuracy of this sensor-driven model may look better, the skill in building models has played no role in it. The improvement is achieved through surveillance. It is as if we had hired a 365/24/7 guard to look over every dishwasher in the building.

A similar situation has played out in the digital advertising world in the last 5 years or so. A fast-growing segment of this market is called "remarketing." This marketing tactic explains why many of us are stalked by corresponding ads after we browsed a certain product webpage, or placed an item into the shopping cart, or purchased something and paid for it.

Any model will tell you that the further you have ventured into the purchasing cycle, the more likely you will consummate the sale. For example, those who have browsed and placed an item into the shopping cart are more likely to purchase this item than those who merely browsed and did not place the item in the shopping cart. So if the analyst knows that you have placed something into the shopping cart, he or she is smart to stalk you around cyberspace with ads for that product - the remarketing ads placed by the analyst then has a better chance of being attributed the "cause" of the transaction.

But the act of placing something in the shopping cart is not a real cause of the transaction. It is merely a necessary step in the buying process, closer to the desired final act of paying for the item. Every final transaction must pass through this step. To the extent that putting the item in the shopping cart predicts a purchase, it is due to active surveillance of the process. Our understanding of why the customer did what he or she did has not improved at all.

Remarketing models will predict purchasing behavior better than other types of models, and this advantage did not arise from better skills. The analyst is just utilizing data collected closer to the time of the subsequent transaction. The role of the remarketing ad is facilitating the execution of an already-made decision. It sometimes gets it wrong, usually because of an already-executed purchase, or a prior decision not to purchase the item (after browsing it). In these cases, the model falls for false-positive signals.

The trouble with remarketing is that it tends to send money to the wrong players: the remarketing ad is crowned the "cause" of the transaction while the hidden true cause of the transaction (e.g. a celebrity endorsement) is afforded no compensation. As advertising dollars shift more and more to remarketing, due to its apparent predictive prowess, the system robs the other channels that may have created the demand for the product.

***

In the above example, the problem of predicting who will consummate a purchase has been turned into a race of who can insert an ad closest to the purchase event, which has in turn become the problem of who can gather the clearest signals as close to the purchase event as possible. No sophisticated analysis is required.

I now return to the sensor-driven predictive model of water leaks, and the same dynamic applies. The sensors are so close to the action that we don't need complex analysis to get a good prediction.

Is that last point really true? I think there are pre-conditions for this type of models to work well. The theory behind the leakage, i.e. that the dishwater valve is the sole, known cause of water leaks, must be rock-solid. If other valves in an apartment are also defective, this model will miss some of the leaks. If the dishwater valve fault is necessary but not sufficient for leaking water, then the model will suffer from some false-positive errors.

The success of the sensor-driven model also presumes the sanctity of the data. It is possible - though the odds need to be measured - that these cheap sensors may once in a while generate false signals, or fail to transmit, say due to end of life. In a world filled with gazillions of sensors (imagined by those tech execs pushing the era of "Internet of things"), even a tiny per-sensor fault rate generates a huge number of failures.