Big Data, Plainly Spoken (aka Numbers Rule Your World)

The most famous calendar in America right now

In the leadup to today’s hearing, U.S. Supreme Court judge nominee Brett Kavanaugh produced what he claimed to be calendar entries from the summer of 1982 as evidence that he did not attend a specific party, and therefore he could not have committed the unsavory acts alleged of him.

This story perked my interest from the data perspective. What kind of information is contained in calendar datasets? And how can such data be used to support or invalidate hypotheses?

This discussion is of great relevance beyond the Kavanaugh case because the majority of the data being collected today - surveillance data, transaction data, clickstream, etc. - are all event data: they record when someone did something.

***

When I started this post, no actual pictures of the most famous calendar right now has been published. USA Today did publish a few pictures yesterday. Here is one page to give us a visual:

This post is not about the specifics of Kavanaugh's youthful activities but about the nature of calendar datasets.

A calendar dataset consists of calendar entries. When converting the above image into a spreadsheet, one would create a row for each event (so multiple rows for multiple events on the same day). Each row should contain the following data, laid out in columns:

• The date and day of week of an event
• The time of the event
• The location of the event, address and/or directions to it
• Other people involved in the event
• Duration of the event
• Miscellaneous notes relevant and specific to the event
• Other notes unrelated to the event

In industry parlance, the above list is called a “data dictionary.” It describes the structure of the data. Any data analyst who has used such things realize that these definitions are imprecise. If one randomly selects any two calendar datasets, there will be variance in what one actually gets.

A lot depends on how the calendar owner uses the calendar! Here are some considerations:

Planner or diary. Most people I know use calendars for planning future events. Kavanaugh appeared to use it also as a diary of sorts. Some entries record past events, e.g. a basketball score. Other entries pertain to future events, e.g. going to a movie. This is a key issue in interpretation: if it’s used as a planner, then the data become frozen in time once the event is over but if it’s a diary, the analyst should assume that edits or appends may change the data even after the event date.

Planner entries are tricky because the information may not hold true beyond the date of the event – an event might get cancelled, the calendar owner might decide to skip the event, times or venues or attendees may shift, etc. To prevent mis-interpreting the data, the analyst should strive to annotate the data with whether a calendar entry is for planning or diary.

How changes are made. If the calendar is used as a diary, the owner may revise the information after the event happened, e.g. correct the list of attendees. Even if the calendar is purely used for planning, edits will be inevitable, e.g. the venue of an event might change before the event happens. How are such changes enacted? Some owners may erase and overwrite; other owners may black out and append; still others may strike out and append. In the first case, we have no trace of the change at all; in the second case, we know something has changed but not the specifics; in the third, we have both the old and the new versions of the information.

How complete is the data? A fallacy is to assume WYSIWYG. There is no guarantee that every event in someone’s life is recorded on the calendar. We can assume that every event that the calendar owner wants to record is found on the calendar. Even that assumption is not safe. It happens a few times a month when I forgot to put meetings down on my calendar (most of these I did attend, and then there are the ones that I forgot about because they are not recorded). Missing events appear as missing rows on the dataset.

Kavanaugh did not seem to care as much about the time of events. There are many entries that did not contain when he was supposed to do something. So his calendar dataset contains a large number of missing entries under the column of “time of event.”

How consistent is the data? An analyst can’t even assume that a specific person would follow strict guidelines in terms of filling out a specific item on the data dictionary. Take Kavanaugh’s calendar as an example. On many entries, he did not record the attendees of the event. But on some, he did. When the dataset shows no attendee names for a particular event, does that mean he was alone, or does that mean he chose not to list the attendees? Even if there is a list of attendees, the analyst is hard-pressed to know if the list is complete.

How reliable is the data? Some people are sloppy about details, others are meticulous. Some people correct the entries, other don’t. Some data elements might be deemed not important enough to correct.

How the data was generated. Most of the data will be transcribed (by hand or by optical character recognition software) from the paper calendar to a database. Note, however, some of the data must be inferred. A good example is item (a), the date and day of week of an event. When we read the calendar, we know by the location of the text which date the event is supposed to occur but there is no handwriting of the date! So, in order to generate that column, the analyst must extract (using my own calendar as an example) the year from the cover, the month from the page title and the day from the column name.

Wait – it gets more complicated. Imagine you fill out the space for a given day, so the additional entries are written in the margins with a guiding line to that entry. (Ouch!)

***

I can keep expanding the list of issues but you get the idea. Data is a dirty business. The worst thing that an analyst can do is to presume that the data in front of him/her is (a) complete and (b) accurate.

Now, let’s get to the logic of the Kavanaugh defense. The basic premise is that the calendar did not contain an entry showing that he was at the same party as the accuser, and therefore he was not at such a party, and therefore he could not have committed those alleged acts.

For that logic to hold, we have to believe a further set of assumptions that are not proven by the calendar dataset:

1. That the calendar contained every party that Kavanaugh attended during those months
2. That Kavanaugh primarily used the calendar as a diary recording past events, rather than as a planner for future events, that is to say, information related to the event is true or corrected after the event
3. That for the specific event in question, he was in “diary” mode rather than “planning” mode so that the absence of the party can be inferred to mean he wasn't there.
4. That when he revised entries in his calendar, he never erased past writing.
5. That none of the blacked-out entries contained relevant information to the current situation.
6. That he always listed all attendees of parties he went to.

It’s really hard to use data to prove “absence,” and this is no exception.

***

Be careful next time you interpret your event dataset!

 

Chip cards: example of predicting invariables

A few years ago, I wrote an article for Harvard Business Review, pointing out that data scientists should pay more attention to predicting "invariables" rather than predicting "variables". See here. 

The gist is: it is a lot more reliable to predict an outcome that is invariable, and there are plenty of problems out there of this nature that are being ignored. So those are low-hanging fruits.

It's more sexy to say you can predict when someone dies, or when the peak flu season occurs, etc. but because the outcomes have high variability, the predictive accuracy is "upper bounded". 

This past month, I have been waiting and waiting to see if my credit card company would become smart. The chip on my chip card died. Every time I use the card, at least once a day, I am accumulating data points - each one tells the same story: the chip is dead - the chip is dead. The chip is DEAD.

It usually takes at least three failures for the tablet software to allow me to swipe. Sometimes, the software is so dumb that it will keep asking for re-tries infinitely. When will the software learn that my chip is dead? Ideally, some system is monitoring the data and be able to predict that my chip has failed which triggers a new card to be sent to me.

Think twice before you Waze

Among many commuters, Waze (acquired by Google) has become a staple - some might say, a drug. Waze is an app that finds you the "best" (i.e. shortest) route to your destination. It is famous for routing people through strange, local roads, and intuitively dubious routes (e.g. it once told me to drive through an airport). Because it takes crowd-sourced congestion data as one of its inputs, there are reports that some residents of previously quiet neighborhoods are sending fake signals into the system to discourage Waze from sending traffic onto their streets.

***

If you are a fan of Waze, let me pose you a few questions to think about.

Let's say you are trying to get from point A to point B. 

Let's say there are 50 different possible routes to get from A to B. Some of these are marginally different (e.g. take a left turn one block later) while others are substantively different (e.g. avoid a stretch of highway). 

At any moment in time (for example, at the time you enter your destination), one of those 50 routes is the shortest route between A (where you are) and B (where you are going).

Also, there are a bunch of other drivers also trying to get from A to B. 

If Waze sends all the drivers to this shortest route, then this route will no longer be the shortest route. 

So some cars will be sent down a route that is not currently the shortest route. Do these drivers know they are not on the shortest route? (Do you?)

Do Waze users know how the algorithm decides who to send down which route?

Should the software vendor disclose how they make decisions like this?

***

Of course, these GPS route planners have transformed our driving experiences in a majorly positive way. Gone are the days where I am juggling and folding a paper map while trying to steer the car and paying attention to the traffic around me. 

When you compare Waze to say a standard GPS route planner, what is the real difference?

***

P.S.

(a) If we allow the road system to run for a long term, at "equilibrium," one should expect all 50 routes to have the same travel time (but different amounts of traffic). If there is an identifiable shortest route, then some drivers will shift there, making some other route better.

(b) Almost all press coverage of the "bias" of algorithms are related to identity politics. However, the truly difficult issues related to bias have nothing to do with race, income, gender, etc. Here is an example of a fairness issue that should get much more attention.

A purely mathematical theory of risk is not enough 2

,

This is the continuation of the last post on Chapter 5 of John Adams's excellent book on Risk. In the previous post, I discussed the problems of measurement that undermines the mathematical theory of risk. This post focuses on the particularly compelling example of traffic black spots, which reveals some risks in the statistical interpretation of risk data.

***

What Adams said:

Parts of the [road] network that have experienced bad spell are defined as accident black spots. When they are "treated", the numbers of accidents usually go down - but they probably would have gone down anyway.

He goes on to show a table of data used to identify traffic black spots. By convention, you list all the sites by the rate of accidents in descending order. The top sites with the most accidents are grouped together as black spots. In that table, 14 sites were identified as the worst sites with on average 3.53 accidents in a "before" period. In the "after" period, the accident rate dropped to 1.50.

Here's the catch: there was no treatment separating the "before" and "after" period. The analysis is an "A/A test," which I have frequently advocated as a way for data analysts (or our clients) to get a feel for "uncertainty". So the halving of the accident rate has zilch to do with any safety measure. One plausible explanation for the sudden drop in accident rate is the phenomenon of regression to the mean: if you select the worst sites for any given time, then in the next window of time, it is likely that those sites will experience below-average accident rates. (Vice versa, if you focus on the best sites for any given time, it is likely that those sites will suffer above-average accident rates in the after period.)

If you read the table of data carefully, you will notice that the 2000+ sites that were perfectly safe (zero accidents) in the "before" period subsequently suffered an increase in the accident rate in the "after" period. The average accident rate in the "after" period was 0.19, still low as an average number, but when multiplied by 2,000 sites, it can't be ignored. To see this even more clearly, in the "after' period, the black spots contributed 1.50x14 = 21 total accidents while the "no accident" spots contributed 2000x0.19 = 380 total accidents, 18 times more than the black spots!

The disease of the "Top X" analysis is an epidemic in our Big Data era; the OCCAM nature of these datasets worsens the disease (link). Pouring money on the "top" prospects is folly when one has not established causation of treatment and effect.

***

"Top X" type analysis is regrettably common in practice. You take a set of data and divide up the data into groups (often called deciles, quartiles, etc. depending on how many groups). Then you can compute some metric of the "lift". For example, in the "before" period, in the aggregate (> 2,600 sites), the average accident rate was 0.26. So if we predict that accidents will happen in the "black spots" (i.e. the worst class of sites), then the lift is 3.53/0.26 = 13.5 times. That is to say, the black spots are predicted to see 13.5 times more accidents than the average site.

Unexpectedly, in the "after" period, this lift number plunged to below 6 times. In other words, the predictive power is poor (when computed on data not even collected at the time of model-building). Given that no treatment was applied, if the predictive model captured the underlying dynamics, we should expect to see the lift number to be close to 13 times again. (We're assuming no strong known other factor that could cause the accident rates to collapse during the "after" period - in fact, in aggregate, the total number of accidents slightly increased.)

Here is the more subtle error: usually, the "after" period is after some safety measure is applied to the black spots (and not the other sites). The expected result is a drop in accident rate at the black spots, providing evidence that the safety measure actually caused the reduction in accidents. It is therefore very convenient for analysts (and policymakers who hire them) to conclude that the drop from 13.5 times to 6 times is evidence of the effectiveness of the safety measure. What the above analysis shows is that even in the absence of a safety measure, the accident rate could have dropped by half! So if you did apply a safety measure, it would need to produce a massive effect beyond the regression to the mean.

There are two reasons why analysts may fall prey to this type of bad analysis: one is confirmation bias - seeing in the data what you want or expect to see; the other is being what Nassim Taleb calls "fooled by randomness".

***

Adams goes on to discuss "accident migration". This is a second-order phenomenon: sites near to "treated black spots" often see an increase of accidents - the hypothesis is that since drivers take more caution when passing through black spots, they end up being more careless after the black spots are treated and so accidents spike up in nearby areas. (Adams does point out that it is possible that accident migration is not a separate thing from regression to the mean.)

Some related reading is the material on highway congestion in my book, Numbers Rule Your World (link). The problem of road congestion is slippery. If you successfully reduce the amount of traffic along one route, drivers will then change their habits to return that route to congestion - for example, they may have chosen to go along local roads before.

A purely mathematical theory of risk is not enough 1

I have been reading the excellent book by John Adams titled Risk (link). This is a geographer's treatment of a subject that is a staple of mathematics, particularly probability math. A mathematical treatment creates objects called probability distributions, which are then taken as complete representations of risk. Adams challenges that construct, bringing a social scientist's sensitivity to the table. In particular, he points out how the mathematics of risk is undermined by measurement issues (i.e. data issues) and statistical issues. He is not invalidating math, just pointing out large cracks that are often ignored.

I will provide a more comprehensive review of the book eventually. I'm very excited by Chapter 5, titled "Measuring Risk", and specifically the example of "traffic black spots". This example is very instructive for anyone who is interested in the practical implications and interpretation of risk measures.

The post got long so I have split it into two parts. The second part will be posted on Monday, and it concerns a delicious bit of analysis related to traffic black spots.

 

 

 

***

Here are some key sentences related to the quality of the collected data:

1. The connection between potentially fatal events [accidents] and actual fatal events is rendered extremely tenuous by the phenomonen of risk compensation.... avoiding action is taken, with the result that there are insufficient fatal accidents to produce a pattern that can serve as a reliable guide to the effect of specific safety interventions. As a consequence, safety planners seek out other measures of risk, usually - in ascending order of numbers but in decreasing order of severity - they are injury, morbidity, property damage and near misses.
2. It is much easier to achieve "statistical significance" in studies of accident causation if one uses large numbers rather than small. Safety researchers therefore have an understandable preference for non-fatal accident or incident data over fatality data... in exercising this preference, they usually assume that fatalities and non-fatal incidents will be consistent proportions of total casualities.
3. [Evoking some data from London] The correlation between fatality rates and injury rates is very weak. Is the weak correlation real or simply a recording phenomenon? How many injuries equal one life?
4. Uncertainty in the data increases as the severity of the injury decreases. The fatality statistics are almost certainly the most accurate and reliable of the road accident statistics... the categorization and recording of injuries is generally not informed by any evidence from a medical examination.... [A British Medical Asssociation report said that] only one in four casualties classified as seriously injured are, in fact, seriously injured and many of those classified as slightly injured are in fact seriously injured..... some 30 percent of traffic accident casualties seen in hospital are not reported to the police, and that at least 70 percent of cyclist casualties go unreported.

This last point is widely applicable in the data science/analytics world. We often have large amounts of unreliable data, and small amounts of more reliable data. What we hope to have - and we don't - is large amounts of reliable data. A lot of bad analyses results from assuming we have large amounts of reliable data.

 

Good job Bloomberg, and some comments on ad measurement

I'm continuing to see important articles as our journalists shine light on the tech companies. This Bloomberg article reveals a previously secret relationship between Google and Mastercard. It is important also as a reminder that the intense focus on Facebook is misguided - as other big tech companies engage in similar practices.

I'd recommend reading the article in full. Here I'd like to bring out a few key observations for you to chew on.

1) The payment processors have become part of the surveillance network that has been constructed by the tech industry. In the past (when I was working in that industry almost two decades ago), payment processors played the role of toll booths that happened to be a very lucrative business collecting cash as the transactions stream through. Users' purchase data were safeguarded and remained private.

Sometime in the last 5 to 10 years, the payment processors decided that the purchase data is a goldmine. They now monetize this data through mostly secret deals with all kinds of companies. Most card-users are not aware that their purchases are being disclosed to third parties.

An analogy is Gmail and other web-based email systems. Gmail in particular pioneered the concept of reading users' emails to extract information. It is now known that Gmail allows selected partners to extract all kinds of information from emails, such as shopping receipts, boarding passes, etc. We knew this because of the scandal about the Unroll.me service in which the service provider extracted shopping information from emails under the pretense of finding email lists from which users can be unsubscribed from.

 

2) Next time reporters hear the tech companies talk about "anonymized" data or hiding of "personally identifiable information", they really need to press the companies for details. Look out for clever word play. It is simply not possible to get value from these datasets without access to information about each person.

In this instance, Google is merging two datasets, the clickstream (what ads people clicked on) and Mastercard purchases, with the goal of claiming that person A purchased item X after clicking on some Google ad. The Mastercard data contain your card number, and/or your name. The clickstream data contain one or more of the following: your Google "cookie", your Google user name, or your email. You can't merge the datasets unless there is a common field. In this case, it will most likely be your name or email address. How could that be "anonymous"? Good question if you're asking it. And the reporters should be pushing on this point.

Further, if the data are fully anonymized, the potential utility of the data is much lower, and the price marketers are willing to pay for them is much lower. If I don't have your name and address, how can I send you a piece of junk mail?

 

3) While the article points out the violation of privacy of this partnership, it inadvertently legitimizes the highly dubious science behind matching the clickstream data and the Mastercard purchase data. The reporter described the program as "powerful" and "potent" but should have talked to third-party experts to understand this science better. I'll just list several major problems with this approach:

• As far as I know, Mastercard does not have line-item level details on our purchases. It may know that you spent $500 at Best Buy, but it doesn't know that that total includes an iPad and a TV, for example. So if the ad shows a TV, how does Google know that Best Buy purchase included a TV? The reporter claims that Google knows that someone bought "red lipstick" with her Mastercard. I find that dubious.
• Do you remember what ads you clicked on 30 days ago? 15 days ago? 5 days ago? This system counts any purchases made on Mastercard as long as 30 days ago as "caused" by an ad click.
• Does Google know what these users are doing on non-Google websites? For example, does Google know that the user who saw a Google ad 10 days ago also saw a Facebook ad for the same product 3 days ago?
• Does Google know anything about the users' interactions offline? For example, the user who clicked on the Google ad 10 days ago may have heard a radio ad, seen a TV ad, talked to a friend about the product, been handed a flyer at the mall, saw a piece of direct mail, etc.

The reporter suggested that Facebook, the other digital advertising behemoth, is also pursuing similar deals with payment processors. If Facebook were to adopt this same method for measuring ad performance, then both Facebook and Google would have claimed credit for the same purchase, as each would have matched that purchase to its own clickstream data. One solution to this particular issue is to merge the Facebook and Google data, which simply merges two surveillance networks into one extra-large surveillance network.