Big Data, Plainly Spoken (aka Numbers Rule Your World)

New Facebook data breach, and the importance of Business Intelligence

 

On Friday, Facebook announced that hackers have gained access to personal data of at least 50 million users (see here for example). Analysts immediately connected this incident to the Cambridge Analytica scandal. How is this data breach different from the Cambridge scandal?

How the data was breached

One should take any announcement of how a data breach occurred with a grain of salt: it's obvious that companies will not publish anything that attracts lawsuits; plus, it's unclear that there is a penalty for lying about the real reasons for a data breach. There is no external auditing, and the companies control the narratives and statistics surrounding these events.

Based on what Facebook told us, the Cambridge Analytica scandal involves Facebook partners gaming the system to obtain data about Facebook users. At worst, it is a violation of community standards, i.e. claiming to be doing academic research. The research firm uses tools provided by Facebook to obtain the data. (The firm maintains that it disclosed to users that it was using the data for non-research purposes.)

In the present case, Facebook claims that a combination of coding bugs (i.e. unintended features) enabled unknown partners to gain access to the user's entire account. Not even that, but any accounts on third-party apps or websites that the user signs on to using Facebook.

 

How the breach was discovered

The current scandal demonstrates the value of the business intelligence/business analytics functions within companies. We were told that Facebook first realized that certain metrics were showing unusual trends, and upon investigation, they discovered the bugs.

This is entirely believable. That's what happens when you have good data reports. They surface anomalies. These then have to be investigated. These investigations are extremely tricky because all you know is the trends are different. There are a thousand reasons for the shift. The analyst's job is to establish a cause-effect. Especially since the development community adopted "agile" practices, all kinds of self-imposed changes are occurring all the time with no warnings. For a site as large and complex as Facebook, it takes a huge effort just to get a list of all site changes within some specified time window!

What complicates this situation more is that the vulnerability was traced to multiple bugs, not just one. I could imagine the twists and turns and the false alarms that were generated during the investigation.

The nature of this problem is no different from an investigator trying to chase down an e-coli outbreak, which is detailed in Chapter 2 of Numbers Rule Your World (link).

The data science community is guilty of talking down on the business intelligence function. There is a misperception that BI is for less skilled people doing boring things. The reality is there is more science in BI than in so-called data science (defined here as software engineering). Science, after all, is about figuring out why things are as they are. Engineers, by contrast, use our understanding of science to change the way things are.

 

You've been warned

In my debut Youtube video, released a few weeks ago, I explain how Facebook collects data about you. My biggest tip about protecting yourself is to not use Facebook to log onto other websites. Convenience is the drug that lures you into the trap.

Think about those one-password services. Instead of hundreds of password, you have one. So the thief needs only one password to assess hundreds of sites. Using Facebook to log onto other websites makes Facebook the centralized point of attack by the bad players.

Note that this is not limited to Facebook. Using Google, Yahoo, Amazon, etc. to log onto other services is no different!

If it's more convenient for you, it's also more convenient for your enemies. Remember that.

Also in the video, you will learn

• why did Facebook invent the Like button
• how do Facebook know what you are doing outside Facebook
• why not clicking doesn't mean you're not being tracked
• why you shouldn't use Facebook to log into other services

See the video here, and send me ideas for future episodes!

The most famous calendar in America right now

In the leadup to today’s hearing, U.S. Supreme Court judge nominee Brett Kavanaugh produced what he claimed to be calendar entries from the summer of 1982 as evidence that he did not attend a specific party, and therefore he could not have committed the unsavory acts alleged of him.

This story perked my interest from the data perspective. What kind of information is contained in calendar datasets? And how can such data be used to support or invalidate hypotheses?

This discussion is of great relevance beyond the Kavanaugh case because the majority of the data being collected today - surveillance data, transaction data, clickstream, etc. - are all event data: they record when someone did something.

***

When I started this post, no actual pictures of the most famous calendar right now has been published. USA Today did publish a few pictures yesterday. Here is one page to give us a visual:

This post is not about the specifics of Kavanaugh's youthful activities but about the nature of calendar datasets.

A calendar dataset consists of calendar entries. When converting the above image into a spreadsheet, one would create a row for each event (so multiple rows for multiple events on the same day). Each row should contain the following data, laid out in columns:

• The date and day of week of an event
• The time of the event
• The location of the event, address and/or directions to it
• Other people involved in the event
• Duration of the event
• Miscellaneous notes relevant and specific to the event
• Other notes unrelated to the event

In industry parlance, the above list is called a “data dictionary.” It describes the structure of the data. Any data analyst who has used such things realize that these definitions are imprecise. If one randomly selects any two calendar datasets, there will be variance in what one actually gets.

A lot depends on how the calendar owner uses the calendar! Here are some considerations:

Planner or diary. Most people I know use calendars for planning future events. Kavanaugh appeared to use it also as a diary of sorts. Some entries record past events, e.g. a basketball score. Other entries pertain to future events, e.g. going to a movie. This is a key issue in interpretation: if it’s used as a planner, then the data become frozen in time once the event is over but if it’s a diary, the analyst should assume that edits or appends may change the data even after the event date.

Planner entries are tricky because the information may not hold true beyond the date of the event – an event might get cancelled, the calendar owner might decide to skip the event, times or venues or attendees may shift, etc. To prevent mis-interpreting the data, the analyst should strive to annotate the data with whether a calendar entry is for planning or diary.

How changes are made. If the calendar is used as a diary, the owner may revise the information after the event happened, e.g. correct the list of attendees. Even if the calendar is purely used for planning, edits will be inevitable, e.g. the venue of an event might change before the event happens. How are such changes enacted? Some owners may erase and overwrite; other owners may black out and append; still others may strike out and append. In the first case, we have no trace of the change at all; in the second case, we know something has changed but not the specifics; in the third, we have both the old and the new versions of the information.

How complete is the data? A fallacy is to assume WYSIWYG. There is no guarantee that every event in someone’s life is recorded on the calendar. We can assume that every event that the calendar owner wants to record is found on the calendar. Even that assumption is not safe. It happens a few times a month when I forgot to put meetings down on my calendar (most of these I did attend, and then there are the ones that I forgot about because they are not recorded). Missing events appear as missing rows on the dataset.

Kavanaugh did not seem to care as much about the time of events. There are many entries that did not contain when he was supposed to do something. So his calendar dataset contains a large number of missing entries under the column of “time of event.”

How consistent is the data? An analyst can’t even assume that a specific person would follow strict guidelines in terms of filling out a specific item on the data dictionary. Take Kavanaugh’s calendar as an example. On many entries, he did not record the attendees of the event. But on some, he did. When the dataset shows no attendee names for a particular event, does that mean he was alone, or does that mean he chose not to list the attendees? Even if there is a list of attendees, the analyst is hard-pressed to know if the list is complete.

How reliable is the data? Some people are sloppy about details, others are meticulous. Some people correct the entries, other don’t. Some data elements might be deemed not important enough to correct.

How the data was generated. Most of the data will be transcribed (by hand or by optical character recognition software) from the paper calendar to a database. Note, however, some of the data must be inferred. A good example is item (a), the date and day of week of an event. When we read the calendar, we know by the location of the text which date the event is supposed to occur but there is no handwriting of the date! So, in order to generate that column, the analyst must extract (using my own calendar as an example) the year from the cover, the month from the page title and the day from the column name.

Wait – it gets more complicated. Imagine you fill out the space for a given day, so the additional entries are written in the margins with a guiding line to that entry. (Ouch!)

***

I can keep expanding the list of issues but you get the idea. Data is a dirty business. The worst thing that an analyst can do is to presume that the data in front of him/her is (a) complete and (b) accurate.

Now, let’s get to the logic of the Kavanaugh defense. The basic premise is that the calendar did not contain an entry showing that he was at the same party as the accuser, and therefore he was not at such a party, and therefore he could not have committed those alleged acts.

For that logic to hold, we have to believe a further set of assumptions that are not proven by the calendar dataset:

1. That the calendar contained every party that Kavanaugh attended during those months
2. That Kavanaugh primarily used the calendar as a diary recording past events, rather than as a planner for future events, that is to say, information related to the event is true or corrected after the event
3. That for the specific event in question, he was in “diary” mode rather than “planning” mode so that the absence of the party can be inferred to mean he wasn't there.
4. That when he revised entries in his calendar, he never erased past writing.
5. That none of the blacked-out entries contained relevant information to the current situation.
6. That he always listed all attendees of parties he went to.

It’s really hard to use data to prove “absence,” and this is no exception.

***

Be careful next time you interpret your event dataset!

 

Chip cards: example of predicting invariables

A few years ago, I wrote an article for Harvard Business Review, pointing out that data scientists should pay more attention to predicting "invariables" rather than predicting "variables". See here. 

The gist is: it is a lot more reliable to predict an outcome that is invariable, and there are plenty of problems out there of this nature that are being ignored. So those are low-hanging fruits.

It's more sexy to say you can predict when someone dies, or when the peak flu season occurs, etc. but because the outcomes have high variability, the predictive accuracy is "upper bounded". 

This past month, I have been waiting and waiting to see if my credit card company would become smart. The chip on my chip card died. Every time I use the card, at least once a day, I am accumulating data points - each one tells the same story: the chip is dead - the chip is dead. The chip is DEAD.

It usually takes at least three failures for the tablet software to allow me to swipe. Sometimes, the software is so dumb that it will keep asking for re-tries infinitely. When will the software learn that my chip is dead? Ideally, some system is monitoring the data and be able to predict that my chip has failed which triggers a new card to be sent to me.

Think twice before you Waze

Among many commuters, Waze (acquired by Google) has become a staple - some might say, a drug. Waze is an app that finds you the "best" (i.e. shortest) route to your destination. It is famous for routing people through strange, local roads, and intuitively dubious routes (e.g. it once told me to drive through an airport). Because it takes crowd-sourced congestion data as one of its inputs, there are reports that some residents of previously quiet neighborhoods are sending fake signals into the system to discourage Waze from sending traffic onto their streets.

***

If you are a fan of Waze, let me pose you a few questions to think about.

Let's say you are trying to get from point A to point B. 

Let's say there are 50 different possible routes to get from A to B. Some of these are marginally different (e.g. take a left turn one block later) while others are substantively different (e.g. avoid a stretch of highway). 

At any moment in time (for example, at the time you enter your destination), one of those 50 routes is the shortest route between A (where you are) and B (where you are going).

Also, there are a bunch of other drivers also trying to get from A to B. 

If Waze sends all the drivers to this shortest route, then this route will no longer be the shortest route. 

So some cars will be sent down a route that is not currently the shortest route. Do these drivers know they are not on the shortest route? (Do you?)

Do Waze users know how the algorithm decides who to send down which route?

Should the software vendor disclose how they make decisions like this?

***

Of course, these GPS route planners have transformed our driving experiences in a majorly positive way. Gone are the days where I am juggling and folding a paper map while trying to steer the car and paying attention to the traffic around me. 

When you compare Waze to say a standard GPS route planner, what is the real difference?

***

P.S.

(a) If we allow the road system to run for a long term, at "equilibrium," one should expect all 50 routes to have the same travel time (but different amounts of traffic). If there is an identifiable shortest route, then some drivers will shift there, making some other route better.

(b) Almost all press coverage of the "bias" of algorithms are related to identity politics. However, the truly difficult issues related to bias have nothing to do with race, income, gender, etc. Here is an example of a fairness issue that should get much more attention.