Big Data, Plainly Spoken (aka Numbers Rule Your World)

Come hear me talk about A/B Testing next Tuesday #optimizely

Coming Tuesday, I'm talking about A/B Testing at the Optimizely Experience event in New York. It's at the Intrepid Museum, a great setting. My talk will address questions that come up frequently in practice when you start doing a lot of online experiments. Testing is one big area of business analytics, and an area in which practical experience has to complement textbook learning. Hope to see you there.

You can register for free here.

Book reviews

I have neglected to acknowledge the good folks who have posted reviews of my books in the last several months. Thank you, readers!

***

Tom Peters, the business guru and author of the business classic In Search of Excellence, tweeted about Numbersense: "On my 13-hour Boston-Dubai flight, I re-read cover-2-cover Kaiser Fung's superb-useful-fun book Number Sense".

Numbers Rule Your World is #7 on Gregory Ciotti's list of 25 underrated books on persuasion. This sentence makes me happy:

It straddles statistics, persuasion, and psychology, and through clear writing, addresses what could be an incredibly boring topic for some readers (the application of statistics and how they affect you) and turns it into a really easy read.

Here is a sentence from Christopher Rollyson's detailed review of Numbers Rule Your World:

Fung is an talented storyteller who revels in thinking about information in creative ways.

Here is Ryan Bower on Numbersense:

In his second book, Fung captures the same informative style of his blog in an entertaining long-form version, with a terrific perspective on Big Data.

Uri Bram, author of Thinking Statistically, has a collection of his favorite books.

Wayne Folta said: "Remember not to overlook the Epilogue! It’s a Real Data Science story, that illuminates the part of Data Science that you don’t usually read about."

Numbersense is on Blue Bottle Biz's Top Five Big Data books.

Rakesh Arya reviewed Numbersense:

Numbersense is not another book of statistics or data analysis that tell about how to manage, represent and analyze data but it shows, using many examples, how to develop the sense to recognize ‘misleading behaviour’ of big data.

***

If you like the book, please consider writing a review. If you have private comments, you can write me here.

Get your copies of the books here and here.

 

Davenport on targeted marketing

Tom Davenport is one of the leading voices on business analytics, and he has a new piece titled "Why are most 'targeted' marketing offers so bad?" in which he expanded on a question I raised in my HBR article. Tom's book Competing on Analytics is a classic. He has a great appreciation for the business of the data business.

In the new feature, Davenport classifies marketing offers he gets into five types:

retargeted offers, well-meaning but poorly-targeted offers, offers that benefit the offerer rather than the potential consumer, offers that are OK except for the context, and well-targeted offers that benefit you

and he certainly speaks some truths.

On retargeted offers, he reminds marketers "for the most part, if we abandon a search or purchase, we intended to do so."

On well-meaning and poorly-targeted offers (like sending men offers for women's clothing), he suspects that the retailers didn't try hard enough to mine their data.

***

I think there are some technical deficiencies partially responsible for these issues.

Firstly, human behavior and preferences can never and will never be reduced to a set of equations. Thus, every targeting algorithm has to balance false positives and false negatives. I have written about this a lot. Start with Chapter 4 of Numbers Rule Your World or the Groupon and Target chapters in Numbersense.

Secondly, the existence of "retargeting" as a business is entirely due to perversion of measurement, which I address in Chapters 1-2 of Numbersense. I also wrote about how online marketing is measured here. Briefly, the more you flood customers with impressions, the more likely your impression is close in time to a purchase event, the more credit you get for "influence".

Thirdly, the data is noisy and few are investing any time in getting rid of bad data. Just think about it for a second. Let's say you are a guy. If your son let his classmate use your iPad to buy something from a girl's clothing site just once, you are forever tagged as a girl's clothing buyer.

The Big Data mindset to solving this problem: they want to be even more creepy; they want "all of your data". If everything is being tracked, but by hundreds or thousands of different entities, that wouldn't work either, so the Big Data end game is one all-knowing monopolist of all of your data.

But this path is entirely a dead end. Here's something to ponder - the fact that you visited a particular website is today equated to an expression of interest in that website. The data measure what you do, and why you do it.

The solution is humility, and accepting a level of uncertainty. Enhance observed data with more direct, even qualitative data. Remove noise, which is a way of managing the uncertainty.

 

 

 

Upcoming Talks #Dataviz #ABTesting

This is a cross-post to both my blogs.

Thanks to the ~200 or so people who showed up at last week's Data Scientist Meetup in Cambridge, Mass., hosted by John Baker. I gave a brief introduction to the concept of "numbersense", and was part of a panel of "chief data scientists" talking about how to run data teams. Thanks to those who asked questions.

This month, I am back in New York, and will be giving two talks.

First up is the Data Visualization New York Meetup organized by Paul Trowbridge. The link to register is here but it looks like all slots have been taken within days. You should get on the wait list as some registrants will eventually drop out. This event is on Aug 20 (Wed).

 

 

 

 

 

 

On Aug 26 (Tues), I am giving the "thought leader" presentation for the Optimizely Experience. I will be talking about statistical testing for online marketing aka A/B testing. The title of the talk is "Five Questions About Testing You Wanted to Ask But Didn't" unless I come up with something better. You can register here.

This will be a brand-new presentation, and I look forward to sharing my ten-plus years of running online experiments. See you there!

***

Also, please let the organizers at SXSW know you want to hear me and other data viz experts talk about visualizing data in Austin. Jon Schwabish has put together a fabulous panel with people from different parts of the spectrum, and it promises to be an engaging conversation.

To hear us, you have to vote for our panel. The voting starts today. Here is the link to the voting site (updated link), and here is our pitch on Slideshare. Thank you!

 

 

 

 

A report from #JSM Joint Statistical Meetings 2014

Stephen Stigler, the preeminent historian of statistics, gave a great talk at JSM, the annual gathering of statisticians on Monday afternoon in Boston. He outlined seven core ideas ("pillars of wisdom") in statistical research that sets the field apart; these are ideas developed by statisticians that represent significant advances to science and to human knowledge.

As he remarked, each of these advances overturned then-established science, but even today, many people outside statistics are not aware of these learning.

I will briefly recount three of these discredited beliefs about data and uncertainty:

1. Fallacy: Throwing individual level data away reduces the amount of information. What Statisticians Learned: Throwing individual level data away, such as aggregating or averaging, can increase information.

I have written a few posts about this topic, especially on the Junk Charts blog. Take a look at this post from Monday (link) or this post about "loss aversion".

2. Fallacy: Information increases linearly and proportionately with the number of samples. i.e. small data, a few insights; big data, a lot of insights. What Statisticians Learned: Information increases only at a rate of square-root of the sample size, meaning there is diminishing returns to increasing sample size.

A corollary of this is that it will take a lot more effort to squeeze out ever decreasing amounts of marginal information in big data.

3. Fallacy: The only correct way to run an experiment is to alter one factor at a time while keeping everything else unchanged. What Statisticians Learned: The one-factor-at-a-time dogma is wrong; we should ask Nature many questions at the same time.

***

I loved Stigler's talk. I do wonder if we also need to look ourselves in the mirror. If we were to test our students after they took Intro Stats about the seven pillars of wisdom, I suspect we will learn a very unfortunate result, that they will not have appreciated any or most of these points. The intro curriculum is much too focused on mechanics.

***

Here are some of the talks I attended so far:

• David Banks showed an application of LDA (topic models) to a corpus of posts from political blogs. The topic distribution is allowed to vary with time. They attempted a semi-successful automated naming of topics by matching words to Wikipedia articles.
• Madeleine Cule described how Google uses a mixture of experiments and statistical adjustments to compute the effect of display advertising on brand interest. Brand interest, since this is Google, is defined as searching for branded terms on Google. How times have changed... digital advertisers have returned to the old world of measuring indirect brand metrics, abandoning clicks and direct responses.
• Phillip Yelland showed how Google created a machine that generates sales pitches for its sales team whose objective is to increase advertiser's spending with Google. Like Cule's talk, this methodology is heavily influenced by end-user input: in this case, those are explicitly represented in a Bayes Belief Network. For those paying attention, the Google researchers discussed how they do not have all of the data, how true "controls" are almost impossible on the Web, how they are restricted by data collection practices of third parties (i.e. adapted data), how they write "lots of SQL".
• Phillip Yu gave an overview of statistical models of ranking data.
• Cynthia Rudin described a novel predictive model to find clusters (in space and time) of crimes in Cambridge, MA. Good questions from the floor.
• Someone from Nielsen (I only heard the second half of this talk) mentioned a lot of practical problems with set-top box data. In this setting, you are supposed to have "all the data". In reality, you don't and what you have are problematic. Just one example: lots of people turn off their TV but not their set top boxes, and it's hard to know if you are still watching the same channel or have gone to bed. Also, the really tricky business of adjusting such data: you need different models at the user level from the aggregate level but those models then are inconsistent with each other.
• R. Mazumder offered a reformulation of Factor Analysis as an optimization problem.
• Pouliot tries to predict which restaurants in San Francisco may have health problems using text analysis from Yelp reviews. Another application of LDA although in my mind, not successful. First attempted as an unsupervised problem, then as a supervised problem using past inspections as the training data. The problem with using past inspections is that the model is now conditioned by past rules. Not sure if Yelp reviews is the way to go but this is an interesting open problem.
• Twitter was supposed to give a talk but cancelled.
• Facebook is as far as I know absent, perhaps because of the weird controversy about scientific research. If so, it's tragic.

 

UPDATE (8/8/2014):

Here are other sessions I attended, including one on statistical graphics I somehow missed in the first roundup:

• Grace Wahba gave the Fisher Lecture. Her research from decades ago on kernels and splines has made a great impact on the field. It's one of those ideas that has lifted off once we reached a certain level of computational power. The support vector machine has been found to be a specific implementation of her more general ideas. She presented a paper on using smoothing-spline ANOVA to look at mortality within family members.
• There was an important session on reproducibility of statistical research, and I heard talks by Phillip Stark and Y. Benjamini. Stark lays out the issues nicely, and calls a spade a spade: a lot of today's research, especially but not limited to computational statistics, amount to "hearsay" or "advertising" for unpublished content. This is because referees or readers do not have the ability to replicate such findings. He introduces the Berkeley Common Environment (BCE), which is a set of tools that his and other research teams use to keep track of codes, environments, etc. to make research replicable. Sounds interesting to me.
• Benjamini's talk is primarily about what he calls "selective inference". One way to describe this is that one should not use the same data to both parametrize one's model and to generate estimates from that model. If ignored, the estimates will very likely to too optimistic (i.e. over-fitted to the observed data). Perhaps this is more understood in the context of modeling but it also is widely practised in statistical testing, where we first select significant effects based on p-values, and then issue interval estimates on those selected effects. As his provocative subtitle asserts, "it's not the p-value's fault". He then ran out of time but was getting into another issue of needing to estimate interaction effects due to clusters, e.g. medical centers in a clinical trial.
• Another session brings together the business, legal and ethical dimensions of data privacy. I wish this session was structured more like a conversation than a series of presentations with a few questions at the end. The lawyer, Paul Ohm, contends that anonymization is impossible, and the notion of PII (personally identifiable information) is useless. I don't think I heard what the solution is. Dick Deveaux, the statistician, reviews recent controversies, such as the Google Maps, Facebook research, and OK Cupid experiments. Clearly our community is ill-prepared to make constructive contributions right now to this debate.
• Leland WIlkinson discussed "scagnostics" which apparently originated with John Tukey. Scagnostics are summary statistics of scatter plots. Once you reduce those plots to numbers, you can then classify and cluster these plots, and Wilkinson shows how this is done and then how to explore a corpus of scatter plots in this way. Useful if you have that type of data.
• Max Ghenis presented Glassbox, an R package created by Google HR analytics group, for visualizing the response surface of models. I have used this sort of tool for years. Definitely a worthy project.
• Andrea Kaplan demonstrated Gravicom, a tool to manually generate clusters of nodes from network data. This is still preliminary work. Would like to see it again once she incorporates algorithmic approaches to complement the manual approach.
• Github and Shiny win the Supporting Actors awards for that section of Statistical Graphics that I attended.
• Carson Sievert presents a way to explore topics and words in LDA topic models. I like what I saw and think it would be very useful to people building these models. I hope he will expand this project from purely exploratory to allowing users to take actions based on what they see.
• M. Majumder's talk was entitled Human Factors Influencing Visual Statistical Influence. It's really about "line-up plots" which is a brilliant concept by A. Buja. You generate multiple sets of simulation data from the null distribution and insert the observed data into this set at a random position. Then the analyst tries to pick out the observed data from the noise. If you can't do it, then we don't have a signal. This research used Mechnical Turks to test what are covariates that affect the performance of this task. I was a bit disturbed by the extreme variance of the "percent accurate" metric - the boxes in many of the boxplots went from almost 0 percent to almost 100 percent. I believe Majumder's conclusion is that the variance is almost entirely explained by the difficulty level of the tasks at hand. I would like to see this work repeated with a choice of tasks that do not range so much in difficulty level.

Bad SQL habits

For those in Boston/Cambridge, I will be speaking at the Chief Data Scientist meetup on Wednesday night. See you there.

***

Warning: this post may be hard to understand if you don't know SQL.

SQL is one of the most fundamental tools in data science. It is used to manipulate data. Its simplicity is a big reason for its popularity. There are lots of things it can’t do but the few tasks it supports cover the majority of required tasks.

Over the years, I have noticed some bad habits of SQL coders. These habits tend to prevent the coders from “seeing” the imperfections in their data. Here are a few:

“Select top N” to “spot check” the data

Most analysts realize that they need to check the integrity of a data set. The easiest “check” is to eyeball the top N rows of data. In most cases, the data set is ordered in some way, not necessarily known to the analyst, so the top N rows do not form a representative sample of all rows.

Even if those top rows were as if random, it’s not clear what checks the analyst is performing mentally as he or she scrolls up and down a printed list of say 100 rows and 20 columns of data. Is the analyst looking for missing data? For extreme values? For discontinuity in the distribution? For out-of-range values? None of these tasks are simple enough to do in one's head.

Further, if there is a problem with the data, it usually comes from extreme or missing values, which are rare. Or if the data contain text, it may be that a few rows contain bad characters that will trip up SQL during a routine task.

Here’s the bottom line: if the data problem affects a huge chunk of the data, you will find it using a spot check, or any kind of checking. But most data problems affect a small corner of the data.  A spot check will almost always miss these, leading to a false negative problem. The real trouble is when the analyst issues a bill of health after a spot check.

Assume that a data table has no duplicate rows

When merging data sets, it’s very easy to generate duplicate rows, if one or both of the data sets contain duplicate rows of the same match key. For instance, the analyst is merging the customer sales history data with the customer contact information data. The match key is the customer id number.

It is normal to assume that the contact database has only one row for each customer (who would design this table in any other way?), and nine times out of ten, this assumption will be correct.

The one time is when your CEO needs an updated sales number right now for a board meeting. Oops, the sales number is double the expected value. The culprit is that duplicate customer ids made their way into the contact history table so that when it is merged with the transactions history, each sales record is replicated one or more times.

It may sound like a waste of time but before merging any data, check each table for duplicates.

Use open-ended time windows

In business analytics, we are always counting events over time, be they sales transactions, clickthroughs, responses to offers, etc. I have a pet peeve: code that does not have explicit accounting windows, meaning a starting time and an ending time.

Such code is not auditable. Every time you run the code, it will generate a different count (unless your business has infrequent events). If you wrote the code yesterday, and I ran it today, the counts would be different. How will I know if the difference is entirely due to the longer accounting window or if there are problems with the underlying data? 

The usual excuse for this coding practice is that the business wants the “most updated” number, up to the very last microsecond. Let me assure you: a day-old number that has been verified is preferred to a second-old number that cannot be audited.

The other excuse is the code is hard to maintain since you have to hard-code the ending time. But that is letting the tool limit your analytical ability. There are plenty of tools for which this is not a limitation.

***

When I talk about numbersense, I am also talking about the habits of the analysts. Bad habits doom many analyses before takeoff.

Have you encountered these issues? Do you have your own list of bad habits? Let me know!