Big Data, Plainly Spoken (aka Numbers Rule Your World)

Title IX, causal claims, and plausibility in statistics

Andrew Gelman discusses a paper and blog post by Ian Ayres on the Freakonomics blog. Their main result is summarized as:

We find that a ten percentage-point increase in state-level female sports participation generates a five to six percentage-point rise in the rate of female secularism, a five percentage-point increase in the proportion of women who are mothers, and a six percentage-point rise in the proportion of mothers who, at the time that they are interviewed, are single mothers.

Andrew finds these claims implausible, so do I.

Ayres uses the econometrics methodology called instrumental variables regression to support these claims. Since the data is observational, and as Andrew pointed out, there wasn't even a period of time in which one could find exposed and unexposed populations (since the TItle IX regulation was federal), one must treat such regression results with a heavy dose of skepticism.

It is useful to understand that causal claims are possible here only if we accept all the assumptions of the instrumental variables method.

Besides, plausibility is assisted by the ability to outline the causal pathways. It should be obvious that more females competing in college sports does not directly cause more females to become secular. The data on sports competition and on secularism come from different sources and this presents a hairy problem. The analysis would have been more convincing if it found that among the women who participated in college sports, more became secular; what the analysis linked was higher participation rate and higher secularism among all women in the state.

What is it about sports participation that would cause people to become secular? (The visual evidence from professional American sports would lead me to hypothesize the opposite--that sports participation may be associated with higher religosity!) Is this specific to the female gender? Do we find male secularism increase as sports participation by men went up?

As Andrew pointed out, the magnitude of the estimated effect seems too large to believe. I'd prefer to see these effects reported at more realistic increments. A jump of 10% participation is very drastic. For example, according to the chart here (the one titled "a dramatic, 40-year rise"), the percent of women participating in high school sports has moved just 2 percent from 1995 to 2011.

***

Andrew is right that this is an instance of "story time". And we are not saying that statisticians should not tell stories. Story-telling is one of our responsibilities. What we want to see is a clear delineation of what is data-driven and what is theory (i.e., assumptions). The plausibility of a claim depends on the strength of the data, plus whether we believe the parts of the theory that are assumed.

***
See how Andrew articulates these--and other--concerns on his post.

Dealing with adapted data

Reader Daniel T. is unhappy about this analysis of the intraday Internet usage by OS and device types. He doesn't like their choice of index, which I'll get to in a second post. (Link appears here when ready.)

There is something else wrong with this type of analysis.

Let's do a thought experiment. If you are a marketer interested in the diurnal variability in Internet usage, what are some of the factors you might investigate? My list would include whether the user is logging in from work or from home; whether the user is working or unemployed or on vacation; whether the user is male or female, young or old, a student, retired, etc. 

Does your list include operating system? (i.e. Apple, Microsoft, Unix,...) I bet not. 

But OS is exactly what the blogger analyzed, and thousands of marketers around the world do so on a daily basis. That's because they are using what data they can get their hands on. Web log data are adapted, that is to say, they were collected by engineers for the purpose of debugging, and now they are used by marketers to explain consumer behavior. It's not hard to see why such data cannot tell the full story.

This goes back to the O and the A in my OCCAM framework for Big Data (link). Web log data is the prototypical example of data collected by tracking devices indiscriminately without purposeful design, and then adapted to marketing applications.

***

One way to cope with using adapted data is to be clear about our model of the world. Assume OS really does affect Internet usage. How does OS affect Internet usage? Are you assuming that the features of an OS directly condition a user's behavior? Or are you assuming that the choice of OS is an indicator of the type of user? 

Another way to cope with adapted data is to find or collect the data you really want (e.g. demographics, occupation) rather than analyzing data you don't understand. Recall Sean Taylor's advice to collect your own data (link).

Numbersense Pros: Interview with Larry Cahoon Part 2

Yesterday, Larry Cahoon, a 29-year veteran at the Census Bureau, answered some questions. The rest of the interview is printed below.

***

KF: How can a data analyst improve his or her skills?

LC:

I have to say my best training has been many, many hours of just playing with statistics, playing with graphics, and reading the analysis others have done. The more data I see, the more analysis I do, the more graphics I look at and produce, the more I learn about how to look at data and how to see the pitfalls in the data analysis. This is getting down in the mud and dealing with real data with all of its warts. My wife tells people I’m a statistician through and through as every time she looks at my computer, there is a graphic of some type on the screen.

Finally, I have always been an avid reader of Science Fiction. No one can read Science Fiction without being forced to consider any problem from different perspectives and take into considerations differing assumptions. This in turn has helped me develop the ability to question the assumptions being made in any data analysis.

***

KF: What are your pet peeves with published data interpretations?

LC:

I seem to return again and again to the same issues with the analyses I see in the newspapers, online, and just about anywhere. The most basic problem is one of incomplete analysis.

We see so many papers and news reports where a data difference is observed and then based on no data whatsoever, the author goes off with an entire line of speculation without any data to justify that speculation. This line of thinking then frequently ends up with the claim that these two things are correlated, and therefore we have cause and effect.

The media then fans these reports by writing a story without asking basic questions, such as is the data itself any good or have they any evidence for the claims that are being made. The media acts as if the claims have been proven – especially in how they headline the story.

My second pet peeve is what I call an emphasis on a one dimensional world. This is usually reflected in simple statements like: A causes B. The world is much more complex than that. Those who investigate airline accidents have been telling us for some time that there is seldom just one cause for each accident. Rather there are a number of causes. We need to carry that knowledge over to our statistical analysis and reporting.

***

KF: Which source(s) do you turn to for reliable data analysis?

LC:

I can’t say that I have any favorite source for data analysis. If forced to name one, I would say that I tend to like the work of the Pew Research Center (link) Their surveys seem to be well designed, the questions they ask well thought out, and the analysis something I can trust.

I like the data that is available from the Federal Government. But the government agencies rightly avoid most detailed data analysis in an effort to remain nonpartisan.

***

KF: Thank you so much for your time. We're lucky that you continue to blog in your retirement.

You can view the earlier installments of the interview series here.

Numbersense Pros: Interview with Larry Cahoon Part 1

A short while ago, I introduced Larry Cahoon's blog, GoodStatsBadStats. He started the blog almost two years ago after retiring from the US Census Bureau (site offline due to government shutdown), where he spent 29 years working on the statistical design of most of the household surveys conducted by the Bureau. Cahoon received his PhD from Carnegie Mellon University.

Cahoon spent the final seven years of his career working on the 2010 decennial census. He was involved in almost all areas of census design and operations. His work included focusing on the many issues of both over coverage and under coverage in the census, the effectiveness of the publicity campaign, and the use of the American Community Survey as a replacement for the long form of the census.

Larry was very gracious and offered great insights and long answers. I have divided the interview into two parts. Part 2 will appear tomorrow.

***

KF: What are the key skills of statistical reasoning?

LC:

There are three things on the top of my list when I think about statistical reasoning skills. A solid knowledge of statistical methods and principles is a necessary starting point. I extend this to include the ability to think in terms of probability. This is a way of thinking or a way of viewing the world at the most basic level. 

Equally important is a solid foundation in logical reasoning. The third piece is recognizing and dealing with the assumptions that must be made in any statistical analysis. One needs to be able to look at the problem from a multitude of perspectives and ask if the assumptions being made are good ones. Can I make a logical argument why they are true and why a contrary set of assumptions is false? The ability to consider alternate assumptions and confounding factors that may not be obvious is very important to any statistical analysis.

KF: Was graduate school useful training for your career?

LC:

In my own background, my first year of graduate school just about drove me crazy as I did nothing but statistics. Most of it was extremely theoretical work. I have always said that the masters’ degree they gave me after that year was in some sense worthless as I got very little exposure to the practical side of the statistical profession.

But what I did learn from the full immersion was to think in terms of probability. I came to see statistics not just as theory but as a way of thinking. Thinking in probability terms became second nature. It helped that I was trained in a Bayesian environment long before it reached the levels of use that we now see. 

I saw statistical testing as a sometimes useful but limited tool. It became clear to me that much more important questions are what is the best estimate and what is the decision rule that would come out of the analysis. It was just as important to know how that data is going to be used as it is to know how the analysis is to be done.

KF: How did your work at the Census Bureau influence you?

LC:

To do good statistics, knowledge of the subject matter it is being applied to is critical. I also learned early on that issues of variance and bias in any estimate are actually more important than the estimate itself. If I don’t know things like the variability inherent in an estimate and the bias issues in that estimate, then I really don’t know very much.

A favorite saying among the statisticians at the Census Bureau where I worked is that the biases are almost always greater than the sampling error. So my first goal is always to understand the data source, the data quality and what it actually measures.

But, I also still have to make decisions based on the data I have. The real question then becomes given the estimate on hand, what I know about the variance of that estimate, and the biases in that estimate, what decision am I going to make.

KF: I want to reiterate this point to my readers who are not statisticians. In data analysis, we are using available data (the sample) to make a general statement, say use the response of subjects enrolled in a clinical trial to describe the effect of a new drug on all potential patients. Imagine you are trying to hit the bull's eye. Shotgun #1 produces a wide scatter around the target while Shotgun #2 produces a narrow scatter but the average shot lands wide of the target. We say that #1 has high variance and low bias while #2 has high bias but low variance. Both types of errors contribute to the shot being off the bull's eye.

Because the Census Bureau typically uses large samples, the sampling error (variance) is very manageable. What is hard to control is bias, meaning the entire sample is not representative of the population under study. This is Shotgun #2.

LC continues:

As I matured in my career, I learned that diversity matters. The greater the exposure and the more diverse the exposure to real world statistics, the better the practitioner would become. So while I worked for many years in the area of survey design, when I went to the annual statistical meetings, I always made the effort to maximize my exposure to other areas of statistics. I would always return home with a few textbooks on areas of statistics outside of what I was actually working on at the job.

A curious nature with a desire to continue learning is essential. Today a good training route is to read as many statistical blogs as you can find the time for. Especially important is to seek out and read the work of those who disagree with me. This forces me to think much more critically of the work I am doing.

***
Part 2 will follow tomorrow.

Read all the previous interviews here. 

 

Know your data 15: How to get robbed by Google

You walk into Walmart, and there is a big notice to customers, informing you that by walking into the store, you consent to being continuously photographed, and being used as a spokesperson for any brands that are carried in the store. You will not be compensated for endorsing these brands because Walmart opens its doors to you without an entrance fee. You should feel blessed that Walmart won’t charge you an entrance fee—in fact, Walmart promises never to charge that. It's none of your business that the brands are paying Walmart for your endorsements.

You will not be notified which brands you are endorsing, when you are endorsing them, how you are endorsing them, or to whom your endorsement will be peddled. If you find the preceding terms distasteful, you can exercise your all-American right to shop elsewhere. Before you turn away, give a little thought to those everyday low prices.

Next week, the big notice will be removed because by shopping here once, you have issued a blanket consent. Every time you walk into the store, you are renewing your consent.

You won't see this notice at Walmart (just yet), but last week, Google announced that this is how they will treat you from now on (link). To Google’s credit, they are simply copying what Facebook has been doing for some time. Now that Google’s doing it, it must not be evil.

***

Next, pity the unsuspecting marketer who spends real money on Google's "shared endorsements" (or "sponsored stories" to use Facebook's terminology.) According to the New York Times reporter:

Such product endorsements, especially from friends and acquaintances, are a powerful lure to brands, replicating word-of-mouth marketing on a broad scale. Social advertising — which includes a wide range of ads, including endorsements — is a $9.5 billion business, according to eMarketer, accounting for 8 percent of digital ad spending.

Most of that money is being poured down the drain. These marketers are getting robbed by not thinking about the counterfactual! (I explain this in detail in the marketing chapters of Numbersense. Learn more about the book here.)

"Word of mouth" operates in the shadows but marketers know it's happening. They even attempt to quantify it through something called the "Net Promoter Score," (link) computed based on answers to the question of whether one would recommend the brand to a friend.

But word of mouth has always been free marketing. This was happening before Google or Facebook even existed. Why should marketers pay social media companies for something that is free? Are they just getting robbed?

***

Here is an example of the kind of ads that would show up featuring your friend or acquaintance.

If Roger Willis is someone I know and trust, such as my Dad or a close friend, I don't need this ad to tell me the obvious. If he is an acquaintance, not a friend, I'm unlikely to buy something based on an ad like this. If he is a friend of a friend, I'm even less likely to care.

Saying the effect is small is not saying no one will ever be influenced. In my view, marketers should not take the bait.

 

 

 

 

These journal editors probably forgot to drink their morning coffee

Here we go again. Another useless study published in a peer-reviewed journal (Mayo Clinic Proceedings) with a relatively high impact factor and promoted as "Breaking News from the Editor" to the press who then attached a sensational headline and reported it as "science".

This is what caught my eye: "Drinking more than 28 cups of coffee a week may be harmful for people younger than 55, according to a study." I saw this on USA Today (link)... but many other outlets also carried the story, including NPR, CBS, AJC, Guardian, etc.

Just the headline should sound alarm bells. Why 28 cups a week? Is it ok to have 27.5 cups a week? Why 55 years old? On your 55th birthday, should you stop worrying about the number of cups?

Of course, even that speculation betrays a causation creep. Nothing in this study, nor any study of this type, can prove a causal link between the accused food and harm.

Now let's check out the summary by the authors:

Conclusion: In this large cohort, a positive association between coffee consumption and all-cause mortality was observed in men and in men and women younger than 55 years. On the basis of these findings, it seems appropriate to suggest that younger people avoid heavy coffee consumption (ie, averaging >4 cups per day). However, this finding should be assessed in future studies of other populations.

Note that construction: "a positive association between .. and .. was observed ... on the basis of these findings, it seems appropriate to suggest that". All those weasel words. What they really mean is "we have data showing a positive association and we make an assumption that correlation = causation, therefore you should ..."

Do you think you'll learn anything in the journal paper about the biological or chemical mechanism by which coffee causes death? Take a guess.

***

USA Today commits the other sin of health reporting: failure to explain the level of harm, and the context to interpreting such. USA Today tells its readers:

Men younger than 55 who drank more than 28 cups of coffee a week (four cups a day) were 56% more likely to have died from any cause.

56% compared to what? Turns out it's compared to men younger than 55 who do not drink coffee at all. There is a wide gap between drinking over 28 cups a week and drinking zero.

What's also missing is the error bar. According to the paper, the 95% interval is 30% to 87%. Not kidding, it's 25% in each direction.

Absent is the context for understanding what 56% means. How many additional deaths for every 10,000 heavy coffee drinkers? Amusingly, you can't figure this out even after reading the entire paper. The authors got away with presenting data in aggregate (33,900 males, 2198 male deaths, etc.) without showing age group breakouts (Where were the editors??) Stymied, I glance at their other result, the one for "all" men.

In men, those who drank more than 28 cups of coffee weekly had a 21% higher risk of dying compared with their non-coffee-consuming peers.

By the way, the error bar on this result is 4% to 40%. Now, I can't interpret this result either. The baseline death rate in the study was 6.48%. Nowhere in the paper does it break out the number of deaths by the level of coffee drinking. There is no way to know how many of those 2,198 male deaths were men who did not drink coffee at all.

While the important data on the outcomes being analyzed are not published, the authors of the paper regale us with numbers such as the N=11 people who were excluded from the population because they had a history of stroke.

So I ask again: where were the editors? how did they miss this?

***

Readers should also check out Andrew Gelman's recent rant about the excessive reliance on statistical significance, and the neglect of data quality in journal editing (link). It's not that hard to draw up a list of data required for publication.

***

I may write another post about the other issues with the analysis. But here is the most important one in case I don't get to it. This is the chart that supports their finding about men under the age of 55 who drink more than 28 cups of coffee a week. They want us to look at the blue bar on the far right of the chart.

Look at the other blue bars. Need I say more?

 

 

An ignored issue in Big Data analysis

The Sunday Review section of New York Times on August 11 contains two pieces I'd like to discuss.

The first piece, by Ian Urbina, an investigative reporter, is called "I Flirt and Tweet. Follow Me at #Socialbot". He tells a fascinating story about how programmers create "bots" that impersonate people surfing the Web. For instance, researchers at a Brazilian university created a bot named "Carina Santos" who was rated by certain social media ranking companies as being more influential than Oprah Winfrey.

Urbina reports that:

Last year, the number of Twitter accounts topped 500 million. Some researchers estimate that only 35 percent of the average Twitter user's followers are real people. In fact, more than half of Internet traffic already comes from nonhuman sources like bots or other types of algorithms.

Somehow Urbina missed the implication of these bots. A huge part of Big Data is internet traffic, including tweets. Recall the Library of Congress made a big deal about saving every tweet ever produced (link). What does this say about the use of tweets to analyze "human" behavior and psychology? Would you ever believe a study that does not adjust for bot traffic?

***

Ironically, readers have the answer in the same section of the paper. There is another article, titled "Dr. Google Will See You Now", penned by Seth Stephens-Davidowitz, a recent Harvard PhD in economics. This article takes Google search data and draws conclusions about "depression" in America.

I could barely get past these words:

Not every health-related search using "depression" is a sign that someone is depressed, and not everyone who is depressed queries Google. But thanks to the incredibly large sample size, meaningful patterns emerge.

This has the same symptom as "causation creep," in which the researcher admits he/she doesn't have evidence of causation but immediately ignores that inconvenient fact, and acts as if he/she has proved causation. Here, Stephens-Davidowitz goes on and assumes that every health-related search using "depression" is a sign that someone is depressed, and that everyone who is depressed queries Google.

It's a sad state of academia if a PhD at Harvard thinks that an "incredibly large sample size" is a cure for badly conceived analysis.

 

A letter to reporters on the economy

The New York Times took over 1,000 words to tell us that Big Data won't change the economy (or is it the economists' profession?) ("Is Big Data an economic Big Dud?") I'm less pessimistic; I think the collection of vast troves of observational data is ultimately beneficial but only if (a) we set a high bar for analytics, such as requiring multiple corroborating data sources pointing to the same conclusion; (b) we initiate efforts to collect specific data through well-designed surveys or experiments with the goal of providing a degree of quality control; and (c) we advance the methodologies of analyzing observational data.

The bar is not high, given the woeful state of research on the economy, exposed by the Great Recession. Even worse are the "consensus forecasts" issued about business performance. However, don't for a second fall into the trap of thinking that more data automatically leads to better research. More data does not automatically lift that low bar; quite the contrary, without meeting the preconditions listed above, more data could push that bar even lower than it is now. That is one of the central concerns of Numbersense.

***

Now, reporters take note. Please don't repeat the following fallacies that pollute most articles written about Big Data.

The NYT starts by telling us how fast the volume of data is growing: "The astounding rate of growth would make any parent proud." Give me a break. The simplest way to generate tons of data is to introduce bugs and mistakes into code. I will give you two examples which count as "data" generation.

First, when you import data to a data base, a log file is produced containing entries like "Import started.", "Read 50,000 entries." "Warning: Found date in non-standard format", etc. plus overhead entries on the time of execution, the name of the software, etc. Such log files are regarded as data, specifically "unstructured" data.

You can use say an ODBC connection to import an external file but in many such arrangements, the database only processes a small number of rows per instruction, say 500 rows. So, if your data comprises 20 million rows, the import requires 40,000 instructions for one upload. Even without errors or warnings, you've generated 1.2 to 2 million rows in the log file. If you have a column of "non-standard dates", you've generated at least 20 million rows of warnings. Your log file now has more rows than your data. You've truly contributed to the growth of data.

The second example is really an analogy. i am reminded of college days when I accidentally printed machine code in computer lab. The human-readable code might be 50 pages long but one typo and the printer started churning out thousands of pages of gibberish. We did the sensible thing in those days--we threw out the gibberish.

***

The second fallacy is encapsulated in this statement: "the economy is, at best, in the doldrums and has stayed there during the latest surge in Web traffic." Behind this kind of blather is the presumption of a simple, direct link between data and "the economy".

Maturity in data-analytic thinking starts with complex, multivariate, stochastic and dynamic systems. There will never be F=ma or E = mc^2 for the "economy". What does the "economy" mean anyway? Is it GDP growth? Is it employment level? Is it inflation? Is it happiness? Is it social wellbeing?

Such statements show a shallow appreciation for the factors leading to the economic collapse. Chief among these is the housing bubble, which was promoted by bad public policy, and in many cases, blatant fraud. Data cannot solve such problems; we need political will, ethics, and enforcement of the law. In fact, as I argue in Numbersense, data worsens these problems as more data allows more theorists to cite "evidence".

***

The third fallacy is causation creep, something I discuss here frequently. This involves first acknowledging correlation is not causation, and then slipping in the causal interpretation anyway. For example, this sentence contradicts itself: "The overall economic trends are complex, but an argument could be made that the slowdown began around 2005 — just when Big Data began to make its appearance." If you believe the antecedent, then you have to conclude that the second clause is too simplistic; conversely, if you believe the second clause, then the antecedent is falsified.

***

The point of Chapters 6 and 7 (on economic data) in Numbersense is to raise awareness of the complexities of developing metrics for something as amorphous as the "economy". One then realizes the challenges of interpreting such metrics. To get the full argument, get a copy of the book today.

 

 

 

Numbersense Pros: Avinash Kaushik

Today, I have the honor of interviewing Avinash Kaushik, author of the bible known as Web Analytics 2.0, and a digital marketing evangelist at Google. He also has a must-read blog called Occam's Razor. Occam's Razor is a principle championed by statisticians that can be summarized as "as simple as possible but not too simple". It is a principle and therefore it also draws controversy from some quarters. Kaushik's blog is a place where you can pick up Numbersense. He writes about Web data, which is a huge part of Big Data.

***

KF: How did you pick up your impressive statistical reasoning skills?

AK:

I'm a Mechanical Engineer with a MBA in Finance and Marketing. Somewhere along the way, data, analysis, statistical reasoning became an integral part of my life. :)  Beyond what our traditional education blesses us with... using new analytical software has been an opportunity to refresh old concepts and learn new ones. I'm also partial to books. I'm attaching a photo of one that's on my desk right now!

I've also come to believe that application of cool, sexy, statistical techniques yields poor results not because someone was unfamiliar with them, rather because they don't quite understand the underlying business drivers, data collection mechanisms, impact of the business culture, the people involved and other "soft factors."  I've attempted to invest an enormous amount of time in all of those areas to improve the impact of my analytical efforts.

***

KF: What are your pet peeves with published data interpretations?

AK:  Ignorance of the simplest rules. 

I get mad, still, when I see the classic correlation = causation. My expectation was that after that problem has been made fun of in Dilbert comics, we've reached a level where no one would fall for this trap. And yet, every day!

Not stating the assumptions clearly or, worse, not understanding their impact is another cardinal sin that I've come to dislike a lot. Assumptions are everything! Dig, dig, dig, find them and understand their impact on the data you are looking at.

Perhaps my favourite is the utter unwillingness to actually think about the data in front of them. For example, a prestigious digital company shared the other day that "67% of US buyers check the social profile of the company before they buy the product." Now that is complete crap. You just have to think about all the people making purchases, or yourself, and how often do they/you actually check a company's social presence before making the purchase? Yet this fact was provided with a pretty infographic, and reposted on many "popular tech/social blogs." So few people bring their nose close to the data to check if it smells. It is distressing.

***

KF: Which sources do you turn to for reliable data analysis?

AK:

(This is 100% genuine, and not an attempt to play to the gallery...) For years and years the posts on this blog, Junk Charts, and the comments have been a valuable source of learning for me.

I'm a big fan of the Guardian's Data Store (link). Such a great collection of data visualizations, types of data, thought provoking analysis.

I also use information aesthetics (link), as a gateway to discovering new sources.

Attribution in online marketing: a Big Data problem

Avinash Kaushik's masterful post on the "mutli-channel attribution problem" in Web analytics is required reading for anyone seeking an understanding of what Big Data is really about. Kaushik's posts are marathons; I provide here a little background, plus some highlights from his post to save you some time. But you absolutely should read the whole thing!

I will start from the elementary. Big Data is big because the Internet was set up in such a way that there are extensive log files (Weblogs) documenting details of how information flows from point A to point B. When you click on a link in your browser, it sends a bunch of requests to a remote server, asking for the elements (images, text, buttons, etc.) that form a Web page. The remote server keeps a log of all such requests from all browsers from everywhere around the world. One of these requests goes to the Facebook server (this is why Facebook blankets huge numbers of pages with their Like buttons.) The Facebook server then has a record of what browsers asked for what pages, in other words, Facebook knows you navigated to that page. Something called a cookie is involved here; the cookie identifies your browser.

Let's say you buy flowers at FTD.com. The retailer uses Google Analytics to track user movement on its website (This is a hypothetical example. I don't know what tools they use.) This means that some computer code is inserted into the web pages at FTD.com, and each time your browser requests a page, a call is made to the Google Analytics server with your cookie information.

The point is that lots of data reside in these Weblogs. They tell when you visited which Web page. Eventually, you may end up buying flowers - that is known as an "action". If you purchased flowers, the FTD.com marketing team wants to understand how you arrived at that decision. They assume that the path by which you ended up on the purchase web page provides hints as to why you bought flowers that day. Note I used the words "assume" and then "hints".

Here is an example of a path analysis:

Take a look at the first row, which represents all the recorded interactions that took place prior to a user making the purchase. Read from right to left and you are reading time backwards from more recent interactions. The "action" of purchasing is off the chart on the right side. The five interactions immediately preceding the purchase are classified as "direct", which basically means we don't know how the user arrived at the site. The next, earlier interaction was traced to "organic search" meaning someone came to the website from (most likely) Google, and (most likely) searching directly for our company name. Looking further back, we also see a social network generating leads, and six more unknown interactions.

Marketers stare at this diagram and impose a causal explanation on it. Every interaction is assumed to be a contributor to the final purchase, albeit with different levels of importance. (In Chapter 3 of Numbersense, I caution that such a path is not causal. To see this, think counterfactually. If for example you remove Social Network spending, so that the purple arrows disappear, would the purchase still have occurred? If yes, the Social Network interaction had zero causal effect.)

The concept of "attribution" is to distribute credit among one or more of these prior interactions. Kaushik walks through a bunch of models that can be used to divide credit.

Kaushik then points out that analyzing the above figure (especially when it has thousands of other rows) is a waste of time: "There are too many paths, and you can't actually control the path that a potential customer can take."

***

While Kaushik dives into the mechanics, here are some high-level takeaways from his post:

• The attribution problem has no intrinsic answer. There is no single correct answer. Everything is subjective.
• Many decisions affect the attribution outcomes. e.g. which sources are credited and which are not, which positions in the path are privileged and which are not, the time window for eligiblility, what counts as an "action" and what doesn't. Different decisions lead to different attribution schemes.
• Having more data creates more complexity but does not reduce subjectivity. On the contrary, more data creates more levers resulting in more assumptions.

See Kaushik's full post here.

 

PS. Let me make a small clarification. I'm not against subjectivity. I'm saying every attribution scheme is subjective. One can spend time arguing over the details of one scheme versus another but the new scheme is also subjective. See the section on "Perversion of Measurement" in Chapter 2 of Numbersense for further discussion of this important point.