There have been few updates as I was working on things for other people. One of these things showed up today. Here is an excerpt from the beginning of my new article on HBR:
For over 10 years and at three companies, I set up and ran A/B testing programs, in which we test a new offer with half a sample against a control group which doesn’t get a new offer. Executives quickly pick up on the headline benefit of testing: that A/B tests provide reliable answers to “why” questions. This comes as no surprise, as such testing has long been held up as the “gold standard” for learning cause-and-effect in scientific research, clinical studies and direct marketing. However, many executives eventually reach a mid-life crisis, developing doubts about the direction of the A/B testing program.
From my experience, here are three of the most common questions that arise from those doubts, and how managers should think about them.
Coming Tuesday, I'm talking about A/B Testing at the Optimizely Experience event in New York. It's at the Intrepid Museum, a great setting. My talk will address questions that come up frequently in practice when you start doing a lot of online experiments. Testing is one big area of business analytics, and an area in which practical experience has to complement textbook learning. Hope to see you there.
I have neglected to acknowledge the good folks who have posted reviews of my books in the last several months. Thank you, readers!
Tom Peters, the business guru and author of the business classic In Search of Excellence, tweeted about Numbersense: "On my 13-hour Boston-Dubai flight, I re-read cover-2-cover Kaiser Fung's superb-useful-fun book Number Sense".
It straddles statistics, persuasion, and psychology, and through clear writing, addresses what could be an incredibly boring topic for some readers (the application of statistics and how they affect you) and turns it into a really easy read.
Here is a sentence from Christopher Rollyson's detailed review of Numbers Rule Your World:
Fung is an talented storyteller who revels in thinking about information in creative ways.
Numbersense is not another book of statistics or data analysis that tell about how to manage, represent and analyze data but it shows, using many examples, how to develop the sense to recognize ‘misleading behaviour’ of big data.
If you like the book, please consider writing a review. If you have private comments, you can write me here.
Thanks to the ~200 or so people who showed up at last week's Data Scientist Meetup in Cambridge, Mass., hosted by John Baker. I gave a brief introduction to the concept of "numbersense", and was part of a panel of "chief data scientists" talking about how to run data teams. Thanks to those who asked questions.
This month, I am back in New York, and will be giving two talks.
First up is the Data Visualization New York Meetup organized by Paul Trowbridge. The link to register is here but it looks like all slots have been taken within days. You should get on the wait list as some registrants will eventually drop out. This event is on Aug 20 (Wed).
On Aug 26 (Tues), I am giving the "thought leader" presentation for the Optimizely Experience. I will be talking about statistical testing for online marketing aka A/B testing. The title of the talk is "Five Questions About Testing You Wanted to Ask But Didn't" unless I come up with something better. You can register here.
This will be a brand-new presentation, and I look forward to sharing my ten-plus years of running online experiments. See you there!
Also, please let the organizers at SXSW know you want to hear me and other data viz experts talk about visualizing data in Austin. Jon Schwabish has put together a fabulous panel with people from different parts of the spectrum, and it promises to be an engaging conversation.
A number of folks have reacted to various blogs and talks I have recently given. I'm glad that my writing has inspired others, and I recommend reading these wonderful responses.
Diane Ravitch, the eminent scholar of New York education and author of several great books, found my 2011 post about Bill Gates's view of education. Here is her reaction:
How refreshing to know that statisticians like Kaiser Fung are keeping an eye on what is called “reform,” but turns out to be the pet ideas or hobbies or whims of very wealthy people who know little or nothing about education.
Thanks Diane for the kind words, and the retweet!
Alberto Cairo, whose book The Functional Art I recommended, writes about a topic that he and I have discussed at length, and I agree with much of what he writes here about the "new" data journalism. He complains that much of the pieces in FiveThirtyEight, Vox, NYT Upshot, etc. are fluff. I am particularly interested in the business model questions.
Alberto is a great example of someone who tunes his "numbersense" by reading and learning from experts. He isn't someone who will program in R or write down regression equations. As I have said all along, it is possible to appreciate good (and detect bad) data analysis without having to learn all of the math.
Ellis Booker, at Data Informed, attended my keynote in Chicago, and wrote a summary of my talk. Very well-done. Thank you! Here's a chance for you to "hear" my talk without being there!
Finally, the ASA (American Statistical Association) has published a white paper on how statistics and Big Data science can work together. Here is the introduction.
From Andrew Gelman's blog, I learned about a paper that makes the claim that daylight savings time could kill you. (Andrew links to this abstract, which is from a poster presentation at a meeting of the American College of Cardiology, and later published as a supplement in the ACC Journal; one of his readers found the published paper.) There is also a press release sponsored by the Journal with the fearmongering headline "Daylight saving impacts the timing of heart attacks". In case you don't get the message, there is a subhead, pushing the idea that "Setting clocks ahead 1 hour may accelerate cardiac events in some, a large study shows".
Given that heart attacks are all about "timing", this headline will be interpreted as daylight saving causes heart attacks. This is probably the intention of the publicist who wrote that headline. But the headline distorts the researcher's conclusion, which was stated as (in the poster):
In the week following the seasonal time change, daylight savings time impacts the timing of presentations for acute myocardial infarction but does not influence the overall incidence of this disease.
First of all, the researchers clearly states that daylight savings time (DST) does not influence overall incidence of AMI. That should have been end of story. Secondly, there is a world of difference between DST "accelerating cardiac events" and observing an increased number of "presentations for acute myocardial infarction". The researchers did not analyze data on heart attacks--the data they had were admissions for AMI undergoing PCI (percutaneous coronary intervention).
Thirdly, anyone looking at the accompanying chart (from the poster) should be asking a lot of questions about the conclusion:
The most plausible conclusion is what the researchers said in the poster: there is no effect, and nothing to see here. But you can't get a paper published or get the press's attention with that conclusion! So you put a magnifying glass on that one blip you see on the Monday after Spring Forward (top chart). That is highly problematic because the two weeks of data is not enough to understand whether that level of a jump from Sunday to Monday is abnormal. In fact, if you look at the Sunday to Monday jump in the bottom chart on the left side, you'll see that it is similar in scale... the only reason why this increase is not commented on is that it does not occur after a DST time shift!
Later in this post, we'll examine how research methodology turns a blip into a supposedly important finding that deserves publicity.
Before getting into the methodological issues, one needs to ask the most basic question. Did the researchers check the quality of the data or just take the data as is? In my experience, the data surrounding DST time shifts are often inaccurate. Let me talk about an example from Web server type systems which I'm familiar with. Some systems fail to switch time zones, an analyst or a customer notices the error some hours later, or a day later, and the administrator corrects the oversight at this time. The administrator does not modify the erroneous past data sitting in logs in some remote servers because it's out of process, because it requires an expensive surgical procedure to isolate the wrong entries, and because, understandably, he/she is only concerned about whether the application understands the current time to serve current and future users. The IT department does not know that in the future, a data analyst will use the log data to investigate the effect of DST change.
The fact that this is a "large study" (large for this type of study) makes this an even bigger problem. The data come from a registry that "encompasses all non-federal hospitals in the state of Michigan". This means that people from many different organizations are handling the data. The more people are involved, the more likely there will be mismatches. For example, any of the hospitals could have database servers that fail to switch time zones. These are some of the challenges of using Adapted and Merged data, which I previously outlined in the OCCAM framework for understanding Big Data. Also, if you are in Chicago, come hear my talk on Wednesday at Predictive Analytics World on OCCAM and Numbersense.
Another basic question is the practical implication of such a result (assuming that we can believe it). Looking back at the top chart shown above, you see that the excess on Monday was balanced by deficits on Tuesday (most of it) and Thursday. Thus, if we believed the result, then DST has the effect of accelerating the heart attack by one or three days. In other words, you will still have a heart attack but just a few days earlier.
Wait a minute, you might challenge me. Didn't I make the assumption that the excess people who suffered heart attacks on that Monday would have gotten heart attacks during the same week otherwise? That would be a great question! That's numbersense! The reason why it is a reasonable assumption is that if you didn't make it, you are making a different assumption: you now have to explain how DST simultaneously induces some heart attacks (to explain the excess on Monday) and prevents other heart attacks (to explain no overall effect).
Moreover, this finding is purely based on a correlation in observational data. No one is arguing that the act of having a DST policy induces heart attacks. However, because we don't have a causal factor to work with, doctors have no action to take as a result of this analysis.
As this article is getting long, I will leave comments on the methodology to a next post.
For those who weren't able to attend my recent talks, a few have surfaced online.
JMP put up the video of the webcast from last Friday with Alberto Cairo, a data visualization expert and author of The Functional Art. You can access it from here. This event is part of their Analytically Speaking series with recent guests such as David Hand and Michael Schrage. I also appear on this recording of the panel celebrating the International Year of Statistics.
Agilone, an emerging vendor of self-service marketing analytics software, hosted me at their recent user conference, as well as a webcast. Here is a clip, in which I explain the structure of analytics teams that I have assembled.
Last year, I gave a fun, lightning talk at the Leaders in Software & Art conference. The recording is here.
Joe Dager did several long interviews with me that is well worth listening to. Here's Part 1, and then there's Part 2.