« Talking shop about probability | Main | Notes on evaluating predictions, or Background to my airfare predictor article »


Feed You can follow this conversation by subscribing to the comment feed for this post.

Sherman Dorn

Why is it a test failure when you discover something? I would think that "hey, we don't have to worry about stupid color choices!" is a pretty good finding.

Tom West

@sherman dorn: you're right, but management may not see it that way.

The random number thing is generally less important than you might think. If the server looks at the time and produces option A when the seconds is odd, and B when its even, then it's highly unlikely you'll get a systematic bias.

Also, I always say that A/B testign lets you pick the perfect shade of blue - but what if the best option was actually a red?


Tom: My point about random number generators is that there is no industry standard. I have come across a lot of fishy ones.

As for your method, I have always wanted to ask someone where the server gets the time? Is there any chance that that query fails?

As for your tongue-in-cheek complaint, why not test red against blue?

Sjors Peerdeman

I'm wondering why you didn't discuss existing A/B-testing tools, such as Visual Website Optimizer and Optimizely? I'm interested in the pros and cons of these tools from a data scientist's (?) standpoint.

Tom West

@Kaiser: For sure, if your server time fails, then the test doesn't work. But that's the same as saying if your random number generator fails, then the test doesn't work.

Yes, you coudl test red against blue... but my point was that A/B testing is often used to zoom in on a specific solution within narrow constraints - but the best answer may be outside those constraints.


Sjors: I am a happy user of Optimizely. We use both their SaaS platform and have a homegrown solution similar to PlanOut. The Facebook solution is written for developers while the Optimizely style solution is created for business people. A SaaS solution has limitations on what tests can be run.

The issues I outlined above are not solved by having tools. In fact, I encountered some of them in tests which used tools. What you need are brains - what I call numbersense. The Facebook solution provides structure and ingredients that the analyst finds helpful in diagnosing problems. I'm not saying that deploying PlanOut will magically prevent those problems.

Tom: On testing large versus small changes, I think the more interesting debate is whether one should test a complete redesign in which dozens of changes are introduced all at once against the existing design.

beli followers instagram

bad very easily happen if the measurement data collection is not done directly. especially if the facebook users are doing other activities simultaneously

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)

Marketing and advertising analytics expert. Author and Speaker. Currently at Columbia. See my full bio.

Spring 2015 Courses (New York)

Jan 26: Business Analytics & Data Visualization (14 weeks) Info

Feb 23: Statistics for Management (10 weeks) Info

Mar 28: Careers in Business Analytics & Data Science (one-day seminar) Register

Apr 7: The Art of Data Visualization Workshop (6 weeks) Register

Next Events

Sep: 28 Data Visualization New York Meetup, New York, NY

Oct: 5 Andrew Gelman’s Statistical Communications class, Columbia University

Oct: 13 AQR ProSeminar, NYU Sociology

Oct: 22 Leading Business Change Through Analytics, Columbia Business School

Oct: 30 Ray Vella’s Designing Infographics class, NYU

Past Events

See here

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee


  • only in Big Data