Big Data, Plainly Spoken (aka Numbers Rule Your World)

Self-checkout and fast passes

In Chapter 1 of Numbers Rule Your World (link), I featured Disney's Fast Pass program, which is a virtual queueing setup that has delighted lots of amusement park goers.

The value of Fast Pass is commonly perceived to be reducing waiting time. It actually does nothing of this sort. What it does is to let people "multi-task" - do something else instead of standing in line. It's a masterpiece of managing perception.

This point is not obvious. Think about any optimization method applied to the queueing problem. A typical goal is stated as minimizing average waiting time. Waiting time is a loaded term - it is technically the difference between one's arrival time and service time (when one gets to the front of the line). If you take any person in the virtual queue, the "waiting time" is the same whether the person is physically present. (They did tell us that we keep our position in the line!) The average time between arrival and service has not changed, but guests are much happier. It's a win-win.

For a full analysis of how Fast Pass works, read my book (link),

***

A similar scenario is playing out in our supermarkets that have adopted self-checkouts.

Wait, am I saying self-checkouts do not actually reduce checkout times?

Let's compare a shopper doing your own groceries versus an experienced checkout clerk. Given the same basket of goods, who does it faster? Who knows where the buttons are? Who can look up codes on fruits and vegetables faster? Who knows how to apply coupons? Who has more practice? It's a slam dunk. 

Nevertheless, the lines for human clerks are frequently much longer. Are our eyes deceiving us?

At the Whole Foods near me, there are about 20 checkout counters with service clerks and maybe 10 self checkout stations. Typically when I'm there, fewer than 5 humans are working. Thus, if the experienced clerks work at twice my rate at bagging groceries, the line for human service is still going to be roughly the same length as the self-checkout line.

The value of self-checkout may not be coming from more efficiency; it comes from more capacity. The average self-service time may be longer than the average clerk service time, but the self-checkout user will still have waited less time because the supermarket management decides to devote more capacity to self-checkout.

As with the case of Disney, managing user perception matters.

***

One last thing.

Self-checkout also benefits from a self-selection effect. Shoppers who are buying just one item are much more likely to use self-checkout. In addition, those with baskets full of fruits and vegetables and don't want to deal with looking up codes are likely to choose to use human clerks. 

Thus, the mix of jobs experienced in the self-checkout line relative to the human service line is not the same, and the human clerks are likely to get the bulk of the jobs requiring longer service times.

***

If we set up a fair comparison, I'm betting that the store clerk finishes the job faster!

 

P.S. [7/27/2022] Minor edits to improve readability

Predicting invariables

Some years ago, I lamented that data scientists are not spending enough time on predicting "invariables", while we spend a lot of time predicting "variables" (HBR link).

While writing a future blog post earlier today, I encountered a perfect example. Apple Keynote.

One of the most annoying features in Keynote is that a text box opens up by default with a "centered" alignment, rather than anchoring the text at the top left corner.

No matter how many times I immediately changed the alignment to top and left upon inserting a new text box, the software does not predict that this is what I want, and continues to use the centered default. In many years of usage, I must have sent this same signal thousands of times.

The beauty of "invariables" is that they are easy to detect, and the accuracy rate is super high.

What is AB testing? 2

In a previous post, I mused about the practice of AB testing in web design. Today's post is another example that I have recently come across.

On one of the apps I use, they pop up "notifications" on the bottom left of the screen. These notifications are unsolicited messages - kind of like the paper pin animation that used to show up in Windows.

Here is what is behind this pop-up window:

Notice that similar to the pop-up from Twitter (see the previous post), there is nothing on the pop-up that allows the user to remove it. I am forced to click on it if I want to sign out of this account. When I click on it, it goes to another pop-up that has an X icon.

***

Imagine you're the AB tester. Version A of the test is this pop-up. The metric of the test is how many users click on the notification message. In the online world, clicks is the proxy for user engagement - click gold as they say.

Does the test result represent what the users want? Or what the designer wants?

What is AB testing: a 2020s view

I don't mean what is the concept of AB testing. If you've been reading this blog, you already know what it is (for those trained in design of experiments, it's the very first lecture.). Here are some prior posts about AB testing (1, 2, 3, 4). I even made a video about AB testing from a business operations perspective here.

All big websites have people whose job is to run AB tests. These tests are supposed to help figure out the optimal page designs.

The devil is in the details. Every AB test has one or more objectives it's trying to optimize. Most AB testers believe that the overarching objective is to make users "happy".

It takes very little time to debunk that myth.

I'll drop an example in this post.

***
In recent times, Twitter has been running an AB test. I know because I have been on both arms of this test. They are testing a pop-up that shows up after a small amount of scrolling. (I am not logged on.)

Here is Version A.

Here is Version B.

That's my reconstruction of Version B as I think they have locked me into a specific test group, or they have already discarded Version B from the test at the time of writing.

Notice that Version A completely blocks access to the feed unless the user logs into or sets up an account while Version B has an X in the corner that the user can use to remove the pop-up and continue to read the feed without logging in or signing up.

***

What are some obvious metrics the AB tester would use to evaluate this test?

What is the goal of the test? What is the tester's hypothesis prior to running the test?

If your goal is to make users happy, is this the test you'd design?

***

Bonus question: AB testing, or randomized experiments in general, has been heralded as the gold standard for causal inference. It's the best way to learn what is the "cause" of observed effects. In all likelihood, version A outperforms version B on the objective of generating more sign-ups or log-ins. What is the cause of this effect?

Is it the presence/absence of the "X" in the corner? Is it the decision to block/allow users to work around the pop-up? Is it blocking/allowing users to access the site contents? 

Let's say the test shows no effect. What did we learn about causality (or not)?

[Metrics: number of new sign-ups, number of log-ins, time on site, bounce rate are the most obvious.]