Big Data, Plainly Spoken (aka Numbers Rule Your World)

Andrew screaming about professional fouls

In a recent blog post, Andrew announced that "he will keep screaming about this sort of thing forever" (link).

And I will be right there with him.

The "sort of thing" that upset him this time is when authors of a peer-reviewed journal article chose a misleading title. The accused title is "....: A long-term experimental study of ..." Anyone who reads more than the abstract learns that by "long term", the authors meant three days. As Andrew quipped, since they aren't studying amoeba, three days do not long term make.

Frustratingly, this paper is not a special outlier in the world of academic publishing. Such misleading titles or abstracts are common enough, and no one seems to be willing to call them out.

This is a nice example of what I've been calling "professional fouls" (see this post), a term borrowed from sports. A professional foul is an act recognized by outsiders as a foul but not considered a foul by insiders since "everyone does it".

In football (soccer), the so-called diving is a professional foul. Most players perform this act and most players aren't agitating for strict refereeing to root out this practice. In pro basketball, I find the last few minutes of a game unwatchable because it consists of a series of professional fouls leading to free throws. The entire profession pretends that this is "smart strategy". Smart strategy is if the community agrees to forbid this ridiculous tactic.

I complained about similar abuse of journal titles before. For example, what Pfizer titled "six-month safety and efficacy" suggests that participants in the clinical trial has been observed for at least six months to determine the vaccine's efficacy after six months. Based on my analysis, 80 percent of the participants in that study were observed for five months or fewer - and would never arrive at the 6-month point because the FDA had allowed placebo participants to get vaccinated, and almost all decided to take the shots. Just because a small number of participants were observed for six months, Pfizer called the study a "six-month" study.

 

 

There is no such thing as objective data

One of the persistent delusions about people running numbers is that they produce "objective" data. This delusion is frequently delivered in a self-righteous manner, directed at people who advocate adjusting the raw data. No no no, they say, you can't change the data because you are injecting subjectivity into the sacred raw data!

They claim that any adjustment is like injecting additives to pure, wholesome, raw foods.

In fact, all data are subjective. Even if the data quality is perfect, all data are still subjective because of measurement. This is one of the running themes in my second book  Numbersense (link). The first chapter addresses how school administrators can manipulate formulas to gain an edge in school rankings. The formulas are fixed but an endless number of tactics  delivers data that result in different values for the same underlying condition.

The latest demonstration of this malfeasance is trending news due to an investigation by a Columbia math professor (link). US News has now suspended Columbia from this year's rankings. What's shocking is their one bad apple framing. As documented in Chapter 1 of Numbersense (link), fraudulent data collection pervades all rankings at all levels of education: in any jurisdiction that matters enough to merit investigation, fraud has been revealed.

One of my favorite examples of fraudulent data collection from that chapter also happens to be an example of a "professional foul", i.e. a foul that is obvious to any outsider but deliberately oblivious to all insiders who partake and benefit from the foul.

This amusing example is the pushing of the "Common App", which lowers the barrier of application for any college applicant. Whereas in the past, each college has its own idiosyncratic application, post Common App, it takes about the same time to apply to one school as 15 schools. 

What the emergence of the Common App does is to "raise all boats". All selectivity rates (proportion of applicants admitted) will go up at all schools that participate in the Common App because the denominator (number of applicants) is jacked by the Common App. If schools are increasing their class sizes at all, the rate of increase in enrollment does not catch up to the rate of growth in applicants due to the Common App.

So - is selectivity rate of colleges an objective data point? In my view, it is a subjective metric, just like every other metric, easily manipulated.

***

This post is inspired by a statement I read about Wework - the controversial startup that rents office space without long-term contracts. (Disclosure: I'm a fan of Wework. I think it markets a product that delivers clear value to users without doing harm, unlike many startups that either delivers zero value or hides the harm done behind "convenience" and other positives. Wework's problem has always been figuring out a business model.)

The CEO claims that Wework's occupancy rate in this last quarter matched that of prepandemic levels, defined as late 2019.

Occupancy rate sounds like one of those pure, raw metrics that has an official definition, thus an analyst can report an objective value for it, beyond scrutiny.

Far from it. Let me point out a couple of ways to dress up this number. I'm not accusing anyone of doing anything. The following is merely a demonstration of possibility - as I disclosed above, I hope Wework finds a business model.

Like selectivity rate, occupancy rate is a fraction: the proportion of available office space that has a tenant. Like selectivity rate, let's focus on the denominator, the amount of available offices.

The occupancy rate may be 72% in both Q4 of 2019 and Q1 of 2022, while at the same time the number of tenants is lower in 2022 than 2019. It all depends on the number of offices available. Since the pandemic, Wework has closed locations - incidentally, any business should want to get rid of its lowest performers so this statement isn't about gaming numbers; I'm pointing out how the metric attaining the same value does not contain the same meaning.

Nevertheless, managers can choose to game the metric. If you get rid of the bottom 20% instead of the bottom 10%, you'd have pushed the occupancy rate even higher.

Now, let's take a tour of one of these buildings. You'll notice that some of the offices are being used for storage. There are chairs, desks, etc. stacked up in them. If I am the accountant, I'd remove these offices from "available" space.

There are also offices that have been converted to common areas for users who are just there for the day. An accountant has at least two options: a) remove these offices from available space, or b) count them as available space, and count them as "occupied". To legitimize option b), the accountant can also stipulate that such an office is counted as "occupied" so long as there is at least one user working in that space for x% of the days of the month.

***

If you came here because you're read Numbersense (link), those last two paragraphs might remind you of Chapter 6, in which I discuss the official unemployment rate. That chapter also portrays data's subjective nature. Unemployment levels depend on how the government defines employed and unemployed. In the book, you learn that someone may have only worked for one hour in a month, and the US government counts him/her as "employed". In addition, the government removes people from the employment count, classifying them as "not in labor force", just like the accountant who subtracts offices used as storage space as "not available".

tldr; If you don't know how something is measured, you don't know anything about the data.

 

 

 

 

Data flows behind "secure" debit cards

The tech industry constantly promises "security". Is it possible for anything to be secure?

Here's something to ponder.

One of my debit cards recently got scammed. I am not sure what the scammer was after because the fraudulent transactions came in two sets, each set consisting of the identical transaction repeated 8 to 10 times on the same day for the same amount from the same merchant name. In other words, it would not take a genius to recognize these transactions as fraudulent.

What is interesting are the following facts:

• the account was less than a year old
• the debit card has never left my wallet except for when I used it at ATMs
• the only ATMs I ever used were the ones belonging to the bank that issued the debit card
• I have never presented this card to any merchant
• I have never used this card at any merchant (self checkout, etc.)

Remember, just a few years ago, we were promised that the new chip cards are so secure they will end the fraud problem. And so on for every new tech going back decades.

in order for these charges to appear on my card, presumably the scammer has stolen my card number, my address, the various codes on the card, etc. The bank's system must have approved those transactions without flagging them as fraudulent.

How did the scammer know those numbers? Why didn't the bank's fraud detection software reject those transactions? Where are the holes in the system?

 

Self-checkout and fast passes

In Chapter 1 of Numbers Rule Your World (link), I featured Disney's Fast Pass program, which is a virtual queueing setup that has delighted lots of amusement park goers.

The value of Fast Pass is commonly perceived to be reducing waiting time. It actually does nothing of this sort. What it does is to let people "multi-task" - do something else instead of standing in line. It's a masterpiece of managing perception.

This point is not obvious. Think about any optimization method applied to the queueing problem. A typical goal is stated as minimizing average waiting time. Waiting time is a loaded term - it is technically the difference between one's arrival time and service time (when one gets to the front of the line). If you take any person in the virtual queue, the "waiting time" is the same whether the person is physically present. (They did tell us that we keep our position in the line!) The average time between arrival and service has not changed, but guests are much happier. It's a win-win.

For a full analysis of how Fast Pass works, read my book (link),

***

A similar scenario is playing out in our supermarkets that have adopted self-checkouts.

Wait, am I saying self-checkouts do not actually reduce checkout times?

Let's compare a shopper doing your own groceries versus an experienced checkout clerk. Given the same basket of goods, who does it faster? Who knows where the buttons are? Who can look up codes on fruits and vegetables faster? Who knows how to apply coupons? Who has more practice? It's a slam dunk. 

Nevertheless, the lines for human clerks are frequently much longer. Are our eyes deceiving us?

At the Whole Foods near me, there are about 20 checkout counters with service clerks and maybe 10 self checkout stations. Typically when I'm there, fewer than 5 humans are working. Thus, if the experienced clerks work at twice my rate at bagging groceries, the line for human service is still going to be roughly the same length as the self-checkout line.

The value of self-checkout may not be coming from more efficiency; it comes from more capacity. The average self-service time may be longer than the average clerk service time, but the self-checkout user will still have waited less time because the supermarket management decides to devote more capacity to self-checkout.

As with the case of Disney, managing user perception matters.

***

One last thing.

Self-checkout also benefits from a self-selection effect. Shoppers who are buying just one item are much more likely to use self-checkout. In addition, those with baskets full of fruits and vegetables and don't want to deal with looking up codes are likely to choose to use human clerks. 

Thus, the mix of jobs experienced in the self-checkout line relative to the human service line is not the same, and the human clerks are likely to get the bulk of the jobs requiring longer service times.

***

If we set up a fair comparison, I'm betting that the store clerk finishes the job faster!

 

P.S. [7/27/2022] Minor edits to improve readability