Big Data, Plainly Spoken (aka Numbers Rule Your World)

Interpreting margins of error in tennis calls

A commentator of the French Open recently complained that the human line judge made a mistake: "Hawkeye's error is 3 cm and the ball was out by 4 cm. So the line judge is wrong to call it in."

***

The commentator got this all wrong. The divergence of opinion should reduce one's confidence in Hawkeye's estimate. Let me explain why.

Hawkeye's goal when it comes to judging line calls can be stylistically described as determining the center of the landing spot of the tennis ball. It's helpful to first look at what happens without a margin of error. From this estimated location, we draw a ball given the diameter of a tennis ball, and figure out if the ball overlaps with the line on the court. If it doesn't overlap, then the computer decides that the ball is outside the line.

The idea of a margin of error of 3 cm is visualized as drawing a circle of radius 3 cm around the estimated location. Now, from any point inside this circle, we draw tennis balls as before; and if none of these balls overlap with the line on the court, then the computer decides that the ball is outside the line. By involving the margin of error, we explicitly embrace the uncertainty of estimation. For sure, fewer out calls will be issued relative to the case when we just use a single location estimate.

The fundamental problem in statistics is that Hawkeye gets one chance (one sample) to get this right. If Hawkeye used a deterministic process, then given the same inputs (videos, etc.), it would always generate the same estimated ball location. In real-world systems, whether it's because of noise in the system, or some stochastic element in Hawkeye's process, the same inputs lead to different estimates. The margin of error describes how much these estimates vary.

The reported margin of error holds that Hawkeye's estimate is unlikely to be off by more than 3 cm. In other words, that expanded circle of radius 3 cm is expected to capture the "true" center of the ball's landing location.

The word is "unlikely" rather than "impossible". All margins of error comes with a confidence level; usually it is 95% confidence. This means there is a 5 percent chance that Hawkeye may be off by more than 3 cm from the true location.

Hawkeye's estimate is not error-free as the commentators assumed, even after allowing for the margin of error.

I'm curious about the margin of error associated with humans inspecting ball marks on the clay - I suspect it's small. (The error of judging the balls in flight, by contrast, is certainly much higher.)

***
At tournaments that use Hawkeye, the players are forbidden from challenging calls. Let's subvert the process and exchange the roles of humans and machines.

Assume Hawkeye makes the first call, in or out. If the player disagrees with the call, he or she raises a challenge, and the umpire (and/or line judge) goes to inspect the mark on the clay. Now, the umpire's word is final, no complaints allowed.

As an example, Hawkeye determines that the ball is 2.5 cm outside the line, which is less than 3 cm, thus the machine rules it "in". A player protests. The umpire decides that the mark on the ground is wholly outside the line, and changes the call to out. How will the commentators react?

If their reaction is not colored by a preference for machines over humans, they will say that the machine has made a mistake - and to accord with their current behavior (in reverse), they should then recommend that the tournament removes line-calling machines because they are not accurate.

This is an instance in which reversing the players makes clear one's biases.

***

If we take a Bayesian view of this, we should combine the evidence. In the first step, we have one estimate. Now, if the second estimate conforms with the first, then the evidence becomes stronger. But if the second estimate contradicts the first, then the evidence weakens. This is why I said at the start that the divergence of opinion causes me to lower my confidence in Hawkeye's estimate.

Even more, I believe that the human estimate derived from the mark on the ground is more accurate anyway so I'd give that even more weight.

 

P.S. Outside of clay courts, the situation is more complicated as there are no ball marks to look at. I'm not against the technology. I'm against the illusion of perfection, and I'm against black-box technology that stifles dissent. Both these issues can be addressed by how technology is applied.

 

 

 

Electronic line calling vs ground truth in tennis

The American commentators at the French Open have been making a fuss over the tournament's decision to favor human line judges over electronic line calling. Their whining centers on two arguments:

• Certain cases in which the electronic call differed from the umpire's decision, for which they claim one of the player was robbed of a point
• The process of letting players dispute close calls wastes too much time

***

These commentators treat computers as infallible. The attitude is apparent when they say things like "The umpire ruled the ball in even though Hawkeye [a brand name of such technology] says it's out by 4 cm."

Here, we have two sources of opinion about where the ball landed. Hawkeye issues its opinion, based on collating video images, and some modeling. The umpire's opinion comes primarily from the evidence on the ground (on a clay court, the ball leaves a mark on the ground), aided by the prior call of the line judge. In the view of these commentators, when the two opinions differ, the computer wins.

In effect, the computer animation is taken as ground truth against which the umpire's calls are evaluated. The computer animation is never wrong. So, it's not the evidence that favors the computer, but a presumption of its superiority.

And yet, on a clay court, the truth is literally on the ground. The ball leaves a mark on striking the surface. Before the computer age, a player can dispute a close call, the umpire will inspect the mark, and confirm or overturn the line judge's call. I don't see any problems with this arrangement. In fact, it uses the best evidence available - the ground truth.

Instead of the actual evidence, the commentators prefer a "modeled" truth - the abstract reconstruction of the ball's landing, and when they chastise the umpire for making the wrong call, they effectively invalidate the ground truth.

***

What about the time saving argument? The current practice of the umpire inspecting the ground truth takes little time, almost always less than a minute. It may take a little longer if the player makes a scene, even though no umpire is going to take the player's words over their own eyes.

It's not that computer technology doesn't take time. It would have taken a similar amount of time to watch the animated video of the ball hitting the ground. The reason why electronic calls save time is because electronic line judges are designated as dictators - players are disallowed from contesting any call.

The time saving comes not from checking the call but from banning challenges! No player can make a scene since no challenges are allowed.

But, they could have achieved the same result by making line judges dictators. Or, if they want to allow the ground truth to be inspected, then make the umpires dictators. The umpire can be called to inspect the marks, but his/her decision is final, and any player making a scene would get a demerit. That would move the game along, as these commentators seem to want.

***

See my previous post about the illusion of perfection in automated line calling.

 

 

Metrics for evaluating passkeys

Long-time reader Antonio sent me an article that has the following intriguing line:

Microsoft revealed today that 68 percent of all password sign ins fail. In other words, only 32 percent of all Microsoft users manage to sign in when they are prompted to do so when they use passwords.

This statistics is shocking but this line for me raises more questions than it provides answers.

The claim that two out of three attempts to use a password failed seems unrealistically high. (The author does not reveal the basis of this number.)

In my experience, most people don't log out after they log in. They think it's more convenient. Developers also think it's more convenient, and thus many application interfaces actively hide the log-out buttons. These users will only attempt log-ins if they unexpectedly got kicked off. They are unlikely to remember any password.

Even though I try to log off frequently, I still can't retain all passwords. I tend to forget passwords for those accounts I don't use often. For those who don't log off, it's a miracle if they could remember their passwords. For this segment of users, the password success rate is very low.

For users like me, who log in and log out constantly, the success rate should be much higher. But I suspect we are a small minority.

So the 32% password success rate may be explained by selection bias - most of the attempts are made by people who don't care to remember passwords.

***

What other group might contribute log-in attempts?

Of course, malicious actors. I have no idea whether the data analysts filtered those out, or, indeed, whether they are able to differentiate between a legitimate user mistyping a password, or a bad actor trying to guess the password. Let's assume they can't tell those apart. Then, it's not true that 32 percent of the time, legit users failed the password test. The real percentage depends on how much malicious activity there is.  

***

The article pitches an alternative to passwords, known as passkeys. It makes a further claim:

Users who sign in with passkeys manage to do so successfully 98 percent of the time.

Ironically, this statistic makes me nervous. The purpose of user authentication is to stop imposters from entering one's account. Analogously, we put locks on our front doors to prevent strangers from entering our homes.

If the door lock salesperson boasts that their lock lets in 98% of entry attempts, do you feel convenienced or insecure?

I feel insecure, because I believe all services face a healthy amount of malicious activities so I expect a lower success rate.

***

I was chatting with Perplexity.ai about passkeys, and it offered another statistic - that attacks via stolen passwords have plunged as more users switch to passkeys. No kidding. Since passkeys don't use passwords, bad actors aren't going to need stolen passwords, so password stealing has crashed.

To properly measure this, the analysts must figure out how malicious actors would adjust their tactics. They can certainly try to steal passkeys, or session tokens. If these developers switched back to passwords, passkey theft would also collapse!

***

The point is that it's important to define the right metrics. Passwords and passkeys are security measures, and the highlighted metric concerns convenience, which may be negatively correlated with security.

 

 

 

Causal skimming

The Wall Street Journal reports that all the rich people in New York City are moving downtown. The headline claims "New York's Wealthiest All Want to Live Downtown Now". (link) I'm not surprised by this statement, I want to believe it, but as a data scientist, I'd like to know how they prove it.

I suppose they can run a survey and interview a bunch of rich people. I imagine that might be a tall order, as rich people might not be very responsive. Perhaps they found their way to a wonderful dataset that shows the movement of rich people. That's not out of the realm of the possible. We know that someone was able to trace private planes around the world (link). I imagine the dataset would be quite small, as uptown/downtown NYC are not the only desirable addresses for rich people. Maybe they have good data on housing inventory. Let's find out.

There are two classes of relevant evidence. First, I'm looking for evidence of the trend. Second, for any causal explanation they provide, I'd like to see evidence of those linkages.

***

Let's skim through this article, and I'll mark down each piece of evidence I encounter.

A financier sold a West Village apartment for $60 million recently, which the couple had purchased for $29 million nearly a decade ago. This is a "record" for downtown Manhattan.

This is presented as evidence for the trend. What kind of evidence is this?

It's an anecdote. It's an extreme value of an anecdote. This anecdote uses house prices as a proxy for people movement - it assumes that if house price increases, then a wealthy person has moved from uptown to downtown. In fact, the $60 million record doesn't mean anything unless we also know that the $29 million was not extreme.

Let's keep going.

The buyer worked at Jane Street Capital, a quant trading firm, which has recently expanded its headquarters located nearby.

Already, the journalist has made a causal claim - that the downtown migration trend is due to rich folks wanting to live close to their work. The evidence for that is another anecdote. It'd have been more convincing had Jane Street moved its HQ from uptown but that's not what happened.

Downtown real estate prices are starting to "rival the city's most expensive uptown enclaves".

This assertion is not supported by any statistic. We don't know how wide the gap was, and how much it changed. Should there be data, we'd have an idea which districts are considered uptown and which downtown.

"As finance and tech firms have migrated downtown—alongside new retail, parks, and cultural institutions—a new wave of luxury condo development along the West Side Highway is luring wealthy buyers to neighborhoods like the West Village, Tribeca and Chelsea."

I left the above as a direct quote. This one citation contains a few more causal assertions, with zero data. Notably, the anecdote of Jane Street Capital should not even count because Jane Street did not "migrate" downtown. In subsequent paragraphs, they mention the names of some of these new developments but they never named any new retail, parks and cultural institutions.

Next:

A real-estate agent says that wealthy people are "buying up the West Side Highway".

Expert testimony. It's possible that these experts know what they are talking about, I'll give them that. Still no data or statistic in sight.

"There were more $30 million-plus home sales below 34th Street in the past five years than in the previous decade, according to data from Corcoran Sunshine Marketing Group. Since 2023, the area has seen more than $1 billion worth of home sales above $20 million."

Now arrives the first bit of data. The first sentence compares downtown sales across two periods of time. This can't prove a preference for downtown since we don't know the trend in uptown neighborhoods for those time periods - it's totally reasonable to believe that uptown prices may also have inflated a lot during the last 15 years. The second sentence gives a snapshot value of home sales in one time period, which does little to support the claimed downtown migration trend.

Since we are talking about economic value stretching back 15 years, those figures ought to have been inflation-adjusted. Another problem is the varying definition of downtown - two paragraphs back, a different sentence mentions "sales and listings below 14th Street".

Three more sales transactions are cited, two with just the current sales price, and one with both current and prior sales prices.

This raises the count of anecdotes to four. Since each transaction has a buyer as well as a seller, we'd also need to know where the seller moved to. If buying a home suggests preference for the neighborhood, does selling a home indicate preference for some other district?

A set of current listings with sky-high listing prices are mentioned.

Assuming these homes eventually find buyers at the listing prices, they would add to the anecdotal evidence.

"Downtown has long drawn wealthy buyers".

I'd label this anti-evidence. This statement undermines the claim of a recent trend.

The next few paragraphs run down a list of financial and tech firms that have set up shop in Hudson Yards. 

The alleged link between employment and residence is questionable because we are talking about $50 million homes, not $5 million homes. Sure, Google is "one of the weathliest corporations" in the world but how many Google employees can afford a $50 million home?

The supply of homes for the ultrawealthy is limited downtown. New developments are making larger homes with luxe amenities comparable to those uptown.

This also works as counter-evidence. Are prices downtown increasing because of shifting patterns of demand, or is it because these homes are larger and have better amenities? This section further muddies the causal picture. 

***

Let's summarize. The entire article contains one paragraph that cites two statistics, there are a bunch of anecdotes, and there are lots of unsupported story-telling.

Regarding the existence of a downtown migration trend, all the included evidence is anecdotal. We only learned that some rich people paid a lot of money to buy downtown homes. We don't know where they moved from. We don't know where the sellers moved to. We don't know the trend in real-estate prices uptown, with the unspoken assumption that they did not rise, or did not move as drastically. In the end, we're asked to trust the experts.

Regarding why rich people are preferring downtown, the main causal explanation is tech and financial firms relocating downtown. It would have been useful to know, for example, what proportion of these workers own $50 million homes. Several other causes are also mentioned, such as larger homes, homes with better amenities, new parks and cultural institutions. All evidence is anecdotal, and we again must just trust the experts.