Big Data, Plainly Spoken (aka Numbers Rule Your World)

Primer on regression adjustments 3

This is the third post in a series on using regression to correct for biases in observational data.

Let's review what we've learned so far. In the first post, I gave an example that shows regression will correct bias in a biased dataset. We looked at a sample of height data, in which women are over-represented relative to the underlying population. Without adjustment, the sample average height under-estimates the population average since women on average are shorter than men. After the regression adjustment, the gender-adjusted sample average height is almost identical to the population average.

In the second post, I expose a hidden assumption in the so-called regression-adjusted estimates, that everything is evenly distributed. In a gender adjustment (with only males and females in the population), the assumption is 50% males, 50% females. In an age-group adjustment (with 5 levels), the assumption is 20% in each level. If age is entered as a continuous variable, the age-adjusted estimate concerns persons of average age in the sample.

Needless to say, most real-world populations are not evenly balanced, nor do they contain only average persons. Therefore, if the goal is to estimate the population value, the regression adjustment may only correct the bias partially, and may even backfire, moving the sample average further from the population average. In other words, don't assume that dumping data into a regression software is a magic wand that cures all biases. And don't trust "experts" who wave this magic wand.

***

Chapter 3 of Numbers Rule Your World (link) is titled "the dilemma of being together." The chapter explores a fundamental problem in statistical analysis: the tension between capturing complex relationships in the data and finding generalizable features. The analyst resolves this question by deciding how much to aggregate the data. In regression models, this relates to how many averages do we include in the equation.

In the most naive model, the analyst summarizes the entire sample in one number, the sample average, and asserts that it is the best estimate of the population average. Thus:

[Naive] Population Average Height = Sample Average Height

Recall the statistics of our sample B of 900 people:

The naive model simply states:

[Naive] Population Average Height = 66.1

The average height of Population B is 67.1 inches, and the error of 1 inch is massive, equal to the width of a 95% confidence interval, as explained in post #1. This model makes systematic mistakes: the errors for males tend to be negative while the errors for females tend to be positive. A good model is one in which errors are as if random.

The underlying problem is that this naive model is too simple. On average, males are taller than females. So we err by summarizing everyone to one average.

To deal with this issue, we add complexity to the regression model. The so-called gender adjustment adds a gender term:

[Gender-stratified] Population Average Height = 66.6 + 2.6 if Male - 2.6 if Female

This is the regression output if I throw the data into software. It's the same equation I showed in post #2. People then take the constant term (66.6) and call that the "gender-adjusted" sample average height.

Note that the above equation is equvalent to saying:

[Gender-stratified 2] Population Average Height for Males = 69.2 while Population Average Height for Females = 64.0.

The reason for calling this "gender-stratified" should now be clear. In fact, this is a better representation of the model because the value of the constant term is not unique.

A mathematically equivalent formulation of the above model is:

[Gender-stratified 2b] Population Average Height = 69.2 - 5.2 if Female

This little wrinkle catches many software users by surprise. Depending on how the software developer codes the data, the regression outputs are different, even though they represent the same model. If you don't believe it, check that the new formula is also identical to:

[Gender-stratified 2] Population Average Height for Males = 69.2 while Population Average Height for Females = 64.0.

***

The standard way of thinking about gender adjustment is that it makes gender irrelevant. A gender-adjusted estimate removes the effect of gender, allowing us to focus on the outcome of interest.

A different - in my view, better - way of thinking about gender adjustment is that it acknowledges gender's effect on our outcome of interest. The regression adjustment involves building gender into the structure of the model.

The gender-stratified model finds that males and females have different heights: for males, we estimate, an average of 69.2 inches while for females, we estimate an average of 64.0 inches. By contrast, the naive model treats everyone as average, and we estimate the average height to be 66.1.

Gender adjustment does not prove that gender is unimportant. It's the opposite: building gender into the model means gender cannot be ignored. If gender does not matter, then we expect the gender-stratified model to reduce to the naive model. In other words, the coefficient of the gender term is zero (statistically speaking).

[Gender-stratified 3: Gender not important] Population Average Height for Males = 66.1 while Population Average Height for Females = 66.1.

And, formulated with a gender term:

[Gender-stratified 3: Gender not important] Population Average Height = 66.1 + 0 if Male - 0 if Female

So the average women in this special place is as tall as the average men.

What about the gender-adjusted sample average height of 66.6? This is the number that researchers in certain fields would have extracted from the stratified models -- recently, they have featured prominently in studies of vaccine effectiveness, including those by CDC scientists.

Mechanistically, these researchers have built gender-stratified regression models, then they ignore the gender coefficient, and report the constant term as gender-adjusted. As explained in post #2, this procedure is equivalent to averaging the gender-level height estimates, placing equal weights on male and female.

In statistics textbooks, we routinely warn readers that the coefficients of a regression model ought to be interpreted holistically. Should this rule apply to vaccine studies? If not, why not?

I'll pick this question up in the next part of this series.

***

If you missed the previous posts, you can still catch up on post #1 and post #2. 

Watching statistical gravity in action: elections and vaccines

A quick post today to let you know what's on my mind this week - I'm still chewing on these ideas, so if you have opinions, feel free to comment below. Will follow up on some of these in future posts.

Four Quick Takes on the Election Aftermath

1. In 2016, we mocked most election forecasters who said Hillary Clinton would win with 99% probability while agreeing that Nate Silver won the least bad award. Fast forward to 2020. Nate gave Biden a 89% chance of winning while the Economist said 96%. Who's got bragging rights here?
   
   
2. A former colleague pointed out to me that people mistakenly confuse a high chance of winning with a wide margin of victory. In a simulation model, a 96 percent chance of winning just means that Biden was predicted to win the most electoral votes in 96 percent of the simulated scenarios -- the 96 percent number does not contain information about the margin of victory in any scenario.
   
   
3. What Trump voters witnessed in Wisconsin, Michigan, Georgia and Pennsylvania was "statistical gravity."
   
   
4. In a Bayesian forecasting model, we get a posterior probability distribution as the output. This is typically explained as the proportion of scenarios in a simulation that end up with a given electoral-vote share. Tail events are rare and capture unusual scenarios - e.g. James Comey in the last days of 2016. Within this framework, how does one express the view that in the forthcoming election, one expects to see a tail event?

 

Four Quick Takes on the Coronavirus Vaccine

We finally got some news yesterday via Pfizer*, which sent out a press release claiming that its vaccine candidate may have "over 90 percent efficacy". They have not published a formal report, so there isn't much we can go on right now. Given the urgent need for a viable vaccine, this is one step in the right direction.

1. The claim is based on what happened seven days after the second dose so what the statement meant was that people who got the vaccine was 10 times less likely to be counted as a case within seven days of the second dose than those who got the placebo. This is an impressive early finding. Will "statistical gravity" kick in later?
   
   
2. People keep misinterpreting vaccine efficacy. The leader of the current White House coronavirus task force tweeted that the Pfizer vaccine "prevented infection in 90% of its volunteers". As I said before, this is not what it means. The case rate for vaccinated volunteers was about 0.1 percent while that for unvaccinated volunteers was 1 percent (99 percent of placebo takers did not get infected in the first 7 days). The former is 90% smaller than than latter.
   
   
3. I was scrambling to read the Pfizer protocol. I had assumed theirs and Moderna's would be similar since their vaccine candidates belong to the same family. I was surprised to learn that Pfizer was going to measure efficacy based on seven days as I already knew that Moderna doesn't count cases until 14 days after the second dose. So if we had run these two clinical trials side by side, Pfizer's analysis would have been finished before Moderna even started theirs. When both trials report results, we will have a much richer understanding of this type of vaccines.
   
   
4. People are calling for subgroup analysis. That will take time. This interim analysis is based on a total of about 90 infections. If you break that number down to subgroup level, there's not enough data yet to provide a useful anaswer. Be patient. Allow statistical gravity to work.

* Should give credit to BioNTech which is the company in Germany that develops this vaccine. Pfizer is running the vaccine trial.

If people would buy the real thing, there would be no counterfeits

In Chapter 3 of Numbersense (link), I told the story of the software industry complaining that pirates stole billions worth of software each year. This estimate of the value of software embeds the fantasy that people who use pirated software would have purchased legal software at list price should piracy be eradicated. It's a failure to think counterfactually. Many people who use pirated software cannot afford, or do not want, to pay for the legal version.

The tech industry invokes this logical flaw frequently. Earlier this week [a month ago by the time this post is published], news broke that the U.S. Customs and Borders Protection (CBP) intercepted "counterfeit" Apple Airpods but quickly, tech-savvy observers pointed out that those earbuds are made by OnePlus for Android phones, an Apple competitor. The photos included in the CBP press release even showed the seized products in OnePlus packaging.

CBP estimated that the seized products had a value of $389,000, by multiplying the number of products with the list price of Apple Airpods. Such an estimate assumes that if OnePlus earbuds were banned, consumers would have all purchased the much more expensive Apple product. The list price for OnePlus earbuds is less than half that of Airpods.

In this ArsTechnica article, the CBP claims $1.5 billion worth of counterfeit products are seized in a year. This number appears to suffer from the same counterfactual fallacy. What would consumers do if OnePlus earbuds were banned? A proportion of them will go back to wired headphones. Another group will migrate to other earbuds that are in a similar price range as OnePlus. These are not merely speculation - we do know that these customers elected to buy OnePlus instead of paying double for Apple.

If everyone who purchased the counterfeits would buy the real thing at list price, why didn't they buy the real thing in the first place?

 

Numbers indeed rule our world, 10 years out

Statistics is the subject of generalizing data. In statistics, we study how to use imperfect data to draw general conclusions. We often have to wait for the future to reveal the evidence of success.

Ten years ago, McGraw-Hill published my first book, a best-seller, Numbers Rule Your World (link). I’ve hoped to explain five of the most profound concepts from my field in a way that anyone can understand and apply to their daily lives. I attempted the balancing act of featuring specific stories while bringing out the general principles. In the last couple of months, the Covid-19 pandemic has generated immense attention to data and statistics, and provided me confirmation, ten years on, that the contents of Numbers Rule Your World (link) have general applicability. It affirms that statistics is a great subject!

If you’re approaching my book for the first time, or re-reading it, here’s how you can work your way through it, making connections at various points to the Covid-19 crisis.

***

Chapter 2: Bad Spinach / Bad Scores

This is your perfect starting point. Half of the chapter details a disease outbreak investigation. You learn about the information networks the CDC puts in place to monitor unusual patterns of disease around the country. You discover data analysts whose job is to set off the alarm. How do they decide if the first handful of cases foretell an epidemic?

You encounter “shoe leather,” the painstaking collection of data to pinpoint the origin of an outbreak. It’s much harder than you might think. The U.S. politicians are making much noise about the early days of Wuhan. An epidemiologist may learn that 10 out of 15 first patients all visited the Wuhan seafood market before they got sick. But the Wuhan market might be a hugely popular destination, like Central Park in New York City. Finding 10 out of 15 people went to Central Park over a weekend could be more common than you think. If all 10 touched a particular rare species of wildlife, that's another story.

Another controversy of the moment is the huge cost of shutting down the economy, against the benefit of saving lives. I invite readers to think about this cost-benefit tradeoff. Because the disease outbreak was traced to a spinach processing plant in California, the FDA ordered a recall of all spinach, which depressed sales of salad greens for six months or so. The cost was much lower than for this pandemic, but the bacteria is well-known and thus predictable, and the lives saved were most likely zero, and even harder to pin down than in the current crisis.

People are already complaining that social distancing and other measures have not improved our numbers, by saying deaths were “unexpectedly low”. You discover how hard it is to prove the effectiveness of a policy decision because you need the counterfactual – what would have happened if the FDA did not recall the spinach.

Chapter 4: Timid Testers / Magic Lassoes

Everything you should know about the statistics of diagnostic testing and predictive models in one place. The motivating examples are (a) the fight against performance enhancing drugs in sports via anti-doping tests and (b) the false promise of terrorism prediction algorithms.

I wrote several posts (1, 2, 3) about the flawed Stanford antibody study that reported 50 times more infections in Santa Clara county in California than reported cases. Despite this headline, the raw data is also consistent with a scenario of zero proportion having antibodies. The statistical reason is the low underlying prevalence of antibodies, which caps the number of true positives in any sample. The number of false positives will be large relative to true positives, because the false positive rate applies to those without antibodies, which is the bulk of the population.

An extreme case of this phenomenon is terrorist prediction using a dragnet surveillance dataset. The set of terrorists is a drop in the ocean of people under mass surveillance. For a predictive model to capture most of these terrorists (low false negative rate), it must flag a lot of people – even a tiny false positive rate when applied to a huge population results in a lot of false accusations.

You see the distinction between a “chemical” false negative (positive) and a “practical” false negative (positive). To evade steroids testing, dopers use all kinds of tactics, such as handing in someone else’s urine. The diagnostic test finds a true negative but in practical terms, it’s a false negative. For swab testing, one may test negative if the swab doesn’t reach a virus-rich area, or if the patient is tested at the “wrong” time.

All tests have errors. What are the consequences of errors? Also, how likely will the error be discovered? For anti-doping, do dopers inform the test lab they made a mistake?

Chapter 1: Fast Passes / Slow Merges

The first chapter gives important insights on managing capacity, which is a pivotal issue for hospitals during the coronavirus crisis. I cover two case studies: (a) Disney’s management of theme-park capacity, and (b) state’s management of highway congestion.

The strategy of building more capacity is unavailable during a crisis, and wasteful in general. In this extraordinary time, some localities were able to bring up temporary hospitals. Transportation experts discovered that policies of regulating congestion are more effective. Highway ramps slow down the influx of new vehicles onto the highway, and are turned on before congestion arrives. The key is to delay as much as possible the onset of congestion. Sounds familiar? It’s the analogue to “flattening the curve”. Social distancing is supposed to reduce contacts, slowing down community spread. This tactic delays the peak demand for hospital resources. The chapter explains the scientific support for why such tactics work.

Chapter 5: Jet Crashes / Jackpots

Statistical significance is the topic of this chapter, one of the most common and also most commonly mis-used concepts in statistics.

In the Stanford antibody study referenced above, statisticians argued that the observed level of positive results is consistent with zero prevalence. This is a way of saying the result is not statistically significant. What we mean is that the study’s result (50 positives out of over 3,000 tests) would be considered normal even if nobody in the population actually had antibodies. This conclusion does not assert that the true prevalence is zero. The topic of statistical significance is made inaccessible to non-mathematicians by textbook authors piling on equations, but it need not be so. I even made a video on explaining "not statisitical significant" (link).

Before you apply a test of statistical significance, you must make sure you’re comparing apples to apples. In the context of air safety, you shouldn’t compare U.S. and foreign airlines on all routes; you should only compare them on routes they share, in which case, you’ll find no statistical difference. One of the bad stats that is circulating during this pandemic is the purported harm of ventilators. This is usually stated as the vast majority of patients put on ventilators eventually die (presumably, relative to those patients not put on ventilators). But only the most serious patients are placed on ventilators. The proper analysis should focus on patients who qualify for ventilators, and compare those who were put on ventilators to those who got other treatments.

We have already seen a host of invalid comparisons. Comparing complete data from years ago with incomplete data this year. Comparing regions that are in different phases of the epidemiological timeline. Comparing countries that have taken different approaches to managing the crisis.

Chapter 3: Item Bank / Risk Pool

This chapter addresses a deeper issue in using statistical averages. An illustrative example is educational testing like the SAT. Critics pointed to the gap in the average scores across racial groups as evidence of bias. But the racial grouping masks a third factor, which is differential ability between students. Due to various socio-economic factors, white students have higher average ability than black students. So the real question is: Among those students with comparable abilities, does the test still show bias toward one racial group? (The answer is not straightforward.)

During the pandemic, the “denier” faction has argued that the mortality rate of Covid-19 is “on a par with” that of seasonal influenza. Are those two rates comparable? The simple answer is no. Comparing mortality rates masks a third factor, which is the level of immunity in the population. Even at the most optimistic, the current level of immunity to the novel coronavirus is less than 20 percent (This statement generously presumes that (a) having antibodies confer immunity without question and (b) the New York City test result is not a scam, and the test is 100% accurate). By contrast, because of vaccination and recurrence, a good chunk of people have immunity to influenza. The mortality rate of Covid-19 should be compared to that of influenza among a population that has no immunity to either, or a population with similar levels of immunity to each.

The other example in this chapter explains how the insurance industry is an outgrowth of the law of large numbers. In any given year, health or auto insurers cannot predict with high accuracy which covered individuals will file claims, but the magic of statistics allows them to predict with great accuracy the aggregate claims. Disaster insurers run into trouble when rare events happen, such as when Class 5 hurricanes hit Florida. Suddenly, potentially all covered individuals tap into the insurance pool all at once. Insurers try to pool together risks of different types and different areas. The global pandemic is a huge threat to the insurance industry. Perhaps there are lawyers reading this, and they might tell me there are natural disaster exclusions. If those don’t apply, we will see a gigantic number of covered entities filing claims all at once.

Florida’s hurricane insurance market was dysfunctional also because the risk is predictable along some parameters. Those living inland realized they were subsidizing the coastal dwellers and wanted out. The coronavirus also creates different risk subgroups. The younger generations have much lower risks than the elderly. When social-distancing and other practices are applied to the whole community, the subgroup with lower risk peels away. A key difference is community transmission – the risk of hurricane damages does not spread by contact.

***

The above covers less than half of the book. Almost everything in those pages has direct relevance to the Covid-19 world. It’s quite heartening to find out ten years later numbers indeed rule our world.

Here’s a link to get your copy of Numbers Rule Your World. There are also translated copies in Chinese (simplified and traditional), Japanese, Korean, and Portuguese, as far as I know.

Deep-dive analysis goes off track: do women earn lower test scores because it's too cold?

A couple of researchers recently claimed to have found evidence of “a battle of the thermostat”: specifically, they argued that women perform significantly worse in a math test when it is administered at lower room temperatures while men’s performance does not decline. They did not find a similar gender gap in a word test.

The analysis strategy is similar to one that is employed by business analysts so readers here might be interested in how it went off track.

First, the researchers looked at the effect of temperature on test scores in aggregate. They found none. See the chart below, which is a replicate of similar charts in the research paper.

Note that in my chart, the scores have been standardized (to be explained below). For both math and word scores, the difference in scores between the extreme temperatures of 15 to 32 celsius (60 to 90 Fahrenheit) is smaller than 0.25 standard units, clearly not statistically significant by any measure.

The next step in the analysis strategy is to rotate through other factors that might influence test scores, such as gender, language spoken, and college major. In the business world, this is often called a deep-dive analysis. The question being asked is whether the effect of temperature on test scores differed by [factor X]. The researchers struck gold when splitting the data by gender. Here are the relevant charts:

In the top chart displaying math scores, it appears that the trend line for men is almost flat while women’s scores progressively increase as temperature increases. According to the usual convention of statistical significance, the effect is strong enough to be published. It is even possible to explain this observation using "battle of the thermostat".

***

Many other researchers have expressed skepticism that the reported effect can be replicated in other settings. See, for example, here.

I explored the data a bit more. Here is the raw data shown in side-by-side boxplots. I have arranged the sessions by the average temperatures, ordered from lowest to highest.

 

The following features caught my attention:

• The math scores of women (top chart, red boxplots) show unusually low variability. The dispersion of these scores is much below that of men’s math scores. The dispersion of these women’s math scores is also much below that of women’s word scores (bottom chart, red boxplots).
• Focusing only on the math scores (top chart), and inspecting the data session by session, from the right to the left, I found that the gender difference is hard to see (often obscured by the high variability of the male scores), except in a few sessions with the lowest temperatures (boxed area).

In terms of math scores, the shape of the distribution for men is clearly different from that for women. The scores for men were generally higher, and contain more extreme values on the high side. If a male and a female student score the same in absolute terms, the scores do not mean the same thing in relative terms – each relative to other students of the same gender.

***

One way to deal with this difference in variability is to standardize the scores by gender. This leads to the following chart, in which the scores are expressed relative to each gender’s distribution.

Notice how the math and word charts look almost identical. Also compare to the chart above using the raw data. This shows the fragility of statistical significance: the top chart shows marginally significant while the bottom chart shows not significant.

***

[Technical note: Here is a scatter plot of the standardized math scores against the raw scores, grouped by gender. You can see that the raw women's scores are in a narrower range, and that men's scores are more disperse and contain some high extremes.]

***

Deep-dive analysis sometimes lead to unexpected discoveries. Aggregate data sometimes mask subgroup differences. But at other times, the subgroup difference is a mirage. At the disaggregated level, the sample size is smaller, and there may also be a difference in variability.

For more discussion of subgroup analysis, see Chapter 3 of Numbers Rule Your World (link), in which I discuss how the insurance industry and the educational testing community tackle this problem.

 

 

 

 

Reading some wild traffic statistics from New Zealand

Only 6% of crashes in New Zealand involve foreign drivers, according to the latest figures provided by the Ministry of Transport.

But in some remote regions of the South Island particularly popular with tourists for their scenery... foreign drivers are involved in about a quarter of all crashes.

These sentences come from a CNN article about a vigilante movement in those regions popular with tourists. The vigilantes snatch car keys from tourists who annoy them by holding up traffic along the scenic routes. 

My friend Tonny saw this article and thought about Numbers Rule Your World. I love to hear stories about how you're able to relate the stories in my book to other real-world situations.

***

The 6% aggregate number hides the effect of tourists on the rate of accidents. The effect of tourists is different depending on which region one is looking at. For this particular article, only the 25% number is relevant--and even this point is not clearcut. I'd like to know whether the vigilante incidents are exclusively occurring in those "remote regions of the South Island", as implied.

The 25% figure does not address the more important question of whether locals or tourists are more likely to get into accidents in those regions. While the tourists accounted for one out of four accidents, they also comprise a large proportion of the traffic. If, say, 50% of the cars on those roads are driven by tourists, then they are disproportionately less likely to be involved in accidents. The base rate of tourist traffic is the missing data.

Further, a margin of error is useful, especially if few accidents occur in those remote areas.

Chapter 1 of Numbers Rule Your World deals with the notion of the statistical average and Chapter 3 investigates when it is appropriate to aggregate data and when it's not. Learn more about the book here. 

 

I don't like 401(k) either

Felix Salmon hates the 401(k), and he explains his reasoning here. His strongest argument is the data, which shows that the first generation of retirees who grew up with these individual retirement savings accounts find themselves with meager retirement savings (average: $120,000, excluding those with zero).

I have always disliked 401(k), and here are some reasons:

I hate the myth of individual control. These accounts (just like health savings accounts and other similar things) are sold to us as an expression of individual freedom and choice. But it is a mirage.

First, it is never a good idea to chop up money into little pockets. The same banks who every day praise the benefit of diversifying our portfolios also lobby the government to establish these individual accounts. If you earn $60,000 a year, and get paid bi-weekly, then your paycheck is $2,500 less tax withholding. If you put 5% of your earnings into the 401(k), then every two weeks, you put away an extra $125. Imagine you then split the $125 into five or ten investments. It just doesn't make much sense.

And this doesn't even consider the per-trade fees that Charles Schwab, or E*Trade, or any of those firms charge us, as a percentage of the expected gains from your trade. Transaction costs are where individuals will always lose to funds.

Second, how many of us can truly beat investment professionals at the investment game? Most of us don't have the time or inclination to be monitoring our portfolio 24/7. The professionals have access to better information. They are able to react faster when something unexpected happens. In fact, our too-big-to-fail banks routinely have zero days of trading losses in a given quarter when they trade on their own accounts. (This does not mean these banks will perform the same when it comes to our money.) I'm aware that most professional fund managers are shown to do no better than some simulated simple-minded strategy. Is there a study that takes the investment performance of the average Joe and compare them to the average fund manager?

Third, moving from a defined benefit plan to a 401(k) is to shift risk from institutions to individuals. When we were sold the myth of individual control, we were also gifted the Trojan Horse of financial risk. Now, we are responsible for our own individual bad decisions. This is an important point. More people in the pool lowers the average risk.

Fourth, the 401(k) is regressive. The lower your salary, the smaller your biweekly contribution. The risk borne by the individual is higher when the principal is smaller.

If those are not enough, individual control is simply a lie, and becoming more so each year. Most of us find that the banks that run the 401(k) only allow us to put our money in one of the funds they administer. They charge a fee to put the money elsewhere. In one case, I was told if I really want to invest in funds not in the approved list, I must first buy one of the designated funds, and then transfer the money to the fund I really want. Needless to say, the in-and-out is not free. There is another company who matches my contribution but in company stock, which is the opposite of diversification; and if I want to transfer the money to a different asset, I have to pay fees.

It's a steep price to pay to enjoy that "individual control."

 

The most cited and most butchered statistical law

The past week, you are unable to avoid, if you read the business news, mention of the "law of large numbers". We are led to believe that tech giants like Apple and Amazon are suffering from this statistical law. For example:

Apple: Newest Victim of the Law of Large Numbers (CNet)

New York Times Paywall Growth Slows (Columbia Journalism Review): "Much of this is due to the law of large numbers"

Amazon Growth Slows While Profit Margin Expands (Yahoo News/Reuters): "The law of large numbers is affecting Amazon, too."

Is WebRTC a Threat to Skype? (UC Strategies): "The law of large numbers provides protection [to Skype against the threat of a new technology]."

***

This is time to re-read my 2010 entry, called "The Many Laws of Large Numbers" (link). Also Chapter 3 of Numbers Rule Your World covers disaster insurance, and Chapter 5 covers lotteries, both of which are proper applications of the statistical Law of Large Numbers.

Use of statistics in setting insurance rates

Reader Mark Johnstone, from across the pond, points me to some fascinating materials that are highly irrelevant to those who have read Chapter 3 of Numbers Rule Your World, where I explain the statistical underpinning of insurance policies. It's unfortunate that our policy-makers do not understand the probabilistic nature of this business, and create rules that are self-defeating.

The EU Court recently decreed that insurers are not allowed to use gender as a factor in determining how much one pays for insurance. (WSJ article here; also PDF of the ruling). For example, life insurers sell policies to women at lower average prices than to men--well, women on average live longer than men. The argument against such differential pricing is that it discriminates between people. That fundamentally usurps the entire concept of insurance.

As I said in the book, insurance is an ingenious and extremely human (and humane) concept. It turns what seems like a bad thing (cross-subsidies) into a good thing (lower average prices for everyone). It protects the unlucky few. It is always the case that some people subsidize other people. But we accept that because it is not known beforehand who would suffer the early death and who would have the long life.

The insurance scheme fails if some customers feel like the subsidy has been set up unfairly, that is, you think you are a sure loser who subsidizes others in the scheme. This is why insurers would give women lower prices. Otherwise, the women would drop out of the insurance pool if and when they realize they would always be subsidizing the men.

***

This ruling may have broad implications on all kinds of businesses because any predictive models used in marketing or credit scoring or online ad targeting,etc. all contain gender as one of the factors. I have always understood that if gender is only one of many factors in a multi-factorial model, then we are acting in good faith.

In Chapter 2 of the book, I explained how a simple credit-scoring model works. If gender is only one of many factors in the model, then not all women receive the same credit score. Amongst women, other factors such as income, education, the kinds of magazines you read, etc. determine one's particular score.

That said, if someone were to remove all other factors from the model, that is to say, if you aggregate everything else and come up with the credit score for the average woman and that for the average man, they are certainly not identical. But that is a very poor use of statistical aggregation. That's when you throw out a lot of important data to arrive at an overly simplistic conclusion.

Here's an example of such aggregation (link):

In Chapter 2, I made the argument that for credit-scoring applications, statistical models based on correlations are sufficient. It doesn't matter whether it is gender that causes the lower propensity to be responsible for car accidents. If it is consistently the case that if someone satisfies a set of criteria (one of which may be female), one has much lower risk, then to me, it is enough to give such people lower rates. The Court rejects this argument and suggests that only causal models can be applied (which is as good as saying, no models can be used.)

***

Mark told me one of the proposals out there is "to fit 'black boxes' into cars so more individual data can be collected, as opposed to relying heavily on aggregates". Presumably, this suggestion is made to automobile insurers.

You know what, the statistics cannot be suppressed, and the result of using this method will not be materially different. If you now take the rates set by these "black boxes" and aggregate them up by gender, as I described above, it is again certain that the rate paid by the average man will be different from that paid by the average woman. The only way by which they would be the same would be that male and female drivers cause the same proportion of accidents, which is demonstrably false.

***

Would insurance premiums go up as a result of this?

It would seem to me that women will pay more and men may pay slightly less. Eventually, some women will realize that their premiums essentially subsidize men. If they start dropping out of the pool, then everyone's premiums would rise.

***

Would the EU Court stop at gender? Arguably, discriminating on age, on education, on incomes, on what journals you subscribe to, etc. are all bad.

One other point I made in Chapter 2 is that automated models are just super-charged versions of manual rate-setting. If a human being is asked to set rates for different customers, he or she would end up setting lower prices for the average woman anyway. In fact, from the very beginning, actuarial tables have gender breakdowns because longevity clearly differs by gender.

 

 

Speaking analytics

I was a guest on the Analytically Speaking series, organized by JMP. In this webcast (link, registration required), I talk about the coexistence of data science and statistics, why my blog is called "Junk Charts", what I look for in an analytics team, the tension between visualization and machine algorithms, two modes of statistical modeling, and other things analytical.