Calling something a spade doesn't make it a spade

Andrew Gelman pointed readers of his blog to a really great press release / report for those interested in how statisticians look at real-world data. This is work done by the Highway Loss Data Institute (HLDI) which made the Freakonomics-style claim "Texting bans don't reduce crashes; effects are slight crash increases."

This is a long post as I'm going to walk through the analysis in detail. In short, great methodology, bad conclusion. (Bear with me, I'll get to the spade part.)

***

What's good about the piece is how they found a creative and yet credible way to approach a seemingly impossible problem. Ideally, to answer the question "Do texting bans reduce crashes?", you would ban texting for a random group of drivers, and allow texting for another random group of drivers, and then compare the accident rates for the two groups. Of course, we can't do that. Like most public policies, once you make the policy, you don't have test and control groups to do direct comparisons.

What helped is that not all states have banned texting (yet). So they came up with the concept of comparing a text-ban state with its surrounding states that have not banned texting. The chart on the right showed the accident rates in California (text-ban state) compared to Arizona, Nevada and Oregon, before and after the ban went into effect in 2009.

The two lines are essentially the same, implying that the texting ban did not change the accident rates.

What's impressive about this analysis is that they did not make the classical mistake of looking only at the red line. (If the red line shifted a lot after the text ban but so did the blue line, then something else other than the text ban was probably causing the accident rates to change.)

***

Now we get to the spade part. If you persisted and read to the last 25% of the press release, you learned something super-important. There, they tell us:

Noncompliance is a likely reason texting bans aren't reducing crashes. ... Many respondents who knew it was illegal to text said they didn't think police were strongly enforcing the bans. (my italics)

So, here are the three super-important words: nonawareness, noncompliance, nonenforcement. Sounds like the "texting ban" has not stopped anyone from texting while driving, and this can explain the unchanged accident rates. When they state the claim "texting bans don't reduce crashes", it sends the message that despite drivers not texting while driving, accidents remain as frequent as before. That's absolutely not what the research is saying.

In other words, the test group is called "texting ban" but in reality, the behavior of these drivers was no different before or after the ban. (According to surveys, of the 48% drivers who admit they text, 45% said they ignored the ban.) But naming something is very powerful. We think text-ban states have drivers who don't text but with compliance close to zero, text-ban states are essentially the same as non-text-ban states.

Imagine in a clinical trial: drug A is a pill, drug B is administered by a painful injection. Drug B is in fact extremely effective. However, in this trial, drug A beats drug B handily. But looking more deeply, you  discover that nobody in the test group B actually took drug B because the pain was too much. So in reality, the trial compared drug A to no drug at all. A good conclusion for this study is: taking drug A is better than nothing, and no one wants to take drug B. A poor conclusion is: taking drug A is better than taking drug B.

***

Beyond this, savvy readers should be asking these questions:

• How is accident rate measured? The collision claims per 100 insured vehicles, how is it correlated with the accident rate due to texting?
• Why are vehicles older than 9 years old excluded? What impact might this exclusion have?
• What do they mean by the nearby states where texting laws "weren't substantially changed"? Substantially and substantively are very different; substantially seems to me like they were changed but not by much.
• Notice the implicit assumptions: that neighboring states are comparable

***

Finally, what is the evidence for "effects [of texting bans] are slight crash increases"? The HLDI President elaborated: "In a perverse twist, crashes increased in 3 of the 4 states we studied after bans were enacted."

Problem: that is simply not true. It turns out California is considered one of the 3 states. Look back at the graph shown above, and try to convince yourself that accident rates have increased after the text-ban went into effect. I can't. You don't even need statistical software for this.

Still not convinced... this is the chart for Minnesota compared to Iowa and Wisconsin, where the accident rate for Minnesota after the ban went up by 9%, the largest among the 4 text-ban states. First thing you noticed is that there is virtually no difference between the states after the law went in. All of that 9% difference had to do with something that happened about 5 to 10 months prior to the ban, and if anything, that period should be investigated. But I can't see how one can conclude that the text-ban caused accident rates to go up.

***

But the researchers didn't stop there. They now declare story time! First, they managed to convince themselves that noncompliance was not really an issue:

But [noncompliance] doesn't explain why crashes increased after texting bans. If drivers were disregarding the bans, then the crash patterns should have remained steady. So clearly drivers did respond to the bans somehow...

Absolute shocker. They say that because the texting ban increased accident rates (not even true), that proves that there is compliance, and therefore, they could conclude that texting bans did not reduce accidents!  My advice: go find statistics of how many drivers have been booked for texting while driving in those states.

Then, they spin more fiction, following on that last sentence:

... what they might have been doing was moving their phones down and out of sight when they texted, in recognition that what they were doing was illegal. This could exacerbate the risk of texting by taking drivers' eyes further from the road and for a longer time.

Great sounding story but the problem: this sounds smart if the accident rate went up, it sounds silly if it didn't. Imagine if I buy this story--then the onus is on the researchers to explain why drivers moving their phones down and out of sight did *not* in fact lead to increased accident rates.

This is the discontent of story time. These stories are made up to explain one particular interpretation of the observed data, and do not hold up if the data were to be interpreted differently. Either the stories had to be reversed, or ever more elaborate extensions have to be developed.

 

 

 

Comments

[Jon Peltier]

Have you discussed your analysis (a very good one, by the way) with HLDI?

Will they ever discuss it?

[twitter.com/UltraNurd]

Are the spikes on either side of the MN texting ban (in August) just the cyclical increase of accidents during winter? The states in the first graph don't have much of a winter spike because they mostly don't get snow/ice (except in the mountains), right?

[Tom West]

UltraNurd makes a good point. Daily, weekly or monthly crash rates are far to prone to cycical variations. A moving annual average would be far more useful.

The real message seems to be "text bans don't stop peopel texting". The main practical upshot is that if you are involved in a collision while texting, your insurance company may decide you were driving illegally, and make you pay.

[Joe Lotz]

UltraNerd is exactly right. This is where comparing teh trends between the non-banned and banned states are important, as Kaiser suggests. These spikes could (and probably) are cyclical affects, like snow season, Christmas/New Year holiday season.

The sensationalism of their conclusion is pathetic. "...what they MIGHT have been doing was moving their phones down..."

Further more, what they MIGHT have been doing were math problems causing aliens to destract them resulting in accidents. The conclusion? Math is dangerous and should never be taught!!

[Basil]

Nice Ultranurd. Great write up! Perhaps behavior will change as the rules are enforced. It's like trying to stop me from biting my finger nails! AGH

[Kaiser]

Jon: unfortunately I don't have the time to contact HLDI or any of the other organizations featured on the blog. It would be nice if they would correct their press release though.

UltraNurd and others: if your suggestion is true, then their choice of states to compare against each other is flawed!

[bbis]

Another potential issue with the analysis could be that a large percentage of the people most likely to comply with the ban on texting had already stopped texting prior to the ban taking effect. Being aware of the upcoming ban and the reasons behind it, it seems reasonable to suppose many would have chosen to self-implement prior to the legal requirement. The numbers presented during the discussion indicating a roughly 3-fold increase in accident rate while texting are quite compelling.

[David Norman]

Hi, Kaiser,
Reading your blog from the first post, so though this is way after the post date, it's still new to me.

Just wanted to chime in that I disagree that:
"When they state the claim "texting bans don't reduce crashes", it sends the message that despite drivers not texting while driving, accidents remain as frequent as before. That's absolutely not what the research is saying."

While one can infer that the statement means: "Texting bans, when perfect, enforced, and followed, do not reduce crahes" the simple fact is the statement is "Texting bans don't reduce crashes" and that means nothing more than "Implementing a law banning texting doesn't reduce crahses." The reason for the non-reduction, especially non-compliance, is not inferable, and the statement remains very true.

Great blog, by the way!