Does run defense win Super Bowls?

Rush defense is a strong barometer of championship potential. More than three-quarters of Super Bowl teams had a top-10 rush defense, according to the Elias Sports Bureau. Nine of the past 14 participants were in the top five.

A Times reporter thus fell victim to the seductive power of data in analyzing the New York Giants' chance of getting to the Super Bowl of American football.  In the past seven years, 64% (9/14) of Super Bowl finalists ranked among the top 5 in run defense; since the Giants currently rank 6th, their chance of getting to the Super Bowl must be about 64% (+/- statistical error).  Thus, run defense "is" (not "can be" or "was" or "has been") a strong barometer or predictor.

Sadly, Giants fans, I bear bad news.  The reporter has put the cart before the horse.  He asked the right question: Are the Giants good enough?  but used the wrong data.  Consider the following two questions:

• Of those teams that made it to the Super Bowl, how many were ranked top 6 in run defense?
• Of those teams that were ranked top 6 in run defense, how many made it to the Super Bowl?

Since the Giants are currently ranked 6th, I'm using top 6 which is more appropriate than top 5 or top 10.  These two questions have different answers as the following pair of histograms show.

What is our data and what is our prediction?  We want to predict the Giants' chance of becoming a Super Bowl team based on our knowledge of their run defense rank.  So putting the horse before the cart, we should use the second chart rather than the first.  In other words, given that their run defense ranking at #6, they have about 24% chance of getting into the Super Bowl! (about 1/3 lower than the reported estimate).

To appreciate the difference, one has to realize that in those seven years, 32 teams that were ranked top 6 in run defense did not reach the Super Bowl (as opposed to 10 teams which did).

The reporter's assertion, however, may still hold.  A team has a 6% chance (2/32) of getting into the Super Bowl, completely at random.  Even assuming that one team in each division is a bottom-dweller with no chance at all, the remaining teams still only have an 8% chance (2/24) of getting there.  Thus, knowing that run defense is in the top 6 has tripled our estimate of the chance of getting to the Super Bowl.

Finally, the data came from this table, which does little to help our comprehension.

Reference: "Giants' Defense Jells, Then Hardens", New York Times, Dec 11, 2005.

For formula crunchers, the above is Bayes rule in practice.  Prob(SB/1<=RD<=6) = Prob(1<=RD<=6/SB) * P(SB) / P(RD) but P(SB) = 2/n while P(RD) = 6/n where n=total # of NFL teams.  So the correct estimate is 1/3 of reported estimate.  I think the histograms above demonstrate the intuition a lot better.

Comments

[John S.]

This type of error is also called the "prosecutor's fallacy".

[Egami]

More like the fallacy of "anal over-analysis". The journalist was simply using numbers to show that basically "Rush defense is a strong barometer of championship potential."

Journalism shouldn't automotically be subject to such critical scrutiny unless the article is trying to draw concrete facts. This is simply an artistic, non-scientific, fanatical, ray of hope analysis.

I mean heck, if you're a Giants fan it's about all you can rely on. ;)

[Kaiser]

to be cynical, sports journalism is nothing more than sports marketing; so if you take that view, then there is not much more to say

but still when the reporter cites data ("concrete facts") as in this case, he should not misrepresent it. calling a 25% chance 70% is quite a leap.

You call that "ray of hope"; I call that "false hope". It's all in your perspective.

[Egami]

He didn't say anything about "chance". Technically, there isn't anything statistically incorrect about what he stated. The data was accurate.

Granted, the manner in which the data is presented can lead itself to a misperception, but that is the fault of the reader as much as the writer.

If anything, what is in error is the extent at which the critic read into what the journalist wrote because the journalist, from this excerpt, doesn't state anything about "chance".

The critic himself makes a fairly substantial statistical error, if you want to look at it via "perspective", he writes:

"To appreciate the difference, one has to realize that in those seven years, 32 teams that were ranked top 6 in run defense did not reach the Super Bowl (as opposed to 10 teams which did)."

Ok, that's asinine...in 7 years exactly 14 teams will make it the SB. 10 of those 14 had a top 6th ranked defense. So, you could just as easily argue, using the critics own logic analysis, that they have a ~71% chance.

Back to the point...I think the facts stated support the assumption stated which was simply: Rush defense is a strong barometer of championship potential.

Both the writer and the critics data could support that.