In the second part of the behind-the-scenes blog post, they discussed how they visualized work by some economists. This is how the research was summarized: "across all players and positions, teams only picked a player better than the person who went next at that position 52 percent of the time."
That's an eye-popping statement, particularly when the use of 52 percent evokes the proverbial fair coin. It leads readers to think that draft picking must be almost as random as throwing a fair coin.
My reaction is this statement is disbelief. On the first order, as discussed in my other post (link), there is a clear and strong correlation between the round of the draft pick and average value. See the chart on the right which shows that the subsequent value of the players decline drastically as the order of the pick increases. When the first-order relationship is this strong, any result that contradicts it must be scrutinized.
The biggest issue in this analysis is the concept of "value". I'm speaking of value as defined in the ChartsNThings blog post: it's a number from a site called pro-football-reference, and a combination of games started and pro bowls, plus some adjustment for position played.
It's not hard to see that this metric is more discriminating for good players and we should expect the worse players to score essentially nothing. The graphics designers noted later in the post that it is not meaningful to talk about one benchwarmer being marginally better than another benchwarmer. (At this point, they stopped talking about the academic paper.)
In other words, the value gaps between the good players are large but the value gaps between middling and bad players will be small, or even nonexistent. The variance of value is small in the middle to lower rounds. This is a problem when you're comparing the value of consecutive picks in the draft: for the most part, you're just picking up noise.
Another potential hiccup are the outliers like Tom Brady who got picked in a later round. The criterion shown above would highlight one particular comparison: if you chose Spergon Wynn (Pick 183 in the 2000 draft) over Tom Brady (Pick 199), your guy is not better than the next guy in the draft. But there are five other QBs picked before Wynn in that draft; none of those teams picked Brady and yet this criterion did not penalize those teams.
Yet another concern is the cohort effect. The more seasons someone has played, the higher the total number of games started and pro bowls. The pro-football-reference site is very unclear as to what they do about this. It seems like they have a value for each season for each player but the NY Times team seemed to have picked out either one season or averaged all seasons or did something to end up with only one value per player. It's very easy to get cohort effect wrong.
Even if the adjustment for the player's tenure is made, we still cannot escape the high variance associated with younger players with short histories.
One last point: any proper analysis has to consider other factors that affect whether the draft pick could perform up to his potential. A quarterback playing behind a poor offensive line isn't going to do well. Injuries reduce playing time. Playing second-string behind an oft-injured player increases playing time, etc. While I haven't read the original paper, it doesn't sound like they accounted for these other factors.