No Team Can Beat the Draft
During this week’s NFL draft, 32 team executives will select 256 prospects in the most-hyped, most-scrutinized event of its kind.
Whatever happens will make or break talent-evaluation careers, and help
plot the course of each franchise over the next I'mdecade or more. And it
all revolves around what is essentially a very public set of
predictions.
Like traders bidding for commodities and speculating on their relative worth, each pick a team makes is essentially a statement about how it expects a player’s career to turn out. Overvalue the commodity (i.e., draft a guy too early) and you end up with a bust; undervalue it and risk another team walking away with a prized prospect. Because of all of the effort and examination being poured into these predictions, the draft is a robust market that, in the aggregate, does a good job of sorting prospects from top to bottom.1 Yet despite so many people trying to “beat the market,” no single actor can do it consistently. Abnormal returns are likely due to luck, not skill. But that hasn’t stopped NFL executives from behaving with the confidence of traders.
The efficient-market hypothesis states that — with certain caveats — markets are informationally efficient. Since any one investor theoretically operates with the same set of information as any other,2 the EMH claims that no individual can consistently achieve risk-adjusted returns in excess of the market-wide average. This conclusion, most notably proposed by University of Chicago professor Eugene Fama in the 1960s, isn’t perfect (it can’t explain speculative bubbles, for instance), but it’s a testament to the power of an ideal market.
The NFL’s draft market differs slightly from the financial markets Fama analyzed. There are legal opportunities for teams to gather inside knowledge through prospect workouts and interviews, which a buyer can’t do with stocks.3 But a large proportion of the information teams use to make their picks — tape of prospects’ college games, their college statistics, biometric data from the pre-draft combine — is available to every team. Teams, of course, differ in how they interpret this data, which is why not everybody wants the same players. That’s where teams’ scouting and, increasingly, quantitative analysis departments come in.
If certain teams had superior talent-evaluation abilities then we’d expect them to achieve a greater return on their draft picks than the average team, after adjusting for where the picks were made in the draft. But if the NFL Draft follows the same general guidelines financial markets do (at least, according to the efficient-market hypothesis), there wouldn’t be much of a relationship between a team or an executive’s drafting performance4 across multiple years’ worth of drafts.
We can test this empirically. Remember when we said the NFL draft does a good job of sorting prospects? We know this because there’s a strong relationship between the performance of a player and where he was picked in the draft.5
Fluctuations happen all the time around the red line, which represents a smoothed average value for each pick slot based on the typical NFL performance of players drafted there. Players routinely play better — and worse — than these long-term averages. But teams can’t regularly predict which prospects will outperform or underperform relative to where they were drafted.
If teams showed any consistency in their ability to out-draft the market, it would show up in these deviations. But, as Chase Stuart of FootballPerspective.com has also found, there’s practically no correlation6 between a team’s picking performance from one draft to the next.
Perhaps limiting ourselves to the team level isn’t quite the best way to look at draft returns. After all, this is as much (or more) a question of the predictive powers of individual decision-makers, and teams can churn through those folks rather quickly. We wouldn’t want to hold it against one general manager that his predecessor made poor selections.
Luckily, Pro-Football-Reference.com keeps an executives database,7 which allows us to isolate the draft decisions of individual general managers. This means we can perform the same test at the GM level as well — and, once again, there’s virtually no relationship8 between how well a GM drafts, relative to average, from one year to the next.
Even if we look at executives’ drafts in three-year segments — which is, by definition, conditional on a GM retaining his job for six seasons (an eternity in the what-have-you-done-for-me-lately world of the NFL) — the relationship between drafting performance from one three-year span and the next is weak9 at best.
While some veteran general managers were able to sustain positive returns above average over six or more years, even theirs were not unqualified success stories. Along with former Green Bay Packers GM Ron Wolf, ex-San Diego Chargers GM A.J. Smith and ex-Indianapolis Colts GM Bill Polian were the three best drafting executives in our data set on a per-pick basis.10 But as Pro-Football-Reference’s Stuart notes, despite Smith and Polian’s track records, both were fired from their posts after a series of poor drafts.
In fact, Polian and Smith merely might have been examples of what’s called the “Wyatt Earp Effect.” It’s named for 19th-century gunslinger, whose fame came from the seeming improbability of an individual surviving countless consecutive gunfights. Any feat seems improbable in hindsight from the perspective of the people involved, but given the volume of gunfights in the Old West, the odds were actually pretty high that someone would make it through a large number of battles unscathed, simply by chance alone.
Likewise, even over a half-decade or more, some GMs would appear to beat average by chance alone. But as we saw with Polian and Smith, eventually that luck runs out.
All of this means that the NFL draft’s mechanism for sorting players is largely an efficient system, in the sense that none of its individual actors have the ability to “beat the market” in the long run. Some do see short-term deviations from the mean, but those prove unsustainable over larger samples. The implication is that much of what each team gets from its draft picks — the very entryway to the league for almost every NFL player — is determined by pure chance.
This doesn’t have to be a knock on the NFL’s talent evaluators. The author Michael Mauboussin has written about what he calls the “Paradox of Skill,” a counterintuitive theory that states that as the aggregate skill level of a market’s participants increases, the proportion of outcomes attributable to luck also increases. Put another way, the smaller the variation in skill between competitors, the more opportunity for randomness to be a differentiating factor. By this reading, NFL general managers are the victims of their own obsessive pre-draft preparations — their skill level has increased so much that only the effects of chance remain.
But there’s another interpretation. Cade Massey and Richard Thaler’s seminal paper (PDF), “The Loser’s Curse,” argues that NFL decision-makers shouldn’t be so quick to attribute the apparent efficiency of the draft market to an abundance of picking skill. To do so is hubris.
As Massey and Thaler point out, the more that teams study players and gather information about them, the more assured they become in their ability to differentiate among prospects of roughly the same talent level. This leads to overconfidence, and the tendency to make what they call “non-regressive predictions” — forecasts that don’t appropriately account for the uncertainty in projecting college players’ performance in the NFL — about the future value of potential draftees.
This isn’t hard to show empirically, either. After examining 1,078 draft-pick swaps between 1983 and 2008, Massey and Thaler found that teams’ behavior when trading picks corresponds incredibly well to the famous draft-value chart popularized by former Dallas Cowboys and Miami Dolphins coach Jimmy Johnson.
Like our earlier draft-pick value curve, Johnson’s table of draft values — or, as it’s called in NFL circles, “The Chart” — provides estimates for the relative worth of each pick. Although it’s been 15 years since Johnson last coached in the NFL, teams still rely on his chart as a guideline in the hopes of extracting equal (or better) value out of trades. It also gives us great insight into the overconfidence phenomenon Massey and Thaler wrote about. Here’s the Johnson chart, recalibrated to the same scale as Approximate Value.
While the empirical chart reflects the inherent uncertainty of draft-day success (even for high picks), and tails off gradually as the draft progresses, Johnson’s chart assigns extremely large value to high picks, and slopes downward sharply after the top 10 to 20 picks — implying that the drop-off in talent between a high first-rounder and any other pick is immense.
If The Chart is an accurate gauge of how teams value each draft slot, then NFL decision-makers place an incredible premium on high draft picks. But the huge disparity between the observed performance of each pick and its apparent market value supports Massey and Thaler’s hypothesis that teams are not being realistic about their own ability to differentiate among prospects.
They should be. Research by TheBigLead’s Jason Lisk (then writing for Pro-Football-Reference) shows that teams with top-five picks in the draft correctly identify the player who goes on to have the best career only 10.3 percent of the time, a success rate that only gets worse as things progress deeper into the draft.11 So a team that believes it could somehow beat the market if only it controlled its own fate can end up doing more harm than good if it trades away lower picks to move up in the draft. This is especially the case if a team uses Johnson’s unrealistically optimistic chart as justification for such behavior.
Similarly, Massey and Thaler point out that even if estimates of a player’s potential fluctuate around his true value in an unbiased way, the team whose evaluation is off by the most on the high side will fall victim to the “Winner’s Curse” — and draft the player at a much higher pick than he merits.
These cognitive biases are working against most if not all teams, and their presence suggests that there is room to improve the drafting process, even if no team has historically demonstrated an ability to out-predict the crowd over a long period of time.
Keep that in mind when you watch the draft Thursday, Friday and Saturday. While the odds are that your team won’t be able to use the proceedings as a springboard to a series of highly successful future drafts, there’s always the hope that it can improve its chances with a more rational process. And if that fails — hey, there’s always luck.
Correction (May 8, 2:45 p.m.): An earlier version of this story misstated the number of picks in the NFL draft as 224. The number is in fact 256 when including the 32 compensatory picks.
Like traders bidding for commodities and speculating on their relative worth, each pick a team makes is essentially a statement about how it expects a player’s career to turn out. Overvalue the commodity (i.e., draft a guy too early) and you end up with a bust; undervalue it and risk another team walking away with a prized prospect. Because of all of the effort and examination being poured into these predictions, the draft is a robust market that, in the aggregate, does a good job of sorting prospects from top to bottom.1 Yet despite so many people trying to “beat the market,” no single actor can do it consistently. Abnormal returns are likely due to luck, not skill. But that hasn’t stopped NFL executives from behaving with the confidence of traders.
The efficient-market hypothesis states that — with certain caveats — markets are informationally efficient. Since any one investor theoretically operates with the same set of information as any other,2 the EMH claims that no individual can consistently achieve risk-adjusted returns in excess of the market-wide average. This conclusion, most notably proposed by University of Chicago professor Eugene Fama in the 1960s, isn’t perfect (it can’t explain speculative bubbles, for instance), but it’s a testament to the power of an ideal market.
The NFL’s draft market differs slightly from the financial markets Fama analyzed. There are legal opportunities for teams to gather inside knowledge through prospect workouts and interviews, which a buyer can’t do with stocks.3 But a large proportion of the information teams use to make their picks — tape of prospects’ college games, their college statistics, biometric data from the pre-draft combine — is available to every team. Teams, of course, differ in how they interpret this data, which is why not everybody wants the same players. That’s where teams’ scouting and, increasingly, quantitative analysis departments come in.
If certain teams had superior talent-evaluation abilities then we’d expect them to achieve a greater return on their draft picks than the average team, after adjusting for where the picks were made in the draft. But if the NFL Draft follows the same general guidelines financial markets do (at least, according to the efficient-market hypothesis), there wouldn’t be much of a relationship between a team or an executive’s drafting performance4 across multiple years’ worth of drafts.
We can test this empirically. Remember when we said the NFL draft does a good job of sorting prospects? We know this because there’s a strong relationship between the performance of a player and where he was picked in the draft.5
Fluctuations happen all the time around the red line, which represents a smoothed average value for each pick slot based on the typical NFL performance of players drafted there. Players routinely play better — and worse — than these long-term averages. But teams can’t regularly predict which prospects will outperform or underperform relative to where they were drafted.
If teams showed any consistency in their ability to out-draft the market, it would show up in these deviations. But, as Chase Stuart of FootballPerspective.com has also found, there’s practically no correlation6 between a team’s picking performance from one draft to the next.
Perhaps limiting ourselves to the team level isn’t quite the best way to look at draft returns. After all, this is as much (or more) a question of the predictive powers of individual decision-makers, and teams can churn through those folks rather quickly. We wouldn’t want to hold it against one general manager that his predecessor made poor selections.
Luckily, Pro-Football-Reference.com keeps an executives database,7 which allows us to isolate the draft decisions of individual general managers. This means we can perform the same test at the GM level as well — and, once again, there’s virtually no relationship8 between how well a GM drafts, relative to average, from one year to the next.
Even if we look at executives’ drafts in three-year segments — which is, by definition, conditional on a GM retaining his job for six seasons (an eternity in the what-have-you-done-for-me-lately world of the NFL) — the relationship between drafting performance from one three-year span and the next is weak9 at best.
While some veteran general managers were able to sustain positive returns above average over six or more years, even theirs were not unqualified success stories. Along with former Green Bay Packers GM Ron Wolf, ex-San Diego Chargers GM A.J. Smith and ex-Indianapolis Colts GM Bill Polian were the three best drafting executives in our data set on a per-pick basis.10 But as Pro-Football-Reference’s Stuart notes, despite Smith and Polian’s track records, both were fired from their posts after a series of poor drafts.
In fact, Polian and Smith merely might have been examples of what’s called the “Wyatt Earp Effect.” It’s named for 19th-century gunslinger, whose fame came from the seeming improbability of an individual surviving countless consecutive gunfights. Any feat seems improbable in hindsight from the perspective of the people involved, but given the volume of gunfights in the Old West, the odds were actually pretty high that someone would make it through a large number of battles unscathed, simply by chance alone.
Likewise, even over a half-decade or more, some GMs would appear to beat average by chance alone. But as we saw with Polian and Smith, eventually that luck runs out.
All of this means that the NFL draft’s mechanism for sorting players is largely an efficient system, in the sense that none of its individual actors have the ability to “beat the market” in the long run. Some do see short-term deviations from the mean, but those prove unsustainable over larger samples. The implication is that much of what each team gets from its draft picks — the very entryway to the league for almost every NFL player — is determined by pure chance.
This doesn’t have to be a knock on the NFL’s talent evaluators. The author Michael Mauboussin has written about what he calls the “Paradox of Skill,” a counterintuitive theory that states that as the aggregate skill level of a market’s participants increases, the proportion of outcomes attributable to luck also increases. Put another way, the smaller the variation in skill between competitors, the more opportunity for randomness to be a differentiating factor. By this reading, NFL general managers are the victims of their own obsessive pre-draft preparations — their skill level has increased so much that only the effects of chance remain.
But there’s another interpretation. Cade Massey and Richard Thaler’s seminal paper (PDF), “The Loser’s Curse,” argues that NFL decision-makers shouldn’t be so quick to attribute the apparent efficiency of the draft market to an abundance of picking skill. To do so is hubris.
As Massey and Thaler point out, the more that teams study players and gather information about them, the more assured they become in their ability to differentiate among prospects of roughly the same talent level. This leads to overconfidence, and the tendency to make what they call “non-regressive predictions” — forecasts that don’t appropriately account for the uncertainty in projecting college players’ performance in the NFL — about the future value of potential draftees.
This isn’t hard to show empirically, either. After examining 1,078 draft-pick swaps between 1983 and 2008, Massey and Thaler found that teams’ behavior when trading picks corresponds incredibly well to the famous draft-value chart popularized by former Dallas Cowboys and Miami Dolphins coach Jimmy Johnson.
Like our earlier draft-pick value curve, Johnson’s table of draft values — or, as it’s called in NFL circles, “The Chart” — provides estimates for the relative worth of each pick. Although it’s been 15 years since Johnson last coached in the NFL, teams still rely on his chart as a guideline in the hopes of extracting equal (or better) value out of trades. It also gives us great insight into the overconfidence phenomenon Massey and Thaler wrote about. Here’s the Johnson chart, recalibrated to the same scale as Approximate Value.
While the empirical chart reflects the inherent uncertainty of draft-day success (even for high picks), and tails off gradually as the draft progresses, Johnson’s chart assigns extremely large value to high picks, and slopes downward sharply after the top 10 to 20 picks — implying that the drop-off in talent between a high first-rounder and any other pick is immense.
If The Chart is an accurate gauge of how teams value each draft slot, then NFL decision-makers place an incredible premium on high draft picks. But the huge disparity between the observed performance of each pick and its apparent market value supports Massey and Thaler’s hypothesis that teams are not being realistic about their own ability to differentiate among prospects.
They should be. Research by TheBigLead’s Jason Lisk (then writing for Pro-Football-Reference) shows that teams with top-five picks in the draft correctly identify the player who goes on to have the best career only 10.3 percent of the time, a success rate that only gets worse as things progress deeper into the draft.11 So a team that believes it could somehow beat the market if only it controlled its own fate can end up doing more harm than good if it trades away lower picks to move up in the draft. This is especially the case if a team uses Johnson’s unrealistically optimistic chart as justification for such behavior.
Similarly, Massey and Thaler point out that even if estimates of a player’s potential fluctuate around his true value in an unbiased way, the team whose evaluation is off by the most on the high side will fall victim to the “Winner’s Curse” — and draft the player at a much higher pick than he merits.
These cognitive biases are working against most if not all teams, and their presence suggests that there is room to improve the drafting process, even if no team has historically demonstrated an ability to out-predict the crowd over a long period of time.
Keep that in mind when you watch the draft Thursday, Friday and Saturday. While the odds are that your team won’t be able to use the proceedings as a springboard to a series of highly successful future drafts, there’s always the hope that it can improve its chances with a more rational process. And if that fails — hey, there’s always luck.
Correction (May 8, 2:45 p.m.): An earlier version of this story misstated the number of picks in the NFL draft as 224. The number is in fact 256 when including the 32 compensatory picks.
No comments:
Post a Comment