During this week’s NFL draft, 32 team executives will select 256 prospects in the most-hyped, most-scrutinized
event of its kind.
Whatever happens will make or break talent-evaluation careers, and help
plot the course of each franchise over the next I'mdecade or more. And it
all revolves around what is essentially a very public set of
predictions.
Like traders bidding for commodities and speculating on their
relative worth, each pick a team makes is essentially a statement about
how it expects a player’s career to turn out. Overvalue the commodity
(i.e., draft a guy too early) and you end up with a bust; undervalue it
and risk another team walking away with a prized prospect. Because of
all of the effort and examination being poured into these predictions,
the draft is a robust market that, in the aggregate, does a good job of
sorting prospects from top to bottom.
Yet despite so many people trying to “beat the market,” no single actor
can do it consistently. Abnormal returns are likely due to luck, not
skill. But that hasn’t stopped NFL executives from behaving with the
confidence of traders.
The
efficient-market hypothesis
states that — with certain caveats — markets are informationally
efficient. Since any one investor theoretically operates with the same
set of information as any other,
the EMH claims that no individual can consistently achieve
risk-adjusted returns in excess of the market-wide average. This
conclusion, most notably proposed by University of Chicago professor
Eugene Fama in the 1960s, isn’t perfect (it can’t explain speculative
bubbles, for instance), but it’s a testament to the power of an ideal
market.
The NFL’s draft market differs slightly from the financial markets
Fama analyzed. There are legal opportunities for teams to gather inside
knowledge through prospect workouts and interviews, which a buyer can’t
do with stocks.
But a large proportion of the information teams use to make their picks
— tape of prospects’ college games, their college statistics, biometric
data from the pre-draft combine — is available to every team. Teams, of
course, differ in how they interpret this data, which is why not
everybody wants the same players. That’s where teams’ scouting and,
increasingly, quantitative analysis departments come in.
If certain teams had superior talent-evaluation abilities then we’d
expect them to achieve a greater return on their draft picks than the
average team, after adjusting for where the picks were made in the
draft. But if the NFL Draft follows the same general guidelines
financial markets do (at least, according to the efficient-market
hypothesis), there wouldn’t be much of a relationship between a team or
an executive’s drafting performance across multiple years’ worth of drafts.
We can test this empirically. Remember when we said the NFL draft
does a good job of sorting prospects? We know this because there’s a
strong relationship between the
performance of a player and where he was picked in the draft.
Fluctuations happen all the time around the red line, which
represents a smoothed average value for each pick slot based on the
typical NFL performance of players drafted there. Players routinely play
better — and worse — than these long-term averages. But teams can’t
regularly predict which prospects will outperform or underperform
relative to where they were drafted.
If teams showed any consistency in their ability to out-draft the
market, it would show up in these deviations. But, as Chase Stuart of
FootballPerspective.com
has also found, there’s practically no correlation between a team’s picking performance from one draft to the next.
Perhaps limiting ourselves to the team level isn’t quite the best way
to look at draft returns. After all, this is as much (or more) a
question of the predictive powers of individual decision-makers, and
teams can churn through those folks rather quickly. We wouldn’t want to
hold it against one general manager that his predecessor made poor
selections.
Luckily, Pro-Football-Reference.com keeps an
executives database,
which allows us to isolate the draft decisions of individual general
managers. This means we can perform the same test at the GM level as
well — and, once again, there’s virtually no relationship between how well a GM drafts, relative to average, from one year to the next.
Even if we look at executives’ drafts in three-year segments — which
is, by definition, conditional on a GM retaining his job for six seasons
(an eternity in the what-have-you-done-for-me-lately world of the NFL) —
the relationship between drafting performance from one three-year span
and the next is weak at best.
While some veteran general managers were able to sustain positive
returns above average over six or more years, even theirs were not
unqualified success stories. Along with former Green Bay Packers GM Ron
Wolf, ex-San Diego Chargers GM A.J. Smith and ex-Indianapolis Colts GM
Bill Polian were the three best drafting executives in our data set on a
per-pick basis. But
as Pro-Football-Reference’s Stuart notes, despite Smith and Polian’s track records, both were fired from their posts after a series of poor drafts.
In fact, Polian and Smith merely might have been examples of what’s called the “
Wyatt Earp Effect.”
It’s named for 19th-century gunslinger, whose fame came from the
seeming improbability of an individual surviving countless consecutive
gunfights. Any feat seems improbable in hindsight from the perspective
of the people involved, but given the volume of gunfights in the Old
West, the odds were actually pretty high that
someone would make it through a large number of battles unscathed, simply by chance alone.
Likewise, even over a half-decade or more, some GMs would appear to
beat average by chance alone. But as we saw with Polian and Smith,
eventually that luck runs out.
All of this means that the NFL draft’s mechanism for sorting players
is largely an efficient system, in the sense that none of its individual
actors have the ability to “beat the market” in the long run. Some do
see short-term deviations from the mean, but those prove
unsustainable over larger samples. The implication is that much of what
each team gets from its draft picks — the very entryway to the league
for almost every NFL player — is determined by pure chance.
This doesn’t have to be a knock on the NFL’s talent evaluators. The author Michael Mauboussin
has written
about what he calls the “Paradox of Skill,” a counterintuitive theory
that states that as the aggregate skill level of a market’s participants
increases, the proportion of outcomes attributable to luck also
increases. Put another way, the smaller the variation in skill between
competitors, the more opportunity for randomness to be a differentiating
factor. By this reading, NFL general managers are the victims of their
own obsessive pre-draft preparations — their skill level has increased
so much that only the effects of chance remain.
But there’s another interpretation. Cade Massey and Richard Thaler’s
seminal paper (PDF),
“The Loser’s Curse,” argues that NFL decision-makers shouldn’t be so
quick to attribute the apparent efficiency of the draft market to an
abundance of picking skill. To do so is hubris.
As Massey and Thaler point out, the more that teams study players and
gather information about them, the more assured they become in their
ability to differentiate among prospects of roughly the same talent
level. This leads to overconfidence, and the tendency to make what they
call “non-regressive predictions” — forecasts that don’t appropriately
account for the uncertainty in projecting college players’ performance
in the NFL — about the future value of potential draftees.
This isn’t hard to show empirically, either. After examining 1,078
draft-pick swaps between 1983 and 2008, Massey and Thaler found that
teams’ behavior when trading picks corresponds incredibly well to the
famous
draft-value chart popularized by former Dallas Cowboys and Miami Dolphins coach Jimmy Johnson.
Like our earlier draft-pick value curve, Johnson’s table of draft
values — or, as it’s called in NFL circles, “The Chart” — provides
estimates for the relative worth of each pick. Although it’s been 15
years since Johnson last coached in the NFL, teams
still rely on his chart
as a guideline in the hopes of extracting equal (or better) value out
of trades. It also gives us great insight into the overconfidence
phenomenon Massey and Thaler wrote about. Here’s the Johnson chart,
recalibrated to the same scale as
Approximate Value.
While the empirical chart reflects the inherent uncertainty of
draft-day success (even for high picks), and tails off gradually as the
draft progresses, Johnson’s chart assigns extremely large value to high
picks, and slopes downward sharply after the top 10 to 20 picks —
implying that the drop-off in talent between a high first-rounder and
any other pick is immense.
If The Chart is an accurate gauge of how teams value each draft slot,
then NFL decision-makers place an incredible premium on high draft
picks. But the huge disparity between the observed performance of each
pick and its apparent market value supports Massey and Thaler’s
hypothesis that teams are not being realistic about their own ability to
differentiate among prospects.
They should be.
Research by TheBigLead’s
Jason Lisk
(then writing for Pro-Football-Reference) shows that teams with
top-five picks in the draft correctly identify the player who goes on to
have the best career only 10.3 percent of the time, a success rate that
only gets worse as things progress deeper into the draft. So
a team that believes it could somehow beat the market if only it
controlled its own fate can end up doing more harm than good if it
trades away lower picks to move up in the draft. This is especially the
case if a team uses Johnson’s unrealistically optimistic chart as
justification for such behavior.
Similarly, Massey and Thaler point out that even if estimates of a
player’s potential fluctuate around his true value in an unbiased way,
the team whose evaluation is off by the most on the high side will fall
victim to the
“Winner’s Curse” — and draft the player at a much higher pick than he merits.
These cognitive biases are working against most if not all teams, and
their presence suggests that there is room to improve the drafting
process, even if no team has historically demonstrated an ability to
out-predict the crowd over a long period of time.
Keep that in mind when you watch the draft Thursday, Friday and
Saturday. While the odds are that your team won’t be able to use the
proceedings as a springboard to a series of highly successful future
drafts, there’s always the hope that it can improve its chances with a
more rational process. And if that fails — hey, there’s always luck.
Correction (May 8, 2:45 p.m.): An earlier version of
this story misstated the number of picks in the NFL draft as 224. The
number is in fact 256 when including the 32
compensatory picks.
Posts
.jpg)
Mystery Military Men in Boston Were Part of WMD Response Team
