Fact or Fiction: Analyzing Another Deflategate Article
by Warren Sharp
I wanted to take an opportunity to simultaneously comment on an article written last week which tried to cast doubt on the Deflategate research, while also playing a game of “fact or fiction” because there is a lot of incorrect information floating around regarding my analysis.
First, the key articles in this series (from most recent to oldest):
Point #1 - Something occurred between 2006 and 2007 which allowed the Patriots to fumble the ball at an extremely low rate moving forward when compared to the rate their team fumbled from 2000-06 (Bill Belichick started coaching the team in 2000).
Point #2 - Whatever occurred caused the Patriots to shift from a team who fumbled the football the league average (in 2000-2006) to a team who was so superior when compared any other team the odds it was a mere coincidence are extremely unlikely.
With that said, let’s move on to the “fact or fiction” where we’ll address prior comments made on the article as well as those raised by a new article from Bleacher Report, written by Mike Tanier.
This is exactly what the Bleacher Report article attempts to do. The author, for the entirety of his analysis, uses a 3 year period from 2012-14. This clearly prevents him from disproving either of my Two Primary Conclusions. He is not comparing the early Brady/Belichick years (2000-2006) with more recent ones (2007+), so he can’t disprove Point #1, nor can he show that the Patriots are not superior to the rest of the league since 2007, because his sample is only 3 years. So he can’t disprove Point #2.
Its quite strange, because when introducing my article, he notes: “[Warren Sharp's analysis] found that the Patriots have been avoiding fumbles at a mathematically improbable rate since 2007.” He then said his goal was to “neither to implicate nor exonerate Brady” but “it was important for me to find out whether or not there’s any real evidence.” So instead of seeing if “there’s any real evidence” of this “mathematically improbable” fumble rate since 2007, he decided to look only at 2012-14?
There are two key graphics to look
at
which I created back in January (the first has been updated with red
notations per the Wells Report). The first is the summary graphic
showing where the Patriots ranked in the early period and where they
ranked following 2006. It’s obvious that “something” happened to cause
this shift.
But the next graphic is the key graphic which casts doubt on the objectivity of the Bleacher Report article.
This graphic depicts the huge noticeable shift of the Patriots fumble rate starting in the 2007 season, and is another of my early graphics. It’s really unmistakeable, how they are league average thru 2006. Then they shift to completely unlike the league average starting promptly in 2007.
While the key is the change which occurred in 2007, with regard to the Bleacher Report period, I want to turn your attention to 2013. This happens to be the middle year of the 3 year period grabbed from the Bleacher Report article. As you can see, the Patriots fumble rate was actually WORSE than league average in this season. Why?
It’s entirely because of one, single week 12 game vs Denver. This Sunday night game was played in insane low temps and bitter wind chills. As I’ll discuss below, inclement weather, and in particular, cold (or brutal cold) significantly increases fumbles. In this game, the Patriots fumbled 6 times. Denver had 5 in the game as well (to show you how bad the weather was at causing fumbles). That’s how cold it was. It was 22 degrees pre-wind chill, 6 degrees with the wind of 22 MPH on average. Those 6 fumbles were a huge percentage of the Patriots total fumbles on the season.
Note that the only game which saw more than 11 fumbles (which New England and Denver combined for in that 2013 game) the last 25 years was a game in December, outdoors in Milwaukee County Stadium, when Green Bay and Detroit combined for 12 fumbles. That game was played in temps of 26, with a wind chill of 5 degrees via 14 MPH winds. Five degrees is almost identical to the 6 degree wind chill temp for the 2013 game in New England which saw the Patriots fumble 6 times.
Clearly, the 3 year sample size is suspicious when sandwiched around the only season where the Patriots were below league average in fumble rate (entirely due to that one, 6-fumble game). However, of my selection of a huge 8 year period from 2007-14, the author said it was “customizing a data set to make [the] change look significant”. I believe the opposite to be the case.
Possibly lost within the Bleacher Report article is the statement: “The Patriots’ fumble rates are low, though not extremely or historically low, over the last three years. Factor in the 2010 and 2011 seasons, and the rates get even lower.” This is obvious. And guess what? Factor in 2007-2009 and the rates don’t just “get even lower”, they begin to become improbable to have occurred by chance. The frustrating part of these counter-arguments is the data is there and easily accessible. The authors of the counter-arguments realize if they were to take a 2007-14 period, and then compare it to 2000-06 it would show exactly what I’ve found in my Two Primary Conclusions.
The bottom line with this point is in order to discredit or disprove my Two Primary Conclusions, you have to look at the same time frame and, based on the stats, conclude something different from what I conclude. Looking only at 2012-14 is not going to disprove my argument based on the stats which shows starting in 2007, something happened, and moved the Patriots from average to other-worldly at fumble prevention.
I will jump
ahead briefly to a graphic from Benjamin Morris of FiveThirtyEight who
used a different time period. His was from 2010-14, and even in that
time period, using binomial and Poisson models, he found that the odds
of the Patriots fumble rate during that period occurring by chance was
over 1 in 10,000. The next closest team was 1 in ~800 and most teams saw
completely normal fumble rates, which occur naturally.
I spent inordinate time examining climate affects on fumble rates. I went into great detail on the subject as part of “Argument #3″ of this article I wrote in February. Please read it before continuing. If you simply look at the weather analysis, pulled from one done by Sporting Charts, it is quite apparent that temperature has a measurable effect on fumble rate. Keep in mind, you can’t look at “teams” only, you have to look at game sites and the stats for both teams playing in that particular site. Dome teams infrequently play in cold weather. It’s just a fact.
Look
at the Falcons, for example. I looked at the coldest 6 weeks of the
season, from mid November thru the end of the regular season. And since
2007, the Falcons have played in a TOTAL of 4 games outdoors in the
Northeast! In 8 years! Meanwhile, the Patriots have played 40 games
outdoors in that same sample. Knowing cold weather is more likely to
cause fumbles, why would we group the Falcons in with the Patriots, when
they play 10 times fewer games in the conditions most likely to cause fumbles?
The graphics to the right make it pretty easy to comprehend why its wrong to consider these teams “apples to apples” and compare them side by side. Its pretty obvious that studying those dome teams differently should be done. And that’s exactly what I did.
But, even if you include dome teams, the Patriots are still surprisingly way different than the rest of the NFL. But don’t take my word for it:
Respected websites with long standing records for being committed to understanding data and football have agreed with my findings, as I’ll mention below. Both included dome teams, and their conclusions were the same.
Brian Burke of Advanced Football Analytics studied my results after thinking that they were “so extraordinary they seemed unlikely to be true” but after he ran his own numbers, his conclusions as to the Patriots incredible ball security (aka lack of fumbling) was incredible, and their numbers are “better than the next best team by 20 plays per fumble.” And he included dome teams.
Benjamin Morris of FiveThirtyEight also studied my results and then ran his own. He concluded my study should be taken “more seriously” because “that author correctly identified that the Patriots fumble rate has been absurdly small. I did my own calculations using binomial and Poisson models and found the same.” He also included dome teams.
His results suggest the odds the Patriots could fumble as infrequently as they did was over 1 in 10,000, whereas the next best team in the NFL was 1 in 800, and most teams were below 1 in 5, meaning most teams had fumble rates which were normal. Morris went on to conclude:
But that’s not what happened for Tom Brady and the Patriots. And I’m going to keep this brief because digging in the weeds of individual players isn’t needed to prove either of my Two Primary Conclusions, but I’ll say:
As an aside, the Bleacher Report article criticized my use of “plays per fumble”, rather than using “fumbles per play”. It makes absolutely no difference when comparing to the rest of the NFL. For instance, using “plays per fumble” the Patriots fumbled once approx every 73 plays from 2007-14, which is 74% more than once every 42 plays, their avg from 2000-06. Or, like the author did, you can convert to fumbles per 100 plays, and say the Patriots improved from 2.38 fumbles per 100 plays (in 00-06) to 1.37 fumbles per 100 plays (in 07-14). The rate of improvement is still 74%.
Leaving that fact as it stands, it is also interesting that while criticizing my use of “plays per fumble”, he chooses to conduct his analysis on sacks using “sacks per fumble”. I would have anticipated that he use “fumbles per X sacks” given his seemingly hardline approach to using “per play” stats, but the fact is, it’s irrelevant how you do it as long as you compare the teams similarly.
The Wells report shed light on Jim McNally, who referred to himself as “the deflator” and who started working as the “Officials Locker Room attendant” in 2007. That year is the year after Tom Brady helped get the rule changed, and as you’ve read above, 2007 coincides with the massive change in fumble rates for the Patriots.
The report concluded the Patriots personnel participated in violations of the Playing Rules and were involved in a deliberate effort to circumvent the rules, and Tom Brady was at least generally aware of the inappropriate activities of McNally and Jastremski involving the release of air from Patriots game balls.
Many of the highly suspicious text messages exchanged between McNally and Jastremski occurred throughout the 2014 season, not just immediately prior to and after the Colts game. While the Patriots report attempted to explain that McNally was just an innocent, portly gentleman with a sense of humor who was trying to lose weight, the truth might never be fully known now that Patriots owner Robert Kraft accepted the punishment without appealing.
Clearly, the data concurs with the notion that there was suspicious activity occurring as a result of McNally’s involvement, and it would be fascinating to see text messages dating back to the spring and summer of 2007, which is the start of the period in question. But we will never see them, so its pointless to hypothesize further. From day one, I said the data shows something was abnormal in New England with regard to their fumble rate, which would be a correlated by-product of deflated footballs. I’ve also said there is no way to prove anything with the data. But the Wells Report gave us indications that systemic ball deflation could have been the source. To this day, I still have not seen any stats which disprove either of the Two Primary Conclusions from my analysis.
[I wanted to add that I like and respect Mike Tanier's football knowledge that he has shared for years on Football Outsiders. He has a great football mind, and I've agreed with a lot of his opinions over the years. I just happen to disagree with his response to my research and conclusions on this particular topic.]
I wanted to take an opportunity to simultaneously comment on an article written last week which tried to cast doubt on the Deflategate research, while also playing a game of “fact or fiction” because there is a lot of incorrect information floating around regarding my analysis.
First, the key articles in this series (from most recent to oldest):
- The Wells Report Delivered an Innocuous Bombshell that Few Noticed
- The Wells Report May Have Uncovered the Tip of the Iceberg
- The New England Patriots Became Nearly Fumble-Proof in 2007
- The New England Patriots Prevention of Fumbles is Nearly Impossible
Point #1 - Something occurred between 2006 and 2007 which allowed the Patriots to fumble the ball at an extremely low rate moving forward when compared to the rate their team fumbled from 2000-06 (Bill Belichick started coaching the team in 2000).
Point #2 - Whatever occurred caused the Patriots to shift from a team who fumbled the football the league average (in 2000-2006) to a team who was so superior when compared any other team the odds it was a mere coincidence are extremely unlikely.
With that said, let’s move on to the “fact or fiction” where we’ll address prior comments made on the article as well as those raised by a new article from Bleacher Report, written by Mike Tanier.
Fact or Fiction: The best way to debunk my analysis is to use a period other than the one I chose, which illuminates and demonstrates the controversy.
This is obviously fiction. The data clearly shows “something” happened in 2007 which vaulted the Patriots from an average team at preventing fumbles to one who is insanely great. Any analysis which looks only at the last couple of years is intentionally trying to limit the results for, as I will explain below, suspicious reasons.This is exactly what the Bleacher Report article attempts to do. The author, for the entirety of his analysis, uses a 3 year period from 2012-14. This clearly prevents him from disproving either of my Two Primary Conclusions. He is not comparing the early Brady/Belichick years (2000-2006) with more recent ones (2007+), so he can’t disprove Point #1, nor can he show that the Patriots are not superior to the rest of the league since 2007, because his sample is only 3 years. So he can’t disprove Point #2.
Its quite strange, because when introducing my article, he notes: “[Warren Sharp's analysis] found that the Patriots have been avoiding fumbles at a mathematically improbable rate since 2007.” He then said his goal was to “neither to implicate nor exonerate Brady” but “it was important for me to find out whether or not there’s any real evidence.” So instead of seeing if “there’s any real evidence” of this “mathematically improbable” fumble rate since 2007, he decided to look only at 2012-14?
There are two key graphics to look
at
which I created back in January (the first has been updated with red
notations per the Wells Report). The first is the summary graphic
showing where the Patriots ranked in the early period and where they
ranked following 2006. It’s obvious that “something” happened to cause
this shift.But the next graphic is the key graphic which casts doubt on the objectivity of the Bleacher Report article.
This graphic depicts the huge noticeable shift of the Patriots fumble rate starting in the 2007 season, and is another of my early graphics. It’s really unmistakeable, how they are league average thru 2006. Then they shift to completely unlike the league average starting promptly in 2007.
While the key is the change which occurred in 2007, with regard to the Bleacher Report period, I want to turn your attention to 2013. This happens to be the middle year of the 3 year period grabbed from the Bleacher Report article. As you can see, the Patriots fumble rate was actually WORSE than league average in this season. Why?
It’s entirely because of one, single week 12 game vs Denver. This Sunday night game was played in insane low temps and bitter wind chills. As I’ll discuss below, inclement weather, and in particular, cold (or brutal cold) significantly increases fumbles. In this game, the Patriots fumbled 6 times. Denver had 5 in the game as well (to show you how bad the weather was at causing fumbles). That’s how cold it was. It was 22 degrees pre-wind chill, 6 degrees with the wind of 22 MPH on average. Those 6 fumbles were a huge percentage of the Patriots total fumbles on the season.
Note that the only game which saw more than 11 fumbles (which New England and Denver combined for in that 2013 game) the last 25 years was a game in December, outdoors in Milwaukee County Stadium, when Green Bay and Detroit combined for 12 fumbles. That game was played in temps of 26, with a wind chill of 5 degrees via 14 MPH winds. Five degrees is almost identical to the 6 degree wind chill temp for the 2013 game in New England which saw the Patriots fumble 6 times.
Clearly, the 3 year sample size is suspicious when sandwiched around the only season where the Patriots were below league average in fumble rate (entirely due to that one, 6-fumble game). However, of my selection of a huge 8 year period from 2007-14, the author said it was “customizing a data set to make [the] change look significant”. I believe the opposite to be the case.
Possibly lost within the Bleacher Report article is the statement: “The Patriots’ fumble rates are low, though not extremely or historically low, over the last three years. Factor in the 2010 and 2011 seasons, and the rates get even lower.” This is obvious. And guess what? Factor in 2007-2009 and the rates don’t just “get even lower”, they begin to become improbable to have occurred by chance. The frustrating part of these counter-arguments is the data is there and easily accessible. The authors of the counter-arguments realize if they were to take a 2007-14 period, and then compare it to 2000-06 it would show exactly what I’ve found in my Two Primary Conclusions.
The bottom line with this point is in order to discredit or disprove my Two Primary Conclusions, you have to look at the same time frame and, based on the stats, conclude something different from what I conclude. Looking only at 2012-14 is not going to disprove my argument based on the stats which shows starting in 2007, something happened, and moved the Patriots from average to other-worldly at fumble prevention.
I will jump
ahead briefly to a graphic from Benjamin Morris of FiveThirtyEight who
used a different time period. His was from 2010-14, and even in that
time period, using binomial and Poisson models, he found that the odds
of the Patriots fumble rate during that period occurring by chance was
over 1 in 10,000. The next closest team was 1 in ~800 and most teams saw
completely normal fumble rates, which occur naturally.Fact or Fiction: Dome teams should be eliminated from the study.
The great part of this one is, after actually looking at the data, you’ll see why it makes total sense to remove them, but even if you include them, it doesn’t make much difference – the Patriots still are at the far end of the spectrum.I spent inordinate time examining climate affects on fumble rates. I went into great detail on the subject as part of “Argument #3″ of this article I wrote in February. Please read it before continuing. If you simply look at the weather analysis, pulled from one done by Sporting Charts, it is quite apparent that temperature has a measurable effect on fumble rate. Keep in mind, you can’t look at “teams” only, you have to look at game sites and the stats for both teams playing in that particular site. Dome teams infrequently play in cold weather. It’s just a fact.
Look
at the Falcons, for example. I looked at the coldest 6 weeks of the
season, from mid November thru the end of the regular season. And since
2007, the Falcons have played in a TOTAL of 4 games outdoors in the
Northeast! In 8 years! Meanwhile, the Patriots have played 40 games
outdoors in that same sample. Knowing cold weather is more likely to
cause fumbles, why would we group the Falcons in with the Patriots, when
they play 10 times fewer games in the conditions most likely to cause fumbles?The graphics to the right make it pretty easy to comprehend why its wrong to consider these teams “apples to apples” and compare them side by side. Its pretty obvious that studying those dome teams differently should be done. And that’s exactly what I did.
But, even if you include dome teams, the Patriots are still surprisingly way different than the rest of the NFL. But don’t take my word for it:
Respected websites with long standing records for being committed to understanding data and football have agreed with my findings, as I’ll mention below. Both included dome teams, and their conclusions were the same.
Fact or Fiction: This entire fumble analysis has been debunked before and is not accurate.
Absolute fiction. First, not one analysis (Bleacher Report included) refuted either of my Two Primary Conclusions, as indicated at the very top. Every article that’s been critical of the big picture fumble rate has tried to discredit it by using one of the following tactics:- Looking at smaller, different time periods
- Looking at smaller data sets, like position by position rates, and failing to sum everything back at the end
- Critiquing early, less refined studies I did which initially looked at player fumble rates when playing on different teams, for instance.
Brian Burke of Advanced Football Analytics studied my results after thinking that they were “so extraordinary they seemed unlikely to be true” but after he ran his own numbers, his conclusions as to the Patriots incredible ball security (aka lack of fumbling) was incredible, and their numbers are “better than the next best team by 20 plays per fumble.” And he included dome teams.
Benjamin Morris of FiveThirtyEight also studied my results and then ran his own. He concluded my study should be taken “more seriously” because “that author correctly identified that the Patriots fumble rate has been absurdly small. I did my own calculations using binomial and Poisson models and found the same.” He also included dome teams.
His results suggest the odds the Patriots could fumble as infrequently as they did was over 1 in 10,000, whereas the next best team in the NFL was 1 in 800, and most teams were below 1 in 5, meaning most teams had fumble rates which were normal. Morris went on to conclude:
“the existence of the Patriots’ extremely low fumble rate, as a Bayesian matter, makes it much more likely that the Patriots were intentionally cheating… and more likely that the Patriots have materially benefited from their cheating.”
Fact or Fiction: Studying QB sack rates show the Patriots had no edge in fumble rate, they just don’t get sacked often.
This is what the Bleacher Report article attempts to use as a secondary argument, but if the author expanded the study to the same time period I used, the results support the fact that the 2000-06 period is very different than the 2007-14 period. The author says that because the r-squared is 0.144, sack rates and fumbles are very correlated. I think that conclusion is debatable, but to bypass argument, lets agree and say sack rates and fumbles are quite correlated league wide. And I’d expect for the same QB on the same team, that correlation is even higher. In other words, over a sample of several seasons, we shouldn’t see a huge change for a single QB on a single team when compared to a sample from several years prior. Tom Brady’s fumble rate per sack should be fairly consistent on a year-to-year basis on the Patriots. While sacks vary, his personal fumbles per sack should not vary much, and should be driven by the total times he was sacked.But that’s not what happened for Tom Brady and the Patriots. And I’m going to keep this brief because digging in the weeds of individual players isn’t needed to prove either of my Two Primary Conclusions, but I’ll say:
- from 2007-14, Brady was sacked 182 times and fumbled 36 times.
- from 2001-06, Brady was sacked 182 times and fumbled 59 times.
- His fumble rate of 1 fumble every 5.1 sacks in the later period was 64% better than his rate of 1 fumble every 3.1 sacks in the earlier period.
- from 2007-14, Brady did not have ANY season in which he fumbled at a rate worse than 1 fumble every 3.5 sacks.
- from 2001-06, in 4 of the 6 seasons, Brady fumbled worse than 1 fumble every 3.5 sacks.
As an aside, the Bleacher Report article criticized my use of “plays per fumble”, rather than using “fumbles per play”. It makes absolutely no difference when comparing to the rest of the NFL. For instance, using “plays per fumble” the Patriots fumbled once approx every 73 plays from 2007-14, which is 74% more than once every 42 plays, their avg from 2000-06. Or, like the author did, you can convert to fumbles per 100 plays, and say the Patriots improved from 2.38 fumbles per 100 plays (in 00-06) to 1.37 fumbles per 100 plays (in 07-14). The rate of improvement is still 74%.
Leaving that fact as it stands, it is also interesting that while criticizing my use of “plays per fumble”, he chooses to conduct his analysis on sacks using “sacks per fumble”. I would have anticipated that he use “fumbles per X sacks” given his seemingly hardline approach to using “per play” stats, but the fact is, it’s irrelevant how you do it as long as you compare the teams similarly.
Fact or Fiction: My analysis looks only at fumbles lost.
Fiction. It looks at all fumbles, regardless of whether a team recovers its own fumble or its opponent recovers it.The Bottom Line
With or without dome teams, with or without fumbles/play or plays/fumble, something changed in 2007 to propel the Patriots toward the best fumble rate in the NFL, one which became “extraordinary” and “absurdly small”. Those are words from other professionals who studied the data (independently) after I conducted my analysis and presented my findings, which was in January of 2015.The Wells report shed light on Jim McNally, who referred to himself as “the deflator” and who started working as the “Officials Locker Room attendant” in 2007. That year is the year after Tom Brady helped get the rule changed, and as you’ve read above, 2007 coincides with the massive change in fumble rates for the Patriots.
The report concluded the Patriots personnel participated in violations of the Playing Rules and were involved in a deliberate effort to circumvent the rules, and Tom Brady was at least generally aware of the inappropriate activities of McNally and Jastremski involving the release of air from Patriots game balls.
Many of the highly suspicious text messages exchanged between McNally and Jastremski occurred throughout the 2014 season, not just immediately prior to and after the Colts game. While the Patriots report attempted to explain that McNally was just an innocent, portly gentleman with a sense of humor who was trying to lose weight, the truth might never be fully known now that Patriots owner Robert Kraft accepted the punishment without appealing.
Clearly, the data concurs with the notion that there was suspicious activity occurring as a result of McNally’s involvement, and it would be fascinating to see text messages dating back to the spring and summer of 2007, which is the start of the period in question. But we will never see them, so its pointless to hypothesize further. From day one, I said the data shows something was abnormal in New England with regard to their fumble rate, which would be a correlated by-product of deflated footballs. I’ve also said there is no way to prove anything with the data. But the Wells Report gave us indications that systemic ball deflation could have been the source. To this day, I still have not seen any stats which disprove either of the Two Primary Conclusions from my analysis.
[I wanted to add that I like and respect Mike Tanier's football knowledge that he has shared for years on Football Outsiders. He has a great football mind, and I've agreed with a lot of his opinions over the years. I just happen to disagree with his response to my research and conclusions on this particular topic.]
No comments:
Post a Comment