Football and Sample Size

Football is starting to grate on my nerves for a few reasons. The ongoing concussion problems and the way it destroys player's bodies is one of them, but another has to do with one of the fundamentals of football itself: Sample size.

In statistical studies, results can be skewed by selecting a sample size that's too small. Let's use Skittles as an example.

In a typical bag of Skittles, there are five flavors: Strawberry, orange, lemon, lime and grape. They are in equal proportion to each other when they're made.

If I grab five Skittles at random, will I have five Skittles each of a different color? Maybe not. I might have two grape, one strawberry, one lime and one lemon. I might have three orange, one grape and one lemon. I might have four lime and one strawberry.

In this case, bad statistical analysis would be to take that miniscule sample size and conclude that the entire batch of Skittles is skewed in that direction. It would be to say, "I only have four lime and one strawberry in this group of five. That means that 80% of all Skittles are lime, 20% are strawberry, and there are no other flavors in this bag."

In football, there are only 16 games played during the year. Is that enough time to determine which are the best teams? How much statistical variance is there that we're not seeing?

We can't really figure that out using football itself. From year to year, the teams change so much that trying to extrapolate long-term data using several seasons worth of games would be absolutely ridiculous and pointless. However, we could try figuring this out using another sport: Baseball.

Every Major League Baseball team plays 162 games per year, which means that statistical anomalies are shaken out by the end of the year. A team that has a strong start with weak personnel will fade down the stretch, and a team that has a weak start but a strong team will come on toward the end.

But what happens when we shorten the season to 16 games, like football has?

First, click here to see 2011's final standings. We remember how the season shook out: The Cardinals and the Rangers met in the postseason and the Cardinals won. However, if we shorten the season to just 16 games, the same as football does, what sort of records do we have?

Since teams play at different times, we rolled back the clock to April 18th, 2011 to see what each team's record was, then adjusted the date to make sure we got the proper amount of games for each team. If the date was adjusted, we noted the adjusted date next to the team name. We are not adjusting the standings by division tiebreakers, just by percentage.

American League

EAST

NY Yankees (4/20) 10-6 .625
Toronto 7-9 .438
Tampa Bay 7-9 .438
Baltimore (4/19) 7-9 .438
Boston (4/19) 5-11 .313

CENTRAL

Cleveland 12-4 .750
Kansas City 10-6 .625
Detroit (4/17) 7-9 .438
Chicago Sox 7-9 .438
Minnesota 6-10 .375

WEST
Texas 11-5 .688
LA Angels 10-6 .625
Oakland 8-8 .500
Seattle (4/17) 5-11 .313

National League

EAST
Philadelphia (4/19) 10-6 .625
Florida (4/20) 10-6 .625
Washington (4/20)* 8-8 .533
Atlanta (4/17) 7-9 .438
NY Mets 5-11 .313

CENTRAL
Cincinnati 9-7 .563
Milwaukee 8-8 .500
Chicago Cubs 8-8 .500
Pittsburgh 8-8 .500
St. Louis 8-8 .500
Houston 5-11 .313

WEST
Colorado 12-4 .750
San Francisco 9-7 .563
Arizona 8-8 .500
LA Dodgers 7-9 .438
San Diego 7-9 .438

* Doubleheader played on 4/20.

According to the 16-game standings, four teams would be tied for the worst record in the league: Boston, Seattle, the Mets and Houston. The actual lowest records, in descending order, were Baltimore, Seattle, Minnesota and Houston.

According to the 16-game standings, there would be three teams with the highest records in the league: Cleveland, Colorado and Texas with five other teams tied at a 10-6 record (Yankees, Kansas City, Anaheim, Philadelpha and Florida). The actual top four teams with the highest records, in descending order, were Philadelphia, the Yankees, Texas and Milwaukee.

By using the 16-game schedule, your division champions in the AL would be the Yankees, Cleveland and Texas with the Royals and Angels battling it out for the wild card. The NL would be Philadelpha, Cincinnati and Colorado with Florida getting the Wild Card.

The real division champions? The Yankees, Detroit and Texas with Tampa getting the wild card in the AL, and Philadelphia, Milwaukee and Arizona with St. Louis in the wild card in the NL.

So, is the 2011 16-game schedule way, way off? There is quite a bit of variance. While a few of the really good teams would still be good under this drastically shorted schedule, over half of the true playoff teams would have been excluded under the 16-game schedule.

How significant is the variance? On average, in 2011, a 16-game schedule would have produced a variance of 93.56 percentage points. That's an entire win and a half in football terms. That may not sound like much, but one or two wins would have changed the entire playoff picture. The eventual Super Bowl winners, the Giants, may have missed the playoffs altogether. The Broncos may not have made the playoffs, the Titans may have, the Cowboys, Eagles, Bears or Cardinals may have made it while the Falcons and Lions may not have, and so on.

But we don't want to make the same statistical mistake by cherry picking one season, and calling this a wrap. Let's go to the 2010 MLB season and do the same thing. The true final standings are right here, and here are the 16-game standings (Taken from April 23, 2010 unless otherwise noted):

American League

EAST
Tampa Bay (4/22) 12-4 .750
NY Yankees 11-5 .688
Toronto (4/22) 9-7 .563
Boston (4/22) 6-10 .375
Baltimore (4/22) 2-14 .125

CENTRAL
Minnesota (4/22) 11-5 .688
Detroit (4/22) 9-7 .529
Cleveland 7-9 .438
Kansas City 6-10 .375
Chicago Sox (4/22) 5-11 .313

WEST
Oakland (4/21) 9-7 .563
Seattle (4/22) 9-7 .563
LA Angels (4/21) 8-8 .500
Texas 7-9 .438

National League

EAST
Philadelphia 10-6 .625
Florida 9-7 .563
Washington (4/22) 8-8 .500
Atlanta 8-8 .500
NY Mets (4/22) 7-9 .438

CENTRAL
St. Louis 10-6 .625
Milwaukee 8-8 .500
Pittsburgh 7-9 .438
Chicago Cubs (4/22) 6-10 .438
Cincinnati (4/22) 7-9 .438
Houston 6-10 .375

WEST
San Diego 10-6 .625
San Francisco 9-7 .563
Colorado 8-8 .500
Arizona 7-9 .438
LA Dodgers 7-9 .438

There's not as much variance in the top teams here. The top four teams in the real schedule are Phildelphia, Tampa Bay, The Yankees, and Minnesota. The top four teams in the 16-game schedule are Tampa Bay, the Yankees, Minnesota and a logjam between three different teams, including Philadelphia.

The bottom teams in the real schedule are Pittsburgh, Seattle, Baltimore and Arizona. In the 16-game schedule, the bottom teams are Baltimore, the White Sox, Houston, and a tie with Kansas City and Boston.

The division champions are slightly different here. The real division champions in the AL were Tampa, Minnesota and Texas with the Yankees getting the wild card. The NL was represented by Philadelpha, San Francisco and Cincinnati, with Atlanta getting the wild card.

The 16-game schedule has a minimal effect on the AL, just changing out Texas for Oakland. Just like 2011, though, the effect is more pronounced in the NL. The NL would have been represented by Philly, St. Louis, San Diego, and the wild card going to either Florida or San Francisco.

So, what's the big deal? The results were mostly the same with just a few quirks. Doesn't sound like much, but what's the variance? Try 81.73 percentage points per team, which equates to one win again.

What about 2009? Here are the final standings, and here's the 16-game schedule.

American League

EAST
Toronto 11-5 .688
Boston 10-6 .625
NY Yankees 9-7 .563
Baltimore 8-8 .500
Tampa Bay 8-8 .500

CENTRAL
Kansas City 9-7 .563
Chicago Sox 8-8 .500
Detroit 8-8 .500
Minnesota 7-9 .438
Cleveland 6-10 .375

WEST
Seattle 10-6 .625
Texas 7-9 .438
LA Angels 6-10 .375
Oakland 6-10 .375

National League

EAST
Florida 11-5 .688
Atlanta 8-8 .500
Philadelphia 8-8 .500
NY Mets 7-9 .438
Washington 3-13 .188

CENTRAL

St. Louis 11-5 .688
Pittsburgh 9-7 .563
Cincinnati 9-7 .563
Chicago Cubs 8-8    .500
Milwaukee 7-9 .438
Houston 6-10 .375

WEST
LA Dodgers 11-5 .688
San Diego 10-6 .625
San Francisco 8-8 .467
Arizona 6-10 .375
Colorado 5-11 .313

What’s the variance? 98.53 percentage points. That’s a win, once again.

Let's try 2008. Here's the 16-game schedule, and compare this with the real standings.

American League

EAST
Boston 9-7 .563
Baltimore 9-7 .563
NY Yankees 9-7 .563
Toronto 8-8 .500
Tampa Bay 7-9 .438

CENTRAL
Chicago Sox 10-6 .625
Kansas City 9-7 .563
Minnesota 7-9 .438
Cleveland 6-10 .375
Detroit 5-11 .313

WEST
LA Angels 9-7 .563
Oakland 9-7 .563
Seattle 8-8 .500
Texas 7-9 .438

National League

EAST
NY Mets 10-6 .625
Florida 9-7 .563
Philadelphia 8-8 .500
Atlanta 7-9 .438
Washington 4-12 .250

CENTRAL
St. Louis 11-5 .688
Milwaukee 10-6 .625
Chicago Cubs 10-6 .625
Pittsburgh 7-9 .438
Cincinnati 7-9 .438
Houston 6-10 .375

WEST
Arizona 12-4 .750
San Diego 8-8 .500
Colorado 8-8 .467
LA Dodgers 7-9 .438
San Francisco 6-10 .375

The variance here is 83.9 percentage points. Again, a full win.

We'll stop here, because I think we get the point: Every year, because of football's short schedule, there's a possibility that a team either has one MORE or one LESS win than they deserve.

So why does all of this matter? Football isn't the same as baseball after all, and who cares if your team has one more win or one more loss?

Well, if you're having a historically great season, it might not matter. If your team goes 15-1 or 14-2, that doesn't change the fact that it was a great season. If your team is historically bad, it doesn't really matter. What's the difference between 4-12 or 3-13?

However, it does matter if you're a fan of any number of teams that go 7-9, 8-8, 9-7 or 10-6 during the season. A little bit of good luck, and your team is in the playoffs with a punter's chance at a Super Bowl. A little bit of bad luck and your team misses the playoffs and maybe your coach who really wasn't that bad gets fired.

Maybe your team is better than the record shows. Maybe they just need some time to gel and work together, but over the span of just 16 games, they don't have the opportunity.

"So are you suggesting that they lengthen the season?" Absolutely not. How can you, when you have athletes whose bodies are destroyed, like Jim McMahon:

McMahon now says his brain doesn’t work well.

“Short term memory is not good,” McMahon said. “I won’t remember a hell of a lot about this interview in 10 minutes.”

Take Kyle Turley, who said:

You start on your own five-yard line, and drive all the way down the field—fifteen, eighteen plays in a row sometimes. Every play: collision, collision, collision. By the time you get to the other end of the field, you're seeing spots. You feel like you are going to black out. Literally, these white explosions—boom, boom, boom—lights getting dimmer and brighter, dimmer and brighter.

What about poor Dave Duerson, who shot himself in the chest so he could have his brain examined? No, you can't lengthen the season. That's the nature of football.

Therein lies the problem. You can't get the good teams to shake out statistically when the way to do is to lengthen the season, which you can't do for player safety, and if the best teams aren't in the playoffs, then what's the point?

You may argue, “But the best team doesn’t win the Super Bowl every year! I mean, some of these years, the wild card team won, or the supposed ‘best’ team lost in the first round of the playoffs!” This is true, but once again, we’re dealing with a very small sample size. Even in baseball, during a seven-game series, almost every team has a chance versus any other team.

This study is not concerned with who wins in the playoffs, because that's prone to variance. When a team gets to the playoffs, anything can happen, and countless pro football teams haven't been able to get to the playoffs precisely because there's an underlying problem with the game itself that can't be fixed under any circumstances.