HomeUncategorizedThe surprising power of preseason rankings (College Football Edition)
The surprising power of preseason rankings (College Football Edition)
April 20, 2017
Each offseason, a new batch of college football preseason rankings is unveiled, drawing ire from from fans and analysts alike. It’s not difficult to see why. For example, preseason rankings make disastrous errors each year, seemingly without fail (see: 2016 Notre Dame and Michigan State). Furthermore, one could also argue that preseason rankings, in theory (or, even in practice) undermine the validity of in-season rankings–perhaps, by biasing voters’ perceptions of teams.
Do preseason rankings have any value? Actually, yes. Preseason rankings predict in-season performance quite well, and may have a worse reputation than they actually deserve.
Predictive power of in-season performance
Importantly, in order to assess the value of preseason expectations, we must have something to compare it to. Otherwise, preseason rankings may be held to an unfair or different standard than other sources of information. For example, people may strongly criticize preseason rankings when the rankings make disastrous errors. However, it may also be the case that similarly large errors occur when relying on non-preseason sources of information, such as a team’s previous performance. Without directly comparing preseason rankings to other information, we cannot know how “good” or “bad” preseason rankings actually are.
Let’s focus on the predictive power of in-season performance. To start, there are clearly instances when current, in-season performance is not a great predictor of future success. For instance, in 2016, Louisville was one of the most impressive early-season performers (notably, annihilating current #2 Florida State 63-20). However, they ended the last eight games with a 5-3 record, thus underperforming predictions, based on prior performance.
However, exactly how unpredictable is a team’s performance from game-to-game, on average? Let’s take a look.
As a measure of in-season performance, I used ESPN’s Football Power Index (FPI). The FPI provides a 0-to-100 score (Game Score) for a team’s performance in each individual game, and possesses several advantages over more standard measures of performance, such as win-loss or point differential. Namely, it includes factors that better indicate a team’s “true” performance, such as the difficulty of the opponent, and the degree a team was “in control” of the game from start to end (it also fares reasonably well against other computer ranking systems).
Results from the 2016-2017 season indicate that for each team, Game Scores vary by nearly 20 points (19.6) between games, on average (note: only games 1 through 12 were used, in order to make sure each team played the same amount of games). This is a lot! To put this number into context, check out the below table. The below table provides examples of the degree of variation in quality, on average that occurs every time your team plays.
As previously mentioned, preseason expectations may appear to be poor predictors in isolation, but appear less so when directly comparing them to other sources of information. As seen above, in-season performance varies quite strongly, lowering the bar when assessing the value of preseason expectations.
Comparing predictive power of in-season performance to preseason expectations
Let’s now directly compare the predictive accuracy of in-season performance to preseason expectations. To examine preseason expectations, I again utilize ESPN’s FPI, which also provides a preseason ranking system that accounts for a number of factors, including previous season performance, returning starters, and strength of recruiting class.
The below table looks at the predictive accuracy of two sources of information: a team’s average performance in previous games during the season, and preseason expectations for the team.
To better visualize the trends, let’s take a look at a graph.
The results reveal two, main points. First, in-season performance eventually outperforms preseason expectations, but only quite late in the season. Specifically, it takes around seven games for in-season performance to have better predictive accuracy than preseason rankings.
Second, even when in-season performance outpredicts preseason rankings, it doesn’t do so by much. Specifically, the differences in error between preseason and in-season performance when predicting Game Scores are quite small, late in the season. In fact, results show that in-season performance only predicts Game Scores about one point (out of a score of 100) more accurately than preseason rankings. For all of the negative expectations surrounding preseason rankings, this seems quite small.
Contrary to expectations, results suggest that preseason expectations nearly do as good of a job predicting performance as does in-season data. The results also show that preseason data outpredicts in-season data until around game seven or so, at which point in-season data gains a slight edge.
Are preseason rankings underrated? It appears so. But why? One explanation is that observers are prone to devaluing preseason rankings for reasons that have nothing to do with their validity. For example, whereas in-season performance provides actual, “real” data from the season, preseason rankings feel artificial, and based on more secondary sources of information (e.g., “how well did they play lastyear?”). Furthermore, psychological phenomena such as hindsight bias may cause in-season performance to feel more predictive than it actually is. For example, games are rich with information, and observers may, in retrospect, flexibly identify key “signs” from these games that hint at a team’s future success or demise.