Hidden near the bottom of my most recent post, I mentioned I was going to enter some Football data competitions over the next few months, and this is part one of (???) posts on my entry in the CollegeFootballData.com prediction contest. You can find my entry there under u/wahoo08, and I am super excited to see how I do.
The Competition
Before we get into the nuts and bolts of what is going on in the model we should probably talk about what exactly the model is trying to predict for, ie.) the rules of the prediction contest.
The competition scores your model based on four categories:
Straight Up Accuracy - Do you pick the winners correctly?
Against the Spread (ATS) Accuracy - Did you beat the Vegas closing line?
Mean Absolute Error (MAE) - How far off were your predicted scores on average
Mean Squared Error (MSE) - The same, but the bigger the miss, the more you are penalized
Every week, each modeler must predict at least 70% of the games on the docket. The modeler then receives a score for those predictions in a composite ranking, which averages their performance across four ranking categories. There are individual winners each week, and an overall winner after bowl season (which terrifies me, but I will get to that at a later point).
Though the competition becomes more difficult because your score relies on your over-expectation numbers: your performance minus the score the Vegas closing line would have assigned to the same games the model selects. This means that if you pick easy results correctly, you do not get nearly the level of credit as picking more uncertain games correctly.
Add all of that up, and the model needs to be very well-rounded in order to win. There was someone last year who hit 56.1% ATS, a number most handicappers would be very happy with. That's making nearly $300 if every bet was only $5, but they finished sixth overall.
Nobody has beaten Vegas yet, but here is to hoping we can get there this year!
Building the Base of the Model
This section is going to be decently technical, so I will try to keep it as understandable as possible throughout, but just know that going in. Essentially, if you can kind of understand what I am getting at from the name of the model, you should be able to read this easily. If not, don't be scared. I will try my best to make sure you can follow along!
WISHBONE — Walk-forward Inference on Schedule Hierarchies with Boosted Opponent-Normalized Ensembles
Gobbledygook right? Each part will make sense in time.
The first major decision was what to use as the basis of the model: do we build our own or do we make it market-implied? Market-implied means we use Vegas' lines and history as a major factor in our training set, and while many previous competitors have used this methodology to their advantage, I wanted to build it out without the numbers at first. The main reason I did that is because I wanted the model to learn without relying so heavily on the lines as an input. If we could discover a feature that helped our predictive power and blend that into the end model at a higher influence level, then that could give us the edge in this competition with the novel insight.
Before I go further, a big shout out to Michael Caley of The Double Pivot Podcast and the Expecting Goals newsletter, whose decisions on his PADDLIN' model informed many of my own.
Michael Caley
The first layer of the model was a ridge-regression that treats every game as an equation on the schedule graph (home margin = home rating - away rating + home field advantage + noise) and solves the problem. Added on top of that model were a few things, a simple hierarchical conference effect allowed us to compare teams who rarely meet, or have few games in common, exponential season decay (meaning that older season were less relevant than newer ones) and a near universal home field advantage. We trained this model on data from 2007 up to the test years of 2023-2025.
That "backbone" model was supposed to be simple, and it was. This put our initial MAE at 14.1 against 12.5 for Vegas over the same test period, and an underwhelming coin flip worthy 50.4% ATS. We needed to build on top of this model and make this something worth entering the competition.
The Feature Set
The second layer of the model was a gradient-boosted model (LightGBM) that predicts the residual. Also known as everything the linear backbone missed: efficiency stats, priors, opponent-adjustments, and situational features, amongst others. The training of this model utilized the more feature-rich dataset available from 2014 up to the test years of 2023-2025.
Which takes us to the first interesting findings of the model, specifically those around home field advantage. First, as has been observed across many sports, the value of HFA is falling, starting at around 2.55 to 2.16 points. Not only that, but mythologized home field advantages like Death Valley or The Big House do not show any statistically significant extra advantage compared to other venues. In fact, the two strongest home fields in the data set were FAU and Middle Tennessee State. So the model settled upon a universal HFA.
However, different home field effects proved to be significant. Travel and rest are very real effects, but it does not appear that Vegas currently has them mis-priced, meaning our model cannot one up its baseline competitor there. We all know that altitude is a major factor. The naïve model consistently missed by +2.3 points when visitors climbed over 1,000+ meters (3280 ft), while the LightGBM model did not.
Outside of that, most of the features that were most important were to be expected. Opponent adjusted efficiency using EPA and success rate with garbage time filtered out, scheme, and surprisingly enough, preseason priors showed to help as well, amongst other data points (SP+, FPI, etc.).
The results of this improved the model drastically. We went from a coin flip against the spread to a better, yet still unprofitable 51.8% ATS, and lowered our Vegas MAE gap from 1.6 points down to 0.75 points. There was still work to do, but this was looking like something that would not get dead last at least.
Tuning
Now that there were features, we needed to find which blend helped the performance of the model the best. Turn one knob too far and it breaks the model; turn one not far enough, and the model loses predictive power it could have had.
The biggest discovery in this process was how much signal dominance gives. This is a direct lineage to Caley's PADDLIN' model, where dominance, especially when already winning, was a major feature of his World Cup winning model. Initially, we set the cap at 38 points, meaning that a team winning by 38 or by 70 meant the same to the model, but the point we found diminishing returns after was 60. Meaning obliterating your opponents is one of the most important signs of a good team. In fact, if there were no additional tuning to the model past this, the MAE would have gone to 0.96 with just this single addition.
The only other changes made were like Caley's. The model relies heavily on preseason priors and updates much slower than most others out there. Recruiting rankings and returning production matter that much to team success, that devaluing it too quickly hurts a lot. In fact, the model uses the new information gained for only 25% of its total prediction. There were other more technical changes, like deciding trees in the model smaller and more regular, and fixing of timescales so the model does not rely too heavily on dead data later in the year.
Portal Adjustments
The more astute among you may have realized a missing puzzle piece in the data set. The transfer portal. It has changed the way college football functions since its introduction in 2018, but especially since the pandemic. That means at best in my data set I have one or two years of true transfer effects in my data set and am testing against years where those effects are much more pronounced.
This was a really big problem, as my model relies so heavily on returning production, and the portal has destroyed the nature of returning production. There were two main takeaways from this.
The first is that the portal era has hurt the strength of returning production as a predictor. In fact, returning production becomes less predictive each year to roughly the same degree. Once we built that into the model, our performance in the portal era increased.
The second is just how much a returning QB matters. I have always stated that a first-year college quarterback is almost useless, and people should expect nothing from them, and my thinking was correct. The model grades the level of production given from a QB from 0 to 1 and makes an adjuster for a team based on that scale. This applied to transfer QBs too, as their production after transferring showed a correlation with their pre-transfer numbers with a decent level of predictability. To highlight how much this matters, in a little sneak preview of a further piece, the model decreased Texas Tech from a 19.3 point favorite against a dead-average team on a neutral site to a 16.7 point favorite once the Brendan Sorsby transfer was removed.
Let's Go Win!!!
Week 0 begins tomorrow, and so does the competition. If you want to follow my progress, you can find the results here. However, I may provide a weekly recap of the model's results, performance, adjustments, and any other lessons learned during this experience!