Last summer, I had the idea of conducting a kind of prospective trial of NHL game prediction. So I pulled together game logs from the past eight seasons, and mined information on shot differential, event rate, and shooting and save percentages to develop an algorithm for predicting games. In order to provide a good comparison, I decided to track “the wisdom of the crowd” via the gambling markets as well. And, if you’ve been checking in regularly, you’ve probably seen my daily predictions alongside those of gamblers (via the Odds Shark website). Well, we’re about a quarter of the way through the 2013-14 season, so it seems like as good a time as any to take a closer look at how the model is performing.
I pulled every game from my spreadsheet between opening night and November 19, 2013; after this date, each team had played at least 20 games. This left a total sample of 321 games. The table below shows how well the model performed against Vegas overall, and in a variety of scenarios, over those games.
A few points here:
- I’ll answer the obvious question right away: a t-test on the accuracy of the Puck Prediction model versus Las Vegas does not suggest a significant difference (p=0.42). Overall, the model was only 41% accurate when picking against Vegas.
- I wondered initially how much overtime and shootout games (which are as close to a coin-flip as you get in NHL games) would matter in determining the effectiveness of the model. The answer: a little bit. Limiting the analysis to games decided in regulation improves Puck Prediction’s relative performance a little bit, but not enormously. When picking against Vegas, the model was only 43% accurate in regulation games.
- Picking road teams hurt the model a lot more than picking home teams did.
- I can’t say I expected greater accuracy picking against the team with a higher save percentage. My guess is this is one of those “fooled by randomness” things.
I also looked at the relative performance of the model by teams, to see whether certain teams gave me more trouble than others. These results are depicted in the table below.
Some things to note:
- On the plus side, it makes sense that I have so many Devils fans following me on Twitter, because Puck Prediction is better than Vegas at picking their games. Other teams I’ve done well with: Philadelphia, the Rangers, and the Jets. Also, it’s good to know that I know enough about my local team to pick their games effectively.
- Puck Prediction and Vegas picked games involving Buffalo, Carolina, St. Louis, and Ottawa equally well. Either it’s easy to pick for or against these guys, or no one knows what to expect from their games.
- In case you were surprised that my first Playoff Forecast picked the Wild to win the President’s Trophy, wonder no more: Puck Prediction did a terrible job of calling their games. Other teams I struggled to understand: Boston, Chicago and the Islanders.
- I’m into statistics, so it probably makes sense that my model sucks when it comes to predicting the Maple Leafs and the Capitals. One possible explanation: in smaller samples, the model may underappreciate penalty-killing.
- Apparently no one knows what’s going on with the Red Wings.