Random Noise: How is the Puck Prediction Model Doing at the Quarter-Season Mark?

Last summer, I had the idea of conducting a kind of prospective trial of NHL game prediction. So I pulled together game logs from the past eight seasons, and mined information on shot differential, event rate, and shooting and save percentages to develop an algorithm for predicting games. In order to provide a good comparison, I decided to track “the wisdom of the crowd” via the gambling markets as well. And, if you’ve been checking in regularly, you’ve probably seen my daily predictions alongside those of gamblers (via the Odds Shark website). Well, we’re about a quarter of the way through the 2013-14 season, so it seems like as good a time as any to take a closer look at how the model is performing.

Photo credit: Flickr user Hakan Dahlstrom. Use of this image does not imply endorsement.

I pulled every game from my spreadsheet between opening night and November 19, 2013; after this date, each team had played at least 20 games. This left a total sample of 321 games. The table below shows how well the model performed against Vegas overall, and in a variety of scenarios, over those games.

quarterseason_tb1

A few points here:

  • I’ll answer the obvious question right away: a t-test on the accuracy of the Puck Prediction model versus Las Vegas does not suggest a significant difference (p=0.42).¬†Overall, the model was only 41% accurate when picking against Vegas.
  • I wondered initially how much overtime and shootout games (which are as close to a coin-flip as you get in NHL games) would matter in determining the effectiveness of the model. The answer: a little bit. Limiting the analysis to games decided in regulation improves Puck Prediction’s relative performance a little bit, but not enormously. When picking against Vegas, the model was only 43% accurate in regulation games.
  • Picking road teams hurt the model a lot more than picking home teams did.
  • I can’t say I expected greater accuracy picking against the team with a higher save percentage. My guess is this is one of those “fooled by randomness” things.

I also looked at the relative performance of the model by teams, to see whether certain teams gave me more trouble than others. These results are depicted in the table below.

quarterseason_tb2

Some things to note:

  • On the plus side, it makes sense that I have so many Devils fans following me on Twitter, because Puck Prediction is better than Vegas at picking their games. Other teams I’ve done well with: Philadelphia, the Rangers, and the Jets. Also, it’s good to know that I know enough about my local team to pick their games effectively.
  • Puck Prediction and Vegas picked games involving Buffalo, Carolina, St. Louis, and Ottawa equally well. Either it’s easy to pick for or against these guys, or no one knows what to expect from their games.
  • In case you were surprised that my first Playoff Forecast picked the Wild to win the President’s Trophy, wonder no more: Puck Prediction did a terrible job of calling their games. Other teams I struggled to understand: Boston, Chicago and the Islanders.
  • I’m into statistics, so it probably makes sense that my model sucks when it comes to predicting the Maple Leafs and the Capitals. One possible explanation: in smaller samples, the model may underappreciate penalty-killing.
  • Apparently no one knows what’s going on with the Red Wings.

About Nick Emptage

Nicholas Emptage is the blogger behind puckprediction.com. He is an economist by trade and a Sharks fan by choice.
This entry was posted in Original Analysis, Predictions and tagged , , , . Bookmark the permalink.

One Response to Random Noise: How is the Puck Prediction Model Doing at the Quarter-Season Mark?

  1. Pingback: Down the Stretch: How is the Puck Prediction Model After Three-Quarters of the 2013-14 Season? | Puck Prediction

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s