Now that the day-to-day grind of the 2013-14 NHL season is over, and attentions are shifting to things like the draft, coaching hires, and free agency, I’ve been thinking a bit about the future of analytics in hockey. Recently, Jonathan Willis published an interesting article over at Edmonton Journal, making the case that data aren’t driving decisions by NHL teams as much as some analysts would like to believe. And certainly, the examples he cites support his point. San Jose’s reputation as a forward-thinking, analytically-minded team has taken on some water recently with the inexplicable signing of Mike Brown to a multi-year deal, the rumored trades of Joe Thornton and Patrick Marleau, and an interesting exchange (via Twitter) between Sharks beat reporter Kevin Kurz and Derek from Fear the Fin suggesting that the team doesn’t realize they have the best shot-creating power play in the NHL. Other examples of this phenomenon include:
- In 2011, Edmonton started up an internal analytics shop to, as David Staples says, “help the Oilers enter in to the new world of sports analytics”, and comments from GM Craig MacTavish and coach Dallas Eakins suggest that both have an appreciation for the value of numbers. Yet, as Willis notes, moves like claiming Luke Gazdic off waivers, playing Justin Schultz in a top-pairing role, and overpaying for mediocre veterans like Andrew Ference and Nikita Nikitin – not to mention the team’s disastrous defensive play and on-ice results – don’t suggest a team making much use of numbers to guide its decisions.
- The Pittsburgh Penguins used the work of the Sports Analytics Institute in making the 2011 decision to trade for star winger James Neal, and, in the words of Director of Player Personnel Dan MacKinnon, “I don’t think we’ve made an impact decision since then without consulting the analytics”. The value of a shot-quality analysis from a model that SAI actually refers to as a “black box” is, of course, open to question, but more importantly, it’s hard to believe the Penguins consulted the analytics when trading two draft picks for one of the league’s worst defensemen, trading a top D prospect to rent a declining winger for a couple months in 2013, signing a badly declining Rob Scuderi to a four-year contract, signing fourth-liner Craig Adams to multiple deals, firing the winningest head coach in franchise history after a 107-point season, or trading Neal to Nashville this week.
- Stats-savvy Toronto fans were thrilled at the suggestion that new team President Brendan Shanahan was willing to embrace possession metrics in charting a direction for the team. For the time being, I’ll reserve judgment, but the decision to retain coach Randy Carlyle and GM Dave Nonis, the faces of the organization’s notorious hostility to numbers, is not a good sign. Nor is the acquisition of Roman Polak from St. Louis.
- The Tampa Bay Lightning employ Michael Peterson, a full-time statistical analyst who assists the front office and coaching staff in making all sorts of decisions. One imagines that Peterson isn’t too happy with the six-year deal the team just gave Ryan Callahan.
- New Flyers GM Ron Hextall reportedly views analytics as a “huge part of what we do going forward”. This sounds great, but it’s a bit at odds with the notion that Andrew MacDonald is “going to be a big part of this team moving forward”, and with the trade of a useful player in Scott Hartnell for a . . . less than useful one in R.J. Umberger.
The point of this isn’t to be discouraging or contrarian, but rather to suggest that the adoption of analytic thinking and results by NHL teams is still in its early stages. It’s certainly fair to say that the profile of this work is higher now than it’s ever been before. Thanks in no small part to the Maple Leafs’ wretched 2013-14 season, as well as the popularity of Moneyball (book or film) and the work of people like James Mirtle, Neil Greenberg, and Sean McIndoe (to name just a few), it’s hard to avoid encountering advanced stats in even mainstream hockey writing, and in some cases (the Kings and Sabres come to mind), local TV broadcasts are bringing possession numbers into their coverage. Willis’s point, which I agree with, is that none of this means that teams are actually leveraging the numbers to make decisions. But we do appear to have reached the point where teams can’t publicly brush the numbers aside the way Nonis and Carlyle did a year ago. Coaches and front-office officials who don’t have at least a vaguely positive answer (like Hextall’s, for example) to questions about their use of analytics risk being painted as behind the curve, or (as we saw in Toronto) having their inattention to statistics become a story on its own.
Still, it appears that much of the NHL’s interactions with the advanced-stats community these days involves using numbers to validate pre-existing notions or observations. While it’s not a bad thing for analysts to be involved in the decision-making calculus of NHL teams, even peripherally, it’s hard to imagine unlocking the potential of analytics in such a limited role. The real value of this work lies in using numbers to challenge popular assumptions about the game, and using those insights to build successful teams in less conventional ways. And obviously, this isn’t going to happen if the numbers are only used (in effect) to justify choices after they’ve been made. Much has been made of the reported hiring of Sunny Mehta, a contributor at Vic Ferrari’s old site, as New Jersey’s Director of Hockey Analytics; while he may be an excellent choice for the position, the real question is the level of influence he’ll have in an organization that Lou Lamoriello has ruled (with something of an iron fist) since the late 1980s. This article from Mirtle suggests that Lamoriello isn’t entirely enthusiastic about embracing the insights the numbers can offer, which is potentially a critical problem.
None of this, of course, is cause for pessimism. At this point, it’s simply a matter of trusting that bigger opportunities for analytics will come with time. It may not happen in the league’s more prominent and wealthy markets, but eventually, teams that are desperate to win will start to look for new ways to gain an edge on the competition. In the meantime, some things analysts can do to keep this process moving forward are as follows:
- Even though data analysis can be conducted without attention to basic principles of sound statistics, remember that it isn’t a good idea. While the proliferation of analysis software and web-based data sources has undoubtedly been a boon to researchers, one downside is that people without a grasp of concepts like statistical power, sampling error, and non-causal correlation can easily crunch numbers and disseminate the work online. But a big reason why decision-makers in NHL teams are reluctant to trust the work of analysts is that they don’t have the training needed to critique statistical work; from their perspective, it’s easier to ignore it than risk making a bad decision based on shoddy analysis. Put another way, analysts need to remember that the burden is on them to prove the value of what they do, and that starts with producing work that’s valid and reliable.
- Don’t give bad advice. Most hockey analysts are fans of the sport as well, and few fans can resist the pull of simplistic narratives or emotional reactions to short-term results (i.e., your favorite team losing in the playoffs). The problem with this is that many of the mistakes of mismanaged teams come down to focusing on short-term results at the expense of seeing a process through. One of the biggest favors the stats community could do for teams would be convincing them that there’s no magical formula to winning in the playoffs, beyond assembling a strong team and hoping the bounces go their way. More generally, tempting as it might be to gravitate towards emotionally satisfying conclusions, and to pick and choose data points to fit them, in the long run this does the cause of hockey analytics no favors.
- Take advantage of the community of analysts around you. This is pretty self-explanatory: it’s a lot easier to do high-quality work, and to generate better ideas, when you can bounce ideas off of others in the field. While different analysts obviously don’t agree on everything, and have different preferences when it comes to methodology, it’s rarely the case that good work is produced in isolation. Fortunately, thanks to Twitter, jumping into the conversations of analysts and getting the feedback you need has never been easier.
- Remember that the future is very bright. Hockey analytics have taken a big step forward this season, and with analysts like Corey Sznajder, Eric Tulsky, and Chris Boyle spearheading efforts to produce new data sources based on tracking events by hand, it’s only going to get better. These data sources, as well as ubiquitous video, will make it increasingly possible to connect patterns in the data to specific plays on the ice; this will allow analytics to become more prescriptive, and to incorporate a new dimension of face validity. It might be a long time before NHL GMs are using hockey stats to run their teams, but there’s never been a better time to be involved in them.