One of the hardest things that people have to do is separate process from results. Instinct leads us to think that if we do something that produces a good result, the process that led there must be sound. We expect that repeating that process will lead to further good results.
Unfortunately, that’s not the way that the world works. Most areas of human endeavor have some space in which we can screw things up and get great outcomes or vice versa. In the long run, a bad process tends to get found out but in the short term, we can get fooled.
This is particularly true in today’s NHL. The gap between good teams and bad teams is smaller than it’s ever been. The NHL’s rules, with some games worth two points and some games worth three points, introduces more chance into the standings than there’s ever been. It’s harder than ever to identify good process from results because the league is tighter.
Which brings us to Randy Carlyle and the Maple Leafs’ decision to extend him for two years. Entering the 2013-14 season, there was a consensus amongst stats-friendly hockey fans that there were problems with Carlyle’s process and that the results the 2012-13 Maple Leafs produced were not really representative of the process.
How did people arrive at this conclusion? They looked at the Corsi% achieved by the Ducks under Carlyle and then under Bruce Boudreau. They looked at the Corsi% achieved by the Maple Leafs under Ron Wilson and then under Carlyle. They noted that subtracting Carlyle seemed to make things better and that hiring him seemed to make things worse.
When Boudreau replaced Carlyle in Anaheim, the Ducks Corsi% got about 4.5 percent better. When Carlyle replaced Wilson in Toronto, the Maple Leafs’ Corsi% got about 4.5 percent worse.
While, 4.5 percent may not seem like a lot, it’s actually a massive swing. The average team had 7,070 Corsi events in their games this season, and 53.5 percent of Corsi events became shots on goal. If you assume average shooting percentage (8 percent) and save percentage (.920), a team that adds 4.5 points of Corsi% will improve by 27 goals, which is likely at least nine points in the standings. The same is true in reverse for a team that loses 4.5 points of Corsi%.
Hockey analytics people will tell you that Corsi% can tell you a lot about your process. If you’re posting a strong Corsi% or trending in the right direction, you can be reasonably assured that you’re on the right track, regardless of your results in the short term. If you’re getting badly outshot most nights, even if you’re winning games, you should be deeply skeptical of either your process or your players.
One of the cutting edge questions in hockey analytics is sorting out the extent to which a good or bad Corsi% is tied to the players vs. the tactics. Obviously, different cases will have different answers. Buffalo posted a terrible Corsi% this season but I wouldn’t start by looking at Ted Nolan. In a case like this though, where we have two examples of Carlyle’s presence resulting in a certain Corsi% and the presence of a different coach resulting in a Corsi% that’s much higher, focusing on the coach and, in particular, the tactics he employs, seems wise.
When you break games into discrete pieces, as with this video showing Carlyle’s Maple Leafs and Boudreau’s Ducks following offensive zone faceoff losses, you can see differences in how the teams play. Boudreau’s teams respond to lost offensive zone faceoffs by forechecking. Carlyle’s teams respond by falling back. You can then check those differences against the results and determine which approach produces better results. As it happens, Boudreau’s teams have been far more effective than Carlyle’s teams after an offensive zone faceoff loss.
It seems reasonable to think that Carlyle, Wilson and Boudreau have different philosophies in terms of how to approach certain situations and that the tactics that Carlyle’s philosophies have dictated have led to an inferior Corsi%.
This isn’t to say that Carlyle’s necessarily a bad coach. There’s reason to question the tactics that he’s employed over the past few years, but there’s more to coaching than tactics and he may well have strengths in other areas that assistant coaches with an eye for tactics can complement.
To that end, the Maple Leafs’ decision to replace their assistant coaches could produce the results that the Leafs are hoping for. The former assistants either weren’t able to identify the issues or weren’t able to convince Carlyle to implement change. If new assistants can do so, then Toronto’s coaching problem has been fixed and the Leafs will still enjoy the benefits of Carlyle’s other strengths.
Of course, the real trouble arises if the new assistants aren’t able to convince Carlyle that he needs to change what he does and the Leafs make the playoffs anyway. Were that to happen, the Maple Leafs would fall back into the same purgatory in which they spent the summer of 2013: due to collapse and not sufficiently aware of the problem and its causes.
If that happens, a repeat of the 2013-14 season is pretty much inevitable. At that point, questions about process will likely be asked of people higher up the ladder than Carlyle.