Friday, July 6, 2012

The Value of Subjective Evaluation

I've written many times about how I do not put a high degree of emphasis on subjective evaluations of goaltenders because there are extraneous and often subconscious factors that can impact how a performance is judged. Even on the individual game level, where the task should presumably be at its very easiest, luck and the performance of the goalie's teammates still have a big impact on how they are rated.

TSN analyst and former NHL goaltender Jamie McLennan scored every goaltending performance in the 2012 Stanley Cup playoffs based on a subjective scale from 1-5, which gives an interesting point of comparison between what the numbers show in terms of saves and goals against and what an informed expert concludes from their individual judgment. I do not question McLennan's scouting ability or his knowledge of goaltending, but the data set he has provided has some interesting properties that force me to question how much value his analysis actually adds.
  • In 86 playoff games, McLennan never gave the losing goalie a better score than the winning goalie.  He gave the two the same score only 6 times, meaning that 93% of the time the goalie on the winning team was judged to have played a better game.  Eleven times the two goalies had a save percentage within .005, suggesting there was likely very little difference in their play, yet the winning goalie routinely received a higher degree of recognition.  When you factor in score effects that typically end up inflating the winning goalie's numbers slightly, it is likely that the losing goalie often managed to match or outperform the winner, yet they never got a better score and only rarely were even graded on the same level as their counterpart
  • The correlation between a goalie's average game score and his overall save percentage was 0.924, which is an extremely high degree of correlation
  • Goalies received a 5 every single time they had a save percentage of .970 or better, regardless of how many shots they faced
  • If you compare the save percentage rankings with the game score rankings, only 4 out of the 17 goalies with at least 3 games played in the 2012 playoffs have a ranking differential of more than two (Niemi, Howard, Brodeur and Holtby)
  • Using regression, each goalie's average game score can be predicted from their overall save percentage.  Every goalie's predicted average score was within 0.25 of their actual average game score, with the exception of Jon Quick, who was 0.37 higher, and Antti Niemi, who came in a whopping 0.67 lower
In summary, McLennan's rankings did not seem to add much information.  He really liked Jon Quick, but he was hardly unique in that viewpoint; the Conn Smythe Trophy is proof enough that most observers liked what they saw from the Kings' goalie. McLennan really wasn't a fan of Niemi's playoffs, giving him scores of 2 for games of .906 and .923, both very unusual for his ranking system (there were only two other games total where a goalie got a 2 for a .900+ save percentage outing).  Perhaps we can conclude from this that the numbers flattered the Sharks' netminder a bit relative to his actual performance.  However, it is also at least possible that McLennan has some sort of bias against Niemi, who had strong numbers at even strength (.940) but was ventilated on the penalty kill (.806), a unit that has been a point of serious weakness for San Jose over the past two seasons.

Beyond those two, comparing the save numbers to the game scores indicates that McLennan thought that Marc-Andre Fleury, Cory Schneider, Jimmy Howard and Martin Brodeur were all a bit better than their numbers suggest, while Tim Thomas, Jose Theodore and Corey Crawford were all a bit worse.  That makes sense for Fleury, awful performances do tend to disproportionately drag down a goalie's overall averages, as well as Brodeur who is noted for his non-save skills (although McLennan also openly admitted to giving the veteran a few sympathy marks for his play in the Stanley Cup Finals).

I would subjectively agree that Thomas and Crawford may both have been a bit worse than their numbers suggest, particularly if you factor in situational leverage and their impact on win probability.  A previous post of mine looked at save percentage by game score, and Thomas and Crawford were certainly outplayed by their opposite numbers when the game was close.  How much of that is randomness and how much of that was each goalie's fault is an open question, but I'm not surprised that someone subjectively rating goalies would take their situational performance into account to some extent.  Still, if I turn the analysis around to predict save percentage based on game score, McLennan was subjectively downgrading both goalies by only about .006-.008, not a very significant change at all for a 150-200 shot sample.

Imagine two individuals, one who didn't watch a second of playoff hockey but was armed with detailed stat sheets showing the save efficiency of every goalie, and one who also didn't watch a second of playoff hockey but still somehow faithfully tuned in every night to hear Jamie McLennan's Post2Post segment on TSN.  Which one would be in better position to rate the goaltenders in the 2012 playoffs?  I think it probably would be the McLennan fan, but their advantage would be very slight as the two would still agree on almost everything.

Over and over, analysis tends to show that the subjective factors often hyped by broadcasters and hockey insiders (shot quality, clutch saves, etc.) really do not have that much of an impact at the end of the day.  That's not always the most intuitive finding for us narrative-obsessed sports fans, but it's hard to argue with the evidence.  I think we should be careful to completely dismiss potential factors that over time could have a significant impact, particularly at the career level, but given that nearly every observer seems to vastly overrate the value of goaltending contributions that are not encapsulated in save percentage I think it reinforces the idea that a goalie's save rate should remain the primary and most trusted method of evaluation.  Subjective opinions from people who know what they are talking about should not be disregarded, but they should at least be treated with some level of skepticism and compared with the statistical record to test their validity before given much weighting in the final evaluation.

Thursday, June 28, 2012

Why Couldn't Mats Sundin Score on the Power Play?

Mats Sundin was named to the Hall of Fame in his first year of eligibility, creating some controversy as many thought he was not fully deserving of that honour.  Some observers seemed especially miffed that Sundin was inducted ahead of Brendan Shanahan.

I don't really see the injustice there to be honest as I think Sundin is fully deserving of the Hall.  If you gave me the choice of Sundin or Shanahan I would have taken Sundin at virtually any point during their careers.  Sundin's record of consistent production is pretty strong, and I don't particularly care about his lack of team success or failure to win any major trophies.  His international scoring record, in particular, is fantastic (18 goals, 21 assists for 39 points in 30 games played in best-on-best tournaments, plus 18-26-44 in 35 games in his world championships career).

Many have wondered why Sundin was unable to duplicate the same excellent results in the NHL, particularly in the playoffs. North American observers are sometimes quick to resort to the traditional explanation that European players are more motivated to perform well in international tournaments than in Stanley Cup postseason games, but I'm
pretty skeptical of that being a major factor. I think a better explanation is that Mats Sundin was one of the best 5 on 5 forwards in the world and was playing with better linemates for Tre Kronor than he was in Toronto.

Sundin has a very impressive record of even strength scoring. From 1996-97 to 2001-02, he finished in the top 20 in the league in even strength scoring in six consecutive seasons while amassing 338 points at even strength over that span, the second-best total in the league behind only Jaromir Jagr. When you take into account Sundin's usual lack of top linemates and coach Pat Quinn's favoured strategy of rolling four lines that had the byproduct of reducing the available ice time for his #1 centre relative to other stars around the league, Sundin's scoring rates are even more impressive.

Here are the even strength and power play scoring rates during the regular season and playoffs for 8 of the top centers in the league from 1997-98 to 2003-04 (plus Brendan Shanahan, given all the recent discussion of whether he was more deserving than Sundin):


Sundin ranks second to only Forsberg in both the regular season and the playoffs in his rate of even strength point production. Sundin also had the best even strength goalscoring rate in the regular season as well as the fourth-best pace in the playoffs.  
However, the former Leaf captain ranks dead last in power play scoring in the regular season, and only Lindros (who had all of 7 playoff games played during this period) ranks below him in playoff power play scoring.

I checked the participation rates for the centers (percentage of team goals while a player was on the ice on which they recorded either a goal or an assist) to see if there were any major discrepancies:

Forsberg: 86.4% EV, 69.8% PP
Sundin: 83.9% EV, 64.4% PP
Sakic: 87.1% EV, 66.1% PP
Lindros: 81.5% EV, 64.4% PP
Turgeon: 85.7% EV, 70.0% PP
Modano: 83.8% EV, 64.7% PP
Yashin: 83.3% EV, 74.7% PP
Yzerman: 78.5% EV, 59.3% PP

Sundin's rates are pretty typical in both game situations. His power play rate is slightly below the group average, but is identical to that of Lindros and very close to Modano's.  Yzerman's PP number is interesting, given that it is much lower than the others.  To add to that, Brendan Shanahan's participation rate in the same unit was 61.4%.  These numbers suggest that the Red Wings' dominant power play unit was more of a team effort than, say, the Colorado Avalanche power play which was very dependent on Sakic and Forsberg.

Was Mats Sundin a poor performer on the power play, or was he merely a victim of a poor special teams unit in Toronto? It's probably at least a bit of both. Sundin apparently wasn't good enough to singlehandedly lift his team's unit above average, but he did score 47 points with the man advantage with Quebec in 1992-93 as a member of a standout PP lineup also consisting of Sakic and Steve Duchesne. Sundin also saw his rate jump in 2002-03 to 5.1 PPP/60 after several seasons in a row in the 3s or low 4s.  Over the remainder of his career Sundin never again dropped below 5.2.
Maybe he finally figured out how to score as a 31-year old, or maybe adding teammates like Nolan, Mogilny and Nieuwendyk had an impact and helped boost Sundin's scoring statistics.

It is interesting to compare the power play numbers for Sundin's Maple Leafs with Brendan Shanahan's Detroit Red Wings during the same period (1998 to 2004). Putting the top 10 in power play goals for each team side by side really illustrates the difference in quality, and does seem to suggest that Sundin would have most likely been able to rack up a lot more points if he had better teammates to share the load with the man advantage.

Detroit Red Wings, Power Play Goals ('98-04):

1. Brendan Shanahan, 81
2. Steve Yzerman, 52
3. Nicklas Lidstrom, 46
4. Sergei Fedorov, 46
5. Tomas Holmstrom, 43
6. Brett Hull, 29
7. Martin Lapointe, 25
8. Igor Larionov, 21
9. Vyacheslav Kozlov, 20
10. Luc Robitaille, 16

Toronto Maple Leafs, Power Play Goals ('98-04):

1. Mats Sundin, 69
2. Gary Roberts, 28
3. Sergei Berezin, 27
4. Bryan McCabe, 22
5. Steve Thomas, 21
6. Darcy Tucker, 21
7. Jonas Hoglund, 17
8. Igor Korolev, 14
8. Alexander Mogilny, 14
10. Mikael Renberg, 13

I think Sundin was disadvantaged by team factors, particularly from 1997 to 2002, which also happens to be his peak period of even strength scoring.  As the team's best player he should shoulder some of the blame for Toronto being so mediocre with the man advantage, but results from earlier and later in his career show that when Sundin did have the good fortune to play together with star linemates then he was able to post better power play scoring numbers.  Pumping up his PP scoring stats could have moved Sundin from the 75-85 point range to a consistent 90+, which would have made him a more significant factor in the overall scoring race and in turn would have seen him viewed in a much more positive light today.

One final stat to compare Sundin and Shanahan:  In the 18 seasons where both of them played in the NHL, Shanahan scored more even strength points than Sundin only twice.  I'd take Sundin over Shanny every single time, and I think he's a deserving first ballot Hall of Famer.

Wednesday, June 6, 2012

Darryl Sutter and Goaltending

Darryl Sutter is getting a lot of credit for his impact on the Los Angeles Kings this season, which is perhaps not too surprising given that the team was 15-14-4 prior to his hire and 40-15-11 since, playoffs included.  Jon Quick is all over the headlines for his ridiculous postseason numbers, and certainly much of that is deserved, but there is also some historical evidence that suggests Sutter may be in the class of coaches that have a positive impact on the statistics of their goaltenders.

Throughout his career, Sutter's goaltenders have routinely been above average.  Sutter-led teams have only posted a below-average save percentage in two out of his dozen seasons as an NHL coach, and in every one of the remaining ten his team was at least .006 above the league benchmark in save rate:

1992-93: .901 in Chicago (.885 avg)
1993-94: .902 in Chicago (.895 avg)
1994-95: .907 in Chicago (.901 avg)
1997-98: .896 in San Jose (.906 avg)
1998-99: .915 in San Jose (.908 avg)
1999-00: .911 in San Jose (.904 avg)
2000-01: .914 in San Jose (.903 avg)
2001-02: .918 in San Jose (.908 avg)
2002-03: .897 in San Jose/Calgary (.909 avg)
2003-04: .919 in Calgary (.911 avg)
2005-06: .917 in Calgary (.901 avg)
2011-12: .930 in Los Angeles (.914 avg)

Overall:  .910 under Sutter, .903 league average

Sutter had Belfour for three seasons, Kiprusoff for two and Quick this past year, so he was partially lucky to benefit from some good goaltending.  However, the numbers before and after he arrived in the different towns seem to suggest that there was a consistent save percentage effect as a result of Sutter's hiring.  I looked at the full season prior to Sutter being hired and the full season after he was fired, with partial in-season results before he was hired/after he was fired also included in the before and after sample:

Chicago (before & after):  4352 SA, .898 save %, .893 average, +.005
Chicago under Sutter:  5940 SA, .903 save %, .892 average, +.011

San Jose (before & after):  6434 SA, .906 save %, .908 average, -.002
San Jose under Sutter:  11781 SA, .910 save %, .906 average, +.004

Calgary (before & after):  5738 SA, .909 save %, .907 average, +.002
Calgary under Sutter:  5545 SA, .914 save %, .906 average, +.008

Los Angeles (before):  3258 SA, .917 save %, .913 average, +.004
Los Angeles under Sutter:  1271 SA, .930 save %, .914 average, +.016

That's a consistent bump of .006 at each of Sutter's first three stops, with his L.A. numbers looking even better so far.

I should point out however that selection bias probably has an impact here, given that teams with low save percentages would be more likely to fire coaches and hire replacements.

Combined before sample:  .901 save percentage, .905 league average, -.004
Combined sample with Sutter:  .910 save percentage, .903 league average, +.007
Combined after sample:  .914 save percentage, .906 league average, +.008

Looking at it this way gives a potentially much less charitable interpretation of Sutter's true impact:  Maybe he simply coached some good goalies and was the beneficiary of teams bouncing back from poor seasons.  On the other hand, it does seem reasonable that the impact of a coach would continue in at least some fashion even after they leave town.  If Sutter played a key role in developing young players or established a style of play that the team continued to use under his successor then he could be partially credited for some of those continuing effects.

It is always important to take regression to the mean into account when evaluating coaches, or else guys hired by underachieving teams will look like heroes nearly every time as the team's results gradually change to more closely match their overall level of talent.  Sutter is not a miracle worker, and I think that Jack Johnson for Jeff Carter trade had a larger impact on the Kings' amazing transformation into a playoff juggernaut than their mid-season coaching change.  On the other hand, Los Angeles is certainly doing a great job of protecting their goaltender through great defensive play at the moment, and that reflects well on their head coach.

In the competitive world of the NHL, slight edges can sometimes end up being important, and Sutter's coaching may just be providing Jonathan Quick the defensive advantage needed to help the Kings' goaltender complete his memorable season with a great postseason run that currently has him holding the highest official playoff save percentage of all-time among goalies that participated in more than one playoff round.

Monday, May 28, 2012

Shootout Records Don't Predict Playoff Success

There was an interesting statistical pattern this year in the first round of the playoffs:  Only one of the goalies on a winning team had a better shootout win/loss record this year than his counterpart on the losing team.

Rangers vs. Ottawa:  Lundqvist 4-3, Anderson 6-1, Rangers win
Boston vs. Washington:  Thomas 7-1, Holtby 0-1, Caps win
Florida vs. New Jersey:  Theodore 4-7, Brodeur 7-2, Devils win
Pittsburgh vs. Philadelphia:  Fleury 9-2, Bryzgalov 3-5, Flyers win

Vancouver vs. Los Angeles:  Luongo 6-6 (Schneider 2-1), Quick 6-8, Kings win
St. Louis vs. San Jose:  Halak 3-7 (Eliott 1-3), Niemi 8-4, Blues win
Phoenix vs. Chicago:  Smith 6-8, Crawford 6-4, Coyotes win
Nashville vs. Detroit:  Rinne 4-5, Howard 7-2, Predators win

Combined totals:
Winning goalies:  34-42 in shootouts this year
Losing goalies:  55-28 in shootouts this year

This result was surely largely influenced by randomness, especially given that the trend somewhat reversed itself in subsequent rounds with Brodeur and the Devils continuing to win.  However, some matchups probably looked closer than they were in the standings because the weaker team had the benefit of a regular season shootout edge.  The Flyers, for example, would have had a better record than the Penguins if all shootouts were counted as ties (although the Penguins' goal differential advantage indicates that Pittsburgh probably still should have been a slight favourite in that series).

The shootout looks like it is here to stay in the NHL, but it does still seem unfair that it has an impact on the regular season standings that are used to determine playoff seedings.  When comparing two teams to make predictions for a playoff series, shootout results should obviously be disregarded.  That said, regular season records aren't even the best way to predict which playoff teams to bet on anyway (as the Kings and Devils are currently demonstrating), with metrics such as score-tied Fenwick and goal differential showing more predictive power anyway.

Tuesday, May 22, 2012

Mike Smith: Good Season or Good Goalie?

Case A:  2011-12 Was A Fluke Season

The fact that goalie results are heavily influenced by randomness and have a lot of variance from year-to-year has been clearly established by statistical analysis.  It would be very unlikely for an ordinary starting goalie to hit .930 just by luck, but it is not beyond the realm of possibility.  If the goalie is perhaps slightly above-average and aided by some relevant team factors then that explanation becomes a lot more compelling.  The sudden massive improvement in numbers for Mike Smith makes it reasonable to question where his true talent lies.  After all, it is not very often that a 30-year old goalie is able to bump up his career save percentage by .008 in the course of a single season.

The most commonly advanced argument by those seeking to downgrade Mike Smith is the team argument that claims that Dave Tippett is a defensive genius that helps all goaltenders succeed in Phoenix.  It's generally a good idea to be skeptical of shot quality arguments, but on the other hand there are three goalies now that have superior numbers under Tippett then they managed everywhere else since the lockout (regular season and playoffs combined):

Ilya Bryzgalov:
Without Tippett:  7517 SA, .911
With Tippett:  4481 SA, .919

Jason LaBarbera:
Without Tippett:  2703 SA, .905
With Tippett:  1474 SA, .916

Mike Smith:
Without Tippett:  4430 SA, .907
With Tippett:  2617 SA, .934

It is probably also worth noting that Smith played in the Pacific Division, a division that had a higher average save percentage than any other division in the league.  Part of that is surely because the Pacific has some great, top-flight goalies, but the league-leading number was also likely influenced at least to some degree by the tight style of play that is more typical for teams on the West Coast.

In the playoffs and regular season combined this year, Smith has faced just 15% of his shots against on the penalty kill.  Throughout the rest of his career his average was 19% (although part of the decrease was because penalties were down in 2011-12).  Smith also posted a .909 save percentage on the PK, which vastly surpasses his previous career PK rate of .874.  This year was only the second time in his career that Smith's PK rate even went above .880.

If Smith had faced 19% of his shots against on the PK with a PK save percentage of .880, his seasonal save percentage would have dropped .006 to .924.  Assuming that Phoenix has some additional shot quality effect at even strength relative to the rest of the league, he may only have played at a level of around .920 in a neutral team situation.  Obviously Smith had a terrific campaign, but perhaps those potential team factors caused some skeptical GMs around the league to drop him out of the top three in Vezina voting, even though his numbers suggest he probably deserved that ranking (Smith led the league in GVT, a measure that historically tracks very well with Vezina winners).

One large negative indicator for Mike Smith heading into this season is that he had an extreme home/road split.  Prior to this year he had a career save percentage of .918 at home compared to just .896 on the road.  That's usually not a good sign, given that road save percentages are subject to less potential scorer bias since they are spread across a number of different arenas.  This year, Smith's degree of improvement on the road (+.030 to .926) was almost twice as much as the increase in his home save percentage (+.016 to .934).  It is perhaps interesting to note that both LaBarbera (.922 on the road) and Bryzgalov (.921) have excellent road numbers under Tippett in Phoenix (both actually did better away from home than they did in Glendale). Phoenix has also finished in the top eight in road record in each of the past three seasons, despite finishing 16th in points at home this season for the second year in a row.

To summarize, Smith's track record doesn't support anything close to a season of 67 GP at .930, and there is evidence that suggests Phoenix may be a place that helps its goalies out a bit.  As a result, Smith is probably headed for a major regression in his numbers for 2012-13.

Case B:  Mike Smith Is A Good Goalie

Smith has been very impressive in the playoffs, causing some to argue that his game has developed and he is in the process of joining the game's goaltending elite.  The caveat is that subjective arguments are affected by all kinds of observer biases, and sometimes it is hard to differentiate between a good goalie and a lucky goalie or a hot goalie, but Smith's numbers have been so outstanding that it is likely they do represent a significant improvement, even taking into account some of the other factors that may be in play.

One of the biggest question marks in evaluating Mike Smith is determining how much of an impact the concussion he suffered late in 2008 had on his career development.  It's certainly possible to make the numbers fit a narrative that portrays Smith as a promising goalie early in his career before he was brought low by a concussion, and that it shouldn't be that surprising that Smith has blossomed into a quality NHL starter now that he has fully recovered.

Smith was concussed in December 2008.  His career save percentage as of December 31, 2008 was .913, which is very strong taking into account the league average of .907 from 2007 to 2009.  Smith's 2008-09 season on a really bad Tampa team looks quite impressive in retrospect.  The Lightning fired Barry Melrose in mid-season, ended up second last overall, and went 10-22-9 with a 3.62 GAA with their backup goalies in the net (Smith himself was 14-18-9 and 2.62).  Unfortunately Smith's campaign was cut short by his concussion symptoms.

How much did the effects of his head injury affect Smith in 2009-10 and 2010-11?  It's difficult to tell, particularly because he faced a relatively small sample size of 1600 shots.  In 2010-11 he would likely have been much closer to full recovery, but rarely played in a backup role behind Dwayne Roloson.  Smith's combined numbers for those two seasons, together with January 2009 when he was playing with concussion symptoms before getting shut down for the year, come to .899 on 2096 SA, which is well below where he was prior to the injury.

Smith have simply have been somewhat lucky early in his career and then unlucky later on, we've seen that happen many times over a couple of 2000 shot samples.  However, it is also entirely possible that his injury had a major impact on his results and that he has had above-average talent since the start of his career.

It has been widely reported that Phoenix goalie coach Sean Burke wanted to bring in Mike Smith because Burke saw potential there.  Maybe Burke was correct that Smith was a star in waiting, although a cautionary note is that those are the types of things that are easy to say in hindsight if you want to make a specific talent evaluator or coach look great, mainly because they don't account for any of the misses (if Smith didn't work out then nobody would be writing stories about how Sean Burke got it wrong).

Smith's two-year, $4 million deal doesn't exactly suggest that Phoenix thought they were going to be getting Vezina-calibre goaltending, but at the same time with LaBarbera in the backup role and no real other options on the farm the Coyotes were clearly betting their season on Smith.  If they didn't think he was going to be at least average it is questionable whether they would have made that move.  Goalies don't always follow linear career paths, and maybe the change of scenery combined with coaching and his personal development helped Smith make the jump and join the games' elite.


It seems to be an absolute rock-solid bet that Mike Smith's numbers are going to regress next season, and that his true talent level doesn't quite measure up to the level of performance he has displayed this year.  On the other hand, it seems clear that Smith's concussion had a negative impact on his career, and as a result his career numbers are at least somewhat understated.

There is a chance that Smith is now one of the game's best goalies, and there is a chance that this was his career season.  The most likely case is that Smith is not an elite goalie who can be counted on to remain near the top of the league year after year, but he may be an above-average starter who still has a number of useful years ahead of him.  Smith is probably still a good candidate to put up another season of around .920 or maybe even a little better next year, given Phoenix's team discipline and the fact that he outperformed league average by .006 over a three season stretch earlier in his career when he was likely not as good as he is right now.

Thursday, May 10, 2012

Understanding When to Make A Save

"Goaltending in the playoffs so much is about when you give up goals, giving your team a chance to stay in a game when the momentum's going against you or when you've had it.  Brian Elliott simply didn't do that...Conversely, Jonathan Quick, he was timely, he was strong on his game.  The St Louis Blues scored six goals in this series on Jonathan Quick, largely due in part to him not only being a talented goaltender, but to being dialed in to the time of the game and understanding that a save needed to be made, he was there all the way through the series."  (Craig Button, TSN panel)

I'm pretty sure that Jonathan Quick probably does understand when a save needs to be made a lot better than Craig Button does.  If any goalie is in a position to realize the importance of every goal against, it is the backstop for a Los Angeles team that ranked 29th in the league in scoring and played in nine 1-0 games this season.  Even with the Kings' improved playoff offence, it seems very unlikely that Quick was taking anything for granted.  Obviously any goalie who allows just six goals against in four games is making a lot of saves, not just a lot of "timely" saves, and Quick has been great so far in these playoffs because he has stopped nearly everything that has been thrown at him.

If you had to make a case for anything about the Kings being timely, it would be their scoring and possession game more so than their goaltending.  Against St. Louis Quick was pretty great regardless of score, posting a save percentage of .947 or better during each of the key score differentials (down by 1, tied, leading by 1).  The Kings' offence scored 1/4 (25%) while trailing by one, 6/33 (18.2%) while tied, and 3/35 (8.6%) when leading by one.

Los Angeles was also dominant on the shot clock over the Blues with the score tied.  In game one shots were 16-16 and goals were 1-1 with the game tied, and a strong effort from Quick was a big factor in the result.  In games two through four, however, the Kings outshot St. Louis 17-6 and outscored them 5-0 with the score tied.  To score almost as many goals as shots allowed is amazing.  In addition, the average distance on those half-dozen shots against was 42.2 feet and probably only a couple of them could even be marked down as scoring chances.

The biggest problem with Button's logic, though, is the implication that there is choice involved in goaltending.  His absurd premise is that any goaltender can choose to stop any puck, if only they have the necessary clutchness or understanding of clutch play to know that it would be best for them to make that save, and that the difference between a goalie who performs well in a pressure situation and one that doesn't is merely a matter of knowledge or understanding.  Anyone making that claim obviously doesn't understand how much making saves requires a netminder to play the percentages, particularly in today's NHL where key goals are often scored through screens or from deflections or on pucks ping-ponging around the crease or slot area.

Even if goalies can increase their focus or energy level and actually boost their results (which is debatable), that still doesn't mean they have any chance at all at stopping a screened double-deflection into the top corner no matter how well they understand the delicate balance of momentum at that exact point in the game.  It really doesn't take a mathematical background or a detailed knowledge of expected win probabilities to understand that goals against are a bad thing in close hockey games.  That simple knowledge is surely the most basic of prerequisites to tend goal in the NHL, and it seems preposterous that it actually has an impact on the outcome of any games.

Thursday, April 26, 2012

Two of the Closest Series Ever

Last night's overtime victory by Washington completed what was likely the closest series in NHL history in terms of score.  Not only was every game decided by a single goal, but 99.4% of the entire series was played with a goal differential of one or less.  The only time either team managed to take a two goal lead over the seven games was when Jay Beagle scored in the second period of game five to put the Caps up 2-0, a margin that lasted less than three minutes before the Bruins cut it to 2-1.

The West had a matchup that was also a strong contender for the title of closest ever right up until the final period of the series.  Phoenix and Chicago went to overtime in each of the first five games of the series, and by the time of the second intermission in game six had spent 99.3% of their series tied or with one of the teams leading by a lone goal.  However, in that final period Phoenix finally broke through and pulled away for a decisive 4-0 win.

I thought it would be interesting to look at numbers broken down by score in these two series to see why they were so tight.

Boston vs. Washington:

Washington Up One:
Shots:  Washington 30, Boston 72
Goals:  Washington 1, Boston 7
Save %:  Holtby .903, Thomas .967

Score Tied:
Shots:  Washington 145, Boston 162
Goals:  Washington 11, Boston 7
Save %:  Holtby .957, Thomas .924

Boston Up One:
Shots:  Washington 32, Boston 11
Goals:  Washington 4, Boston 0
Save %:  Holtby 1.000, Thomas .875

There were major score effects during the series, as would be expected.  Including the brief period when Boston was down by 2, the trailing team outshot the team in the lead by the whopping margin of 107-41 and scored 12 goals in just 141.7 minutes of play for an amazing rate of 5.08 goals per 60 minutes.  The leading team managed just one goal (0.42 per 60), scoring on just 2.4% of their shots taken.

It is typical that the trailing team generates more shot attempts, but usually their percentages drop as they put more pucks on the net and take more risks, leaving themselves open to chances going back the other way.  For whatever reason both the Caps and Bruins were able to get away with almost everything in their own end when pushing for the tie.

The overall shot statistics flatter the Bruins a bit because they spent more time trailing.  Boston outshot Washington by 20% overall but by just 12% with the game tied.  It was still a very close series, and Washington was pushed over the top by an impressive effort from rookie netminder Braden Holtby.

The Chicago-Phoenix series was similarly close in terms of scores, but the underlying numbers suggest that balance of play wasn't nearly as tight.

Phoenix vs. Chicago:

Phoenix Up Two Goals or More:
Shots:  Phoenix 11, Chicago 11
Goals:  Phoenix 2, Chicago 1
Save %:  Smith .909, Crawford .818

Phoenix Up One:
Shots:  Phoenix 40, Chicago 76
Goals:  Phoenix 2, Chicago 5
Save %:  Smith .934, Crawford .950

Game Tied:
Shots:  Phoenix 73, Chicago 124
Goals:  Phoenix 9, Chicago 6
Save %:  Smith .952, Crawford .877

Chicago Up One:
Shots:  Phoenix 35, Chicago 30
Goals:  Phoenix 4, Chicago 0
Save %:  Smith 1.000, Crawford .886

In this series the Blackhawks outshot the Coyotes by 51% overall, and an impressive 70% with the game tied, yet somehow managed to get outscored 17-12 over the course of six games, as well 9-6 in tie-game situations.  The strong goaltending of Mike Smith compared to the inconsistent play of Corey Crawford was the decisive factor in the series.

Smith was particularly strong in game six, a game where everything was massively tilted in favour of Chicago except for the scoreboard.  I also found it interesting that Smith had the highest save percentage of any of the four goaltenders while his team held a one goal lead, which was perhaps unexpected given that the Blackhawks fought back three times to tie the game late in the third period.  Several posters in this HFBoards thread specifically downgrade Smith's first-round performance because of his supposed lack of clutch play in allowing late game-tying goals, but to me that's making the classic mistake of evaluating playoff performance:  Letting a few memorable events have too much influence while failing to properly appreciate the larger picture.

For the sake of comparison, Braden Holtby gave up a lead in five out of seven games, including twice in the third period, yet nobody would call him unclutch.  Allowing goals against in the final 10 seconds obviously has a huge negative impact on a team's win probability, but when 99% of the series is within one goal then every goal against has a major impact on the chance of victory.  Other than one third period in Chicago, pretty much every situation these four goalies have faced so far in the 2012 playoffs was a clutch situation.

Just as with Washington and Boston, the trailing team rode both a large outshooting advantage and insanely high percentages.  Overall, shots were 122-81 in favour of the team playing catch-up, with turned into a 10-4 advantage on the scoreboard.  That works out to a rate of 3.16 goals per minute for the trailing team, compared to just 1.26 for the team holding on to a lead.

In some respects it is amazing that Chicago even came as close to winning the series as they did, given that Phoenix's scoring rate per shot was over twice as high.  The repeated late-game comebacks to force OT kept them in it, but unfortunately the goaltending disparity was perhaps never more apparent than in extra time where Smith managed a .923 save percentage while Crawford put up a mere .813 and let in two very soft goals to help the Coyotes to the franchise's first second round appearance since 1987.

It would be interesting to see detailed scoring chance numbers or a breakdown of odd-man rush chances to see whether the leading teams were generating chances and just missing their shots, or whether their defensive focus meant they were not creating many dangerous opportunities to score.  In the absence of compelling evidence, however, I'd guess it was mainly a tremendous run of hot goaltending that kept the scores close (in both series combined, goalies on teams trailing by one stopped 97.3% of the shots against) and created 13 games of razor-thin margins in these two memorable series.

Sunday, April 15, 2012

Sitting on a Lead Isn't Working in 2012

Fifteen games in, the story of the 2012 playoffs so far has been third-period comebacks. Twelve out of 15 games have featured a one goal lead at some point in the third period, and in seven of those twelve games the trailing team fought back to tie. Five times the team that was trailing eventually ended up winning the game, four times in overtime and once in regulation (the Flyers' roller coaster 8-5 win over the Penguins on Friday night).

I would have guessed that the trailing teams were riding very high percentages to be able to put up those kind of results, but that's not actually the case. Both the leading and trailing teams so far have shot 8.0% in the third period with one team up a goal, although the leading teams' numbers are slightly inflated because of empty netters (the shooting percentage with a goalie in the net for teams up by one is 6.1%). The primary reason for all the comebacks is that the trailing teams have been absolutely dominant in terms of possession, outshooting the opposition 88-49. That is a rate of 39-22 per 60 minutes of play.

It is normal for teams to play to the score. According to Behind the Net's stats, the Nashville Predators were the only team in the league this season that did not have over 50% of the Fenwick events while trailing by a goal, and teams down by one had an outshooting advantage of 29.6 to 24.5 per 60 minutes. Those numbers include results from the whole game, and would likely show a greater split for third period results only (some of my past findings indicate that the shot rate skews even more in the third period alone).

Despite generating more shots, however, the trailing teams were actually outscored during the regular season (2.54 to 2.43 per 60 minutes), which indicates that playing to the score was the right move for teams holding a late lead. It probably should be noted though that in a regular season sample team strength might also be a factor, given that the better teams would be more likely to spend time in the lead than the league doormats, whereas the playoffs do not have the same broad range of team quality.

The regular season results give a trailing team shooting percentage of 8.2%, very close to the 2012 playoff numbers. During the regular season the numbers for the leading team went up to 10.4%, which indicates that the comeback teams in this year's playoffs have probably been a bit lucky or had their comebacks enabled by strong goaltending in the third period. That said, even if the leading teams' percentages rise as the playoffs go on, there will still be a lot of comebacks as long as the shot ratio remains in the vicinity of 2-to-1 in favour of the trailing teams.

Utilizing a collapsing defence to preserve a late lead is the best strategy in some cases, but some teams (Pittsburgh and Phoenix, perhaps?) are probably currently reviewing their game tape to determine whether their late-game play is doing more to help them than to hurt them. You can trade quantity for quality in terms of chances against, but if the quantity gets too high than some pucks are going to go in anyway and the strategy is no longer optimal. It will be interesting to see whether teams can improve their ability to hang on to late leads or whether the comebacks will continue in 2012.

Thursday, April 12, 2012

One of These Does Not Belong...

GoalieSv%MaxMinAvgPO Tms
M. Smith.930.974.851.926.936
J. Quick.929.963.881.927.930
J. Halak.926.963.856.931.937
P. Rinne.923.960.880.921.919
J. Howard.920.940.881.919.919
R. Luongo.919.949.869.920.930
A. Niemi.915.950.863.914.915
C. Crawford.903.931.867.901.896

Sv% is total save percentage, Max and Min are each goalie's best save percentage marks over seven consecutive games during the 2011-12 season, Avg is the goalie's average save percentage per 7 game stretch (naturally, this metric will track very closely with the overall save rate), and PO Tms is each goalie's combined save percentage against the other 7 Western Conference playoff teams.

The obvious takeaway is that there is some really good puckstoppers out West, with six teams possessing demonstrably above-average goaltending. That leaves just the San Jose Sharks with the very average Antti Niemi, and the Chicago Blackhawks, who look to be the one team in the Western Conference that is in a substantial amount of trouble in goal heading into the playoffs.

In a previous post, I pointed out that a typical Cup winning goaltender needs to put together a streak of approximately .930 over 600 shots. This season Crawford barely even managed to hit that during his best seven game stretch. His highest mark over 20 consecutive games was only .912. If the playoff version of Corey Crawford is the same one that showed up during the regular season, it is very unlikely that Chicago will get the percentages needed to overcome four straight difficult opponents and end up with a Stanley Cup. Chicago did manage to win the 2010 Cup despite Antti Niemi's mediocre .910 save percentage, but most observers would agree that version of the Hawks with multiple star players on ELCs was stacked compared to this year's roster.

That said, all hope is not lost for Blackhawks fans. Despite Crawford's mediocre play, Chicago still managed to go 13-10-3 against the other seven Western Conference playoff teams with him in net. That was still the second-worst record of any of the eight expected playoff starters (ahead of only Niemi), but it was not far behind most of the other teams as only Smith and Luongo managed to post a win percentage of .600 or better against that tough slate of opposition. Chicago has a very good team in front of Crawford, especially if captain Jonathan Toews is able to get back into the lineup and contribute at his usual level. In score-close Fenwick, one of the best measures of overall team strength, Chicago ranked 5th in the league.

The other thing to keep in mind is that Crawford acquitted himself quite well in the first round against Vancouver last year and had a much better overall season in 2010-11. Perhaps Crawford's true ability is closer to the .918 he managed in the 2010-11 regular season and playoffs combined than this year's .903, which would make him far more likely to be able to deliver Cup-calibre goaltending over the next two months.

For what it is worth (probably not much), Crawford finished the season strong with a .921 and an 8-2-1 record over the last two months, including six wins over playoff teams, although Chicago's offensive output of 32 goals in the 11 games and stingy shot prevention of 23.5 SA/60 obviously also played a major role in that recent success.

The Western Conference playoffs are as usual going to be a grind filled with tight games where hot goaltending can make a difference. The Blackhawks have a strong team that could be a contender, except for the question mark related to the fact that their goaltending this season has been well behind the rest of the teams in the mix out West. It seems safe to say that the pressure is on firmly on the shoulders of Corey Crawford because Chicago will likely need much better goaltending than they have been given so far in 2011-12 to be able to put together another Cup run.

Tuesday, April 10, 2012

Why Lundqvist Should Win the Vezina

Even Strength Save Percentage in 2011-12 by Division:

1. Northwest, .925
2. Pacific, .925
3. Central, .923
4. Northeast, .919
5. Southeast, .917
6. Atlantic, .916

Quick and Lundqvist ended up with almost identical situational save percentage stats (both .933 at even strength, with Quick just slightly ahead on the penalty kill .908 to .905). The Kings faced a higher percentage of shots against on the PK, which caused Lundqvist to slightly edge out Quick in terms of overall save percentage by the narrow margin of .001. The closeness of those results, combined with Quick's extra workload and the Kings' late-season playoff charge, has made Quick a recent trendy Vezina pick, an impressive comeback given that it looked like Lundqvist had the award sewn up by the All-Star break.

Assuming both goalies were competing in the same environment, it would indeed be very difficult to separate them by the numbers. However, if each goalie's numbers are adjusted relative to their individual conferences, then Lundqvist opens up a decisive edge over Quick:

Conference-adjusted situational save percentages:

Quick: .930 EV, .907 PK, .941 PP
Lundqvist: .936 EV, .906 PK, .974 PP

Multiplying those out by the league-wide average frequency of shots against in each situation, Lundqvist ends up with a conference and situationally adjusted save percentage of .932 compared to .927 for Quick.

Another option would be to adjust each goalie's numbers relative to their division, although in that case the much smaller samples means it would be important to remove each goalie's results from the overall numbers (i.e. the Pacific without Quick and the Atlantic minus Lundqvist). That calculation only increases the margin in favour of Lundqvist, given that his .933 looks much more impressive when stacked up against the combined .912 put up by the rest of the Atlantic division, with the other netminders in the Pacific still combining for an above-average .924 mark even without the contributions of that division's Vezina candidate. Using the divisional numbers and the same correction for shots against by situation, Lundqvist's adjusted number moves well ahead of Quick .935 to .927.

To me, anything close to a tie suggests Lundqvist should win because his elite track record means that it is much less likely that his terrific season was based on luck or other secondary factors. I would pick Lundqvist even assuming that both goalies faced identical shots against for this reason alone. Adjusting for the east/west disparity only makes the choice that much more obvious, in my opinion.

Relying on historical records to evaluate a single season of goaltending is somewhat unfair to the goalie who is four years younger, but it is simply the reality of dealing with uncertainty in goalie evaluation (and why I don't think single-season awards are all that significant). This was Lundqvist's third season in a row at .920 or better while Quick's previous career best was .918. Future years may well prove that Quick's true talent is in the mid- to upper-.920s, but as of right now that's probably not the smart way to bet.

Another relevant piece of statistical info is that Quick's numbers were much better at home (.936 at home compared to .922 on the road), whereas Lundqvist's numbers were higher away from home (.934) than at MSG (.926). That doesn't necessarily mean much, variance is naturally going to be higher over 900 shot samples than over full-season results, but since road numbers are counted by a variety of different scorers they are less likely to be subject to bias. It is also worth noting because both goalies have been pretty consistent on the road over the last three seasons, with Lundqvist maintaining a steady .011-.013 gap over Quick:

Quick: .915, .916, .922
Lundqvist: .926, .929, .934

This year Quick's home numbers shot up while Lundqvist's improved slightly. Again that could be reflective of improved play for both goalies, but it could also be partially related to shot counting or team factors. The other interesting thing to note is that this is the third straight year that Lundqvist's save numbers have been better on the road. MSG is known for poor stat recording in general and it is possible that his numbers are being at least partially suppressed at home, especially given that Lundqvist faced an average of 5.6 more shots against per 60 minutes on the road compared to at home in 2011-12, coming on the heels of a 6.0 difference in 2010-11.

On the other hand, Quick's shots against rates were 26.5 at home compared to 28.0 on the road in 2011-12. His shots against rate at home has actually increased in each of the last three seasons, at the same time as his road shots against rate has been continually decreasing. That's not the typical statistical profile of a goalie being disadvantaged by his home scorekeeper. Overall, the home/road numbers are just one more reason to be a bit more confident in the Swede than in the American.

Taking their histories into account along with the conference disparity, I think Henrik Lundqvist deserves to win his first best goalie award. It could be argued that the shot quality allowed by the Rangers was not typical of the rest of the Atlantic Division, or that goaltending in the Western Conference in general or the Pacific Division specifically was simply a whole lot better than it was out East, either of which would mean that the adjustments above are unfair to Quick. I would certainly listen to anybody willing to make those arguments, but right now I don't see enough supporting evidence on the table. Shot quality arguments are always particularly murky because of the lack of good data, and subjective comparisons are very difficult, particularly for two teams in opposing conferences.

As an aside, this season has made me wonder at times whether we can continue to rely on the general assumption that EV shot quality is relatively constant between teams. The two major pieces of evidence in that direction are probably St. Louis' 2011-12 goalie stats and the conference splits displayed above. In the aggregate I think 5 on 5 shot quality is still probably not all that important for most of the league, but if there are some significant effects on the margins that would be important to know for goalie evaluation.

Thursday, March 22, 2012

Ilya Bryzgalov and Regression

Ilya Bryzgalov, first 27 games of 2011-12: .890
Ilya Bryzgalov, next 27 games of 2011-12: .929

Ilya Bryzgalov, last 27 games of 2008-09: .898
Ilya Bryzgalov, first 27 games of 2009-10: .925

It may still be too soon to proclaim that Ilya Bryzgalov is back in the kind of form that earned him a $51 million contract in Philadelphia, but the results have certainly turned around as of late for the Flyers' quirky netminder (8-1-1, 1.27, .955 so far in March).

As I pointed out in my post on Jon Quick, things that happen at the start of the season always seem to take on much greater significance. And when you have a slump in your first year in a new city that is notoriously tough on its goaltenders after signing a big-money contract in the off season and then end up increasing your exposure through a starring turn in HBO's 24/7 series, then any kind of early season struggles make it dead easy for sportswriters to start crafting narratives. There are all kinds of places to assign blame, from Bryzgalov's weak playoffs last year to his attitude and work ethic after signing his massive contract to his mental toughness and the pressure of the Philly market. It's possible to come up with all kinds of theories about the situation because while the performances were not up to par nobody knew for sure why exactly that was the case.

Is it possible that one of those things was affecting Bryzgalov? Probably, maybe even several of them at the same time. Insiders like Elliotte Friedman are most likely on the right track with some of the possible explanations for Bryzgalov's early-season play. Bryzgalov did look awful at times in games, worse than he did even when going through tough stretches in Phoenix, and it seems to be fairly unanimous within the team that he was struggling to cope with his greatly increased level of public exposure. At the same time, however, it is important to keep in mind the likely role of chance and variance in the goalie's results. As shown above, he has posted similar splits over a similar number of games before, even without the specific factors affecting his current season in Philly.

When a talented goalie goes into a slump, it can seem like they will never get out again. It's not always entirely bad luck, often they have to work on a few issues with their game or they have some off-ice problems, but with enough training and time those things are usually overcome and the goalie ends up returning to his elite form.

That's why regression to the mean is always the way to bet. If it's simply a matter of luck, then that is going to even out over time as shooters will stop making their shots at an unusual rate. If the goalie has issues to fix, then in most cases he will put in the time and effort to deal with them. At the other end of the scale, if a goalie's style of play is unusually effective then the rest of the league will do their homework on them to try to take advantage of any weaknesses and drag them back towards the norm.

Some have criticized stats guys' seemingly relentless focus on regression as removing the human variable and treating players like robots, but I think it is actually a natural consequence of human behaviour. When competition is involved, everyone copies everyone else and unsustainable advantages get erased fairly quickly. Those who excel may not always keep up the effort needed to separate them from the pack, while those who lag behind will often increase their intensity to try to make up ground. This is probably also one of the reasons why it is difficult to find persistent team advantages in terms of shot quality in today's NHL. A new system might initially work wonders but not once everyone else figures it out. In the long run, that often means there is not a lot of separation between the pack, especially when looking at a job like NHL starting goalie which only employs the top 30 best in the world at their craft.

Extended slumps by top level goaltenders are not a rare phenomenon. Sometimes they appear to have an obvious cause and other times they do not. Pretty much every goalie will struggle through them over the course of their pro careers. A particularly good example is when Henrik Lundqvist, whose consistently elite track record stacks up among the best in the game, had a 33 game stretch at .888 in the middle of the 2007-08 season that ended with him finishing third in Vezina voting. Dominik Hasek also found himself below .900 twenty games into the 1997-98 season that concluded with a second straight Hart Trophy. The Dominator recorded an identical .945 save percentage in the twenty regular season games both immediately prior to and immediately following those brief early-season struggles.

My philosophy is to deal with large sample sizes and bet on track record rather than recent form. Others prefer a "what-have-you-done-for-me-lately" approach that tries to take into account all kinds of other factors such as a goalie's mental state, his recent technical performance or the way the rest of the team has been performing. Both methods will get some right and some wrong, but I think the historical evidence suggests that betting on established talent over recent form is a far more successful method (just read Arctic Ice Hockey, Gabe Desjardins has been beating that drum for years). Ellen Etchingham also recently provided a great read on the subject of variance.

Even if there was someone out there who could identify all of Bryzgalov's early-season technical flaws and could see between his ears well enough to perfectly measure how the pressure of his new environment would impact his play, they still might not do any better at predicting his next 27 games than somebody who simply knows that Bryzgalov is a .915 career goalie on 10,000+ shots. All Bryzgalov may need to do is tweak a few things in his game, change his mental approach, or have his team restructure some aspect of his environment and it might be enough to reset everything and put him back to his usual self, leaving that detailed analysis of his struggles pretty much obsolete.

I don't mean to oversimplify the difficulty of fixing holes that have developed in your game, that is not always an easy process or something that happens overnight. And if you are the Philadelphia goalie coach, then obviously nothing is more important than diagnosing exactly what is going awry. But for proven talent, things usually end up working out in the long run, or at least until they hit the inevitable decline because of age or injuries.

Bryzgalov is unquestionably a proven NHL talent, but while he has quieted the critics for the moment the playoffs are certainly going to be a pressure cooker where every small performance sample will be blown ridiculously out of proportion. Hopefully for him when those games roll around he will be closer to his March form than his level of play in October, or the media will no doubt be dusting off the exact same stories about his inability to handle the pressure of Philadelphia.

Friday, March 16, 2012

The Curse of Obama

In the 20 games since the Boston Bruins' White House visit that Tim Thomas famously did not attend, the defending Vezina Trophy winner has gone 9-9-0 with a very uncharacteristic 3.07 GAA and .890 save percentage.

That .890 would look ugly for pretty much any goalie in today's NHL, but it looks especially glaring for Thomas considering his excellence over the past three seasons. Going back to the start of 2008-09, Thomas' numbers have been consistently elite even when divided up into the relatively small sample size of 20 games (playoffs included):

Nov '11-Jan '12: .935
May '11-Nov '11: .943
Mar '11-May '11: .929
Jan '11-Mar '11: .931
Nov '11-Jan '11: .939
Mar '10-Nov '11: .941
Dec '09-Mar '10: .911
May '09-Dec '09: .921
Feb '09-May '09: .936
Dec '08-Feb '09: .930
Oct '08-Dec '08: .937

The .911 and .921 came in the latter half of 2009-10 when Thomas was limited by a hip injury. If that is enough to excuse those two results, it has been over three calendar years since a healthy Thomas had to endure a 20 game stretch below .929. And then he decided to skip a team event for personal reasons and everything fell apart.

Obviously I don't actually think the flap surrounding Thomas and President Obama made any significant impact on his play, but there is no question that regression has hit the Bruins hard in net in 2012. Not only have Thomas' numbers nosedived, but Tuukka Rask is 0-3-1, 3.07, .886 since the All-Star break. The .940 team save percentage that the team was rocking early in the year was never going to hold up, but I didn't expect the wheels to come off this heavily either.

At 37, Thomas is in the age range where performance levels can change pretty suddenly, and he has had a lot of recent mileage considering his workload this year combined his playoff run last season. Thomas is already in the top-10 all-time this year for games played by a 37-year old, and including playoffs only one goalie has ever played more games in his age 36 season than Thomas' 82 last year. That doesn't necessarily mean that an awful 20 game stretch is a sign of impending doom, but it's certainly something to keep an eye on. Thomas' recent form has to be at least somewhat concerning for Bruins fans with the playoffs just around the corner, given that elite goaltending has been a big part of the team's success over the past two seasons.

Wednesday, February 29, 2012

Why the Counting Trophies Method of Goalie Evaluation Is Flawed

One of the biggest problems with evaluating goalies by career value is that there aren't any good commonly-used counting statistics. A goalie's career line merely shows games played, wins, and shutouts, plus the rate stats of GAA and save percentage. Wins and shutouts are very heavily influenced by the rest of the team, with shutouts also varying widely depending on the level of league scoring. Games played is important in determining overall value, but does not take into account level of performance at all other than reflecting how long the netminder was able to convince an NHL team to keep giving them starts.

There has been a shift towards a greater focus on save percentage, but as long as save percentage remains a rate stat it is difficult to understand intuitively whether a goalie with a higher save percentage but a lower workload is contributing more value than one of his peers with the opposite. There are a number of good ways to turn save percentage from a rate stat into a cumulative stat (typically by comparing to league average or to replacement level and then multiplying by the shots against). Hockey Prospectus' Goals Versus Threshold, which is slightly more complex but based on a similar foundation, is a number that is recognized at least within the online hockey stats community, but there is not a widely accepted standard.

As a shortcut, therefore, many people look at a goalie's trophy case, and use that to determine which netminder had the better career. Intuitively that makes sense, as when we evaluate athletes we want to know things like how many times they were considered the best in the league. However, I do not believe this is the best approach for evaluating NHL goaltenders.

If somebody just sweeps the awards year after year, like Dominik Hasek did in the 1990s, then that is definitely showing something meaningful. Or when Glenn Hall kept getting voted ahead of other Hall of Famers while repeatedly overcoming the heavy bias towards the GAA leader, that's also probably revealing something important about how his play was viewed by his contemporaries. But goalies with a handful of awards are exceedingly rare. When you are comparing two veteran goalies where one has a Vezina or a Smythe while the other one doesn't, I don't think that trophy really adds much information at all. Some trophy-focused individuals might, for example, try to argue that Miikka Kiprusoff's Vezina Trophy means he has had a better career than Roberto Luongo, even though the performance gap between them is likely at least 100 goals in Luongo's favour (Lou's career GVT is more than double Kipper's number).

The problem with goalies is that one season is not a large enough sample to properly rate anybody, because results are not accurately tied to performance. Given that awards are handed out primarily based on results, this means that luck and team factors play a disproportionate role in winning awards for goalies.

For a .914 goalie who faces 1800 shots in a season, the 95% confidence interval of his save percentage based on binomial probability would put his performance anywhere between .903 and .925. To prove that this is not just a theoretical exercise, we can just look at two goalies with .914 career save percentages: Ilya Bryzgalov and Ryan Miller. Both have a low mark of .906 as a starting goalie, and despite their strong career track records both have struggled for the majority of this season. Bryzgalov looks likely to set a new career low mark this year, although Miller looks to have turned things around as of late.

Their peaks, on the other hand, are much higher, with Bryzgalov hitting .921 last season and Miller topping out at .929 during his '09-10 Vezina season. Those ranges are in fact almost exactly what would be expected if their numbers varied by random chance alone. Between the two of them, their average high is .925 and if Bryzgalov ends up at his current .899 while Miller stays above .906 then their average low would be .903, which again would exactly match the predicted range given above.

In addition to simple performance variation, a goalie's teammates could raise or lower his save percentage by up to about .005 or so depending on how many penalties they take and whether they are good at preventing shots against on the penalty kill. The official scorer in the goalie's home rink could also assist in cutting or boosting shot totals by a shot or so per game, which again could have an impact in the neighborhood of .005 compared to a goalie on the other end of the spectrum. And maybe a goalie plays for Ken Hitchcock or Jacques Lemaire and his team has a good year in front of him defensively in terms of reducing scoring chances, which could easily add another few thousandths to the final save percentage number.

Over a career these effects often mostly wash out, as a goalie will benefit from them in some seasons and suffer because of them in others. But when looking at a single season, as is the case when considering awards nominations, these factors can further accentuate the already heavy effect of random chance.

A goalie with at least a .920 save percentage over 1800 shots has about a 50% chance of being a Vezina nominee, based on seasonal results since the lockout. Bump that save percentage up to .925, and you probably have about a 50% chance of winning the trophy, given that half the goalies who met both cutoffs won the Vezina. Consider that binomial probabilities suggest that a goalie with league average talent will put up a .920 or better over 1800 shots 1 time out of every 5 just by chance, and you can see how it is entirely possible that a goalie who carves out a decent NHL career will probably luck into at least one good season somewhere along the line even before considering the other factors that could help his statistics. Fortunately there are selection effects that limit the number of flukes, since most average goalies won't be given that many starts by their teams, but it does still happen with some regularity.

Whether a goaltender wins the Vezina or not depends not just on his own play, but also what other goalies are doing around the league. To return to the Luongo/Kiprusoff comparison above, it's easy to see the impact of external factors when you consider that if Martin Brodeur had his 2006-07 season in 2005-06 and vice versa, Luongo would be the guy with a trophy while Kiprusoff would have been shut out.

It's not hard to find examples of goalies doing more or less the same thing year after year even as their voting numbers vary widely. Take Patrick Roy over the last five seasons of his career when he was the very picture of elite consistency, rattling off 61-63 starts per season, overall save percentage typically in the .915-.920 and EV SV% in the .925-.930 range and a GAA usually 2.20-2.30. In 2001-02, his numbers all improved a bit, particularly his GAA and shutouts, although his EV SV% was just .005 better than his five-year average. His team's improved shot prevention and penalty discipline helped as well. There's certainly a chance that Roy played better than normal in 2001-02, but there is also a pretty strong possibility that he was more or less the same goalie all the way through and the breaks went his way in '01-02.

Over that period Roy typically got a few Vezina votes per season, until 2001-02 when he almost won the award. That year the high-minute goalies (Brodeur, Kolzig, et al) had down seasons, while Hasek was playing at a lower, post-injury level. It turned out that only one starting goalie other than Roy posted a save percentage above .921.

Unfortunately for Roy's trophy case, that goalie happened to be Jose Theodore, who put up a .931 save percentage on his way to the Vezina/Hart combo. If you look at Theodore's 2001-02 numbers in context, they are major outliers. He also had a bunch of the indicators of one-year flukes, including very high special teams numbers and much better numbers at home than on the road.

Knowing what we know now, it is very likely that Patrick Roy was a better goalie in '01-02 than Jose Theodore. That just wasn't immediately apparent from that 82 game sample, and that's why Theodore won the award. Using awards as the primary evaluation criteria, Theodore's first five seasons as a starter ranks ahead of Roy's last five seasons as a starter (unless you're one of those guys who thinks only the playoffs matter and you really love Roy's 2001 Cup/Smythe combo, and even in that case you'd probably give Roy just a slight edge), even though Roy's .929 EV SV% on 6165 SA is quite a bit better than Theodore's .921 on 6522 (about 50 goals better, or nearly two wins per season).

In contrast, you won't find a skater win a scoring title primarily because of luck. Assume a typical first liner with 15 minutes per game at even strength and 3.5 minutes per game on the power play, who either doesn't play much on the PK or doesn't score any points when does get an occasional shorthanded shift. Let's say his team takes shots at an average rate while he is on the ice (27 shots per 60 at 5 on 5, 45 shots per 60 on the power play). That player and his linemates would need to put up a ridiculous shooting percentage, and he would have to be involved in an unusually high number of plays to get on the scoresheet enough to be in Art Ross contention.

For example, if the player's team shot 13% at evens and 20% on the power play, both numbers above what any regular player managed last year, and got a point on 90% of his team's even strength goals and 75% of his team's power play goals, both extremely high and unusual participation percentages, the player would still end up with 97 points. That's a pretty high total and it would end up being near the very top of the league, especially this year, but it still isn't high enough for a top-3 finish in any of the post-lockout seasons. And again, this hypothetical player would need to get all the breaks just to get that close. If any of the luck factors drop off, then he's not in contention, and there are many ways that could happen (e.g. he misses a few games, his ice time decreases, his teammates don't score at a high rate, he gets unlucky with second assists or the power play runs more often through a teammate which reduces his scoring opportunities).

Jordan Eberle is pretty much that guy this year (at even strength his team is shooting 13% at even strength with him on the ice and he has points on 88% of his team's goals), and he's still tied for 10th in scoring. Joffrey Lupul is another guy that has flirted with the top of the scoring charts this season based on unsustainable percentages, and while he still sits just above Eberle in 9th it was always just a matter of time before he was going to left behind by the likes of Malkin, Stamkos and Giroux.

Very high points finishes are meaningful because they are unlikely to be flukes. Pure trophy counting would still undervalue someone like Mats Sundin who was consistently productive although never near the very top of the league, but in general a player with a couple of high finishes can be considered to have had a better peak than a similar player who never climbed the table to the same degree (although of course context and team factors need to be taken into account).

That doesn't happen to goalies. Flashes in the pan have won the Vezina, and average goalies have found themselves in trophy contention simply because fortune ended up favouring them over a 50-60 game stretch. Of course observers can sometimes identify when other factors or luck are at play; they don't always just follow the numbers to the exclusion of anything else. However, it is particularly difficult when rating a young goalie without much of a track record who has a big season. Is he breaking out, or is he getting lucky? Is he Pekka Rinne or Steve Mason? Only time will tell.

It is my belief that goalies can only be properly evaluated in a multiple season context. With that in mind, single season awards should be considered relatively insignificant in terms of career evaluation.

Friday, February 24, 2012

Score One For the Kings

Los Angeles defencemen with at least 100 games played between 2007-08 to present, ranked by plus/minus rating per 82 games played:

1. Sean O'Donnell, +8
2. Rob Scuderi, +6
3. Drew Doughty, +4
4. Willie Mitchell, +4
5. Matt Greene, +2
6. Davis Drewiske, -3
7. Peter Harrold, -5
8. Jack Johnson, -21

Obviously, Jack Johnson is not very good, yet apparently he and a first round pick are worth Jeff Carter, a guy who scored more goals in the three seasons prior to this one than every player in the league save Ovechkin, Crosby, Stamkos and Marleau.

Just as I prefer to look at multiple seasons' worth of data for goalies, I think there is at least some value in multiple years' worth of plus/minus given how the percentages tend to work themselves out over the larger sample, although obviously matchups and usage context are still important, especially for players utilized in a specific role.

However, when you're supposed to be one of the best defencemen on your team and you're that much of an outlier in terms of getting outscored over a 338 game sample despite not playing tough minutes, it's pretty glaring. It's also very tough to make a linemates excuse given that Johnson had four different defencemen rank as his highest defensive teammate TOI over that span, not to mention also having the benefit of playing more minutes with Anze Kopitar than any other Kings forward in four of the five seasons (it probably would have been all five if Kopitar played 82 games in 2010-11). At some point there's really nobody else left to blame. And the bad news for Blue Jackets fans is that their team is on the hook for paying Johnson $4.357 million per year through 2017-18.

Friday, February 10, 2012

The Hall of Fame Committee

I've often been critical of the decisions of NHL decision-makers in seasons past. But I have found myself being generally less and less critical of their moves and choices in recent seasons (and as my last post shows, it is clear that their perspectives have certainly changed over time). I think NHL teams collectively have a much better sense of the value of goaltending and what makes a good goalie than ever before.

I don't think the general managers from two decades ago were stupid. I do think quite a few of them were uninformed, unaware of the importance of objective analysis of goaltenders and focusing instead on the things that tradition dictated were important (like winning). I'm making that claim based on award voting and based on the number of bad goalies and washed-up veterans that kept getting NHL jobs long after they were deserving of them.

The same logic applies to Hall of Fame voters. I've gotten into a few discussions about the Hall of Fame, both here and in other places, and when people ask me if I think certainly goalies will end up being elected, I generally say that I really don't know. I do know what kind of voting patterns have existed in the past, but I don't think that those historical rules remain valid given how perceptions have changed enough over the past two decades.

Just because the voters may have overvalued Cups and wins in the past does not mean they are guaranteed to do so until the end of time. Some may have that kind of negative, defeatist approach when discussing the really borderline Hall of Famers, but I think that's unfair to the voting group because it implies they haven't learned anything over the last 20 years, and the progression of Vezina voting makes it absolutely clear that hockey insiders have in fact learned some lessons.

Every goalie now up for voting has an entire career's worth of official save percentage numbers and an entire career's worth of Vezina voting done by the league's general managers. Even though it's almost been three decades since those changes were introduced, Ed Belfour was just the second Hall of Fame goalie who played his entire career in the official save percentage and modern Vezina eras, joining Patrick Roy. Both Belfour and Roy would have been slam-dunk picks in any year, and therefore are hardly litmus tests for how the Hall factors in these new developments. We simply have not seen how much those two things have changed the game in terms of rating goalie careers. As such, any attempt to use prior voting results to predict how the voters rate future HOF goalie candidates is probably little more than guesswork.

I know for sure that there is at least one member on the Hall of Fame committee who doesn't care about goalie wins. A long time ago I linked an article by HOF member Michael Farber in which he quoted fellow HOF voter Serge Savard as saying that goalie was not the most important position, since in almost all cases teams make the goalie, rather than the other way around.

In the past a guy like Scotty Bowman may well have been a traditionalist and rated goalies based on wins, I don't know. But he was there when the Red Wings waived Chris Osgood, and he remains a special advisor to the Chicago Blackhawks, who turfed out their Stanley-Cup-winning netminder to save a few bucks. Seems pretty clear he's either been overruled multiple times by his team's management, or else Bowman is using evidence other than merely wins and Cups to rate goalies. If he's able to do that while helping to run a team, there's no reason he shouldn't apply the same logic when debating Hall of Fame candidates.

Any of the currently active writers or broadcasters on the committee have to be aware that teams are spending less money on goaltending, that mediocre goalies are winning Cups, and that save percentage is being used more than ever to rank and rate goalies. With less of a tradition of rating goalies based on winning in Europe, it's also possible that the two European voters would be more open to rating goalies who were good but not "winners" ahead of average goalies with terrific teammates.

All this is pure guesswork, of course, and the secretive nature of the Hall's election process means that we really have little evidence to go on. Maybe Mike Vernon fell one vote short last year, maybe nobody even mentioned his name, I don't know, but it is at least a positive sign that nobody has voted him in yet, and there doesn't seem to be a huge groundswell of support to get that to happen any time soon. I do think over the next few years we will have a lot better idea of how the Hall rates modern goalies. Perhaps surprisingly to some I am actually fairly optimistic that they are going to get the choices more right than wrong, although only time will tell on that one.

Tuesday, January 17, 2012

Vezina Trends

"First, he leads the NHL in the one stat that trumps all others: wins." (Scott Burnside, ESPN)

Going after Scott Burnside on goalie analysis is not dissimilar to shooting fish in a barrel, but my real beef is with how he is parroting the conventional wisdom that people within hockey consider wins to be extremely important. It might be Burnside's opinion that wins are the most vital stat, which is obviously misguided but he is allowed to personally believe whatever he wants. The problem is that when he explicitly claims to be handicapping the Vezina race, then at a minimum I would expect that he should be aware of what stats have actually been considered to be important in past voting.

Here is how the last 20 Vezina winners have ranked in five key goalie stats (GVT is Goals Versus Threshold):


Some readily apparent observations from the above table:

1. It is very rare for the consensus best goalie to win the most games. Only 4 of the last 20 Vezina winners led the league in wins. In contrast, for each of the other four stats, the Vezina winner was more likely to lead the league than not. Wins are in fact easily trumped by save percentage, GAA, shutouts, and GVT.

2. The data suggests that the emphasis on shutouts may be decreasing as well, although that could just be variance.

3. The 1996, 2003 and 2004 decisions stand out quite starkly relative to the others. The unwillingness of voters to rank non-playoff goalies as the best in the league was a big factor in the '96 and '04 votes, which is at least somewhat understandable although I disagree with the logic. That leaves the '03 Vezina as the most unusual result of the last two decades. Voters overlooked a playoff goalie that had 1-1-10-5-1 ranking based on the above table, a pattern that much more closely matches the overall averages than that year's winner.

4. The historical pattern that goalies require an excellent GAA to win a Vezina has continued. I would suggest that a low GAA on a strong defensive team seems to be the biggest source of error in the current voting, as those goalies appear on ballots much more frequently than average goalies that rack up a lot of wins on strong offensive teams.

Tuesday, January 10, 2012

Grant Fuhr and Effort

One of the things people like to say about goalies on dynasty teams is that they didn't try as hard when it didn't matter, and as a result their stats were understated. I think the evidence generally suggests that while there are some score effects from changing team strategies, goalies usually do try to keep the puck out of the net at all times. With only 30 starting jobs available goalie competition is fierce, which makes less likely that goalies would be willing to slack off while in the game. Even good teams have to fight to make the playoffs these days (see the '10-11 Chicago Blackhawks as an example), which means that there can be major team consequences for a netminder with a habit of trying to coast though a game here or there. Finally, in today's low-scoring environment there is not a lot of garbage time so goals against in blowouts simply will not have a material impact on a goalie's stats.

However, while those things may be true at the moment, they don't necessarily apply to results from several decades ago where the competitive balance and scoring level was much different than it is in today's salary capped NHL. I do think it is probably worth checking truly dominant teams to see whether there are some kind of unusual effects at play, since the incentives for players on those teams are not exactly the same as they are for everyone else.

For example, I suspect that some members of the 1980s Edmonton Oilers may not have been playing much of a 200 foot game during the second half of the regulation schedule during the peak of their dynasty simply because they were already dozens of points in front of everyone else in the standings. With 16 of 21 teams making the playoffs in those days, there was really nothing to left to do by that point in the season other than pad their offensive stats while trying to stay healthy and in good shape for another attempted Cup run.

From 1983-84 to 1987-88 (numbers from the Hockey Summary Project and Hockey Reference), there is a noticeable downward trend in Grant Fuhr's save percentages by month as the season wore on:

Oct: 1121 SA, .898
Nov: 1288 SA, .889
Dec: 1181 SA, .870
Jan: 1286 SA, .890
Feb: 1128 SA, .871
Mar: 1158 SA, .880
Apr: 232 SA, .871

It could be argued that Fuhr was experiencing fatigue, but that seems unlikely as a factor (other than potentially in 1987-88 when he played in 75 games) because he was usually used in a platoon scenario together with Andy Moog. On top of that, Fuhr's numbers jumped back up again to October levels as soon as the playoffs started.

To summarize:

Oct-Jan: 4876 SA, .887
Feb-Apr: 2518 SA, .875
Playoffs: 2268 SA, .899

By the end of January, the Oilers were always sitting very comfortably in the standings.

1983-84: 38-9-5, 41 pts ahead of 5th
1984-85: 36-9-6, 45 pts ahead of 5th
1985-86: 36-11-5, 40 pts ahead of 5th
1986-87: 34-14-11, 36 pts ahead of 5th
1987-88: 29-17-7, 26 pts ahead of 5th

The only season the Oilers were not ranked first overall in the league was 1987-88, where they sat in third place but were still far above the playoff cut line. In all five seasons the team had more wins at the end of January than the last place team in their division would finish with at the end of the season, meaning they could have lost every game they played after January 31 and still made the playoffs. In short, the Oilers had very little to play for as a team from February onwards in any of those seasons.

Looking at Andy Moog's monthly numbers, there is some reason to believe that the rest of the team was having a big impact on the late-season statistical slide:

Andy Moog, 1983-84 to 1987-88:
Oct-Jan: 3363 SA, .890
Feb-Apr: 1572 SA, .876
Playoffs: 343 SA, .866

Moog showed a very similar save percentage decline, suggesting that the Oilers as a group were less committed defensively once they had the division well locked up. Either that or Moog and Fuhr both had a similar lack of focus late in the season when the games became less meaningful. However, given that the two were mostly alternating starts, and were at least in some level of competition for the starting job come playoff time, I would guess that team defence may have been a more significant factor than the effort level of each individual goaltender.

I think there is a reasonable argument to be made that Grant Fuhr's true talent in terms of save percentage was understated by his regular season numbers in the mid-1980s. From 1983-84 to 1987-88, his overall regular season save percentage was .884, a decent mark given the league average of .876 over the same period of time. However, if his February to April numbers are excluded as not being representative of a team giving 100% defensive effort in front of him, with his playoff numbers substituted instead, Fuhr's save percentage would jump to .891, nearly doubling his advantage relative to league average. That rate would also rank Fuhr up near the top of the league over that period of time, rather than merely the upper middle of the pack.

It is still worth pointing out that Moog would be at .888 based on the same assumptions. That suggests that Edmonton's shot quality against was probably not nearly as bad as some suggest, at least when the team felt the game mattered and wanted to play defence, although Moog was an above-average goalie in his own right.

Other goalies may have suffered slightly from this effect as well during the unbalanced '80s, but it seems likely that it would have had the biggest impact in Edmonton given their prolific offence and incredible team success. Fuhr may still be a bit overrated by some fans who rate him as one of the main keys to the Oilers' championships, but this is at least some evidence that supports the contention that he had Hall of Fame talent in his prime.