I've got a new post up on my site on the relative importance of the four factors: How NBA Games Are Won
The basic idea is to assess how important each factor is by summing up its associated win probability added (with respect to the winning team). The results indicate that shooting is even more important than prior assessments of the four factors (although I'm not entirely convinced this isn't just a mathematical mirage). Here is how my results compare to prior estimates from Dean Oliver and EvanZ:
I offer my own speculation as to why this is at the end of the blog post, but would be curious to hear from the group here. I also have further breakouts by quarter. Each factor has about the same importance throughout the game, but the fourth quarter is more than twice as important than the third in terms of win probability added.
I think the discrepancy can be explained by the large amount of luck (random variance) at the single game level. Just doing some back-of-envelope calculations, the SD for random variance for a team with a true FG% of 50% would be about 7.5 points. That is, in 33% of games a team would score 7.5 or more "extra" points, or 7.5 fewer points, than predicted by their actual talent. In contrast, the SD for random variance in offensive rebounding should be about 3 points. So some of what you are picking up with WPA is just noise -- a team's shooting just happens to be "hot" or "cold" that day. That happens with the other 3 factors too, of course, but the impact is smaller.
By using team SD, Evan was looking at something much closer to the variation in true talent among teams. Thinking about it now, he probably should have regressed his SDs a bit based on respective sample sizes. But I doubt it would change his estimates much.
Guy wrote:I think the discrepancy can be explained by the large amount of luck (random variance) at the single game level. Just doing some back-of-envelope calculations, the SD for random variance for a team with a true FG% of 50% would be about 7.5 points. That is, in 33% of games a team would score 7.5 or more "extra" points, or 7.5 fewer points, than predicted by their actual talent. In contrast, the SD for random variance in offensive rebounding should be about 3 points. So some of what you are picking up with WPA is just noise -- a team's shooting just happens to be "hot" or "cold" that day. That happens with the other 3 factors too, of course, but the impact is smaller.
By using team SD, Evan was looking at something much closer to the variation in true talent among teams. Thinking about it now, he probably should have regressed his SDs a bit based on respective sample sizes. But I doubt it would change his estimates much.
Good points. One way I was trying to wrap my head around this is if the NBA settled tie games (after four quarters) with a coin flip instead of overtime. If I created a fifth factor called "Winning the Coin Flip", it would show up as an important factor in my win probability analysis. But it would be meaningless from a team evaluation perspective (all variation is due to luck). I think this is the point you're trying to make with FG%.
I'm trying to see if there are ways to account for the random variation within my win probability framework.
I'm not sure this will work, but you might want to try comparing the WPA of winning and losing teams. For example, sum the WPA of all teams >.600 WP%, and compare to those <.400. There will still be noise in that data, but proportionately much less. That might give you more meaningful proportions for the four factors, in terms of what distinguishes winning teams from losing teams.
Guy wrote:Do you post team-level WPA totals somewhere on your site?
As luck would have it, I've been working on such a feature. It's in soft launch mode now, but here's a link: http://stats.inpredictable.com/nba/ssnT ... order=DESC. See the "Win Probability Added" columns on the right. Note that these are adjusted to a per 100 possessions basis. This is, in effect, an explanation of a team's win-loss record. If you start at 0.50 and then add each WPA component, you should end up with the team's win-loss percentage (give or take a point).
When I look over the past four seasons, about 80-90% of the variance in team win-loss records can be explained just by field goal shooting (offense+defense). Each of the remaining three factors only explain about 10-20% of the variation in any given season. Which seems more consistent with my original result regarding the importance of shooting. Thoughts?
These %s are still based on the play by play model, right?
In past I've converted team four factor data into "factor wins". Seeing your WPAs is interesting, but I wonder if you'd also find use in presenting your findings in terms of wins added by factor? Simple enough to convert though.
Any interest in extending this type of play by play analysis to player level? Traditional boxscore metrics are based on season level data, so this would look different. EZPM is play by play based but not win probability based (I don't believe). RPM and BPM are adjusted based on game situation (for some games situations but not all, yes?) but your method adjusts every play so this would be new conceptually I believe, except for maybe some version of Wayne Winston's or perhaps an academic paper I am forgetting. (FWIW Matt Steinmetz of twitter / radio expressed interest in such a context sensitive player level metric. I shared this thread link with him but didn't hear anything further.)
One thing I've never seen regarding WPA-like metrics is whether it is actually more predictive than non-WPA versions. IOW, how much does it actually matter whether a player hits a shot or gets a rebound when the game is "on the line" vs. the same team being up by 20 points with 10 minutes left?
Has it been shown that this actually means anything? Is a 3-pt shot "worth more" later in the game than it is earlier in the game? Guys like Steinmetz certainly think so. And is a player who hits that 3pt shot later in the game more valuable than the one who hit it earlier?
The only way I think this can be resolved is by creating a WPA-based player valuation metric such as Crow is suggesting and then using it to make predictions. Either it is better or it isn't. I don't know. I guess I hope it is, but I'm a bit skeptical after reading Tango and others who have found that your best players tend to be your best clutch players, in general.
(One more related thought, if it is more predictive, is it due to anything more than simply accounting for strength of opponent or some other such "hidden" variable? For example, I could buy the argument that garbage time stats are less meaningful, but it could be that if we corrected for strength of opponent and teammates, we wouldn't need WPA to tell us that.)
Looking at 2012-13 and 2013-14, and using the SD of WPA in each category, I get this distribution among the four factors:
FG 49%
TO 20%
RB 17%
FT 14%
However, if we just sum the WPA for winning teams, we see a different story: about 70% of WPA generated by shooting, and 10% for each of the other factors. If the question is the relative impact of each factor on winning, I think the SDs tell the real story. WPA is telling us that success/failure on the other dimensions is having a real impact. However, the fact that winning teams in recent seasons owe almost all of their success to shooting efficiency (and preventing opponent efficiency) is certainly interesting. Is that just a coincidence, or has it been true in prior seasons as well? If it's generally true, then it suggests these factors are not independent, and the correlations may be very different. For example, OREB and oFG have been negatively correlated in recent years. If that is generally true, it may suggests that a team pays a price for getting offensive rebounds. Those rebounds still have value, but if they are systematically "packaged" with lower efficiency, then it's going to be very hard to build winning teams by boosting OREB.
Crow wrote:These %s are still based on the play by play model, right?
Correct.
In past I've converted team four factor data into "factor wins". Seeing your WPAs is interesting, but I wonder if you'd also find use in presenting your findings in terms of wins added by factor? Simple enough to convert though.
Not sure I follow. What is the difference between "WPA" and "wins added"?
Any interest in extending this type of play by play analysis to player level? Traditional boxscore metrics are based on season level data, so this would look different. EZPM is play by play based but not win probability based (I don't believe). RPM and BPM are adjusted based on game situation (for some games situations but not all, yes?) but your method adjusts every play so this would be new conceptually I believe, except for maybe some version of Wayne Winston's or perhaps an academic paper I am forgetting. (FWIW Matt Steinmetz of twitter / radio expressed interest in such a context sensitive player level metric. I shared this thread link with him but didn't hear anything further.)
By wins added I meant estimated wins added for a season as opposed to win probability added stated in %s. Your link is what I was interested. I get tunnel vision sometimes when making a comment in the moment. I may have briefly seen this part of your site but apparently forgot. I will spend some more time looking at these.
EvanZ wrote:One thing I've never seen regarding WPA-like metrics is whether it is actually more predictive than non-WPA versions. IOW, how much does it actually matter whether a player hits a shot or gets a rebound when the game is "on the line" vs. the same team being up by 20 points with 10 minutes left?
Has it been shown that this actually means anything? Is a 3-pt shot "worth more" later in the game than it is earlier in the game? Guys like Steinmetz certainly think so. And is a player who hits that 3pt shot later in the game more valuable than the one who hit it earlier?
The only way I think this can be resolved is by creating a WPA-based player valuation metric such as Crow is suggesting and then using it to make predictions. Either it is better or it isn't. I don't know. I guess I hope it is, but I'm a bit skeptical after reading Tango and others who have found that your best players tend to be your best clutch players, in general.
(One more related thought, if it is more predictive, is it due to anything more than simply accounting for strength of opponent or some other such "hidden" variable? For example, I could buy the argument that garbage time stats are less meaningful, but it could be that if we corrected for strength of opponent and teammates, we wouldn't need WPA to tell us that.)
At least in football, WPA is less consistent for QBs than EPA.