Is WP a legitimate stat?

[Neil Paine]

Well, here's my research. I took RAPM, ASPM, PER, Win Shares/48, & Wins Produced/48 data from 2001-2012, and tried to predict future team wins from how a team's players did in a metric 1, 2, and 3 seasons before. In all cases, players who had less than 250 minutes in the season from which I drew the metric (Y-1, Y-2, or Y-3) were assigned the league averages of 0.0 RAPM/ASPM, 15 PER, and 0.100 WS48/WP48. (This is essentially an update of the Rosenbaum study, with new metrics added.)

Here's an example of one team, the 2007 Orlando Magic:

Code: Select all

player_id       mp-1    mp      per-1   rapm-1  aspm-1  ws48-1  wp48-1  per-2   rapm-2  aspm-2  ws48-2  wp48-2  per-3   rapm-3  aspm-3  ws48-3  wp48-3
------------------------------------------------------------------------------------------------------------------------------------------------------
howardw01       3021    3023    19.3     4.5     1.5     0.137   0.211   17.2    -0.2     0.9    0.131   0.227   15.0    0.0     0.0     0.100   0.100
nelsoja01       1784    2331    19.5     0.6     2.1     0.132   0.121   14.5    -2.5    -0.8    0.074   0.045   15.0    0.0     0.0     0.100   0.100
turkohe01       2615    2268    16.7     0.4     0.9     0.140   0.138   16.0    -0.6    -1.5    0.092   0.041   14.1    2.5     1.7     0.155   0.164
hillgr01         613    2009    19.0    -0.1     1.2     0.130   0.117   20.0     0.0     1.6    0.136   0.146   15.0    0.0     0.0     0.100   0.100
milicda01        767    1913    15.2    -1.1    -1.1     0.073   0.044    4.7    -2.0    -6.5   -0.031  -0.208   15.0    0.0     0.0     0.100   0.100
battito01       2215    1575    12.2    -1.5    -0.7     0.083   0.037    8.6    -1.8    -1.6    0.051   0.010   11.9   -1.1    -2.0     0.074   0.045
doolike01       1137    1435    12.6    -2.2    -1.9     0.046   0.008   10.1    -2.9    -2.7    0.061  -0.008   10.0   -3.6    -2.7     0.013  -0.042
arroyca01       1194    1304    15.6    -1.2    -1.0     0.107   0.088   11.3    -3.9    -2.2    0.043   0.000   16.8   -0.9     0.9     0.112   0.074
arizatr01        999    1278    11.8     1.2    -0.9     0.043   0.154   13.3    -0.5    -1.4    0.073   0.118   15.0    0.0     0.0     0.100   0.100
boganke01       1912     990    10.7    -1.3    -1.7     0.058   0.063   10.4    -1.5    -3.4    0.002  -0.019   11.0   -2.5    -2.6     0.034   0.110
redicjj01          0     622    15.0     0.0     0.0     0.100   0.100   15.0     0.0     0.0    0.100   0.100   15.0    0.0     0.0     0.100   0.100
outlabo01        355     460    11.8    -1.5    -0.4     0.101   0.149   15.0     0.0     0.0    0.100   0.100   13.1    2.8     1.1     0.107   0.167
dienetr01        246     288    15.0     0.0     0.0     0.100   0.100   15.0     0.0     0.0    0.100   0.100   15.0    0.0     0.0     0.100   0.100
garripa01        938     277     9.1    -4.0    -3.0     0.062  -0.031    9.1    -4.4    -3.6    0.054  -0.070   15.0    0.0     0.0     0.100   0.100
augusja01          0       7    15.0     0.0     0.0     0.100   0.100   15.0     0.0     0.0    0.100   0.100   15.0    0.0     0.0     0.100   0.100

That team had the following minute-weighted averages in each metric:

Code: Select all

year_id team_id wpct    per-1   rapm-1  aspm-1  ws48-1  wp48-1  per-2   rapm-2  aspm-2  ws48-2  wp48-2  per-3   rapm-3  aspm-3  ws48-3  wp48-3
----------------------------------------------------------------------------------------------------------------------------------------------
2007    ORL     0.488   15.865  0.251   0.136   0.103   0.109   13.379  -1.367  -1.383  0.074   0.052   14.153  -0.181  -0.204  0.095   0.093

Do that for every team, and run the correlations between each metric and winning percentage in Year Y, and this is what you get:

Code: Select all

       --Correlation vs Wins-- 
Metric   Y-1     Y-2     Y-3    
---------------------------------
PER     0.638   0.546   0.502
RAPM    0.751   0.646   0.568
ASPM    0.723   0.610   0.532
WS/48   0.694   0.547   0.494
WP/48   0.654   0.492   0.440

If we're sorting that by correlation in each period of time:

Code: Select all

Metric   Y-1            Metric   Y-2            Metric   Y-3
-------------           -------------           -------------
RAPM    0.751           RAPM    0.646           RAPM    0.568
ASPM    0.723           ASPM    0.610           ASPM    0.532
WS/48   0.694           WS/48   0.547           PER     0.502
WP/48   0.654           PER     0.546           WS/48   0.494
PER     0.638           WP/48   0.492           WP/48   0.440

Wins Produced does beat PER in prediction if we look at player performance in each metric last year. But the more time that goes by, the worse and worse WP gets relative to the competition. For anything beyond one year ago, you'd be better off knowing each player's PER (much less his RAPM, ASPM, and WS48) than his WP48 if you wanted to make accurate predictions.

[v-zero]

Excellent stuff Neil, very interesting. Any chance of including previous year SRS as a sanity check?

[DSMok1]

Excellent, Neil!

Do you have some idea why you see the trends you do? It is my thought, as I mentioned, that WP is by far the worst when dealing with "new" lineups, but is fine enough (as any stat summing to eff. dif. would be) if we are looking at the same lineups as the previous year.

Any way to look at only lineups that are "new"? Looking at Y+3 would be a proxy for that; almost every lineup would be new.

[Neil Paine]

Excellent stuff Neil, very interesting. Any chance of including previous year SRS as a sanity check?

Sure, here were the correlations vs WPct in Year Y:

Code: Select all

SRS from...     Corr vs Wins Y
------------------------------
  Y-1               0.616
  Y-2               0.405
  Y-3               0.247

So knowing any of these metrics does at least help you predict better than if you just knew how the team as a whole did the previous year, but remember as well that the metrics have a huge unfair advantage vs SRS: they know the distribution of minutes to each player ahead of time.

[Neil Paine]

Excellent, Neil!

Do you have some idea why you see the trends you do? It is my thought, as I mentioned, that WP is by far the worst when dealing with "new" lineups, but is fine enough (as any stat summing to eff. dif. would be) if we are looking at the same lineups as the previous year.

Any way to look at only lineups that are "new"? Looking at Y+3 would be a proxy for that; almost every lineup would be new.

I think that's the case as well. WP relies so heavily on team adjustments and other aspects of production that don't "belong" to the player himself (overvaluing rebounding, which has been shown by Phil Birnbaum, Guy Molyneaux & others to be more a product of coaching/strategy/role on the team than player talent; undervaluing shot creation, which is a more "portable" skill; etc.) that when you take the player out of the context he put up his WP in, the metric becomes completely useless. Even for Win Shares, the team defensive adjustment seems to "wear off" over time, to the point that you should be indifferent whether to use WS/48 or PER after 2-3 years.

I agree on testing using teams with a lot of roster turnover. That's something I've espoused here and here in the past, but haven't really tested for these advanced metrics. Perhaps we could have the seeds for a future ESPN article!

[v-zero]

Last sanity-check suggestion: give each player a rating equal to one-fifth their team's rating per 100 pos or per 48 minutes, and use that as a very simple player metric. That will fulfil the famed 95% (or whatever it is) R-squared that wins produced gets in-sample, and could be considered the 'baseline' for any metric that sums to point differential.

Thanks for everything you've done so far, very useful - a good use of the BBRef database! Also congrats to DSMok1, very nice result for ASPM there.

[deepak]

Thanks Neil.

So, you show predictions based on player stats 1 season ago, 2 season ago, and 3 seasons ago. For which season(s) did you get those final correlation results?

Not sure which version of RAPM you looked at, but if its the version using the previous year as a prior wouldn't it be at an advantage, since it is inherently incorporating multiple years of information?

I'm curious how each stat would fare if the prediction was made based on multiple years of information. E.g. weighted Y1 + 0.5*Y2 + 0.25*Y3.

For roster turnover, here are some results I got recently that might help. For each team/season, it gives the % of players on the roster who've played at least 20% of the team's available minutes and also played at least 1 minute for the team in the preceding year. I would suggest that teams for which the percentage is < 50% may be of particular interest.

Code: Select all

   Tm      2013    2012    2011    2010    2009    2008    2007    2006    2005    2004    2003    avg     
   okl     87.5%   100.0%  100.0%  66.7%   50.0%   60.0%   88.9%   70.0%   80.0%   81.8%   75.0%   78.2%   
   san     83.3%   80.0%   88.9%   60.0%   80.0%   88.9%   90.0%   80.0%   60.0%   62.5%   70.0%   76.7%   
   lal     40.0%   70.0%   77.8%   87.5%   100.0%  80.0%   75.0%   77.8%   22.2%   63.6%   88.9%   71.2%   
   det     70.0%   66.7%   83.3%   50.0%   72.7%   66.7%   77.8%   85.7%   85.7%   57.1%   66.7%   71.1%   
   uta     75.0%   72.7%   50.0%   90.0%   88.9%   87.5%   77.8%   63.6%   50.0%   66.7%   55.6%   70.7%   
   den     88.9%   63.6%   77.8%   77.8%   66.7%   100.0%  66.7%   90.0%   60.0%   40.0%   36.4%   69.8%   
   ind     66.7%   66.7%   72.7%   60.0%   44.4%   80.0%   50.0%   72.7%   81.8%   80.0%   90.0%   69.6%   
   por     66.7%   62.5%   77.8%   77.8%   66.7%   66.7%   50.0%   66.7%   72.7%   77.8%   72.7%   68.9%   
   bos     55.6%   55.6%   100.0%  80.0%   100.0%  50.0%   81.8%   66.7%   60.0%   40.0%   66.7%   68.8%   
   hou     44.4%   77.8%   72.7%   60.0%   72.7%   87.5%   87.5%   55.6%   40.0%   83.3%   66.7%   68.0%   
   phi     55.6%   88.9%   66.7%   54.5%   70.0%   66.7%   66.7%   87.5%   80.0%   66.7%   44.4%   68.0%   
   bro     60.0%   54.5%   30.0%   72.7%   40.0%   80.0%   72.7%   100.0%  44.4%   88.9%   88.9%   66.6%   
   sac     75.0%   54.5%   63.6%   55.6%   75.0%   66.7%   88.9%   50.0%   55.6%   66.7%   77.8%   66.3%   
   dal     44.4%   63.6%   77.8%   66.7%   77.8%   72.7%   77.8%   80.0%   50.0%   28.6%   80.0%   65.4%   
   mia     87.5%   80.0%   45.5%   77.8%   60.0%   70.0%   90.9%   50.0%   44.4%   42.9%   70.0%   65.4%   
   atl     45.5%   54.5%   100.0%  87.5%   75.0%   66.7%   70.0%   55.6%   20.0%   60.0%   75.0%   64.5%   
   pho     50.0%   60.0%   60.0%   88.9%   62.5%   66.7%   100.0%  37.5%   37.5%   77.8%   66.7%   64.3%   
   mem     60.0%   66.7%   77.8%   50.0%   50.0%   62.5%   70.0%   55.6%   90.0%   70.0%   54.5%   64.3%   
   chi     66.7%   90.9%   36.4%   75.0%   77.8%   72.7%   55.6%   77.8%   40.0%   37.5%   70.0%   63.7%   
   gsw     37.5%   54.5%   62.5%   71.4%   54.5%   100.0%  66.7%   80.0%   70.0%   25.0%   77.8%   63.6%   
   nyk     55.6%   50.0%   33.3%   87.5%   71.4%   80.0%   81.8%   40.0%   66.7%   44.4%   88.9%   63.6%   
   cle     60.0%   70.0%   54.5%   63.6%   88.9%   80.0%   90.0%   44.4%   40.0%   40.0%   54.5%   62.4%   
   orl     33.3%   88.9%   60.0%   60.0%   54.5%   75.0%   90.0%   87.5%   22.2%   44.4%   70.0%   62.4%   
   nor     40.0%   54.5%   37.5%   63.6%   66.7%   87.5%   55.6%   50.0%   30.0%   70.0%   88.9%   58.6%   
   tor     50.0%   81.8%   60.0%   30.0%   70.0%   70.0%   40.0%   50.0%   75.0%   37.5%   66.7%   57.4%   
   lac     50.0%   50.0%   44.4%   75.0%   20.0%   60.0%   87.5%   55.6%   54.5%   45.5%   72.7%   55.9%   
   cha     44.4%   54.5%   77.8%   55.6%   50.0%   62.5%   72.7%   75.0%   0.0%                    54.7%   
   min     55.6%   72.7%   45.5%   40.0%   50.0%   44.4%   62.5%   44.4%   88.9%   22.2%   70.0%   54.2%   
   mil     80.0%   54.5%   50.0%   36.4%   60.0%   55.6%   40.0%   45.5%   63.6%   44.4%   63.6%   54.0%   

Here is another table which shows the percentage of player-minute allocations in the current season that did not exist in the preceding season. So, for each player, I took the lesser of X and Y where X is the percentage of minutes he played for the team this season and Y is the percentage of minutes he played for the team in the preceding season. Add up the percentages for each team, and I get this result. So if all the players from the preceding season returned, but their minutes allocation changed, this will reflect that.

Code: Select all

   Tm      2013    2012    2011    2010    2009    2008    2007    2006    2005    2004    2003    avg
   san     67.3%   63.1%   74.4%   49.7%   66.2%   79.7%   74.8%   73.2%   51.5%   54.3%   62.5%   65.1%
   okl     70.7%   74.4%   81.9%   61.7%   52.7%   55.7%   63.8%   63.7%   64.8%   57.2%   63.4%   64.5%
   det     62.7%   62.0%   68.5%   43.4%   67.9%   68.7%   69.5%   86.6%   65.6%   36.5%   62.0%   63.0%
   lal     39.0%   67.9%   79.7%   70.8%   73.7%   64.4%   64.2%   46.1%   22.4%   53.8%   83.4%   60.5%
   pho     42.8%   61.7%   59.3%   64.5%   53.0%   74.6%   76.7%   36.8%   48.9%   63.0%   63.9%   58.7%
   dal     36.4%   59.4%   62.5%   62.4%   64.4%   70.7%   69.6%   72.5%   43.8%   35.4%   67.3%   58.6%
   phi     45.3%   84.1%   57.9%   56.0%   74.1%   62.2%   55.4%   61.0%   48.0%   53.2%   40.8%   58.0%
   uta     65.2%   56.6%   44.2%   75.7%   76.2%   75.0%   60.8%   52.4%   38.1%   34.1%   57.7%   57.8%
   mem     61.6%   61.4%   71.0%   58.5%   41.6%   47.5%   47.4%   49.6%   77.6%   58.8%   56.9%   57.4%
   sac     76.6%   46.1%   54.3%   46.8%   58.4%   55.5%   67.5%   42.3%   42.5%   58.9%   77.4%   56.9%
   bos     53.4%   53.1%   62.5%   75.5%   81.9%   36.2%   54.9%   56.8%   50.2%   39.8%   59.0%   56.7%
   ind     54.5%   58.1%   58.5%   49.0%   46.7%   58.7%   42.0%   64.3%   59.3%   65.6%   63.8%   56.4%
   chi     52.3%   75.8%   45.1%   52.1%   61.2%   68.1%   61.8%   71.2%   37.8%   37.2%   43.1%   55.1%
   atl     41.4%   54.3%   83.5%   73.3%   65.4%   63.3%   69.5%   45.5%   8.8%    44.1%   49.0%   54.4%
   por     45.2%   50.9%   55.4%   56.3%   65.5%   52.9%   41.2%   47.6%   51.4%   57.4%   72.1%   54.2%
   den     59.3%   43.5%   67.8%   70.7%   50.2%   62.9%   46.0%   72.7%   62.1%   28.2%   20.0%   53.0%
   cle     46.7%   50.2%   45.9%   61.9%   56.8%   67.5%   75.2%   54.6%   42.4%   40.3%   38.6%   52.7%
   mia     79.0%   63.9%   38.8%   61.9%   38.8%   43.6%   77.0%   42.9%   38.7%   37.5%   41.0%   51.2%
   hou     19.8%   55.8%   57.4%   47.8%   58.0%   70.5%   59.1%   45.8%   28.6%   58.8%   59.3%   51.0%
   tor     50.8%   67.8%   47.2%   35.3%   64.3%   62.8%   36.6%   48.7%   53.5%   31.6%   50.7%   49.9%
   bro     27.4%   34.2%   28.7%   58.8%   28.3%   62.6%   60.0%   61.5%   39.3%   69.2%   73.7%   49.4%
   nyk     44.7%   37.0%   28.9%   63.3%   45.8%   63.5%   67.6%   33.1%   41.6%   40.6%   77.1%   49.4%
   orl     28.6%   68.5%   51.8%   51.4%   56.3%   58.2%   64.4%   66.8%   5.1%    29.6%   56.5%   48.8%
   nor     33.6%   38.8%   38.0%   54.6%   64.9%   56.2%   47.8%   28.7%   34.6%   65.2%   67.8%   48.2%
   gsw     36.1%   46.6%   38.0%   43.9%   40.2%   66.1%   46.1%   75.7%   47.5%   22.6%   66.7%   48.1%
   lac     49.3%   43.0%   33.8%   53.7%   19.5%   61.8%   74.0%   45.0%   54.1%   39.5%   52.5%   47.8%
   min     47.3%   67.0%   29.6%   32.0%   50.5%   25.4%   57.6%   53.8%   68.5%   27.2%   61.7%   47.3%
   cha     43.7%   38.4%   55.2%   49.8%   50.1%   56.6%   52.9%   53.8%   0.0%                    44.5%
   mil     59.1%   45.1%   53.4%   36.3%   34.1%   49.6%   32.3%   36.4%   57.8%   24.6%   59.3%   44.4%

[Neil Paine]

Thanks Neil.

So, you show predictions based on player stats 1 season ago, 2 season ago, and 3 seasons ago. For which season(s) did you get those final correlation results?

Not sure which version of RAPM you looked at, but if its the version using the previous year as a prior wouldn't it be at an advantage, since it is inherently incorporating multiple years of information?

I'm curious how each stat would fare if the prediction was made based on multiple years of information. E.g. weighted Y1 + 0.5*Y2 + 0.25*Y3.

The 1-year correlations were against team-season winning %s from 2002-2012. 2-year, 2003-12; 3-year, 2004-12.

I used the RAPMs you can still find on Jerry's site (i.e., http://stats-for-the-nba.appspot.com/ratings/2010.html). Those only use previous seasons as a prior, so I never "predicted" using any information that you wouldn't have had available before the season being predicted. In other words, when predicting the 2007-08 season, I used Jerry's 2006-07 RAPM, and Daniel's 2006-07 ASPM, Justin's 2006-07 Win Shares, etc. The only quibble you could have in this department is that ASPM was trained on a RAPM dataset that spanned all of 2001-12, so in that case I'm using a model that was built from data that wouldn't have been completely available at the time, but the model itself isn't using inputs with any future knowledge.

I can test the predictive accuracy of a weighted version of each metric at some point; that's been suggested a couple of times and I definitely think that's a reasonable request. What should the weighting be, though? The Simple Projection System uses 6-3-1 with 1000 minutes of the mean thrown in as well. I can do it that way if everyone deems that a fair measure. Then I wouldn't even have to set everyone whose MP-1 < 250 to average -- the regression-to-the-mean term would take of that automatically.

[deepak]

I used the RAPMs you can still find on Jerry's site (i.e., http://stats-for-the-nba.appspot.com/ratings/2010.html). Those only use previous seasons as a prior, so I never "predicted" using any information that you wouldn't have had available before the season being predicted. In other words, when predicting the 2007-08 season, I used Jerry's 2006-07 RAPM, and Daniel's 2006-07 ASPM, Justin's 2006-07 Win Shares, etc. The only quibble you could have in this department is that ASPM was trained on a RAPM dataset that spanned all of 2001-12, so in that case I'm using a model that was built from data that wouldn't have been completely available at the time, but the model itself isn't using inputs with any future knowledge.

I can test the predictive accuracy of a weighted version of each metric at some point; that's been suggested a couple of times and I definitely think that's a reasonable request. What should the weighting be, though? The Simple Projection System uses 6-3-1 with 1000 minutes of the mean thrown in as well. I can do it that way if everyone deems that a fair measure. Then I wouldn't even have to set everyone whose MP-1 < 250 to average -- the regression-to-the-mean term would take of that automatically.

That's sounds good. I was just curious about RAPM doing relatively better than the other stats. Its a result that surprised me, frankly, because I usually think of all APM variants as have some noise problem. Maybe I underestimated how good a job J.E. did in stabilizing the results. It uses a 1-year prior, but really even that is based on multiple past seasons worth of information, isn't it? So, I think maybe looking at a weighted version for all stats could be a more "fair" comparison.

[v-zero]

I would suggest that 9:3:1 is a better ratio, but otherwise that sounds like a good idea. You could also simply include all three years of metrics in a single regression and have the regression decide the weights.

[xkonk]

It uses a 1-year prior, but really even that is based on multiple past seasons worth of information, isn't it? So, I think maybe looking at a weighted version for all stats could be a more "fair" comparison.

That's my intuition as well, and I think was mentioned in one of the Sport Skeptic posts in that series. As Neil mentions, though, it would be a bit of an enterprise to figure out what those weights should be.

Another idea to keep in mind is that the varieties of RAPM (and by extension ASPM) were built to be predictive models while PER, WS, and WP were built to be explanatory models. This post http://sportskeptic.wordpress.com/2011/ ... g-problem/ showed that simply regressing WP scores a bit toward the mean improved its predictive ability. That could be a sign that the latter group of models are suffering because they are more likely to have outlier scores that (possibly) describe what actually happened but are unlikely to occur again.

[v-zero]

If they are accurately measuring what happened, and they are representative of a measurement of some underlying statistical quantity, then their mean represents an unbiased estimate of that quantity (assuming a few things hold). What that implies is that they should be the best unbiased estimate of themselves for future prediction, so if they fail to predict then they also are failing to accurately explain.

I.E. If you claim to measure how well a player played, then that measurement should also predict how well that player plays in future, unless the player's performance is drawn entirely at random - if player game-to-game performance isn't entirely random, then accurate measurements of it should lead to better future predictions. Ergo if WP predicts badly in comparison to others it is because it is less able to accurately measure the level of play of players from game to game, and hence fails because it is flawed, not because it creates outliers. Maybe it does create outliers, but if it does it is because it is flawed.

[TheSpiceWeasel]

Not sure which version of RAPM you looked at, but if its the version using the previous year as a prior wouldn't it be at an advantage, since it is inherently incorporating multiple years of information?

Wait, so you're saying that using more information can confer an advantage? No way!!!

[DSMok1]

I used the RAPMs you can still find on Jerry's site (i.e., http://stats-for-the-nba.appspot.com/ratings/2010.html). Those only use previous seasons as a prior, so I never "predicted" using any information that you wouldn't have had available before the season being predicted.

Note: that RAPM is actually xRAPM, which uses box-score information in the prior (which is a large reason why it is so stable). Search this forum for more information on that change.

Another idea to keep in mind is that the varieties of RAPM (and by extension ASPM) were built to be predictive models

ASPM was not built on a RAPM basis, it was built on a long-term average APM as its basis (ASPM was built to be descriptive of APM, and it's predictive ability hinges on it being better at being accurate descriptively).

[jbrocato23]

Note: that RAPM is actually xRAPM, which uses box-score information in the prior (which is a large reason why it is so stable). Search this forum for more information on that change.

Yes, and this distinction is very important because the use of the box score (as a prior and blended into the results themselves) provides stability, which ultimately helps the predictive ability of the metric.

J.E. discusses xRAPM in this thread: viewtopic.php?f=2&t=8025&start=30