Adj. +/- peaks&valleys; perspective&pathforward 08
Posted: Mon Apr 18, 2011 4:54 am
Author Message
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Sat Sep 13, 2008 12:59 am Post subject: Adj. +/- peaks & valleys; perspective & path forward Reply with quote
Looking at the last 3 season datasets the only players I see who were -5 or worse the last 3 straight years were Warrick and Wilkins.
Does that suggest that few players are consistently really and truly that bad or just that odds are the measure is not going to catch them 3 times in a row?
For how many cases are the strong bad marks significantly influenced by bad luck? Or especially bad fit / management as opposed to fundamental strong negativity? Given the very low number consistently that bad in the population does that suggest that many or most of the players with marks worse than -5 are getting an unfavorable reading relative to true?
Only 3 players were +8 or better the last 3 seasons in a row. KG, Kobe and LeBron. Do many / most? above that mark for a season or two have estimates that are shining too brightly or it is real but reflects cases of players who happened to be especially well optimized or just that estimates may not have captured their true superior impact 3 times in a row? How much share for each explanation ?
How sure are we that the 2 consistently low on 3 estimates are really that low or the 3 consistently that high are truly that high? What can we say about the range in true? Is it about that size or is it more likely less or greater?
From 2006-7 to 2007-8 Ryan Gomes improved his adjusted +/- score by over 15 points. From 2005-6 to 2006-7 Richard Hamilton did too. The former is probably a more extreme swing in estimate than true and the later case has already been noted as an outlier. Andrew Bynum improved about 20 points from 2005-6 to 2007-8 from very very poor to modest positive impact. Probably more extreme swing in estimate than true but maybe not and regardless what is true or next true? I guess that example and the general thrust of this comment gives hope in the case of Stuckey.
How likely is a 5 point move in estimate reflecting a 5 or 3 point move in true? The standard errors allow that calculation but it is rarely done in discussions. I guess for any change from one estimate to the next the chances are 50/50 the true change is more or less. The probablity that a 5 point move in estimate reflects at least a 3 point move in true may get closer to 65% but still leaves a fair share of doubt.
I think that these other questions / comments can be replaced with statistically sound probabilistic statements too if given the work-up and would welcome that and a better summary or commentary if any qualified wanted to address parts or all of it. I think it would inform the use of adjusted +/- information and give better perspective on movements in estimates.
Last edited by Mountain on Sat Sep 13, 2008 5:26 pm; edited 4 times in total
Back to top
View user's profile Send private message
Neil Paine
Joined: 13 Oct 2005
Posts: 774
Location: Atlanta, GA
PostPosted: Sat Sep 13, 2008 1:51 am Post subject: Reply with quote
Let's face it, due to the nature of regressing on such (relatively) small sample sizes, APM is an extremely volatile metric; the best players in one season can turn up as negative the next season, even despite similar box-score stats. That's why Dan originally advocated balancing pure APM with stat +/-, the regression of pure APM on boxscore stats -- it still measures performance better than existing metrics like TENDEX, Wins Produced, etc., but it's also not subject to the large standard errors and wild fluctuations of pure APM. So, to me, the combination of pure and statistical +/- suggested by Dan (each weighted by the degree of standard error in the pure APM estimate) would give the clearest plus-minus-related picture of a player's on-court performance. Because, frankly, the standard errors of the pure regression results are much too large, and the results can fluctuate much too wildly from season to season to use as a standalone metric.
Back to top
View user's profile Send private message Visit poster's website
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Sat Sep 13, 2008 1:32 pm Post subject: Reply with quote
Yes a blend of pure and statistical is more sensible & supportable than single season pure adjusted and I have advocated that though I have previously fallen back into using just the pure, with caution, for simplicity in discussion. I think I'll try to stop doing that- for now.
You have updated statistical according to Dan R.'s weights and I think David Sparks' metric is of this standard too. Whether the right blend on average is roughly 4-1 statistical over pure or something different or if any other ingredients are added to the meta-metric including subjective scores it is probably time to try to move beyond pure adjusted use in discussions.
The danger of an "overall +/-" composed of statistical and pure, as has been noted before, is that offensive impact gets overweighted as shot defense is not included. In the predictions thread your meta-metric uses defense rating though I'd ask if you are sure defense is properly weighted to 50% of the total score. How much did defense weigh in your subjective component? I'd be interested in the full scoring breakdown. I might try to incorporate counterpart defense into a meta-metric of my own later.
When Steve has time to finish 3 or more season combined adjusted +/- or a adjusted +/- version 2.0 that will be interesting to see/use, if either or both steps reduce the errors significantly for most players.
Back to top
View user's profile Send private message
Scott S
Joined: 10 Feb 2008
Posts: 46
Location: East Rutherford, NJ
PostPosted: Wed Sep 17, 2008 12:52 pm Post subject: Reply with quote
Trying to lower this variation in (estimated) adjusted +/-, I compared data for players with adjusted +/-'s available in both 2007 and 2008. Using maximum full credibility of 2730 minutes for statistical +/- (after applying credibility to adjusted +/-) and applying the rest to a normalized adjusted +/- rating from NBA 2k8, which I assume to be subjective, I get the following absolute mean errors:
Code:
Estimate This Year Other Year
Model 1.91 3.54
Zero 3.87 3.87
Adj +/- 0.00 4.45
The average absolute error between years for my model was 2.20, less than half of Adjusted +/-'s absolute mean error.
Note that just because the absolute mean error of assuming everyone has a 0 adjusted +/- produces a lower absolute mean error than assuming they stay the same from year to year, definately does not mean that it is a better model.
Back to top
View user's profile Send private message Visit poster's website
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Thu Sep 18, 2008 8:53 am Post subject: Reply with quote
Can you clarify for me even in rough terms what the weight of statistical ends up being vs pure adjusted in your better performing model?
Last edited by Mountain on Thu Sep 18, 2008 5:21 pm; edited 1 time in total
Back to top
View user's profile Send private message
Scott S
Joined: 10 Feb 2008
Posts: 46
Location: East Rutherford, NJ
PostPosted: Thu Sep 18, 2008 4:08 pm Post subject: Reply with quote
Since weights for adjusted plus minus' tend to be around 25%-50%, if I assume full credibility for minutes of 2730 attributable to statistical plus minus, the formula for credibility is, at most, (minutes/2730)^.5. (Total weights don't exceed 1).
The model I used, for the population I compared, it is almost exactly the same as Dan R's weights of adjusted and statistical +/-'s, since most players have enough minutes not to apply any weight to the subjective measurement. The model I mentioned would have more additional usefulness for rating those who don't have many minutes at all.
Upon reviewing, I noticed a slight error in my average absolute error term, there was actually less than 1% of prediction models that predicted itself, since I used the same subjective rating for both years, removing this would only increase the error to about 2.22. Obviously, this model is more consistent. It is only more accurate because of the inconsistency from adjusted plus minus. If we had a much larger, more varied sample to use for adjusted +/-, that would be the most accurate predictor.
Note that applying some additional weight to subjective plus minus would make the model more accurate since subjective ratings combine the consistency lacking from adjusted +/- and the intangibles lacking from statistical +/-. Statisical +/- tends to be the best predictor, followed by Subjective ratings (provided they are good) and then adjusted +/-.
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Sat Sep 13, 2008 12:59 am Post subject: Adj. +/- peaks & valleys; perspective & path forward Reply with quote
Looking at the last 3 season datasets the only players I see who were -5 or worse the last 3 straight years were Warrick and Wilkins.
Does that suggest that few players are consistently really and truly that bad or just that odds are the measure is not going to catch them 3 times in a row?
For how many cases are the strong bad marks significantly influenced by bad luck? Or especially bad fit / management as opposed to fundamental strong negativity? Given the very low number consistently that bad in the population does that suggest that many or most of the players with marks worse than -5 are getting an unfavorable reading relative to true?
Only 3 players were +8 or better the last 3 seasons in a row. KG, Kobe and LeBron. Do many / most? above that mark for a season or two have estimates that are shining too brightly or it is real but reflects cases of players who happened to be especially well optimized or just that estimates may not have captured their true superior impact 3 times in a row? How much share for each explanation ?
How sure are we that the 2 consistently low on 3 estimates are really that low or the 3 consistently that high are truly that high? What can we say about the range in true? Is it about that size or is it more likely less or greater?
From 2006-7 to 2007-8 Ryan Gomes improved his adjusted +/- score by over 15 points. From 2005-6 to 2006-7 Richard Hamilton did too. The former is probably a more extreme swing in estimate than true and the later case has already been noted as an outlier. Andrew Bynum improved about 20 points from 2005-6 to 2007-8 from very very poor to modest positive impact. Probably more extreme swing in estimate than true but maybe not and regardless what is true or next true? I guess that example and the general thrust of this comment gives hope in the case of Stuckey.
How likely is a 5 point move in estimate reflecting a 5 or 3 point move in true? The standard errors allow that calculation but it is rarely done in discussions. I guess for any change from one estimate to the next the chances are 50/50 the true change is more or less. The probablity that a 5 point move in estimate reflects at least a 3 point move in true may get closer to 65% but still leaves a fair share of doubt.
I think that these other questions / comments can be replaced with statistically sound probabilistic statements too if given the work-up and would welcome that and a better summary or commentary if any qualified wanted to address parts or all of it. I think it would inform the use of adjusted +/- information and give better perspective on movements in estimates.
Last edited by Mountain on Sat Sep 13, 2008 5:26 pm; edited 4 times in total
Back to top
View user's profile Send private message
Neil Paine
Joined: 13 Oct 2005
Posts: 774
Location: Atlanta, GA
PostPosted: Sat Sep 13, 2008 1:51 am Post subject: Reply with quote
Let's face it, due to the nature of regressing on such (relatively) small sample sizes, APM is an extremely volatile metric; the best players in one season can turn up as negative the next season, even despite similar box-score stats. That's why Dan originally advocated balancing pure APM with stat +/-, the regression of pure APM on boxscore stats -- it still measures performance better than existing metrics like TENDEX, Wins Produced, etc., but it's also not subject to the large standard errors and wild fluctuations of pure APM. So, to me, the combination of pure and statistical +/- suggested by Dan (each weighted by the degree of standard error in the pure APM estimate) would give the clearest plus-minus-related picture of a player's on-court performance. Because, frankly, the standard errors of the pure regression results are much too large, and the results can fluctuate much too wildly from season to season to use as a standalone metric.
Back to top
View user's profile Send private message Visit poster's website
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Sat Sep 13, 2008 1:32 pm Post subject: Reply with quote
Yes a blend of pure and statistical is more sensible & supportable than single season pure adjusted and I have advocated that though I have previously fallen back into using just the pure, with caution, for simplicity in discussion. I think I'll try to stop doing that- for now.
You have updated statistical according to Dan R.'s weights and I think David Sparks' metric is of this standard too. Whether the right blend on average is roughly 4-1 statistical over pure or something different or if any other ingredients are added to the meta-metric including subjective scores it is probably time to try to move beyond pure adjusted use in discussions.
The danger of an "overall +/-" composed of statistical and pure, as has been noted before, is that offensive impact gets overweighted as shot defense is not included. In the predictions thread your meta-metric uses defense rating though I'd ask if you are sure defense is properly weighted to 50% of the total score. How much did defense weigh in your subjective component? I'd be interested in the full scoring breakdown. I might try to incorporate counterpart defense into a meta-metric of my own later.
When Steve has time to finish 3 or more season combined adjusted +/- or a adjusted +/- version 2.0 that will be interesting to see/use, if either or both steps reduce the errors significantly for most players.
Back to top
View user's profile Send private message
Scott S
Joined: 10 Feb 2008
Posts: 46
Location: East Rutherford, NJ
PostPosted: Wed Sep 17, 2008 12:52 pm Post subject: Reply with quote
Trying to lower this variation in (estimated) adjusted +/-, I compared data for players with adjusted +/-'s available in both 2007 and 2008. Using maximum full credibility of 2730 minutes for statistical +/- (after applying credibility to adjusted +/-) and applying the rest to a normalized adjusted +/- rating from NBA 2k8, which I assume to be subjective, I get the following absolute mean errors:
Code:
Estimate This Year Other Year
Model 1.91 3.54
Zero 3.87 3.87
Adj +/- 0.00 4.45
The average absolute error between years for my model was 2.20, less than half of Adjusted +/-'s absolute mean error.
Note that just because the absolute mean error of assuming everyone has a 0 adjusted +/- produces a lower absolute mean error than assuming they stay the same from year to year, definately does not mean that it is a better model.
Back to top
View user's profile Send private message Visit poster's website
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Thu Sep 18, 2008 8:53 am Post subject: Reply with quote
Can you clarify for me even in rough terms what the weight of statistical ends up being vs pure adjusted in your better performing model?
Last edited by Mountain on Thu Sep 18, 2008 5:21 pm; edited 1 time in total
Back to top
View user's profile Send private message
Scott S
Joined: 10 Feb 2008
Posts: 46
Location: East Rutherford, NJ
PostPosted: Thu Sep 18, 2008 4:08 pm Post subject: Reply with quote
Since weights for adjusted plus minus' tend to be around 25%-50%, if I assume full credibility for minutes of 2730 attributable to statistical plus minus, the formula for credibility is, at most, (minutes/2730)^.5. (Total weights don't exceed 1).
The model I used, for the population I compared, it is almost exactly the same as Dan R's weights of adjusted and statistical +/-'s, since most players have enough minutes not to apply any weight to the subjective measurement. The model I mentioned would have more additional usefulness for rating those who don't have many minutes at all.
Upon reviewing, I noticed a slight error in my average absolute error term, there was actually less than 1% of prediction models that predicted itself, since I used the same subjective rating for both years, removing this would only increase the error to about 2.22. Obviously, this model is more consistent. It is only more accurate because of the inconsistency from adjusted plus minus. If we had a much larger, more varied sample to use for adjusted +/-, that would be the most accurate predictor.
Note that applying some additional weight to subjective plus minus would make the model more accurate since subjective ratings combine the consistency lacking from adjusted +/- and the intangibles lacking from statistical +/-. Statisical +/- tends to be the best predictor, followed by Subjective ratings (provided they are good) and then adjusted +/-.