Appr. 5.x year reg. adj. +/- (J.E., 2010)

[J.E.]

I was just pointing out that you no longer are using a method that is unbiased or theoretically statistically appropriate. I know it's more accurate for most players (it's an informed prior!), but it is now biased towards older players, significantly. The more years you string together as priors, the worse it will get, if you don't reduce the weight of the older years while keeping the weight of the prior the same.

What do you mean by "weight of the prior"?

The version which uses zero as prior for everyone was "unfair" to the absolute top players because there was a good chance they couldn't reach the rating they deserved. Also, it was unfair for all those who played with the worst players because the very, very bad players didn't get a rating as bad as they deserved and thus dragged their teammates rating down with them.

You'll probably always find someone to whom whatever method currently in use is "unfair"; and I would argue that the version with less out of sample error is less "unfair" than the one with higher error

[Mike G]

'03, '04 and '05 are now online. '06 and forward still needs to be updated with the new priors...
http://stats-for-the-nba.appspot.com/ranking04
...
Garnett is always at the top. In '04 he's more than twice as good as the player ranked at #7...

I see some 4.5 pages of above-zero players and 12.5 pages of below-zero, roughly 25 per page.
How many times better than the median (-1) player is Garnett (+8.6) in '04? In that range: Donyell Marshall, Maggette, Reggie Evans, Haslem.

[J.E.]

'03, '04 and '05 are now online. '06 and forward still needs to be updated with the new priors...
http://stats-for-the-nba.appspot.com/ranking04
...
Garnett is always at the top. In '04 he's more than twice as good as the player ranked at #7...

I see some 4.5 pages of above-zero players and 12.5 pages of below-zero, roughly 25 per page.
How many times better than the median (-1) player is Garnett (+8.6) in '04? In that range: Donyell Marshall, Maggette, Reggie Evans, Haslem.

That "twice as good"-quote was a poor choice of words on my part.
What I meant was: if he were to play on a team that accumulated a point differential of 0 per game without him, he would boost that team's point differential by more than twice the amount which the #7 ranked player would boost that team's point differential.
Further, if Haslem and Garnett played every minute and you replaced Haslem with Garnett, Haslem's old team should see a (8.6 - -1)= 9.6 improvement in point differential. Obviously, no one plays every minute, so realistically replacing Haslem with Garnett in that year should have given you a boost of ~0.75*9.6

[bbstats]

I agree with Daniel - players' ratings should be regressed or at least modified according to the estimated aging curve if they have played too few possessions.

I think this would, over the course of time, satisfy the condition of improving the system by decreasing per-possession error.


On a related note, someone needs to help me write about rAPM and its value in determining causal, rather than correlational, relationships.

[EvanZ]

On a related note, someone needs to help me write about rAPM and its value in determining causal, rather than correlational, relationships.

I've thought about this a little before. Here's a question. If you wanted to know the "true" effect that a particular player has on the outcome of possessions, what experiment would you do? I think it's fairly clear that - ideally - you would be able to substitute that player for other players of various talent levels while keeping all other 9 players on the court "constant". We can't actually do this experiment, but we can look at the "natural" experiments that take place over the course of a season that result in players being substituted for each other. APM and its variants essentially try to set up a framework for keeping track of the results of these in-game experiments, and to the extent that is possible given the nature of the problem, the results *do* - I would argue - represent causation.

[schtevie]

I have not really been paying attention to what's been going on for quite awhile, but Jeremias, this is really, really great stuff. A few questions, observations, and suggestions - not that you need any more of the latter.

First (and apologies if I have missed a discussion of this elsewhere) what is the relative out-of-sample predictive performance of the various permutations of RAPM and APM? It would be great to have a referential chart of such comparisons. (And better still if non-APM-based metrics were also included.) Opinions on the shape of the earth shouldn't be allowed to differ (much) and having a (yearly) scorecard should help a lot.

Regarding your single-year ratings with the previous year's ratings as priors, it seems to me that these would be significantly improved (the earlier years, especially, of course) by beginning the series with a prior that approximates the generally expected range of estimates. What seems clear is that the overall distribution of results, year by year, is playing catch-up to reality. (For example, the 10th best player seems to become "twice as good" between 2002 and 2011.) Perhaps a multi-year SAPM for 2001?

More generally, it will be great, great, great when aging curves become part of the prior. And in an even more perfect world, having the prior being informed by injury history. (Not that I would expect the latter to change things that much, but for completeness...)

Finally, upon the "perfection" of the prior, I will be especially curious about comparisons between RAPM estimates for 2007-08 and the multi-year, weighted APM of Ilardi and Barzilai (http://www.82games.com/ilardi2.htm). I am not sure if this is even the proper way to think of things, but I&B estimates +/- two standard errors still aren't generally compatible with your currently preferred RAPM results. Should one expect eventual compatibility?

[J.E.]

First (and apologies if I have missed a discussion of this elsewhere) what is the relative out-of-sample predictive performance of the various permutations of RAPM and APM? It would be great to have a referential chart of such comparisons. (And better still if non-APM-based metrics were also included.)

Here is some recent work on the subject
http://sportskeptic.wordpress.com/2011/ ... e-results/
We did our own little contest here
http://sonicscentral.com/apbrmetrics/vi ... ?f=2&t=244
but it's just one season. I plan to put up a google document soon where one can see retrodiction results for standard RAPM and prior-informed RAPM for every season since '04. This document will also include other metrices of which I have forecasts

Regarding your single-year ratings with the previous year's ratings as priors, it seems to me that these would be significantly improved (the earlier years, especially, of course) by beginning the series with a prior that approximates the generally expected range of estimates. [..] Perhaps a multi-year SAPM for 2001?

That would definitely be a cool thing. But I would need SPM numbers for 2001 for that. And to make things perfectly fair, the SPM weights need to be determined only with seasons prior to 2001

Finally, upon the "perfection" of the prior, I will be especially curious about comparisons between RAPM estimates for 2007-08 and the multi-year, weighted APM of Ilardi and Barzilai (http://www.82games.com/ilardi2.htm). I am not sure if this is even the proper way to think of things, but I&B estimates +/- two standard errors still aren't generally compatible with your currently preferred RAPM results. Should one expect eventual compatibility?

I don't think so, the regularization has a huge effect. There's no real reason to compare the two. I'm 99.9% sure RAPM does better. (Because if it wasn't I would see lower lambdas when doing crossvalidation)

I did a couple of minor tests and to save future researchers some time, here are the results:
- Multiplying the prior with a factor (<1) results in worse out-of-sample-prediction. Unless the factor is >0.9 at which point things become a wash: Anything between 0.9 and 1.0 is fine.
- Two year RAPM with zero as prior and the first season weighted at 0.5 gives better results than one year RAPM (not exactly news).
- Prior informed RAPM brings us three times the error reduction over 1yr RAPM as the aforementioned two year RAPM brings over 1yr RAPM

[DSMok1]

Whenever you can get me your 8 year average APM, with possessions played, I'll work on revising ASPM. ASPM could perhaps be an excellent informed prior to use.

I would use the ASPM for the year in question as the prior, because we are trying to get as accurate as possible an estimate of what actually occured. I would be very interested to see, for example, a comparison of using multi-year RAPM vs. using 1 year RAPM, regressed toward ASPM rather than 0. To see how good the projection is, of course, we'd have to use the next year, because ASPM is hard to get in partial-year chunks after the fact.

A good starting point for the aging curve would be this graph:

The equation for that line is:

Code: Select all

y = -0.0005647x3 + 0.0566511x2 - 1.9327991x + 21.8411933

The points on that curve are:

Code: Select all

Delta Age Deltas (Model)
18.5      1.898
19.5      1.506
20.5      1.162
21.5      0.861
22.5      0.601
23.5      0.378
24.5      0.188
25.5      0.029
26.5     -0.103
27.5     -0.212
28.5     -0.300
29.5     -0.372
30.5     -0.431
31.5     -0.479
32.5     -0.521
33.5     -0.560
34.5     -0.599
35.5     -0.641
36.5     -0.691
37.5     -0.751
38.5     -0.824
39.5     -0.915
40.5     -1.026
41.5     -1.162
42.5     -1.324
43.5     -1.517

On another note: the weighting for previous years in a sample. I found, for my projections, that it was best to weight like this: current year (if during year), 1.84. Last year, 1.0, previous to that 0.439^number of years prior.

[J.E.]

Whenever you can get me your 8 year average APM, with possessions played, I'll work on revising ASPM. ASPM could perhaps be an excellent informed prior to use.

This will probably take two more days.

Thanks for that formula.

Do you ever plan to make different curves for offense and defense? There's one coach here who thinks basketball players improve until the age of 35 because playing good defense is more about experience. I'm not saying I agree but I think the curves would look different if you look at offense and defense seperately

[schtevie]

Finally, upon the "perfection" of the prior, I will be especially curious about comparisons between RAPM estimates for 2007-08 and the multi-year, weighted APM of Ilardi and Barzilai (http://www.82games.com/ilardi2.htm). I am not sure if this is even the proper way to think of things, but I&B estimates +/- two standard errors still aren't generally compatible with your currently preferred RAPM results. Should one expect eventual compatibility?

I don't think so, the regularization has a huge effect. There's no real reason to compare the two. I'm 99.9% sure RAPM does better. (Because if it wasn't I would see lower lambdas when doing crossvalidation)

I think there is a very good reason to compare the two: it is really important to have confidence that point estimates on the tail of the "talent" distribution (the upper, in particular) are accurate. The "magic" of RAPM is that despite imposing a zero prior - what is a very strong assumption - it yields not only better line-up predictions than unbiased APM, but it apparently defeats all comers as well. Formidable.

However - and it is a big however - it has a glaring weakness: it accomplishes this while clearly - and I mean clearly - biasing the contributions of star players downward (most conspicuously with a zero prior imposed). The notion that in 2002, for example, at the height of their powers, that Tim Duncan was only worth 3.0 points per 100 possessions and Shaq was at 3.6, well, I don't think you would find anyone on this board or elsewhere adhere to that point of view.

But now we are talking about improving matters with a more rational specification of the prior. And, as such, the issue remains: how to establish confidence that, after having done so, upper tail estimates are accurate? And this is not a trivial concern, basketball-wise, I should think. If RAPM becomes the referential ranking - deservedly so, based on its predictive power - you want to have confidence in the cardinal as well as the ordinal on the high-end, for reasons relating to payroll, for awards, both within-season and post-career, etc. This is kind of a big deal.

Which gets to the point of comparisons with multi-year APM. Again, perhaps I am not thinking of the technical issues correctly, but the thought is that if you take the estimates from multi-year APM, say for the top X% of the distribution, that these results should be compatible, statistically, with those generated by multi-year APM (and perhaps single-year) if the best prior is chosen (in terms of predictive power).

[DSMok1]

Whenever you can get me your 8 year average APM, with possessions played, I'll work on revising ASPM. ASPM could perhaps be an excellent informed prior to use.

This will probably take two more days.

Thanks for that formula.

Do you ever plan to make different curves for offense and defense? There's one coach here who thinks basketball players improve until the age of 35 because playing good defense is more about experience. I'm not saying I agree but I think the curves would look different if you look at offense and defense seperately

I don't have enough data to run them separately right now, but when I have a chance, I'll look at it. Probably, I should generate new ASPM formulas using your data (you calculate O and D independently, right?), and generate regressed true talent estimates for each year played, and then run new aging curves for offense and defense separately.

I'm really busy right now (will be moving across the country and starting a new job beginning next month), so my time will be limited. Fortunately, I have most of the spreadsheets already constructed for such an endeavor.

[bbstats]

schtevie -

I don't think 2002 has all of its data yet? So the player are still being penalized to 0 a lot? Maybe?


It is interesting, to see the differences in the tails from year to year. I don't feel like ANY statistical measure for players well-represents those on the extreme ends.
The year-to-year variation in extremes perhaps speaks less of the players and more of how the system interpreted the season as a whole.

here's the top 5 in the past 4 years...

2011: 8.6, 8.2, 8.1, 8.0, 7.0
2010: 10.5, 7.9, 7.2, 6.3, 6.2
2009: 9.4, 7.8, 6.4, 6.1, 5.9
2008: 8.1, 7.3, 6.0, 5.8, 5.5

[Crow]

In the post Shaq era, Kobe has always been at least +1 on this 1 year RAPM. (No weird huge negative here as I believe 1 year traditional APM was showing for this season and had been highlighted by an APM critic.) In the 4 years he was above +3, they went to the finals 3 times. He was +4 or above in those 4 seasons. 2005-6 was the only time he was elevated above +3 and didn't go far in the playoffs.


Compared to the ratings they got on separate teams on the most recent previous 1 year RAPM and the 5 and 10 year versions, the ratings sum for the combination of Wade, James and Bosh fell about 30-50% on this season's 1 year RAPM.


Dirk's 1 year rating this season was his highest in the 10 year period.


Dwight Howard had his second best 1 year rating this season. He was estimated as having a -1.4 impact in his rookie 2004-5 season.


Durant has moved up nicely the last 3 seasons. Westbrook was estimated at +0.6 in 2008-9, +3.3 in 2009-10 but down to +0.5 in 2010-11.

Duncan still over +3 this season but barely and that was his worst in 10 years. Parker also at his lowest and barely above neutral. Ginobili was fine at over +6, except when he wasn't fully fine physically in the playoffs.

Zac Randolph had his first season over +2, after usually being a bit under 0.

The Celtics big 4 combined were almost at +14 on RAPM but Rondo's health, matchups and the worst turnover rate, the 2nd lowest FT/FGA and a weak offensive rebounding performance in the playoffs had there affects.


Amare Stoudemire was the only 2011 All-Star with a negative 1 year RAPM estimate. Love, Westbrook and Horford were between 0 and +1.

For whatever reason(s) there are fewer below -3 in 2011 than in any other season since 2001-2 (when only 1 player pulled a rating below that level) and in most cases far fewer than those other seasons.

On 2008-9 had the most estimated at or over +4 (17). 2001-2 had no estimates over +4.

[J.E.]

However - and it is a big however - it has a glaring weakness: it accomplishes this while clearly - and I mean clearly - biasing the contributions of star players downward (most conspicuously with a zero prior imposed). The notion that in 2002, for example, at the height of their powers, that Tim Duncan was only worth 3.0 points per 100 possessions and Shaq was at 3.6, well, I don't think you would find anyone on this board or elsewhere adhere to that point of view.

I don't think 2002 has all of its data yet? So the player are still being penalized to 0 a lot

bbstats is right. The ratings are low because there are only 3 months of data. With one full season of data the ratings can go as high as those http://stats-for-the-nba.appspot.com/ranking11.

DSMok1, I am not sure I interpret that chart correctly but to me it seems that every 20 year old that enters the league improves by ~+4.5 (per 200 poss.) by the time he is 25. If we assume rookies to start at -2 it means that they all improved to +2.5 at 25. And a team of 25 year olds would beat a team of 20 year olds by (5*4.5) by 22.5 points. Seems a bit drastic to me. I don't think you did something wrong but could you comment on why the deltas are that high?

[DSMok1]

DSMok1, I am not sure I interpret that chart correctly but to me it seems that every 20 year old that enters the league improves by ~+4.5 (per 200 poss.) by the time he is 25. If we assume rookies to start at -2 it means that they all improved to +2.5 at 25. And a team of 25 year olds would beat a team of 20 year olds by (5*4.5) by 22.5 points. Seems a bit drastic to me. I don't think you did something wrong but could you comment on why the deltas are that high?

You are interpreting the chart correctly, but average rookies are a lot worse than that. A major issue is that there were very few 20 year olds in my sample; once I have a larger sample, I can nail down the numbers better on the aging curve.

Here are the average rookies (and I found no evidence that age was a factor in how good each rookie was):



Here are the exact numbers:

Code: Select all

Pick	Expected	Stdev
1	0.51	1.65
2	-0.22	1.65
3	-0.67	1.65
4	-1.01	1.68
5	-1.28	1.72
6	-1.52	1.76
7	-1.73	1.80
8	-1.92	1.84
9	-2.09	1.88
10	-2.25	1.92
11	-2.40	1.96
12	-2.54	1.99
13	-2.68	2.03
14	-2.81	2.07
15	-2.94	2.11
16	-3.06	2.15
17	-3.18	2.19
18	-3.29	2.23
19	-3.40	2.27
20	-3.51	2.31
21	-3.62	2.35
22	-3.72	2.39
23	-3.82	2.43
24	-3.92	2.46
25	-4.02	2.50
26	-4.12	2.54
27	-4.21	2.58
28	-4.31	2.62
29	-4.40	2.66
30	-4.49	2.70
31	-4.58	2.74
32	-4.67	2.78
33	-4.76	2.82
34	-4.85	2.86
35	-4.94	2.90
36	-5.02	2.94
37	-5.11	2.97
38	-5.19	3.01
39	-5.28	3.05
40	-5.36	3.09
41	-5.45	3.13
42	-5.53	3.17
43	-5.61	3.21
44	-5.69	3.25
45	-5.77	3.29
46	-5.85	3.33
47	-5.93	3.37
48	-6.01	3.41
49	-6.09	3.44
50	-6.17	3.48
51	-6.25	3.52
52	-6.33	3.56
53	-6.40	3.60
54	-6.48	3.64
55	-6.56	3.68
56	-6.63	3.72
57	-6.71	3.76
58	-6.79	3.80
59	-6.86	3.84
60	-6.94	3.88

Realize, of course, that once you get below, say, pick 15, many to most of the rookies don't actually play. To calculate the curve above, I assumed that the players that actually played were the top of a bell curve--if 15% of the players played, I was looking at 85th percentile and above at that pick.