Poll: RPM's degree of efficacy in sorting players

[Crow]

PM, which box score metrics do you consider more reliable at sorting overall player impact than RPM? Without names, I can't consider. Almost every box score suffers from having no or vague shot defense data and not enough dealing with the impacts of a player on the offensive production of others.


Is no one else going to answer call for longer list of cases where RPM may be or is in their opinion more than modestly off? I would add a few names to my list, including Leonard. We've heard folks go on about a half dozen cases (1.5% of league cases); but if you really think RPM is not reliable at all, shouldn't be used it should be pretty easy to list dozens or many dozens or many, many (many?) dozen of cases. Without listing those dozens of cases where RPM is too far off the sweeping dismissal is lightly argued / basically untested / unsupported to any meaningful depth. Of course that is the easiest path to take.

Often unspoken but asserted just above is that box score metrics or simple stats or eye tests are more reliable. I consider these claims to be claims less supported than those for RPM. RPM and its rougher, simpler predecessors has beat box score metrics on projecting team performance regularly for 5-10 years. It is difficult to find another or better test. If someone has one, present it, let's discuss it, perhaps put it to use. Predicting single games or playoff series (or lineup stints?) may be alternatives but probably less good than a full season. I think but tell me if you disagree.

[permaximum]

PM, which box score metrics do you consider more reliable at sorting overall player impact than RPM? Without names, I can't consider. Almost every box score suffers from having no or vague shot defense data and not enough dealing with the impacts of a player on the offensive production of others.


Is no one else going to answer call for longer list of cases where RPM may be or is in their opinion more than modestly off? I would add a few names to my list, including Leonard. We've heard folks go on about a half dozen cases; but if you really think RPM is not reliable at all, shouldn't be used it should be pretty easy to list dozens or many dozens of cases. Without listing dozens or many dozens of cases where RPM is too far off the sweeping dismissal is basically unsupported, untested. Of course that is the easiest path to take.

Often unspoken but asserted just above is that box score metrics or simple stats or eye tests are more reliable. I consider these claims to be claims less supported than those for RPM. RPM and its rougher, simpler predecessors has beat box score metrics on projecting team performance regularly for 5-10 years. It is difficult to find another or better test. If someone has one, present, let's discuss it, perhaps put it to use.

It's my belief that noise, bias, multicollinearity issues exceed the advantages of shot defense data, synergy etc. That's how I described my foundings. Besides, I haven't seen a single case where RPM's player value assigning power was tested properly. Have you? RPM is untested, unsupported and lacks any documentation. I privately tested it ofc. That's a different story.

I worked on this topic alone for a month about a year ago. Then I tried to make a significantly better metric at assigning player value which I failed. I could only get marginal improvements. But I still believe there's room for improvement. I probably have some of those files still.

As for your request for absurdly simple linear box-score metrics that are better than RPM at assigning player impact, here are two examples.

Tom Thibodeau: (2*fg-fgx+ft-0.5*ftx+3*3pm-1.5 * 3px+reb+ast-pf+stl-tov+blk)/mp
David Lewin/Dan Rosenbaum - Alternate Win Score: (pts+0.7*orb+0.3*drb+stl+0.5*blk+0.5*ast-0.7 * fgx-fg-0.35*ftx-0.5*ft-tov-0.5*pf)/mp
PER is a bit better than RPM but worse than the previous two.

I couldn't test many box-score metrics but I'm pretty sure there are more out there that's better than RPM and BPM. The difference doesn't make them reliable either. Just better than RPM but not good enough to be reliable at all.

[Crow]

I understand that you have high standards and do a lot of private work. I appreciate hearing both opinion and detailed background supporting that finding and / or opinion.

Does that work include testing box score - rpm blends? There has been published work and prediction results suggesting that is better than either by themselves.

At first glance I am not sure if Tom Thibideau metric is his creation or something else he adopted. It is probably his; I recall hearing something about it but would prefer confirmation / clarification. Either way, where is any data analysis of it being tested to be found alone or side by side? Link?

Alt Win Score did well in past and probably recent past by screenname Skeptic whatever but it is important to keep testing. He was credible to me and I believe he found metric blends to do best, with RPM and several other metrics.

Back when Dan Rosenbaum published his first paper on APM (outside of APBRMETRICS discussions) he reported that a blend of box score and APM was better than pure APM. But that point was largely lost til RPM incorporated priors and some folks advocated and / or presented blends that were both pre and post blends of RPM and boxscoremetrics.

On "reliable", there are different standards. Reliable to scientific standards, betting world standards or "reliable" relative to other available options in the same range of reliability (some maybe better, most worse). I used reliable in the last way. I know you object to that use. I never claimed more than that level of validity and yes it is not much. But I am mainly urging the step up from the worst metric answers or the least proven approaches (eye test and/ or subjectively selective use of discrete stats, where people tend to only report a few cases that look good to them and only the good parts and not everything and not the mistakes. and then there is the issue of there being 50 million versions of these approaches...). Using RPM along with other tools and the hunt for better. In blends or something new...

[permaximum]

I understand that you have high standards and do a lot of private work. I appreciate hearing both opinion and detailed background supporting that finding and / or opinion.

Does that work include testing box score - rpm blends? There has been published work and prediction results suggesting that is better than either by themselves.

At first glance I am not sure if Tom Thibideau metric is his creation or something else he adopted. It is probably his; I recall hearing something about it but would prefer confirmation / clarification. Either way, where is any data analysis of it being tested to be found alone or side by side? Link?

Alt Win Score did well in past and probably recent past by screenname Skeptic whatever but it is important to keep testing. He was credible to me and I believe he found metric blends to do best, with RPM and several other metrics.

Back when Dan Rosenbaum published his first paper on APM (outside of APBRMETRICS discussions) he reported that a blend of box score and APM was better than pure APM. But that point was largely lost til RPM incorporated priors and some folks advocated and / or presented blends that were both pre and post blends of RPM and boxscoremetrics.

On "reliable", there are different standards. Reliable to scientific standards, betting world standards or "reliable" relative to other available options in the same range of reliability (some maybe better, most worse). I used reliable in the last way. I know you object to that use. I never claimed more than that level of validity and yes it is not much. But I am mainly urging the step up from the worst metric answers or the least proven approaches (eye test and/ or subjectively selective use of discrete stats, where people tend to only report a few cases that look good to them and only the good parts and not everything and not the mistakes. and then there is the issue of there being 50 million versions of these approaches...). Using RPM along with other tools and the hunt for better. In blends or something new...

Yes, it includes insane amount of blends. Blends do really help with the next year's team win or point differential prediction. However it's a different story for the isolated player impact. You can't improve it in a meaningful way. Marginal improvements in the base case scenerio.

Well I saw the paper myself. It's Tom Thibideau's metric.

Creator of Alternate Win Score probably didn't give much needed attention to prediction accuracy/roster turnover rate graph.

For me, reliable means ~75% prediction accuracy (because of HCA and current metric standards) at +65% roster turnover rate (because this is the line metrics really start to struggle), per game basis. Because you can get 63% with HCA alone.

The thing is no one's making retrodiction tests at the game level and even if they do they don't account for roster turnover rate for each game. Each game has different roster turnover. I will publish some of the results if I find them.


However, I would still strongly encourage everyone to use RPM, BPM blends for the yearly prediction contests because they're optimized for it. I don't have any SportsVU data so I haven't done any tests for it but we saw AJBaskets' success so I assume RPM-BPM-PTPM blends can probably do even better.

I do suspect simple team-based metrics that "properly" adjusted by player-based metrics depending on the signings, trades and rookies can do even better at predicting next year. But I don't have the time for tests. It takes insane amount of time.

[Crow]

Thanks for the follow-up comments.

[EvanZ]

Another way to frame this question is in terms of AUC. For example, an AUC of 0.9 would mean that if you take two players and ranked them, you'd have a 90% chance of ranking the better player ahead of the worse player.

So, with that metric, what would people estimate is the "AUC" of RPM? I'd say around 0.8 conservatively.

[Crow]

RPM currently estimates Ibaka, Dirk and Aldridge in 25-30 range for PFs. Derrick Favors 41. Saric up to 70. Odds the first 4 have a better overall impact than Saric? In this setup, very high imo. What doubt exists probably mainly exists with Ibaka and Favors. They may be too high on own right. Still unlikely worse than Saric. Does anyone think so? Think RPM is wrong on that... for season to date? You could argue they are closer and getting closer and I wouldn't have a problem with that. The future? Up likely for Saric. Ibaka and Favors? Probably depends a lot on coaching / utilization compared to current.

[Crow]

Cases on my quick list with large (over 2 pts) BPM - RPM differential:

K Love
A Johnson
R Covington (almost)
Lucas Nogueira
Patrick Patterson
Isaiah Thomas
Hassan Whiteside
Nikola Vucevic (almost)
Kenneth Faried
Reggie Jackson
Kyle Korver

So only about a quarter of the cases where I raised a question have this sign of value dilemma. I'd look further at these cases to consider whether one or both metrics is off much or more.

Disappointing that no one else has taken the call to provide a list of more cases where they believe or even think maybe RPM is off or way off. Oh well.

[Mike G]

About 3/4 of the way down page 1, I listed 10 players each whose RPM seems too high / low, relative to their BPM, PER, and WS/48.
I don't really 'believe' much about players, other than what the numbers express; I don't watch that much b-ball.

[Crow]

Yes you provided names, early, that might have issues. Sorry for forgetting.

And then I was looking for more, especially if someone had strong opinions about RPM being off.

[Crow]

Others with a 2 pt or more differential:

Muscala
Mirotic
Gibson
D Powell
J Nelson
Mason Plumlee
Baynes
KD
McCaw
Harden
E Gordon
R Anderson
Harrell
Stuckey
Deng
Conley
J Martin
Giannis
Hawes
Towns
LaVine
Noah
O'Quinn
Westbrook
Sabonis
Okafor
Jerami Grant
A Williams
TJ Warren
Aminu
D Lee
Kuofos
Siakam

Not that many, among minute qualifiers. Regression pulls down some of the top guys down on RPM from what BPM shows. The average differential is a bit under 1.2.

Comparing RPM and DRE is another step. DRE weights are derived using RPM. Looking at it. It appears that there are bigger differences between RPM and this old version of DRE, with DRE running notably higher. Will want to see new version.

[tarrazu]

Seems most of the disagreement or issues arise on defense. Most publicly available metrics do a decent job of quantifying offense.

[Mike G]

Among players with >1500 minutes this year, RPM seems to like these guys more than (an avg of) BPM and DRE

Code: Select all

R-BD   Player         Team   RPM    BPM   DRE   WS/48   PER
2.7   Al-Farouq Aminu  POR   1.3   -1.5  -1.3   .042   10.8
2.4   Malcolm Delaney  ATL  -3.2   -6.0  -5.3  -.003    7.2
2.4   Domantas Sabonis OKC  -2.5   -5.0  -4.8   .018    6.6
2.2   Marcus Morris    DET   1.1   - .4  -1.8   .072   12.4
2.1   Amir Johnson     BOS   4.4    2.2   2.2   .146   14.9

2.1   Paul Millsap     ATL   4.7    2.8   2.3   .132   17.9
2.1   Isaiah Whitehead BKN  -3.0   -5.0  -5.3  -.024    7.8
2.0   Robert Covington PHI   3.4    1.5   1.2   .074   13.2
2.0   Kevin Love       CLE   4.4    1.0   3.8   .169   21.7
1.9   Solomon Hill      NO    .7   - .3  -2.1   .060    8.1

1.9   Luol Deng        LAL   -.4   -2.6  -2.0   .025   10.1
1.9   Eric Gordon      HOU   1.3   - .9   -.3   .089   13.7
1.9   Victor Oladipo   OKC   1.5   -1.0    .1   .097   14.2
1.9   Draymond Green    GS   7.1    5.6   4.7   .164   16.7
1.9   Ryan Anderson    HOU   1.2   -1.0   -.5   .115   13.2

DRE seems to be on the same scale, so I used it in the calc. WS/48 and PER are just shown for visual corroboration.
Among these 15, DRE beats BPM 8-6-1 on being closer to RPM.

Among 16 most disfavored by RPM, BPM is closer than DRE on all but 2 --

Code: Select all

R-BD    Player        Team   RPM   BPM   DRE   WS/48   PER
-4.5   Isaiah Thomas   BOS   1.7   5.4   7.0   .238   26.5
-4.3 Russell Westbrook OKC   6.4  15.1   6.3   .223   30.6
-3.7   Giannis A       MIL   4.3   7.7   8.5   .215   26.3
-3.7   Kevin Durant     GS   5.4   8.1  10.1   .278   27.7
-3.7   James Harden    HOU   4.9  10.3   6.9   .255   27.7

-3.2   Karl-A Towns    MIN   1.8   4.8   5.3   .196   25.4
-3.2   Andre Drummond  DET   -.7    .7   4.3   .135   21.4
-3.1   Zach LaVine     MIN  -3.0   -.2    .5   .087   14.7
-2.9   Kyle Korver  ATL/CLE -3.1   -.3    .0   .096   11.9
-2.7   TJ Warren       PHX  -2.6   -.4    .6   .092   15.3

-2.5   Kawhi Leonard    SA   6.2   7.8   9.5   .265   27.6
-2.4   Mike Conley     MEM   3.4   5.6   5.9   .203   22.8
-2.2  Mason Plumlee DEN/POR   .8   2.9   3.2   .139   18.5
-2.2   Dwight Howard   ATL   1.2   2.5   4.2   .183   21.0
-2.1   Marquese Chriss PHX  -3.8  -1.8  -1.5   .048   11.9
-2.1   Dwyane Wade     CHI   -.5   1.3   1.8   .093   19.0

Quick correlations between RPM and the others, for the 1500+ minute players and for those with 500-1500 min.

Code: Select all

corr.  1500+  5-15
BPM    .866   .761
DRE    .845   .684
WS/48  .837   .663
PER    .713   .517

[Crow]

RPM measures individual shot defense along with everything. BPM and DRE don't directly. List where RPM is higher could be higher because of good shot defense (and / or other things) and mostly looks plausible for that. List where RPM is lower, possible/ probable for lesser shot defense than teammates. Some cases might cause question like D Howard so one would want to look at player tracking data... and hope for 4 factors of RPM. Most cases of wonder about RPM accuracy could be aided by them. Will JE provide them or will someone else step again as in last to provide them for RAPM?

[Crow]

Actually the RPM - BPM differential for Howard is almost entirely on offense. It is common for RPM to show a negative impact on offense for big men that BPM doesn't and maybe can't. RPM sees what happens to offense with and without the big. BPM just looks at the box score. The real world impact is probably the reason coaches go small. RPM for these guys in general is consistent with that coaching action. Score points for coaches and RPM I'd say.