APBRmetrics

Which of these statements most closely matches your current assessment of RPM's efficacy in sorting overall player impacts (for players in year 3 or beyond)?

Average vote so far is a bit above option 4 (I think RPM can get within 1.5 pts of impact level plus or minus for at least 70% of league). That is where I put it too.

I think it's better than "I think RPM can do a good job sorting most players into maybe 5 groups (great, vg, avg, below avg, awful)" but not as accurate as the next option up.

Okay.

But splitting between Options 3 and 4 as written would be pretty hard. The terminology is different. 70% of players within plus or minus 2 (or 2.5) gets real close to the 5 bin split with qualitative labels if the middle bin is centered on zero.

Perhaps I could have had 6-7 options. I am sometimes told or guess based on reaction that I make things too complicated for others, so 5 options was using a bit of restraint.

Crow wrote:Okay.

But splitting between Options 3 and 4 as written would be pretty hard. The terminology is different. 70% of players within plus or minus 2 (or 2.5) gets real close to the 5 bin split with qualitative labels if the middle bin is centered on zero.

Perhaps I could have had 6-7 options. I am sometimes told or guess based on reaction that I make things too complicated for others, so 5 options was using a bit of restraint.

Yeah I wasn't criticizing your choice of poll options, I think they're fine, just giving my personal perspective as specifically as possible.

Although on second thought, 70% +/- 2 or 2.5 really isn't that high of a standard, so maybe it is around that level. Given the range of RPM is about -6 to +8, those two options would be pretty similar.

I like the idea behind this poll, but here are a few thoughts and observations:

First, I find the categories somewhat confusing. For example, options two and three, when I try to express each as an explicit empirical criterion, akin to four and five, I read them as implying that RPM is more "efficacious" than their rank in the poll (I read two as being roughly equivalent to four and three implying - perhaps - the highest degree of accuracy of RPM ratings).

Second, I wonder if the modal vote (so far) for option three truly represents these voters' informed beliefs.

I find it useful (as a first approximation) to think of the errors in any given year's estimates of RPM (or xRAPM or whatever) as being normally distributed. In such a light, the poll could be expressed as one's belief in the standard deviation of this distribution, and option three is pretty much asking if folks believe that to be about 1.5 points.

Do most people really believe that it is that high?

I would ask interested (and actual) voters to explicitly pore through the data at http://www.espn.com/nba/statistics/rpm/_/sort/RPM and check their intuition.

Two critical stylized facts about RPM data (and whatI believe I have discussed previously) are: The best player in the league maxes out at between +8 to +10ish; About 150 players, year in year out (roughly 1/3 of the participating players) have positive ratings (the number/share of positive ORAPM being somewhat lower and DRAPM being somewhat higher). Also, the range of the "average" players, the middle third, is about 0 to -2.

So, in this context, what ought one's belief about the standard deviation of the error distribution be?

One simple exercise to help specify one's beliefs would be to eyeball the data (across the almost four complete seasons of RPM) and identify the perceived extreme errors. Divide that ("true" RPM - ESPN RPM estimate) by 3 or 4 and that might be a considered poll-vote.

When I look at the data, I don't perceive any estimates being off by 5+ points, but that's just me.

Since even a perfect system won't draw a line between 'average' and 'bad', I skipped those and checked this one:

I think RPM can get within 1.5 pts of impact level plus or minus for at least 70% of league

If that refers to 70% of the players in the league, I regret my choice. Low minute players are going to have bigger uncertainties in any system.

Last year, it looks like 37 players got an RPM based on less than 100 minutes. Of these just 3 have a BPM within 1.0 or their RPM. For most their BPM is dramatically worse -- on avg -5.5 vs -1.4 (minutes weighted)

If the poll choice refers to 70% of the minutes in the league (via the players getting the minutes), then maybe.
BPM isn't necessarily my best choice for impact level, but it's on the same scale as RPM; and it's more agreeable than PER, WS/48, or eWins.

By the BPM standard, just 55% of NBA minutes in 2015-16 were with players whose BPM and RPM were within 1.0 of each other, and 73% were within 1.5

Converting WS/48 to plus-minus, just 40% are within 1.0 of RPM, and 59% fall within 1.5

Mike, it is not for me to speak for Crow, but in the first post he writes "(for players in year 3 or beyond)". So, the intention is to refer only to players who have had their ratings stabilized from the rookie season prior and three years experience (which is the vast majority of total players in the league).

This aside, I am not sure how your approach of comparing the correlation between metrics is relevant for determining the precision of RPM estimates?

Might I ask you to play my game? Go through the 2015-16 RPM list (or any other or all of the 3+ seasons for which the statistic is provided) and identify the most egregious single error (or several errors) according to your view of the world. Then share with us your estimate, in plus-minus terms, of what you believe the "true" value of said player. This is not to judge the analytic framework you employ, but based on the absolute value of this supposed error, what might this imply about the precision of RPM as a metric?

I wondered how anyone would have any quantitative idea of whether they agree with an RPM list within 1.0 or 1.5, so I admitted my ignorance and just used a handy point of reference. 'Player impact' by subjective means is at least as variable as our known stat measures.

Last year, Draymond Green led the league in RPM 'wins'. Here's how other stats rated him in plus-minus equivalents:
RPM - 9.0
BPM - 5.8
WS - 3.4
eWins 3.2
PER - 1.8

A newer measure I've worked on, faux RPM -- another boxscore RAPM-correlated thing -- gives him 6.0; but that was based entirely on 2015-16 stats, so may be less valid for other seasons.

Draymond Green is an interesting, possible "extreme". As I am sure you have noted, the preceding year he clocked in at 6.8 and this year (as of today) he reverts to 6.53, so there was something about the data last year that bumped him up 2+ points relative the the abutting years.

Are there other eligible candidates? Who would rank 2nd and 3rd?

The poll is not perfect but it is what I came up with at the time. It is a small sample response. Mostly supportive of RPM. Few critics voted, no absolute rejections registered in the poll.

A discussion extending beyond the poll is probably better at this point than focus on the poll. But if someone with extensive league contacts wanted to do a poll of insiders (GMs, Coaches, "Analytic Directors") that would be interesting. Never been done to my knowledge. Overdue.

I meant 70% of non-rookie players in option 4 but I probably should have excluded low minute players too.

Mike G's comparison of BPM and winShares / 48 data shows fairly close to what I suggested in that option. Of course BPM and Winshares each have "issues" for some users / evaluators. It would be very useful to see the comparative table, sooner or later. I build them occasionally but don't have a current crosswalk... and no media source that I am aware of does it.

schtevie's proposed exercise could be interesting though time consuming. I might do later. It would be interesting if a few skeptics or non-believers / non-users of RPM or active critics of RPM did this exercise (and named the names of the believed true value - RPM outliers).
If any do, it will increase the chances I will also.

On the Green case, I assume much of the metric discrepancy comes from RPM trying to estimate individual shot defense impact while on the court, while everything else uses team as whole data, not individual and not just when on court, or nothing for shot defense.

According to BRef there are 299 non-rookies with over 500 minutes played this season. 260 over 800 minutes. 236 over a 1,000. I'll run thru the 236 very fast. It is less than a full season so variance from "expected" RPM may be higher but lets start there. They are a better group for RPM (or anything else) to estimate than any larger sample. It is essentially main rotation minus some fookies, about 8 players per team. Deep bench probably harder to peg / less important.

Earlier I expressed less confidence in values for guys with less than 3 years experience. I still feel that way but left 2nd and 3rd year players in. If they were out, there would be less cases of doubt but I didn't want to slice that far and go under 200 considered. If I had, it probably would make RPM look stronger. But I forgot and will stick with what I said.

Dropping eWins from the comparison, here are top and bottom 10, relative to RPM, of net points avg of BPM and plus-minus inferred from PER and WS/48.In most cases, it seems BPM is closest to RPM, while PER-PM is furthest.

I went thru the RPM values. I gave guys anywhere from a fraction of a second thought to maybe 5 seconds. These are the 41 that I thought it would be most appropriate to think about further as to whether the RPM estimate is close. I am not yet willing to say I think the variance might be over 1.5 points plus or minus. Maybe I'll get there on pass 2 or 3.

J Butler
Gobert
K Love
Pachulia
A Johnson
G Hill
Beverley
O Porter
C Zeller
D Dedmon
Ariza
R Covington
P Patterson
Nogueira
M Gasol
J Ingles
Rubio
Danny Green
D Carroll
Isaiah Thomas
D Howard
T Hardaway
W Ellington
Whiteside
Jerian Grant
R Hood
Middleton
Vucevic
J Val.
D Collison
A Bradley
Henson
Faried
Beasley
D Schroeder
R Jackson
K Korver
Crabbe
Mozgov
S Livingston
D Booker

Only 40% of Mike's extreme variance 10's bothered me before seeing them.

I am not yet saying the other 195 are accurate within 1.5 pts but on lightning review I didn't have above average doubts about RPM's rough accuracy.

Not reliable at all. No metric is reliable at sorting players at all. There are some absurdly simple linear box-score metrics that are more reliable than RPM at sorting player impact.

RPM is reliable at predicting next year's team point differential or wins at a moderate rate. It's the best single metric at that, while BPM is a close second.