Poll: RPM's degree of efficacy in sorting players

[permaximum]

interesting. any thoughts on why predictiveness of mpg somehow becomes higher as there's more turnover? (compared to itself going from say 50% to 60-65%)
for the rpm sample, all 3 seem to get better so that might just be part of the sample size issue. but for the large sample, it happens for mpg but not bpm.

I was actually surprised by this too when I first found out. Actually all metrics see very small improvements until 60-65% of RT but MPG and a few other metrics see bigger improvements. I think the general improvement is completely related to sample. Roster turnover rate was not really high in the 80s and that's where the metrics struggled. That's understandable.

As for MPG, PER and two simple linear box score metrics, I believe individual player quality difference shows itself more when there's less synergy, lineup effect, set running, player-coach politics while trying to get used to other teammates. However, when the RT rate gets too crazy and games become really unpredictable all metrics start to see sharp declines but the better metrics hold their ground more. This whole situation confirms the very first theory.

[permaximum]

Thanks for the correction. The print was real tiny on my phone screen and I too quickly misread Chart 2 & 3 and was overfocused on RPM. So RPM always beat minutes in Chart 1. I am not fully understanding or convinced that very high or 100% roster turnover is the ideal or only important test point (given how far it is from average).

No. MPG catches RPM at 61%. That's where it matters. Actually even more roster turnover rate was needed and MPG would eventually surpass RPM.

For isolated player impact, very high roster turnover rate is a must. BPM and RPM try to share team success to players. But that team success has more things than player themselves.

[Crow]

With RPM and minute performance pretty close to very close, why not take that as a compliment for each? Reinforcing. Maybe the contest is over-rated. Use what you have / like depending where you sit. Gets you to close to about the same spot right?

[Crow]

No. MPG catches RPM at 61%. That's where it matters...

But that team success has more things than player themselves.

MPG catches RPM at 61% in your chart from Saturday morning but not chart 1 from Sunday evening. Admittedly I am going too quickly but why the difference?

Team success includes coaching right? Management too in all its aspects? Then I ask so what? What I supposed to think or do differently after acknowledging this?

[permaximum]

@Crow

You're clearly reading charts wrong my friend. Please read them again. I shared RPM chart only once. You're probaby confusing it with RAPM which I shared Saturday.

Anyways, why do you need the urge to defend RPM? You're on the wrong path. If you want to compare players, it's even worse than MPG along with other advanced metrics. It's simple as that. There are no ifs or whats. And MPG itself is worse than simple linear box-score metrics I tested.

If you want to predict next-year team wins or if you want to bet on games use RPM/BPM blends. That's simple too. RPM is optimized for it and RAPM's core lies on lineups and multicollinearity issues which's predicted by BPM thus making it indrectly optimized for it too.

Why do you try to make it complicated? Because of your beliefs? I don't see anything else there besides what I just shared. I believe Kevin Pelton tried to use RT weighting method for a retrodiction test of BPM, PER, WS (which I did too and found out weighting - normal, ^2, ^4, ^8, ^16, ^32 - is not reliable even one bit) that was fundemantally wrong.

Again, you are on the wrong path if you want to capture isolated player impact.

[DSMok1]

This is a very interesting subject, one I'm following closely as I make adjustments to BPM.

How large are the samples for each level of roster turnover? Are we talking 50 team seasons with 80% turnover or 5 team seasons?

[permaximum]

This is a very interesting subject, one I'm following closely as I make adjustments to BPM.

How large are the samples for each level of roster turnover? Are we talking 50 team seasons with 80% turnover or 5 team seasons?

Tests were done on game-level. Each roster turnover level has the amount of games predicted in parentheses.

[Nathan]

-The sample size issue in games played with 100% or near-100% roster turnover is compounded by the fact that many of those games were probably played by the same players.

-Instead of simply predicting winners, you should probably try predicting margin instead, which holds a lot more information.

-Your are misinterpreting your results. If stats like RPM, BPM, etc. are having a hard time predicting impact at high roster turnover, that means one of two things. Either they're far less accurate than we thought, or player impact itself tends to fluctuate a lot under conditions of high roster turnover. Obviously the latter is more likely. When Harrison Barnes moved from GS to DAL, his role changed completely, and as a result his impact likely changed too. This doesn't mean that estimates of his impact in GS were bad, it just means that his impact depends on what role he's asked to play.

[permaximum]

-The sample size issue in games played with 100% or near-100% roster turnover is compounded by the fact that many of those games were probably played by the same players.

-Instead of simply predicting winners, you should probably try predicting margin instead, which holds a lot more information.

-Your are misinterpreting your results. If stats like RPM, BPM, etc. are having a hard time predicting impact at high roster turnover, that means one of two things. Either they're far less accurate than we thought, or player impact itself tends to fluctuate a lot under conditions of high roster turnover. Obviously the latter is more likely. When Harrison Barnes moved from GS to DAL, his role changed completely, and as a result his impact likely changed too. This doesn't mean that estimates of his impact in GS were bad, it just means that his impact depends on what role he's asked to play.

1. Wrong. Because I know the data.
2. Completely useless in this big of a sample and potentially deceiving. Blowouts, white-flag players and veterans' tendency to not give a crap about scores may affect the results on the deceiving side about 0.5-1% at best. (Actually I still tested with margin until the very end of all tests and realised it was completely unnecessary. The results differed around 0.5% )
3. I'm not. They're far less accurate and you pretty much summed the purpose of this test. Evaluating player impacts regardless of teams, lineup synergies, coaches and roles. Players' skill level shouldn't change depending on where he plays at.

[mystic]

First: Which RAPM version did you use? Prior informed RAPM does better in such tests than 1yr RAPM.

Regarding your latest response:

1. I think you should check the data again, because what Nathan said is likely true. Also, when I tested that on season basis (using y-1 data to "predict" SRS in y) the "prediction became worse for bad teams with a lot of younger players on the roster. I can see how that will screw up the results on a game by game basis.

2. From my tests and my betting experience: Some metrics (including RAPM derivates) do slightly better when predicting margin instead of the winner.

3. The metrics aren't saying much about the player's skill level, but rather something about his impact. When players changing their role, they may use different skills. It makes a big difference for a lot of players, whether they are featured as 1st/2nd option with more iso plays or whether they are more often off-ball players with more catch&shoot situations on offense. The latter will usually lead to less turnovers and higher scoring efficiency.

As a comment regarding the results: I'm not surprised that mpg is doing pretty well. Coaches aren't idiots and they know what they are doing. It makes much more sense to question the results of various metrics instead of claiming that coaches have no clue what they are doing. Also, there is a recency bias; the further away the prediction is, the worse it becomes. PER suffers less from that phenomena than any other popular metric I tested (I did not test BPM). Last but not least: Age adjustment improves PA a little bit for basically all metrics.

[Nathan]

"Evaluating player impacts regardless of teams, lineup synergies, coaches and roles. Players' skill level shouldn't change depending on where he plays at."

I disagree strongly with this. There is no such thing as player impact regardless of team/role. Even under the dubious assumption that players' skills don't change much from year to year, different lineups, coaches, at teammates, push different skills to the forefront, and that certainly affects a player's impact.

[Crow]

Whoops, I did it again. I read the Saturday RAPM chart and saw the R but not the A. My apologies again. I still have similar thoughts about the RPM graph but am sorry to have repeatedly confused it with other graphs.


My history with RAPM and RPM shows I take the defensive side with critics but I have also frequently taken the offensive side with the designers of APM, RAPM, RPM, BPM, PER, WS, WP, etc. to critique them, push them to be better. Dialogue sometimes helps and is worth a try. I get off track some especially when I rush but I'll also make good points too, recognized and followed up on or not. Now thar we are here with others engaged in the conversation I'll probably step back a bit and re-read more carefully. Among the things I hope others will discuss further is whether coach / organization / system should be included full-time in models / future models that are better at recognizing player talents /roles and what colors them.

[DSMok1]

permaximum, this is very useful research you are doing!

It appears you are taking season-long stats from Y-1 and using them to predict each of the 82 games in year Y. Is that correct?

Something that you will find with that approach, which may cause issues with the findings, is that unusual games will show high "roster turnover".

• Blowouts typically end up with unusual minutes distributions. Bad players play more when the team plays well! (Similar to the causation issue of the football running=winning idea.)
• Games with many players injured or resting. These would have high turnover, but may not reflect how those players would play together if they played together on a normal basis.
• End of season games, where one or both teams have little to play for. These games often feature unusual minutes patterns and players who haven't played get a chance to play.

[permaximum]

First: Which RAPM version did you use? Prior informed RAPM does better in such tests than 1yr RAPM.

Regarding your latest response:

1. I think you should check the data again, because what Nathan said is likely true. Also, when I tested that on season basis (using y-1 data to "predict" SRS in y) the "prediction became worse for bad teams with a lot of younger players on the roster. I can see how that will screw up the results on a game by game basis.

2. From my tests and my betting experience: Some metrics (including RAPM derivates) do slightly better when predicting margin instead of the winner.

3. The metrics aren't saying much about the player's skill level, but rather something about his impact. When players changing their role, they may use different skills. It makes a big difference for a lot of players, whether they are featured as 1st/2nd option with more iso plays or whether they are more often off-ball players with more catch&shoot situations on offense. The latter will usually lead to less turnovers and higher scoring efficiency.

As a comment regarding the results: I'm not surprised that mpg is doing pretty well. Coaches aren't idiots and they know what they are doing. It makes much more sense to question the results of various metrics instead of claiming that coaches have no clue what they are doing. Also, there is a recency bias; the further away the prediction is, the worse it becomes. PER suffers less from that phenomena than any other popular metric I tested (I did not test BPM). Last but not least: Age adjustment improves PA a little bit for basically all metrics.

1. Here I'm looking at the data. It's not true.

2. Correct but trust me it's very slight. I don't have the data tested against margin because I deleted it afterwards because of the very slight difference and it was only making things unnecessarily complicated. It's pretty tiresome to work GBs of sheets.

3. I both agree and disagree. Agree that it's more about impact, disagree that they're even good enough at sharing the total impact among players. Teammate interactions and synergies have considerable effect on the total impact and it's not ideal to share them to individual players if you want to evalute sole player impact.

Age adjustmend does indeed improve it a bit (I tested it too for a small amount of time) but it doesn't affect the purpose of this test at all in any scenerio.

[permaximum]

"Evaluating player impacts regardless of teams, lineup synergies, coaches and roles. Players' skill level shouldn't change depending on where he plays at."

I disagree strongly with this. There is no such thing as player impact regardless of team/role. Even under the dubious assumption that players' skills don't change much from year to year, different lineups, coaches, at teammates, push different skills to the forefront, and that certainly affects a player's impact.

Perhaps player skill is the better word although I believe all players have an "impact range" that reflects their skill level.