Poll: RPM's degree of efficacy in sorting players

[permaximum]

I want to repeat: Using current advanced metrics such as RPM, BPM, WS, WP, PER, RAPM etc. to compare player impacts is a HUGE mistake. Also, individual TS% (or FG%, eFG% etc.) is the most overrated stat (even if it's compared to league average and position).

These two are the biggest mistakes that people tend to do.

This is just a very small part of the results of tests I did last year. I tested more metrics and I did different kinds of tests but I'll reveal them later.

Code: Select all

|     All      |     BPM     |     MPG     |
+---------------+-------------+-------------+
| Full          | 0.645144601 | 0.591365306 |
| half(19443)   | 0.655145811 | 0.611994034 |
| quarter(9741) | 0.659480546 | 0.634637101 |
| 50%+(4475)    | 0.661452514 | 0.646480447 |
| 55%+(2693)    | 0.662829558 | 0.656145563 |
| 60%+(1502)    | 0.662450067 | 0.663115846 |
| 65%+(796)     | 0.645728643 | 0.655778894 |
| 70%+(371)     | 0.582210243 | 0.603773585 |
| 75%+(166)     | 0.560240964 | 0.590361446 |
| 80%+(87)      | 0.551724138 | 0.574712644 |
+---------------+-------------+-------------+

0.63=HCA

Against simple linear box-score metrics the graph becomes even much worse for RPM and BPM. I won't get to them yet.

[Crow]

There are many things about this post that need explanation. I can't guess what the first column is, don't know what PA or RT are. Look forward to the explanation.

[sndesai1]

first column might be roster turnover %, thoguh not sure what full/half/quarter would mean in that case

i can't even guess what pa and rt mean.

[permaximum]

Code: Select all

+-----------------+-------------+-------------+
|    2002-2016    |    RAPM     |     MPG     |
+-----------------+-------------+-------------+
| Full            | 0.624676802 | 0.588737201 |
| half(9727)      | 0.630204585 | 0.611596587 |
| quarter(4868)   | 0.631881676 | 0.641947412 |
| 50%+(2584)      | 0.627321981 | 0.643962848 |
| 60%+(851)       | 0.614571093 | 0.652173913 |
| Top300(305-tie) | 0.603278689 | 0.639344262 |
+-----------------+-------------+-------------+

RPM next.

[Crow]

Thinking for a minute, RT is probably roster turnover. But that is where is stops for me.

[rlee]

PA= Prediction Accuracy

[Crow]

Ok thanks. Always good to label graphs clearly. Some figured it but two letter abbreviations and little write-up is not very user friendly when folks are jumping in and out of slowly developing threads.


If it is a huge mistake to use any or all of what permaximum dismissed, what's left?

Eye test? Key stats and current metrics aren't as good as desired but they tend to be better on average than or better in combination with eye test in my experience- including in team win contests and in the betting world- than subjective, usually partial, imprecisely aggregated eye test evaluations. Eye tests are usually not fully communicated or systematically evaluated. Yeah you can form an opinion about the opinion and you might be right. But you might not be and not know it til later, if ever.

What else? Minutes alone may be pretty good but are off a lot too in many cases. Minutes only has a small lead in part of the results and trails in more.

Simple linear metrics? I guess I should wait to see these results. But I can't help but be concerned by the undeniable and imo huge absence of shot defense, defensive help and impact on the offense of others. How do you settle on something with big gaps when there is even more data to try to get a handle on probably 20-40% of all player impacts in these areas that simple linear metrics don't address? Even if current metrics fall short, the effort may get better over time- but only if you try.



Blends of simple linear metrics with RPM (or other) I hope to see those results also. I did see the results the sports skeptic guy produced a few years ago testing many combinations. He found a blend of metrics including RPM did best, better than any alone. Rather than discrediting separately, I have preferred to blend and get farther than anything else. Provably still will. Including adding eye test with care, where it adds / seems necessary.

[tarrazu]

Still don't understand what is being presented here despite deciphering the acronyms. Is it your intent to be mysterious and ambiguous? Sounds like you potentially have some interesting findings, but I'd rather not try and guess as to what you're presenting.

[sndesai1]

the overarching idea seems to be that with a high % of roster turnover, these all-in-one metrics are worse at predicting value than something as simple as minutes per game

but like tarrazu said, it would be good to know the specific details and analysis instead of us all guessing

[Crow]

When is roster turnover real high in just one off-season? Not very often.

[permaximum]

Since I did the test last year, not even one game from this season was predicted. RPM only had 3 years of data then. RAPM had 16 years of data. I found out in-sample was not an issue for BPM in 2001-16 so all games were predicted for BPM.

Prediction Target: 1984/85-2015/2016 - 38658 games (Regular and Playoff)
Source Data: 1983/84-2014/2015 (Regular: Playoff games were excluded after finding out the undesirable punishment effect in metric scores for players that take part in post-season games because of high intensity)

Tested Metrics: All popular advanced public metrics - RPM (2627 games were predicted - 2 years), RAPM (19345 games were predicted - 15 years), BPM, WS and PER. As for WP it was dropped very quickly because of very subpar performance. On top of those; MPG + USG + 2 different non-empirical simple linear boxscore metrics were included.

Retrodiction Method: Each game was predicted by calculating previous year's per-possesion or per-minute metric score depending on the metrics' formula and regular season average in the previous year for players that take part in the game. Then each team assigned a Total Metric Score by using the actual minutes or possessions and thus winner was predicted. On 2-6 rare occasions where the metric scores were equal to each other home team claimed winner because of the factor of HCA. Players below 250 MP in the previous year and rookies were assigned average values. Then each game's unique roster turnover rate was calculated depending on the new founding of teams, signings, trades, rookies etc and most importantly their in-game minutes for that particular game. E.g: 90% RT minutes from team A and 80% RT minutes from team B makes the game's turnover score 85%. For those that curious there were only 6 games where the roster turnover was 100%. It means both teams had 100% roster turnover and all minutes came from completely new players for those teams. Average roster turnover rate for games was around 33%.

PA/RT: Prediction Accuracy / Roster Turnover Graph.
Y axis: Prediction Accuracy
X axis: Roster Turnover.
In-Parentheses: Game Count. Full means the complete data (38658 games or 19345 games for RAPM comparisons and 2627 games for RPM comparisons). Half and quarter are obvious. % percentage numbers represent Roster Turnover Rate. Top 500-400-300 etc. means games with the most roster turnover rate. There has been lots of ties for roster turnover positions between the games so in those cases all of them were included. That's why half or quarter or top 300 200 etc. may actually represent a bit bigger numbers.

As promised; RPM. RPM comparison became problematic because of the sample problem. I couldn't go to high roster turnover rates because the sample became very small and randomness took part. Still the graph pretty much shows the obvious trend and since RPM and BPM follows an extremly similar path it can be concluded RPM suffers the same consequences. If I could go to higher roster turnover rates both metrics would suffer and MPG would prevail.



RPM-BPM following extremly similar path when it's limited to RPM data size for comparison.



And BPM-MPG on the whole data (38658 games) once again.



PER, WS, USG and simple linear box-score metrics will come later.

Shortly, PER is a bit better. WS is worse. Simple linear box-score metrics do shine but still it's not enough.

As for Usage, I included it because once Kevin Pelton said something along the lines of Player Usage Rate translates to other teams and scenerios very well and that's why PER was closing the gap between BPM when the roster turnover rate became greater. It couldn't be more wrong. Usage was a lot worse than other metrics at any roster turnover rate.

Edit:For those that are curious, blends made it worse. In the best case scenerio, you could only get marginal improvements if you optimize for ONLY 1 RT LEVEL by sacricing the trend, graph and pretty much everything. That marginal improvement for an exact roster turnover rate is completely artificial. Blends do suffer overfitting a lot. But I see some still confuse this with predicting next year's team wins which is a completely different thing where RPM is directly and BPM is indirectly optimized for it.

[sndesai1]

interesting. any thoughts on why predictiveness of mpg somehow becomes higher as there's more turnover? (compared to itself going from say 50% to 60-65%)
for the rpm sample, all 3 seem to get better so that might just be part of the sample size issue. but for the large sample, it happens for mpg but not bpm.

[Crow]

Thanks for this much.

So 3 charts. Chart 1 RPM always leads. Chart 2 RPM almost always leads. Chart 3 RPM leads most of tge way, including at the average turnover rate of 33%. How does this square with the initial dismissal of RPM? So far my understanding is that RPM is better except in limited, pretty rare cases.

Do you plan to test multi-season RPM? It is available I believe.

Or go the other way. This might be possible now but perhaps in future: test RPM from quarter of season in second quarter and so on. Or use the latest rolling RPM estimate for the next game. Does it do better than previous season RPM? Is the critique appropriate for RPM fundamentally or is it really about the RPM lag? Players do change; they are not static. It may be wrong to expected old estimates to be precise today or estimates from a partial time period to the best when estimates from longer tine periods are available. Maybe there is an ideal between long term RPM, last season RPM and recent RPM for one purpose or another.

A side use of this data could be to look at RPM vs Minute by coach for full career or last few years. I'd be really in that. Then use it to think about organization disposition to RPM data. Not tgat we know in most cases but this would stimulate further speculation / reaction.

Has any time am tried to use RPM or RAPM at factor level and lineup or principal pair or trio level to game plan usage distribution on offense and other strategies (risk taking to the rim, offensive rebound effort, etc.) and defensive priorities / strategies? I doubt it but if there are believers in RPM equal to od above other guidance systems, one should try to make use of it at a practical level (i.e. calculate matchup and lineup differentials for offense / defense or at factor level and with consideratiob of factor level interactions in play sequences.

[permaximum]

1. Ideally, the best player metric is the one that's the best at 100% roster turnover rate. But the sample size at 100% RT becomes a big problem so we analyse these charts and check the trend and momentum. +65-70% is the sweet spot of RT and sample size for my test. So, prior roster turnover spots are insignificant for our purpose which is isolating player impact from any team-related scenerio.
2. Actually, there's only 1 chart for RPM where BPM passes it and MPG catches it.
3. Like I said, BPM eventually passes RPM and MPG eventually catches RPM. But I ended the graph at 61% roster turnover rate where MPG catched RPM because of the small sample size. However it's 100% clear MPG eventually passes RPM at 61%+ roster turnover rates too just like it passed BPM because of the reasons I mentioned. The graph itself and momentum is a pretty big indicator.
4. Actually I "know" MPG passes RPM while they both start to get worse but since the sample becomes very small I can't call it statistically significant. That's why I didn't include it. It's questionable even a sample size of 100 games is significant or not.
5. Simple linear box score metrics are yet to come. And I can confirm things will get even uglier for advanced metrics starting with RPM and BPM.

So far; I tried to show RAPM, RPM and BPM are all worse than "even" MPG at sole player impact.

Edit: I don't plan to test anything further neither do I see any point in it for my purpose. Because all metrics were tested fairly and equally. For example I could test multiple years of PER, BPM etc. too just like you mentioned for RPM. However I know from my previous tests those kind of multi-year values, sygnergy effect, quarterly predictions, game to game update of metrics scores etc. are immensely pointless AND WRONG for various reasons. You can find some documents about that on the internet. These tests should be sufficient enough to prove these advanced metrics are really bad at evaluating player performance. Besides, doing these kind of tests takes enormous time and I did these tests about a year ago. There are some neceassary files missing for calculations.

[Crow]

Thanks for the correction. The print was real tiny on my phone screen and I too quickly misread Chart 2 & 3 and was overfocused on RPM. So RPM always beat minutes in Chart 1. I am not fully understanding or convinced that very high or 100% roster turnover is the ideal or only important test point (given how far it is from average).