Poll: RPM's degree of efficacy in sorting players

[permaximum]

Alright I'll generate those results.

BTW I have to correct something. HCA=0.615 and Average HCA Margin is = 3.63.

[permaximum]

Quick results. They're all worse than TS% (0.544) and USG% (0.538) except FTR. It's close. OFC these don't take roster turnover rates into account. Roster turnover comparisons next.

Code: Select all

+----------+-------------+
|   FTR    | 0.542656113 |
+----------+-------------+
| AST%*RB% | 0.530343008 |
| TS*USG   | 0.518366186 |
| 3PTA/100 | 0.506699778 |
| 3PAR/FTR | 0.495705934 |
+----------+-------------+

[permaximum]

Interesting results.

Higher roster turnover rates made TS%*USG% shine above others although it was initially behind FTR and AST%*REB%. I come to the conclusion that individual FTR and AST%*REB% are very well affected by teams and different situations that players are in unlike TS%*USG% which will better translate to other teams and situations compared to previous stats.

[Crow]

Thanks. Worth remembering when tinkering with lineups or making trades.

[Jinxed]

permaximum,

I have a bit of a philosophical disagreement with your approach of using high roster turnover rate as way of evaluating a player's skill.

A player who plays well with high RT is not a necessarily more skilled player than one who plays well with normal RT, he's just more skilled at playing with high RT. It's analogous to being good at pickup basketball (when any random person can be on your team) vs being good in a league setting with a coach, a defined role, and familiarity with teammates.

These are different skill sets. And I see no reason why we should call the former "the better evaluator of player skill".

Your tests don't show that these simple metrics Tom Thibs/AWS are better than RPM at evaluating player skill, they just show they are better at evaluating which players are better at playing with high RT than RPM.

[permaximum]

permaximum,

I have a bit of a philosophical disagreement with your approach of using high roster turnover rate as way of evaluating a player's skill.

A player who plays well with high RT is not a necessarily more skilled player than one who plays well with normal RT, he's just more skilled at playing with high RT. It's analogous to being good at pickup basketball (when any random person can be on your team) vs being good in a league setting with a coach, a defined role, and familiarity with teammates.

These are different skill sets. And I see no reason why we should call the former "the better evaluator of player skill".

Your tests don't show that these simple metrics Tom Thibs/AWS are better than RPM at evaluating player skill, they just show they are better at evaluating which players are better at playing with high RT than RPM.

What you're saying (being good in a league setting with a coach, a defined role, and familiarity with teammates) can be a small factor but I don't believe it's significant. All NBA-level players should fit into different systems well enough. Any impact change "that's related to player skill" that happens because a given player is in a different role should be very minimal almost to the extend of non-existence. It's more about their environments instead of themselves. At this stage, player impact can be limited or maximized by outside interference... not the other way around. The effect of player adaptability in such a huge data shouldn't have changed results this dramatically. Actually it shouldn't have changed anything at all. In this research, 38658 games were predicted by metric values which were affected by thousands of players. In core, metrics were in the race. Not the players.

For example, AWS being better than all metrics at higher roster turnover rates mean that the metric predicted game outcomes better than others because it's more detached from teams' synergistic effects on players.

RPM or any +/- related metric try to distribute "TEAM SUCCESS" to players that take part in those games. What those metrics miss is that TEAM SUCCES is not equal to PLAYERS. The equation should be;

TEAM SUCESS = a.P1 + b.P2 + c.P3 + d.P4 +e.P5 - f.P6 - g.P7 - h.P8 -i.P9 - j.P10 + HCA + REST + CONSTANT

CONSTANT = Lots of things but mainly something like Brand Value, Score Margin, Game Time, Timeout, Momentum, Garbage Time etc.

a,b,c,d,e,f,g,h,i,j = Coefficients: Synergistic effect of lineups, roles, coach, team and the city itself on player impacts.

Compared to other +/- metrics, RPM try to adjust for HCA, REST, Score Margin etc. but it can't adjust for those coefficients since it still has to use the core principle of APM. Deduction. These APM based metrics share those synergistic effects and constants almost equally to players on the court which is unfair and completely misleading. That's why some players in good teams that barely pass the eye-test tend to score very well on PM based metrics. Like Nick Collison. Like Amir Johnson. Like every role player in top teams. On top of that, these metrics are biased by priors and effected by multi-collinearity issues of APM.

Although these linear box-score metrics are very simple, they use the method of induction. It's the valid approach and that's why they're better at evaluating player peformances that are isolated from sygnergistic effects. Because they don't capture those effects at all. Although they suffer the consequences of not using an empirical approach and advanced data of today, their advantages still exceed the disadvantages although even they are not good enough considering HCA alone is 0.615. This only proves the fact that current all-in-one metrics' state is pretty poor.

Long story short, evaluating metric performances at very high roster turnover rates is a must if we want to detach players from synergistic effects and these linear box-score metrics don't even have that problem.

[Crow]

If one accepts what permaximum presents, then RPM and other complex metrics that go beyond simple linear accounting might be less reliable when considering trades than simple linear accounting. At least compared to other usages of metrics. But on same team all those non-player factors are the same for everybody on team, except role. For decisions about playing time on same team RPM and other complex metrics capture those beyond player components and explain the totality involved in games with typical roster turnover rates better than those simple linear metrics that lack consideration of those extras. Shot defense is one of those extras that is partly coach / system but I'd still think also speaks to player skill, skill not considered or captured by simple linear metrics. Neither is most of offensive impact on others beyond assists and offensive rebounds.

[mystic]

If one accepts what permaximum presents, then RPM and other complex metrics that go beyond simple linear accounting might be less reliable when considering trades than simple linear accounting.

Well, I would say the conclusion should be that such metrics have trouble with players for whom very little information is available. Teams with higher roster turnover tend to have players getting more playing time with a low or no sample size in y-1. His 250 minutes threshold is not sufficient to give you more reliable results for pm-based metrics.

Boxscore-based metrics tend to be more reliable for low minutes players; there will be less extreme outliers in comparison to pm-based metrics, and usually it is more likely that those outliers will be eliminated by the 250 min threshold.

I would also argue that using an average value for players below the threshold has different consequences for different metrics. For some metrics it makes more sense to use a replacement level value instead of an average value. Also, using prior information will help pm-based metrics more than boxscore-based metrics. I don't follow the argument, that NPI has to be used instead of PI, when it is clear that PI is better than NPI. Why should I leave available information out?

[Nathan]

I do agree that interaction terms have perhaps been applied too liberally in making BPM and related stats. Personally, I have had more success making piecewise-linear models as a first refinement on a simple linear model rather than introducing quadratic terms or other poorly behaved interactions.

[Crow]

Sensible points to my ears.

[permaximum]

If one accepts what permaximum presents, then RPM and other complex metrics that go beyond simple linear accounting might be less reliable when considering trades than simple linear accounting. At least compared to other usages of metrics. But on same team all those non-player factors are the same for everybody on team, except role. For decisions about playing time on same team RPM and other complex metrics capture those beyond player components and explain the totality involved in games with typical roster turnover rates better than those simple linear metrics that lack consideration of those extras. Shot defense is one of those extras that is partly coach / system but I'd still think also speaks to player skill, skill not considered or captured by simple linear metrics. Neither is most of offensive impact on others beyond assists and offensive rebounds.

Very useful find about comparing players in the same team. I agreed at first. But I have to disagree after some consideration.

Each matchup that happens on the floor has BOTH lineup synergies that are distributed among ALL players on the court regardless of the team. Since +/- of matchups of the league adjusted afterwards, it's a complete mess and impossible to know which players in the same team were awarded by playing teams with very low synergistic effects while the others were punished.

It's simply because equation is wrong. We need to add a few more "variables" and "player coefficient adjusters" to regressions. I don't know how...yet.

[permaximum]

If one accepts what permaximum presents, then RPM and other complex metrics that go beyond simple linear accounting might be less reliable when considering trades than simple linear accounting.

Well, I would say the conclusion should be that such metrics have trouble with players for whom very little information is available. Teams with higher roster turnover tend to have players getting more playing time with a low or no sample size in y-1. His 250 minutes threshold is not sufficient to give you more reliable results for pm-based metrics.

Boxscore-based metrics tend to be more reliable for low minutes players; there will be less extreme outliers in comparison to pm-based metrics, and usually it is more likely that those outliers will be eliminated by the 250 min threshold.

I would also argue that using an average value for players below the threshold has different consequences for different metrics. For some metrics it makes more sense to use a replacement level value instead of an average value. Also, using prior information will help pm-based metrics more than boxscore-based metrics. I don't follow the argument, that NPI has to be used instead of PI, when it is clear that PI is better than NPI. Why should I leave available information out?

I will generate a "minutes from <250MP players / roster turnover chart" for you. I believe 250 MP cutoff should be more than sufficient and PM based metrics shouldn't have any problem with it because it stays roughly the same for any roster turnover rate so the line should be straight and your conclusion should be wrong. But I don't know yet. Maybe you're right. We'll see.

I calculated average values from all metrics myself instead of taking known average values for those metrics. I was considering going for the replacement level approach but I decided this was the right thing to do. Replacement approach hasn't been properly tested and accepted yet and it's not fair to use average values for some while using replacement values for others.

RPM represents everything stands for PI, multi-year (which is actually not fair) and more than them. That's why I used NPI-RAPM. I didn't use previous years' information for box-score metrics either. I bet it would increase their prediction power too.

Edit: Instead of making assumptions based on theories I'm spending time and testing them for no gain to prove something that's very obvious... at least for me. And then I share them. I'm not complaining but I wanted to reveal what I think.

[DSMok1]

Just a few quick notes--I'm following this thread, but very busy right now.

If the high roster turnover games are very rare, sprinkled here and there and not in continuous or season-long stretches, I agree that players that thrive in disorganized "street ball" may do better.

Box Plus/Minus is less stable than linear box score models, and will struggle more than them with very low minutes players. It is far more stable than APM models, though.

Nathan, I would be interested in hearing your ideas on piecewise linear models!

Multi-year would definitely help APM models the most. Box score models get helped less since they are more stable.

[permaximum]

Just a few quick notes--I'm following this thread, but very busy right now.

If the high roster turnover games are very rare, sprinkled here and there and not in continuous or season-long stretches, I agree that players that thrive in disorganized "street ball" may do better.

Box Plus/Minus is less stable than linear box score models, and will struggle more than them with very low minutes players. It is far more stable than APM models, though.

Nathan, I would be interested in hearing your ideas on piecewise linear models!

Multi-year would definitely help APM models the most. Box score models get helped less since they are more stable.

On the opposite. High roster turnover games are continous and spread around season-long stretches. I think this should have been very obvious.

It's definetely wrong to think low minute players take part in these games with high roster turnover and eventually coming to a conclusion that it's why +/- based metrics suffer. I haven't started working on a proof for what I'm saying but eventually you'll see.

I think by this page of this thread, it's pretty obvious some try to make baseless theoritical excuses for the poor showing of PM based metrics instead of accepting the simple and very obvious truth.

[permaximum]

Well, I know the results now but I want to ask something first before I share something unnecessary.

Do you guys have problem with <250 MP players that have been assigned average values for all metrics and blame it for the poor performance of PM metrics for some unknown reason although it's fair and the same for all metrics or do you have problem with players that played between 250 and let's say 1000 minutes and have their metric values calculated?

I think you said 250+ minutes is not enough for PM metrics to reliably capture performances right? So you want the second thing calculated. How many minutes do you guys think these PM metrics need? 1000? 1250? 1500?