Poll: RPM's degree of efficacy in sorting players

[permaximum]

permaximum, this is very useful research you are doing!

It appears you are taking season-long stats from Y-1 and using them to predict each of the 82 games in year Y. Is that correct?

Something that you will find with that approach, which may cause issues with the findings, is that unusual games will show high "roster turnover".

• Blowouts typically end up with unusual minutes distributions. Bad players play more when the team plays well! (Similar to the causation issue of the football running=winning idea.)
• Games with many players injured or resting. These would have high turnover, but may not reflect how those players would play together if they played together on a normal basis.
• End of season games, where one or both teams have little to play for. These games often feature unusual minutes patterns and players who haven't played get a chance to play.

Actually I did these tests a long time ago and I wasn't going to share the results. I thought this thread was very related to them and decided to share some of the results. As for my method, I take regular-season per-minute or per-possesion averages (depending on the metric) from Y-1 (some exceptions for rookies, injured players and below 250MP players) and predict all games that happen both in regular season and playoffs in the next season depending on the actual possession or minutes that take place in those games.

For more detail, you can check here where I explained it in a previous post.

Retrodiction Method: Each game was predicted by calculating previous year's per-possesion or per-minute metric score depending on the metrics' formula and regular season average in the previous year for players that take part in the game. Then each team assigned a Total Metric Score by using the actual minutes or possessions and thus winner was predicted. On 2-6 rare occasions where the metric scores were equal to each other home team claimed winner because of the factor of HCA. Players below 250 MP in the previous year and rookies were assigned average values. Then each game's unique roster turnover rate was calculated depending on the new founding of teams, signings, trades, rookies etc and most importantly their in-game minutes for that particular game. E.g: 90% RT minutes from team A and 80% RT minutes from team B makes the game's turnover score 85%. For those that curious there were only 6 games where the roster turnover was 100%. It means both teams had 100% roster turnover and all minutes came from completely new players for those teams. Average roster turnover rate for games was around 33%.

1. You're right. But the sample size is too big for it to become a problem and a better metric should actually be good in those situations too. That's what I was looking for. And finally, it's the exact same playing field for all metrics where it's completely fair.
2. That's exactly the situation we're looking for.
3. That's exactly the situation we're looking for.

To summarize, more chaos is better for what I tested which is individual player impact or skill.

[Nathan]

"Evaluating player impacts regardless of teams, lineup synergies, coaches and roles. Players' skill level shouldn't change depending on where he plays at."

I disagree strongly with this. There is no such thing as player impact regardless of team/role. Even under the dubious assumption that players' skills don't change much from year to year, different lineups, coaches, at teammates, push different skills to the forefront, and that certainly affects a player's impact.

Perhaps player skill is the better word although I believe all players have an "impact range" that reflects their skill level.

I think that's true. In situations with little/no roster turnover, individual players can be expected to have about the same impact year after year (adjusted for aging). In situations with high roster turnover, players end up at all different places than before in their respective impact ranges. As a result, previous year impact is, as you observed, less predictive there. At the very least, this exposes a substantial limitation of (at least some popular) advanced stats. They may be able to measure impact, but they can't really tell you where a player is in his impact range, and thus they can't accurately predict how a player's performance might change under different circumstances.

It's a mystery to me why MP is a relatively strong predictor in high-turnover situations. Perhaps things along the lines of what DSMok1 mentioned. It's an interesting but daunting idea to try to come up with a metric to predict how player performance changes in high roster turnover situations.

[Crow]

Minutes as a predictor by coach would get at how many coaches are near the average and how many are outliers high and low. How many should be praised, passed by fairly quickly or critiqued hard.

If simple linear metrics do good or "best", then BPM maybe should do away with assist * rebounds component?

"I want to repeat: Using current advanced metrics such as RPM, BPM, WS, WP, PER, RAPM etc. to compare player impacts is a HUGE mistake."
COMPARE. How about "analyze" or "assess"? Comparison is a problem. Maybe over-experienced but probably not completely avoidable either.

"Again, you are on the wrong path if you want to capture isolated player impact."
why do we want to capture isolated player impact as opposed to non-isolated or player in context? Presumably for trade talk or for "comparative greatness" talk. I am not really interested in latter. I am somewhat interested in former but GMs are going to do and not do what they want so it is tempting but fairly pointless chatter.

Discrete stats get at many but not all skills.

People talk about position-less basketball but traditional positions & roles still overlap more than they don't. But perhaps we should have role sensitive metrics. Several assessments of how well a player "fits" a traditional role, even though maybe holding onto roles might not always be a good idea. It think seems to be a common practice likely to continue though, regardless if it is wise.

[permaximum]

The first box-score metric I tested:

David Lewin/Dan Rosenbaum - Alternate Win Score: (pts+0.7*orb+0.3*drb+stl+0.5*blk+0.5*ast-0.7 * fgx-fg-0.35*ftx-0.5*ft-tov-0.5*pf)/mp



Like I said before considering the size of the data sample, somwhere between 65% and 70% RT is the sweet spot. I tried lots of other stats; stat interactions, metric interactions, blends. That's why I can comfortably call it the sweet spot in this case. When the sample becomes less than 300 games or so, several variables (stats, all possible normal and advanced stat interactions, all kinds of blends, metrics) had shown erratic behaviours which confirms the sample issue. So, I don't advise you to look further than 70% RT but I included higher roster turnover rates for the graph anyways.

And yes I really did test all possible stat interactions, metric interactions, blends along with basic/advanced stats and metrics to create a better metric for evaluating individual player skill. Improvements were artificial. Overfitting was too great. I guess after a few weeks of focusing on this thing alone I couldn't come with something significantly better. A new metric has to be developed. Anyways, I still suspect there are even better box-score merics out there that do fine regardless of high roster turnover rates.

[permaximum]

The second box-score metric.

Tom Thibodeau: (2*fg-fgx+ft-0.5*ftx+3*3pm-1.5 * 3px+reb+ast-pf+stl-tov+blk)/mp



PER



Comparison of Box-Score Metrics:



Unfortunately, a significantly more complex metric called PER is good at nothing. There isn't even a minor reason to use PER at any case or any occasion.

[permaximum]

WS



Usage for curious people:



It's worse than all metrics and MPG by a large margin although it does improve compared to BPM, RPM and WS with higher roster turnover.

[permaximum]

If you want, I can compare all metrics at once or any combination of metrics and/or stats depending on the request. And I can give results for any year depending on the request.

Data: 1983-2016 (Cutoff is 1983 because of the fundemental change of the game with the inclusion of 3pt line)

Comparable stats:
ORB%
DRB%
AST%
STL%
BLK%
TOV%
TS%+ (compared to league average)
TS%
USG
MPG
2P-A (A stands for compared to league average)
3P/A
FT/A
2PX/A
3PX/A
FTX/A
2P/100 (Per 100 possession)
3P/100
FT/100
2PX/100
3PX/100
FTX/100
TSA-A
TSA/100
PF-A
2P%
3P%
FT%
EFG%
FT/FGA
3P/FGA
3PAR FTR
AST/TO
STL/TO

and All Possible Interactions

Comparable Metrics:
PER
WS
BPM
Thibodeau
AWS
RAPM (single-year to make the comparison fair)
RPM
USG (Signicant Stat)
MPG (Signicant Stat)

For example TS%+'s PA is 0.544 at full data prediction

2015-2016's prediction accuracy for metrics from best to worst. (All games) With higher roster turnover rates the same parttern is observed for each year. I can give per-year roster-turnover charts too if anyone wants.

BPM 0.676291793
RPM 0.671732523
WS 0.670212766
Thibodeau 0.655775076
RAPM 0.655015198
AWS 0.654255319
PER 0.636778116
MPG 0.574468085
USG 0.559270517

[Crow]

You dismissed the usefulness of blending but I haven't seen any charts demonstrating their non -helpfulness. There are a near infinite number of blends so either you search them all or you can't definitively dismiss their performance as a whole.

But how about starting with showing charts for these two:

The explanatory blend was created by adding together .5* old WP48, .35*ASPM, and .15*WS48. The predictive blend was created with .5*ASPM, -.5*WP48, .35*RAPM, and .3*WS48 and old WP48, then dividing the sum by .95.

https://sportskeptic.wordpress.com/2012 ... ect-blend/

Would you be willing to show at least one chart of metric performances for a single coach or team? Maybe Popovich (or Pol before 2006-7 vs. after), Stevens, S Van Gundy... or whoever you prefer

On that last set of results, after all the criticism BPM and RPM score best ahead of WS and Thibideau and minutes was second worst??

[permaximum]

You dismissed the usefulness of blending but I haven't seen any charts demonstrating their non -helpfulness. There are a near infinite number of blends so either you search them all or you can't definitively dismiss their performance as a whole.

Would you be willing to show at least one chart of metric performances for a single coach or team?

On that last set of results, after all the criticism BPM and RPM score best ahead of WS and Thibideau and minutes was second worst??

1. I tried every possible metric blend and lots of blends with stats etc. (mainly to create a better metric). Results were almost always the same. Artificial very slight improvement (not even 2%) at a given roster turnover rate (e.g.65%) at the cost of sacricing prediction accuracy for games with higher and lower roster turnover rates (almost up to 15%). This is not an exact science where you can optmize all the data when you doesn't take roster turnover into account. It's more complicated than that.

2. A coach or a team? Hmmm... I can do that I guess. But it would take some time. A year ago while I was working on this I could quickly give it but now files of steps to produce these results are missing. What's the point in it anyways?

3. Not sure if you're serious or drunk or something's wrong with you nowadays. Because the last set of results don't take roster turnover into account and it's 2015-16 season only. The whole point of this research was proving although those metrics are good at yearly predictions, they're not actually good at evaluating players. Even the worst metrics pass them when the roster turnover gets higher. First you misinterpreted RAPM with RPM and now you missed the whole purpose of pages of discussion. I hope you're OK my friend.

[permaximum]

YThe explanatory blend was created by adding together .5* old WP48, .35*ASPM, and .15*WS48. The predictive blend was created with .5*ASPM, -.5*WP48, .35*RAPM, and .3*WS48 and old WP48, then dividing the sum by .95.

https://sportskeptic.wordpress.com/2012 ... ect-blend/

1. Ideally a retrodiction test should be done at game level. Seasonly predictions are not reliable enough. But I know retrodiction at game-level takes too much time so laziness is the factor here.
2. That doesn't take roster turnover into account. Creating a blend that's constantly better than any single metric at higher roster turnover rates is impossible. You have to create a new metric.
3. Yes, blends are generally better than single metrics at yearly team win or margin predictions. I guess everyone knows that by now and I have been saying that for a long time. So, what's your point?

[Crow]

I'm fine. Not a simple or perfect reader / direction follower / correspondent but I try.

The point of the coach test is to get at how coach behavior compares to metrics / discrete stats. Which is more similar? Do they align more with PER or Winshares or RPM?
With a comprehensive chart one could have a better case that a coach gives more or less weight than usual to shot defense, usage or whatever. How do coaches / organizations compare thru this lens? Could be very interesting application of your valuable work. Can help find attractive trades where stat orientation is different. Could explain player minutes, lineup preferences to some greater degree than without the aid of this info.

Yes the results at end are of a specific kind and different than your fixation on accuracy at the unusually high roster turnover level (where everything does the worst). But why do you cite these results if you then react negatively to my mention of them? I didn't say they were the main point, everything or more important or push the other results off the table. Just that you cited them and that they are contradictory to your main narrative from the high roster turnover rate datapoints.



So you dismissed blends at game level but now it comes out you apparently haven't tested them there or much and are not too interested in doing it in future. (I know season prediction is different). I see. Thanks for that info.

You did what you did and thanks for sharing. I have tried to understand it. Made some mistakes and apologized for them. Much of my commentary goes other angles. That may be confusing / unwelcome but is my reaction, what I thought added to what you done & said, what might provide opportunities for you (and others) to say & do more. You can engage on those angles or not in addition to what you intend, my fellow analytics traveler.

[permaximum]

I'm fine. Not a simple or perfect reader / direction follower / correspondent but I try.

The point of the coach test is to get at how coach behavior compares to metrics / discrete stats. Which is more similar?
With that information one could have a better case that a coach gives more or less weight than usual to shot defense, usage or whatever. How do coaches / organizations compare thru this lens? Could be very interesting application of your valuable work. Can help find attractive trades where stat orientation is different. Could explain player minutes, lineup preferences to some greater degree than without the aid of this info.

Yes the results at end are of a specific kind and different than your fixation on accuracy at the unusually high roster turnover level (where everything does the worst). But why do you cite these results if you then react negatively to my mention of them? I didn't say they were the main point, everything or more important or push the other results off the table. Just that you cited them and that they are contradictory to your main narrative from the high roster turnover rate datapoints.



So you dismissed blends at game level but now it comes out you apparently haven't tested them there or much and are not too interested in doing it in future. (I know season prediction is different). I see. Thanks for that info.

You did what you did and thanks for sharing. I have tried to understand it. Made some mistakes and apologized for them. Much of my commentary goes other angles. That may be confusing / unwelcome but is my reaction, what I thought added to what you done & said, what might provide opportunities for you (and others) to say & do more. You can engage on those angles or not in addition to what you intend, my fellow analytics traveler.

Well, I can do coach reasearch but I'm not interested in that. We'll see.

I think you're not interested in player evaluation then if you don't give much attention to roster turnover because the league average for games is 33%.

Unfortunately I can't put a proof for blends here so you're free to don't believe me but I tested them. My foundings for blends were like I said before, worse. I wish I didn't delete those files because I deemed them unnecessary. But like I said before, I did this research almost a year ago.

Edit: I found some of the missing files. After a couple weeks of work or so I can be ready to generate results for blends. But we'll only find what I was saying and I'll only waste weeks of my free time.

[Crow]

Roster turnover rates is an interesting angle but I'll stay more interested in rates near average than focusing on unusually high.

Our interests and priorities vary somewhat. Just the way it is. Maybe I focused on the differences too much but that seemed potentially more useful.

You say blends are worse but earlier you said they were better but not at certain low and high turnover rates. They don't sound the same to me but enough on that.

Do what you want and only what you want and don't blame me. I have interests, suggestions, opinions but it is your choice what to do or not.

Might suggest a new stat mix / metric if you don't pursue further metric blend analysis.

Still interested in hearing more thoughts from others.

[permaximum]

For what you're looking there are 2 good metrics. RPM and BPM. Their blend will even be better. You just need to give more weight to RPM.

There's probably some misunderstanding about blends. I say blends are better for yearly team win and margin predictions. But at high roster turnover rates they get worse than single metrics. If I try to produce a blend that's optimized for let's say 65% roster turnover, it will only be artificially better at 65% and it will get even worse at higher AND lower turnover rates also.

Long story short, BPM, RPM, RAPM, WS, PER and blends are not good for evaluating players. Simple linear box-score metrics are better than those although they're not good at capturing defense data. Still I don't consider them good enough neither. When someone eventually looks into what I researched and develop a metric that stays stable above the prediction accuracy of 0.7 (considering HCA alone is 0.63) at all roster turnover rates, we'll finally have a metric that's good enough to individually compare players.

[Crow]

If you want to test some stats how about:

ast% * reb%
3pta per 100 possessions
Ft rate
3pt rate / FT rate
TS% * usage