Hi,
I'm reading about popular all-in-one metrics such as RAPTOR, EPM, BPM and it seems like many of them try to predict the long-run (often 3 years?) RAPM. My question here is: if you only need 3 years of data to compute RAPM as the target, why not just use RAPM, instead of training a new model to predict RAPM, since most impactful players have been in the league for more than 3 years?
A second question is instead of using long-run RAPM as the target, why not directly try to predict the plus minus of each of those lineups with a regularized method?
Thanks!
Newbie question about why RAPM is used as the target for so many metrics
Re: Newbie question about why RAPM is used as the target for so many metrics
Hello, welcome!
Pure Adjusted Plus/Minus is the only "unbiased" way to estimate a players complete impact on the team's point differential.
Pure APM, though, is unstable even in multi year samples due to the collinearity of players and lack of connection between groups of players leading to very small samples "determining" how player A and player B relate to one another.
Pure RAPM (Ridge Regression) puts a finger on the scale to bias every player toward the mean. This greatly improves stability of the output, but you still need several years of data to get reasonably stable results.
Even 3 year RAPM still has noise in it. It still has error due to the same issues as pure APM, but much less pronounced.
That said, if you're trying to build a metric, the RAPM noise/random errors for individual players will generally balance out--so it is an excellent target to use as a regression basis.
Now pure RAPM still has some issues--the regression towards "average" is not truly appropriate. Low minutes players are most regressed toward this prior, but low minutes players are actually, as a group, worse than average. So an improved version of RAPM regresses toward a prior that accounts for that issue. (I use this improved version of RAPM when developing BPM).
The other big reason to not just use 3 year RAPM is that it's looking at 3 full years of data--if you want to look at one year, or even smaller periods, then RAPM just won't work.
Back in 2011 I wrote up a blog post explaining APM and RAPM, though RAPM was still very new at the time: http://godismyjudgeok.com/DStats/2011/n ... ilization/
As for the second question--That would work, but the data set becomes much, much larger and harder to manage.
Pure Adjusted Plus/Minus is the only "unbiased" way to estimate a players complete impact on the team's point differential.
Pure APM, though, is unstable even in multi year samples due to the collinearity of players and lack of connection between groups of players leading to very small samples "determining" how player A and player B relate to one another.
Pure RAPM (Ridge Regression) puts a finger on the scale to bias every player toward the mean. This greatly improves stability of the output, but you still need several years of data to get reasonably stable results.
Even 3 year RAPM still has noise in it. It still has error due to the same issues as pure APM, but much less pronounced.
That said, if you're trying to build a metric, the RAPM noise/random errors for individual players will generally balance out--so it is an excellent target to use as a regression basis.
Now pure RAPM still has some issues--the regression towards "average" is not truly appropriate. Low minutes players are most regressed toward this prior, but low minutes players are actually, as a group, worse than average. So an improved version of RAPM regresses toward a prior that accounts for that issue. (I use this improved version of RAPM when developing BPM).
The other big reason to not just use 3 year RAPM is that it's looking at 3 full years of data--if you want to look at one year, or even smaller periods, then RAPM just won't work.
Back in 2011 I wrote up a blog post explaining APM and RAPM, though RAPM was still very new at the time: http://godismyjudgeok.com/DStats/2011/n ... ilization/
As for the second question--That would work, but the data set becomes much, much larger and harder to manage.
Re: Newbie question about why RAPM is used as the target for so many metrics
Thank you for the response! I previously came across and read your blog post already 
I had a few followup questions:
"That said, if you're trying to build a metric, the RAPM noise/random errors for individual players will generally balance out--so it is an excellent target to use as a regression basis."
>> I'm trying to wrap my head around this but I find this justification a little surprising. Essentially RAPM is a noise reduction preprocessing step and then we rely on a second model to fit the data imperfectly and hopefully balance out the noise?
"The other big reason to not just use 3 year RAPM is that it's looking at 3 full years of data--if you want to look at one year, or even smaller periods, then RAPM just won't work."
>> I think I have some knowledge gap about how these models are trained. Let's say we train a model using box stats and other things to predict RAPM: is this done in a standard time-series machine learning way e.g. take all the players stats (and other things) up until game x to predict the RAPM of the player for game x + 1 where RAPM is calculated over the past 3 years? In that case wouldn't our new model still be predicting the 3 year RAPM? It would be a win in not needing 3 years of data but not necessarily better than 3 year RAPM if we have 3 years of data?
Thanks for explaining everything! Btw do you know if there is a standard RAPM database or github/tutorial to calculate RAPM with standard adjustments (like luck-based FT% or removal of garbage time stats or playoff/regular season weighting)?
I had a few followup questions:
"That said, if you're trying to build a metric, the RAPM noise/random errors for individual players will generally balance out--so it is an excellent target to use as a regression basis."
>> I'm trying to wrap my head around this but I find this justification a little surprising. Essentially RAPM is a noise reduction preprocessing step and then we rely on a second model to fit the data imperfectly and hopefully balance out the noise?
"The other big reason to not just use 3 year RAPM is that it's looking at 3 full years of data--if you want to look at one year, or even smaller periods, then RAPM just won't work."
>> I think I have some knowledge gap about how these models are trained. Let's say we train a model using box stats and other things to predict RAPM: is this done in a standard time-series machine learning way e.g. take all the players stats (and other things) up until game x to predict the RAPM of the player for game x + 1 where RAPM is calculated over the past 3 years? In that case wouldn't our new model still be predicting the 3 year RAPM? It would be a win in not needing 3 years of data but not necessarily better than 3 year RAPM if we have 3 years of data?
Thanks for explaining everything! Btw do you know if there is a standard RAPM database or github/tutorial to calculate RAPM with standard adjustments (like luck-based FT% or removal of garbage time stats or playoff/regular season weighting)?
Re: Newbie question about why RAPM is used as the target for so many metrics
The key here is that you're not looking at the RAPM for any individual player but the whole group of players based on their box-score stats to value each of those box-score components. So even if Player A's contributions might still be overvalued and Player B's undervalued, across them and many other players the average of their stats is unlikely to benefit or be hurt by that error.I'm trying to wrap my head around this but I find this justification a little surprising. Essentially RAPM is a noise reduction preprocessing step and then we rely on a second model to fit the data imperfectly and hopefully balance out the noise?
Re: Newbie question about why RAPM is used as the target for so many metrics
Got it. Is this because we are giving universal weights to each box score component and since RAPM is unbiased and BPM is underparameterized this holds?I'm trying to wrap my head around this but I find this justification a little surprising. Essentially RAPM is a noise reduction preprocessing step and then we rely on a second model to fit the data imperfectly and hopefully balance out the noise?
The key here is that you're not looking at the RAPM for any individual player but the whole group of players based on their box-score stats to value each of those box-score components. So even if Player A's contributions might still be overvalued and Player B's undervalued, across them and many other players the average of their stats is unlikely to benefit or be hurt by that error.
Re: Newbie question about why RAPM is used as the target for so many metrics
You could get everyone's last 3 years' RAPM, and a reliable boxscore metric for those years, and assign a difference between them, to each player. Some (per 36 min.) 17 and 10 guys are seriously "overrated" by the boxscore, and some 12 and 6 players are undervalued.
If the 17-10 guy is measured 5 points less by RAPM; and then he drops to 15-8; he might be near unemployable.
A player's "true" value is not necessarily closer to his RAPM or to his BPM; mostly it will be somewhere in between. But the parameter (RAPM-BPM) may be more stable than either one alone.
Playoff performance, relative to regular seasons, is kind of a built-in "clutch" measure. This does seem to recur for certain players, for better (Horry) or worse (Cliff). It also follows an aging trajectory, peaking a couple years later than RS.
I keep hearing how Aaron Gordon really should be doing more on offense. But I've never covered Kevin Durant for 35 minutes per game, so I don't know.
If the 17-10 guy is measured 5 points less by RAPM; and then he drops to 15-8; he might be near unemployable.
A player's "true" value is not necessarily closer to his RAPM or to his BPM; mostly it will be somewhere in between. But the parameter (RAPM-BPM) may be more stable than either one alone.
Playoff performance, relative to regular seasons, is kind of a built-in "clutch" measure. This does seem to recur for certain players, for better (Horry) or worse (Cliff). It also follows an aging trajectory, peaking a couple years later than RS.
I keep hearing how Aaron Gordon really should be doing more on offense. But I've never covered Kevin Durant for 35 minutes per game, so I don't know.
Re: Newbie question about why RAPM is used as the target for so many metrics
I see. In this case does last 3 year's RAPM mean this point in time - 3 years till now? When training, technically you could use the future to calculate RAPM, e.g. 1.5 years ago - 1.5 years in the future. Would this be more accurate as a target to regress against?You could get everyone's last 3 years' RAPM, and a reliable boxscore metric for those years, and assign a difference between them, to each player. Some (per 36 min.) 17 and 10 guys are seriously "overrated" by the boxscore, and some 12 and 6 players are undervalued.
Re: Newbie question about why RAPM is used as the target for so many metrics
That's an option, but AFAIK you can either combine (sum) everyone's (of those in the lineup) BoxScore stats and have each stat once as a variable
Or you can do a positionally dependent model where each stat then appears 5x
Neither of these seem like a great solution