Flaws with RAPM

[J.E.]

Jerry, do you still discount pure garbage time, but not give a premium for clutch time or is it now both or neither?

Everything gets the same "weight", but you're required to score more when behind - due to the 'effect of being up/behind' - to break even, and vice versa

[Mike90]

I have a hunch that the surprisingly high plus-minus figures for some low-minute guys are caused mostly by selection bias, as in a coach purposely plays his specialists only in situations where he thinks they would do well and sits them otherwise. Example: Jason Collins is considered a great interior defender, so he's used more against star big men and doesn't play when he's not needed.

It seems like the old usage-efficiency debate. One way to test it would be to identify high RAPM, low minute players and check how well their teams do when the players are forced to play more minutes due to injuries or suspensions.

[Crow]

Yeah. Probably.

But the critics would still find things to pick at.

It isn't perfect but it is pretty good or better most of the time.
Probably.

[Mike90]

Good, metrics should be criticized. Understanding where a metric fails is necessary for correcting it. Long-term APM is a fantastic metric, particularly for measuring defensive impact, but we should still examine the outliers and try to understand why guys like Amir Johnson, Nick Collison, and Jason Collins have this huge plus-minus impact despite never being heavy minutes players.

I think my test should work. Not really sure how to go about getting the data and what not, but we can get closer to judging the true impact for high APM, low minute guys.

[Crow]

I am fine with fair criticism and criticism aimed at prompting improvement.

[AcrossTheCourt]

That low minute, opponent selection stuff is mostly BS. This isn't baseball where one reliever is only brought out for right-handers. Jason Collins was a starter for a long time and often played high minutes. The one year he played 32 MPG he was fourth in RAPM -- don't blame low minutes on that. Amir Johnson is a starter too and they don't select his minutes for only opponents where he can be successful. He's not a matchup post defender like Collins.

By the way, due to the regularization, RAPM hurts low minute guys more than other metrics.

[Mike90]

Nick Collison's RAPM numbers are more interesting to me than Collins. In the 14-year RAPM data set, Collison has a +3.9, which ranks him 37th out of the 1383 players listed. He's in the top 3% - truly elite.

Below are his games started, minutes per game with team rank in parentheses, and team's record. I would include games played too, but Nick's been pretty durable in his career and has never missed a lot of games. It has an unusual pattern imo:

04-05: 4, 17.0 (9th), 52-30
05-06: 27, 21.9 (8th), 35-47
06-07: 56, 29.0 (5th), 31-51
07-08: 35, 28.5 (3rd), 20-62
08-09: 40, 25.8 (7th), 23-59
09-10: 5, 20.8 (7th), 50-32
10-11: 2, 21.5 (9th), 55-27
11-12: 0, 20.7 (7th), 47-19
12-13: 2, 19.5 (7th), 60-22
13-14: 0, 16.7 (10th), 59-23

Collison never became a full-time starter and never played 30 mpg. The year where he was 3rd on the team in mpg, he was still behind a rookie Kevin Durant and Earl Watson. When his teams were actually good, he was somewhere between the 2nd and 4th guy off the bench.

If you just looked at the above minute’s distribution, you would assume Collison is an average player.

Keep in mind that I am not doubting the RAPM number itself. Look at 82 games.com, and you’ll see that Collison’s had really good plus-minus numbers year after year. I am curious about why he looks like a borderline superstar by plus-minus, but his coaches didn’t play him. I’ve thought of a few possible reasons:

1) Plus-Minus is doing exactly what it’s supposed too – capturing non box score-based contributions.

Collison’s not playing heavy minutes so when he does play, he can play with complete 100% intensity – making hard cuts, setting screens, diving for loose balls, rotating quickly on defense, playing physical post defense. All of these things aren’t captured by the box score, or noticed by the general public, but they clearly help teams win games. Collison is considered an intelligent, unselfish player, so maybe can make better use of the increased energy than other players.

2) He gets tired quickly.

Big guys probably don’t have the same stamina that little guys do. Plus, Collison is constantly guarding bigger guys in the post, which I’m sure can be tiring.*

3) His coaches are simply underrating the value he brings to the table.

This is absolutely a possibility. I discount it a bit though. Take a look at that 08-09 season. Collison plays 26 minutes a game for a team that goes 23-59. The next year, his minutes dip to 21 a game, and the team wins 50. I’m not sure what to make of it. It’s hard to believe that OKC is leaving that many wins on the table. Not impossible but unlikely.

4) Selection bias

Nick has pretty obvious strengths and weaknesses. He’s a smart player. He works hard. He’s the type of player that coaches tend to appreciate. He’s also a tweener – too small to play center but too slow to go against many of the elite power forwards. It makes sense that coaches might play him off the bench where he would face generally weaker opponents. It also makes sense that his coaches would play Nick less in games where’s facing a really quick or really big frontcourt player.

* This might explain why Jason Collins didn’t play many minutes per game, even at his peak. AcrossTheCourt mentioned the season when the season with 32 mpg, but that was the only year he averaged over 30, and even then he was just 5th on the team in mpg.

[Statman]

Nick Collison's RAPM numbers are more interesting to me than Collins. In the 14-year RAPM data set, Collison has a +3.9, which ranks him 37th out of the 1383 players listed. He's in the top 3% - truly elite.

Below are his games started, minutes per game with team rank in parentheses, and team's record. I would include games played too, but Nick's been pretty durable in his career and has never missed a lot of games. It has an unusual pattern imo:

04-05: 4, 17.0 (9th), 52-30
05-06: 27, 21.9 (8th), 35-47
06-07: 56, 29.0 (5th), 31-51
07-08: 35, 28.5 (3rd), 20-62
08-09: 40, 25.8 (7th), 23-59
09-10: 5, 20.8 (7th), 50-32
10-11: 2, 21.5 (9th), 55-27
11-12: 0, 20.7 (7th), 47-19
12-13: 2, 19.5 (7th), 60-22
13-14: 0, 16.7 (10th), 59-23

Collison never became a full-time starter and never played 30 mpg. The year where he was 3rd on the team in mpg, he was still behind a rookie Kevin Durant and Earl Watson. When his teams were actually good, he was somewhere between the 2nd and 4th guy off the bench.

If you just looked at the above minute’s distribution, you would assume Collison is an average player.

Keep in mind that I am not doubting the RAPM number itself. Look at 82 games.com, and you’ll see that Collison’s had really good plus-minus numbers year after year. I am curious about why he looks like a borderline superstar by plus-minus, but his coaches didn’t play him. I’ve thought of a few possible reasons:

1) Plus-Minus is doing exactly what it’s supposed too – capturing non box score-based contributions.

Collison’s not playing heavy minutes so when he does play, he can play with complete 100% intensity – making hard cuts, setting screens, diving for loose balls, rotating quickly on defense, playing physical post defense. All of these things aren’t captured by the box score, or noticed by the general public, but they clearly help teams win games. Collison is considered an intelligent, unselfish player, so maybe can make better use of the increased energy than other players.

2) He gets tired quickly.

Big guys probably don’t have the same stamina that little guys do. Plus, Collison is constantly guarding bigger guys in the post, which I’m sure can be tiring.*

3) His coaches are simply underrating the value he brings to the table.

This is absolutely a possibility. I discount it a bit though. Take a look at that 08-09 season. Collison plays 26 minutes a game for a team that goes 23-59. The next year, his minutes dip to 21 a game, and the team wins 50. I’m not sure what to make of it. It’s hard to believe that OKC is leaving that many wins on the table. Not impossible but unlikely.

4) Selection bias

Nick has pretty obvious strengths and weaknesses. He’s a smart player. He works hard. He’s the type of player that coaches tend to appreciate. He’s also a tweener – too small to play center but too slow to go against many of the elite power forwards. It makes sense that coaches might play him off the bench where he would face generally weaker opponents. It also makes sense that his coaches would play Nick less in games where’s facing a really quick or really big frontcourt player.

* This might explain why Jason Collins didn’t play many minutes per game, even at his peak. AcrossTheCourt mentioned the season when the season with 32 mpg, but that was the only year he averaged over 30, and even then he was just 5th on the team in mpg.

Another reason he doesn't play more minutes is his foul rate.

[Mike90]

You're right - 4.8 fouls per 36 minutes is pretty high. Maybe that's the only reason.

[Crow]

It is a reason, but having watched his entire career, some set of your reason list also applies.

[Mike G]

Statman (and everyone else) -- do you suppose you could refrain from reproducing the entire preceding post? It just makes more scrolling for everyone else reading after you.

A player who picks up a foul every few minutes doesn't just get a quick return to the bench; it also may reveal something about how he can be as effective as he is. When you're overly aggressive, you get some rebounds and steals and picks that you wouldn't get if you intended to stay on the floor for 30 to 40 minutes. So yeah, it's kind of a 'specialist' situation when a Collison comes in and wreaks some mayhem for his 15 minutes.

Not just the fouls put him back on the bench. In 17 mpg, he averages 2.6 PF, and he hasn't fouled out since Feb of '13.
The opponent also adjusts, neutralizing him somewhat. It's like he's got a bag of tricks, but he can't make up new ones as fast as the league figures them out.

He's not a great rebounder, but his OReb% is .60 of his DReb%. Only 10 players have done this for as many minutes, with TRb% >14.
http://bkref.com/tiny/t5BWa

[colts18]

I was watching a game recently and saw a bench player come in the game late in the quarter. He gets fouled then heads to the free throw line for 2. That play made me realize another flaw of RAPM. That bench player is getting credit from him teammates drawing fouls and getting the team into penalty. Bench player comes in and gets 2 Free Throw shots despite the fact that he had nothing to do with the 5 earlier fouls drawn that got his team in the bonus.

Things like will artificially boost the offense of some players and depress the Offensive RAPM's of the players who don't get the benefit of playing in the penalty. The same thing can happen on the defensive side of the ball where the bench player gets a bad RAPM because his teammates fouled a lot.

[rainmantrail]

I just read through this thread for the first time as I'm new here, and this thread started several years ago. The majority of concerns raised here (adjusting for FT%, FG%, "luck", which minutes someone gets to play in, etc.) would all amount to very minor differences in the overall RAPM values, if any. And only for a small number of players. You're much more likely to introduce unwanted bias against strong defenders if you're not extremely careful with how you adjust for opponent's FG% in each situation. As J.E. points out, you'd really have to know everything about that shot and how it was defended, and even the height and probably even wingspan of the nearest defender. Perhaps even a quickness/speed factor of the defender. All of these things matter tremendously when contesting a shot, and if you're not accounting for how these factors affect the probability of a shot going in, then you're making a mistake by adjusting for "luck" when it comes to opponent's FG% on shots. FTs are tricky, but there's obviously a much better argument for accounting for those, but again, as J.E. points out, even in the most extreme differences, we're probably talking about less than a 0.2 +/- effect on the final result, and the vast majority of players will see no change at all. Sample sizes of FT attempts are simply too large for anyone playing sufficient minutes. It shouldn't be a big issue.

My main concern and one which I'm surprised no one here has raised, has to do with the statistical fundamentals of how a regularized regression model works mathematically to begin with. This is by far the biggest flaw with RAPM in my opinion. A regularized regression model of this sort assumes that the player values follow a Gaussian distribution (think "bell curve"). It then regresses the player values toward zero, yielding an output distribution that resembles the Gaussian bell shaped curve but with an inflation around 0, caused by the regularization. To better understand what I'm talking about, here is a plot from my database of RAPM values across 7 seasons worth of data. Notice the shape of the distribution is approximately normal (or at least symmetric) but with an inflation near the mean of 0.




This 0 inflation is necessary to account for the multicollinearity that occurs in the PBP data. This bias is introduced as a way of reducing the variance we would otherwise have, and it is a very helpful approach. It is also what gives RAPM its strength and advantage over something like APM without the regularization component. But the fundamental flaw here with RAPM that I'm addressing doesn't have to do with the 0 inflation component. The fundamental flaw here is the assumption that the model has to make regarding player values being Gaussian distributed. Without question, this assumption is violated in the real world. NBA player values do not follow a Gaussian distribution. They would look like the right tail of this distribution. Their true distrubution should look something like the shaded purple region of this plot below.




This is very problematic for RAPM player values. If we want a comprehensive player evaluation metric, then the distribution of that metric must mirror the true underlying distribution of player values itself. Currently, we have an RAPM metric that says player values are approximately Gaussian, but we know otherwise. They are not Gaussian. They are heavily skewed and right-tailed.

That doesn't mean that the RAPM values are useless however. The Central Limit Theorem is helpful here. We could use the RAPM outputs as a target variable to create a more accurate model that yields the distribution we want. I haven't built any BPM metrics yet myself to test this, but I assume the shape of the output distribution from such a metric would actually follow the right-tailed shape that we would ultimately want the RAPM values to have. The trick is getting the BPM model to be able to capture enough of the factors that actually impact a player's on-court performance, namely the stuff that doesn't show up in box scores like boxing out, contesting shots, creating space, etc. Being able to incorporate this into the model is key for improving it. A skilled data scientist should be able to yield a metric that outperforms RAPM and BPM by incorporating player tracking data to account for as much of the non-box-score impacts as possible. This is on my to-do list, but I'm pretty late to the game here, as I've only recently starting working on my NBA predictive models, and most of this community is years ahead of me.

[DSMok1]

I 100% agree with your criticism, and that is my biggest concern as well.

My somewhat crude attempt to solve it was to introduce a prior based on MPG and team efficiency that generally follows that shape (i.e. many low minutes players have a prior around -3, a few high minutes players on good teams have a prior around +3). However, that was just applied as a pre-processing step and the actual penalty/regression approach uses a normal L2 loss function. If there is an elegant approach, I would love to see it.

Here was Tangotiger's discussion on the talent distribution in Baseball, from around 2000: http://tangotiger.net/talent.html

[J.E.]

I don't think it's as big of an issue as it's made it to be

You're taking an (originally tail-end-of-population) subset of people, but then you're providing them with resources that aren't available to the outside world, but are the same for all ~600 players in this "bubble".
Resources such as coaches, physical therapists, nutritional info, (money)

I think the end result is a mix of tail-end and gaussian

Distributions of various simple BoxScore stats will tell the same story. None of the histograms look like a pure tail-end distribution

Either way, BoxScore priors are a simple way to reasonably deal with it