Flaws with RAPM

[permaximum]

I'm no expert but the biggest flaw imo is trying to reduce noise in every possible way to increase prediction power of the algorithm. I'm even OK with pure multi-year RAPM (simply bigger data - as long as each season's weights are the same) but I hate priors and box-score data, height, age etc. Plus-minus is the result of players' all actions and skills. However box-score stats are some of those things that lead to the result. It sounds completely off and I can't take better prediction rate as a good argument. And I think those priors, box-score, different weights etc. introduce different kind of bias (different than regularization) into the algorithm which we shouldn't want.

If I lack a huge pile of information and sound like a fool pardon me and please enlighten me on the subject. BTW if better prediction is everything try an another variable. Black/white. Then RAPM will start to like blacks too just like it likes tall people on the defense.

[xkonk]

colts18 identifies free throw shooting for special mention, but there is nothing really special about this category of scoring attempts. Random variation of outcomes of two and three point shooting attempts are conceptually identical and should similarly be taken into account. And indeed the contribution of these two categories to per possession scoring variance is very much larger. (The only thing special about free throw shooting is that there is zero defensive input in the instance, what might make adjustments for this effect more straightforward.)

I'll put in a disclaimer about perhaps misunderstanding, but if you mean that players should get a bit of a pass for all shots, and not just free throws, I think that's wrong. Colts suggested adjusting for opponent free throws because a defender has no control over if they go in or not (although I guess it's important to point out they should still be dinged for giving up the attempts in the first place). Repeating my example from before, if you're the guy who happens to be on the court when Dwight Howard hits 7 of 8, that isn't on you. On the other hand, players presumably do have control over twos and threes going in via their defense - forcing a shot from further away, getting a hand up, changing the trajectory, etc. If you're on the court when Dwight goes 7 of 8 from the field, as opposed to the line, that presumably is in some part your fault.

I suppose I could see an argument that even those shots should be adjusted to the shooter's typical percentage, but to do it right you would have to have shot data at such a fine level (from a certain place on the floor, with a defender nearby or not, with a hand up or not, so on and so forth) that even if you had the camera data to do it, the sample sizes would be so tiny as to make the adjustment unreliable.

[J.E.]

I suppose I could see an argument that even those shots should be adjusted to the shooter's typical percentage, but to do it right you would have to have shot data at such a fine level (from a certain place on the floor, with a defender nearby or not, with a hand up or not, so on and so forth) that even if you had the camera data to do it, the sample sizes would be so tiny as to make the adjustment unreliable.

I think one could up with a decent model that can predict FG% for certain spots on the floor for each player relatively well, without sample size being too big of an issue.
I feel uncomfortable doing this 'adjustment' for shots that can potentially be rebounded though. Say someone misses a dunk, which had a 90% chance of going in, gets his own rebound and dunks it. Did he 'score' 0.9*2*2 = 3.6 points in the possession?

[xkonk]

Well, he missed the first dunk, so there has to be a minus in there somewhere. But his expected points added have that 3.6 in there in some fashion, I suppose.

More to the point of the defensive adjustment, I think it's an issue of how fine-grained you want to be in setting the expected value. As mentioned, it's pretty straightforward with free throws. But say we want to evaluate Tony Allen's defense on Durant, and we're looking at a shot where Durant takes a long two from the top of the key. Do we just take Durant's average % from the top of the key and Allen is penalized the difference? If Allen was right on him, we should limit to Durant shots from the top of the key where the defender was close; otherwise the baseline was too high (it includes open shots). If Allen has his hand up (or not), we should include that. Do we limit it further to Durant shots where he was guarded by a guard as opposed to a small forward? Do we limit to guys around Allen's height of 6'4", or his wingspan, since that presumably affects how much the shot is affected? Do we limit to time left on the shot clock, or in the game, or if the play was a spot-up or an iso, or assisted or not? Durant takes a lot of shots, so maybe even with all these filters he has a decent number of shots for creating a baseline/expected FG%. But I would be surprised if that's true for lots of players. In exchange you could probably create an average by going across seasons or combining players who are similar in some sense, but then I think you could be generalizing too much and/or losing season-specific changes.

[schtevie]

It will remain an enduring mystery to me how the high priests of (R)APM specifically set priors and make adjustments. But now I am also confused by the general approach suggested for dealing with the issue in question.

If the problem is that ratings are being adversely impacted by the predictable effects of randomness on scoring attempts, why not just directly account for this in total, after the fact, rather than trying to make adjustments on each individual scoring attempts (if I am understanding the suggestion)?

Taking FTAs as the simplest case, for each player, the "unexpected" number of free throws scored (on both defense and offense) can be estimated, ex post, based on realized season-averages for FT%. The player's proportionate share of this differential could then be directly added to his per possession +/- rating. Why would such an approach be problematic? Yes, the correction wouldn't be "correct" in that realized season-averages wouldn't represent the true free throw shooting abilities, but at least we should expect the error of this average to be very small.

Trickier would be dealing with FGAs, albeit not conceptually. You would have to come up with expected points per shot by zones, reflecting the underlying line-ups, then carry out the same exercise. Indeed, isn't http://stats-for-the-nba.appspot.com/ra ... ooter.html such a step in that direction?

[ryannow]

Related: what's the explanation for Nick Collison's gigantic yearly RAPM? I've heard that there was some weird interaction one season, and the error has carried over (maybe.)

This is a bit of a digression from other conversations going on, but I wanted to address it. The short explanation is that Collison is an excellent defender and a great screener, on a team with a lot of scorers who really benefit from having a good screener around. Per Basketball-Reference, here's the raw +/- per 100 possessions on and off court for Collison since 09-10:

Code: Select all

Year   On    Off    Diff
09-10   +9.9  +0.6   +9.3
10-11  +11.0  +0.2  +10.8
11-12  +11.4  +3.7   +7.7
12-13  +12.0  +8.2   +3.8
13-14  +11.9  +4.2   +7.7

This shows that it definitely doesn't look like a one-year thing is doing it. He's also got positive values all five years before then, although the differences were never quite as big, and he's even got highly positive +/- values every year in the playoffs in those small samples, for what it's worth.

Since these are raw numbers, perhaps you could credit collinearity to a degree. For example, in 11-12, 1154 of Collison's 1307 minutes were with Harden on the floor, and you could say Harden deserves a good portion of the credit for the difference there. OKC never did as well with Harden without Collison than they did with the pair of them together, though, so Collison was clearly a pretty important part of it, and considering his skillset, that's not a huge surprise. Harden benefits a lot from having a great screen to get going, and he's never been quite as efficient as he was in that ridiculous .660 TS% year.

Harden's not the only guy with this difference, though, nor is 11-12 the only year. Checking NBAWowy for this past year, I see that Durant had 1.18 points per possession on 31.8 USG with Collison off the floor, and Westbrook was at .99 PPP on 34.2 USG. With Collison on the floor, those numbers went up to 1.30 PPP on 36.8 USG (!!!!!!) for Durant, and 1.12 PPP on 34.8 USG for Westbrook. Look at the year before, and once again, Durant, Westbrook, and Martin all show increases in their scoring efficiency with Collison on the floor. This change seems to be quite consistent across multiple years and for multiple players, and I think it would be a mistake to dismiss it as a fluke.

[J.E.]

With Collison on the floor, those numbers went up to 1.30 PPP on 36.8 USG

Haha. That's pretty sick


A couple of thoughts on the 'adjusting for FT/FG%':

- This should work for 'Adjusted eFG' (PPS) without having the 'offensive rebounding'-problem (see below). The question then becomes whether to include things like 'defender distance' in your model for expected FG%. If you do include it you'll need to somehow credit defenders that manage to keep a close distance, and you'd need to credit offensive players that managed to create seperation. If you don't include defender distance in your model, players with good opponent FG% will probably look worse than they should. Although I suppose one could take some sort of linear combination of exp. points and actual points to migitate that effect. Then the next problem arises though: Some defenders are good at allowing shots from mostly low % zones, and not allowing shots in high % zones. We'd need to account for that, too

- My concern for doing this with standard RAPM with reboundable shots was that good offensive rebounders will create lots of additional shots for their team. If we just plug in exp. points for all of these shots then the rating for those offensive rebounders will go through the roof. See the example from an earlier post

Say someone misses a dunk, which had a 90% chance of going in, gets his own rebound and dunks it. Did he 'score' 0.9*2*2 = 3.6 points in the possession?

I now think a possibly simple fix for this issue could be to multiply the exp. points of the second shot with the chance of the first shot not going in. So, instead of 0.9*2*2 = 3.6 exp. points scored, it would be 0.9*2+0.1*0.9*2 = 1.98. At first glance this makes me feel a lot better since the exp. points don't exceed '2'

[colts18]

Adjusting for FT% is huge. Among players with 1,000+ MP this season, their opponents ranged in FT% from 70.8% to 79.8%. That's a 9% range which the player has no control over.

My intuition (which is correct about 50% of the time) leads me to think that this still won't amount to much of a material change in RAPM values for almost all players, especially in models that are prior informed (via box, or prior rapms, or whatever). That said, I'm happy to take on the role of crash test dummy here, so we'll test the "Colts Conjecture" after the playoffs here are our end and shall report back.

My guess is that adjusting for FT% on offense and defense might be worth at least 1-2 points in RAPM. That could be a huge difference and move some guys up like 30 spots in the rankings. For example, if you compare the guy with the worst luck (Mike Miller) to the best luck (Kendall Marshall), Miller would have gained 2.4 points per 100 possessions if his opponents shot FT's like Kendall Marshall's opponents. That's a massive difference.

[J.E.]

My guess is that adjusting for FT% on offense and defense might be worth at least 1-2 points in RAPM. That could be a huge difference and move some guys up like 30 spots in the rankings. For example, if you compare the guy with the worst luck (Mike Miller) to the best luck (Kendall Marshall), Miller would have gained 2.4 points per 100 possessions if his opponents shot FT's like Kendall Marshall's opponents. That's a massive difference.

Miller would share the gained points with 4 teammates though, so we're down to .5. Add the fact that we really aren't supposed to be doing this for 'X of X' free throws (without further adjustments) because those can be rebounded, but only for 'Y of X' where Y!=X, so we're probably down to .25
.. and that's completely ignoring that you just looked at opponent FT% without looking at who actually shot these free throws. Maybe Marshall is often on the court when the opponent has bad free throw shooters on the court, or he's good at fouling the not-so-great free throw shooters

[permaximum]

With the recent advancements in machine learning, deep learning, artificial general intelligence and rise of new algorithms which use totally different aprroaches, I'm curious if "ridge regression" is still the best way to solve the problems with APM. Perhaps the R in RAPM is the biggest flaw by today's standards. What do you think?

[J.E.]

With the recent advancements in machine learning, deep learning, artificial general intelligence and rise of new algorithms which use totally different aprroaches, I'm curious if "ridge regression" is still the best way to solve the problems with APM. Perhaps the R in RAPM is the biggest flaw by today's standards. What do you think?

If there's going to be a non-regression version of a player metric I doubt that it will have 'APM' in its name.

I have actually tried a bunch of new-ish regression techniques to see whether they might perform better than Ridge Regression:
- LASSO: is worse
- ElasticNet: is worse
- Stochastic Gradient Descent: the same

Further, xRAPM actually uses 'modified Ridge Regression', and Ridge Regression has, in theory, another aspect I/we can potentially mess with to maybe further improve prediction accuracy - it's just a matter of finding the time

To further improve player metrics I think we'd get the most gain by further improving SPM, especially on the 'defense' side, and using SportVU data to do so

[TheSpiceWeasel]

4) I really wish intentional fouls were kept officially because we'd have to guess when they were happening. But you can always weigh possessions less when the lead is too big.

They are tracked. Look at 1:38 remaining in the 4th period of this game: http://www.nba.com/games/20140519/OKCSA ... #nbaGIPlay. They are referred to as "Personal Take" fouls. You can see the accompanying video here: http://stats.nba.com/cvp.html?GameID=00 ... ventID=482.

[TheSpiceWeasel]

So, just to be clear, we're talking here about the effect of random variation in free throw shooting outcomes biasing (R)APM estimates, both offensive and defensive, right?

I must say that my initial reaction was that this should be a rather small effect, but upon further consideration, I think I'm of a belief that the combination of this and related effects could be quite large, indeed.

colts18 identifies free throw shooting for special mention, but there is nothing really special about this category of scoring attempts. Random variation of outcomes of two and three point shooting attempts are conceptually identical and should similarly be taken into account. And indeed the contribution of these two categories to per possession scoring variance is very much larger. (The only thing special about free throw shooting is that there is zero defensive input in the instance, what might make adjustments for this effect more straightforward.)

Maybe I'm thinking about this incorrectly, but here's the simple thought experiment, breaking scoring attempts into the aforementioned three categories (2PA, 3PA, and FTA), using data for the last season's NBA average* (with a minor asterisk that can be explained), and calculating the resulting standard deviation of points per 100 possessions (assuming no covariation between types of scoring attempt). I then get the following for players playing 70 games and either 1/4, 1/2, 3/4, or the entire game (of which, of course, there are none of the latter): 2.6, 1.9, 1.5, and 1.3 respectively. And these would correspond to offense and defense separately.

Now imposing a prior should eliminate a significant portion of the problem of such variation, as we don't expect players to be serially (un)lucky, but if these numbers are in the ball park, they are large enough to be of concern, no?

Congratulations. You've conceptually re-derived the statistical notion of the "residual sum of squares."

[ampersand5]

The Raptors have had a far better defensive efficiency with Demar Derozan out. Many Raptor fans argued that the variance is influenced by the amount of fouls Demar draws, thus putting the other team into foul trouble and impacting their play.

I'm not sure how valid it is, but I think its interesting because it would allow a player to have an impact on the +/- of a game even when hes off the court.

Also, I'll just throw it in here - points are not equally valuable. A point scored with one minute remaining is far more valuable than a point scored in the first minute of the game. The time remaining in the game can serve as a factor in XRAPM (I vaguely recall Wayne Winston using something similar to this).

[Crow]

Jerry, do you still discount pure garbage time, but not give a premium for clutch time or is it now both or neither?