Possible Steps to improve SPM
Posted: Fri Oct 10, 2014 3:29 pm
In the last couple of days/weeks I've been trying to "improve" my SPM (Statistical Plus Minus). As most of you probably know, SPM serves as a prior ("starting point") for xRAPM.
I would like to improve my SPM's accuracy since that will, in turn, lead to better xRAPM accuracy. No SPM will ever be perfect and, to some degree, the RAPM-part of xRAPM will be able to rectify some of the most egregious errors of SPM, but I think there's more room for improvement in the SPM-world than there's in the RAPM-world, so that's what I'm working on.
Why are inaccurate SPM ratings a problem for xRAPM? Imagine the (offensive) SPM for two random players A and B (that play the same position one one team) to be +0 (A) and +4 (B), when in reality A is actually +4, and B is 0 - this case can easily occur with two players where one is more of a "stat stuffer" who fails to do all the little things (think Ricky Davis), and the other is doing all those things not recorded in the BoxScore like setting good screens etc. (think Battier, Nick Collison etc.).
To get xRAPM, we feed those "somewhat off" SPM values into RAPM. Even if we're lucky and lineups of player A perform 4 points better than lineups with player B (they might not because of noise), it will take some time and data before A's rating approaches 4 and B's rating approaches 0. With just half a season, both players might be at +2. Due to the SPM's being off, players that primarily play with B will be expected to have a higher +/- than players that play with A. Since they probably won't have a higher +/- with B (because A is the better player) their xRAPM rating will (incorrectly) suffer. As you can see, SPM values that are off not only give a wrong impression of the player in question, but maybe also for teammates (and to a small degree, opponents)
To test my own SPM for glaring weaknesses I came back to an idea I had a couple of years back:
- For every year in the dataset compute SPM ratings
- Use SPM ratings from year X-1 and lineup data from year X to compute "expected team points scored while on court" and "expected opponent points scored while on the court" for every player
- compute the difference between "expected points" <-> "actual points" for every player
- sum up the differences over the entire sample to identify players that are regularly over- or underrated by SPM. You could instead look at things on a year to year basis, but for now I'll work with the totals
Here's the list of most under- and overrated players, for offense, since 2002 (Please note that you could create such a list of "most blatant errors" for every NBA rating system that humankind will create.)This is basically saying that, since 2002, Dirk Nowitzki's lineups have scored 1000 points more than SPM expected them to score. As such, Nowitzki is responsible for a lot of SPM's OOS error. Our goal should be, to some degree at least, to find ways so that SPM can better seperate the Nowitzki's from the Vince Carters.
Here's the pitfall: We probably shouldn't do so at all cost. If we start adding terms to SPM that might not actually make sense, from a basketball standpoint, we might reduce in sample error, but probably not out of sample error. But obviously if we find a glaring weakness of (any kind of) SPM we should be trying to attack and improve it.
My first reaction after seeing Nowitzki and Nash at the top was to add numbers from a shot metric like EvanZ had created where a player's shooting efficiency from specific spots on the court gets compared to the average efficiency in those spots. Nowitzki and Nash generally scored pretty well in that metric. I don't actually have shot location data, so my next thought was to use 3P%*2P%*FT% (Nash is part of the 50/40/90 club and Nowitzki came close), or possibly 3P%*2P%*FT%*(FGA+FTA*0.5). The list above was actually generated after adding these terms. (Nowitzki's error was even higher before adding it) Nowitzki, unfortunately, is not easy to seperate using BoxScore stats. He doesn't excel at rebounding/assists/blocks/steals.
Another problem is that Kobe Bryant and Vince Carter are at completely opposite sides of the spectrum even though their statistical profiles are similar
So, if anyone has any ideas about possible interaction terms or stats that might be extractable from PlayByPlay, which could help lower the SPM rating of the overrated and raise the rating of the underrated, let me know. It can't be '+/-' though, as that's something that the RAPM-part of xRAPM will take care of, anyway. The plan is to add useful terms to SPM, recompute the list of "most under/overrated" and then look for new terms. It's a process that can go on forever, and at some point should be using other sources of data
I would like to improve my SPM's accuracy since that will, in turn, lead to better xRAPM accuracy. No SPM will ever be perfect and, to some degree, the RAPM-part of xRAPM will be able to rectify some of the most egregious errors of SPM, but I think there's more room for improvement in the SPM-world than there's in the RAPM-world, so that's what I'm working on.
Why are inaccurate SPM ratings a problem for xRAPM? Imagine the (offensive) SPM for two random players A and B (that play the same position one one team) to be +0 (A) and +4 (B), when in reality A is actually +4, and B is 0 - this case can easily occur with two players where one is more of a "stat stuffer" who fails to do all the little things (think Ricky Davis), and the other is doing all those things not recorded in the BoxScore like setting good screens etc. (think Battier, Nick Collison etc.).
To get xRAPM, we feed those "somewhat off" SPM values into RAPM. Even if we're lucky and lineups of player A perform 4 points better than lineups with player B (they might not because of noise), it will take some time and data before A's rating approaches 4 and B's rating approaches 0. With just half a season, both players might be at +2. Due to the SPM's being off, players that primarily play with B will be expected to have a higher +/- than players that play with A. Since they probably won't have a higher +/- with B (because A is the better player) their xRAPM rating will (incorrectly) suffer. As you can see, SPM values that are off not only give a wrong impression of the player in question, but maybe also for teammates (and to a small degree, opponents)
To test my own SPM for glaring weaknesses I came back to an idea I had a couple of years back:
- For every year in the dataset compute SPM ratings
- Use SPM ratings from year X-1 and lineup data from year X to compute "expected team points scored while on court" and "expected opponent points scored while on the court" for every player
- compute the difference between "expected points" <-> "actual points" for every player
- sum up the differences over the entire sample to identify players that are regularly over- or underrated by SPM. You could instead look at things on a year to year basis, but for now I'll work with the totals
Here's the list of most under- and overrated players, for offense, since 2002 (Please note that you could create such a list of "most blatant errors" for every NBA rating system that humankind will create.)
Code: Select all
╔══════════════════════╦══════════════════════╗
║ Name ║ Actual-Exp (offense) ║
╠══════════════════════╬══════════════════════╣
║ Dirk Nowitzki ║ 1016 ║
║ Steve Nash ║ 731 ║
║ Kobe Bryant ║ 619 ║
║ Kevin Garnett ║ 483 ║
║ Mike Conley ║ 478 ║
║ Chris Paul ║ 472 ║
║ Hedo Turkoglu ║ 470 ║
║ Zach Randolph ║ 459 ║
║ Eduardo Najera ║ 441 ║
║ Carlos Boozer ║ 394 ║
║ .. ║ ║
║ Antoine Walker ║ -354 ║
║ Jacque Vaughn ║ -368 ║
║ Ricky Davis ║ -379 ║
║ Paul Pierce ║ -405 ║
║ Jason Collins ║ -408 ║
║ Shareef Abdur-Rahim ║ -411 ║
║ Gerald Wallace ║ -412 ║
║ Mark Blount ║ -417 ║
║ Richard Jefferson ║ -458 ║
║ Vince Carter ║ -592 ║
╚══════════════════════╩══════════════════════╝Here's the pitfall: We probably shouldn't do so at all cost. If we start adding terms to SPM that might not actually make sense, from a basketball standpoint, we might reduce in sample error, but probably not out of sample error. But obviously if we find a glaring weakness of (any kind of) SPM we should be trying to attack and improve it.
My first reaction after seeing Nowitzki and Nash at the top was to add numbers from a shot metric like EvanZ had created where a player's shooting efficiency from specific spots on the court gets compared to the average efficiency in those spots. Nowitzki and Nash generally scored pretty well in that metric. I don't actually have shot location data, so my next thought was to use 3P%*2P%*FT% (Nash is part of the 50/40/90 club and Nowitzki came close), or possibly 3P%*2P%*FT%*(FGA+FTA*0.5). The list above was actually generated after adding these terms. (Nowitzki's error was even higher before adding it) Nowitzki, unfortunately, is not easy to seperate using BoxScore stats. He doesn't excel at rebounding/assists/blocks/steals.
Another problem is that Kobe Bryant and Vince Carter are at completely opposite sides of the spectrum even though their statistical profiles are similar
So, if anyone has any ideas about possible interaction terms or stats that might be extractable from PlayByPlay, which could help lower the SPM rating of the overrated and raise the rating of the underrated, let me know. It can't be '+/-' though, as that's something that the RAPM-part of xRAPM will take care of, anyway. The plan is to add useful terms to SPM, recompute the list of "most under/overrated" and then look for new terms. It's a process that can go on forever, and at some point should be using other sources of data