Retrodiction (schtevie, 2009)
Posted: Fri Apr 22, 2011 2:08 am
schtevie
Joined: 18 Apr 2005
Posts: 408
PostPosted: Wed Mar 11, 2009 10:57 am Post subject: retrodiction Reply with quote
Perhaps the forces of entropy are too great, but I like to think that it would be productive to, at least, try to achieve some sort of consensus about the relative merits (and shortcomings) of various methods for assessing NBA player (and team) value.
This point, to me, was underscored by John Hollinger's Cuban-baiting at the Sloan School Conference, regarding the wisdom of the Harris/Kidd trade. Though certainly amusing, it was, in its essence, a simple case of PER said vs. APM (or Cuban's brand thereof) said. Clearly, Kidd's PER of 17.4 this year vs. Harris' PER of 22.7 was the basis of John's smug questioning of the wisdom of the trade. And by contrast, Cuban's somewhat exasperated "you haven't a clue" reply was surely based on a fact similar to that shown by current APMs which show that the Mavs got the upper hand. Considering just the principals (as the contributions of remaining players involved in the trade essentially offset each other, and never mind the implications of the draft picks for now) a minutes-weighted estimate of the 3+ APM differential suggest that the Mavs will get about an extra 6 wins this year from the trade.
Now, one should not expect consensus on matters such as these, but at least there should be explicit recognition of and respect for plausible interpretations.
In a recent string on APM, Dean made the correct point that isolated predictions are necessarily of limited value, and that a possibly useful starting point for assessing varying methods of appraisal would be by retrodiction. As one of the apparent, unaffiliated champions of APM on this board, I did the calculations for this method, retrodicting the 2007-08 season and will post the summary statistics soon. (Additionally, Dave Berri has unwittingly agreed to play along, with his posted retrodictions on his site.)
Before presenting these results, however, I would like it if the journalists and modern historians on this board would take the time to discuss the unexpected. By this I mean, which players (hence, by summation, teams) had outcomes that might be expected to be poorly predicted by any method of player/team evaluation? In particular, I am thinking of two phenomena, though there may be others.
First, the expectation of the life-cycle of a player is that he improves then deteriorates with age. To the extent that a metric doesn't take this into account, it will be expected to make prediction errors. Note that here I am only looking to identify "ex-ante" the players that played significant minutes that really exhibited "unexpected" maturation or degradation.
Second, there are player injuries that are insufficiently significant to take them out of games, but significant enough to affect their productivity, hence leading to apparent errors in retrodiction. In discussing the performance of the Celtics, I surmised that Ray Allen's off-season surgery might have accounted for Ray Allen's "unexpectedly" low APM the following year.
If the collective wisdom could come up with a list of such prominent instances throughout the league (or suggest other such categories of expected errors) prior to the posting of any results, it would set the stage for an interesting discussion of results.
And in the interim, may I encourage everyone who has a preferred method for evaluation, to post a retrodiction of the 2007-2008 season, explicitly on the basis of such a method (which is to say, with no modification of the results based upon personal interpretation). And Hopefully Neil from basketball-reference will chime in with a statistical APM retrodiction.
Back to top
View user's profile Send private message
Ryan J. Parker
Joined: 23 Mar 2007
Posts: 708
Location: Raleigh, NC
PostPosted: Wed Mar 11, 2009 11:18 am Post subject: Reply with quote
The key point I took from the debate had to do with the business aspect. What players are going to be available in the future (based on each team's salary cap figures) that would make us want to execute specific deals?
I think this is the point Cuban was trying to make. Maybe I'm in the minority, but I've never thought about this sort of thing. Seems to me it would be valuable to know which players are more likely to move in future years based on this kinda stuff.
Certainly there are non-zero probabilities associated with these kinds of things that we can base decisions on.
_________________
I am a basketball geek.
Back to top
View user's profile Send private message Visit poster's website
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Wed Mar 11, 2009 12:19 pm Post subject: Reply with quote
Cuban's released numbers showed Kidd as really great in the clutch. Being really good in the clutch helps win the close games. Cuban's valuation of Kidd probably relies on this more than any simple average metric and perhaps more than any GM.
A recent study of team consistency had Dallas as one of the most volatile but what wasn't displayed winning versus offensive and defensive performance. For which teams did their volatility pattern add wins above expected win % by the averages? Dallas is tied for first in win% - expected win % this season. How much is because of Kidd? Would they be better or worse with Harris? Would need to know his consistency pattern and impact on team volatility pattern- does he add wins above expected win %? I assume Mark has this or at least should. If he does maybe he knows what he did or why he took the risk. If Harris was not the equal of Kidd on this then Cuban has a rationale beyond what most are aware of or understand. I see Dallas was the 4th biggest under-performer on win % - expected % among playoff level teams last season. How much was Avery? How much was Harris vs Kidd under Avery by part of season? Would Dallas be a better contender with Harris right now or would they be out of the race? I don't have an immediate answer to that but there are huge incentives for Mark to avoid the latter- pride and financial. I do see that Harris had the best on the court win % on the team last season but Kidd has that status on team this season. I've only studied the issue a few minutes but I'd say it is a hard to say what was right quickly. In some ways it come down to trying something different before the Nowitski window closes. I can understand that.
If you expect to have to go thru a valley in the near future maybe you can rationalize the loss of those draft picks if you think you will or plan to suck for awhile and get better picks later, or cut payroll and cut losses or even add profit for awhile then go back to maxing out when you have the next set of talent. Or if you just think really young guys don't fit with late stage try to still compete with Dirk-Jason which I can understand. Maybe the new economic climate and perhaps CBA will leave lots of vets looking for a good enough team in a desired city. Draft picks beyond 3rd or 5th probably have been overrated by many owners, GMs and fans in my opinion compared to what you can get in the free agent market. I think the future will shift further in favor of good teams using the expected lower cost, better value, quicker to get what you want free agent market.
Back to top
View user's profile Send private message
Mike G
Joined: 14 Jan 2005
Posts: 3558
Location: Hendersonville, NC
PostPosted: Wed Mar 11, 2009 3:09 pm Post subject: Reply with quote
Schtevie,
Shall we 'retrodict' 2008 based on 2007 performance metrics, but applying actual 2008 player-minutes?
If my '08 prediction did not contain any 'aging' factor, might it now include such?
_________________
`
36% of all statistics are wrong
Back to top
View user's profile Send private message Send e-mail
Neil Paine
Joined: 13 Oct 2005
Posts: 774
Location: Atlanta, GA
PostPosted: Wed Mar 11, 2009 3:32 pm Post subject: Reply with quote
Mountain wrote:
If you expect to have to go thru a valley in the near future maybe you can rationalize the loss of those draft picks if you think you will or plan to suck for awhile and get better picks later, or cut payroll and cut losses or even add profit for awhile then go back to maxing out when you have the next set of talent. Or if you just think really young guys don't fit with late stage try to still compete with Dirk-Jason which I can understand. Maybe the new economic climate and perhaps CBA will leave lots of vets looking for a good enough team in a desired city. Draft picks beyond 3rd or 5th probably have been overrated by many owners, GMs and fans in my opinion compared to what you can get in the free agent market. I think the future will shift further in favor of good teams using the expected lower cost, better value, quicker to get what you want free agent market.
I think we're seeing a lot of teams decide to go against the grain recently, because the economic crisis really changed the game on everyone -- the vaunted summer of 2010 suddenly comes with a lower tax ceiling, etc. Besides, if everyone is geared towards the 2010 FA class, you could go bargain-hunting elsewhere.
So I agreed with Cuban's financial rationale, especially when he posted this last May:
Quote:
But the current year wasn’t really my personal deciding factor. Looking at our future cap structure was. In doing a deal for JKidd, we created a situation where Devin, Mo Ager, Hass and what we would have paid Ghana would no longer be on our cap. Which put us in a position for the future that I looked at as follows:
2008-9 We have a full training camp with a very motivated JKidd, the rest of our starters back, an improving Nasty Bassty (had to get that in there), plus anyone we can add. We all thought (and still do), we would have a very strong nucleus to build around. We would also have a 1st round pick.
2009-10 Depending on how the previous season went, we would have several last year contracts available, the option of potentially having some cap room, and other options to improve the team. With the much lower potential salaries, not only could we use cap room if we went that direction to enable roster flexibility, we could also buy a pick. (there are almost always teams willing to sell a pick in the 20s for 3mm dollars)
2010-11 In this year we only have 2 fully guaranteed contracts and in 2011-12 We dint have any fully guaranteed contracts. So the options are endless in both years. Plus , its in one of these years that the new CBA comes up. With so little committed, depending on how hard a line the owners take, things could get very, very inter sting. Having so little contractually committed could be a great place to be when other teams look to dump salaries to avoid the risks of a lockout.
So with the changes, while we lost a great player in Devin, we felt like we were picking up someone who could spark the team and add energy on the court. At best, we re energized, at worst, it doesn’t help, but we have improved the roster flexibility for the future and improved our opportunity to re invent ourselves, just as we had after the 04 05 season. And as far as the draft picks, there is no question there is risk there, but in the NBA, there is always the option to buy low first round draft picks, so we weren’t as concerned there.
So from my my viewpoint, we accelerated salaries, which wasn’t cheap, and in exchange, got a player that our basketball people thought could energize and help us.
Now, that's actually even more sensible in light of the financial meltdown that transpired since the post... But then why did they turn around and re-sign Diop a few months later for 5 years, $32.4 million? Only to trade him to Charlotte for Carroll, who's on the books through at least 2012 for about $4.5 million per year? And that's not even considering the tax threshold, which a good-but-not-championship-caliber Mavs roster will be very close to with the $69 million they've already committed for next year. Did any of that make sense in the context of the post I quoted?
All of which is to say, I don't get Cuban sometimes. One second, he'll look like he's ahead of the market... and the next, he'll do something to undermine that position. However, IMO, the Kidd trade wasn't an horrendous misstep, certainly not on par with some of Phoenix's moves since 2007.
Back to top
View user's profile Send private message Visit poster's website
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Thu Mar 12, 2009 2:17 am Post subject: Reply with quote
Thanks for the Cuban quote I hadn't seen that.
Don't know the Cuban - Donnie Nelson dynamic well. Wonder who was the bigger fan of Diop / if there was disagreement and who liked Carroll more.
Some guys are used as currency and might be on the move again.
Hollins might be under-talked about and a bigger part of that deal in th elong-run. Still a mixed bag on boxscore stats but I see +10.5 on 2 yr adjusted, though that may be mostly from Charlotte. Excellent counterpart defense in Dallas. Diop's turned quite weak this season.
Back to top
View user's profile Send private message
BobboFitos
Joined: 21 Feb 2009
Posts: 190
Location: Cambridge, MA
PostPosted: Thu Mar 12, 2009 2:58 am Post subject: Reply with quote
Quote:
Now, that's actually even more sensible in light of the financial meltdown that transpired since the post... But then why did they turn around and re-sign Diop a few months later for 5 years, $32.4 million? Only to trade him to Charlotte for Carroll, who's on the books through at least 2012 for about $4.5 million per year? And that's not even considering the tax threshold, which a good-but-not-championship-caliber Mavs roster will be very close to with the $69 million they've already committed for next year. Did any of that make sense in the context of the post I quoted?
This is a good point that didn't make sense to me either. The Diop signing seemed like a gross misuse of the MLE and is oppositional logic as far as what Cuban said he wanted to do w/ his team.
_________________
-Rob
Back to top
View user's profile Send private message AIM Address
schtevie
Joined: 18 Apr 2005
Posts: 408
PostPosted: Mon Mar 16, 2009 9:20 am Post subject: Reply with quote
Let me return things now to the topic of retrodiction. Here are the results for APM and Wages of Wins retrodiction of the 2007-2008 season, using the 2006-2007 estimates of individual productivities and actual 2007-2008 season minutes.
For the APM retrodiction, the data come from Steve Ilardi's 82games article about the 2006-2007 season (which in turn imposes a one quarter weight for 2005-2006 APMs. As the results are on a per-minute basis, the exercise was to convert these based on average team pace to a per possession basis. For replacement level players (those not getting 400 minutes per season) I assumed a value of -7.
For the Wins Produced retrodiction, these come directly from Dave Berri's site.
To put the metrics on a common basis of predicted wins, I used a regression of the difference between offensive and defensive rating on actual 2007-2008 wins. The argument is not that this linear relationship provides the best fit, but it surely does a serviceable job of providing a common baseline for comparison.
So, the results:
For APM, the average number of wins retrodicted is 38.4 for WoW it is 35.9. That the APM comes close to 41 is not surprising, given that the process provides unbiased estimates of individual contributions. And if the actual 2006-2007 replacement value was slightly greater (less) than -7, the difference would be somewhat less (greater) still.
In terms of the average error and its dispersion, here WoW does better. An APM retrodiction of the 2007-2008 season yields an average error of 10.8 games, compared to 8.8 for WoW. Much of this difference is provided by the most extreme outliers. The standard deviations for APM and WoW are 13.2 and 9.1, respectively.
Before discussing these results, let me again encourage others again to add their retrodictions, the goal being a better informed conventional wisdom. Neil Paine suggests that statistical +/- (at least for this season) might yield an error around 6+ points. Let's see what others come up with.
Back to the story at hand.
As I tried to lead the conversation, before presenting results, it is the discussion of unexpected vs. expected errors which is, to me at least, perhaps the most interesting thing about this exercise in retrodiction. When retrodictions are nailed, if mitigating circumstances suggest that they actually failed, these should not be considered successes, and vice versa.
Let me try again to induce some commentary, focusing on the "failed" retrodictions of both APM and WoW. (Though the same discussion could take place with the "successes".
APM did a bad job, in order of absolute terribleness, of retrodicting the records of the following teams (I pick the worst five): (1) Washington, (2) Indiana, (3) Utah, (4) Portland, and (5) Minnesota. All it should be noted were underestimates.
By contrast WoW's "top" five are: (1) Portland, (2) Orlando, (3) Detroit, (4) Denver, and (5) Chicago. All of these except the last are underestimates, perhaps less surprising than with APM as WoW retrodictions are 5.1 wins short on average. And note that only Portland is a common egregious error.
In terms of the teams which provided the greatest difference between the retrodictions of the two approaches, these are: (1) Boston, (2) Orlando, (3) Detroit, (4) Philadelphia, and (5) Washington.
So, to put a more focused question to the floor, in terms of explaining the collective and individual failures listed here, what are the stories that you all have to tell about these aforementioned teams in 2007-08 as to how they performed vs. expectations?
In terms of unexpected outcomes, which teams were plagued by extraordinary number of minutes played by players with nagging injuries? Which teams had players with "unexpected" break-out seasons? Which teams seemed to have a disproportionate number of general age-related declines?
My hope is that the intersection of such informed interpretation and empirical methods might give us all a firmer grounding for our prior beliefs.
Back to top
View user's profile Send private message
erivera7
Joined: 19 Jan 2009
Posts: 178
Location: Chicago, IL
PostPosted: Mon Mar 16, 2009 1:24 pm Post subject: Reply with quote
schtevie wrote:
In terms of the teams which provided the greatest difference between the retrodictions of the two approaches, these are: (1) Boston, (2) Orlando, (3) Detroit, (4) Philadelphia, and (5) Washington.
So, to put a more focused question to the floor, in terms of explaining the collective and individual failures listed here, what are the stories that you all have to tell about these aforementioned teams in 2007-08 as to how they performed vs. expectations?
In terms of unexpected outcomes, which teams were plagued by extraordinary number of minutes played by players with nagging injuries? Which teams had players with "unexpected" break-out seasons? Which teams seemed to have a disproportionate number of general age-related declines?
My hope is that the intersection of such informed interpretation and empirical methods might give us all a firmer grounding for our prior beliefs.
I can speak about the Magic.
Last year, the "surprises" that occurred for Orlando were two main things, a.) a career-year from Hedo Turkoglu and b.) an improved defensive squad, despite the fact Rashard Lewis "became" a power forward and the team ratcheted up the pace [25th in '07, 9th in '08]. Not really a surprise, but a trend was that the Magic stayed relatively healthy throughout the year. No one on the squad, except for Tony Battie, suffered a serious injury (which would have required substantial time in street clothes). Due to these two main factors, Orlando was able to become a 50+ win squad after many before the preseason, surmised that the team would win roughly 43-47 games.
_________________
@erivera7
I cover the Orlando Magic - Magic Basketball
Back to top
View user's profile Send private message
Ilardi
Joined: 15 May 2008
Posts: 262
Location: Lawrence, KS
PostPosted: Mon Mar 16, 2009 2:01 pm Post subject: Reply with quote
Schtevie,
The value of a replacement player for that 06-07 season was -3.82; that's a lot lower than the -7.0 value you used. I suspect this discrepancy could account for at least part of the APM model's less-than-stellar retrodiction. But there are at least three other issues:
1) The APM estimates from that 06-07 article were still pretty noisy, since they only had the benefit of 2 seasons' worth of data to push down standard errors. The APM estimates for 07-08 (i.e., the ones Aaron and I published) are dramatically less noisy (using 5 seasons' worth of data), and should provide much better retrodiction if you're willing to repeat this exercise at the conclusion of the 08-09 season next month.
2) Individual player APM values display a predictable year-to-year pattern of improvement/decline based on player age, position, experience, and injury. I haven't yet refined this predictive model - it's nonlinear and tricky - but hope to have it completed soon. Once I've got it, you're welcome to use it to adjust each 07-08 player value for use in the 08-09 retrodiction study. (In fairness, though, you should also apply a similar procedure to the WoW estimates.)
3) Finally, rookies should probably be treated as special cases - with values that will be (on average) higher than that of low-minute replacement players, but nevertheless lower than the overal league average. When I have a chance, I'll try to come up with a rookie APM forecast algorithm based on draft position, and will pass it along once it's completed . . .
ssi
schtevie wrote:
Let me return things now to the topic of retrodiction. Here are the results for APM and Wages of Wins retrodiction of the 2007-2008 season, using the 2006-2007 estimates of individual productivities and actual 2007-2008 season minutes.
For the APM retrodiction, the data come from Steve Ilardi's 82games article about the 2006-2007 season (which in turn imposes a one quarter weight for 2005-2006 APMs. As the results are on a per-minute basis, the exercise was to convert these based on average team pace to a per possession basis. For replacement level players (those not getting 400 minutes per season) I assumed a value of -7.
For the Wins Produced retrodiction, these come directly from Dave Berri's site.
To put the metrics on a common basis of predicted wins, I used a regression of the difference between offensive and defensive rating on actual 2007-2008 wins. The argument is not that this linear relationship provides the best fit, but it surely does a serviceable job of providing a common baseline for comparison.
So, the results:
For APM, the average number of wins retrodicted is 38.4 for WoW it is 35.9. That the APM comes close to 41 is not surprising, given that the process provides unbiased estimates of individual contributions. And if the actual 2006-2007 replacement value was slightly greater (less) than -7, the difference would be somewhat less (greater) still.
In terms of the average error and its dispersion, here WoW does better. An APM retrodiction of the 2007-2008 season yields an average error of 10.8 games, compared to 8.8 for WoW. Much of this difference is provided by the most extreme outliers. The standard deviations for APM and WoW are 13.2 and 9.1, respectively.
Before discussing these results, let me again encourage others again to add their retrodictions, the goal being a better informed conventional wisdom. Neil Paine suggests that statistical +/- (at least for this season) might yield an error around 6+ points. Let's see what others come up with.
Back to the story at hand.
As I tried to lead the conversation, before presenting results, it is the discussion of unexpected vs. expected errors which is, to me at least, perhaps the most interesting thing about this exercise in retrodiction. When retrodictions are nailed, if mitigating circumstances suggest that they actually failed, these should not be considered successes, and vice versa.
Let me try again to induce some commentary, focusing on the "failed" retrodictions of both APM and WoW. (Though the same discussion could take place with the "successes".
APM did a bad job, in order of absolute terribleness, of retrodicting the records of the following teams (I pick the worst five): (1) Washington, (2) Indiana, (3) Utah, (4) Portland, and (5) Minnesota. All it should be noted were underestimates.
By contrast WoW's "top" five are: (1) Portland, (2) Orlando, (3) Detroit, (4) Denver, and (5) Chicago. All of these except the last are underestimates, perhaps less surprising than with APM as WoW retrodictions are 5.1 wins short on average. And note that only Portland is a common egregious error.
In terms of the teams which provided the greatest difference between the retrodictions of the two approaches, these are: (1) Boston, (2) Orlando, (3) Detroit, (4) Philadelphia, and (5) Washington.
So, to put a more focused question to the floor, in terms of explaining the collective and individual failures listed here, what are the stories that you all have to tell about these aforementioned teams in 2007-08 as to how they performed vs. expectations?
In terms of unexpected outcomes, which teams were plagued by extraordinary number of minutes played by players with nagging injuries? Which teams had players with "unexpected" break-out seasons? Which teams seemed to have a disproportionate number of general age-related declines?
My hope is that the intersection of such informed interpretation and empirical methods might give us all a firmer grounding for our prior beliefs.
Back to top
View user's profile Send private message
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Mon Mar 16, 2009 2:04 pm Post subject: Reply with quote
All five of APMs unders were bottom half on crude unweighted average team age. Past season APM data might miss the learning improvement and might need an adjustment for that.
3 of WOWs 4 underestimates were in oldest 11.
The biggest differences were (1) Boston, (2) Orlando, (3) Detroit, (4) Philadelphia, and (5) Washington and they represent 5 of the top 7 offense in the east. And 4 of the top 5 in east on defense. But how do the methods vary- consistently one higher or scattered? Probably need a strength of schedule adjustment and the elite east teams I assume would have gone up quite a bit from that last season.
I checked a few other things quickly- home/road, playoff/lottery opponent, 3 pt game. turnovers, fouls shots but didn't find any other obvious trend.
Back to top
View user's profile Send private message
schtevie
Joined: 18 Apr 2005
Posts: 408
PostPosted: Mon Mar 16, 2009 2:37 pm Post subject: Reply with quote
Steve,
Thanks for chiming in. I presume the -3.82 is also on the per 40 minute basis and not a recalculated per possession basis? I will modify the results and repost accordingly.
Regarding your three points:
1) I share your belief that an upcoming retrodiction based on five years of data would be less noisy. Interestingly however, I performed the same exercise with one year '05-'06 data as well, and it didn't seem to improve the retrodictive quality. Regarding who should do the '08-'09 retrodiction, might I nominate you and Aaron? You got the data sitting there all ready to go....
2) Has there ever been a doubt that incorporating the effects of age, injury, etc. wouldn't improve the predictive quality of the model? I note that the residuals from the retrodictions that I reported are significantly related to minute-weighted roster ages, both in terms of size and statistical significance (not the case with WoW residuals, by contrast). Sorting this out at the player level will matter.
3) I look forward to the rookie retrodiction, but given the typically low minutes played by this cohort, they shouldn't drive the results of this exercise.
Thanks again.
Steve
Back to top
View user's profile Send private message
gabefarkas
Joined: 31 Dec 2004
Posts: 1313
Location: Durham, NC
PostPosted: Mon Mar 16, 2009 4:26 pm Post subject: Reply with quote
schtevie wrote:
So, to put a more focused question to the floor, in terms of explaining the collective and individual failures listed here, what are the stories that you all have to tell about these aforementioned teams in 2007-08 as to how they performed vs. expectations?
In terms of unexpected outcomes, which teams were plagued by extraordinary number of minutes played by players with nagging injuries? Which teams had players with "unexpected" break-out seasons? Which teams seemed to have a disproportionate number of general age-related declines?
I think you've answered your own question, at least partly. I think age has a lot to do with it. Portland was under-retrodicted because their key players are all approaching what is usually their peak production age.
Back to top
View user's profile Send private message Send e-mail AIM Address
Ilardi
Joined: 15 May 2008
Posts: 262
Location: Lawrence, KS
PostPosted: Mon Mar 16, 2009 4:44 pm Post subject: Reply with quote
Steve,
Yes, that -3.82 replacement value is a per-40-minute metric.
Also, to echo gabefarkas' point, it looks to me like 3 of the worst fits for the APM retrodiction - Portland, Minnesota, Utah - were young teams that saw major improvement for key personnel.
The other 2 outliers - Washington and Indiana - suffered major injury to key starters (Arenas on the former, O'Neal and Tinsley for the latter), and this may have introduced estimation bias via some previously undetected interaction effects (wherein, for example, players like Jamison and Butler exhibited greater APM value when playing without Arenas, since they had more significant roles on the team.) Presumably, such estimation challenges could be addressed to some extent through the use of 5-year APM estimates (with more varied lineups, which - on average - could help some interaction effects cancel each other out) and (in future) more explicit modeling of interactions.
- ssi
schtevie wrote:
Steve,
Thanks for chiming in. I presume the -3.82 is also on the per 40 minute basis and not a recalculated per possession basis? I will modify the results and repost accordingly.
Regarding your three points:
1) I share your belief that an upcoming retrodiction based on five years of data would be less noisy. Interestingly however, I performed the same exercise with one year '05-'06 data as well, and it didn't seem to improve the retrodictive quality. Regarding who should do the '08-'09 retrodiction, might I nominate you and Aaron? You got the data sitting there all ready to go....
2) Has there ever been a doubt that incorporating the effects of age, injury, etc. wouldn't improve the predictive quality of the model? I note that the residuals from the retrodictions that I reported are significantly related to minute-weighted roster ages, both in terms of size and statistical significance (not the case with WoW residuals, by contrast). Sorting this out at the player level will matter.
3) I look forward to the rookie retrodiction, but given the typically low minutes played by this cohort, they shouldn't drive the results of this exercise.
Thanks again.
Steve
Back to top
View user's profile Send private message
Mike G
Joined: 14 Jan 2005
Posts: 3558
Location: Hendersonville, NC
PostPosted: Mon Mar 16, 2009 5:34 pm Post subject: Reply with quote
http://sonicscentral.com/apbrmetrics/vi ... 5&start=45
Last year, not retrodicting but predicting, Mountain was off by an average of 8.4 wins; Neil (daviswylie), Mike G, and John H were off by 9.2-9.5 .
We were all within 5 W on the Wolves. Only Neil was worse than 4 off on Utah. Blazers surprised everyone by 15-22 extra wins.
Portland's big unpredictable boost came from the great rookie season by Roy. Little was expected of James Jones, yet he produced. Only Rodriguez and Jack regressed, of their whole roster.
_________________
`
36% of all statistics are wrong
Author Message
schtevie
Joined: 18 Apr 2005
Posts: 182
PostPosted: Tue Mar 17, 2009 10:52 am Post subject: Reply with quote
Let me repost the APM retrodiction results, based on replacement value being -3.82 points per 40 minutes as opposed to -7 points per 100 possessions. It doesn't improve things much, but it does a little. The average team wins increases from 38.4 to 39.1, so a little less "unbiased". The average error decreases from 10.8 to 10.5. And the standard deviation of these errors decreases from 13.2 to 13.0. So, there is still some explaining to do with the outliers.
In terms of the ranking of the teams that had the largest retrodiction errors, this did not change.
This said, let me reply to a variety of comments.
Thanks erivera7 for the observations on Orlando. The lack of nagging injuries comment is of interest. However, I don't get the defensive improvement story. They actually got a bit worse in '07-'08, but their offense was much improved. In terms of how APM fared vs. WoW, this was an instance where APM nailed it, with an error of just +0.3 wins with respect to the predicted wins figure. WoW by contrast underpredicted by 18.4 wins. which was "explained" by underestimates of multiple player contributions. By contrast, the APM story is essentially that the improvement over the previous year was more than explained by the rise of Dwight Howard, with remaining players, Turkoglu included, falling short of retrodicted expectations. Significant negative contributions obtained to Nelson and Arroyo and smaller positive contributions than expected obtained to Turkoglu and Lewis.
Which of these stories is more plausible is what I am trying to get at with this discussion.
On to MikeG.
Mike, does your prediction/retrodiction method generated exclusively by a straight formula or are such results edited? I am looking to compare empirical methods first and foremost, not expert interpretation thereof. So, please offer up a retrodiction using actual '07-'08 minutes. And by all means, include also a retrodiction for whatever method you have subsequently come to advocate for age adjustments.
On to Gabe.
What I am interested in advancing is the notion of an improved statistics-based conventional wisdom. Everyone, I think, would agree on the general factors that account for deviations from expected performance. In the fullness of time, some of these factors will be formally incorporated into the models. But for others, interpretation of results will remain key. So, for example, take the just discussed case of Orlando. If nagging injuries were not their affliction last year, score a tentative point for APM vs. WoW in determining how we should think about the world. Then add all other methods into the mix as well. Does anyone know how PER works in relation to such matters?
Finally, as I tend to pay attention to Boston, essentially to the exclusion of other clubs, and that the performance of Boston last year generated the biggest outlier in the comparison of APM and WoW, let me rephrase some previously expressed commentary in the current context.
Retrodicting with just '06-'07 data (with the caveat that the APM numbers are partially based on the previous season as well) APM overpredicts the "actual" win total by 10.7 games whereas WoW underpredicts it by 14.6 games, a 25.3 game retrodiction gap. WoW accounts for its underestimates broadly, with essentially only Tony Allen doing worse that retrodicted. APM by contrast finds the two biggest "underachievers" as being Kevin Garnett and Ray Allen - a story which is consistent with a nagging injury explanation (never mind the aging factor). I recall that contemporaneous APM numbers showed a decline corresponding to his injuries last year (perhaps Aaron has archives to back up this claim?) and I wish I had had the time to pester Ray Allen at the MIT event to ask him how he perceived his relative effectiveness between last season and this.
Anyway, here too I would score one - a three pointer in fact - for APM, but I explicitly note that its howlers have yet to be discussed in depth. Does anyone have specific commentary to offer about: (1) Washington, (2) Indiana, (3) Utah, (4) Portland, and (5) Minnesota?
Back to top
View user's profile Send private message
Ilardi
Joined: 15 May 2008
Posts: 105
Location: Lawrence, KS
PostPosted: Tue Mar 17, 2009 4:22 pm Post subject: Reply with quote
schtevie wrote:
Does anyone have specific commentary to offer about: (1) Washington, (2) Indiana, (3) Utah, (4) Portland, and (5) Minnesota?
I wonder if you missed this comment (below) from earlier in the thread?
Quote:
Also, to echo gabefarkas' point, it looks to me like 3 of the worst fits for the APM retrodiction - Portland, Minnesota, Utah - were young teams that saw major improvement for key personnel.
The other 2 outliers - Washington and Indiana - suffered major injury to key starters (Arenas on the former, O'Neal and Tinsley for the latter), and this may have introduced estimation bias via some previously undetected interaction effects (wherein, for example, players like Jamison and Butler exhibited greater APM value when playing without Arenas, since they had more significant roles on the team.) Presumably, such estimation challenges could be addressed to some extent through the use of 5-year APM estimates (with more varied lineups, which - on average - could help some interaction effects cancel each other out) and (in future) more explicit modeling of interactions.
The basic gist is that the 3 young teams saw great improvement in numerous starters/rotation guys (and perhaps also above-average APM contribution from rookies), while the other 2 outliers (Washington & Indiana) had major injuries to key starters, possibly unmasking player interaction effects that led remaining non-injured personnel to "over-perform" their historic APM numbers that season.
Back to top
View user's profile Send private message
erivera7
Joined: 19 Jan 2009
Posts: 52
PostPosted: Wed Mar 18, 2009 3:27 am Post subject: Reply with quote
schtevie wrote:
Thanks erivera7 for the observations on Orlando. The lack of nagging injuries comment is of interest. However, I don't get the defensive improvement story. They actually got a bit worse in '07-'08, but their offense was much improved. In terms of how APM fared vs. WoW, this was an instance where APM nailed it, with an error of just +0.3 wins with respect to the predicted wins figure. WoW by contrast underpredicted by 18.4 wins. which was "explained" by underestimates of multiple player contributions. By contrast, the APM story is essentially that the improvement over the previous year was more than explained by the rise of Dwight Howard, with remaining players, Turkoglu included, falling short of retrodicted expectations. Significant negative contributions obtained to Nelson and Arroyo and smaller positive contributions than expected obtained to Turkoglu and Lewis.
Which of these stories is more plausible is what I am trying to get at with this discussion.
If you want to be technical, the Magic got "worse", yeah .. when looking at Orlando's defensive efficiency in 2007 (6th, 104.1) compared to 2008 (5th, 105.5). However, the reason why the team did improve (in my eyes) is because they sped up the tempo last year. So even though by statistical technicality, the Magic appeared worse on paper, they were better and more impressive defensively given the increased pace. Add the fact head coach Stan Van Gundy played Rashard Lewis at the power forward position, and I feel given the totality of the situation, Orlando improved just a smidgen on defense (in my opinion).
Obviously the main reason why the Magic were a better team last year was, more so, due to the offense becoming a juggernaut statistically. There's no doubt about that (I was alluding to that point when I was referring to Hedo Turkoglu's career year last season).
_________________
Contributing Editor
Third Quarter Collapse (An Orlando Magic blog)
Back to top
View user's profile Send private message
Mike G
Joined: 14 Jan 2005
Posts: 2363
Location: Delphi, Indiana
PostPosted: Wed Mar 18, 2009 9:08 am Post subject: Reply with quote
OK, this was a godawful amount of work, but obviously a useful exercise. After changing my 'predictions' spreadsheet to account for minutes and trades, my results are worse. Average error from Pythagorean-expected is 10.7 (worse than from actual 2008 wins, 10.1).
I had predicted the Wiz to win 39.5, and their Pyth was 40.2 -- pretty good, except that it was all luck. Knowing the actual minutes, I'd have predicted 23 wins. In order of minutes played, postdicted and actual eWins, and eW/1000 (average is about 2.0 ) :
Code:
eWPre eWAct eW/1000 min : 2007 2008
9.0 9.9 Jamison,Antawn 2.94 3.24
3.7 3.3 Stevenson,Deshaw 1.45 1.29
6.2 6.9 Butler,Caron 2.66 3.00
3.8 5.4 Haywood,Brendan 1.72 2.43
2.5 2.8 Daniels,Antonio 1.14 1.29
.1 3.0 Mason,Roger .04 1.76
2.0 3.8 Blatche,Andray 1.18 2.26
1.5 2.1 Songaila,Darius .99 1.37
.7 1.8 Young, Nick .60 1.52
.3 .2 McGuire, Dominic .40 .30
1.7 1.2 Arenas,Gilbert 3.99 2.75
.2 .5 Pecherov, O .50 1.49
Jamison and Butler improve nicely but not dramatically. Hugely improved are Haywood and Blatche. Little to nothing were expected from Mason and Young.
Total postdicted eWins were 32, and 41 were delivered. Expected wins, though, are twice as disparate.
xW = eW*2 - 41 = 23.
It's a two-edged sword.
_________________
`
There's no I in analysys.
Back to top
View user's profile Send private message Send e-mail
schtevie
Joined: 18 Apr 2005
Posts: 182
PostPosted: Wed Mar 18, 2009 12:57 pm Post subject: Reply with quote
Steve,
Sorry, I hadn't intended to leave the impression that your summary was being ignored. I was just trying to solicit commentary from folks who follow those teams listed with special passion. In particular, I am interested in testimony that speaks to the issue of injuries, of two types. Injuries that upon returning to the line-up left the player less productive than before and injuries which were more of the nagging type, with similar expected effects on productivity.
I am of a belief that these factors are very significant, at least in APM land. For example, the single biggest factor accounting for the wretchedness of the APM retrodiction of the Wizards had to do with the contribution of Antawn Jamison. Were his 2005-06 APM estimate to have been used instead, this would have eliminated a huge chunk of the retrodiction error. Was 2006-07 anomalous because of a noisy estimate, or were there relevant injuries? He did miss all of February 2007 with a sprained left knee. Was the estimated APM as of the end of January, higher than at season's end? And so on. And Caron Butler too had anomalously crappy 2006-07 according to APM. His injury report shows a series of injuries starting at the end of February 2007, beginning with back spasms. Perhaps these are significant.
Before interaction effects are addressed, I would first be curious what the residual is upon dealing with the more obvious fundamentals of age and injury (not that identifying all relevant injuries is that easy to do). Alchemy should follow.
Mike,
Thanks for the numbers. To make sure I understand, you are saying that your method retrodicts on average 32 out of 41 wins? Also, what is the standard deviation of the errors? And finally, which teams produced the five worst retrodictions?
Back to top
View user's profile Send private message
Kevin Pelton
Site Admin
Joined: 30 Dec 2004
Posts: 828
Location: Seattle
PostPosted: Wed Mar 18, 2009 1:49 pm Post subject: Reply with quote
schtevie, perhaps I am misunderstanding, but what makes you think injury and interaction effects are more problematic for an adjusted plus-minus model than for those using box-score stats? The former would seem to affect both equally, and aren't the interaction effects in large part what we're seeking to tease out with adjusted plus-minus?
The question here isn't how well adjusted plus-minus predicts team performance. It's how well it predicts team performance vis-a-vis box-score stat methods, or how well various forms of adjusted plus-minus predict relative to each other, which is an interesting test of the practical viability of combining seasons.
Getting away from this central question tends to distract from the issue, I think.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jkubatko
Joined: 05 Jan 2005
Posts: 668
Location: Columbus, OH
PostPosted: Wed Mar 18, 2009 1:55 pm Post subject: Reply with quote
I guess I'll throw in some results as well. I decided to retrodict the 1979-80 through 2007-08 seasons using Win Shares. The rules:
* The player's minutes played total for the retrodicted season is known.
* The player's rate of Win Shares per minute played from his last season with at least 250 minutes was used as his predicted rate of production.
* If the player was a rookie, then a rate of 0.00134 WS/MP was used (that value is the average rate for all rookies during the study's time period).
* If the player was not a rookie and he did not play at least 250 minutes in any previous season, then he was given a value of 0. In other words, rather than try to fudge some replacement-level value, I decided to err on the side of caution.
This is mainly going to be a data dump without commentary, but let me state that I was pleasantly surprised with the results.
First, here are the results for each season individually:
Code:
+---------+----------+-------------+-------------+
| year_id | avg_wins | avg_abs_err | std_dev_err |
+---------+----------+-------------+-------------+
| 1980 | 40.56 | 8.06 | 10.48 |
| 1981 | 39.16 | 7.94 | 8.93 |
| 1982 | 40.84 | 6.46 | 8.29 |
| 1983 | 40.74 | 4.93 | 6.27 |
| 1984 | 42.04 | 3.96 | 5.15 |
| 1985 | 39.03 | 7.72 | 9.16 |
| 1986 | 40.23 | 3.82 | 5.40 |
| 1987 | 40.74 | 6.57 | 7.80 |
| 1988 | 40.90 | 5.59 | 6.73 |
| 1989 | 39.83 | 7.44 | 9.16 |
| 1990 | 40.07 | 6.58 | 8.14 |
| 1991 | 40.48 | 4.53 | 6.15 |
| 1992 | 41.10 | 7.41 | 8.54 |
| 1993 | 41.28 | 7.25 | 8.87 |
| 1994 | 39.46 | 6.69 | 8.54 |
| 1995 | 41.16 | 7.32 | 9.34 |
| 1996 | 40.96 | 5.51 | 7.22 |
| 1997 | 40.64 | 7.05 | 8.85 |
| 1998 | 41.85 | 8.21 | 10.19 |
| 1999 | 25.60 | 4.42 | 5.38 |
| 2000 | 40.80 | 6.63 | 8.64 |
| 2001 | 40.99 | 5.76 | 7.29 |
| 2002 | 41.27 | 5.84 | 7.44 |
| 2003 | 40.80 | 6.10 | 7.91 |
| 2004 | 40.81 | 6.41 | 7.91 |
| 2005 | 40.13 | 7.57 | 9.62 |
| 2006 | 40.86 | 6.73 | 7.94 |
| 2007 | 41.08 | 6.02 | 7.86 |
| 2008 | 41.49 | 8.30 | 10.13 |
+---------+----------+-------------+-------------+
Here are the cumulative results:
Code:
+----------+----------+----------+-------------+-------------+
| year_min | year_max | avg_wins | avg_abs_err | std_dev_err |
+----------+----------+----------+-------------+-------------+
| 1980 | 2008 | 40.15 | 6.46 | 8.07 |
+----------+----------+----------+-------------+-------------+
The ten biggest misses:
Code:
+---------+---------+------+-------+--------+
| year_id | team_id | wins | pred | error |
+---------+---------+------+-------+--------+
| 1980 | BOS | 61 | 36.67 | 24.33 |
| 2005 | PHO | 62 | 38.05 | 23.95 |
| 2008 | MIA | 15 | 38.16 | -23.16 |
| 1990 | SAS | 56 | 33.03 | 22.97 |
| 1998 | DEN | 11 | 33.68 | -22.68 |
| 1989 | PHO | 55 | 34.25 | 20.75 |
| 2008 | BOS | 66 | 45.35 | 20.65 |
| 1982 | CLE | 15 | 35.35 | -20.35 |
| 1998 | TOR | 16 | 36.12 | -20.12 |
| 2000 | LAL | 67 | 46.96 | 20.04 |
+---------+---------+------+-------+--------+
The ten closest calls:
Code:
+---------+---------+------+-------+-------+
| year_id | team_id | wins | pred | error |
+---------+---------+------+-------+-------+
| 1993 | LAL | 39 | 38.97 | 0.03 |
| 1986 | UTA | 42 | 41.95 | 0.05 |
| 1984 | MIL | 50 | 50.07 | -0.07 |
| 1989 | BOS | 42 | 42.08 | -0.08 |
| 2005 | SAS | 59 | 59.09 | -0.09 |
| 2003 | POR | 50 | 49.90 | 0.10 |
| 1981 | DET | 21 | 20.90 | 0.10 |
| 1991 | ATL | 43 | 42.88 | 0.12 |
| 1984 | DAL | 43 | 43.13 | -0.13 |
| 1983 | SEA | 48 | 48.14 | -0.14 |
+---------+---------+------+-------+-------+
_________________
Regards,
Justin Kubatko
Basketball-Reference.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Mike G
Joined: 14 Jan 2005
Posts: 2363
Location: Delphi, Indiana
PostPosted: Wed Mar 18, 2009 5:21 pm Post subject: Reply with quote
schtevie wrote:
... you are saying that your method retrodicts on average 32 out of 41 wins? Also, what is the standard deviation of the errors? And finally, which teams produced the five worst retrodictions?
5 worst:
Code:
tm retro W PythW PyEr
Mia 43.3 15 19.0 -24.3
Por 20.2 41 38.3 18.1
Chi 50.2 33 32.7 -17.5
Was 23.3 43 40.2 16.9
Cha 46.2 32 29.4 -16.8
In our season-predicting threads of the last 2 seasons, I've gone by actual wins; but in this thread, I'm ranking by pythagorean.
I postdicted 32 eWins for the Wiz, knowing their minutes. It was just my sense of irony that prompted me to report that hindsight (of player minutes) led to a much-worse estimate of their win rate.
On average, 2007 eW/min * min postdict an average error of 10.7 wins per team.
_________________
`
There's no I in analysys.
Last edited by Mike G on Thu Mar 19, 2009 10:29 am; edited 1 time in total
Back to top
View user's profile Send private message Send e-mail
schtevie
Joined: 18 Apr 2005
Posts: 182
PostPosted: Wed Mar 18, 2009 6:12 pm Post subject: Reply with quote
In order:
Kevin,
I don't have particularly strong beliefs that injury and interaction effects are more problematic for an adjusted plus-minus model than for those using box-score stats. If the argument is that APM picks up all the box score stuff and also all the other little things, then there is a reason to believe that injuries matter more in APMville. If Ray Allen's notionally tender ankles affected his jumpshot last year, they probably also affected his determination to fight through picks, etc.
As for the interaction effects, I don't think that this is what we're seeking to tease out with adjusted plus-minus per se. If they are real and significant (in both senses of the term) by all means lets find them, by whatever approach. However, we have a simple theory that age matters, and a not unrelated theory that injury matters. Interaction effects can be thought of as a wordier way of saying "chemistry" which always has been a way of invoking the residual. My point is let's see what is left to be explained after we account for "real" things.
Justin,
Thank you for the Win Shares results. It was interesting to see that 2007-08 was the worst year "ever" for the retrodiction business.
It is perhaps not surprising that this method does well on the important criterion of actually including all wins, on average, given that it is based on Offensive and Defensive Rating. That Boston's performance was completely unanticipated, however, caught me a bit by surprise. I would have though that an approach, again, based on Offensive and Defensive Ratings would have done better.
Ex post, what accounts for the failed retrodiction in this instance? That is, who performed worse than expected?
Finally, would you also share the five worst failures of this method. I think that comparing the howlers is potentially very informative. I note that Miami was retrodicted by APM to have an essentially identical record - 37.4 - as what Win Shares came up with - 38.16. Strange, it seems, that Boston and Miami should perform so differently between the two approaches. Or perhaps not.
Mike,
I am still a bit confused about what your method predicted in terms of average wins overall. Wages of Wins left 5.1 wins on the table, on average, per team. APM 2.0 wins. Win Shares offered 0.49 wins too many. And your method?[/i]
Back to top
View user's profile Send private message
Mike G
Joined: 14 Jan 2005
Posts: 2363
Location: Delphi, Indiana
PostPosted: Thu Mar 19, 2009 5:59 am Post subject: Reply with quote
Ah, gotcha. The average '08 record was 39.2 wins predicted.
So expected wins were 1.8 short of 41 . Equivalent (eWins) were 0.9 short. Team minutes were all above 19,000, so some 98% accounted-for.
Every Heat player had a bad year in '08. RDavis, he of 2960 minutes, was but 61% of his 2007 self; by himself creating 6 fewer wins than predicted. Other shortfalls: Wade 8.3, JWill 3.3, Shaq 3.2, Haslem 2.5, Blount 1.5, Marion 1.4 . No gainer over 1.8 (Quinn). These add up (or down) dramatically.
Among 8-10 players, expect half to improve and half to regress. When >3/4 of a lineup goes one way, that's unpredictable.
_________________
`
There's no I in analysys.
Back to top
View user's profile Send private message Send e-mail
jkubatko
Joined: 05 Jan 2005
Posts: 668
Location: Columbus, OH
PostPosted: Thu Mar 19, 2009 1:01 pm Post subject: Reply with quote
schtevie wrote:
It is perhaps not surprising that this method does well on the important criterion of actually including all wins, on average, given that it is based on Offensive and Defensive Rating. That Boston's performance was completely unanticipated, however, caught me a bit by surprise. I would have though that an approach, again, based on Offensive and Defensive Ratings would have done better.
Ex post, what accounts for the failed retrodiction in this instance? That is, who performed worse than expected?
I assume you mean who performed *better* than expected for Boston, not who performed worse. Offensively, the youngsters (Rondo, Perkins, and Powe) all took big steps forward, and Garnett was much more efficient in '08 than he had been in '09, but obviously the biggest changes were on the defensive end.
Quote:
Finally, would you also share the five worst failures of this method. I think that comparing the howlers is potentially very informative. I note that Miami was retrodicted by APM to have an essentially identical record - 37.4 - as what Win Shares came up with - 38.16. Strange, it seems, that Boston and Miami should perform so differently between the two approaches. Or perhaps not.
I did share the ten worst failures of this method, but I'm guessing you mean the five worst failure of 2007-08. Here they are:
Code:
+---------+---------+------+-------+--------+
| year_id | team_id | wins | pred | error |
+---------+---------+------+-------+--------+
| 2008 | MIA | 15 | 38.16 | -23.16 |
| 2008 | BOS | 66 | 45.35 | 20.65 |
| 2008 | CHI | 33 | 51.84 | -18.84 |
| 2008 | NYK | 23 | 37.52 | -14.52 |
| 2008 | NOH | 56 | 42.66 | 13.34 |
+---------+---------+------+-------+--------+
_________________
Regards,
Justin Kubatko
Joined: 18 Apr 2005
Posts: 408
PostPosted: Wed Mar 11, 2009 10:57 am Post subject: retrodiction Reply with quote
Perhaps the forces of entropy are too great, but I like to think that it would be productive to, at least, try to achieve some sort of consensus about the relative merits (and shortcomings) of various methods for assessing NBA player (and team) value.
This point, to me, was underscored by John Hollinger's Cuban-baiting at the Sloan School Conference, regarding the wisdom of the Harris/Kidd trade. Though certainly amusing, it was, in its essence, a simple case of PER said vs. APM (or Cuban's brand thereof) said. Clearly, Kidd's PER of 17.4 this year vs. Harris' PER of 22.7 was the basis of John's smug questioning of the wisdom of the trade. And by contrast, Cuban's somewhat exasperated "you haven't a clue" reply was surely based on a fact similar to that shown by current APMs which show that the Mavs got the upper hand. Considering just the principals (as the contributions of remaining players involved in the trade essentially offset each other, and never mind the implications of the draft picks for now) a minutes-weighted estimate of the 3+ APM differential suggest that the Mavs will get about an extra 6 wins this year from the trade.
Now, one should not expect consensus on matters such as these, but at least there should be explicit recognition of and respect for plausible interpretations.
In a recent string on APM, Dean made the correct point that isolated predictions are necessarily of limited value, and that a possibly useful starting point for assessing varying methods of appraisal would be by retrodiction. As one of the apparent, unaffiliated champions of APM on this board, I did the calculations for this method, retrodicting the 2007-08 season and will post the summary statistics soon. (Additionally, Dave Berri has unwittingly agreed to play along, with his posted retrodictions on his site.)
Before presenting these results, however, I would like it if the journalists and modern historians on this board would take the time to discuss the unexpected. By this I mean, which players (hence, by summation, teams) had outcomes that might be expected to be poorly predicted by any method of player/team evaluation? In particular, I am thinking of two phenomena, though there may be others.
First, the expectation of the life-cycle of a player is that he improves then deteriorates with age. To the extent that a metric doesn't take this into account, it will be expected to make prediction errors. Note that here I am only looking to identify "ex-ante" the players that played significant minutes that really exhibited "unexpected" maturation or degradation.
Second, there are player injuries that are insufficiently significant to take them out of games, but significant enough to affect their productivity, hence leading to apparent errors in retrodiction. In discussing the performance of the Celtics, I surmised that Ray Allen's off-season surgery might have accounted for Ray Allen's "unexpectedly" low APM the following year.
If the collective wisdom could come up with a list of such prominent instances throughout the league (or suggest other such categories of expected errors) prior to the posting of any results, it would set the stage for an interesting discussion of results.
And in the interim, may I encourage everyone who has a preferred method for evaluation, to post a retrodiction of the 2007-2008 season, explicitly on the basis of such a method (which is to say, with no modification of the results based upon personal interpretation). And Hopefully Neil from basketball-reference will chime in with a statistical APM retrodiction.
Back to top
View user's profile Send private message
Ryan J. Parker
Joined: 23 Mar 2007
Posts: 708
Location: Raleigh, NC
PostPosted: Wed Mar 11, 2009 11:18 am Post subject: Reply with quote
The key point I took from the debate had to do with the business aspect. What players are going to be available in the future (based on each team's salary cap figures) that would make us want to execute specific deals?
I think this is the point Cuban was trying to make. Maybe I'm in the minority, but I've never thought about this sort of thing. Seems to me it would be valuable to know which players are more likely to move in future years based on this kinda stuff.
Certainly there are non-zero probabilities associated with these kinds of things that we can base decisions on.
_________________
I am a basketball geek.
Back to top
View user's profile Send private message Visit poster's website
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Wed Mar 11, 2009 12:19 pm Post subject: Reply with quote
Cuban's released numbers showed Kidd as really great in the clutch. Being really good in the clutch helps win the close games. Cuban's valuation of Kidd probably relies on this more than any simple average metric and perhaps more than any GM.
A recent study of team consistency had Dallas as one of the most volatile but what wasn't displayed winning versus offensive and defensive performance. For which teams did their volatility pattern add wins above expected win % by the averages? Dallas is tied for first in win% - expected win % this season. How much is because of Kidd? Would they be better or worse with Harris? Would need to know his consistency pattern and impact on team volatility pattern- does he add wins above expected win %? I assume Mark has this or at least should. If he does maybe he knows what he did or why he took the risk. If Harris was not the equal of Kidd on this then Cuban has a rationale beyond what most are aware of or understand. I see Dallas was the 4th biggest under-performer on win % - expected % among playoff level teams last season. How much was Avery? How much was Harris vs Kidd under Avery by part of season? Would Dallas be a better contender with Harris right now or would they be out of the race? I don't have an immediate answer to that but there are huge incentives for Mark to avoid the latter- pride and financial. I do see that Harris had the best on the court win % on the team last season but Kidd has that status on team this season. I've only studied the issue a few minutes but I'd say it is a hard to say what was right quickly. In some ways it come down to trying something different before the Nowitski window closes. I can understand that.
If you expect to have to go thru a valley in the near future maybe you can rationalize the loss of those draft picks if you think you will or plan to suck for awhile and get better picks later, or cut payroll and cut losses or even add profit for awhile then go back to maxing out when you have the next set of talent. Or if you just think really young guys don't fit with late stage try to still compete with Dirk-Jason which I can understand. Maybe the new economic climate and perhaps CBA will leave lots of vets looking for a good enough team in a desired city. Draft picks beyond 3rd or 5th probably have been overrated by many owners, GMs and fans in my opinion compared to what you can get in the free agent market. I think the future will shift further in favor of good teams using the expected lower cost, better value, quicker to get what you want free agent market.
Back to top
View user's profile Send private message
Mike G
Joined: 14 Jan 2005
Posts: 3558
Location: Hendersonville, NC
PostPosted: Wed Mar 11, 2009 3:09 pm Post subject: Reply with quote
Schtevie,
Shall we 'retrodict' 2008 based on 2007 performance metrics, but applying actual 2008 player-minutes?
If my '08 prediction did not contain any 'aging' factor, might it now include such?
_________________
`
36% of all statistics are wrong
Back to top
View user's profile Send private message Send e-mail
Neil Paine
Joined: 13 Oct 2005
Posts: 774
Location: Atlanta, GA
PostPosted: Wed Mar 11, 2009 3:32 pm Post subject: Reply with quote
Mountain wrote:
If you expect to have to go thru a valley in the near future maybe you can rationalize the loss of those draft picks if you think you will or plan to suck for awhile and get better picks later, or cut payroll and cut losses or even add profit for awhile then go back to maxing out when you have the next set of talent. Or if you just think really young guys don't fit with late stage try to still compete with Dirk-Jason which I can understand. Maybe the new economic climate and perhaps CBA will leave lots of vets looking for a good enough team in a desired city. Draft picks beyond 3rd or 5th probably have been overrated by many owners, GMs and fans in my opinion compared to what you can get in the free agent market. I think the future will shift further in favor of good teams using the expected lower cost, better value, quicker to get what you want free agent market.
I think we're seeing a lot of teams decide to go against the grain recently, because the economic crisis really changed the game on everyone -- the vaunted summer of 2010 suddenly comes with a lower tax ceiling, etc. Besides, if everyone is geared towards the 2010 FA class, you could go bargain-hunting elsewhere.
So I agreed with Cuban's financial rationale, especially when he posted this last May:
Quote:
But the current year wasn’t really my personal deciding factor. Looking at our future cap structure was. In doing a deal for JKidd, we created a situation where Devin, Mo Ager, Hass and what we would have paid Ghana would no longer be on our cap. Which put us in a position for the future that I looked at as follows:
2008-9 We have a full training camp with a very motivated JKidd, the rest of our starters back, an improving Nasty Bassty (had to get that in there), plus anyone we can add. We all thought (and still do), we would have a very strong nucleus to build around. We would also have a 1st round pick.
2009-10 Depending on how the previous season went, we would have several last year contracts available, the option of potentially having some cap room, and other options to improve the team. With the much lower potential salaries, not only could we use cap room if we went that direction to enable roster flexibility, we could also buy a pick. (there are almost always teams willing to sell a pick in the 20s for 3mm dollars)
2010-11 In this year we only have 2 fully guaranteed contracts and in 2011-12 We dint have any fully guaranteed contracts. So the options are endless in both years. Plus , its in one of these years that the new CBA comes up. With so little committed, depending on how hard a line the owners take, things could get very, very inter sting. Having so little contractually committed could be a great place to be when other teams look to dump salaries to avoid the risks of a lockout.
So with the changes, while we lost a great player in Devin, we felt like we were picking up someone who could spark the team and add energy on the court. At best, we re energized, at worst, it doesn’t help, but we have improved the roster flexibility for the future and improved our opportunity to re invent ourselves, just as we had after the 04 05 season. And as far as the draft picks, there is no question there is risk there, but in the NBA, there is always the option to buy low first round draft picks, so we weren’t as concerned there.
So from my my viewpoint, we accelerated salaries, which wasn’t cheap, and in exchange, got a player that our basketball people thought could energize and help us.
Now, that's actually even more sensible in light of the financial meltdown that transpired since the post... But then why did they turn around and re-sign Diop a few months later for 5 years, $32.4 million? Only to trade him to Charlotte for Carroll, who's on the books through at least 2012 for about $4.5 million per year? And that's not even considering the tax threshold, which a good-but-not-championship-caliber Mavs roster will be very close to with the $69 million they've already committed for next year. Did any of that make sense in the context of the post I quoted?
All of which is to say, I don't get Cuban sometimes. One second, he'll look like he's ahead of the market... and the next, he'll do something to undermine that position. However, IMO, the Kidd trade wasn't an horrendous misstep, certainly not on par with some of Phoenix's moves since 2007.
Back to top
View user's profile Send private message Visit poster's website
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Thu Mar 12, 2009 2:17 am Post subject: Reply with quote
Thanks for the Cuban quote I hadn't seen that.
Don't know the Cuban - Donnie Nelson dynamic well. Wonder who was the bigger fan of Diop / if there was disagreement and who liked Carroll more.
Some guys are used as currency and might be on the move again.
Hollins might be under-talked about and a bigger part of that deal in th elong-run. Still a mixed bag on boxscore stats but I see +10.5 on 2 yr adjusted, though that may be mostly from Charlotte. Excellent counterpart defense in Dallas. Diop's turned quite weak this season.
Back to top
View user's profile Send private message
BobboFitos
Joined: 21 Feb 2009
Posts: 190
Location: Cambridge, MA
PostPosted: Thu Mar 12, 2009 2:58 am Post subject: Reply with quote
Quote:
Now, that's actually even more sensible in light of the financial meltdown that transpired since the post... But then why did they turn around and re-sign Diop a few months later for 5 years, $32.4 million? Only to trade him to Charlotte for Carroll, who's on the books through at least 2012 for about $4.5 million per year? And that's not even considering the tax threshold, which a good-but-not-championship-caliber Mavs roster will be very close to with the $69 million they've already committed for next year. Did any of that make sense in the context of the post I quoted?
This is a good point that didn't make sense to me either. The Diop signing seemed like a gross misuse of the MLE and is oppositional logic as far as what Cuban said he wanted to do w/ his team.
_________________
-Rob
Back to top
View user's profile Send private message AIM Address
schtevie
Joined: 18 Apr 2005
Posts: 408
PostPosted: Mon Mar 16, 2009 9:20 am Post subject: Reply with quote
Let me return things now to the topic of retrodiction. Here are the results for APM and Wages of Wins retrodiction of the 2007-2008 season, using the 2006-2007 estimates of individual productivities and actual 2007-2008 season minutes.
For the APM retrodiction, the data come from Steve Ilardi's 82games article about the 2006-2007 season (which in turn imposes a one quarter weight for 2005-2006 APMs. As the results are on a per-minute basis, the exercise was to convert these based on average team pace to a per possession basis. For replacement level players (those not getting 400 minutes per season) I assumed a value of -7.
For the Wins Produced retrodiction, these come directly from Dave Berri's site.
To put the metrics on a common basis of predicted wins, I used a regression of the difference between offensive and defensive rating on actual 2007-2008 wins. The argument is not that this linear relationship provides the best fit, but it surely does a serviceable job of providing a common baseline for comparison.
So, the results:
For APM, the average number of wins retrodicted is 38.4 for WoW it is 35.9. That the APM comes close to 41 is not surprising, given that the process provides unbiased estimates of individual contributions. And if the actual 2006-2007 replacement value was slightly greater (less) than -7, the difference would be somewhat less (greater) still.
In terms of the average error and its dispersion, here WoW does better. An APM retrodiction of the 2007-2008 season yields an average error of 10.8 games, compared to 8.8 for WoW. Much of this difference is provided by the most extreme outliers. The standard deviations for APM and WoW are 13.2 and 9.1, respectively.
Before discussing these results, let me again encourage others again to add their retrodictions, the goal being a better informed conventional wisdom. Neil Paine suggests that statistical +/- (at least for this season) might yield an error around 6+ points. Let's see what others come up with.
Back to the story at hand.
As I tried to lead the conversation, before presenting results, it is the discussion of unexpected vs. expected errors which is, to me at least, perhaps the most interesting thing about this exercise in retrodiction. When retrodictions are nailed, if mitigating circumstances suggest that they actually failed, these should not be considered successes, and vice versa.
Let me try again to induce some commentary, focusing on the "failed" retrodictions of both APM and WoW. (Though the same discussion could take place with the "successes".
APM did a bad job, in order of absolute terribleness, of retrodicting the records of the following teams (I pick the worst five): (1) Washington, (2) Indiana, (3) Utah, (4) Portland, and (5) Minnesota. All it should be noted were underestimates.
By contrast WoW's "top" five are: (1) Portland, (2) Orlando, (3) Detroit, (4) Denver, and (5) Chicago. All of these except the last are underestimates, perhaps less surprising than with APM as WoW retrodictions are 5.1 wins short on average. And note that only Portland is a common egregious error.
In terms of the teams which provided the greatest difference between the retrodictions of the two approaches, these are: (1) Boston, (2) Orlando, (3) Detroit, (4) Philadelphia, and (5) Washington.
So, to put a more focused question to the floor, in terms of explaining the collective and individual failures listed here, what are the stories that you all have to tell about these aforementioned teams in 2007-08 as to how they performed vs. expectations?
In terms of unexpected outcomes, which teams were plagued by extraordinary number of minutes played by players with nagging injuries? Which teams had players with "unexpected" break-out seasons? Which teams seemed to have a disproportionate number of general age-related declines?
My hope is that the intersection of such informed interpretation and empirical methods might give us all a firmer grounding for our prior beliefs.
Back to top
View user's profile Send private message
erivera7
Joined: 19 Jan 2009
Posts: 178
Location: Chicago, IL
PostPosted: Mon Mar 16, 2009 1:24 pm Post subject: Reply with quote
schtevie wrote:
In terms of the teams which provided the greatest difference between the retrodictions of the two approaches, these are: (1) Boston, (2) Orlando, (3) Detroit, (4) Philadelphia, and (5) Washington.
So, to put a more focused question to the floor, in terms of explaining the collective and individual failures listed here, what are the stories that you all have to tell about these aforementioned teams in 2007-08 as to how they performed vs. expectations?
In terms of unexpected outcomes, which teams were plagued by extraordinary number of minutes played by players with nagging injuries? Which teams had players with "unexpected" break-out seasons? Which teams seemed to have a disproportionate number of general age-related declines?
My hope is that the intersection of such informed interpretation and empirical methods might give us all a firmer grounding for our prior beliefs.
I can speak about the Magic.
Last year, the "surprises" that occurred for Orlando were two main things, a.) a career-year from Hedo Turkoglu and b.) an improved defensive squad, despite the fact Rashard Lewis "became" a power forward and the team ratcheted up the pace [25th in '07, 9th in '08]. Not really a surprise, but a trend was that the Magic stayed relatively healthy throughout the year. No one on the squad, except for Tony Battie, suffered a serious injury (which would have required substantial time in street clothes). Due to these two main factors, Orlando was able to become a 50+ win squad after many before the preseason, surmised that the team would win roughly 43-47 games.
_________________
@erivera7
I cover the Orlando Magic - Magic Basketball
Back to top
View user's profile Send private message
Ilardi
Joined: 15 May 2008
Posts: 262
Location: Lawrence, KS
PostPosted: Mon Mar 16, 2009 2:01 pm Post subject: Reply with quote
Schtevie,
The value of a replacement player for that 06-07 season was -3.82; that's a lot lower than the -7.0 value you used. I suspect this discrepancy could account for at least part of the APM model's less-than-stellar retrodiction. But there are at least three other issues:
1) The APM estimates from that 06-07 article were still pretty noisy, since they only had the benefit of 2 seasons' worth of data to push down standard errors. The APM estimates for 07-08 (i.e., the ones Aaron and I published) are dramatically less noisy (using 5 seasons' worth of data), and should provide much better retrodiction if you're willing to repeat this exercise at the conclusion of the 08-09 season next month.
2) Individual player APM values display a predictable year-to-year pattern of improvement/decline based on player age, position, experience, and injury. I haven't yet refined this predictive model - it's nonlinear and tricky - but hope to have it completed soon. Once I've got it, you're welcome to use it to adjust each 07-08 player value for use in the 08-09 retrodiction study. (In fairness, though, you should also apply a similar procedure to the WoW estimates.)
3) Finally, rookies should probably be treated as special cases - with values that will be (on average) higher than that of low-minute replacement players, but nevertheless lower than the overal league average. When I have a chance, I'll try to come up with a rookie APM forecast algorithm based on draft position, and will pass it along once it's completed . . .
ssi
schtevie wrote:
Let me return things now to the topic of retrodiction. Here are the results for APM and Wages of Wins retrodiction of the 2007-2008 season, using the 2006-2007 estimates of individual productivities and actual 2007-2008 season minutes.
For the APM retrodiction, the data come from Steve Ilardi's 82games article about the 2006-2007 season (which in turn imposes a one quarter weight for 2005-2006 APMs. As the results are on a per-minute basis, the exercise was to convert these based on average team pace to a per possession basis. For replacement level players (those not getting 400 minutes per season) I assumed a value of -7.
For the Wins Produced retrodiction, these come directly from Dave Berri's site.
To put the metrics on a common basis of predicted wins, I used a regression of the difference between offensive and defensive rating on actual 2007-2008 wins. The argument is not that this linear relationship provides the best fit, but it surely does a serviceable job of providing a common baseline for comparison.
So, the results:
For APM, the average number of wins retrodicted is 38.4 for WoW it is 35.9. That the APM comes close to 41 is not surprising, given that the process provides unbiased estimates of individual contributions. And if the actual 2006-2007 replacement value was slightly greater (less) than -7, the difference would be somewhat less (greater) still.
In terms of the average error and its dispersion, here WoW does better. An APM retrodiction of the 2007-2008 season yields an average error of 10.8 games, compared to 8.8 for WoW. Much of this difference is provided by the most extreme outliers. The standard deviations for APM and WoW are 13.2 and 9.1, respectively.
Before discussing these results, let me again encourage others again to add their retrodictions, the goal being a better informed conventional wisdom. Neil Paine suggests that statistical +/- (at least for this season) might yield an error around 6+ points. Let's see what others come up with.
Back to the story at hand.
As I tried to lead the conversation, before presenting results, it is the discussion of unexpected vs. expected errors which is, to me at least, perhaps the most interesting thing about this exercise in retrodiction. When retrodictions are nailed, if mitigating circumstances suggest that they actually failed, these should not be considered successes, and vice versa.
Let me try again to induce some commentary, focusing on the "failed" retrodictions of both APM and WoW. (Though the same discussion could take place with the "successes".
APM did a bad job, in order of absolute terribleness, of retrodicting the records of the following teams (I pick the worst five): (1) Washington, (2) Indiana, (3) Utah, (4) Portland, and (5) Minnesota. All it should be noted were underestimates.
By contrast WoW's "top" five are: (1) Portland, (2) Orlando, (3) Detroit, (4) Denver, and (5) Chicago. All of these except the last are underestimates, perhaps less surprising than with APM as WoW retrodictions are 5.1 wins short on average. And note that only Portland is a common egregious error.
In terms of the teams which provided the greatest difference between the retrodictions of the two approaches, these are: (1) Boston, (2) Orlando, (3) Detroit, (4) Philadelphia, and (5) Washington.
So, to put a more focused question to the floor, in terms of explaining the collective and individual failures listed here, what are the stories that you all have to tell about these aforementioned teams in 2007-08 as to how they performed vs. expectations?
In terms of unexpected outcomes, which teams were plagued by extraordinary number of minutes played by players with nagging injuries? Which teams had players with "unexpected" break-out seasons? Which teams seemed to have a disproportionate number of general age-related declines?
My hope is that the intersection of such informed interpretation and empirical methods might give us all a firmer grounding for our prior beliefs.
Back to top
View user's profile Send private message
Mountain
Joined: 13 Mar 2007
Posts: 1527
PostPosted: Mon Mar 16, 2009 2:04 pm Post subject: Reply with quote
All five of APMs unders were bottom half on crude unweighted average team age. Past season APM data might miss the learning improvement and might need an adjustment for that.
3 of WOWs 4 underestimates were in oldest 11.
The biggest differences were (1) Boston, (2) Orlando, (3) Detroit, (4) Philadelphia, and (5) Washington and they represent 5 of the top 7 offense in the east. And 4 of the top 5 in east on defense. But how do the methods vary- consistently one higher or scattered? Probably need a strength of schedule adjustment and the elite east teams I assume would have gone up quite a bit from that last season.
I checked a few other things quickly- home/road, playoff/lottery opponent, 3 pt game. turnovers, fouls shots but didn't find any other obvious trend.
Back to top
View user's profile Send private message
schtevie
Joined: 18 Apr 2005
Posts: 408
PostPosted: Mon Mar 16, 2009 2:37 pm Post subject: Reply with quote
Steve,
Thanks for chiming in. I presume the -3.82 is also on the per 40 minute basis and not a recalculated per possession basis? I will modify the results and repost accordingly.
Regarding your three points:
1) I share your belief that an upcoming retrodiction based on five years of data would be less noisy. Interestingly however, I performed the same exercise with one year '05-'06 data as well, and it didn't seem to improve the retrodictive quality. Regarding who should do the '08-'09 retrodiction, might I nominate you and Aaron? You got the data sitting there all ready to go....
2) Has there ever been a doubt that incorporating the effects of age, injury, etc. wouldn't improve the predictive quality of the model? I note that the residuals from the retrodictions that I reported are significantly related to minute-weighted roster ages, both in terms of size and statistical significance (not the case with WoW residuals, by contrast). Sorting this out at the player level will matter.
3) I look forward to the rookie retrodiction, but given the typically low minutes played by this cohort, they shouldn't drive the results of this exercise.
Thanks again.
Steve
Back to top
View user's profile Send private message
gabefarkas
Joined: 31 Dec 2004
Posts: 1313
Location: Durham, NC
PostPosted: Mon Mar 16, 2009 4:26 pm Post subject: Reply with quote
schtevie wrote:
So, to put a more focused question to the floor, in terms of explaining the collective and individual failures listed here, what are the stories that you all have to tell about these aforementioned teams in 2007-08 as to how they performed vs. expectations?
In terms of unexpected outcomes, which teams were plagued by extraordinary number of minutes played by players with nagging injuries? Which teams had players with "unexpected" break-out seasons? Which teams seemed to have a disproportionate number of general age-related declines?
I think you've answered your own question, at least partly. I think age has a lot to do with it. Portland was under-retrodicted because their key players are all approaching what is usually their peak production age.
Back to top
View user's profile Send private message Send e-mail AIM Address
Ilardi
Joined: 15 May 2008
Posts: 262
Location: Lawrence, KS
PostPosted: Mon Mar 16, 2009 4:44 pm Post subject: Reply with quote
Steve,
Yes, that -3.82 replacement value is a per-40-minute metric.
Also, to echo gabefarkas' point, it looks to me like 3 of the worst fits for the APM retrodiction - Portland, Minnesota, Utah - were young teams that saw major improvement for key personnel.
The other 2 outliers - Washington and Indiana - suffered major injury to key starters (Arenas on the former, O'Neal and Tinsley for the latter), and this may have introduced estimation bias via some previously undetected interaction effects (wherein, for example, players like Jamison and Butler exhibited greater APM value when playing without Arenas, since they had more significant roles on the team.) Presumably, such estimation challenges could be addressed to some extent through the use of 5-year APM estimates (with more varied lineups, which - on average - could help some interaction effects cancel each other out) and (in future) more explicit modeling of interactions.
- ssi
schtevie wrote:
Steve,
Thanks for chiming in. I presume the -3.82 is also on the per 40 minute basis and not a recalculated per possession basis? I will modify the results and repost accordingly.
Regarding your three points:
1) I share your belief that an upcoming retrodiction based on five years of data would be less noisy. Interestingly however, I performed the same exercise with one year '05-'06 data as well, and it didn't seem to improve the retrodictive quality. Regarding who should do the '08-'09 retrodiction, might I nominate you and Aaron? You got the data sitting there all ready to go....
2) Has there ever been a doubt that incorporating the effects of age, injury, etc. wouldn't improve the predictive quality of the model? I note that the residuals from the retrodictions that I reported are significantly related to minute-weighted roster ages, both in terms of size and statistical significance (not the case with WoW residuals, by contrast). Sorting this out at the player level will matter.
3) I look forward to the rookie retrodiction, but given the typically low minutes played by this cohort, they shouldn't drive the results of this exercise.
Thanks again.
Steve
Back to top
View user's profile Send private message
Mike G
Joined: 14 Jan 2005
Posts: 3558
Location: Hendersonville, NC
PostPosted: Mon Mar 16, 2009 5:34 pm Post subject: Reply with quote
http://sonicscentral.com/apbrmetrics/vi ... 5&start=45
Last year, not retrodicting but predicting, Mountain was off by an average of 8.4 wins; Neil (daviswylie), Mike G, and John H were off by 9.2-9.5 .
We were all within 5 W on the Wolves. Only Neil was worse than 4 off on Utah. Blazers surprised everyone by 15-22 extra wins.
Portland's big unpredictable boost came from the great rookie season by Roy. Little was expected of James Jones, yet he produced. Only Rodriguez and Jack regressed, of their whole roster.
_________________
`
36% of all statistics are wrong
Author Message
schtevie
Joined: 18 Apr 2005
Posts: 182
PostPosted: Tue Mar 17, 2009 10:52 am Post subject: Reply with quote
Let me repost the APM retrodiction results, based on replacement value being -3.82 points per 40 minutes as opposed to -7 points per 100 possessions. It doesn't improve things much, but it does a little. The average team wins increases from 38.4 to 39.1, so a little less "unbiased". The average error decreases from 10.8 to 10.5. And the standard deviation of these errors decreases from 13.2 to 13.0. So, there is still some explaining to do with the outliers.
In terms of the ranking of the teams that had the largest retrodiction errors, this did not change.
This said, let me reply to a variety of comments.
Thanks erivera7 for the observations on Orlando. The lack of nagging injuries comment is of interest. However, I don't get the defensive improvement story. They actually got a bit worse in '07-'08, but their offense was much improved. In terms of how APM fared vs. WoW, this was an instance where APM nailed it, with an error of just +0.3 wins with respect to the predicted wins figure. WoW by contrast underpredicted by 18.4 wins. which was "explained" by underestimates of multiple player contributions. By contrast, the APM story is essentially that the improvement over the previous year was more than explained by the rise of Dwight Howard, with remaining players, Turkoglu included, falling short of retrodicted expectations. Significant negative contributions obtained to Nelson and Arroyo and smaller positive contributions than expected obtained to Turkoglu and Lewis.
Which of these stories is more plausible is what I am trying to get at with this discussion.
On to MikeG.
Mike, does your prediction/retrodiction method generated exclusively by a straight formula or are such results edited? I am looking to compare empirical methods first and foremost, not expert interpretation thereof. So, please offer up a retrodiction using actual '07-'08 minutes. And by all means, include also a retrodiction for whatever method you have subsequently come to advocate for age adjustments.
On to Gabe.
What I am interested in advancing is the notion of an improved statistics-based conventional wisdom. Everyone, I think, would agree on the general factors that account for deviations from expected performance. In the fullness of time, some of these factors will be formally incorporated into the models. But for others, interpretation of results will remain key. So, for example, take the just discussed case of Orlando. If nagging injuries were not their affliction last year, score a tentative point for APM vs. WoW in determining how we should think about the world. Then add all other methods into the mix as well. Does anyone know how PER works in relation to such matters?
Finally, as I tend to pay attention to Boston, essentially to the exclusion of other clubs, and that the performance of Boston last year generated the biggest outlier in the comparison of APM and WoW, let me rephrase some previously expressed commentary in the current context.
Retrodicting with just '06-'07 data (with the caveat that the APM numbers are partially based on the previous season as well) APM overpredicts the "actual" win total by 10.7 games whereas WoW underpredicts it by 14.6 games, a 25.3 game retrodiction gap. WoW accounts for its underestimates broadly, with essentially only Tony Allen doing worse that retrodicted. APM by contrast finds the two biggest "underachievers" as being Kevin Garnett and Ray Allen - a story which is consistent with a nagging injury explanation (never mind the aging factor). I recall that contemporaneous APM numbers showed a decline corresponding to his injuries last year (perhaps Aaron has archives to back up this claim?) and I wish I had had the time to pester Ray Allen at the MIT event to ask him how he perceived his relative effectiveness between last season and this.
Anyway, here too I would score one - a three pointer in fact - for APM, but I explicitly note that its howlers have yet to be discussed in depth. Does anyone have specific commentary to offer about: (1) Washington, (2) Indiana, (3) Utah, (4) Portland, and (5) Minnesota?
Back to top
View user's profile Send private message
Ilardi
Joined: 15 May 2008
Posts: 105
Location: Lawrence, KS
PostPosted: Tue Mar 17, 2009 4:22 pm Post subject: Reply with quote
schtevie wrote:
Does anyone have specific commentary to offer about: (1) Washington, (2) Indiana, (3) Utah, (4) Portland, and (5) Minnesota?
I wonder if you missed this comment (below) from earlier in the thread?
Quote:
Also, to echo gabefarkas' point, it looks to me like 3 of the worst fits for the APM retrodiction - Portland, Minnesota, Utah - were young teams that saw major improvement for key personnel.
The other 2 outliers - Washington and Indiana - suffered major injury to key starters (Arenas on the former, O'Neal and Tinsley for the latter), and this may have introduced estimation bias via some previously undetected interaction effects (wherein, for example, players like Jamison and Butler exhibited greater APM value when playing without Arenas, since they had more significant roles on the team.) Presumably, such estimation challenges could be addressed to some extent through the use of 5-year APM estimates (with more varied lineups, which - on average - could help some interaction effects cancel each other out) and (in future) more explicit modeling of interactions.
The basic gist is that the 3 young teams saw great improvement in numerous starters/rotation guys (and perhaps also above-average APM contribution from rookies), while the other 2 outliers (Washington & Indiana) had major injuries to key starters, possibly unmasking player interaction effects that led remaining non-injured personnel to "over-perform" their historic APM numbers that season.
Back to top
View user's profile Send private message
erivera7
Joined: 19 Jan 2009
Posts: 52
PostPosted: Wed Mar 18, 2009 3:27 am Post subject: Reply with quote
schtevie wrote:
Thanks erivera7 for the observations on Orlando. The lack of nagging injuries comment is of interest. However, I don't get the defensive improvement story. They actually got a bit worse in '07-'08, but their offense was much improved. In terms of how APM fared vs. WoW, this was an instance where APM nailed it, with an error of just +0.3 wins with respect to the predicted wins figure. WoW by contrast underpredicted by 18.4 wins. which was "explained" by underestimates of multiple player contributions. By contrast, the APM story is essentially that the improvement over the previous year was more than explained by the rise of Dwight Howard, with remaining players, Turkoglu included, falling short of retrodicted expectations. Significant negative contributions obtained to Nelson and Arroyo and smaller positive contributions than expected obtained to Turkoglu and Lewis.
Which of these stories is more plausible is what I am trying to get at with this discussion.
If you want to be technical, the Magic got "worse", yeah .. when looking at Orlando's defensive efficiency in 2007 (6th, 104.1) compared to 2008 (5th, 105.5). However, the reason why the team did improve (in my eyes) is because they sped up the tempo last year. So even though by statistical technicality, the Magic appeared worse on paper, they were better and more impressive defensively given the increased pace. Add the fact head coach Stan Van Gundy played Rashard Lewis at the power forward position, and I feel given the totality of the situation, Orlando improved just a smidgen on defense (in my opinion).
Obviously the main reason why the Magic were a better team last year was, more so, due to the offense becoming a juggernaut statistically. There's no doubt about that (I was alluding to that point when I was referring to Hedo Turkoglu's career year last season).
_________________
Contributing Editor
Third Quarter Collapse (An Orlando Magic blog)
Back to top
View user's profile Send private message
Mike G
Joined: 14 Jan 2005
Posts: 2363
Location: Delphi, Indiana
PostPosted: Wed Mar 18, 2009 9:08 am Post subject: Reply with quote
OK, this was a godawful amount of work, but obviously a useful exercise. After changing my 'predictions' spreadsheet to account for minutes and trades, my results are worse. Average error from Pythagorean-expected is 10.7 (worse than from actual 2008 wins, 10.1).
I had predicted the Wiz to win 39.5, and their Pyth was 40.2 -- pretty good, except that it was all luck. Knowing the actual minutes, I'd have predicted 23 wins. In order of minutes played, postdicted and actual eWins, and eW/1000 (average is about 2.0 ) :
Code:
eWPre eWAct eW/1000 min : 2007 2008
9.0 9.9 Jamison,Antawn 2.94 3.24
3.7 3.3 Stevenson,Deshaw 1.45 1.29
6.2 6.9 Butler,Caron 2.66 3.00
3.8 5.4 Haywood,Brendan 1.72 2.43
2.5 2.8 Daniels,Antonio 1.14 1.29
.1 3.0 Mason,Roger .04 1.76
2.0 3.8 Blatche,Andray 1.18 2.26
1.5 2.1 Songaila,Darius .99 1.37
.7 1.8 Young, Nick .60 1.52
.3 .2 McGuire, Dominic .40 .30
1.7 1.2 Arenas,Gilbert 3.99 2.75
.2 .5 Pecherov, O .50 1.49
Jamison and Butler improve nicely but not dramatically. Hugely improved are Haywood and Blatche. Little to nothing were expected from Mason and Young.
Total postdicted eWins were 32, and 41 were delivered. Expected wins, though, are twice as disparate.
xW = eW*2 - 41 = 23.
It's a two-edged sword.
_________________
`
There's no I in analysys.
Back to top
View user's profile Send private message Send e-mail
schtevie
Joined: 18 Apr 2005
Posts: 182
PostPosted: Wed Mar 18, 2009 12:57 pm Post subject: Reply with quote
Steve,
Sorry, I hadn't intended to leave the impression that your summary was being ignored. I was just trying to solicit commentary from folks who follow those teams listed with special passion. In particular, I am interested in testimony that speaks to the issue of injuries, of two types. Injuries that upon returning to the line-up left the player less productive than before and injuries which were more of the nagging type, with similar expected effects on productivity.
I am of a belief that these factors are very significant, at least in APM land. For example, the single biggest factor accounting for the wretchedness of the APM retrodiction of the Wizards had to do with the contribution of Antawn Jamison. Were his 2005-06 APM estimate to have been used instead, this would have eliminated a huge chunk of the retrodiction error. Was 2006-07 anomalous because of a noisy estimate, or were there relevant injuries? He did miss all of February 2007 with a sprained left knee. Was the estimated APM as of the end of January, higher than at season's end? And so on. And Caron Butler too had anomalously crappy 2006-07 according to APM. His injury report shows a series of injuries starting at the end of February 2007, beginning with back spasms. Perhaps these are significant.
Before interaction effects are addressed, I would first be curious what the residual is upon dealing with the more obvious fundamentals of age and injury (not that identifying all relevant injuries is that easy to do). Alchemy should follow.
Mike,
Thanks for the numbers. To make sure I understand, you are saying that your method retrodicts on average 32 out of 41 wins? Also, what is the standard deviation of the errors? And finally, which teams produced the five worst retrodictions?
Back to top
View user's profile Send private message
Kevin Pelton
Site Admin
Joined: 30 Dec 2004
Posts: 828
Location: Seattle
PostPosted: Wed Mar 18, 2009 1:49 pm Post subject: Reply with quote
schtevie, perhaps I am misunderstanding, but what makes you think injury and interaction effects are more problematic for an adjusted plus-minus model than for those using box-score stats? The former would seem to affect both equally, and aren't the interaction effects in large part what we're seeking to tease out with adjusted plus-minus?
The question here isn't how well adjusted plus-minus predicts team performance. It's how well it predicts team performance vis-a-vis box-score stat methods, or how well various forms of adjusted plus-minus predict relative to each other, which is an interesting test of the practical viability of combining seasons.
Getting away from this central question tends to distract from the issue, I think.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jkubatko
Joined: 05 Jan 2005
Posts: 668
Location: Columbus, OH
PostPosted: Wed Mar 18, 2009 1:55 pm Post subject: Reply with quote
I guess I'll throw in some results as well. I decided to retrodict the 1979-80 through 2007-08 seasons using Win Shares. The rules:
* The player's minutes played total for the retrodicted season is known.
* The player's rate of Win Shares per minute played from his last season with at least 250 minutes was used as his predicted rate of production.
* If the player was a rookie, then a rate of 0.00134 WS/MP was used (that value is the average rate for all rookies during the study's time period).
* If the player was not a rookie and he did not play at least 250 minutes in any previous season, then he was given a value of 0. In other words, rather than try to fudge some replacement-level value, I decided to err on the side of caution.
This is mainly going to be a data dump without commentary, but let me state that I was pleasantly surprised with the results.
First, here are the results for each season individually:
Code:
+---------+----------+-------------+-------------+
| year_id | avg_wins | avg_abs_err | std_dev_err |
+---------+----------+-------------+-------------+
| 1980 | 40.56 | 8.06 | 10.48 |
| 1981 | 39.16 | 7.94 | 8.93 |
| 1982 | 40.84 | 6.46 | 8.29 |
| 1983 | 40.74 | 4.93 | 6.27 |
| 1984 | 42.04 | 3.96 | 5.15 |
| 1985 | 39.03 | 7.72 | 9.16 |
| 1986 | 40.23 | 3.82 | 5.40 |
| 1987 | 40.74 | 6.57 | 7.80 |
| 1988 | 40.90 | 5.59 | 6.73 |
| 1989 | 39.83 | 7.44 | 9.16 |
| 1990 | 40.07 | 6.58 | 8.14 |
| 1991 | 40.48 | 4.53 | 6.15 |
| 1992 | 41.10 | 7.41 | 8.54 |
| 1993 | 41.28 | 7.25 | 8.87 |
| 1994 | 39.46 | 6.69 | 8.54 |
| 1995 | 41.16 | 7.32 | 9.34 |
| 1996 | 40.96 | 5.51 | 7.22 |
| 1997 | 40.64 | 7.05 | 8.85 |
| 1998 | 41.85 | 8.21 | 10.19 |
| 1999 | 25.60 | 4.42 | 5.38 |
| 2000 | 40.80 | 6.63 | 8.64 |
| 2001 | 40.99 | 5.76 | 7.29 |
| 2002 | 41.27 | 5.84 | 7.44 |
| 2003 | 40.80 | 6.10 | 7.91 |
| 2004 | 40.81 | 6.41 | 7.91 |
| 2005 | 40.13 | 7.57 | 9.62 |
| 2006 | 40.86 | 6.73 | 7.94 |
| 2007 | 41.08 | 6.02 | 7.86 |
| 2008 | 41.49 | 8.30 | 10.13 |
+---------+----------+-------------+-------------+
Here are the cumulative results:
Code:
+----------+----------+----------+-------------+-------------+
| year_min | year_max | avg_wins | avg_abs_err | std_dev_err |
+----------+----------+----------+-------------+-------------+
| 1980 | 2008 | 40.15 | 6.46 | 8.07 |
+----------+----------+----------+-------------+-------------+
The ten biggest misses:
Code:
+---------+---------+------+-------+--------+
| year_id | team_id | wins | pred | error |
+---------+---------+------+-------+--------+
| 1980 | BOS | 61 | 36.67 | 24.33 |
| 2005 | PHO | 62 | 38.05 | 23.95 |
| 2008 | MIA | 15 | 38.16 | -23.16 |
| 1990 | SAS | 56 | 33.03 | 22.97 |
| 1998 | DEN | 11 | 33.68 | -22.68 |
| 1989 | PHO | 55 | 34.25 | 20.75 |
| 2008 | BOS | 66 | 45.35 | 20.65 |
| 1982 | CLE | 15 | 35.35 | -20.35 |
| 1998 | TOR | 16 | 36.12 | -20.12 |
| 2000 | LAL | 67 | 46.96 | 20.04 |
+---------+---------+------+-------+--------+
The ten closest calls:
Code:
+---------+---------+------+-------+-------+
| year_id | team_id | wins | pred | error |
+---------+---------+------+-------+-------+
| 1993 | LAL | 39 | 38.97 | 0.03 |
| 1986 | UTA | 42 | 41.95 | 0.05 |
| 1984 | MIL | 50 | 50.07 | -0.07 |
| 1989 | BOS | 42 | 42.08 | -0.08 |
| 2005 | SAS | 59 | 59.09 | -0.09 |
| 2003 | POR | 50 | 49.90 | 0.10 |
| 1981 | DET | 21 | 20.90 | 0.10 |
| 1991 | ATL | 43 | 42.88 | 0.12 |
| 1984 | DAL | 43 | 43.13 | -0.13 |
| 1983 | SEA | 48 | 48.14 | -0.14 |
+---------+---------+------+-------+-------+
_________________
Regards,
Justin Kubatko
Basketball-Reference.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Mike G
Joined: 14 Jan 2005
Posts: 2363
Location: Delphi, Indiana
PostPosted: Wed Mar 18, 2009 5:21 pm Post subject: Reply with quote
schtevie wrote:
... you are saying that your method retrodicts on average 32 out of 41 wins? Also, what is the standard deviation of the errors? And finally, which teams produced the five worst retrodictions?
5 worst:
Code:
tm retro W PythW PyEr
Mia 43.3 15 19.0 -24.3
Por 20.2 41 38.3 18.1
Chi 50.2 33 32.7 -17.5
Was 23.3 43 40.2 16.9
Cha 46.2 32 29.4 -16.8
In our season-predicting threads of the last 2 seasons, I've gone by actual wins; but in this thread, I'm ranking by pythagorean.
I postdicted 32 eWins for the Wiz, knowing their minutes. It was just my sense of irony that prompted me to report that hindsight (of player minutes) led to a much-worse estimate of their win rate.
On average, 2007 eW/min * min postdict an average error of 10.7 wins per team.
_________________
`
There's no I in analysys.
Last edited by Mike G on Thu Mar 19, 2009 10:29 am; edited 1 time in total
Back to top
View user's profile Send private message Send e-mail
schtevie
Joined: 18 Apr 2005
Posts: 182
PostPosted: Wed Mar 18, 2009 6:12 pm Post subject: Reply with quote
In order:
Kevin,
I don't have particularly strong beliefs that injury and interaction effects are more problematic for an adjusted plus-minus model than for those using box-score stats. If the argument is that APM picks up all the box score stuff and also all the other little things, then there is a reason to believe that injuries matter more in APMville. If Ray Allen's notionally tender ankles affected his jumpshot last year, they probably also affected his determination to fight through picks, etc.
As for the interaction effects, I don't think that this is what we're seeking to tease out with adjusted plus-minus per se. If they are real and significant (in both senses of the term) by all means lets find them, by whatever approach. However, we have a simple theory that age matters, and a not unrelated theory that injury matters. Interaction effects can be thought of as a wordier way of saying "chemistry" which always has been a way of invoking the residual. My point is let's see what is left to be explained after we account for "real" things.
Justin,
Thank you for the Win Shares results. It was interesting to see that 2007-08 was the worst year "ever" for the retrodiction business.
It is perhaps not surprising that this method does well on the important criterion of actually including all wins, on average, given that it is based on Offensive and Defensive Rating. That Boston's performance was completely unanticipated, however, caught me a bit by surprise. I would have though that an approach, again, based on Offensive and Defensive Ratings would have done better.
Ex post, what accounts for the failed retrodiction in this instance? That is, who performed worse than expected?
Finally, would you also share the five worst failures of this method. I think that comparing the howlers is potentially very informative. I note that Miami was retrodicted by APM to have an essentially identical record - 37.4 - as what Win Shares came up with - 38.16. Strange, it seems, that Boston and Miami should perform so differently between the two approaches. Or perhaps not.
Mike,
I am still a bit confused about what your method predicted in terms of average wins overall. Wages of Wins left 5.1 wins on the table, on average, per team. APM 2.0 wins. Win Shares offered 0.49 wins too many. And your method?[/i]
Back to top
View user's profile Send private message
Mike G
Joined: 14 Jan 2005
Posts: 2363
Location: Delphi, Indiana
PostPosted: Thu Mar 19, 2009 5:59 am Post subject: Reply with quote
Ah, gotcha. The average '08 record was 39.2 wins predicted.
So expected wins were 1.8 short of 41 . Equivalent (eWins) were 0.9 short. Team minutes were all above 19,000, so some 98% accounted-for.
Every Heat player had a bad year in '08. RDavis, he of 2960 minutes, was but 61% of his 2007 self; by himself creating 6 fewer wins than predicted. Other shortfalls: Wade 8.3, JWill 3.3, Shaq 3.2, Haslem 2.5, Blount 1.5, Marion 1.4 . No gainer over 1.8 (Quinn). These add up (or down) dramatically.
Among 8-10 players, expect half to improve and half to regress. When >3/4 of a lineup goes one way, that's unpredictable.
_________________
`
There's no I in analysys.
Back to top
View user's profile Send private message Send e-mail
jkubatko
Joined: 05 Jan 2005
Posts: 668
Location: Columbus, OH
PostPosted: Thu Mar 19, 2009 1:01 pm Post subject: Reply with quote
schtevie wrote:
It is perhaps not surprising that this method does well on the important criterion of actually including all wins, on average, given that it is based on Offensive and Defensive Rating. That Boston's performance was completely unanticipated, however, caught me a bit by surprise. I would have though that an approach, again, based on Offensive and Defensive Ratings would have done better.
Ex post, what accounts for the failed retrodiction in this instance? That is, who performed worse than expected?
I assume you mean who performed *better* than expected for Boston, not who performed worse. Offensively, the youngsters (Rondo, Perkins, and Powe) all took big steps forward, and Garnett was much more efficient in '08 than he had been in '09, but obviously the biggest changes were on the defensive end.
Quote:
Finally, would you also share the five worst failures of this method. I think that comparing the howlers is potentially very informative. I note that Miami was retrodicted by APM to have an essentially identical record - 37.4 - as what Win Shares came up with - 38.16. Strange, it seems, that Boston and Miami should perform so differently between the two approaches. Or perhaps not.
I did share the ten worst failures of this method, but I'm guessing you mean the five worst failure of 2007-08. Here they are:
Code:
+---------+---------+------+-------+--------+
| year_id | team_id | wins | pred | error |
+---------+---------+------+-------+--------+
| 2008 | MIA | 15 | 38.16 | -23.16 |
| 2008 | BOS | 66 | 45.35 | 20.65 |
| 2008 | CHI | 33 | 51.84 | -18.84 |
| 2008 | NYK | 23 | 37.52 | -14.52 |
| 2008 | NOH | 56 | 42.66 | 13.34 |
+---------+---------+------+-------+--------+
_________________
Regards,
Justin Kubatko