Wins Produced?!

[Guy]

Also, my research is similar, Guy:

Thanks, Daniel. As you probably know, Berri rejects any analysis that evaluates metrics in terms of their correlation with another model -- a point he repeats in the linked thread. Unfortunately, that is probably a fairly convincing argument to the WOW followers: why should we evaluate a model in terms of its ability to predict some other model (which has its own limitations)? My guess is that Berri himself knows that PM models should, in the aggregate, provide an unbiased estimate of player value -- and thus is being disingenous here -- but perhaps he really doesn't understand the point. But either way, it's certain that most non-statisticians (including all of Berri's followers) will never grasp this point.

So I think the best way to educate people about why WP doesn't truly "predict 95% of wins" is by showing that other metrics can better predict what a given lineup will do (as Mystic has) or what a team will do next year. After all, WP's claim to fame is predicting wins and point differential (not RAPM). Perhaps at least some of its fans will reconsider if they learn that it cannot in fact do what it claims.

[mystic]

Interesting. Did you use same season data for the metrics (i.e. are players' 2007 WP being used to project the performance of a 2007 lineup)? I suspect the spread in your correlations would be even higher -- to WP's disadvantage -- had you used players' prior-season ratings on the metrics.

Indeed, I used the data from the same season. And given the results of retrodiction tests with previous season data, I suspect you are correct. Maybe I take the time and check that.

As you probably know, WP also does a relatively poor job of predicting next-season team wins ("Sport Skeptic" looked at this a year or so ago).

Yeah, I was somewhat engaged in the discussion about it here on APBR and made some comments on the Sport Skeptic blog. Especially about using the rookie data from the respective season in the prediction, which made WP48 look closer a bit to the rest.

[Guy]

Daniel: As it happens, Neil Paine reported the results of your analysis over at WOW in the thread linked here. It elicited this response from "reservoirgod," one of the WOW regulars (who runs the Miami Heat Index site):

I burst out laughing when I read this:
“The correlation between Win Shares and Regularized Adjusted Plus/Minus (which, for all of its noise on an individual basis, does begin to separate players by their influence on the game when taken in the aggregate) is 0.531, far greater than the correlation for Wins Produced (0.380).”
That’s PRICELESS…

Unfortunately, this is the kind of statistical ignorance you are up against. Admittedly, there is probably no evidence that would ever change reservoirgod's view -- his position is faith-based, not evidence-based -- but I suspect his reaction here would be shared even by those with an open mind. I think it would be somewhat harder for them to reject evidence that WP fails to predict future differentials/wins by lineups and teams....

[xkonk]

To make a slight clarification, APM provides unbiased estimates in aggregate. RAPM provides, by definition, biased estimates. Correlating other measures with RAPM might demonstrate the ability of those metrics to similarly be relatively better at predicting out of sample but relatively worse at fitting the existent data.

[DSMok1]

To make a slight clarification, APM provides unbiased estimates in aggregate. RAPM provides, by definition, biased estimates. Correlating other measures with RAPM might demonstrate the ability of those metrics to similarly be relatively better at predicting out of sample but relatively worse at fitting the existent data.

At the large sample size I was using (8-year equally-weighted RAPM -- the source is here: http://stats-for-the-nba.appspot.com/dsmok_edition ), APM and RAPM converge, at least for any variables with significant data over that time period. I used a cutoff for the correlation of only players with 3000 possessions played over those 8 years (633 total players); therefore, APM and RAPM would be effectively the same.

[Crow]

I posted a few things to the discussion at WOW. Not totally surprising one post was deleted.


NBA Geek said to another poster: "Read the FAQ first. “WP Overvalues rebounds” was annoying to deal with when it was 2008. When you repeat it now, you just look ignorant."

I replied something like... make sure to also read the 2011 post officially significantly reducing the defense rebound weight.

That was highlighting fact, not expressing opinion. But I guess that fact was judged to embarrass one or more WOW actor.

[J.E.]

From an older thread, because it's a great quote

He is basically a crazy person, ranting at people who haven't heard his spiel before. Would you stand next to a crazy street prophet, explaining to passer-by that the end is not actually nigh? Anybody who buys into DB's whole argument will be able to reason their way out of it eventually, I don't think it's up to us to help them out and at times that can be counter-productive.

[xkonk]

At the large sample size I was using (8-year equally-weighted RAPM -- the source is here: http://stats-for-the-nba.appspot.com/dsmok_edition ), APM and RAPM converge, at least for any variables with significant data over that time period. I used a cutoff for the correlation of only players with 3000 possessions played over those 8 years (633 total players); therefore, APM and RAPM would be effectively the same.

Have you checked how closely they converge at any smaller time period?

[DSMok1]

At the large sample size I was using (8-year equally-weighted RAPM -- the source is here: http://stats-for-the-nba.appspot.com/dsmok_edition ), APM and RAPM converge, at least for any variables with significant data over that time period. I used a cutoff for the correlation of only players with 3000 possessions played over those 8 years (633 total players); therefore, APM and RAPM would be effectively the same.

Have you checked how closely they converge at any smaller time period?

I have not done a full-fledged study on the issue, no.

[wilq]

As an outside observer with no horse in the race I find debate between WP and [opponents] fascinating and entertaining.

Both sides are convinced their way is the only right way.
Both sides act like on the other one are morons who ignore basic things and don't know what are they doing.
Both sides have huge "home-court advantage" - every discussion at WoW blog from [different] perspective is flooded with negative comments [ranged from sceptical to degrading] while every discussion about WoW here quickly turns into a one-sided bashing of their method with no pro-WP people in sight.
Both sides occasionally like to mock PER which in spite of that is probably the closest one to become mainstream stat.

This symmetry and similarity provides the entertainment - I guess it works like every good family feud and reality TV.

The fascinating part for me is both practical [in those conditions there's no way to have a both-sided debate though I would gladly watch/read a "fight" on neutral ground between those camps] and theoretical - if you take look at a non-trivial object from 2 different perspectives there's a strong possibility those views will be different, right? Isn't it the case with WP vs [other metrics]?
For me both camps have different priorities which obviously led to the different results but... they are not mutually exclusive because of the different perspective.

[mystic]

wilQ, it is always interesting when people trying to see a pattern where no such pattern is. That is hardly a debate between WP and APM people here. And if you are able to follow the argumentations, you may as well see why here is way LESS support for WP than on a page which is stricly there to promote WP. You should open your eyes and understand the arguments first before making bold generalizations.

[wilq]

That is hardly a debate between WP and APM people here.

Which is exactly my point: how is it a debate if there's only one side here?

on a page which is stricly there to promote WP

Do you suggest Dre, Arturo and other guys there a) get rewarded for all their posts and b) can only write about certain topics? Otherwise IMO this argument is no longer valid. Berri left the blog basically.

You should open your eyes and understand the arguments first before making bold generalizations.

Yikes, that was a generalization ;-P Your assumption was I haven't done it before the post.

[mystic]

wilQ, I know it is always convienent to think of oneself as being objective and thus have an overview about everything, but you are wrong. First, we have people HERE who are arguing against WP who use boxscore based metrics. Take me for one, I have a metric completely based on the boxscore, and yet, I argue against WP. You just want to see a pattern, you want to put people into a certain box and then apply your ideas to it, because it just fits your personal feeling.

A debate about WP happens like years ago, right now the people can just repeat the argumentation, because in essence WP has the same flaws despite making that adjustment regarding the defensive rebounds. And while you might not see it, we had a discussion with pro WP people here just a couple of month ago, you are just not paying attention. Usually, the discussion gets less, when one side has the better arguments. Your conclusion upon this is: here are only APM people who despise WP, while the reality is that here on APBR is much more diversity. Mike is basically only posting the results of his boxscore metric, Evan has a play-by-play based boxscore metric, myself has a boxscore metric, DSMok is using a boxscore metric, which is just based on regression on RAPM values. Seriously, if you think here are only APM people around trying to trash WP, you are obviously not paying attention at all.

And indeed, wilQ, the writers on that blog are getting probably paid by Berri for providing a content which suits his way of thinking and his books. People, questioning WP and the arguments are getting blocked from the comment section, you may haven't notice that, but I can't comment over there, Evan is blocked and I guess others as well. The site is censored, when you ask question and explain things in too much detail, you get thrown out. There is a reason Berri set a limit on the characters and that limit is not there to improve the debate about WP.

So, do you see now why I said that you haven't looked at the argumentations?

[wilq]

First, we have people HERE who are arguing against WP who use boxscore based metrics. Take me for one, I have a metric completely based on the boxscore, and yet, I argue against WP.

Oh, you meant that part. OK. I should have written "WP vs the rest" and that was indeed a bad generalization. I'm sorry if I offended anybody. I thought you meant entire content not a technical shortcut

And while you might not see it, we had a discussion with pro WP people here just a couple of month ago, you are just not paying attention. Usually, the discussion gets less, when one side has the better arguments.

Do you mean a topic titled "Berri Changes Value of Defensive Rebounds in WP" where, again, nobody here was on the side of WP?

And indeed, wilQ, the writers on that blog are getting probably paid by Berri for providing a content which suits his way of thinking and his books.

Very interesting statement... is it a pure speculation on your part?

[mystic]

Oh, you meant that part. OK. I should have written "WP vs the rest" and that was indeed a bad generalization. I'm sorry if I offended anybody. I thought you meant entire content not a technical shortcut

I meant your content, which is entirly based on the pattern "WP vs. APM". Take that out, and your post is losing basically all of its meaning. The entire idea of yours is based on a seperation between black and white, either APM or WP. When you remove that, the rest makes no sense at all. And no, I don't feel offended at all. Why should I feel offended by that?

You are running into a typical bias, the idea that you can understand completely unfamilar people and their motives by just reading some posts on the internet. Get rid of that and you may start to concentrate more on the content of the discussion.

Do you mean a topic titled "Berri Changes Value of Defensive Rebounds in WP" where, again, nobody here was on the side of WP?

There were actually people arguing or taking side with WP and pointing out the good things the metric is supposed to do. They were just not good arguments. That's how it works. Seriously, if WP would be great at all those things, much more people would actually say that. It is just a weird idea that people would argue against WP without a very good reason.

Very interesting statement... is it a pure speculation on your part?

When you name a blog after a book title, what do you think you want to accomplish with that? When you constantly refer to the books, what purpose has that? And when people only using your metric in their articles and other things are said to be irrelevant, why do you think that is the case? For sure I don't have a copy of their paychecks, but I know that I got money for writing articles, and a lot of the stuff they are doing on WoW is pretty time consuming, if those people are not getting paid, they are even more naiv than I thought.

Look at the reaction by the WoW staff when someone asks too many questions or raises reasonable doubts about WP. They constantly get attacked personally. You should have read some of Berri's rants (a lot of them deleted after a certain amount of times) when he can't answer those questions, it is quite funny, and pretty obvious in the end that he is more concerned with a certain image he wants to protect than with real scientific research. How many articles have you found on WoW which are critical with the methodology?