Is WP a legitimate stat?

[colts18]

I'm wondering what the consensus is in regards to wins produced. Does it have any value. I'm curious because of a recent article where they attributed the 00-02 Lakers success more to role players than Shaq/Kobe.

One of the most famous duos is that of Shaq and Kobe. They powered the Lakers back to prominence. The fact that personal issues shoved Shaq out a few years to early is one of the many pains felt by Laker fans, especially as Shaq would win a ring with another dynamic shooting guard in 2006. Except, the contributions of both Shaq and Kobe are overstated, by a lot! Let’s go back through our the Phil Jackson Laker titles and show what I mean.

The 2000 Lakers

Regular season
Player % of Lakers’ regular season wins
Shaquille O’Neal 32% (1st)
Kobe Bryant 16% (2nd)

Playoffs
Player % of Lakers’ playoff wins
Shaquille O’Neal 40% (1st)
Ron Harper 11% (2nd)
Robert Horry 10% (3rd)
Brian Shaw 8% (tied 4th)
A.C. Green 8% (tied 4th)
Glen Rice 8% (tied 4th)
Kobe Bryant 8% (tied 4th)

I’ve listed all players that put up more wins that Kobe and Shaq in either the regular season or the playoffs. In the regular season the start of the Lakers Kobe-Shaq dynasty was indeed the Kobe and Shaq show. Of course, Shaq was much better than Kobe and Kobe was pretty good himself. The playoffs were a different story. While Shaq kept up his insane play, Kobe dropped from being the second best player on the team and dropped below the likes of Ron Harper and Robert Horry in terms of helping the team win. And it’s worth noting that Kobe’s playoff production was actually below average!

The 2001 Lakers

Regular Season
Player % of Lakers’ regular season wins
Shaquille O’Neal 28% (1st)
Kobe Bryant 17% (2nd)

Playoffs
Player % of Lakers’ playoff wins
Kobe Bryant 20% (1st)
Derek Fisher 19% (2nd)
Shaquille O’Neal 17% (3rd)

The regular season dynamic kept up. Shaq shouldered most of the load. Kobe was a solid second. We do notice that Shaq slipped a little from his prior dominance. The playoffs is where we have to adjust our dials. Kobe was the best Laker on route to their back to back. However, Derek Fisher outplayed Shaq! Read that again. In Shaq’s second title, his performance was below Derek Fisher’s! And Derek Fisher definitely showed an amazing difference. In the 2001 regular season, Fisher was average. In the playoffs? He played over twice as good as an average player! The reason is surprisingly simple. Fisher shot over 50% from beyond the arc and had a true shooting of almost 70% for the whole playoffs! While Shaq certainly can take credit for getting the Lakers to the playoffs and definitely contributed to their second playoff run, Bryant and Fisher were the top dogs this go around.

2002

Regular season
Player % of Lakers’ regular season wins
Shaquille O’Neal 20% (1st)
Kobe Bryant 19% (2nd)

Playoffs
Player % of Lakers’ playoff wins
Robert Horry 30% (1st)
Shaquille O’Neal 24% (2nd)
Rick Fox 15% (3rd)
Kobe Bryant 12% (4th)

Robert Horry! You read that right. Robert Horry overall strong production anchored the final run for Shaq and Kobe. Shaq was still a strong contributor but not top dog. And while he was top four in total wins, it turns out that Kobe’s playoff production was again below average!
Summing Up

If we merely examine the regular season of the Lakers during the Kobe-Shaq era then a very familiar narrative is told. Shaq was an amazing player and Kobe was an excellent sidekick. As the years went on, Shaq declined and Kobe improved until their sad inevitable breakup. But, we know the playoffs are the irrational test we use to define greatness. And here is where the Shaq and Kobe myth crumbles. Shaq was only the top Laker in one of the Lakers’ three titles. Kobe was only a solid contributor in one of the three runs. To be fair, he was the best Laker in that run. And the “role players” of Fisher, Horry and Fox? Yeah, it turns out they were huge “when it mattered”. I can’t tell you how hilarious that is to write. The key we should all note is that the playoffs are very limited number of games. Over 20 games any NBA player can have a great run. Even greats can slump. The playoffs are a great place to find narratives, but may not be the best place to find statistical absolutes. And all I can say is when we examine the Shaq and Kobe playoffs narrative? Well, the numbers don’t back it up.

http://wagesofwins.com/2013/03/11/the-m ... haq-rings/
l
How does WP come up with those kinds of results

[DSMok1]

Look through the archives here for some of the original discussion: http://godismyjudgeok.com/DStats/APBRmetrics_Old/

Also, check out Tom Tango's recent post here (the comments are good as well): http://tangotiger.com/index.php/site/co ... at-concept

In short, I think the consensus is that it is flawed.

[AcrossTheCourt]

Compared to other box score metrics like PER it's alright in some cases, but the problem is the people behind the stat.

Wins Produced is built from offensive and defensive efficiency. It all goes back to the possession. They find the value of a possession in terms of wins, and then derive values for box score numbers. It's pretty basic regression. They also add a boost based on team defense based on how many minutes a player had on that team. So it doesn't matter how "good' of a defender you were; it's just based on how many minutes you played. But it's a small effect.

The problem is the interpretation. They "validate" the model by summing Wins Produced for each player and comparing it to the team's wins that season. The problem is that all they're doing is validating that team point differential (efficiency) leads to wins. Of course it does! We already knew that!

But they proceed to write tons of articles evaluating players solely on Wins Produced and disparaging other writers and stats. They also have a boring writing style and talk very little about the actual game.

People complain about NBA stat guys not taking context into account and only using numbers to judge players. We don't do that. The best NBA stat guys don't. But they do, and they need to be called out for it.

[v-zero]

Beyond the fact that WP is not a good stat in general (poor predictive value), the WoW guys and their often overzealous followers do not take criticism well. They claim that their stat is highly predictive, but when called to prove this they hark either to its correlation with past wins, which is not prediction! - and which as ATC points out above is a given considering their framework, no matter how they weight different box score numbers, or they hark to the fact that player per-minute values for box-score numbers are pretty consistent year to year, but this does nothing to validate the values of their coefficients, but somehow they are convinced it does, and they have managed to convince a large following that this is the case.

It is clear from much of their work, even that of Berri, that they don't have a strong understanding of statistical mathematics, or robust model building, or proper inference techniques - which would be fine if they didn't act as if they wrote the book on regression analysis. They also apply their interesting view on statistics to debunking other attempts to model the NBA, often accusing others of making assumptions or using intuition, yet they cannot see (or refuse to acknowledge) that there is an enormous (and clearly flawed) leap of faith in assuming that a deconstruction of the team-efficiency equation with a few explanatory, non-predictive regressions thrown in, all done at the team level, is enough to best explain player value using the box-score.

[Bobbofitos]

Beyond the fact that WP is not a good stat in general (poor predictive value), the WoW guys and their often overzealous followers do not take criticism well. They claim that their stat is highly predictive, but when called to prove this they hark either to its correlation with past wins, which is not prediction! - and which as ATC points out above is a given considering their framework, no matter how they weight different box score numbers, or they hark to the fact that player per-minute values for box-score numbers are pretty consistent year to year, but this does nothing to validate the values of their coefficients, but somehow they are convinced it does, and they have managed to convince a large following that this is the case.

It is clear from much of their work, even that of Berri, that they don't have a strong understanding of statistical mathematics, or robust model building, or proper inference techniques - which would be fine if they didn't act as if they wrote the book on regression analysis. They also apply their interesting view on statistics to debunking other attempts to model the NBA, often accusing others of making assumptions or using intuition, yet they cannot see (or refuse to acknowledge) that there is an enormous (and clearly flawed) leap of faith in assuming that a deconstruction of the team-efficiency equation with a few explanatory, non-predictive regressions thrown in, all done at the team level, is enough to best explain player value using the box-score.

Perfect summation.

[deepak]

I'm not that familiar with all the studies that have been done to validate or critique WP.

But I just took a quick look at how its calculated (here), and something stuck out that's bothering me.

They develop a model for Win% based on team offensive efficiency and defensive efficiency. They use the standard definition for offensive efficiency, but they define defensive efficiency as "points surrendered divided by possessions acquired (PA)", where:

Code: Select all

   PA =  DFGM + 0.45*DFTM + REBD + DTO + REBTM

Despite the very different formulation, their defensive efficiency yields similar rankings to the more commonly used definition of points surrendered divided by opponent possessions.

Anyway, so they break up offensive efficiency and defensive efficiency into 4 terms: points scored, possessions employed, points allowed, possessions acquired and calculate the marginal value for each factor. For PA, its a 0.033, meaning every possession acquired is equivalent to .033 wins. For points surrendered, its -0.032. So, a made 2-point field goal by the opponent (2 points surrendered, 1 possession acquired) would effectively cost a team .031 wins.

A defensive rebound, by their formula, is (I would think) equivalent to an acquired possession and therefore should add .033 wins. In their table, they have it add .034 wins -- not sure why.

An offensive rebound would be equivalent to subtracting a "possession employed (PE)", and therefore should add .034 wins, which is what they show.

But how you define the formula for "PA" or "PE" changes the value of the individual statistics. So, take PE:

Code: Select all

   PE = FGA + .45*FTA + TOV - REBO

One could have defined it in this way instead:

Code: Select all

   PE = FGA_not_offensive_rebounded + .45*FTA + TOV

The difference is that REBO no longer shows up in the formula. Now, if a player attempts a shot that is offensive rebounded, it does not contribute at all to PE. So, the player isn't penalized and the offensive rebounder isn't credited. Now, I wouldn't say that's fair, but then I don't think its fair to say that a missed field goal is equivalent to a turnover and an offensive rebound is equivalent to stealing it back. And that's effectively what the WP model says simply because of the way they formulate PE/PA, which is, I think, arbitrary on their part.

[v-zero]

Yep, that falls under the veil of assuming effects at team level are the same at player level. At team level the definitions are consistent, at player level they are not, as you point out.

[Neil Paine]

The difference is that REBO no longer shows up in the formula. Now, if a player attempts a shot that is offensive rebounded, it does not contribute at all to PE. So, the player isn't penalized and the offensive rebounder isn't credited. Now, I wouldn't say that's fair, but then I don't think its fair to say that a missed field goal is equivalent to a turnover and an offensive rebound is equivalent to stealing it back. And that's effectively what the WP model says simply because of the way they formulate PE/PA, which is, I think, arbitrary on their part.

Exactly. They like to act as though the WP formula fell organically out of a regression model, but it actually is the product of some very deliberate decisions on the model-builder's part, which colors all of the results it produces.

Yet, because possessions & points for/against are technically accounted for at the team level, it is still rigged to give you that vaunted 95% explanatory power, a self-fulfilling non-prophecy if ever there was one...

[Dr Positivity]

I've been reading WP to kill time for the last half year and I've actually grown fond of Andres and Arturo. I think they're inquisitive and attracted to rationality, have class and love the game. Guys like them are normally a good thing for a field, including basketball. The videocasts/podcasts with Andres are downright excellent and worth it for a basketball fan even if you don't agree with everything they're saying IMO. Especially rec the one that just got put up with Kevin Draper

But WP is flawed. I think part of the problem is Berri himself holding too much name-power and The Godfather esteem for them, and thus his framework for WP hasn't been changed enough as it should have. Just judging a book by its cover, Berri doesn't seem like the type of guy to look at his own creation and ask himself "Is this wrong? Can I make it better?". I think if Arturo had been the guy to come up with WP, it may have changed a lot more from its original framework to now just because he seems more interested in pushing forward towards new ideas

[deepak]

Beyond the fact that WP is not a good stat in general (poor predictive value), the WoW guys and their often overzealous followers do not take criticism well. They claim that their stat is highly predictive, but when called to prove this they hark either to its correlation with past wins, which is not prediction! - and which as ATC points out above is a given considering their framework, no matter how they weight different box score numbers, or they hark to the fact that player per-minute values for box-score numbers are pretty consistent year to year, but this does nothing to validate the values of their coefficients, but somehow they are convinced it does, and they have managed to convince a large following that this is the case.

Are there any available studies that have shown WP to have poor predictive value? That claim seems to be at variance with what the WoW analysts argue.

If WP48 doesn't change much in the following year, then doesn't it follow that if we know how minutes will be allocated in the following season we can come up with a fairly accurate prediction of total wins for every team? Its not obvious to me where the flaw is in that reasoning. If the flaw can be pointed out, or experimental evidence to the contrary can be shown that would be very helpful.

[v-zero]

There have been some, though I don't have them to hand. Google is your friend. The WoW analysts have never attempted to show the predictive nature of WP, only claimed it. It's very easy to make claims; I have a cow made from gold, for instance.

WP is quite consistent year-to-year because rosters are quite consistent year-to-year. When players change team their WP values do not remain consistent, though they likely do in the limit, but that is a given.

[deepak]

All I could find in assessing WP's predictive value was Dan Rosenbaum's presentation of his paper "The Pot Calling the Kettle Black" at NESSIS (2007). Wish the actual paper was available somewhere. Dan used to have it on his website years back, but its no more to be found. Berri wrote a response to some of the points brought up by him, but it wasn't very substantive. I'm surprised the WoW analysts haven't written anything comparing the predictive power of WP to other stats.

[v-zero]

Their stance on their metric is seemingly that it is more or less beyond question as to its formulation.

[DSMok1]

All I could find in assessing WP's predictive value was Dan Rosenbaum's presentation of his paper "The Pot Calling the Kettle Black" at NESSIS (2007). Wish the actual paper was available somewhere. Dan used to have it on his website years back, but its no more to be found. Berri wrote a response to some of the points brought up by him, but it wasn't very substantive. I'm surprised the WoW analysts haven't written anything comparing the predictive power of WP to other stats.

WP is fairly consistent year to year due to high value on rebounds, which are to some extent a measure of a player's typology/role rather than quality.

Thus, they fairly high year-to-year correlation is correct, technically

When tested out of sample (new lineups, new roles, roster turnover), WP does relatively poorly. If we look at the same roles/lineups, it does fine... as any stat that equals point differential would.

We need a better study on this, but here is one that looked at out of sample (though it didn't filter to only focus on new lineups/changes, which would be more informative): http://sportskeptic.wordpress.com/2012/ ... the-goods/

[deepak]

We need a better study on this, but here is one that looked at out of sample (though it didn't filter to only focus on new lineups/changes, which would be more informative): http://sportskeptic.wordpress.com/2012/ ... the-goods/

Hmmm, his results seem very different from what was presented in 2007 by Rosenbaum et al. PER fared very well as a predictor of team performance in that study, where here Alex argues that PER is the very worst. Not sure what to believe.