APBRmetrics

Just a few thoughts about recent output of the ESPN expert panel, which produced both an upcoming season forecast and individual player rankings.

In the former case, the aggregated estimate of the 210 plus member panel seems to have "crowd wisdom" to the extent that the forecast seems pretty close to the betting line shown (link provided by Royce Webb and then integrated with other predictions here: http://imagizer.imageshack.us/a/img912/7204/q8KNwz.png). Please feel free to disagree if this is an unwarranted conclusion.

By contrast, for the player rankings, the crowd appears to be rather unwise, at least seen through a RPM prism.

Half of the forecast Top 20 list includes players whose RPM last year were below the Top 20, with high/lowlights of Anthony Davis (ranked 3rd, but last year 98th in RPM) and DeMarcus Cousins (ranked 19th, but last year 127). And, of course, expected average aging considerations would not propel these two close to the Top 20.

Then another perspective: this year's forecast Top 20 last year had an average RPM rank of 33.85, so 23 slots lower in the table.

Perhaps RPM is a very imperfect forecast basis (we shall see) but digging a little deeper, we can get a clue as to the why a majority of ESPN's experts seem to hate RPM, and as should be expected, it's disproportionately the age-old failure to properly appreciate defensive contributions.

If one compares the actual RPM Top 20 in 2014 with the 2014 RPM ratings of ESPN's forecast Top 20, what one sees is last year's actual best were +1.20 better overall (5.56 vs. 4.37) of which +0.36 was attributed to offense (3.53 vs. 3.16) and +0.83 to defense (2.03 vs. 1.20).

More speculatively (and this doesn't comment on the accuracy of ordinal rankings) there also might be reason to believe that the ESPN collective brain thinks that talent is far more equally distributed at the top than it actually is.

For the purposes of discussion, a reasonable approach might be to translate the ESPN rankings into "implied RPM" by mapping the 0 to 10 quality scale into the observed RPM +/- range for last season (9.08 of LeBron, at the top, down to -8.44, at the bottom, for a range of 17.52).

Doing this, for example, LeBron, the "perfect 10", becomes the table-topping LeBron of 2014 with an RPM of 9.08. Similarly, Chris Bosh, the 20th ranked, with an 8 rating on the 0 to 10 scale, has a corresponding "implied RPM" of 5.58.

Accepting such an approach for the sake of argument, the unweighted average "implied RPM" of the forecast Top 20 was 6.68. By contrast, again, the actual unweighted average of actual Top 20 RPM last season was 5.56, and to get an average of about 6.7, you would have to average the contributions of the Top 8 players last year. Just a thought.

to me, these results simply imply that the panel thinks that xRAPM, while being clearly the best metric they could've considered while making their rankings, is not perfect. or simply that with the current information available to us, no metric is yet (near) perfect.

dont we all agree that Brow and DeMarcus are likely to be much closer to 3rd/19th in the league this coming season (if doing the rankings thing, which i dont much like anyway, but thats a different story), than they are to 98th/127th? ie that they are better than the 50th and 73rd best players in the league, right now?

also, their RPMs at end of 2014 include info from the 2013 season. thats clearly relevant info for the sample of NBA players as a whole. but its probably not that relevant in terms of evaluating these 2 particular guys, especially Brow.

it also doesn't account for them probably getting a decent bit better this past summer (more than a bit in the case of Brow), due to the experience of playing in FIBA, receiving coaching on defense from Thibs, etc.

i think that xRAPM-based ratings probably underrate these 2 particular guys more than they do any 2 other players. so imo, it makes sense that the crowdsourcing approach has these 2 guys as the biggest outliers compared to xRAPM.

Interested in this discussion. The wisdom of the crowd can be phenomenal, particularly when done well, and I'm constantly looking for ways to improve our approach. The results we've seen have been eye-opening to me and we hope to extend that success.

As for #NBArank, please don't assume that the panel is making any judgment of RPM. We do not instruct the panel on how to rate players (the panel provides the ratings; from those we derive the rankings). There are many people and different approaches -- this is indeed a strength of the panel.

We are, on the other hand, not averse to instructing the panel in the future if we find appropriate ways to do so, such as using better stats or techniques. So I'm very interested in figuring out what those might be, and this discussion might enlighten me in that regard.

Royce

I think most of the raters (I am one) think of this as a tier/rank approach. So we care about the sequence more than the delta in the ratings. That may be the biggest potential issue--what do you do with the high end? Do you rank LeBron a 10, KD a 9, CP3 an 8 and then everyone else a 7 or below (which may more accuratly reflect the actual gap in talent)? Or do you put all of those guys at 10 so you can get a better spread in the rest? Is a 10 the top 10% of the league, or is it just LeBron?

That's the most significant issue, I think. Effectively, this is about the rank, not about the rating. It is NBA Rank, after all! Looking at the ratings isn't going to help a lot. I would just look at the ranks for critical analysis of the rankings.

FYI, I used went with a straight rating system based mostly on BPM/VORP and then truncated the top end--LeBron was a 13 and KD a 12; just rounded down the top 5 players to 10s. Couldn't really decide what to do for sure.

As for the Wisdom of the Crowd approach--the wisdom of the crowd with an expert crowd will beat just about any other methodology. That's about what Vegas ratings are anyway.

To follow up, two points:

(1) It is very hard for crowds to be wise (experts included) if the issue for appraisal is not well-defined.

I could offer some gratuitous comments on MVP and HoF voting, but the point is clear?

(2) Even is the issue for appraisal is well-defined, the collective judgement of experts is by no means certain to be wise (calling into question the supposed expertise?)

Rereading the terms for ESPN's individual player ranking (more carefully this time around) it actually does seem to me to be quite clear. From http://espn.go.com/nba/story/_/id/11530 ... gs-401-500:

"We asked our ESPN Forecast panel to predict the overall level of play for each player for the upcoming NBA season. This includes both the quality and the quantity of his expected contributions, combined in one overall rating....We asked our ESPN Forecast panel to rate each player on a 0-to-10 scale, in terms of "the overall level of play for each player for the upcoming NBA season.""

Whereas the focus of my initial remarks was RPM, this pretty much sounds like a direct request to rank players by expected WAR, with the 1 to 10 scale offered for simplicity (were fractional numbers allowed?) So, perhaps I should revisit my remarks taking this more "correct" approach.

And what I find is that my previous conclusion is more firmly founded, i.e. looking at WAR rankings, the overall anti-defense bias on the part of the expert panel can be more clearly seen.

To try to illustrate this as succinctly as possible, there are ten names not in common between the 2014 Top 20 in WAR and the ESPN forecast of Top 20 players. If you look at the difference in the (unweighted) average Offensive and Defensive RAPM between these two groups, you find that those in the Top 20 for 2014 were slightly worse offensively, -0.2 (2.1 vs. 2.3), but much better defensively, +2.0 (2.4 vs. 0.4)

But past isn't (necessarily) prologue. What about expert crowd wisdom, anticipating future changes?

Well, one such change might be the significant deterioration of Tim Duncan. He cannot stay great forever. But removing him from the aforementioned average hardly changes the (anti-defense bias) picture. The differences in Offensive and Defensive RAPM between groups change to +0.1 and +1.6, respectively.

But what about possible expert crowd wisdom in anticipating growth of the previously noted, developing stars: Anthony Davis and DeMarcus Cousins? OK. So, let's suppose approximate Durantesque levels of third year improvement (in addition to the deterioration of TD). Let's add a +3 to both players' offensive and defensive RAPM. Still the notional anti-defense bias remains: the respective difference in the two measures now being -0.5 and 1.0.

Now, perhaps the notion that WAR is the desired metric is fundamentally incorrect (but if not that, or something close, what?) or perhaps it is correct but there is greater wisdom in the expert crowd, to be gloriously revealed at season's end.

Alternatively, maybe the story is simpler (and consistent with NBA history) most "experts" do not properly weight defensive contributions at the individual level. (Should I bring up Kobe Bryant at this point?)

Anyway, supposing this conjecture is correct, it is the an interesting question as to why collective wisdom is observed when measuring team outcomes but not at the individual level.

Royce, would you consider having the ESPN Forecast panel rate each player on a 0-to-10 scale for both offense and defense, then let the aggregators combine them equally? Or even go just 0-5 on defense, if it is consider too hard with current state of stats and knowledge to be able to go as granular as 0-10 on defense with confidence or at least equal confidence (but then adjust the scale to equivalent with offense)? And / or take minutes projection out of the hands of the panel or put that component into the hands of a few of those seriously willing to do that carefully for all the projections?

Another you could do (if you are not already) would be to find the average rating for each member of the panel and scale their individual player ratings to that average so that a 5 from a panelist whose average is 5 is equal to the 7 given by the panelist whose average is 7. This is done with panel ratings if beers in several places. At least also looked at standard deviation from crowd average and I believe adjusted panelist weighting based on that.

As everyone here knows, defense is indeed difficult to assess and properly weigh, and not just for our panel, but for almost everyone and for most systems. And it's all the more difficult because offense and defense work so differently from each other in basketball, and because they can affect each other in big and little ways.

These are constructive suggestions toward addressing those issues. While I don't know yet what is the most practical solution, this discussion helps. Thanks. It might start with my putting more emphasis on defense in the instructions, though I cannot put my thumb on the scale (or emphasize any one factor) without the risk of skewing the results inordinately.

RPM has already influenced the panel and its influence should grow, along with that of other advanced stats systems and metrics.

RoyceWebb wrote:As everyone here knows, defense is indeed difficult to assess and properly weigh, and not just for our panel, but for almost everyone and for most systems. And it's all the more difficult because offense and defense work so differently from each other in basketball, and because they can affect each other in big and little ways.

Point taken, but this is not to say the same basic point cannot be made about offense.

RoyceWebb wrote:These are constructive suggestions toward addressing those issues. While I don't know yet what is the most practical solution, this discussion helps. Thanks. It might start with my putting more emphasis on defense in the instructions, though I cannot put my thumb on the scale (or emphasize any one factor) without the risk of skewing the results inordinately.

I think that it would be a very good idea to put your thumb on the scale (as well as make more explicit what you want estimated/ranked) if for no other reason that the other thumb rests heavily on the other side. With respect, just look at essentially every article published by your employer (no different from other media outlets). Can it be said that a profound offensive bias doesn't exist? (Examples provided upon request.)

RoyceWebb wrote:RPM has already influenced the panel and its influence should grow, along with that of other advanced stats systems and metrics.

A final empirical point. We can look at the ESPN expert panel's prognostications of last year, compared to realized WAR (and this is not to make a fetish about WAR, but just to realize that it is a representative +/- measure of quality * quantity).

Last year's predicted Top 20 realized an (unweighted) average Offensive, Defensive, and Total RPMs of 3.32, 1.00, and 4.31 respectively (and an average WAR of 9.745 and an average WAR ranking of 47.3). By contrast, the actual Top 20 WAR players, saw the following averages: 3.08, 2.18, and 5.26 (with an average WAR of 12.42).

So, last year's expert panel was pretty accurate in terms of the average offensive efficiency of the most "valuable" players, their preferred group yielding 0.25 more points per offensive possession. But they were not so good in appraising and appreciating defensive ability, with their preferred players being -1.18 worse than the "actual" most valuable players (in WAR terms).

And this year's forecast doesn't promise to be much better (in terms of recognizing defensive contributions). Comparing it to last year's realized Top 20 WAR, the differences in terms of Offensive, Defensive, and Total RPM are very similar: +.09, -.98, and -.89.

We shall see.