APBRmetrics

mystic wrote: In that sense, it is not a question of the method used by Neil, but rather that this method is not applicable for all metrics in all years. Other than that I agree, that testing the 2001 to 2013 BPM values in that fashion is not a out-of-sample test. So, not including such years in the test would be the appropiate way.

I sent Neil data back to 1980. I believe he could easily test any number of metrics from 1980 to 2000 if we wanted to be "out of sample" for BPM.

Maybe BPM will correlate better in the recent seasons but not as well from seasons 15 or more years ago - it'll be interesting to find out. I'd love to see my ratings compared to any (& all?) others from 1980 to now if at all possible - to see when & where the correlations most diverge among all the known box score metrics. We'll just have to see what Neil comes up with.

Neil Paine wrote:Hey Jerry, you know the great thing about historical metrics? We can cut them off at 2000 (or whenever you want) and still have a large sample to test on.

Then please do so

You've "tested" models on in-sample data in the past, so I figured that was your plan this time, too. A couple of quotes from this very thread

Neil Paine wrote:I looked at metrics from that perspective here, and found that over the 2001-2012 period, ASPM did better than any other boxscore metric at predicting out-of-sample team performance

Neil Paine wrote:I also ran the same test using data from 1978-2014 for my SPM metric, Daniel's old ASPM (a version behind the current BPM), PER, WS/48 and a plus/minus estimate constructed from Basketball on Paper's ORtg/%Poss/DRtg Skill Curves

just to be clear, ASPM was fit on 2001-2014 RAPM, so "testing" it on the 2001-2012 period doesn't make sense

Neil Paine wrote:In any event, no matter how many times I look at which metric does the best job of predicting future team performance, the Statistical Plus/Minus metrics are always in a class by themselves, particularly as you use data from further out of the sample being predicted

what you call "out-of-sample" is actually in-sample

Hmm. I have been 'fitting' eWins to team plus-minus since 2004.
Does that mean it can't tested for season-to-season consistency in this interval?

Yr1 is the sample; yr2 and yr3 are outside that sample.

Any data used in deriving a model, be that the model formulation or coefficients/parameters is in-sample and no out of sample testing can be done on that data. I think there is some confusion here that perhaps because the data is 'out of sample', that is you are using previous year/s data to predict a future year, then the test is valid...which it isn't.

v-zero wrote:Any data used in deriving a model, be that the model formulation or coefficients/parameters is in-sample and no out of sample testing can be done on that data. I think there is some confusion here that perhaps because the data is 'out of sample', that is you are using previous year/s data to predict a future year, then the test is valid...which it isn't.

100% true. The true test would be testing all metrics on 2014-15 season since most of the metrics probably used 1985+ or 1974+ data to come up with the models. However as far as public metrics go, BPM will destroy all others. Actually I didn't test RAPM or xRAPM. Will do it in a few days.. Let's see which one is worse. Noise or missing info?

permaximum wrote:

v-zero wrote: However as far as public metrics go, BPM will destroy all others.

What's the definition of a "public metric"? The ones on basketball-reference? I'm am pretty certain my metric would "destroy" the correlations those produced also.

I have a very difficult time with the assumption that the '14-15 correlation w/ the '13-14 BPM data would "destroy" what my metrics would do, or maybe even Mike G's? Of course, maybe our stuff isn't considered "public".

It has bugged me for years now that PER, WS, WP (that is Berri's, right?) are considered THE box score metrics. It's like any work the last 10-15 years in box score metrics has ceased to exist. They are THE box score metrics because the were pretty much the first - but big reason I do the player ratings & rankings I do because I think they are a (often big) improvement over what those metrics produced. I don't think Mike G would be bothering to do what he does if HE didn't believe his results were at the very least better than what those metrics produced.

Neil has a number of metrics he can test (he has mine, maybe Mike G's). He might as well test '14-15 as well.

To reinforce my point a while ago about how PER should probably be ignored in general because of how poorly the compiled player PERs match team results SAME SEASON, here's this season's average player rating for each team (for BPM, PER, WS), & how it compares to actual team results:

Sorry I can't make it look "pretty" - maybe one of you guys can fix it for me because I completely forgot how to get everything to line up everything, I so rarely post results (partly because of this). Code don't cut it.

Anyway, it's pretty self explanatory I think. Teams are ranked by W%. I give team rankings in W%, Point Ratio (Pts/OppPts), & SRS. The next columns are the BPM, PER, WS results when the player ratings are compiled & averaged. The last three are team ranks by BPM, PER, & WS.

BPM is strongly tied to SRS, WS to Point Ratio - PER to, uh, pace & O efficiency I guess.

The Mavs are a REALLY strong #1 in PER, The Pelicans are a STRONG 3rd, & the Rockets are 24th - WELL below average and WELL below the Lakers (who are about average). The Celtics & Jazz are bit above average.

Any & all box score metrics, no matter how tested, should outperform (ie "destroy") PER.

BTW, the reason I didn't include my metrics is that everything is compiled at the team level FIRST, THEN the player level (where production, playing time, etc are worked in). My HnR & HnI team rankings (compiling player ratings) will exactly match team ratio, while my projected WAR & WAR/48 will exactly match W%. My metrics really need to be tested against others OUT of season to see how the correlations stand up to other metrics.

@Statman

I mean the metrics in basketball-reference, espn, nba.com or the metrics which are known by a lot of people. Usual stuff you know. EFFICIENCY, PER, ORTG, DRTG, Win Shares, Wins Produced, APM, BPM and even RAPM.

in table format:

sndesai1 wrote:in table format:

Code: Select all

╔═════════════════════════════╦═══════╦═════════╦════════╦════╦═══════╦═════╦═══════╦═══════╦═════════╦═════╦═════╦════╗
║                             ║ W/L%  ║ PtRatio ║  SRS   ║ W% ║ Ratio ║ SRS ║ TmBPM ║ TmPER ║ TmWS/48 ║ BPM ║ PER ║ WS ║
╠═════════════════════════════╬═══════╬═════════╬════════╬════╬═══════╬═════╬═══════╬═══════╬═════════╬═════╬═════╬════╣
║ Toronto Raptors (1)         ║ 0.857 ║ 1.12    ║ 9.6    ║  1 ║     1 ║   3 ║ 2.5   ║ 17.1  ║ 0.169   ║   3 ║   2 ║  1 ║
║ Memphis Grizzlies (1)       ║ 0.857 ║ 1.08    ║ 6.95   ║  2 ║     5 ║   5 ║ 1.8   ║ 16    ║ 0.149   ║   5 ║   7 ║  5 ║
║ Golden State Warriors (2)   ║ 0.846 ║ 1.11    ║ 11.19  ║  3 ║     2 ║   1 ║ 2.7   ║ 16.2  ║ 0.165   ║   1 ║   5 ║  3 ║
║ Portland Trail Blazers (3)  ║ 0.786 ║ 1.1     ║ 9.87   ║  4 ║     4 ║   2 ║ 2.6   ║ 16.5  ║ 0.155   ║   2 ║   4 ║  4 ║
║ Houston Rockets (3)         ║ 0.786 ║ 1.05    ║ 2.84   ║  5 ║     7 ║  10 ║ 0.8   ║ 13.8  ║ 0.135   ║  10 ║  24 ║  7 ║
║ San Antonio Spurs (5)       ║ 0.692 ║ 1.06    ║ 6.63   ║  6 ║     6 ║   6 ║ 1.7   ║ 15    ║ 0.14    ║   6 ║  16 ║  6 ║
║ Washington Wizards (2)      ║ 0.692 ║ 1.03    ║ 0.86   ║  7 ║    10 ║  13 ║ 0.2   ║ 14.9  ║ 0.117   ║  13 ║  18 ║ 12 ║
║ Dallas Mavericks (6)        ║ 0.667 ║ 1.1     ║ 8.85   ║  8 ║     3 ║   4 ║ 2.3   ║ 18.5  ║ 0.165   ║   4 ║   1 ║  2 ║
║ Sacramento Kings (7)        ║ 0.643 ║ 1.03    ║ 6.43   ║  9 ║     8 ║   7 ║ 1.7   ║ 15.3  ║ 0.123   ║   7 ║  14 ║  8 ║
║ Los Angeles Clippers (8)    ║ 0.615 ║ 1.03    ║ 4.31   ║ 10 ║    11 ║   9 ║ 1.1   ║ 16.1  ║ 0.118   ║   8 ║   6 ║ 10 ║
║ Phoenix Suns (9)            ║ 0.6   ║ 1.03    ║ 2.64   ║ 11 ║    12 ║  11 ║ 0.7   ║ 15.2  ║ 0.119   ║  11 ║  15 ║  9 ║
║ Chicago Bulls (3)           ║ 0.6   ║ 1.02    ║ -0.01  ║ 12 ║    14 ║  14 ║ 0.1   ║ 15.5  ║ 0.11    ║  14 ║  13 ║ 14 ║
║ Atlanta Hawks (4)           ║ 0.583 ║ 1       ║ -1.6   ║ 13 ║    15 ║  18 ║ -0.4  ║ 16    ║ 0.104   ║  17 ║   8 ║ 15 ║
║ New Orleans Pelicans (10)   ║ 0.538 ║ 1.03    ║ 4.4    ║ 14 ║     9 ║   8 ║ 1.1   ║ 17    ║ 0.118   ║   9 ║   3 ║ 11 ║
║ Miami Heat (5)              ║ 0.533 ║ 0.99    ║ -1.41  ║ 15 ║    16 ║  17 ║ -0.4  ║ 15.5  ║ 0.101   ║  18 ║  12 ║ 16 ║
║ Milwaukee Bucks (5)         ║ 0.533 ║ 0.97    ║ -5.4   ║ 16 ║    21 ║  24 ║ -1.4  ║ 13.8  ║ 0.09    ║  25 ║  25 ║ 19 ║
║ Denver Nuggets (11)         ║ 0.5   ║ 0.99    ║ -0.37  ║ 17 ║    18 ║  15 ║ 0     ║ 14.9  ║ 0.092   ║  15 ║  20 ║ 18 ║
║ Cleveland Cavaliers (7)     ║ 0.462 ║ 1.02    ║ 2.57   ║ 18 ║    13 ║  12 ║ 0.6   ║ 15.8  ║ 0.112   ║  12 ║   9 ║ 13 ║
║ Indiana Pacers (8)          ║ 0.429 ║ 0.98    ║ -2.89  ║ 19 ║    19 ║  21 ║ -0.7  ║ 13.5  ║ 0.089   ║  20 ║  27 ║ 20 ║
║ Brooklyn Nets (9)           ║ 0.385 ║ 0.99    ║ -2.03  ║ 20 ║    17 ║  19 ║ -0.5  ║ 15    ║ 0.098   ║  19 ║  17 ║ 17 ║
║ Orlando Magic (10)          ║ 0.375 ║ 0.95    ║ -6.04  ║ 21 ║    24 ║  26 ║ -1.6  ║ 13.6  ║ 0.07    ║  26 ║  26 ║ 24 ║
║ Boston Celtics (11)         ║ 0.333 ║ 0.98    ║ -0.8   ║ 22 ║    20 ║  16 ║ -0.2  ║ 15.8  ║ 0.084   ║  16 ║  10 ║ 21 ║
║ Utah Jazz (12)              ║ 0.333 ║ 0.95    ║ -2.38  ║ 23 ║    23 ║  20 ║ -0.7  ║ 15.5  ║ 0.075   ║  21 ║  11 ║ 22 ║
║ New York Knicks (12)        ║ 0.267 ║ 0.96    ║ -7.45  ║ 24 ║    22 ║  27 ║ -1.9  ║ 14.4  ║ 0.073   ║  27 ║  22 ║ 23 ║
║ Charlotte Hornets (12)      ║ 0.267 ║ 0.94    ║ -5.04  ║ 24 ║    27 ║  23 ║ -1.4  ║ 13.8  ║ 0.063   ║  24 ║  23 ║ 27 ║
║ Minnesota Timberwolves (13) ║ 0.25  ║ 0.92    ║ -8.21  ║ 26 ║    28 ║  29 ║ -2    ║ 14.6  ║ 0.046   ║  28 ║  21 ║ 28 ║
║ Detroit Pistons (14)        ║ 0.214 ║ 0.94    ║ -7.7   ║ 27 ║    25 ║  28 ║ -2    ║ 12.9  ║ 0.064   ║  29 ║  28 ║ 26 ║
║ Los Angeles Lakers (14)     ║ 0.214 ║ 0.91    ║ -5.54  ║ 27 ║    29 ║  25 ║ -1.4  ║ 14.9  ║ 0.038   ║  23 ║  19 ║ 29 ║
║ Oklahoma City Thunder (15)  ║ 0.2   ║ 0.94    ║ -3.33  ║ 29 ║    26 ║  22 ║ -0.9  ║ 12.5  ║ 0.065   ║  22 ║  29 ║ 25 ║
║ Philadelphia 76ers (15)     ║ 0     ║ 0.85    ║ -14.62 ║ 30 ║    30 ║  30 ║ -3.6  ║ 11.4  ║ 0.006   ║  30 ║  30 ║ 30 ║
╚═════════════════════════════╩═══════╩═════════╩════════╩════╩═══════╩═════╩═══════╩═══════╩═════════╩═════╩═════╩════╝

I know it's almost certainly a dumb question - how do I reproduce that format here? Maybe create a table in word & copy/paste in code?

I have zero patience to try it by trial and error.

not dumb at all.

assuming you originally had it in excel, just copy and paste the values into: http://www.sensefulsolutions.com/2010/1 ... table.html
use unicode art as the setting and then click create table
copy and paste the output as code into the message board

I found a novel way to build a historical SPM and hopefully I'll have a solution soon if it's not too late to send my values.

sndesai1 wrote:not dumb at all.

assuming you originally had it in excel, just copy and paste the values into: http://www.sensefulsolutions.com/2010/1 ... table.html
use unicode art as the setting and then click create table
copy and paste the output as code into the message board

Thanks for the info, I bookmarked the link. Hopefully next time I try to post some data I can make it halfway readable.

AcrossTheCourt wrote:I found a novel way to build a historical SPM and hopefully I'll have a solution soon if it's not too late to send my values.

I hope you can, I sent Neil an email asking if I could send my WAR/48 values also - just to see how they correlate also. He never responded, so I didn't send the ratings.

Neil might be overloaded, who knows. Can't look a gift horse in the mouth.

Statman wrote: BPM is strongly tied to SRS, WS to Point Ratio

Have you checked out how either of those metrics is calculated? BPM is summed up to fit the adjusted Net rating, and WS uses the team ORtg and DRtg heavily for the creation of the individual numbers (via Oliver's calculation of the individual ORtg and DRtg, check out his book for that). Both of those values are adjusting for the team performance. PER does not do that. Thus, comparing the in-sample results for BPM and WS/48 vs. PER makes about ZERO sense.

Statman wrote:PER to, uh, pace & O efficiency I guess.

The team average PER for this season has a R² of 0.932 to the team ORtg. It is mainly a stats describing the player's offensive impact. Funny thing is that it obviously describes something more meaningful than WS/48 for example in the longrun: http://www.apbr.org/metrics/viewtopic.p ... 343#p15334

And I guess, if the respective period wouldn't have been an in-sample test for ASPM (the prior version of BPM), I can see PER even take that stats over after 3 years and more.

Just check out how your metric would do, if you wouldn't make it fit on the team level. Than compare that to PER. I wouldn't be surprised, if your metric comes up short.