2014-15 MAX Ratings

Home for all your discussion of basketball statistical analysis.
permaximum
Posts: 413
Joined: Tue Nov 27, 2012 7:04 pm

Re: 2014-15 MAX Ratings

Post by permaximum » Thu Dec 04, 2014 11:04 am

@Statman

I can reproduce or calculate OBPM, DPBM, OWS, DWS easily myself. It's not a rocket science. If B-R disappears it won't be a problem for me.

@mystic

I did know OWS, DWS include ORTG and DRG. However, when you work with penealized regressions such as lasso, elastic and ridge, you want to include everything that increases prediction meaningfiully and don't cause overfitting as you can see in xRAPM's box-score partition. Since regression decided the penalty, I gave ORTG and DRTG a chance too. And, I wanted to be as clear as possible so you all can replicate the metric.

However imo, you just need to give some info about what's in sample or not for the metric and that's it. Especially since nobody except a few in your ethic field shares non-publicly available data or steps to reproduce metrics. (BTW I can assure you that, age and position value is not in the blend)


From now on, I'll answer you with the appropriate attitude. So people, stop complaining on what's fair and ethical etc. and come up with something that's better. I promise you when you do that, I'll come with something even better in a few days. Instead of whining you know.

Sigh... I should have sent the ratings to Neil and never written anything here. All I get is like "it's restricted to rush at the beginning" in your typical strategy game. Grow up.

permaximum
Posts: 413
Joined: Tue Nov 27, 2012 7:04 pm

Re: 2014-15 MAX Ratings

Post by permaximum » Thu Dec 04, 2014 2:21 pm

I decided to update the rankings daily from now on... Ratings are updated after yesterday's games.

Statman
Posts: 548
Joined: Fri Apr 15, 2011 5:29 pm
Location: Arlington, Texas
Contact:

Re: 2014-15 MAX Ratings

Post by Statman » Thu Dec 04, 2014 4:03 pm

mystic wrote:
v-zero wrote: Whilst a meta-model such as permaximum's is an interesting thing, it doesn't provide the kind of insight that a model from first principles can provide.
That is another issue with the approach here, let alone that likely a lot of people done something similar anyway. I also don't think that it is a big secret that a blended metric is doing better than a single metric.
To be honest, many metrics now are kinda "blended" from past metrics. My work's base is pretty similar to aspm (or maybe aspm w/ a smidge of PER), but I adjust for league, team, & playing time relative to team quite differently - in a way to bring in a sort of RAPMish factor (an attempt to "see" things not explicate in the box score). Important thing is - I can reproduce my work completely from scratch - and I can do it for ANY league in the world as long as I have all player data from every team - as well as game results (honestly, season PF & PA for teams works fine as long as SoS's aren't completely out of wack - say like college, where I have to have more complex team ratings).

permaximum
Posts: 413
Joined: Tue Nov 27, 2012 7:04 pm

Re: 2014-15 MAX Ratings

Post by permaximum » Fri Dec 05, 2014 4:27 pm

Update:

1. Tweaked the code a bit (league adjustment part). The difference should be minimal. Around 0-0.2. It's final now.

2. Generated ratings for seasons before 1979-80 for more out-of-sample prediction test. 2013-14 is out-of-sample too since BPM was regressed towards RAPM of the same year. However we're predicting team point differental of the next year. Now out-of-sample seasons for prediction test are;

1973-74
1974-75
1975-76
1976-77
1977-78
1978-79
1979-80
2013-14
2014-15

3. 2014-15 ratings are updated. I'll continue to update them daily since it's "the best box score metric that's publicly available" atm.

@Neil Paine.

I've sent you the final data which includes 1973-...1979 seasons and the current one as of 05.12.2014. The ratings are updated very slightly. (0 to 0.2)

permaximum
Posts: 413
Joined: Tue Nov 27, 2012 7:04 pm

Re: 2014-15 MAX Ratings

Post by permaximum » Sat Dec 06, 2014 12:15 pm

Rankings are updated. Russel Westbrook's rating normalizes the more he plays.

talkingpractice
Posts: 194
Joined: Tue Oct 30, 2012 6:58 pm
Location: The Alpha Quadrant
Contact:

Re: 2014-15 MAX Ratings

Post by talkingpractice » Sun Dec 07, 2014 12:19 am

after x weeks of reading this stuff, i just cant help myself any longer...

it appears that this hilarious/dangerous thread re: comparing metrics has taken over where the last hilarious/dangerous thread on comparing metrics left off ->
  • - theres insample stuff (both obvious and less obvious) all over the place in these various metrics/blends/idunnos
    - x% (x being pretty large, prolly) of the results are fake, obviously.
    - etc.
this 'test' needs to be coated in cinnamon to hide the smell lol.

Crow
Posts: 6245
Joined: Thu Apr 14, 2011 11:10 pm

Re: 2014-15 MAX Ratings

Post by Crow » Sun Dec 07, 2014 4:33 am

If Neil is still in planning perhaps it would be wise to restate all the criteria for models to comply with and how the scoring is going to done again. There isn't a rush and hopefully greater agreement about approach can be reached than exists now. If anyone else wants to propose different criteria and scoring and is willing to administer we could have more than one test. I'd propose brand new separate thread(s) for any reporting of findings or restart of the whole exercise. It is complicated. The dialogue of the parties should be able to get to a good place, I think / hope.

permaximum
Posts: 413
Joined: Tue Nov 27, 2012 7:04 pm

Re: 2014-15 MAX Ratings

Post by permaximum » Sun Dec 07, 2014 8:28 am

talkingpractice wrote:after x weeks of reading this stuff, i just cant help myself any longer...

it appears that this hilarious/dangerous thread re: comparing metrics has taken over where the last hilarious/dangerous thread on comparing metrics left off ->
  • - theres insample stuff (both obvious and less obvious) all over the place in these various metrics/blends/idunnos
    - x% (x being pretty large, prolly) of the results are fake, obviously.
    - etc.
this 'test' needs to be coated in cinnamon to hide the smell lol.

1. Well first, I'm not lying.
2. Finding out which seasons are "in-sample" and which seasons are "out-of-sample" for a metric is incredibly simple. Do I really need to explain it? :)
3. You're in the wrong thread. This is not a metric comparison thread.

@Crow

I'm against any rules, restrictions etc. Come up with the best prediction at "anything" is what matters to me. I'm not saying I'm good at anything. I'm just saying if somebody "somehow" predicts anything better than others, great. I won't criticize his approach afterwards.

Edit: BTW, I have an opinion. The best box-score metric and the best RAPM variant should be the only all-in-one metrics to stay updated and used in the future. I know that RPM is the best RAPM variant. And I know that BPM blend (which I call MAX) is the best box-score metric (I thank DSmok1 for the 10th time). So, if you got something to say about this, shoot!

I'm tired of all this talking. It's jut a talk. I rarely see an actual work. If you can't show your work because of "money-related" reasons which is perfectly fine, then why are you talking so much?

I respect those that share their work and talk less. J.E, DSmok1, Crow, Mike G, EvanZ, colts18, hoopdon, bobbofittos and a few others. However some people just "talk".

It wasn't a luck that Jeremias E.'s RPM is used by ESPN and DSMok1's BPM is used by B-R. Thank you guys for your work, again.

v-zero
Posts: 409
Joined: Sat Oct 27, 2012 12:30 pm

Re: 2014-15 MAX Ratings

Post by v-zero » Sun Dec 07, 2014 12:59 pm

permaximum wrote: 2. Finding out which seasons are "in-sample" and which seasons are "out-of-sample" for a metric is incredibly simple. Do I really need to explain it? :)
So how would you do this? To you what does in-sample mean? I'm willing to bet it doesn't mean to you what it does to me.
permaximum wrote: I know that BPM blend (which I call MAX) is the best box-score metric (I thank DSmok1 for the 10th time). So, if you got something to say about this, shoot!
Curious, then, that you should care about any sort of contest. I dare say your confidence is misplaced.
permaximum wrote: I'm tired of all this talking. It's jut a talk. I rarely see an actual work. If you can't show your work because of "money-related" reasons which is perfectly fine, then why are you talking so much?

I respect those that share their work and talk less. J.E, DSmok1, Crow, Mike G, EvanZ, colts18, hoopdon, bobbofittos and a few others. However some people just "talk".
Ah, I think a lot of these people would speak up, but do not feel that it is time well spent. I don't feel this is time well spent, but I am very easy to bait in such situations. J.E. has already spoken up previously, in essence to say 'the vast majority of you clearly don't know the difference between in-sample and out of sample', for which he got an impetuous response from Neil.

talkingpractice
Posts: 194
Joined: Tue Oct 30, 2012 6:58 pm
Location: The Alpha Quadrant
Contact:

Re: 2014-15 MAX Ratings

Post by talkingpractice » Sun Dec 07, 2014 7:12 pm

I disagree with so much of the quasi-mathematics in this thread (and its predecessors) that I don't know where to begin.

So I shall just shush, get out of your thread, and let you enjoy the last words.

Live long and prosper.

permaximum
Posts: 413
Joined: Tue Nov 27, 2012 7:04 pm

Re: 2014-15 MAX Ratings

Post by permaximum » Sun Dec 07, 2014 7:18 pm

v-zero wrote: So how would you do this? To you what does in-sample mean? I'm willing to bet it doesn't mean to you what it does to me.

Curious, then, that you should care about any sort of contest. I dare say your confidence is misplaced.

Ah, I think a lot of these people would speak up, but do not feel that it is time well spent. I don't feel this is time well spent, but I am very easy to bait in such situations. J.E. has already spoken up previously, in essence to say 'the vast majority of you clearly don't know the difference between in-sample and out of sample', for which he got an impetuous response from Neil.
1. I know what's out-of-sample, so do not assume things. You elitists think that you are the only ones that know anything related to this field in the world. Out-of-sample and in-sample prediction accuracy difference should be noticable. Otherwise there's no sample problem for the model. Even if someone used all the data available at the point just before he gave the metric ratings to Neil, there's been a quite some time. 10-15 games should be enough. So, he just needs to check "normalized rmse" of those games that played recently and normalized rmse of previous seasons. If there's a noticable difference in prediction accuracy in those games or at "some point" before this season you'll know the sample's border.

Sometimes "simple logic" is all you need.

2. If it's misplaced, prove it or do not talk. If it's what it is I can say "I bet your metric is worse than MPG+HCA."

3. If the time is not well spent, then do not talk. If you're too lazy to prove your words, do not talk. Why are you wasting time with all the talk?


This elitist attitude is the biggest problem here. All I hear are lots of flashy words but only a few do something useful that an outsider can see. No surprise that the fan vote on wins for the new season beat lots of "experts".

@talkingpractice

More talk?

From now on I won't respect anyone unless I see his work here. It's simple as that. If he can't share it then he shouldn't come and blame me for something. It means he simply can't back up his words.

My blend is here. Beat it without using previous seasons and APM variant then we can talk. I'm getting sick of this "I created those mountains" attitude that most of the people seem to have here. But when you ask "Good then show me your work"... Naaa, I can't do that. The reason can be ethics, money, hope for a job etc.

Update: Player rankings are updated.

permaximum
Posts: 413
Joined: Tue Nov 27, 2012 7:04 pm

Re: 2014-15 MAX Ratings

Post by permaximum » Mon Dec 08, 2014 2:00 pm

Rankings are updated. Kobe Bryant is 170th with -0.29 now.

mystic
Posts: 470
Joined: Mon Apr 18, 2011 10:09 am
Contact:

Re: 2014-15 MAX Ratings

Post by mystic » Mon Dec 08, 2014 3:24 pm

permaximum wrote: 1. I know what's out-of-sample, so do not assume things. You elitists think that you are the only ones that know anything related to this field in the world. 1.1 Out-of-sample and in-sample prediction accuracy difference should be noticable. Otherwise there's no sample problem for the model. Even if someone used all the data available at the point just before he gave the metric ratings to Neil, there's been a quite some time. 10-15 games should be enough. So, he just needs to check "normalized rmse" of those games that played recently and normalized rmse of previous seasons. If there's a noticable difference in prediction accuracy in those games or at "some point" before this season you'll know the sample's border.

Sometimes "simple logic" is all you need.

2. This elitist attitude is the biggest problem here. All I hear are lots of flashy words but only a few do something useful that an outsider can see. 2.1 No surprise that the fan vote on wins for the new season beat lots of "experts".

3. The reason can be ethics, money, hope for a job etc.
1. If that is the case, why do you still have seasons listed as "out-of-sample", which wouldn't be eligible for Neil's proposed test, but would be considered in-sample?

1.1 Not necessarily true based on the used sample (assuming you have figured out which seasons would be in-sample and which would be out-of-sample in the first place). The accuracy of the prediction is varying from season to season based on the shifting in minutes and roles for players. The funny thing is that the season-to-season average shifts are bigger than the differences between metrics in terms of prediction accuracies (based on the 2012 retrodiction test results). Therefore, your proposed idea of checking that wouldn't be sufficient. Sometimes the supposed "simple logic" isn't all you need.

Well, but there are some things, which can give a pretty accurate assessment of the in-sample/out-of-sample issue. One would be checking how stable the predicition accuracy is over the years. Meaning, checking not just the y1 performance, but comparing that to the y2, y3 and y4 performance as well. A metric, which has "less" issues with in-sample stuff should be more stable than those with a bigger issue (see Neil's result for PER and ASPM for example, PER shows more stability than ASPM, Edit: to add here the yr-to-yr variance from the retrodiction test as well: PER has a relative std of 8.9 while ASPM has 16.5). You can also use a simple mpg-based metric to crosscheck the yearly variance. If the yr-to-yr performance by a metric is clearly different than that of that simple metric, it indicates such issue with in-sample exists.

The overall question is: What do you want to achieve? Being able to evaluate players better or just have a metric performing well in a specific test setting? From your recent posts I have the impression that you are more about the latter, while falsely assuming that this would per se equal the former. Also, and that is a point v-zero made before, such a blend may perform exceptional well in such a test, but it really does not give the same insight to improve an individual metrics. Take Crow in the prediction thread for example. Sure, he is leading by a comfortable margin, but how is he able to say with great certainty what caused the existing errors? I can simply check it out and can say why my prediction is off for specific teams, which gives me a really good look at possible improvements in different areas, and therefore can help me improving the underlying method. I see that the minute prediction is off, especially for those injured players (not a surprise), but I also see that the part based on expected player developments isn't particular good and provides a lot of prediction errors. Crow can look afterwards which other prediction was more helpful, but that doesn't mean that this will be the same next year, given that seldom the exact methods for such predictions are revealed (or comparable from season-to-season) or just simply based on "guesses". That is a severe lack of insight given by such "blends".

2. There is no big issue with "elitist attitude", there is a bigger issue with someone always feeling offended when his "work" is critized. Everytime someone points out an issue, you reply with a snarky comment and proclaim that the critique isn't necessary, because you would have already known that. If you really know that, why do you constantly contradict your own knowledge? It starts with the in-sample/out-of-sample issue, where you specifically created a "metric", which would make it incredible tough to find sufficient years for an out-of-sample test as the one proposed by Neil? Why not just create one which does not have that problem in the first place, if you understand what in-sample and out-of-sample means?
2.1 Why do you often use logical fallacies in your argumentation?

3. There are people who do not need your approval or that of anybody else. Also, the quality of the arguments is independent from the "work" someone is presenting on the internet. The argument is either valid or not. It is again a logical fallacy by you that you believe that an argument is more valid coming from a person who presented his work to you. You should rather evaluate the argument based on the merit of the argument itself. (Let alone that addressing talkingpractice with such flawed thinking is hilarious, to say the least, given that they actually presented their work and are doing not only the apbrmetrics community a great favour by supplying us with prior informed RAPM based on their likely high quality raw data, but also every fan who can appreciate good "work with numbers". v-zero also jumped in when J.E. took his RAPM numbers down in 2013. I found that to be very generous, and if you really need the "work" to back up the "words", there would be a starting point.)

permaximum
Posts: 413
Joined: Tue Nov 27, 2012 7:04 pm

Re: 2014-15 MAX Ratings

Post by permaximum » Mon Dec 08, 2014 4:46 pm

1. If that is the case, why do you still have seasons listed as "out-of-sample", which wouldn't be eligible for Neil's proposed test, but would be considered in-sample?
I'm sorry but I couldn't read further. If you can't get that I used 1980-11...2012-13 data in the regression, I don't know what to say. Besides it looks you don't know how did DSMok1 come with BPM since you think 2013-14 should be in-sample too.

Crow
Posts: 6245
Joined: Thu Apr 14, 2011 11:10 pm

Re: 2014-15 MAX Ratings

Post by Crow » Mon Dec 08, 2014 4:56 pm

It was a side issue above but I'll just say that my blend with adjustments was soley aimed at winning a contest where adjustment was within the rules. However I can tell what the blend said and what adjustments I made after and thus can tell what the value of the adjustments are distinct from the blend. I could adjust the blend too and see the impact, usually small since no metric / prediction had more than a 15% weight.

Separately I still support pure metric blends for player measurement and prediction. I mentioned a few blends that Neil could look at if he wants. A metric blend is indistinguishable in results from a new metric that happens to be exactly equivalent to the blend so mathematically it is not a different beast in essence, just in derivation. I know that ideal weights can change and earlier suggested that ideally I would study at least the last five years to try to set them effectively. If the metric contest comes off, a better metric blend may be possible from using the data set created and the evaluative scores. It is not the same nature of work and timing, ultimate credit would be shared but it might end up legitimately producing better predictions. If we get to the point of having scores for the building blocks. If the goal is prediction in X plus 1, it may be that updating the metric is advantageous. Maybe, if done well. Or maybe if league variance is mostly noise not signal, it might be better not to change. That is something we might know 3-5-7-15 years down the line.

Post Reply