Continuation of one metric prediction test discussion

[Crow]

I agree optimal blending is indeed best as a second step.

But as for the power of blending being old news, not outside here in last week really.

Has there been a single major media story about metric blends (beyond a simple explanation of XRAPM)?

Has there been a Sloan paper or even the slightest mention of the concept?

Has a single GM or analytic shop manager ever been asked about it or said anything about it independently? (excluding Jon Nichols' website before he was hired the first time)

Correct me if I am wrong but I think the answer to all these questions is no.

I've been talking about metric blends here for at least 3 years, maybe 5. I probably should have posted one. Right now I am waiting on a metric comparison contest that may or may not happen.

[willguo]

Anyways, I'm curious what do you suggest to use when I bet on tonight's games? My gut feeling besides you know. Because that's the ultimate question interests me. I have a feeling you know a few things about this.

Edit:

IMO the best metric should help me beat Vegas odds by combining it with HCA+REST (+ Coach factor, still not sure on this one though)

You want a basic game prediction strategy? Take the Vegas line, shade 0.5 point against the team that is more heavily bet as determined by the line movement, don't shade if there is no line movement. I'd be willing to wager that beats Box PM and RAPM. (Someone want to retrotest this? http://www.scoresandodds.com for decent line history.)

I guess my fundamental question is - what do you want this one-metric system to do? Predict within the season? Or answer hypothetical questions like if SAC replaced Gay with Anthony? The second is obviously much harder than the first, and this is why betting limits are lower when there is injury uncertainty, and IMO spreads are much weaker when there is a trade, when a coach is fired, when there is a new injury.

Any one metric system will capture Nick Collison's value in the context of having Durant/Westbrook, and that's perfectly fine because Durant/Westbrook are back now. It doesn't matter how you apportion value between OKC's rotation, because there are few enough games where the rotation differs.

You will need to have a system that goes significantly deeper into the lineup level of analysis and matchup analysis, and consider things like diminishing returns, usage trade offs, etc. Ed Davis-Amir Johnson-Tyson Chandler-Tim Duncan-Thiago Splitter - but if you project this as a +10 lineup, then your one-metric won't pass the laugh test. Admittedly, this is unlikely, but so was the Lakers replacing an injured Bynum with Pau Gasol.

Second, defensively, you need to project match-ups at a minimum, and most likely scheme as well. Joakim Noah is fabulous defensively in most situations, but he's not quite All-NBA when he's guarding Goran Dragic, right? If your one-metric doesn't know that Noah is more likely to be guarding Dragic (last year, at least) than he would be guarding Brandon Jennings, then best of luck to you.

These are all doable - clustering is my friend. And of course you trade off sample size vs relevancy, like you do in all other statistical analysis.

Every player value is within a certain context. You need to 1) remove that context 2) put the players into a new context.

[Crow]

I've sent Neil a PM today. But maybe he'll see this instead and give us a heads up on his thoughts / plans.

[permaximum]

Anyways, I'm curious what do you suggest to use when I bet on tonight's games? My gut feeling besides you know. Because that's the ultimate question interests me. I have a feeling you know a few things about this.

Edit:

IMO the best metric should help me beat Vegas odds by combining it with HCA+REST (+ Coach factor, still not sure on this one though)

You want a basic game prediction strategy? Take the Vegas line, shade 0.5 point against the team that is more heavily bet as determined by the line movement, don't shade if there is no line movement. I'd be willing to wager that beats Box PM and RAPM. (Someone want to retrotest this? http://www.scoresandodds.com for decent line history.)

I guess my fundamental question is - what do you want this one-metric system to do? Predict within the season? Or answer hypothetical questions like if SAC replaced Gay with Anthony? The second is obviously much harder than the first, and this is why betting limits are lower when there is injury uncertainty, and IMO spreads are much weaker when there is a trade, when a coach is fired, when there is a new injury.

Any one metric system will capture Nick Collison's value in the context of having Durant/Westbrook, and that's perfectly fine because Durant/Westbrook are back now. It doesn't matter how you apportion value between OKC's rotation, because there are few enough games where the rotation differs.

You will need to have a system that goes significantly deeper into the lineup level of analysis and matchup analysis, and consider things like diminishing returns, usage trade offs, etc. Ed Davis-Amir Johnson-Tyson Chandler-Tim Duncan-Thiago Splitter - but if you project this as a +10 lineup, then your one-metric won't pass the laugh test. Admittedly, this is unlikely, but so was the Lakers replacing an injured Bynum with Pau Gasol.

Second, defensively, you need to project match-ups at a minimum, and most likely scheme as well. Joakim Noah is fabulous defensively in most situations, but he's not quite All-NBA when he's guarding Goran Dragic, right? If your one-metric doesn't know that Noah is more likely to be guarding Dragic (last year, at least) than he would be guarding Brandon Jennings, then best of luck to you.

These are all doable - clustering is my friend. And of course you trade off sample size vs relevancy, like you do in all other statistical analysis.

Every player value is within a certain context. You need to 1) remove that context 2) put the players into a new context.

Decent perspective. Thanks for all the info.

What's your opinion on the trade off assuming you have a system that captures some of the things you mention? Is sacrificing sample size greatly in favor of relevancy worth it?

[willguo]

What's your opinion on the trade off assuming you have a system that captures some of the things you mention? Is sacrificing sample size greatly in favor of relevancy worth it?

That's the million dollar question in statistics, right?

[Crow]

Jon Nichols' composite score article was 7-8 years ago and probably helped him get inside. Not sure if 82games' own simple rating preceded or followed. I recall talking about local / global impact blends back on the first yahoo version of this forum. It is surprising how slow some things are adopted as main approaches.

[v-zero]

The use of ensemble models has long been a feature of the machine learning community. I suggest anybody wanting to learn about the power of 'blends' do lots of reading in that area, and forget about basketball specific research.

[Crow]

Good suggestion.

[Crow]

Via twitter Neil's says he still intends to do metric comparison but not sure if it will be before or after New Year.

[Crow]

I have tweeted him a couple more times briefly on this with no response. If it happens, it will be on his timetable.

[nbacouchside]

I created a new version of Win Score for comparison with Alternative Win Score, which has traditionally been the best simple game to game metric.

New version is this:

Points + .2*Reb + .5*Ast + 1.7*Stl + .535*Blk - .9*FGA - .35*FTA - 1.4*TOV.

Values came from a regression on RAPM & then setting it so that everything was relative to points.

[Crow]

Another entry.

Anyone else want to run the test?

[ca1294]

I think this is a great idea! This is exactly what I was suggesting in the prediction thread.

Some thoughts:
- If the testing framework is built well, it shouldn't matter how many entries there are. So why not allow any submissions, regardless of how their calculated?
- Since the primary goal of this test would be to judge between different player metrics, we can ignore all team adjustments (e.g. Homecourt Advantage or Rest). Not including these adjustments would worsen our results, but it shouldn't hurt one submission more than another.

I'm willing to make the testing framework. I was thinking something like this:
- I make a shared Dropbox folder where everyone can put their submissions
- Submissions would be files named something like 'ca1294_01152015_1.csv' and would contain:

Code: Select all

James Harden,10.14
Chris Paul,8.80
Kevin Durant,9.45
etc.

- A script reads submissions every 3 hours or so and processes any files that have new file names (e.g. if you had ca1294_01152015_1.csv, and you wanted to make a change before the games started, you can just update the file and change it to ca1294_01152015_2.csv)
- In the same folder, the script writes a file like 'ca1294_01152015_report.csv' that lets you know if you are missing any players that are on a team playing on 01/15/2015
- The next day, the script gets the actual minute played in each game and uses ratings from each person's latest submission that made it before the games started for the day
- It writes a file every day with everyones' predicted game results, and another file with everyone's updated ranking

Some other things to account for are people missing a day. I wouldn't want to use outdated ratings which could hurt people's numbers, so I could say not do any calculations unless the date in the submission csv file matches the current day. However, people who miss a lot of games may end up getting lucky by missing the "outlier" games that skew everyones' results. But that's a minor issue that we could sort out at the end.

What do you guys think?

[Crow]

In your example script, I don't know what the values after the players name are.

Game by game prediction with opportunity for metric refinement is one kind of test, probably the cleanest but most of the previous talk was about doing many years all at once though there was argument about out of sample validity. It there resolution to this debate?

If a contest gets run, it should be fully explained in detail , get agreement from the metric producers beforehand and then proceed.

[ca1294]

In your example script, I don't know what the values after the players name are.

Game by game prediction with opportunity for metric refinement is one kind of test, probably the cleanest but most of the previous talk was about doing many years all at once though there was argument about out of sample validity. It there resolution to this debate?

If a contest gets run, it should be fully explained in detail , get agreement from the metric producers beforehand and then proceed.

Sorry, the numbers were just random. But yes I assumed the plan was for game by game prediction.

Wouldn't the contest involve more than just metrics? For example, lets say you're trying to predict the results for 2010-03-12, and the metric you're using is PER. Do you just use the players' season PERs at the end of 2010-03-11 as the prediction, or do you use some sort of weighted average to give recent games more weight? We could have multiple submissions using the same metric, but different prediction strategies.