APBR Retrodiction Challenge for Summer 2011

[J.E.]

Are the Mean Error & RMSE values just based on per-game results?

They're based on actual vs expected end-of-season team efficiency differential

How can you predict team efficiencies without using individual player rates?

Is this question directed at me? I am confused

[Mike G]

I would strongly suggest that you use actual 2010-11 results for rookies, otherwise the contest will depend heavily on who has the best rookie predictions, which I don't think is the intent here.

I don't think that's really possible, because we are predicting team efficiencies/lineup efficiencies (not individual anythings). Since we are not predicting individual numbers, I think we'd best use a set value for the rookies.

This is deeply buried in the thread -- top of page 3 -- after a sudden flurry of posts.

[Crow]

To my understanding the Retrodiction Challenge assumes that the individual impacts on team and lineup efficiencies in the current season are measured in terms of impact on net scoreboard point differential impact per possession and will be found by / estimated as the average such individual impact last season. With that assumption, the metrics are then judged by how well they predict team efficiencies/lineup efficiencies in the current season. Individual and team impacts are both projected in a sense but the metrics are compared at the level of how well they predict team level results. The individual impacts are and will always be just an estimate, the team projection can be scored against a concrete actual season level net efficiency or point differential.



(If one was interested in game level predictive power instead of season level one would probably need to begin to try to go beyond fixed average individual impact estimates as building blocks for team performance projections and predict individual contributions in a context based on such contributions in a similar or fairly similar context and perhaps team and lineup level responses to such context where individual level response to context is considered too hard to capture discretely.)

[Mike G]

OK, so I still don't see the logic in DSMok1's statement that it's not possible to use actual 2011 rookie impacts or rates, since we'd be predicting team results. Those seem to be entirely separate issues.

If last year's average rookie performance is used to predict this year's rookie rates, these rates may impact different metrics in different ways.
Conversely, if this year's actual rookie rates are used, then we're getting a clean comparison between metrics attempting to predict last year's player rates to this year's rates.

[EvanZ]

If last year's average rookie performance is used to predict this year's rookie rates, these rates may impact different metrics in different ways.

How so? It's not clear to me why that would be the case.

[DSMok1]

OK, so I still don't see the logic in DSMok1's statement that it's not possible to use actual 2011 rookie impacts or rates, since we'd be predicting team results. Those seem to be entirely separate issues.

If last year's average rookie performance is used to predict this year's rookie rates, these rates may impact different metrics in different ways.
Conversely, if this year's actual rookie rates are used, then we're getting a clean comparison between metrics attempting to predict last year's player rates to this year's rates.

There are no actual rookie rates to use, Mike.

We are predicting P1, P2...P10, and summing them to get Predicted Eff.Dif.

We are comparing against Measured Eff.Dif.

We do not know, and have no way of knowing precisely, what ACTUAL P1 was. So we can't use the actual rates. What we do have, and it is what we are measuring against, is the actual Measured Eff. Dif.

Since we do not know what the rookies actually did, it seems best to me to assign a generic number for them so they will not be effecting the evaluation.

[Mike G]

Still not getting it.
Why don't you have 2011 rookie rates, or know what they actually did? The season is complete.

[EvanZ]

Still not getting it.
Why don't you have 2011 rookie rates, or know what they actually did? The season is complete.

We all have rookie ratings. Should we use yours? Daniel's? Jerry's? If you think it makes a difference, wouldn't that choice bias the challenge even more? As Daniel implied, there are no "actual" ratings. If there were, we wouldn't be doing this, as I'm sure you would agree.

I would argue (until you or someone else can convince me otherwise) that it doesn't matter (mathematically) whether we choose an arbitrary metric or any arbitrary value (i.e. "0" or "-1") for rookies, as long as we all use the same thing. It's easiest just to assign one number to all rookies, so why not do that?

[Mike G]

If I want to retrodict by my metric, I'd be inclined to use my ratings. I'd assume you'd use yours, as would anyone -- ??

DeMarcus Cousins is rated at .022 WS/48 (b-r.com), and I have him at .131 eW/48 . That's quite a difference, and in each case I assume his rating could be incorporated along with others to retrodict that the Kings win about 27 games (-5.3 PPG differential).
The rates are not interchangeable, though; and I couldn't begin to guess how to reapportion his teammates' rates to accommodate such a radical difference. Nor can I imagine what would be produced by this accommodation.

eWins would be tested by using DMC's 2011 eWins and minutes, along with his veteran teammates' 2010 eWins and 2011 minutes.

[DSMok1]

If I want to retrodict by my metric, I'd be inclined to use my ratings. I'd assume you'd use yours, as would anyone -- ??

DeMarcus Cousins is rated at .022 WS/48 (b-r.com), and I have him at .131 eW/48 . That's quite a difference, and in each case I assume his rating could be incorporated along with others to retrodict that the Kings win about 27 games (-5.3 PPG differential).
The rates are not interchangeable, though; and I couldn't begin to guess how to reapportion his teammates' rates to accommodate such a radical difference. Nor can I imagine what would be produced by this accommodation.

eWins would be tested by using DMC's 2011 eWins and minutes, along with his veteran teammates' 2010 eWins and 2011 minutes.

We're trying to figure out which is right, not which correlates best year-to-year. As such, measuring predicted against actual team efficiency differential is the only way to go. Can't break it down to the player level, since we don't know how good (e.g.) DeMarcus Cousins actually was. We need to measure against actual, not an estimate of actual.

[EvanZ]

DeMarcus Cousins is rated at .022 WS/48 (b-r.com), and I have him at .131 eW/48 . That's quite a difference, and in each case I assume his rating could be incorporated along with others to retrodict that the Kings win about 27 games (-5.3 PPG differential).
The rates are not interchangeable, though; and I couldn't begin to guess how to reapportion his teammates' rates to accommodate such a radical difference. Nor can I imagine what would be produced by this accommodation.

It's a difference, but that's beside the point. As long as we all agree to a set value, our metrics are on equal footing. Why do you want to use 2011 ratings for rookies? That defeats the purpose of retrodiction, since we wouldn't have those values before the start of the season.

I think I understand why this is an issue for you, though, because eWins, perhaps, depends on having those rates. If that is the case, how would you go about making a *prediction* for 2012?

[bbstats]

I just had a great idea -

can someone PLEASE plug in the following

1) Berri's 09-10 numbers converted to points, somehow
and then
2) Multiyear RAPM ending in 09-10


?

[J.E.]

I just had a great idea -

can someone PLEASE plug in the following

1) Berri's 09-10 numbers converted to points, somehow
and then
2) Multiyear RAPM ending in 09-10


?

Not exactly what you wanted, but close
http://sportskeptic.wordpress.com/2011/ ... e-results/

[Crow]

Jerry declined to score my meta-metric blend proposed earlier in this thread. It was a quick, starting point / test weighted blend of 40% ASPM, 25% LambaPM, 20% bbstats' entry and 15% EZPM based on their reported strength. RAPM was left out partly because I didn't immediately have match-up by player ID like I had with the others and because it was not a leading performer so I relied on lambdaPM in its place. The decline to score it alongside the others, if I understand correctly, was because of use of this after the fact performance information. Ok, his choice and I can understand his perspective technically for the specified thread challenge.

My goal though was finding a better performing metric, not winning the thread challenge. There can probably be better blends anyways, especially if the optimal blend is systematically searched for as Jerry suggested by pm he could, if he wished.

I still hope that meta-metric blends will eventually be tested "fairly" side by side against pure metrics by someone, somewhere to check if they can improve performance, which is the goal.

Perhaps using the newest, very best RAPM blended with some other metrics (including an expected forthcoming new version of ASPM). With or without rookies.



Jerry, do you intend to post the score of your newest RAPM using RAPM priors? I would be interested in seeing that performance level.

Is a search for an optimal meta-metric blend something you currently intend to get to eventually?

[Crow]

No rush especially given your move DSMok1 but you plan to eventually use your new ASPM for this exercise?

bbstats would you have enough time & interest to develop and submit a Adjusted Player dWin% entry based on either your current approach or perhaps a later regularized version?

Any other submissions planned?


The challenge structure was a starting point but after phase 1 is over, whenever that is, there would still be room for a phase 2 for refining metrics or tuning them or blending them or rethinking them or whatever. It would be positive if the work continued instead of ending here or soon. If folks want to.


I noticed BB Reference and WOW both posted meta-metric blends fairly recently. Simple 3 and 4 metric blends, not tuned in to fit a set of results, just offered for consideration, as mine was.

While these data posts as is do not fit into the framework of this challenge of using 09-10 to predict 10-11, it could be done if either author were inclined and scoring was given. In the name of examining performance and trying to help build better predictors.

Do blends, or at least the better / thoughtful blends, tend to regress to the mean / regularize the player findings and does that improve the team performance predictions or not? Do they tend to reduce the impact of outlier weights within metrics or does it increase the number of noise production sources and on net increase the average amount of noise? It would take some comparable scoring of blends to gauge if it is a frequently helpful approach or not. Of course there will be better and worse blends as there are better and worse metrics and better and worse weightings.

If that manual search is considered too tedious to offer & score then I'd still hope that an optimal blend would be searched for and found systematically from the entries or other metrics. Seems like an appropriate eventual step in this line of interest, to me.