Reconstructing Box Plus/Minus

[Nate]

...
I have been preparing for some time now to reconstruct Box Plus/Minus (BPM), with a goal of addressing the major existing issues.
...

The "major existing issue" with most of the stats that people put out is that they're trivia: They're created by and for a bunch of people who watch sports (or whatever) without any sense of the decisions that people are making, the freedom the decision makers have, or how the decision makers are going to measure success.

In baseball people paid attention to batting average instead of on-base percentage for a long time because they had some hypothesis of "a way to measure the batter" rather than by looking at what wins games. Without some idea of the decisions you're trying inform or what you care about, it's hard to have any sense that the new formula you come up with is meaningfully better than the old ones.

[DSMok1]

The "major existing issue" with most of the stats that people put out is that they're trivia: They're created by and for a bunch of people who watch sports (or whatever) without any sense of the decisions that people are making, the freedom the decision makers have, or how the decision makers are going to measure success.

In baseball people paid attention to batting average instead of on-base percentage for a long time because they had some hypothesis of "a way to measure the batter" rather than by looking at what wins games. Without some idea of the decisions you're trying inform or what you care about, it's hard to have any sense that the new formula you come up with is meaningfully better than the old ones.

The general objective of Box Plus/Minus is to assess how to credit a team's success (as measured by points) to the players who were on the court.

Certainly, decisions about things like schemes and lineups affect the success as well. This metric only looks at the players. If players were put in a bad position, they will show up as a problem, not the coach who put them in that position.

There are certain limitations to achieving the objective--since the intent is for this to extend to historical contexts, we are limited to the constraints of box score statistics.

[Nate]

....
The general objective of Box Plus/Minus is to assess how to credit a team's success (as measured by points) to the players who were on the court.
....

"Credit" comes from the latin "credere" which means "to believe." And, it really is the case that people can assign credit for wins and losses any way they believe. We could, for example, take the margin of victory (or margin of loss) divide it by the total minutes played, and then multiply it by each player's minutes. That's probably a bad way to do things, but the numbers will add up to points.

The question is, how do we decide whether it's a bad way to do things? Is it just because the formula isn't fancy enough? Is it because that approach goes against some intuition that we have about basketball? Now, we can hope to have some kind of happy accident just cooking up formulas on a computer, but it makes more sense to start with an understanding about how we decide whether a model is good or bad and work from there.

It's also worth pointing out that performance - in terms of points - is non-linear. It's harder to increase points per possession on a team that's already good than on one that's not so good. This probably isn't such a big deal in the NBA, but could easily be more significant in less competitive leagues.

[DSMok1]

....
The general objective of Box Plus/Minus is to assess how to credit a team's success (as measured by points) to the players who were on the court.
....

"Credit" comes from the latin "credere" which means "to believe." And, it really is the case that people can assign credit for wins and losses any way they believe. We could, for example, take the margin of victory (or margin of loss) divide it by the total minutes played, and then multiply it by each player's minutes. That's probably a bad way to do things, but the numbers will add up to points.

The question is, how do we decide whether it's a bad way to do things? Is it just because the formula isn't fancy enough? Is it because that approach goes against some intuition that we have about basketball? Now, we can hope to have some kind of happy accident just cooking up formulas on a computer, but it makes more sense to start with an understanding about how we decide whether a model is good or bad and work from there.

It's also worth pointing out that performance - in terms of points - is non-linear. It's harder to increase points per possession on a team that's already good than on one that's not so good. This probably isn't such a big deal in the NBA, but could easily be more significant in less competitive leagues.

We're pretty deep into the philosophy of basketball statistics here!

Well, RAPM is cross-validated to attempt to determine, by testing out of sample, how much statistical shrinkage should be added to best balance out signal vs. noise in the APM modelling. While RAPM is a crude approach, at least it is not biased towards any one box score statistic. With a multiple 5 year datasets, RAPM should provide a very solid basis. Please note--the RAPM does take into account the "playing from behind" advantage.

When the box score model is then constructed, it relies on the RAPM basis being a reasonably unbiased metric. It will be validated out-of-sample (still within the RAPM space, though).

The nonlinearity of points is a complex issue and one that is difficult to assess. More significant is the nonlinearity of skillset interactions (i.e. some skillsets scale and some don't). This metric cannot address that at all.

[v-zero]

That makes a lot of sense, Colts18. I would like to somehow do things that way, but it certainly adds a lot of computational complexity. For many leagues, that data will not be readily available.

I also don't think it can actually work if the metric is nonlinear. For a linear metric, it would be a very good choice. Effectively, you would have to do the analysis for each game individually, since the players available vary almost every game. This means there would be a ton of outlier box score statistics, so that could only be handled by a linear metric.

Note: developing a truly linear version of Box Plus/Minus would probably be a good idea. It could then be used on individual games like SPR or DRE (https://fansided.com/2015/02/23/introdu ... le-metric/ ). Using individual games and constructing an overall number as you have mentioned above would give a nice counterpoint to a nonlinear, season-long BPM construction.

In terms of nonlinear but relatively simple interaction terms you can use the geometric mean to create linear versions of those interactions.

Going one step further you can usually create a linear term from a nonlinear one by applying some dimensional analysis to the problem.

[Nate]

I'm not sure it's a great idea, but you could try to do something analogous to Elo ratings for players using minutes played and scoring margin in some kind of Bayesian inference model, then estimate "contribution" as rating * minutes played.

So you'd do game-by-game updates, but only using minutes played and score margin. You'd also have to do some per-league calculations to estimate the variance in scoring margin per game.

This is a very different approach from using a single stat line, but splitting into per-game data should be much richer, and probably still be feasible in excel. (Almost certainly feasible in something like R.)

[mewfert]

Couple of random ideas to make BPM potentially more translatable & cross-comparable to NCAA & other leagues:

1) For the minutes variable, explicitly tie it somehow to the length of the game for a particular league. Thinking about NBA vs. NCAA, 48 vs. 40 minutes. Should the minutes input get stretched out by 20% for NCAA? Related: the 4 additional games for the "regressed" MPG could also be tied to some standard of league season length. Since NCAA teams are playing half as many games, this regression has twice as much effect.

2) For the team adjustment, rather than using pure efficiency margin use something like standard deviations above the mean. Problem here is that the EM "spread" in college is much wider than in NBA, which I believe has the effect of ultimately goosing the BPM for players on better teams. Also, different methods of calculating adjusted efficiencies can give different spreads. For example, in NCAA the BB Reference adjusted ratings have a significantly wider spread than the Kenpom ratings: 2019 Virginia is +39 at Basketball Reference and +34 at Kenpom; the 20th ranked team is +24 versus +20; similar differences at the other end.

Anyhow, thanks for doing this and looking forward to seeing what you come up with!

[DSMok1]

Couple of random ideas to make BPM potentially more translatable & cross-comparable to NCAA & other leagues:

1) For the minutes variable, explicitly tie it somehow to the length of the game for a particular league. Thinking about NBA vs. NCAA, 48 vs. 40 minutes. Should the minutes input get stretched out by 20% for NCAA? Related: the 4 additional games for the "regressed" MPG could also be tied to some standard of league season length. Since NCAA teams are playing half as many games, this regression has twice as much effect.

The MPG variable is a little unusual, because it doesn't represent anything at all from a game-state perspective. In fact, it's more of a prior than anything, representative of "the little things that a coach picks up on that aren't shown in the box score." It's highly correlated with a number of the other inputs.

I would love to come up with some better way to handle playing time. Playing time very clearly adds information beyond the box score statistics, and the regression fits reflect that. That is why it was included.

That said, for a true linear "BPM-style" statistic, MPG should not be included (i.e. one that could be used on a one-game sample.) It does not stabilize very quickly at all compared to other statistics since it is measured each game rather than each possession.

I welcome thoughts on the meaning, interpretation, and usefulness of playing time as an input.

2) For the team adjustment, rather than using pure efficiency margin use something like standard deviations above the mean. Problem here is that the EM "spread" in college is much wider than in NBA, which I believe has the effect of ultimately goosing the BPM for players on better teams. Also, different methods of calculating adjusted efficiencies can give different spreads. For example, in NCAA the BB Reference adjusted ratings have a significantly wider spread than the Kenpom ratings: 2019 Virginia is +39 at Basketball Reference and +34 at Kenpom; the 20th ranked team is +24 versus +20; similar differences at the other end.

Anyhow, thanks for doing this and looking forward to seeing what you come up with!

This is an interesting discussion. At it's core, a pure efficiency margin should be straightforward to replicate--as long as you are not using priors. Once priors are included, or garbage time is excluded, it gets much less replicable.

The very center of the BPM method is that it should sum to the team's overall efficiency differential, as the best unbiased measure of the team's overall strength.

I do not necessarily feel that a properly constructed BPM will have the same issue of over-valuing players on good teams. I think that a poor handling of a few of the input variables is messing with outlier statistics that are compiled on some of the good NCAA teams.

[DSMok1]

I'm not sure it's a great idea, but you could try to do something analogous to Elo ratings for players using minutes played and scoring margin in some kind of Bayesian inference model, then estimate "contribution" as rating * minutes played.

So you'd do game-by-game updates, but only using minutes played and score margin. You'd also have to do some per-league calculations to estimate the variance in scoring margin per game.

This is a very different approach from using a single stat line, but splitting into per-game data should be much richer, and probably still be feasible in excel. (Almost certainly feasible in something like R.)

This approach would work well with a purely linear metric. In fact, it should be done on a play-by-play basis ideally. I don't have the time to implement such an approach, but it is certainly a worthy avenue to research.

[Crow]

The "major existing issue" with most of the stats that people put out is that they're trivia: They're created by and for a bunch of people who watch sports (or whatever) without any sense of the decisions that people are making, the freedom the decision makers have, or how the decision makers are going to measure success.

That is a sweeping opinion.

MOST without ANY sense of the decisions people are making?

Does that include you? Who specifically are exceptions? Do you mean MOST or do you really mean All?

Why the heck hasn't a coach or player ever ever stepped in and revealed the perspective they have that most everybody else lacks and needs? Why aren't they sharing it with the world anywhere audibly where a "non-intelligent" fan or analyst has ears open? Does this mystical secret insider knowledge exist? Why isn't shared?

Trivia is defined as details, considerations, or pieces of information of little importance or value. Are you saying most or everything in performance metrics is trivia? I'd disagree. What is trivia and what isn't? Who knows? Does RAPM bother with or credit any trivia? I assume you are alleging BPM credits trivia. Which specific items, if you know. If you don't, can you name one player or coach that does? Have they ever spoken of this truth? Where / what is truth in the sea of alleged trivia?

Without sense of the decisions that people are making...name the decisions that people are making that aren't in metrics or player tracking and aren't trivia. What if anything is non-trivia that is missed by RAPM and on what basis is that claim made if made? If not made, should I assume that RAPM is based on non-trivia and completely captures the non-trivia?


what freedom do the decision makers have that is being ignored or completely missed?

how are the decision makers going to measure success that fans, analysts, metric makers are going to miss in their measures of success that aren't style or trivia? Name the ways.

These opinions need expansion and justification. Or they are not helpful to me.

[Crow]

I'm not sure it's a great idea, but you could try to do something analogous to Elo ratings for players using minutes played and scoring margin in some kind of Bayesian inference model, then estimate "contribution" as rating * minutes played.

So you'd do game-by-game updates, but only using minutes played and score margin. You'd also have to do some per-league calculations to estimate the variance in scoring margin per game.

This is a very different approach from using a single stat line, but splitting into per-game data should be much richer, and probably still be feasible in excel. (Almost certainly feasible in something like R.)

How is this better than the RAPM approach? Do they avoid relying on trivia? How did you come up with it? Are minutes 100% free of trivia? How, why? Is scoring margin free from trivia and free from trivia at the individual player level? Is this raw scoring margin unadjusted by quality of players on court or amount of time with large leads / deficits?

Have metric markers made a mistake ever going beyond minutes and scoring margin? If there is any non-trivia beyond that, what is it and why didn't you include it in the suggested approach? If there any non-trivia beyond that, is it going to go undifferentiated forever and unused by analysis? How do the holders of knowledge of the secret non-trivia know it is there without differentiation and before your proposed approach? Eye test? Channeling the beautiful music of basketball beyond the eye test, based on divine favor or 10,000 hours in the gym?

[Crow]

General question: should metrics be adjusted by home court advantage?

Harder to do: theoretically should metrics be adjusted by team standings, beyond quality of players in a specific player's floor stints?

[DSMok1]

General question: should metrics be adjusted by home court advantage?

Harder to do: theoretically should metrics be adjusted by team standings, beyond quality of players in a specific player's floor stints?

For specific home court advantage? BPM just uses adjusted efficiencies, which in essence include a generic average home court advantage.

What does the second question mean?

[Crow]

If RPM and BPM are adjusted for asserted change of player behavior / performance under certain GAME conditions (big leads, big deficits), wouldn't it be likely players behave / perform differently based on SEASON level conditions (comfortably #1 seed, in playoffs safely, fighting for playoffs, realistically out of playoffs)?

[Crow]

If winning games is the only non-trivia, and that is implied by Elo Ratings, why not take it partially or entirely to playoff game victories or perhaps to title level?