2022-23 team win projection contest

[Crow]

Alright, thanks for the replies.

(I saw a flattening method article that used medians, a choice for them but not relevant here.)

So minutes from last season matter, a proxy for projection but not projection per se.

Entries and exits from top 9 on minutes or matter but are not projected. Share of current minutes can change / matter but are not projected.

[v-zero]

Yeah. I do think it could easily be improved by trying to better guess at changes in player minutes from one season to the next, but the flattening of the averages definitely helps in this regard.

For what it's worth my daily predictions do use a minutes allocation algorithm.

[Crow]

5 of 6 ahead of me by average error fell back from Mar. 24 to Mar. 27, while I improved modestly.



For recent 8 year period, the top averaged about 5.5 on average error. Bottom about 7.5.

[DarkStar48]

... it’s actually a consistent finding in the forecasting literature that the simple arithmetic mean (i.e., with equal weighting) of different forecasts often provides highly accurate overall predictions...

Of course, not all forecasts are improved by sheer number of forecasters. The ESPN experts I think numbered 15-20, but they aren't among the leaders.
avgA is the error produced by averaging the predictions of our 6: emin, ncs, dtka, trzu, vzro, and Crow.

Obviously, the avgA “entry” composed of APBR.org submissions is different from a true ensemble average.

The ensemble is incredibly useful in reasonably extrapolating the highly uncertain outcome of a final team record.

Moreover, the robustness of ensemble projection is driven by:
a) Each component has equal weight, so a prediction isn’t given precedence over the others.
b) The combination is based on a large number of individual forecasts with each consequently having a relatively small weight, such that no single one overwhelms the overall mean.

(See the full “Average” ensemble from the 2022-23 season. Link: https://i.imgur.com/XDACgyj.png)

[Mike G]

That link show's last year's results, and the Average ranked 7th of 30. I'd guess that might be typical.
This year, avg of the whole field of entries would rank 3rd right now in MAE -- dropping one spot, just ahead of Pelton. And 4th in RMSE. So yeah, it seems kind of immune to overall 'bad judgement'.

In the exercise I posted 2 days ago (Mar.28), only 5 of the other entries improved the avgA error by being (individually) included; 8 made the error worse. Not counting dummy entries.

Can you explain what makes a "true ensemble avg"?

[Mike G]

Just 3-4 games left per team.

Code: Select all

.   avg err   rmse   r^2          avg err   rmse   r^2
dtka   4.90   5.90   .67      EBPI   5.94   7.30   .51
avgA   5.24   6.21   .63      MPra   5.96   7.20   .63
vzro   5.41   6.83   .55      538R   6.22   7.78   .49
KPel   5.53   6.50   .58      TmRk   6.53   7.69   .51
LEBR   5.53   6.23   .62     vegas   6.54   7.75   .52
DRKO   5.62   6.59   .58      nuFi   6.65   8.01   .49
ncs.   5.75   6.81   .55      22Re   6.67   8.02   .36
emin   5.80   7.30   .53      EExp   6.68   8.07   .50
Crow   5.80   6.58   .62      AnBa   6.71   7.94   .53
BIPM   5.83   7.19   .67      4141   7.32   9.98   
trzu   5.89   6.76   .61      538E   7.59   9.05   .31

UPDATE Apr.6

Code: Select all

.   avg err   rmse   r^2          avg err   rmse   r^2
dtka   4.81   5.79   .69      MPra   5.84   7.04   .64
avgA   5.07   6.10   .65      EBPI   5.85   7.23   .52
vzro   5.36   6.83   .56      538R   6.17   7.72   .51
DRKO   5.48   6.46   .60      TmRk   6.44   7.60   .52
KPel   5.48   6.52   .59     vegas   6.44   7.66   .53
LEBR   5.49   6.19   .63      EExp   6.57   7.97   .51
emin   5.64   7.19   .55      nuFi   6.60   7.96   .50
Crow   5.67   6.43   .64      22Re   6.65   8.09   .36
ncs.   5.70   6.75   .57      AnBa   6.70   7.87   .54
trzu   5.71   6.60   .63      538E   7.43   8.99   .32
BIPM   5.78   7.12   .67      4141   7.52  10.11   

BIPM and EExp with their smallest errors of the season.

[v-zero]

Can you explain what makes a "true ensemble avg"?

I guess what Darkstar means is that it isn't an ensemble that includes all participants. That in no way makes it not an ensemble, though, an ensemble is simply an average (equally weighted or otherwise) of predictions/models which have not been created within the exact same framework (i.e. there is at least some variability between them). Ensembles work because they in general reduce both the bias and variance of the predictions when compared to the individual models. However, it is important to note than an ensemble of strong predictors (like you have created) would not benefit from the addition of considerably weaker individual predictions.

[DarkStar48]

There is no hard and fast rule in modelling, but I think that a true ensemble combines all reasonable forecasts that are available from different component methods.

The projections need to be non-absurd, i.e.:
-prediction means are not highly irregular, for example, total team wins that are significantly greater than 1230 for an 82-game schedule — slight allowance for asymmetric distributions.
-confidence intervals (when available) do no exceed the possible outcomes, like team wins greater than 82 or less than 0 in a regular NBA season.

The components should include, if possible:
-expectation models (such as, popular polls, betting markets, expert judgments, etc.)
-statistical projections (that is, retrospective, prospective or mixed)
-naive forecasts (e.g., the 4141 “projection”)

[v-zero]

That's ok as your personal definition of an ensemble, but it isn't what ensemble means in a machine learning/statistical framework.

For instance, every day I (well, one of my servers) build lots of prediction models. The majority get discarded during a validation process. What remains makes up an ensemble.

[DarkStar48]

@v-zero, I agree.

However, in my experience, it is very easy to make arbitrary decisions on which projections to select and which ones to leave out.

Therefore, as long as a forecast "makes sense," my inclination is to include it (even "bad" ones), so as to make the ensemble in the long-run more robust in response to novel scenarios and data not in the training set.

Essentially, more or less the same logic of gradient boosting.

Still, there can be some benefit in the near-term to weeding out the weaker models to combine the "fittest" ones that survive.

[Mike G]

Final standings!

Code: Select all

.   avg err   rmse    r^2          avg err   rmse    r^2
dtka   4.67   5.63   .694      BIPM   5.74   7.13   .672
avgA   4.90   5.86   .662      MPra   5.93   6.99   .647
vzro   5.14   6.61   .568      538R   6.10   7.61   .509
LEBR   5.20   5.97   .645      TmRk   6.14   7.40   .537
DRKO   5.26   6.15   .620     vegas   6.15   7.48   .541
Crow   5.33   6.20   .652      EExp   6.33   7.75   .528
ncs.   5.39   6.47   .585      nuFi   6.42   7.72   .516
KPel   5.40   6.34   .587      22Re   6.50   7.83   .370
emin   5.47   6.93   .567      AnBa   6.60   7.79   .546
trzu   5.49   6.42   .637      538E   7.23   8.68   .343
EBPI   5.63   7.05   .531      4141   7.47   9.86   

dtka, Crow, and EExp end up with their lowest errors of the season.

[Crow]

Congrats to dtka for the win.

Congrats to V-zero for next best contest entry and all of us in the 2nd best overall Apbrmetrics average.

Thanks to Mike for the scorekeeping.


I finished 6th / 5th / 4th across the 3 measures. Only 2 were better on all 3 and only 2 more better on 2. Not quite what I hoped but pretty decent.

[Crow]

As noted in mid-season, Pelton projected only one 50 plus game winner. The 8 year average was 6 and it turned out to be 6 this season. The long time leader was highly regressed and appeared to benefit from that but in the end finished 8th partly because of that.

[v-zero]

The extent to which Pelton's predictions fell back, admittedly by a fine margin in a competitive pack, does surprise me.

Whilst I did well on the absolute error, I feel it was more a fluke, I wasn't all that close to the top on RMSE, into which I put more weight.

[Mike G]

This season, team avg departure from 41 wins was 6.95. Almost everyone went beyond that in their predictions.

Code: Select all

depart.-41   err
KPel   6.0   5.40
vzro   7.1   5.14
LEBR   7.4   5.20
EBPI   7.6   5.63
ncs.   7.6   5.39
DRKO   7.7   5.26
538E   7.7   7.23
dtka   8.3   4.67
538R   8.7   6.10
emin   8.7   5.47
nuFi   8.8   6.42
TmRk   9.1   6.14
Crow   9.3   5.33
trzu   9.3   5.49
vegas  9.4   6.15
EExp   9.5   6.33
AnBa   9.5   6.60
BIPM  10.0   5.74
MPra  10.1   5.93

The positive correlation is .34

Pelton lost his lead when the Pacers, Blazers, Wizards kept losing, and the Dubs, Lakers, Sixers started winning.