2016-17 Team Win Projection Contest / Discussion

[Mike G]

Since b-r.com's page is blank today, here we are relative to the 538 forecast

Code: Select all

1-21   RMSE      avg abs err             SMRE
trzu   4.44      BaDo   3.55      BaDo   2.50
lnqi   4.57      RyRi   3.63      RyRi   2.91
RyRi   4.65      lnqi   3.66      amp5   2.96
amp5   4.70      amp5   3.67      kmed   3.09
ATCt   4.83      trzu   3.67      trzu   3.14
sbs.   4.84      sbs.   3.85      sbs.   3.16
BaDo   4.90      kmed   3.90      lnqi   3.16
vegas  5.01      cali   3.98      Crow   3.23
Crow   5.04      vegas  4.01      cali   3.25
Mike   5.13      ATCt   4.01      shad   3.34
kmed   5.18      Crow   4.03      vegas  3.41
cali   5.19      Mike   4.07      yoop   3.42
ncsD   5.27      yoop   4.21      Mike   3.48
taco   5.30      shad   4.26      ATCt   3.51
ncsB   5.35      nrfo   4.33      jg34   3.52
nrfo   5.39      jg34   4.36      538    3.56
sndi   5.39      sndi   4.41      nrfo   3.68
shad   5.43      ncsD   4.41      ncsD   3.72
jg34   5.59      ncsB   4.46      sndi   3.74
yoop   5.65      taco   4.46      ncsB   3.77
GK5.   5.75      538    4.59      taco   3.81
538    5.91      GK5.   4.79      GK5.   4.15

https://projects.fivethirtyeight.com/20 ... edictions/
EDIT: Andrew Johnson's numbers: 4.99, 4.24, 3.78

Back to b -r.com

Code: Select all

1-22   RMSE      avg abs err             SMRE
lnqi   4.31      lnqi   3.27      BaDo   2.35
RyRi   4.34      BaDo   3.31      Mike   2.74
amp5   4.38      amp5   3.37      lnqi   2.75
trzu   4.40      Mike   3.39      amp5   2.79
sbs.   4.55      RyRi   3.47      sbs.   2.97
ATCt   4.66      sbs.   3.57      RyRi   3.00
BaDo   4.68      trzu   3.58      shad   3.06
Mike   4.71      kmed   3.74      trzu   3.09
AnJo   4.73      vegas  3.77      kmed   3.11
vegas  4.80      ATCt   3.77      nrfo   3.17

http://www.basketball-reference.com/fri ... f_prob.cgi

Code: Select all

1-23   RMSE      avg abs err             SMRE
lnqi   4.18-     lnqi   3.17      BaDo   2.32
RyRi   4.21-     BaDo   3.20      lnqi   2.70
trzu   4.27-     amp5   3.29      amp5   2.74
amp5   4.27-     RyRi   3.34      Mike   2.74
sbs.   4.41      Mike   3.36      RyRi   2.83
BaDo   4.51      trzu   3.45      sbs.   2.89
ATCt   4.54      sbs.   3.45      trzu   2.96
Mike   4.61      kmed   3.62      kmed   2.96
AnJo   4.64      vegas  3.65      shad   3.05
Crow   4.65      ATCt   3.65      Crow   3.06

The 4 best RMSE today are better than any previously.

[tarrazu]

BR and 538 projections look pretty similar at this point but with BR more condensed/smaller range.

[Mike G]

For a few days, BasketDork had an astonishing lead in the 3rd column, SMRE -- some .40 better than anyone else.
Yesterday I found he had 7 guesses within 0.3 of the b-r.com projections. Nobody else had more than 3 such direct hits.
Something like half of projections fell within 1.5 of his predictions. I wonder if this is largely luck, even as 538 gave him an equally commanding lead.
The trouble with having such perfect guesses is that either a win or a loss will move a team off that perfect spot.

Code: Select all

1/24   RMSE      avg abs err             SMRE
lnqi   4.11-     lnqi   3.10-     lnqi   2.55
RyRi   4.17-     BaDo   3.27      BaDo   2.67
trzu   4.24      Mike   3.32      Mike   2.68
amp5   4.25      amp5   3.33      amp5   2.80
sbs.   4.40      RyRi   3.34      sbs.   2.80
BaDo   4.50      trzu   3.41      RyRi   2.85
Mike   4.57      sbs.   3.44      taco   2.92
ATCt   4.57      Crow   3.58      trzu   2.93
vegas  4.59      kmed   3.60      kmed   2.93
Crow   4.60      vegas  3.61      shad   3.01
sndi   4.64      taco   3.65      Crow   3.02
AnJo   4.64      ATCt   3.70      nrfo   3.05
ncsD   4.68      shad   3.72      cali   3.06
kmed   4.68      sndi   3.77      ATCt   3.12
taco   4.72      cali   3.79      vegas  3.14
cali   4.82      nrfo   3.85      jg34   3.22
shad   4.85      jg34   3.89      sndi   3.29
ncsB   4.94      AnJo   3.99      538    3.37
jg34   4.95      yoop   4.07      GK5.   3.42
nrfo   5.02      ncsD   4.07      yoop   3.51
yoop   5.12      538    4.12      ncsB   3.59
GK5.   5.29      ncsB   4.15      AnJo   3.61
538    5.46      GK5.   4.18      ncsD   3.66

Some more season bests marked by -

Code: Select all

1/25   RMSE      avg abs err             SMRE
lnqi   4.21      lnqi   3.24      Mike   2.63
RyRi   4.27      Mike   3.31      BaDo   2.73
amp5   4.32      BaDo   3.37      lnqi   2.76
trzu   4.39      amp5   3.43      amp5   2.92
sbs.   4.50      RyRi   3.44      RyRi   2.95
Mike   4.56      trzu   3.50      trzu   2.96
BaDo   4.62      sbs.   3.57      Crow   3.01
ATCt   4.68      Crow   3.63      sbs.   3.01
sndi   4.69      kmed   3.72      kmed   3.09
vegas  4.69      vegas  3.75      shad   3.11

[nbacouchside]

well, this is the worst i've done in a while.

[Mike G]

You aren't doing worse; everyone else is doing better.
On 1/19, your D version RMSE made the top 10 at 4.83. Now at 4.78, good for 14th.
On 1/17, the B side RMSE was 8th from the bottom at 5.18; now 5.08 is 5th worst.

I keep thinking we can't all keep looking better, but it keeps going there. Phenomenal.

[sndesai1]

wow 538 has really fallen off a cliff moving to their new system this year

[Mike G]

A new season best in RMSE

Code: Select all

1/26   RMSE      avg abs err             SMRE
lnqi   4.09-     BaDo   3.207     BaDo   2.49
RyRi   4.18      lnqi   3.210     Mike   2.64
trzu   4.23      Mike   3.33      lnqi   2.76
amp5   4.23      RyRi   3.34      RyRi   2.77
sbs.   4.41      trzu   3.39      Crow   2.85
BaDo   4.47      amp5   3.41      trzu   2.89
Mike   4.49      sbs.   3.49      sbs.   2.89
ATCt   4.54      Crow   3.49      amp5   2.89
Crow   4.59      kmed   3.66      shad   3.01
sndi   4.60      vegas  3.72      kmed   3.04

Abrupt new lows:

Code: Select all

1/28   RMSE      avg abs err             SMRE
lnqi   3.94-     lnqi   3.01-     lnqi   2.50
RyRi   4.04-     RyRi   3.20      BaDo   2.63
trzu   4.04-     BaDo   3.24      Mike   2.69
amp5   4.10      amp5   3.24      shad   2.70
sbs.   4.29      trzu   3.26      RyRi   2.71
BaDo   4.37      Mike   3.37      amp5   2.73
sndi   4.45      sbs.   3.40      kmed   2.79
ATCt   4.45      Crow   3.43      trzu   2.79
AnJo   4.46      vegas  3.48      Crow   2.84
vegas  4.47      kmed   3.51      sbs.   2.93

More new lows, and an avg of our 20 guesses leads all fields.

Code: Select all

1/30   RMSE      avg abs err             SMRE
avg    3.73       avg   2.89       avg   2.34

lnqi   3.78-     lnqi   2.96-     BaDo   2.42
trzu   3.79-     trzu   3.01      RyRi   2.47
RyRi   3.86-     BaDo   3.03      trzu   2.54
amp5   3.95      RyRi   3.03      lnqi   2.54
sbs.   4.06      sbs.   3.13      sbs.   2.55
BaDo   4.13      amp5   3.14      yoop   2.65
vegas  4.20      Crow   3.21      Crow   2.66
ATCt   4.25      vegas  3.28      amp5   2.66
Crow   4.25      Mike   3.31      Mike   2.72
sndi   4.28      kmed   3.32      kmed   2.72
AnJo   4.29      sndi   3.42      vegas  2.79
kmed   4.32      yoop   3.45      sndi   2.82
ncsD   4.36      shad   3.50      shad   2.85
Mike   4.36      cali   3.52      cali   3.03
cali   4.37      ncsD   3.65      ncsD   3.10
shad   4.49      jg34   3.67      taco   3.10
taco   4.56      ATCt   3.70      jg34   3.14
yoop   4.59      taco   3.70      538    3.28
jg34   4.64      AnJo   3.78      nrfo   3.29
nrfo   4.78      nrfo   3.87      ATCt   3.39
ncsB   4.78      GK5.   3.98      GK5.   3.39
GK5.   4.95      ncsB   4.07      AnJo   3.48
538    5.31      538    4.09      ncsB   3.68

RMSE are especially shrinking because outlying successes Hou and Mil have been losing.

Code: Select all

1/31   RMSE      avg abs err             SMRE
trzu   3.83      trzu   3.04      RyRi   2.46
lnqi   3.84      lnqi   3.05      BaDo   2.55
RyRi   3.94      RyRi   3.07      trzu   2.58
amp5   3.96      BaDo   3.10      sbs.   2.61
sbs.   4.14      amp5   3.14      amp5   2.62
BaDo   4.16      sbs.   3.20      Crow   2.65
vegas  4.22      Crow   3.24      kmed   2.66
ATCt   4.31      vegas  3.28      lnqi   2.66
sndi   4.31      kmed   3.31      yoop   2.78
AnJo   4.32      Mike   3.38      vegas  2.78

[GK5]

Glad to see I have moved from the basement in all three categories

Thanks for keeping this updated Mike!

[Mike G]

The worst RMSE today was about the best just 3 weeks ago.

Code: Select all

date     -  err +      - RMSE +
11/17   5.19  6.91    6.42  8.47
11/22   5.08  6.28
11/29   3.99  4.98
12/03   4.55  5.53    5.54  6.83
12/13   3.95  5.16    4.94  6.17
12/21   3.58  4.81    4.59  5.73
12/27   3.45  4.74    4.49  5.60
1/04    3.20  4.66
1/09    3.61  5.18    4.94  6.24
1/13    3.33  4.84    4.59  5.91
1/17    3.31  4.70    4.51  5.73
1/24    3.10  4.18    4.11  5.29
1/31    2.96  4.07    3.78  4.95

[Crow]

wow 538 has really fallen off a cliff moving to their new system this year

They moved from RPM / BPM blend to pure BPM right? Wrong move.

[tarrazu]

wow 538 has really fallen off a cliff moving to their new system this year

They moved from RPM / BPM blend to pure BPM right? Wrong move.

Yeah all BPM and a player similarity model.

https://fivethirtyeight.com/features/wh ... r-2016-17/

"CARMELO projections are generated through a two-step process. First, the system produces a baseline projection for each player based on regression analysis. Then, it adjusts the forecast and generates a distribution of possible outcomes based on the comparable players. Although the second part of this process is almost exactly the same as last year, we’ve put more work into the first step, the baseline projection. The system is now a bit smarter about handling players with limited playing time. It also recognizes that different statistics have different amounts of predictive value. For instance, because shooting can be streaky, players who generated strong performances on the basis of good shooting seasons are more apt to regress to the mean than others. By contrast, generating shots, drawing fouls and taking 3-pointers are correlated with improved performance in future seasons. Rebounding, blocking shots and, especially, accumulating steals are also correlated with stronger future performance."

[sndesai1]

i think we all know bpm will perform worse than rpm here, but i believe mike g's projection using last season's bpm and kevin pelton's mins is doing better than 538 without any fancy similarity model to project performance.

i guess it could be that the minute projections were way off...

[AcrossTheCourt]

No, they often share samples of the minutes projections, and they're usually really solid.

I think they just overfit the similarity/age curve.

Look at MIN's projection again, which did not factor in Thibs -- they were over the optimistic Vegas line by a good distance. They assumed some incredible development.

[Mike G]

i think we all know bpm will perform worse than rpm here, but i believe mike g's projection using last season's bpm and kevin pelton's mins is doing better than 538 without any fancy similarity model to project performance.

i guess it could be that the minute projections were way off...

I didn't use BPM at all. It was a home cooked version I called fauxRPM averaged with eWins.
Both are boxscore stats, to be sure.

Besides player minutes, the big variables seem to be rookie projections and regression to the mean. The optimum extent of regression may vary by metric or by season; seems like plenty is always best.
538 is just having a bad year all around.

Meanwhile, new bests in RMSE

Code: Select all

2/01   RMSE      avg abs err             SMRE
trzu - 3.84      trzu   3.09      BaDo   2.58
lnqi - 3.91      lnqi   3.12      RyRi   2.62
RyRi   3.99      BaDo   3.15      sbs.   2.63
amp5   4.03      RyRi   3.16      trzu   2.63
sbs.   4.18      amp5   3.20      Crow   2.65
BaDo   4.21      sbs.   3.25      amp5   2.66
vegas  4.26      Crow   3.25      kmed   2.69
sndi   4.33      vegas  3.29      lnqi   2.75
ATCt   4.33      kmed   3.35      vegas  2.76
AnJo   4.35      Mike   3.45      yoop   2.88
Crow   4.37      sndi   3.48      sndi   2.93

[sndesai1]

sorry, that's my fault on the wording. i didn't mean your contest entry, but rather the numbers posted here under BPM: http://www.apbr.org/metrics/viewtopic.p ... =15#p28073