2015-16 Team win projections

[kmedved]

Are you asking why we use squared error generally? Or in this particular case.

My answer to why I defaulted to using squared errors isn't terribly sophisticated: because smarter people than me have told me it's usually a better way to do things than mean average error. But apart from ease of calculation (which isn't really salient these days), I can't explain why.

[BasketDork]

What I meant to say was "why the rather large difference in RMSE" differing between the ones up here and bbrf's?

[kmedved]

Sorry, I still don't understand. The "ones up here" are the RMSEs I posted presumably. What are the BBref RMSEs however? All the RMSE and MAEs I've posted are using an average of the Kubatko method, the BBRef simulation, and the TeamRankings projections. But I'm not sure what gap you're referring to?

Thinking about it more, I do think mean average error is probably more appropriate than squared errors here. RMSEs penalize large errors disproportionately, which is for good reason. However, with basketball team projections, I think this might end up just penalizing projections for the most injured teams. Given I don't think anyone here is claiming to have put in a lot of work projecting injuries, I'm not sure of the value of that for our purposes.

But I don't have strong thoughts here. I'd be interested in hearing some more thoughts about this, as I think its a pretty interesting topic.

[Crow]

I'm for mean average error in the end.

[Statman]

Why do we prefer squaring the errors to begin with?
A hypothetical -- For simplicity, suppose we're predicting a league of 8 teams:

Predictor AB has 7 teams exactly right: errors 0, and squared errors are 0. On the 8th team, he's off by 20.
YZ has missed those same 7 teams by 5, and the 8th he misses by 15.

On each of those 7 teams, YZ has a squared error of 25, vs a total of zero for AB. This gives AB an advantage of 175.
On that 8th team, the difference in squared errors is (400-225) also 175.
Despite AB being (equally) better on 7 of 8 predictions, both predictors get the same RMSE: sqrt(400/8) = 7.07

In (unsquared) absolute avg error, its 2.50 to 6.25

Good question. I've wondered this also, but I assumed there was a decent reason for the squared errors that I just didn't realize.

Happens pretty often on this board - someone does something, I don't really understand why, but I don't question it because I'm afraid I'll get slapped in the face with an obvious answer that I was just plain blind to. Squared errors is one of those things.

[Mike G]

One possibility is that before computers, we could actually compute squares and square roots. Because we could, we did. And there are parallels in nature -- gravity, balance, etc.

But with computers, we can just as easily calculate the 1.65 root, or the 0.35 root, or any root or power which seems to give the most predictive ability. Limiting oneself to integral exponents doesn't seem to have much utility.

When we study point differentials, we like to compress or truncate blowout scores, if we are trying to estimate a team's strength. One way to do that is to put a power <1 on the MOV: The square root of the MOV suggests that a 40-point win isn't 4 times as good as a 10-point win; it's only twice as good.

Analagous is the team that has injuries, is clearly out of the playoffs, and then tanks a few games. How significant is it that my prediction is off by 15 and yours is off by 20? It may be that we should weigh more heavily the difference between 0 and 5, i.e. use an exponent <1.

Meanwhile, how do you explain to an average fan what RMSE actually is? Send him to Wikipedia? It's pretty easy to explain avg error: Total error divided by 30.
For several reasons, I'm strongly inclined to no exponent.

[kggk]

There is no general answer for RMSE error vs absolute error, and this forum is not the only place people debate this question. See http://www.geosci-model-dev.net/7/1247/ ... 7-2014.pdf for example.

In general, RMSE is preferred when the errors follow approximately a Gaussian distribution (bell curve). This happens a lot of the time in practice because if the total errors are due to an underlying set of random variables with finite variance (for example player quality or player minutes), the sum tends towards a Gaussian distribution (https://en.wikipedia.org/wiki/Central_limit_theorem). In that case RMSE is the natural way of describing the error distribution. But presenting both types of errors is very common as well, and provides more information.

Fitting models with RMSE is also easier, as doing manipulations like taking the derivative of the absolute value is more difficult, and derivatives are often used in optimizing models.

[kmedved]

I asked Phil Birnbaum if he'd done anything on this, and of course he had, and of course, it's the best explanation I've seen for people like me who are barely mathematically literate: http://blog.philbirnbaum.com/2012/08/th ... olute.html

It doesn't really solve the RMSE vs. average error question (as kggk says, there is no answer), but it did help me think about it cleaner. The closest there is to a takeaway is that squared errors target the mean better, while absolute errors target the median.

[ca1294]

I've heard one reason RMSE is more popular than mean absolute error is because it is readily differentiable whereas functions involving absolute values are not. Optimization algorithms are often trying to minimize some error term, and some algorithms depend on the function being differentiable.

[xkonk]

Agreeing with ca1294, my understanding has always been that squared errors are preferred because that produces a function that can be differentiated, which is important for the calculus that underlies statistics (hence variance and standard deviation but also higher order moments like skewness or kertosis). But for any given purpose, you should choose the error measure that matches your purpose. For a competition like this it would make sense to me to use the absolute error, but if someone wanted to have extra punishment for being far off the RMSE isn't wrong in any sense.

[Dr Positivity]

So if I interpret it correctly in the context of this contest, the farther away a prediction is from the record, the more its damage accelerates. On one hand this helps reduce the impact between projections that are 0 wins away and 2 wins away, which is fair cause the guy 2 wins away made have had a perfect projection and just got caught by a team landing a few wins from point differential and you don't want to punish him too much compared to the 0. OTOH the difference between say 15 wins and 17 wins away has a significant impact, the same as between 0 and 8 Ws away. If the gap between 15 and 17 was also luck compared to point differential, the impact of this is not ideal. I would vote for average error as the most natural way to do it for this contest (it also doesn't hurt that my predictions look better with average error)

[kmedved]

I think the counter to that 15 vs. 17 wins example is that if the errors actually do follow a normal distribution, then a 17 win error really is dramatically more important than a 15 win error. It should be significantly less likely to happen by random chance than a 0 to 2 win error. It signals much more model error. That's why it's key that the errors follow a normal distribution in the first place. If they don't, then using RMSE may not make sense. It's like how there are dramatically fewer people with 150 IQs than people with 145 IQs, while the shift from 100 to 105 doesn't move the needle much.

Anyone have a link to last year's results? I'm curious enough to try and plot these. I suspect they will not follow a normal distribution due to injury issues. 15 win errors will be way too common relative to the frequency of 5-6 win errors (i.e., OKC).

[Crow]

viewtopic.php?f=2&t=8633&start=255

[Dr Positivity]

I think the counter to that 15 vs. 17 wins example is that if the errors actually do follow a normal distribution, then a 17 win error really is dramatically more important than a 15 win error. It should be significantly less likely to happen by random chance than a 0 to 2 win error. It signals much more model error. That's why it's key that the errors follow a normal distribution in the first place. If they don't, then using RMSE may not make sense. It's like how there are dramatically fewer people with 150 IQs than people with 145 IQs, while the shift from 100 to 105 doesn't move the needle much.

This and the IQ example is a good point, and helps explain where in real life RMSE can more useful than average error

[permaximum]

I like RMSE more. Imo a model that predicts x-y-z by +3 +1 -6 is a better model than predicts x-y-z by 0, 0, 10. Especially when it comes to NBA team wins. It feels like the second model lucked out. However this is a random guy's opinion. Surely some stat experts can shed light on this issue. I've read on some research paper that if the expected error distribution is gaussian, RMSE is a better indicator of model performance than MAE.

Do we expect a gaussian error distribution? I guess yes. Then I say, let's use RMSE.