Shortcut Method for Measuring Reads on a Series

[Jeff Fogle]

Was looking over the "statgeek" picks in Henry Abbott's playoff contest. He awards 5 points for a correct team, then 2 bonus points if you get the exact number of games for that team. Looks like a fun contest. Was thinking though that it's possible to have a pretty good read on a series but not get rewarded in the scoring in that format.

Let's say some pick a team to sweep in four straight games...while others like the same team, but think it's going to be a tough series that goes 7 full games. If that team gets upset, everyone takes a zero in the series even though those who thought it would be tough actually had a pretty good read...and those that thought it would be a blowout had a misread.

Or, let's say one guy had the favorite in four in a rout...another had the underdog in a seven-game nailbiter. If the favorite wins in seven games, who had the better read? If the goal is to pick the winner, it's one way. If we're trying to evaluate the analysts...or evaluate our own personal analysis on a group of series, picking a sweep in what turned out to be a very tight series feels like an error.

Came up with a way of grading "reads" on a series if anyone's interested. Imagine a scale where you just circle your pick on the number line. I'll use O for Orlando and A for Atlanta from the first round:

O4-O5-O6-O7-A7-A6-A5-A4

The two extremes are at the sides. Tight competitive series are in the middle.

Once the series is complete, just count the steps between the pick and reality. Nobody in Henry's contest picked Atlanta to win the series. But, a pick of O7 is closer to reality (A6) than O5 or O4, and probably represents a better read on the competitiveness of the series from the predictor.

Went through and tabulated the distances from reality in the first round in that contest...

Morris 9
Stahlhut 11
Ilardi 13
Hollinger 14
Voulgaris 14
Paine 14
Berri 16

With Mother's Day coming up, I'll leave Mrs. Abbott out for now, and put her back in if she surges back onto the leader board with the second round games.

Kevin Pelton picked all but the Chicago/Indiana series over at BP. If I give him 0.5 points for that (because everyone either had C4 or C5 and he probably would have too), he's at 12.5 points on the scale above.

Rick Reilly also posted picks at ESPN. He's at 15. Probably won't be able to monitor Reilly going forward though because he picked the whole event up front, and three of his projected second round matchups ended up not existing (and Denver won't win the West).

Strong performance so far from Morris, averaging just over 1 placement away from reality over the first eight games.

Second round picks are up now. Nobody picked any upsets, so we can assume these are all picks based on "reads of the series" rather than game theory attempts to get back into the contest (meaning, if you fall behind early in Henry's format, you may have to pick underdogs to catch back up or you get locked out from any chance to win, but nobody's done that yet). We may see some game theory strategies in the later rounds...but "later rounds" are only 3 series of the 15 in the whole playoffs...so 80% of the assessments are already on the record.

Not suggesting using "distance from reality" is better for a contest. I can see wanting to reward getting the winner of a series correct when prizes are at stake. But, just in terms of evaluating a read on a series, or reads throughout a collection of series...this seems like a handy way to do it whether you're looking at your own methodologies or the reads of others.

Anyone have other ideas on this? Or, a possibility for the ideal contest format that punishes bad reads, rewards good reads, but also credits correct picks on which teams will advance?

[Crow]

I had some of the same thoughts as you did about caring about / wanting to measure distance from actual results.

There would be other hybrid scoring systems. Say 10 points for being exactly right; 8, 7 or 6 points for calling the series correctly and missing on the # of games by 1, 2 or 3 games; then say 4,3, 2 and 1 points for calling the series incorrectly by 1, 2, 3 or 4 games. That would retain some greater reward for calling a series correctly. If this scale didn't reward that aspect enough for one's taste, it could be adjusted to do so more heavily by moving the top half of the scale up by 1 point or more and / or moving the bottom half down by a point.

[Mike G]

Well, either "Winning is Everything", or it isn't. Winners advance, and the number of games needed is almost peripheral; except to break ties among prognosticators.

If predicting the competitiveness is more important than who wins, then why not judge guesses by point differential?
Orlando outscored Atlanta by enough to win in 7 or 6.

Your proposal is great, really. I'm just supposing how it would play in the world at large.
(I was off by 14 in your system)

[Jeff Fogle]

Like the idea, crow, of starting at 10 points for exactly right and working down from there. I've played around with some options. But, they get so messy in the explanation that they become kind of unusable. You don't want a contest that's confusing to explain. I'll keep playing around.

Agree with your points Mike G. about how different people see the prediction process. Can't argue that picking the winner shouldn't be a top priority. Trying to find a way I guess to reward good reads that just miss...and expose partial misreads. It's easy with one's own work I think to find ways to give yourself credit for stuff that you didn't exactly nail, and to not be harsh enough on the misses.

Haralabob would probably point out that's the beauty of the betting markets. That approach can be painful during the education process though.

You touch on what might be the next key factor for analytics. The goal for teams isn't to accumulate the biggest composite overall point differential. It's to find ways to be ahead each time they play EXACTLY at the 48-minute mark in the NBA, 40-minute mark in the colleges, 60-minute mark in the NFL and college football. How can analytics capture the complex process of running out the clock with a lead, or executing well late in a game in a way that isn't just "Well, the full season composite totals are what matters, and late game stuff is random."

I've heard (or expressed) that criticism in all sports at one time or another I think. The median result in Atlanta/Orlando was Atlanta by 3. Orlando won huge in the game the Hawks tanked to get ready for their home clincher (one could argue). The betting markets are keyed off medians as a general rule (many in the field have stated that the "true" pointspread in a game is the median of possibilities, and "finding value" means looking for bettable spreads that have been moved off that median for whatever reason). I would be opposed to using end of series margin differential as a guide unless predicting that was part of the original contest...and I'd then encourage people to use predicting the median in a contest format too (lol).

Thanks to both of you for responding...

[DSMok1]

I'd use error^2, where error is the number of games different than reality. I'd possibly also put an extra game of error if you picked the wrong overall winner.

That would really penalize someone who was way off in a series (predicted a 7 gamer and got a sweep=3^2=9) but help people who were close but missed.

[Jeff Fogle]

Thanks DSM, I like that idea a lot. Only concern would be if, in a contest format, the penalty is so great that it scares contestants up front into very conservative picks. But, most people usually go with conservative picks anyway so that may not matter. Think your idea is very good for emphasizing the distance in misreads...

[J.E.]

I think there's only two non arbitrary ways to score this. One would require everybody to come up with a player metric and then crown the metric which has the lowest error between forecasted and actual points per possession for all 5 man units. Not very applicable.
The other one would be to only give points for picking the correct winner of the series.
Everything else seems arbitrary to me. How many points you can get, how much you deduct for error etc. is just personal preference.

One, again arbitrary, scoring system that I would like would be for everyone to give their top ~3 scenarios for the series, for example 1. 4-3 2. 2-4 3. 4-1 and then give points if the actual outcome matched one scenario

[DSMok1]

After thinking about it, Jeff, perhaps the error in winning % would be better to use: 4-1 vs. 4-2 is a bigger difference than 4-3 vs. 4-2. If that makes sense...

[Kevin Pelton]

I think I might favor a percentage chance of winning because the likelihood of the higher-seeded team winning at home is such a strange externality in best-of-seven series. Picking the favorite in six minimizes the risk, but also offers a lower chance of being exactly right.

[Jeff Fogle]

Updated through two rounds. 12 of 15 series. And, we're probably seeing some game theory strategies from this point forward. Berri talked about that at the WoW website explaining his pick on Chicago in 7. So, what's in the books so far are measures of reads in the matchups through two rounds. From this point forward there will be some strateg-ery as people try to give themselves a shot to rally back.

Distance from reality on the scale outlined above:
Morris 17
Ilardi 18
Stahlhut 19
Paine 21
Pelton 22.5
Hollinger 24
Voulgaris 24
Berri 24

Kevin Pelton isnt' part of the smackdown, but made selections at BP. He didn't pick Chicago/Indiana, so I gave him a half a point there because all entrants either scored 0 or 1 in a series that largely met everyone's expectations in duration.

Morris missed Dallas/LA by 5 spots (everyone missed that one by a bunch). He's only 12 off in the other 11 series. Stellar. Ilardi missed Dallas/LA by 4, and is 14 off in the other 11 series.

Tough year to read by a lot of methodologies I think because of the personnel matchups. Fun to watch in unfold...

[Crow]

So the contestants have missed on series outcome by 1.5 to 2 wins per series for the 12 series. That seems like a fairly modest sized range from top to bottom but maybe a bit bigger variance from actual that I expected. I guess it is fairly hard to call series length, especially with the way home games are distributed. The NBA is probably happy with the schedule format and unpredictability.

Any interest in breaking the variances down into games over and under the predicted length? I am curious if folks in general are calling series too long or too quick on average and if there is a difference on this over / under pattern from top to bottom of the rankings.

[Jeff Fogle]

I'll play around with that if I get a chance tonight crow.

Noticed an error in earlier standings when I was logging the picks from everyone for this round. Here are corrected "distance from reality" standings.

Distance from reality on the scale outlined above:
Morris 17
Ilardi 18
Stahlhut 19
Paine 21
Voulgaris 22
Pelton 22.5
Hollinger 24
Berri 24

Sorry, misread my notes with Haralabob the first time through. He's at 22 not 24.

Amongst those under 20 in these next two rounds:
Morris: Miami 6, Dallas 6
Ilardi: Miami 6, Dallas 5
Stahlhut: Miami 6, OKC 6

Some game theory from Stahlhut in terms of trying to win the contest though, as he mentioned in his series comments...

[Jeff Fogle]

Quick Glance shows a slight tendency toward too long crow. We've only had one 7th game. Somebody winning in 7 was a reasonably common prediction in the series that were expected to be evenly matched.

Flat out unanimous misses:
Orlando over Atlanta
San Antonio over Memphis
LA Lakers over Dallas

Mostly right:
Chicago over Indiana
Miami over Philadephia
Dallas over Portland with Dallas picks in 6-7 (three erred with Portland)
Oklahoma City over Memphis (most had OKC in 6-7)

Mostly predicted too long:
Boston over NY (influenced by NY injuries--nobody had a sweep)
Oklahoma City over Denver (most had it in 6-7 games)
Miami over Boston (most had it in 6-7, also injury influenced)

Mostly predicted too short:
LA Lakers over New Orleans (everyone had 5-6, it went 7)
Chicago over Atlanta (everyone had 4-5, it went 6)

[Mike G]

Mostly predicted too short:
LA Lakers over New Orleans (everyone had 5-6, it went 7)

Dang, I missed that last game?

[Jeff Fogle]

Man, I've got to stop trying to type so fast, lol...everyone had LAL 4-5 and it went 6...JF