"Total Quarterback Rating" applied to PGs

[Crow]

I saw this article on Dean Oliver & ESPN's new Total Quarterback Rating:

http://espn.go.com/nfl/story/_/id/68332 ... ack-rating
and I thought to myself, hey, it might be nice to apply something similar in approach to PGs in the NBA.

Why and for what impact?

To account (with standard stats and visual interpretation of other details) for the impact of plays involving PGs as a catalyst, leader or "Quarterback" on win probability added. And to account for the varying degree of clutch importance of the plays. Elements not in Offensive Rating or most other traditional statistic-based metrics.

To divide credit between the PG and other teammates on every play or at least most plays. The plays where the PG acts enough like a QB in the overall play implementation to be called that and rated for that role instead of just when a pass from the PG leads immediately to a shot attempt that goes in. To go beyond just registered assists or a pure point guard rating as leadership measures.

With TQBR, quarterbacks are responsible for producing good team performance not just good individual stats. They could look differently under the old version of QBR and TQBR. I'd have interest in a new metric that measured PG win impact. Most of the traditional snapshots of his individual stats don't fully account for all PG impacts on winning. Being a good individual scoring PG or a dominant passer might be ingredients for a good overall PG win impact but the PG win impact on the other 25-50% of plays where they didn't take the shot or get the official assist also impact their overall win impact and sometimes PGs who have great individual stats don't have great impacts in this other set of plays. Hence the discrepancies already visible between the offensive boxscore metric ratings of certain PGs and their estimated offensive Adjusted +/-. This approach would add another perspective to the mix.

A scorer who maneuvers to a better shot or score deserves more credit than one set up for a easy catch 'n shoot but maybe not all the credit (in the absence of an official assist) and, like in football, the quality of pass, reception and run after catch / shot release all matter and could be credited separately if one has the data, time and inclination to be detailed and accurate in accounting of relative performance impacts.

Players beyond the QB / PG and "the receiver" could get credit for useful motion- for picks but also more generally for spacing, motion for the sake of creating alternative perceived threats and deception. Other players beyond the PG (or just those few who do it enough to make it fairly significant) could get similar analysis for the times where they strongly act as the creator or Quarterback. While I have focused most of the discussion about applying the TQBR approach to basketball to learning more about PG win impact, it would produce visual evaluation enhanced ratings for all other players in their receiver and "lineman" roles and perhaps in their creator / QB roles too.

I don't know how ESPN plans to handle running plays in football. Apparently they not giving the QG partial credit but I assume they will still give partial credit to the linemen. In basketball, plays considered totally self-created could be treated as a separate pile and give full credit to the individual actor or one take a broader perspective and could give a bit of "lineman" to create that iso situation and QB credit or blame for it becoming an iso play. I think if TQBR were applied to basketball it would be good to do apply to most or all plays. A choice, a technique, to see what it says.

Defensive adjustment we are told will be left for later in the TQBR analysis for football. I understand the reasoning for going stepwise, but it would seem important to take that step fully before drawing final conclusions about players or making final comparisons. The results of the first step without the second part will remain a partial analysis and similar in that regard to the way most basketball Statistical +/-'s (and most other offensive focused / biased metrics) stop short of adding the defensive adjustment. Based the quality of defense is not that different but you don't know for sure until you check and there might be a few cases where it affects the takeaway. Might want to add a third step ultimately and adjust for teammate quality too or at least recognize that is also has consequence.

With or without adjustments, it would be interesting to match up the TQBR approach, especially for PGs, to their Adjusted +/- and maybe especially Wayne Winston's Impact Score version since it has endeavored to address win probability impacts and clutch context. And probably the Factor level Adjusted +/- too. A detailed statistical and visual evidence metric of win impact could be compared side by side with and used alongside the "pure" Adjusted +/- method (just using scoring as input), pure boxscore metrics and Statistical +/-. 4 legs to the stool, a richer resource set for evaluation and improvement plans.

[EvanZ]

I think the QBR is really interesting. And it's great to see DeanO on tv!

One of the things about the QBR that is not clear to me is how they decided the weight of each stat. For example, they seem to make a big deal about "clutch" plays in the 4th quarter. Is there some evidence that the 4th quarter is any more important to winning than the 1st quarter? Maybe there is, and I'm not aware of it. But it seems to me that it's treated more as an assumption by the folks on ESPN (at least, the talking heads).

Also, how do you account for the fact that some QB's call their own plays, and others never do? If you have a coach who doesn't call the right plays, isn't the QB going to appear worse?

Crow, regarding the PG rating, that was the motivation behind my recent tracking of potential assists on spot-up jumpers using Synergy. It's at least a step in that direction. Did you see that post?

http://thecity2.com/2011/07/29/warriors ... g-synergy/

[Crow]

I saw your post with its use of video to add information beyond the boxscore. I had prepared a longer response but lost it. Maybe I'll recreate it later.

[Crow]

Your article probably helped cultivate my response to TQBR, though I have long felt that PGs should get leadership ratings (beyond assists or Pure Point Rating) and are pretty responsible for what happens and doesn't in plays and the final offensive results most of the time.

The main difference I see in what you did in your post and what I was (suggesting with TQBR applied to basketball is that I was looking broadly at real impact on actual scoring (yet not in the boxscore) while you looked at passer-shooter data to identify both real and potential value on one type of play. But with your coding skills and access to video you could certainly do something broadly covering PG non-boxscore impacts identified by video review though probably limited to real value added to points actually scored, at least for the Warriors, if you choose to invest the time.

I don't know if Dean & ESPN have already considered application of this approach to basketball and PGs or would, but I mention it to at least raise the possibility and perhaps hear a response.

You are right to raise the issue of how much is the QB vs Coach or PG vs Coach. Not likely that an outsider would handle this as well as a team could internally, knowing the calls and the options off them play by play (if they track them comprehensively), but an analyst real familiar with a team and their offensive system or tendencies might be able to do pretty well at identifying called plays leading to intended shots vs improv.

Whatever you found from this approach could and should be compared to the separate credits for PGs, other players and Coaches in Jerry E.'s RAPM for basketball.

If there is anything similar in football, a comparison between TQBR and the value found by an Adjusted +/- method could be done there too.

(This might still be half-baked or off but I wonder what you'd find if you took a standard RAPM run for everyone including the Coach and compared it a run with no Coaches and no PGs. Where are the biggest differences? Or would all the changes be proportional?)

I'll note that when proposing dividing credit in basketball among players for the value of a single play on offense and defense years ago (here and at the predecessor yahoo group) by simple rules based on perception of average value of contributions by various players and not specific plays (before ProTrade and after ProTrade) I frequently got reactions that such rules are subjective and that these observers weren't too interested in a metric that relied on subjective rules. I wonder if fans / analysts will eventually call on ESPN to reveal more about their rules for the subjective evaluation of video (whether they are documented in detail or follow general guidelines or are left to to the brood discretion of the scorer) and if they accept the specific rules used or heavy use of such rules in a metric.

[mtamada]

For example, they seem to make a big deal about "clutch" plays in the 4th quarter. Is there some evidence that the 4th quarter is any more important to winning than the 1st quarter? Maybe there is, and I'm not aware of it. But it seems to me that it's treated more as an assumption by the folks on ESPN (at least, the talking heads).

It flows directly from any study which looks at win probability. Perhaps the easiest (as well as earliest) examples are from baseball. Score is 0-0, and Ken Griffey Junior hits a solo home run in the bottom of the inning. How much has he added to the home team's win probability?

If it's the bottom of the 9th inning, he's raised the probability from a little over 0.50 to 1.00, because he just hit a walkoff home run.

If it's the bottom of the 1st inning, he's raised the probability to ... I don't know, TangoTiger published win probability matrices somewhere, I'm guessing the win probability went from maybe 0.52 to 0.62.

Same score, same homerun by The Kid, but huge difference in his team's probability of winning. Because those close-and-late plays are the ones which have the highest impact (or "leverage" as the sabrmetricians put it). Plays in an early inning, or the first quarter, have less impact on the team's win probability.


If the Clippers are up by 2 points at the end of the 1st quarter, what is their probability of winning?

If the Clippers are up by 2 points at the end of the 4th quarter, what is their probability of winning?

Even the Clippers can't blow a 2-point lead with 0:00 time left. Leads, and plays, in the 4th quarter have higher impact than leads and plays in the first quarter.

Note that I'm note saying that the late field goal is more valuable than the 1st quarter field goal. 2 points is still 2 points. But if we're looking at impact/leverage, there's a clear difference.

[EvanZ]

I'm aware of some of these studies, and am not disputing the math behind win probability. I do question, however, the actual link to player value and prediction of future results.

[xkonk]

I'm aware of some of these studies, and am not disputing the math behind win probability. I do question, however, the actual link to player value and prediction of future results.

I would agree. Alex had a related post (http://sportskeptic.wordpress.com/2010/ ... onsistent/). Brian Burke's win probability metric has pretty poor predictive power while his expected points metric did better, presumably because you take out some of the randomness that occurs in high-leverage situations. Success rate did even better, probably because it takes out the randomness of really great and really poor plays, basically just looking at if you improved your position or not.

[EvanZ]

First real discrepancy between ESPN's QBR and NFL rating?

Alex Smith
C/ATT YDS AVG TD INT
15/20 124 6.2 0 0

Kevin Kolb
C/ATT YDS AVG TD INT
18/27 309 11.4 2 0

Kolb had a 130 NFL rating vs. 90.4 for Smith. Conversely, Smith came out with a 66.2 QBR versus 44.0 for Kolb. Interesting. Just looking at the two lines, one would clearly think Kolb had the better day. (Right?)

Smith was not sacked and did not fumble the ball. Kolb was sacked once and also fumbled twice. Given that, the QBR starts to make a lot more sense.

[xkonk]

Kolb was sacked twice according to the ESPN box score, and lost one of his two fumbles. Smith fumbled once but didn't lose it. Smith also ran for 22 yards and a score, while Kolb did nothing with his legs. So Smith made up for his smaller passing numbers in other ways. On the other hand, Kolb threw a TD pass to tie the game in the fourth, which should have given him extra 'clutch' points. Still hard to say what's really driving what without being able to get under the hood more.

[huevonkiller]

Big time spamming in this thread, let's clean that up mods.

[EvanZ]

Alex Smith had a 28.7 QBR yesterday after leading a huge comeback. His passer rating was 112.1. Not a small difference.

Smith went 21/33 for 291 yds, 2 TDs and 0 int. He fumbled twice, one of which was recovered by Philly.

I'm not sure ESPN has worked out all the kinks of their new metric.

[xkonk]

Alex Smith had a 28.7 QBR yesterday after leading a huge comeback. His passer rating was 112.1. Not a small difference.

Smith went 21/33 for 291 yds, 2 TDs and 0 int. He fumbled twice, one of which was recovered by Philly.

I'm not sure ESPN has worked out all the kinks of their new metric.

See the first two comments at http://www.advancednflstats.com/2011/10 ... nched.html for a short explanation. And if you look at the win probability chart http://live.advancednflstats.com/index. ... 2011100204 , the 49ers were basically out of it until Philly missed a field goal and then they ran (not passed) their way to the winning touchdown. Even that only got them to about 50/50; the rest of the win came when Philly couldn't score. I'm not sure that Smith did much at all. Passer rating mostly cares about completions, so 21/33 and 2 TD looks pretty good.

[EvanZ]

What is the coefficient that is being applied here though? How much is TD worth when the win probability is 5% vs. 50%? Is it worth 1/10 the value? 50%? I have no idea and haven't seen any details of how ESPN determined this.

[xkonk]

Yeah, no idea. But since Smith's last really positive contribution (the TD pass to make it 23-17) still only had the 49ers at a 25% chance of winning, it probably didn't get him a lot of credit. After that, Smith had 3 or 4 incompletions and took a sack on third down versus 3 completions for all of 22 yards; the Eagles lost by missing two field goals, penalties, and giving up big runs. I'm inclined to go with ESPN's (or Brian Burke's) interpretation on this one.

[EvanZ]

It's still not clear to me why the *order* of events matters. Is there any evidence that defenses let up completely when they are ahead by 2 touchdowns? If so, how much? If it's a lot, I can see why it would matter. If it's not, then it shouldn't matter. That's really the crux of the matter for me. The fact that Smith is penalized for throwing touchdowns when "it doesn't matter" seems arbitrary to me. I know that's the conventional wisdom, though. But are there data that support it?