Page 1 of 1

Sloan 2020

Posted: Thu Mar 05, 2020 7:06 pm
by rlee
Searching for the heart of Sloan: On the State of the (Analytics) Union in 2020
By Seth Partnow and John Hollinger 3h ago 7

Editor’s Note: Below is a discussion between Seth Partnow and John Hollinger. Both cover the NBA and basketball analytics for The Athletic. Partnow was formerly the Director of Basketball Research for the Milwaukee Bucks. Hollinger spent seven seasons as the Memphis Grizzlies’ Vice President of Basketball Operations.

Hollinger: So it’s a big week for geeks – most of whom are headed to Boston for the 2020 Sloan Conference.

The fact we’re discussing this at all is kind of amazing. This is the 14th Sloan Conference and it’s grown from a few panelists sitting in front of a small classroom at MIT to a Woodstock-type weekend extravaganza that takes over a large city’s convention center.

The growth of the conference has mirrored the rise of analytics in some ways, as more teams and more sports have been impacted by the field, more individuals in those fields decided they have to learn about it, and more powerful people decided it was OK to use the word “analytics” in polite company. The fun part, however, is that Sloan’s heart has remained in basketball in many ways. Obviously it starts with the founder – this was the brainchild of Rockets GM Daryl Morey and Kraft Analytics CEO Jessie Gelman – and the original conference was top-heavy with hoop nerds. (Including yours truly).

Sloan has changed a lot too, and so has analytics. As the lowest-hanging fruit gets eaten off the analytics tree, we’ve gone much deeper in the weeds on regression-adjusted lineup ratings, camera tracking and other analysis. At the same time, the conference itself has grown into a different beast too – a mostly mainstream event about sports that still has a geeky heart if you look for it.

For you, Seth, is it a bit like the alternative band that went mainstream? I liked Sloan’s early albums better, but they sell more records now.

And more generally, let me start with a question that we’ve each had to answer from both sides of the prism. This event (and the field in general) originally began as basically an academic exercise, with a lot of collegiality and a not-insignificant amount of information sharing. I’m probably romanticizing the past a bit, the intervention of NBA franchises to hire analytics talent has inevitably changed that.

So I’ll throw this out to you: What do you think the impact is of franchise interventions on the field? Is what has been gained privately offsetting what has been lost publicly? Are there areas where more collegiately should be encouraged rather than 30 teams trying to invent their own wheel?

Partnow: You will not find a prouder SSAC “it was better before it went pop” hipster than me, even though the first time I went (I want to say 2014 but could have been a year later) it may have already passed the tipping point between “symposium” and “trade show.”

Going back to my first piece for The Athletic, I’ve consistently said that one of the things that one misses working on the team side is the collaboration you’re talking about. I hesitate to say there’s a lack of collegiality, however. I think that as the role of the analytics professional has expanded into the NBA, we have done a pretty decent job of building a community amongst ourselves, which is a big reason why going to Sloan still has value, even if what happens on stage has become a bit more marketing fluff and storytime.

Fortunately, that community hasn’t been foreclosed to me being back on this side of things. Eric Tulsky, the Carolina Hurricanes VP of Hockey Management and Strategy (their title for ‘head of analytics’) has given talks before at Sloan on the usefulness of working with “public” analysts, which emphasize the importance of the broader range of viewpoints and fresh sets of eyes that come with casting a wider net. Or as I’ve been known to say, “beg, borrow and steal.”

I don’t think that degree of conferring is antithetical to seeking a proprietary competitive advantage. Or at least it doesn’t have to be, as I don’t think it’s well-enough understood how important it is to know not just the statistical methodology of a study, but also the research question itself in terms of application.

It’s funny, as no one ever questions whether coaches attending the same clinics are “giving too much away.” In much the same way, there are plenty of non-proprietary topics where analysts can, and I think should, share more, especially the sorts of “best practices” for communication and research processes that help us all to become better. But that’s to some degree pie-in-the-sky as teams are going to err very much on the side of secrecy.

As to your bigger picture question, I do think that to some degree the extent to which different groups have been walled off from each other has slowed the pace of discovery continuously. I was going back through some of the (partial) archives of the APBR Metrics forum a couple of years ago and was struck by the degree to which many of the same arguments we have today were present, oftentimes among the same people having the debates in both time frames! If you look at the handles on those old board posts you find names like Kevin Pelton (currently at ESPN), Dean Oliver (currently with the Wizards), Dan Rosenbaum (Pistons), Eli Wittus (Rockets), Jon Nichols (Cavs) and so on everywhere.

I also wonder if despite the widespread adoption of “stats” in the broader basketball discussion, if we’ve really gotten much closer to an understanding of the “analytics process” (described succinctly by Paul DePodesta of the Cleveland Browns starting at around 3:09 here) as far as making better decisions in terms of uncertainty, or if we’ve just moved from one set of conventional wisdom to another. What do you think?

JH: Process? What is this “process” thing you speak of?

I joke, but on the team side the research process often degenerates into what I call “drunk-lamppost analytics,” where the analytics research “question” is really a hunt for data to prove somebody’s preconceived point.

Bigger picture, I think understanding uncertainty is still a big fail point. Take the draft, for instance, which is probably the place people are most likely to get it but still often don’t. People intuitively understand that a future draft pick will have some kind of variance to it in terms of where it falls, and another kind of variance to it in terms of the quality of player it will yield. Yet for some people, that lack of certainty becomes a massive impediment that they can’t get around. I’ve actually had team officials tell me they want “pick certainty” in a trade. Fine, call me back in June then. Or better yet, try me 10 years from now and we can go back and figure out what the pick was worth.

Similarly, there’s a tendency to dismiss the possibility of non-lottery picks bearing fruit (even on my old team, where the best player was the 48th pick in the draft, Mark Gasol in 2007) and for lottery picks to be some kind of guarantee … even when the fail rate on a pick as high as second overall has been shockingly high.

But I digress. I want to get back to your point about sharing and collegiately, because that’s at the heart of where Sloan began, and now it is, as you say, a trade show with something a bit different for everyone — high-wattage generalists, powerful execs and owners, analytics-hating coaches taking their reputations to the car wash, etc.

Your point about coaches hits the nail on the head — I think because it’s a) an accepted, delineated profession and b) one where there is so much mixing to begin with given how often they change teams, that to a certain extent it has become accepted that coaches will cross-fertilize ideas. Less so on the analytics side, and the relative newness of the field and the relative lack of mixing between teams and sports both mitigate against it happening.

But where might the best opportunities be? We dipped our toes in the sand with a baseball team but never really got anywhere — the differing calendars of different sports teams make it hard. A lot of the medical analytics happening in the NBA now is coming right out of soccer and Australian Rules Football, so maybe that’s the best current example. But there should be a lot more, right? Teams in the East and West, in particular, could probably share a decent amount between each other at virtually no competitive cost. Obviously other barriers remain (trust, geography, needs, resources, etc.), but one wonders if that’s a start.

Seth — should the Bucks and Grizzlies have been co-conspirators the whole time?

SP: Glad you mentioned The Sloan Redemption Tour! Who is the lucky contestant this year?

Regarding “sharing,” some of that does go on, but it tends to be on sort of background and/or purely technical topics. For example, a few years ago there were some persistent issues with the accuracy of the PBP feeds to start the year. While that was going on there were frequent texts of, “Hey, watch out for the second quarter of Sacramento-Charlotte last night.” Similarly, there have been low-level discussions about pooling resources on officiating given how opaque the league can be in terms of detailed officiating evaluations. But those are pretty low salience examples with minor impacts.

More than that, the natural tendency to want to pull back from open discussion is understandable. I’ve had the experience multiple times at smaller conferences and gatherings where you can almost literally sense people about to have an open conversation about methods and findings and then everyone sort of pulls back at the same time. I’ve had baseball execs share the exact same experience with me as well, so it’s not just a basketball thing, but it might be a sports organization paranoia thing.

I’m glad you mentioned other sports though because breaking that barrier does have a way of allowing for far more earnest and open discussions. I can (and did) learn more about theories of roster building and cap management from NHL and NFL execs than I could from other folks around the NBA, even the ones I consider personal friends. And certainly, as things like tracking data spread across sports, conversations about how to use that data, both from a technical what-is-this-thing standpoint and a practical “how do I make this make sense to coaches and/or GMs” standpoint become more and more helpful.

In terms of the medical/training/sports science realm, that is one area where I do think there should be far more open collaboration. While a team might be able to gain a narrow competitive advantage in that area, it’s better for the sport and league as a whole if the best players are on the floor and playing at the highest possible capacity. You and I have hinted at this before I think, but it’s very easy to forget that all the teams are only competitors to a degree, and as businesses are reliant on one another to a huge extent. Healthy stars mean a healthier product and better conditions for everyone — at least in theory. Plus on a human level, we want, or should want, what’s best for the athletes themselves.

Before turning to something more positive, I want to agree with you regarding understanding and accepting uncertainty into our lives. In a way, I think well-trained analytics and analytics-adjacent professionals are ill-suited to be effective in an environment that can reward the appearance of assuredness bordering on bombast. One of the outside criticisms often leveled at the purveyors of statistical findings that I find frankly hilarious is the notion that we’re so sure of ourselves despite [INSERT FAILING X USUALLY NEVER PLAYED THE GAME].

People, trust me when I say that most often nobody knows more about how wrong models are and why they miss than the people who build them in the first place. Which is sort of the heating vent in the Death Star in group discussion settings, where the more “traditional” might not have explored their own biases in the same way so as to understand why they are also frequently wrong.

Now, making an abrupt turn for the happier, are there any aspects where you are pleasantly surprised with the progress we’ve made over the last decade or so?

JH: Still getting over the “heating vent in the Death Star” reference.

I think we’ve made a ton of progress on health and wellness analytics in particular, and I think the cross-sport cooperating in this area has been a huge catalyst — nobody in English soccer gives a crap if American basketball teams copy what they’re doing, so the sharing has been relatively free and honest.

Similar to the analytics movement, however, the other catalyst is that a bunch of the work in this area has either been publicly available — especially referencing things like sleep and travel — or “partially” publicly available via third-party entities like P3 or Second Spectrum.

The other area where we’ve made progress is that in a lot of areas the public information has become better, even with teams periodically poaching some of the best analysts. As you noted above, despite our best efforts on the team side, the publicly available info is often a great resource. And a lot of it is just the simple stuff — like say, lineup data or advanced stats — just the fact you can get it with one click and don’t have to ask somebody to look it up frees people to pursue loftier tasks.

But on the health/medical side, you just wonder if the current trend can continue with so much sharing and public information. And if not, how we can continue to make progress in spite of much information moving “in house” with individual teams?

And I’ll give you one more question, in the spirit of a conference like Sloan: What’s next for analytics?

SP: There are three areas where I think there are still some low-to-moderate fruit height left. First, on the pure “game stats” side, I’m really pleased with the focus more people are giving to bottom up stats identifying and measuring skillsets as opposed to focusing on top down player value metrics. Right now, a lot of the public stuff in that area is a bit hamstrung by the available data, but I think even with public data — to reiterate for the hundredth time this season, there is so much available on stats.nba.com if you know where and how to look — a lot can be done in this sort of area.

I think there are gains to be made in terms of the intersection of the “on floor production” and sports science realms. For the most part, fatigue/load seems to have been studied with respect to injury prevention and management rather than in terms of performance degradation, the “risks” of the risk-reward trade-offs. But there will be work done on the “reward” side as well in terms of determining when it is and is not worth exposing players to various risks.

But lastly and most interestingly, I think there are going to be advances in better understanding what is currently lumped under the “intangibles” nameplate. I’m not an expert enough in the field to be certain, but I think there are a lot of findings coming out of the industrial psychology discipline that could shed some light on personality fit, culture building and things of that nature. This could affect not just roster decisions, but also hiring decisions up and down basketball operations as getting a better handle on who will work well together brings some fairly obvious benefits.

The thing that I think is less likely to manifest is a similar “eureka” moment to seemingly sudden recognition that 3>2. Much like baseball made the fundamental discovery that not making outs was the most important thing, that was basketball’s apple-on-the-ground moment and we’re unlikely to find another One Big Thing that fundamentally changes the way the game is played in the near future.

Final question to you, is there anything in any of the talks or panels that you’re particularly looking forward to? I have to admit to being a complete sucker for anything that is coming out of FC Barcelona’s Innovation Hub and can’t wait to see what Javi Fernandez and company have in store for us this year.

JH: Sadly, I think you’re right — this sport has already plucked all the lowest-hanging fruit. I’m also envious of football because there is so much obvious money still sitting on the table (stop punting, fools!), but I digress …

As far as what I’m looking forward to, obviously seeing my friend Seth is the first item. But some of the panels are potentially hugely interesting, especially the ones in sports where I either know very little (like say, cricket) or know just enough to sound like an idiot (European football!). In general, I think there are more opportunities for cross-sport learning than people generally acknowledge.

But it call gets back to the “freedom to share” question we raised earlier — the more these speakers are able to let us look under the hood, the more compelling the panels will be. If everybody stops just short of telling us the stuff we don’t already know, it ends up far less interesting.

For me, the heart of the conference has always been the research papers because that’s where it gets back to its academic nerdist* roots. We have a few exciting papers in this one that relate to our sport (a proposal to stop tanking, for instance!), and some others that are on less familiar turf that nonetheless look exciting. (Optimizing batter-pitcher confrontations in cricket? Tell me more …).

As this event morphs into brainiac Bonnaroo, a veritable East by Northeast, my greatest hope is that Sloan can serve its larger audience while still retaining its geeky heart. It’s harder to pull off these days, but somewhere in there, at the core of it, is the idea of sharing and collaborating in an academic pursuit of knowledge.

Re: Sloan 2020

Posted: Fri Mar 06, 2020 5:00 am
by Crow
Thanks.

Re: Sloan 2020

Posted: Wed Mar 11, 2020 12:50 am
by Crow
Conference generated little basketball followup. Probably least ever. There is a podcast I'll get to later but I see little else. If anyone has meaningful basketball conference follow-up, give a link.

Re: Sloan 2020

Posted: Wed Mar 11, 2020 1:15 am
by Crow
That podcast? Don't bother.

Re: Sloan 2020

Posted: Wed Mar 11, 2020 2:42 am
by Crow
Heard half of one of the basketball panels. I'll want to hear it all before saying much. But it sounded typical surface. The other one on 3s seemed even less likely to have new insights but I'll probably listen when the recording is available later.

Anybody else have comments on the basketball panels?