APBR-DraftExpress 2015 NBA Draft Project

[ampersand5]

After seeing the reaction on the web from this article, I find some of the responses to be humorous. I understand the people who are 100% against analytics, but I always get surprised by the people who say things alongs the line of "analytics backs me up in saying X is good, but the analytics must be wrong in saying Y is bad".

It's amazing to see the people condemn analytics because of one bad pick, as if they live in a world where teams not using analytics have never made a sub-optimal pick before.

On the subject of Cleveland, I'm not entirely sure what happened there. I imagine they (Ben Alamar?) had a draft model that liked Waiters, Thompson and Bennett. This gives me further credence in the information we have in front of us. I doubt that many teams are making models to begin with, and from those that are, they likely only have one model. I don't think any NBA team's model would be radically different than the users on here and the fact that we can take a blend of 5 helps ensure further accuracy. Maybe Philadelphia and Houston could have something slightly better, but I imagine the composite ranking is really ahead of the pack.

In regards to what Crow and EvanZ wrote - I agree that subjective rankings clearly have a role to play. That being said, do you think those subjective rankings are not covered in the DraftExpress rankings?

[ampersand5]

I created a new spreadsheet including Layne's Humble model, Kevin Pelton and Andrew Johnson's ranking with a new composite score which can be found here:
https://docs.google.com/spreadsheets/d/ ... =865187672

I then added in a new ranking getting rid of the outlier scores for each player to better reflect consensus within the models.
I also added a ranking that consists of 75% consensus analytic composite and 25% draftexpress ranking to add the subjective component.

Code: Select all

+------+-------------------------+
| Rank |         Player          |
+------+-------------------------+
|    1 | D'Angelo Russell        |
|    2 | Karl Towns              |
|    3 | Jahlil Okafor           |
|    4 | Justise Winslow         |
|    5 | Stanley Johnson         |
|    6 | Tyus Jones              |
|    7 | Frank Kaminsky          |
|    8 | Myles Turner            |
|    9 | Willie Cauley-Stein     |
|   10 | Kevon Looney            |
|   11 | Bobby Portis            |
|   12 | Kelly Oubre             |
|   13 | Delon Wright            |
|   14 | Sam Dekker              |
|   15 | Cameron Payne           |
|   16 | Jerian Grant            |
|   17 | Christian Wood          |
|   18 | Rondae Hollis-Jefferson |
|   19 | Devin Booker            |
|   20 | R.J. Hunter             |
|   21 | Trey Lyles              |
|   22 | Chris McCullough        |
|   23 | Montrezl Harrell        |
|   24 | Robert Upshaw           |
|   25 | Justin Anderson         |
|   26 | Richaun Holmes          |
|   27 | Cliff Alexander         |
|   28 | Dakari Johnson          |
|   29 | Jarell Martin           |
|   30 | Rashad Vaughn           |
|   31 | Aaron White             |
|   32 | Jordan Mickey           |
|   33 | Terry Rozier            |
|   34 | J.P. Tokoto             |
|   35 | Michael Frazier         |
|   36 | Andrew Harrison         |
|   37 | T.J. McConnell          |
|   38 | Alan Williams           |
|   39 | Michael Qualls          |
|   40 | Vince Hunter            |
|   41 | Tyler Harvey            |
|   42 | Seth Tuttle             |
|   43 | Wesley Saunders         |
|   44 | Pat Connaughton         |
|   45 | Josh Richardson         |
|   46 | Larry Nance             |
|   47 | Branden Dawson          |
|   48 | Darrun Hilliard         |
|   49 | Quinn Cook              |
|   50 | Aaron Harrison          |
|   51 | Norman Powell           |
|   52 | Anthony Brown           |
|   53 | Keifer Sykes            |
|   54 | Rakeem Christmas        |
|   55 | Derrick Marks           |
|   56 | Chasson Randle          |
|   57 | Olivier Hanlan          |
|   58 | Jonathan Holmes         |
|   59 | TaShawn Thomas          |
|   60 | Joshua Smith            |
|   61 | Joseph Young            |
|   62 | Chris Walker            |
|   63 | Shannon Scott           |
|   64 | Ryan Boatright          |
|   65 | Treveon Graham          |
|   66 | Brandon Ashley          |
|   67 | Dez Wells               |
|   68 | Corey Hawkins           |
|   69 | Julian Washburn         |
|   70 | Terran Petteway         |
|   71 | Marcus Thornton         |
|   72 | D.J. Newbill            |
|   73 | Travis Trice            |
|   74 | Juwan Staten            |
|   75 | LeBryan Nash            |
+------+-------------------------+

[Crow]

If an analytic / subjective blend is pushed, it probably should included more than one "subjective". Perhaps that could be done here by adding C Ford, maybe NBAdraft.net or others.

[Statman]

I'd love to contribute to the followup articles revealing more methodology/commentary/etc as long as I get enough heads up and can dedicate enough time. With a real job taking up my week days it is hard for me to promise anything unless I have at least a weekend to work on it.

we can plan something for Monday

Well, I think I can try - crazy busy weekend for me, but the least I can do is TRY to make that deadline.

[Crow]

Maybe mid next week or even week from next Monday would be possible? Other's call of course but doesn't seem like it needs to be a rush.


If the modelers were super gungho about, given the positive attention, perhaps a later step could be for some or all to use their method as faithfully as possible to go back and rank one or more of the 2010, 11 and 12 drafts. At least early career performance numbers would be available for evaluating the models.

[ampersand5]

IF the modelers were super gungho about, given the positive attention, perhaps some or all could use their method as faithfully as possible to go back and ranking the 2010, 11 or 12 draft. At least early career performance numbers would be available for that.

I imagine there are 0-very few out of sample models out there that this could be done for.

I'm still curious to see what the correlation is between the DE mocks every year, the actual draft results and a ranking of each class by RAPM/winshares/vorp etc.

[Crow]

Out of sample would be better but I'd rather check anyways than wait 3-5 years. If in-sample performance was relatively weak that would inform and spur further efforts at improvement. If it is strong or real strong, it is not sufficient information but right now I don't know anything specific about the possible strength, just that folks tried their best to model for it. The in sample performance provides some form of baseline. Detailed comparison and analysis of the two performance sets might give clues as to where the adjustments need to be made.

[Statman]

I created a new spreadsheet including Layne's Humble model, Kevin Pelton and Andrew Johnson's ranking with a new composite score which can be found here:
https://docs.google.com/spreadsheets/d/ ... =865187672

I then added in a new ranking getting rid of the outlier scores for each player to better reflect consensus within the models.
I also added a ranking that consists of 75% consensus analytic composite and 25% draftexpress ranking to add the subjective component.

Code: Select all

+------+-------------------------+
| Rank |         Player          |
+------+-------------------------+
|    1 | D'Angelo Russell        |
|    2 | Karl Towns              |
|    3 | Jahlil Okafor           |
|    4 | Justise Winslow         |
|    5 | Stanley Johnson         |
|    6 | Tyus Jones              |
|    7 | Frank Kaminsky          |
|    8 | Myles Turner            |
|    9 | Willie Cauley-Stein     |
|   10 | Kevon Looney            |
|   11 | Bobby Portis            |
|   12 | Kelly Oubre             |
|   13 | Delon Wright            |
|   14 | Sam Dekker              |
|   15 | Cameron Payne           |
|   16 | Jerian Grant            |
|   17 | Christian Wood          |
|   18 | Rondae Hollis-Jefferson |
|   19 | Devin Booker            |
|   20 | R.J. Hunter             |
|   21 | Trey Lyles              |
|   22 | Chris McCullough        |
|   23 | Montrezl Harrell        |
|   24 | Robert Upshaw           |
|   25 | Justin Anderson         |
|   26 | Richaun Holmes          |
|   27 | Cliff Alexander         |
|   28 | Dakari Johnson          |
|   29 | Jarell Martin           |
|   30 | Rashad Vaughn           |
|   31 | Aaron White             |
|   32 | Jordan Mickey           |
|   33 | Terry Rozier            |
|   34 | J.P. Tokoto             |
|   35 | Michael Frazier         |
|   36 | Andrew Harrison         |
|   37 | T.J. McConnell          |
|   38 | Alan Williams           |
|   39 | Michael Qualls          |
|   40 | Vince Hunter            |
|   41 | Tyler Harvey            |
|   42 | Seth Tuttle             |
|   43 | Wesley Saunders         |
|   44 | Pat Connaughton         |
|   45 | Josh Richardson         |
|   46 | Larry Nance             |
|   47 | Branden Dawson          |
|   48 | Darrun Hilliard         |
|   49 | Quinn Cook              |
|   50 | Aaron Harrison          |
|   51 | Norman Powell           |
|   52 | Anthony Brown           |
|   53 | Keifer Sykes            |
|   54 | Rakeem Christmas        |
|   55 | Derrick Marks           |
|   56 | Chasson Randle          |
|   57 | Olivier Hanlan          |
|   58 | Jonathan Holmes         |
|   59 | TaShawn Thomas          |
|   60 | Joshua Smith            |
|   61 | Joseph Young            |
|   62 | Chris Walker            |
|   63 | Shannon Scott           |
|   64 | Ryan Boatright          |
|   65 | Treveon Graham          |
|   66 | Brandon Ashley          |
|   67 | Dez Wells               |
|   68 | Corey Hawkins           |
|   69 | Julian Washburn         |
|   70 | Terran Petteway         |
|   71 | Marcus Thornton         |
|   72 | D.J. Newbill            |
|   73 | Travis Trice            |
|   74 | Juwan Staten            |
|   75 | LeBryan Nash            |
+------+-------------------------+

Wow, this blend at first sight will probably correlate quite well to my model. I gotta get this done this weekend, climb out from under my rock to be part of this info for everyone. Great job Ampersand.

[Statman]

IF the modelers were super gungho about, given the positive attention, perhaps some or all could use their method as faithfully as possible to go back and ranking the 2010, 11 or 12 draft. At least early career performance numbers would be available for that.

I imagine there are 0-very few out of sample models out there that this could be done for.

I'm still curious to see what the correlation is between the DE mocks every year, the actual draft results and a ranking of each class by RAPM/winshares/vorp etc.

I'm trying to go back every season of drafts to 1998 (my complete college stats go back to '97, gives me the 2 seasons of college stats for the '98 draft). Maybe when I'm done fixing some of the college data (ALWAYS stat errors no matter where the data comes from) - I'll at least make the last two seasons of college stats for every college player who played at least 1 minute in the NBA (&/or drafted) available to all involved (easier to get ages from basketball-reference, which is vital). I don't think that'll get me in any data hot water - since I won't be sharing ALL the college data & I "fixed" some of it anyway against many sources, so it's not from just one source.

[Statman]

Can we create a hashtag that we all might try to throw in from time to time so we can find discussions related to this specific project on twitter? Maybe #NBADraftModel ? Maybe something better?

This might get me involved with twitter debates/discussions again if I can find these specific discussions more easily.

[Crow]

Whatever the deadline, it might help to restate what additional input from the modelers is sought. Is it basically anything the modelers want to add including responses about the most over/underrated compared to DX? Can any comment on any of those or is the strong preference to get that from the extreme ranked modeler to comment first or only?


At least two of the modelers used a blend of models to create their presented model. What do folks think of separate models by quality level? Say one for top 20, once for next 20-40, one for rest? Models try to match guys up with those of similar quality in general but might 3 tier model provide more flexibility / ultimate performance? I know there are judgment calls who belongs in what tier but if time didn't constrain why not run as a unified model and a three tier one and see if anything pops up for further consideration or if three tier performs better in long run. The tiering is perhaps a subjective prior? Estimating degree of uncertainty about real talent and / or likelihood of further development?


Anybody want to attempt a small sample RAPM for games against say top 40-65 teams? Possibly multi-season and definitely including the tournaments.

[ampersand5]

BTW I just quickly ran some numbers on the first round for the 2009 draft for NCAA players, using career vorp numbers, a DX mock and a model I chose at random.

Code: Select all

+-----------------------+-------------+
|         Input         | Correlation |
+-----------------------+-------------+
| draft-model           | 0.382868937 |
| draft-mock            | 0.82249742  |
| draft-vorp            | 0.302373581 |
| model-mock            | 0.566563467 |
| model-vorp            | 0.626418989 |
| mock-vorp             | 0.547987616 |
| 75%model/25%mock-vorp | 0.663088011 |
| 50/50-vorp            | 0.663482527 |
+-----------------------+-------------+

[Crow]

So in this point sample, the models and mocks vastly outperformed the actual draft on future vorp. Blending mock with models helped a modest amount and a modest improvement is still worth going after.

More testing data needed but I'd say teams would be wise to build more and better models and employ more folks to do it. If they don't go after individuals for exclusive deals, it might be worth it for a group to come together and market a set of models, unblended and blended, in future years on a nonexclusive or exclusive basis whichever pays / rewards better.

[jessefis]

-I agree with Statman's suggestion on way to follow twitter discussion - I don't know how to find/follow either
-If you want me to do something for followup article please make clear of specifics
-At some point I plan to release historical performance of my numbers on my blog (with comments on how in/out of sample they are)
-FYI I will be continuing to modify my projections with enhancements as we near the draft (I was a bit rushed for this article) so my numbers will likely change a bit. I will likely have v2 released sometime this weekend.

[Statman]

So in this point sample, the models and mocks vastly outperformed the actual draft on future vorp. Blending mock with models helped a modest amount and a modest improvement is still worth going after.

More testing data needed but I'd say teams would be wise to build more and better models and employ more folks to do it. If they don't go after individuals for exclusive deals, it might be worth it for a group to come together and market a set of models, unblended and blended, in future years on a nonexclusive or exclusive basis whichever pays / rewards better.

The first part isn't a surprise - the little historical testing I've done has my model outperforming the actual draft by quite a bit. This is without any scouting (ie subjective) data and combine results. This is why I've decided I will test the last 17 seasons - to fully flush this out. Takes a while. Pretty much every "surprise" quality late pick (in retrospect) my model would have tabbed as being a much better prospect than their actual draft position would suggest (Boozer, DeAndre Jordan, Draymond Green, Milsap, Kevin Martin, Arenas, Chalmers, etc). Almost every horrible bust my model didn't like nearly as well as their actual draft position - although Michael Beasley does stand out as a big whiff in my model, although I think any model ever created would have whiffed there.

There isn't a single true NBA superstar from college that my model wouldn't have tabbed super high. Tim Duncan has the highest single season college rating ever (since '97) by a longshot. Anthony Davis is the highest rated college 18 year old who went pro - followed by Irving (very limited Duke data), Durant, a little drop to Carmelo, another drop to Bosh, another drop to Favors, then we hit the bust Eddie Griffin. Griffin's college rating was 153 compared to AD's 197, Irving's 193, & Durant's 184 - he would not have projected anywhere close to those guys - BUT he would have projected very well in general (if he didn't have any bad limiters like high personal foul rate, etc - which is why I need to fully apply the model to all the past seasons).

Highest rated guards (college rating) 21 years old or under in college who played in the NBA the next season: Steph Curry & Allen Iverson both had 203 ratings as 20 year olds, followed by Kemba Walker, Dwyane Wade, Kyrie Irving, & Ray Allen. Since my model uses the college rating (broken down at every subset) as a base for the NBA career projections, suffice to say these guys would all have been considered certainties for having a good/great NBA career.

I'll take back my model not missing any NBA superstar - Westbrook had a 140 rating as a 19 year old in college, I'm guessing my model would have pegged him somewhere in the 5th to 10th best prospect in his draft - that's probably not being tabbed "super high". Not quite the Irving, Harden, Chris Paul, Mike Bibby, Derrick Rose, John Wall, Mike Conley, TJ Ford, John Wall, Tyreke Evans, & Rondo. Those are the highest rated 18 or 19 year olds PGs/combos who played in the NBA the next season. The next in that list was William Avery, who would be a miss, but he's a ways down from the top guys. Avery (in the '99 draft) rated better than Westbrook in college. There is NO doubt I would have DEMANDED Kevin Love be drafted instead of Westbrook in his draft - Beasley or Love almost certainly would have been my top prospects. I also would have lauded the Timberwolves for taking William Avery 14th in 1999. I would have been wrong.

Looking closer at Avery, he was bad as a frosh, great as a soph. I use weighted two seasons from college, he would not have been rated higher than Westbrook by the model. I'm just glancing over my spreadsheet of draftees that has their last college season rating.

My model doesn't "find" every guy by any means - Eric Bledsoe was not liked by my model. But, I am about certain that any player rated very highly by my model who drops to the 2nd round or goes undrafted is a much better prospect than the scouts give them credit for - and almost any guy my model HATES who ends up going lottery will be a bust. The exceptions from these outliers are few and far between from what I've seen so far. I plan on making all my historical results public before the draft for complete scrutiny.

All that being said - I am certain a quality model COMBINED with quality scouting & taking into account some pertinent combine stuff (w/o going overboard) would do much better than what teams have generally done historically. Much lesser chance of the horrible bust. Much better chance at getting that quality 2nd round or undrafted guy who very well could pan out to be a nice contributor for years. A quality model probably would probably really help D league teams draft. A quality model would help WNBA teams draft.

So Crow, your second point - I completely agree. There is so little risk for NBA teams to hire a team of guys whose whole responsibility is to combine their skills to best model NBA career projections (MAN I wish I could build databases & data mine worth a crap - but others can, no I in TEAM) - coming out of college, from foreign leagues, from D Leagues, etc. One could incorporate the high level AAU leagues now that they have greatly improved their statkeeping (making it easier to adjust for pace, SoS, etc). One could model from all the past Summer League & Exhibition seasons to maybe help tab current players who might already be ahead or behind projections during their summer league & exhibition. Projections could be updated daily. Projections would make salary & free agent decisions easier & better planned out long term. The work could be extended to D League & WNBA, PLENTY of work for the "modelers" to earn their paychecks - which combined might not reach NBA minimum salary for one player (depending on how big the model "team" is).