Home » What is the best advanced statistic for basketball? NBA executives weigh in

What is the best advanced statistic for basketball? NBA executives weigh in

by TLD

Modern basketball debates often include one of the parties citing advanced analytics to prove their point. But are those metrics any good?

While some may shy away from numbers when talking about athleticism, others have embraced the statistical revolution. We were curious, though, which of those numbers we should reference in our player evaluations. Is there a catch-all, all-in-one composite metric that has the best reputation and has the most accurate assessment of a player’s holistic impact on winning?

We wanted to find out so, for this story, we surveyed some of the most trusted thinkers in the basketball community.

HoopsHype received answers from nearly 30 participants, including various media members as well as individuals who have a combined experience with more than half of the teams in the NBA. Answers came from folks at every level within an organization, including those who work on a coaching staff as well as several different directors of analytics departments.

Most who answered spoke on the condition of anonymity because they are currently employed for NBA teams and felt that speaking publicly could reveal proprietary information about their teams.

But some like Cory Jez, who was the head of analytics for the Utah Jazz, were kind enough to walk us through the key principles that a team analytics staffer would want to incorporate in their analysis.

More representative all-in-one metrics need to capture the impact players can have when they don’t log a box score event (see: basically every Rudy Gobert possession ever),” said Jez. “It’s much harder to see the impact a player like De’Anthony Melton has compared to a different substitute like Lou Williams.”

Jez told us his criteria for a good formula would include possession-based data featuring more than just box-score metrics; methods that are Bayesian in nature that consider the individual player contribution and not just possession results; inputs that include tracking data; statistics that properly handle the sample size problem of single-season data by including historical information.

While we received overwhelming interest in this project from individuals like Jez, others declined participation. Some felt that as a whole, catch-all stats are flawed and do not very accurately measure talent or performance.

“I don’t really use any,” said one executive, who is the president of basketball operations for a team in the Eastern Conference. “They are all pretty bad.”

Others were less critical but felt that while all-in-one composite metrics are constantly getting better, the future of analytics is headed away from these measurements altogether.

“If I could add a wrinkle to your story, it would be that all-in-one stats are overused – that the next phase of basketball analytics is all about context-dependent numbers,” said another front office member from the Western Conference. “That would be the most honest quote I could give.”

This executive feels that analytics will move away from ridge-regression-based stats and instead attempt to answer questions about forecast future performance based on roles the player had for their team (e.g. BBall-Index.com and Backpicks.com have metrics on lineup spacing, playmaking value and defensive versatility).

The executive also added that “averaging across a few numbers you trust” (which is something Owen Phillips suggested in this newsletter) is “probably the way to go” to get the best evaluation.

However, the most common feedback to the survey we received was that most teams focus on their own custom-developed systems when evaluating players.

But those measurements aren’t available to the public or to the media and can’t be readily cited. Ultimately, the goal of this project is to provide the most updated feedback on the evaluation tools that you can actually use.

Based on conversations with some of the most trusted names in basketball, here is what we learned about the state-of-the-art, publicly available metrics. The following rankings included below are sorted from least trusted to most trusted.

Full Name: Player Efficiency Rating

Metric Type: Box score (not estimating player plus-minus)

Developer: John Hollinger

Data: ESPN.com

Most Recent Leaders: Nikola Jokic, Joel Embiid, Giannis Antetokounmpo, Zion Williamson, Jimmy Butler

Player Efficiency Rating (PER) is an all-in-one metric created by John Hollinger. He originally introduced this statistic for ESPN.com back in 2007, though he had used the metric much earlier than that.

Hollinger created a unique pace-adjusted, per-minute rating of player productivity and it had a home on the Worldwide Leader of Sports. Not only did PER have a prominent placement but it was easy to interpret. The league average score for PER is always 15.0, which makes it digestible and scalable to compare to someone’s points per game.

As a whole, the introduction of PER was a gigantic step forward to welcome more advanced analytics into the mainstream and it is still regularly cited among even casual fans.

Hollinger was a pioneer in his own right and has enjoyed a long career in basketball – he worked in the front office for the Memphis Grizzlies and is now a writer for The Athletic.

Even though the stat has its shortcomings (you can read about them here, here or here), PER represents an important milestone in the history of basketball analytics. Additionally, due to the fact that PER’s limitations are very well-cited, it’s easy to understand the blind spots.

“I did not list PER as a metric that I trust,” one executive, who is currently the director of analytics for a team in the Eastern Conference, told HoopsHype. “But it is ubiquitous and it is easy to understand what it is and what it isn’t. That makes it useful as a quick and dirty summary statistic.”

SURVEY SAYS: Among the 29 individuals who participated in our survey, 22 responded that they did not trust PER (Player Efficiency Rating) as an effective all-in-one metric. None of the participants who answered said PER was their preferred catch-all statistic.

Full Name: Player Impact Estimate

Metric Type: Box score (not estimating player plus-minus)

Developer: NBA

Data: NBA.com

Most Recent Leaders: Joel Embiid, Nikola Jokic, Giannis Antetokounmpo, LeBron James, Jimmy Butler

Much like PER, Player Impact Estimate (PIE) is an all-in-one measurement, but this one was actually created by the league.

For this metric, when you add up the sum of every player’s score, the results equal 100. Essentially, it aims to answer what percentage of the events in a game each player contributed.

PIE’s formula also seeks to add more value to defense than PER’s equation and it is also more simple to calculate at home. Because of the value on defense, PIE was one of the few advanced metrics where Joel Embiid scored better than MVP winner Nikola Jokic.

SURVEY SAYS: Among the 29 individuals who participated in our survey, 20 responded that they did not trust Player Impact Estimate (PIE) as an effective all-in-one metric. None said PIE was their preferred catch-all statistic.

Full Name: Win Shares per 48 minutes

Metric Type: Box score (not estimating player plus-minus)

Developer: Justin Kubatko

Data: Basketball-Reference.com

Most Recent Leaders: Nikola Jokic, Joel Embiid, Jimmy Butler, Rudy Gobert, Giannis Antetokounmpo

Anyone who read Moneyball (2003) by Michael Lewis is familiar with the concept of win shares in baseball. Originally developed by Bill James for the MLB, the conceptual goal of win shares is to assign credit for team success to individual players.

Using some of the ideas Dean Oliver explained in Basketball on Paper (2004), Basketball-Reference’s Justin Kubatko created a version of the evaluation that translates to the NBA.

It’s a fun measurement because it assigns a score that estimates exactly how many wins a player contributed to his team’s record because after all, getting wins is the ultimate goal of a front office.

But based on results from retrodiction testing by Ben Taylor for Nylon Calculus, Win Shares (much like Box Score Geek’s Wins Produced) yielded particularly volatile results.

SURVEY SAYS: Among the 29 individuals who participated in our survey, 14 responded that they did not trust Win Shares/48 Minutes (WS/48) as an effective all-in-one metric. Only one of the participants who answered said that Win Shares were their preferred catch-all statistic, but curiously that individual was actually the highest-ranking individual with an NBA team of anyone who filled out our questionnaire.

Full Name: Floor Impact Counter

Developer: Chris Reina

Metric Type: Box score (not estimating player plus-minus)

Data: RealGM.com

Most Recent Leaders: Nikola Jokic, Russell Westbrook, Giannis Antetokounmpo, Nikola Vucevic, Luka Doncic

Floor Impact Counter (FIC) is RealGM’s version of PER and PIE. It was developed in 2007 and it differs from other efficiency stats in that it assigns more weight to assists, shot creation as well as offensive rebounding.

It is sometimes presented as FIC40, which is an individual’s FIC per 40 minutes. This metric favors players who can fill up the box score like Russell Westbrook. But anyone who has watched Westbrook over the past few seasons knows that filling up the box score doesn’t always translate to winning.

While it’s not widely cited in the NBA, RealGM has data for other levels of play (such as college basketball, G League and international) so FIC can definitely come in handy in those instances.

SURVEY SAYS: Among the 29 individuals who participated in our survey, eight responded that they did not trust Floor Impact Counter as an effective all-in-one metric. None said FIC was their preferred catch-all statistic.

Full Name: Simple Rating

Metric Type: Box score (estimating player plus-minus)

Developer: Roland Beech

Data: 82Games.com

Most Recent Leaders: Joel Embiid, Giannis Antetokounmpo, Jimmy Butler, Kawhi Leonard, LeBron James

Simple Rating was developed by Roland Beech. He founded 82games.com in 2002 and was then hired by the Dallas Mavericks, where he became the first “stats coach” in the NBA. Beech served as the VP of Basketball Operations for Dallas and served as VP of Basketball Strategy for the Sacramento Kings until 2017.

Beech was a thought leader for on-off stats and his goal for “simple rating” was to measure the productivity of a player on the court compared to the counterpart player on the other team.

For what it’s worth, Simple Rating is one of the only evaluations that did not have Jokic rank among the five best players this past season.

SURVEY SAYS: Among the 29 individuals who participated in our survey, five responded that they did not trust Simple Rating as an effective all-in-one metric. No participants said that Simple Rating was their preferred catch-all statistic.

Full Name: Win Probability Added

Metric Type: Box score (not estimating player plus-minus)

Developer: Mike Beuoy

Data: Inpredictable.com

Most Recent Leaders: Damian Lillard, Stephen Curry, Zion Williamson, Nikola Jokic, Jayson Tatum

Back in 2014, Mike Beuoy introduced win probability contributions, which are weighted to give more credit to clutch shots. He defined some of his methodologies here, here and here.

The most interesting takeaway is that he focuses on missed shots, made shots, turnovers and free throws while ignoring rebounds, assists, blocks and steals. On his site, you can also sort individual performance based on how well someone did in clutch minutes compared to how well they did while in garbage time.

Inpredictable has another catch-all metric named kWPA (kitchen sink win probability added) that factors in all box score stats like rebounds, assists, blocks and steals. They also do great work examining clutch shooting and pace of play in the NBA.

SURVEY SAYS: Among the 29 individuals who participated in our survey, five responded that they did not trust Win Probability Added as an effective all-in-one metric. No participants who answered said that WPA was their preferred catch-all statistic.

Full Name: Real Plus-Minus

Metric Type: Box score using tracking data and on-off impact hybrid (Prior Informed RAPM)

Developer: Jeremias Engelmann, Steve Ilardi

Data: ESPN.com

Most Recent Leaders: Stephen Curry, LeBron James, Rudy Gobert, Paul George, Giannis Antetokounmpo

ESPN’s Real Plus-Minus (RPM), which was introduced in 2014 and is credited as launching a new era of composite metrics, aims to replicate the plus-minus component of a box score while also sifting through the NBA’s play-by-play data.

RPM was next in the lineage that included the groundbreaking WINVAL ratings (developed by Wayne Winston and Jeff Sagarin in 2002), APM (adjusted plus-minus developed by Dan Rosenbaum in 2004) and xRAPM.

Especially with its prominent placement on ESPN, RPM was long considered to be the gold standard among the adjusted plus-minus stats. But then sometime last year, something strange happened.

“RPM created by Jeremias Engelmann is not the one currently displayed on ESPN,” responded one individual, who works as the director of basketball analytics for a team in the Western Conference. “I would rank the former very highly while the latter is in my estimation fairly useless.”

Overall, the opinions about ESPN’s RPM were the most divisive of any metric mentioned. Some even felt that there may be something broken in whatever coding is currently formulating the results.

SURVEY SAYS: Among the 29 individuals who participated in our survey, two said ESPN’s Real Plus-Minus (RPM) was their preferred all-in-one metric. Eight others said they trusted RPM whereas 11 responded that they did not trust RPM.

Full Name: Box Plus-Minus

Metric Type: Box score (estimating player plus-minus)

Developer: Daniel Myers

Data: Basketball-Reference.com

Most Recent Leaders: Nikola Jokic, Giannis Antetokounmpo, Stephen Curry, Jimmy Butler, LeBron James

Box Plus-Minus (BPM) aims to evaluate how much a player contributed when they are on the court. BPM incorporates box score performance, team performance and player position to rate how well a player performed above league average per 100 possessions.

BPM is essentially the same statistic as VORP (Value Over Replacement Player) but the latter accounts for playing time.

For BPM, the league average is set at 0.0 and you can compare individual performances (e.g. 10.0 is considered to be an all-time great season, 8.0 is considered MVP level, 4.0 is someone who deserves All-Star consideration, -2.0 is a bench player, etc.) quite easily.

When he developed BPM 2.0, Daniel Myers wrote that he wanted his formula to only use stats that are readily available.

Unlike ESPN’s RPM, this measurement is comparably simple and doesn’t include tracking data. As such, despite relying on less information, it’s interesting to note that many actually prefer BPM over the more complex RPM.

The results for BPM can be found at Basketball-Reference. But it’s also worth noting that Ben Taylor, author of Thinking Basketball (2016), has his own version of BPM that many said they trusted as well. Those results can be found here.

SURVEY SAYS: Among the 29 individuals who participated in our survey, two said Box Plus-Minus was their preferred all-in-one metric. Eleven others said they trusted BPM as an effective all-in-one metric whereas three said they didn’t trust BPM.

Full Name: Regularized Adjusted Plus-Minus

Metric Type: Pure on-off impact with no box score

Developer: Joe Sill

Data: NBAShotCharts.com

Most Recent Leaders*: Stephen Curry, Chris Paul, LeBron James, Giannis Antetokounmpo, Rudy Gobert

As mentioned in the section about ESPN’s RPM, Regularized Adjusted Plus-Minus (RAPM) works off the basic principles of Adjusted Plus-Minus (APM), which applies linear algebra to standard plus-minus found in a box score.

But the main difference from APM is that RAPM applies a linear ridge regression filter to account for regularization. You can learn more about the Bayesian process used here and you can read a history lesson about RAPM here.

One of the most important things to know about these calculations, though, is the difference between prior informed and non-prior informed when it comes to RAPM.

The use of “priors” to the plus-minus measurements like RAPM allows for certain metrics to pass the “sniff test” at first glance. Utah’s former head of analytics, Jez, offered a simple explanation:

“We know that LeBron James is really really good, we shouldn’t have to wait for hundreds of possessions to tell us that,” Jez told HoopsHype. “Instead, modelers can use a prior to start the model off in the right direction.”

Of course, explained Jez, how those priors are created is subjective and it’s what makes the “secret sauce” behind stronger predictive metrics.

There are some obvious flaws with RAPM when it is used by itself, chief among them being that there are no real “basketball” values like rebounds or assists used in the input beyond just the score of the game when an individual is and isn’t on the floor.

Even though RAPM ignores the traditional box score elements, it can be very helpful as a base component in other popular composite metrics like EPM (more on that later!) to measure on-off factors.

It’s also worth noting that several who responded also told HoopsHype that this metric is at its most useful when studying multi-year results (e.g. three-year RAPM and five-year RAPM, which software engineer Ryan Davis updates here) but that single-season data can be misleading.

“It needs to be considered in [the] context of the player and their role,” said one director of analytics for a team in the West. “But it is a good objective measure when you have multiple seasons.”

SURVEY SAYS: Among the 29 individuals who participated in our survey, more than half (15) said they trust Regularized Adjusted-Plus Minus (RAPM) as an all-in-one metric and only two said they did not. But none said that RAPM was their preferred catch-all metric.

* = Leaders based on 5-year RAPM

Full Name: Robust Algorithm (using) Player Tracking (and) On/Off Ratings

Metric Type: Box score with tracking data and plus-minus hybrid (estimating RAPM)

Developer: Jay Boice, Neil Paine and Nate Silver

Data: FiveThirtyEight.com

Most Recent Leaders: Nikola Jokic, Joel Embiid, Rudy Gobert, Kawhi Leonard, Mike Conley

Perhaps the most household name in the data world is Nate Silver, so it’s no surprise to see that his publication FiveThirtyEight has a widely-cited catch-all basketball metric: RAPTOR.

FiveThirtyEight’s previous basketball metrics (Elo and CARM-Elo) relied on BPM and RPM for calculations. But RAPTOR, introduced in 2019, offers original insight from their data scientists. This evaluation incorporates a blend of box score data, play-to-play data, player-tracking data and an on-off component.

The main thing to know about RAPTOR is that it includes a ton of descriptive data that are as specific as fast-break starts, isolation turnovers and distance traveled for perimeter defenders.

One executive told HoopsHype that they preferred RAPTOR because it is created in such a different way than other metrics and that they consider it to be useful as a “robustness” check.

RAPTOR is deservedly well-respected. However, studies have shown that its limitations are that is not as useful as a predictive tool when there is a smaller sample size of data (minutes played) or when there is a significant amount of year-over-year roster turnover.

SURVEY SAYS: Among the 29 individuals who participated in our survey, six said that RAPTOR was their preferred catch-all metric, which made it the third-most-popular in that regard. Eight others said that they trust RAPTOR as an all-in-one metric but an additional seven said that they did not.

Full Name: Luck-adjusted player Estimate using a Box prior Regularized ON-off

Metric Type: Box score and on-off impact hybrid (Prior Informed RAPM)

Developer: Krishna Narsu, Tim/Cranjis McBasketball

Data: BBall-Index.com

Most Recent Leaders: Nikola Jokic, Rudy Gobert, Giannis Antetokounmpo, Jimmy Butler, Joel Embiid

One of the newer metrics that has become increasingly popular over the past few years is BBall-Index.com’s LEBRON. While it’s fresh to the scene, it has performed pretty well thus far.

This rating uses box score results and on-off calculations (specifically, luck-adjusted RAPM) for an impact score measured per 100 possessions. The box score component of LEBRON uses weights from PIPM (Player Impact Plus-Minus).

PIPM – no longer publicly available – was created by Jacob Goldstein, who worked for BBall-Index before he was hired to work for the Washington Wizards, Washington Mystics and Capital City Go-Go.

For stabilization purposes, they incorporate their offensive archetypes, which labels players based on the role they play for their team. According to the site, LEBRON is the only impact statistic that measures “role-adjusted, stabilized and luck-adjusted values utilizing actual RAPM calculations.”

BBall-Index also mentions that the “next generation” of LEBRON will incorporate tracking data.

SURVEY SAYS: Among the 29 individuals who participated in our survey, four said that LEBRON was their preferred catch-all metric. Fourteen others said that they trust LEBRON as an all-in-one metric while two said that they did not.

Full Name: Estimated Plus-Minus

Metric Type: Box score and on-off impact (Prior Informed RAPM)

Developer: Taylor Snarr

Data: Dunksandthrees.com

Most Recent Leaders: Nikola Jokic, Rudy Gobert, Stephen Curry, LeBron James, Kawhi Leonard

One unsurprising trend we can pick up from this study is that newer metrics tend to be trusted more than the older ones and the popularity of Estimated Plus-Minus (EPM) is a clear example.

EPM was developed by data scientist and former Utah Jazz basketball analytics coordinator Taylor Snarr. When he introduced EPM back in February 2020, he described it as the “most accurate” all-in-one NBA player metric.

Science proves this as well. Based on a comparison study, EPM did the best job at assigning credit for impact to the correct players. You can see the results here, although it is worth noting that the results of the metric comparison are actually very similar to answers provided in this survey.

Here is what a current Eastern Conference analytics staffer, who felt that the study was set up pretty well and points to how well EPM compares to other metrics, told HoopsHype:

“I will say the part I like most about EPM is that he put together a study that shows how well it projects compared to other metrics. I always wonder why more people don’t do this. If your metric is better, show me why it is, don’t just tell me it is.”

The analyst added that all-in-one metrics that typically perform the best are adjusted plus-minus metrics that include priors.

Meanwhile, like the aforementioned RPM and RAPTOR, Snarr’s EPM is a hybrid model that uses player-tracking data as part of its evaluation.

We asked Steve Ilardi, co-creator of ESPN’s Real Plus-Minus who has worked as a consultant for NBA teams including the Phoenix Suns, what he thought about the metric.

“[EPM] does a superb job of reducing the high noise level inherent in basic RAPM modeling estimates by integrating Bayesian priors derived not just from box score stats (a la BPM) but also from player tracking metrics,” said Ilardi, during his recent conversation with HoopsHype. “[EPM] has gone one step beyond the RPM estimates that Jerry Engelmann and I developed for ESPN, as we began that project without access to player tracking metrics.”

“I regard EPM as the obvious gold standard of all-in-one metrics,” added Ilardi.

Additionally, not only is the data trusted but the website also looks pretty modern. That’s a nice change of pace because some of the other sites that host these metrics are outdated and, admittedly, can be really hard to look at it.

Dunksandthrees.com also offers a great user experience that includes fantastic data visualizations and color-coded efficiency ratings for skills like defense and ball handling.

SURVEY SAYS: Among the 29 individuals who participated in our survey, six said that EPM was their preferred catch-all metric. That was the second-most among all metrics. Eleven others said that they trust EPM as an all-in-one metric while only one said that they did not.

Full Name: Daily Plus-Minus

Metric Type: Predictive tool estimating plus-minus

Developer: Kostya Medvedovsky

Data: DARKO.app

Most Recent Leaders: Giannis Antetokounmpo, Kawhi Leonard, LeBron James, Joel Embiid, Kevin Durant

The “winner” of our composite metric survey goes to DARKO. This is an application developed by Kostya Medvedovsky and hosted by Andrew Patton.

Their website defines DARKO as a “machine learning-driven basketball player box-score score projection system” similar to MLB’s PECOTA or ZIPS. Their application updates daily for every player in the NBA. The tool has inputs from NBA.com, Basketball-Reference and PBPStats.com.

DARKO also accounts for time series and sample size using the complex methods of “exponential decay” and “Kalman filters” to take all historical data into account.

“This is a huge breakthrough because when using possession-level data, it can be tricky to tease out the signal from the noise and oftentimes one season of data is not enough to ensure the proper accounting,” explained Jez, the former head of analytics for the Jazz. “Medvedovsky has solved for this with DPM, allowing him to confidently say when a players’ improvement is more signal than noise.”

The results are a Bayesian model that predicts all elements of a box score. The app shows a projected skill curve for each player highlighting elements such as perimeter shooting and other box-score-specific elements.

For this exercise, in particular, we focused on its results for Daily Plus Minus 2.0 (DPM), DARKO’s version of a catch-all metric. The main difference between DPM and the other metrics is that it solely looks forward.

It won’t answer who should have won MVP but it’s a powerful tool for projections. In fact, DPM beats all the other public-facing metrics in predictive power, based on root mean square error (RMSE). The next two best metrics graded out, in order, were EPM and LEBRON – just like in our survey.

While it might not have the same popularity among casual fans as PER, it might be time to add DARKO’s Daily Plus-Minus to your list of metrics that you check if you’re interested at all in NBA player evaluations – especially because like Dunksandthrees.com, DARKO’s application also has some great data visualization tools.

As others begin to consider what to include in their composite metrics, it is also worth noting that a predictive model fared better than more traditional evaluations.

SURVEY SAYS: Among the 29 individuals who participated in our survey, eight (8) said that DPM was their preferred catch-all metric. That was the most among all metrics. Ten (10) others said that they trust DPM as an all-in-one metric while only one (1) said that they did not.

HONORABLE MENTIONS (stats that are not publicly available)

APM (Adjusted Plus-Minus // Developer: Wayne Winston and Jeff Sagarin)

SPM (Statistical Plus-Minus // Developer: Dan Rosenbaum)

WARP (Wins Above Replacement Player // Developer: Kevin Pelton)

IPV (Individual Player Value // Developer: Talking Practice)

DRE (Daily RAPM Estimate // Developer: Kevin Ferrigan)

SPR (Simple Player Rating // Developer: Nathan Walker)

PIPM (Player Impact Plus-Minus // Developer: Jacob Goldstein)

AuPM (Augmented Box Plus-Minus // Developer: Ben Taylor)

Main Image: Coley Cleary / USA TODAY Sports Media Group



Source link

You may also like