A baseball player’s defensive skills are often debated heavily, as old-school minds tend to believe in the “eye test” more, while more analytically-minded fans and analysts lean on advanced metrics. However, even if those sabermetric fans rightfully believe in defensive metrics, there are so many different ones that it’s hard to know which ones to trust. Sometimes, players can rate out completely differently in each of the various metrics. Today, we’ll attempt to conclude which defensive metric is the most reliable.

To assess how reliable a defensive metric is, we’ll look at its stability from year-to-year. Although some players can drastically improve their defensive abilities in a year, most players shouldn’t see their defensive numbers suddenly increase or decrease; the more stable the metric, the more effective it would appear to be.

Before we start, let’s go over which metrics we’ll be analyzing:

• Defensive Runs Saved (Drs) rates defenders based on the number of runs they save.
• Ultimate Zone Rating (UZR) is very similar to defensive runs saved. Notably, it takes into account range, arm, double-plays, and errors. It is also the defensive metric used to calculate WAR at Fangraphs for non-catchers.
• Outs Above Average (OAA) was a recent creation by Baseball Savant, which is able to measure a player’s defensive ability by taking into the account the probability of a play being converted, so it takes into account how effective a fielder is on every play, and credits those who make more difficult plays.

For a complete deep dive into these statistics, I recommend clicking the links that are attached to their names above. Now, let’s get to the results!

We’ll start with OAA. Since it’s calculated differently for infielders and outfielders, unlike the other two metrics, infield OAA and outfield OAA are essentially two different statistics. When comparing the two, it’s obvious that OAA does a much better with one specific position group:

When dissecting the coefficient of determination of these two statistics, it’s telling that outfield OAA (bottom) is almost four times as predictive from one year to the next as infield OAA (top). It was only recently that Baseball Savant came up with infield OAA, so maybe the results will get better over time, but when looking at the calculation of OAA, it’s obvious to tell why it does a much better job of grading outfielders. Whereas outfielders are evaluated based on catch probability, there are a lot more moving parts when judging infielders; part of their calculation of the difficulty of a play is the speed of the runner, so it’s more dependent on outside circumstances.

So, how do infield OAA and outfield OAA compare with drs and UZR? Let’s look at the stability of the latter two metrics:

The coefficients of determination from year-to-year for Drs and UZR are nearly the same, so between those two metrics, each are an important component when assessing a player’s defensive ability. However, when comparing both to OAA, we can come to a more definitive conclusion.

Since Drs and UZR are both roughly twice as stable as infield OAA, we’d be best off using the former metrics to judge infielders, at least until the creators of OAA can fine-tune the statistic to improve its reliability. However, where Baseball Savant is ahead of the curve is their ability to grade outfield defense,; their usage of catch probability really does a proper job of assessing their overall range by looking at their actual success rate versus estimated success rate. Meanwhile, both Baseball Savant and Fangraphs do a reasonable job with catcher defense, so both can be used in collaboration.

The debate between eye-test and defensive metrics in evaluating defense is a completely different conversation. However, for those who realize the flaws of using the eye-test, which inflates highlight-reel plays and isn’t able to take into account consistency, it’s important to have a guide as to which defensive statistics to use. It’s a good idea to use all of these metrics to a certain extent, but to sum it up, Drs and UZR are the superior statistics for infield defense, while OAA is the most stable metric for outfield defense. This is good news for Marcus Semien and Didi Gregorious, infielders who grade out much better with Drs and UZR than OAA, but the opposite is true for the likes of Justin Turner, Freddy Galvis, and Carlos Correa- their OAA may cause people to overrate their defensive abilities. Now, considering that the highest coefficient of determination (outfield OAA) was 40.8%, it’s clear that no defensive metric is completely stable; players have the ability to get worse or better based on the practice they put in and their athletic abilities. Still, it’s what we have to work with, so taking these findings into account can give you the best read of how well a player is supporting their pitcher with quality defense.