Inter-Projection Volatility & The ATC Projections

The 2021 ATC Projections have arrived at FanGraphs!

The Average Total Cost projection system (ATC) gets its name from the fact that it “averages” many other projection systems together. ATC also happens to be my initials.

ATC is a “smart” projection aggregation model. While most other aggregation systems typically apply equal weight to all underlying data sources, ATC assigns weights based on historical performance. The method is similar to what Nate Silver does with his political forecasting at www.fivethirtyeight.com. Read more about how ATC works in last year’s introductory ATC article.

ATC’s advanced methods have paid off. Last year, FantasyPros ranked ATC as the most accurate baseball projections, even more accurate than their own. Annually, I conduct my own projections review using a game theory method, which has consistently shown that ATC has outperformed other projections. You can read the results of last year’s study here.

Starting this year, ATC will include three new volatility metrics to be described later in this article. Aside from introducing the concepts behind these new metrics, we will also discuss how a fantasy manager might use them in practice.

First, let’s quickly remind everyone on where to find the ATC projections at FanGraphs.

Viewing ATC

There are currently three ways to view and utilize the ATC projections on the site.

• Full Projections
• Individual Player Pages
• Auction Calculator

Full Projections

The full ATC projections can be found by clicking the “Projections” dropdown tab on the main menu bar at the FanGraphs site, and then by clicking on “ATC.” Track your favorite leaderboard stats by sorting on any column. The projections are also downloadable with most of the ATC projected statistics.

Here are links to the individual ATC hitter and pitcher projections:

Full 2021 ATC Hitter Projections

Full 2021 ATC Pitcher Projections

Individual Player Pages

The ATC projections are also available on individual player pages. You can take a look at past player performance, and compare ATC with other FanGraphs systems such as Steamer, THE BAT and ZiPS.

Auction Calculator

Finally, if you are looking for ATC player rankings, use the FanGraphs’ Auction Calculator. Positional rankings will differ greatly among various fantasy league formats. Since ATC-generated rankings are not directly published, please use this helpful online tool to generate your own ATC-based rankings.

You can access the Auction Calculator by clicking on the “Projections” dropdown tab on the main menu bar, and then by clicking on “Auction Calculator.”

Once you are on the Auction Calculator page, look under “League Settings” for a pulldown menu where you can choose ATC as your source projection system.

Then, enter your league’s specific roster settings, and the calculator will generate rankings and auction dollar values for all players using FanGraphs’s Z-Score valuation method.

Process & Parameter Risk

Before unveiling ATC’s new volatility metrics, let’s review how player projections work.

Let’s take the Steamer projection system for example. Steamer uses their proprietary methodology to project future player performance. According to MLB.com, Steamer incorporates past performance, aging trends and pitch-tracking data to help forecast players in the upcoming season.

When Steamer generates a 32 HR projection for Freddie Freeman, Steamer is not guaranteeing that the Braves’ 2020 MVP will exactly hit exactly 32 homeruns.

Instead, Steamer projects many possibilities for what could happen in 2021. There is a range of outcomes that Freeman can achieve, and Steamer quantifies them. 38 HRs might represent a 90^th percentile outcome for him, and 23 HRs may depict a 30^th percentile outcome, etc.

What is displayed in the commonly viewed projection line is either the average of all modeled scenarios, or often the median outcome itself. 32 HR is Steamer’s modeled measure of Freeman’s true talent level for power.

Due to the 162-game season sample size, there will always be variations above and below a player’s true average/median expectation. A bit of good luck in one season could produce three more HRs, while extreme bad luck may take away seven.

The term for the inherent variance due to sample size is known as process risk.

If there were ten million games played in a season, with no external or unexpected factors – Steamer would expect that Freddie Freeman would average close to 32 HRs per 162 games (his true talent) in the long run. But in any standard 162-game stretch, there will always be higher and lower performances.

To give an example from my insurance/actuarial background – some [insurance] lines of business exhibit relatively small variations year to year, while some display large annual variations. For example, automobile physical damage insurance is one of the lines with relatively low process risk; results hardly vary from year to year. Earthquake insurance, however, is one of the more highly variable lines of insurance where insurers do not receive any large claims in most years.

The same can be also said regarding major league baseball players. There are some more consistent performers year to year, while many others exhibit a wider range of outcomes.

What if Steamer was wrong about a player’s projection? Perhaps THE BAT’s 34 homerun projection was the more accurate forecast of Freeman’s true talent. What if ZiPS’ 28-HR forecast was truly the better long-term bet?

The term for the uncertainty of the true expectation is referred to as parameter risk.

Process risk cannot be materially minimized, nor should it – but parameter risk has a chance to be reduced. One major strength of the ATC projections is its ability to reduce parameter risk. By utilizing and combining many sets of expectations, ATC reduces the risk of relying on any one single projection system.

Inter-Projection Volatility

How can we quantify player projection risk?

Most projection experts grapple with this difficult question. The ATC Projection system provides us with one way of looking at projection volatility resulting from parameter risk. It does this by looking at the distribution of all of its underlying projections.

To illustrate the concept, take a look at the following two 2021 player projections:

For Jackie Bradley Jr., there is little difference between how the models project HRs and SBs. Assuming that each of the five systems receive equal weight, the standard deviations of just 1.1 for HRs and 0.5 for SBs depict an extremely tight range of projected outcomes.

By contrast, the distributions for Matt Carpenter are a bit more variable. The stolen base totals are low, and therefore are not as important. But look how wide Carpenter’s homerun projections vary – a 2.9 SD on top of a 20.4 average. The possible range of expectations lie from 16 to 23! As opposed to Bradley, projections for Carpenter exhibit less agreement due to the ZiPS 16-homer projection outlier.

If we only looked at a simple average of the five FanGraphs models, the average HR projections would be 19 for Bradley and 20 for Carpenter. But that would not show just how variable the underlying data is. Carpenter’s projected HR totals are more variable than Bradley’s, but they do not skew nearly as far downwards.

The extra information (of the standard deviation and skewness) paints a more vivid picture of the underlying projections.

In 2021, ATC will not list out volatility metrics for every projected category. Instead, ATC will present volatility metrics on a total player-value basis.

To do this, auction values are first internally computed for each underlying system using standard NFBC league settings (15-team mixed, 5×5 roto scoring categories). The valuation method is a Z-Score type approach, similar to the FanGraphs’ auction calculator.

Then, the following two variability metrics are computed for each player:

• Inter-Projection Standard Deviation (InterSD) – The standard deviation of the underlying projections surrounding the ATC average auction value. InterSD describes how much the projections disagree about the value of a player. The larger the InterSD, the more projections differ.
• Inter-Projection Skewness (InterSK) – The skewness of the underlying projections surrounding the ATC average auction value. InterSK describes the symmetry of the underlying projections. A positive InterSK means that a player’s mean is being pulled to the upside; the majority of projections are lower than the ATC average. A negative InterSK means that a player’s mean is being pulled to the downside; the majority of projections are higher than the ATC average.

Intra-Projection Volatility

Now that we have looked at the volatility arising from differences between the underlying systems, ATC also has a measure of the volatility that results from a player’s categorical profile.

Using the same standard NFBC league settings (15-team, 5×5), I define the following:

• Intra-Projection Standard Deviation (IntraSD) – The standard deviation of a player’s categorical Z-Scores. IntraSD is a measure of the dimension of a player’s statistical profile. The smaller the IntraSD, the more balanced the individual player’s category contributions are. The larger the IntraSD, the more unbalanced the player’s category contributions are.

Some players such as Xander Bogaerts are projected to contribute across the board in all five rotisserie categories. Such a player will have a low IntraSD (Bogaerts’ is 0.44). Meanwhile, a more unbalanced contributor like Adalberto Mondesi, will have a high IntraSD (3.17). Mondesi’s rotisserie value arises mostly from his stolen bases, while ATC projects Bogaerts to make major contributions in all categories. Mondesi’s IntraSD is quite a bit higher than Bogaerts’.

While inter-projection volatility serves to quantify the parameter risk of projections, the intra-projection volatility deals with the player’s categorical or profile risk. The smaller the categorical risk of the player, the less effect that any single component of the player’s profile will have on his total rotisserie value.

IntraSD is by no means a perfect metric. But when you are your assembling fantasy rosters, the lower the aggregate IntraSD you can assemble, the less prone you are to upsetting your team’s 5×5 category balance. Yes, the overall rotisserie value projection takes precedence, but for two similarly highly priced players, IntraSD is yet another risk factor that you might try to minimize.

Effect on Rotisserie Earnings

Aside from understanding the shape of the projections that feed ATC, let’s take a look at how the new ATC volatility metrics can affect future rotisserie value.

Observe the following two plots:

Each chart above (one for hitters and one for pitchers) plots the final 2019 accumulated player rotisserie earnings against various InterSD ranges. The InterSD figures are derived from the pre-2019 ATC projections, with ranges selected to break the player pool into groups of relatively equal number.

As the effect of projection volatility might also differ depending upon a player’s value, players were classified into four groupings by ATC projected auction values.

Two other technical notes:

• Negative values were capped at -$10 for this exercise. Read here for a further description of the reasoning behind the capping of values.
• Data for this analysis came from the 2019 season. Volatility metrics had not yet been calculated prior to that point, and the inclusion of the 2020 short season would distort results.

For hitters, the trend for rotisserie values leans downward for player groups with an ATC projected value of $10 and up, and also for replacement level players (~$0-2). Particularly, players with extremely high levels of InterSD, tended to earn $5-10+ less than players with the same projected auction values, but with lower InterSDs. Indeed, players with extremely low levels of InterSD earned values a few dollars higher than the average results for similarly valued players.

Hitters in the $2-10 range showed a small upward trending correlation between risk and reward. Remember – risk works both upwards and downwards. Players in the $2-10 range have higher InterSD values because some of the underlying projection systems showed more upside promise for the player. You typically should want a player with a higher InterSD (showing some upside potential) at this price point, since they are low-cost investments.

As an aside, it is a testament to ATC’s success that the projected $2-10 hitters have stable earnings, even at higher levels of projection volatility.

It is harder to discern a clear pattern with pitcher InterSD. This is partly because of the small sample size used in this study. One data point clearly affected the averages for various groups. As a result, until we obtain larger amounts of data – I will say that most of the player graphs are flat, other than in the $8-15 value range. For that cohort, the trend is clearly downward sloping, with the extreme points particularly worth thinking about. At the moment, the takeaway here is to look more intently at pitcher InterSD only for mid-range values.

The next two graphs plot rotisserie earnings against various InterSK ranges. To remind everyone, a positive InterSK means that the ATC average is being pushed up by one or two bullish projections, while a negative InterSK is the reverse – with outlying bearish projections.

For hitters, we clearly see that historical results favor the negatively skewed players. Especially at the high and low endpoints of the graphs (very negative or very positive skew), the results are more dramatic. For an InterSK greater than about +0.6, take heed of the player’s projection downside, and slightly downgrade. Give a boost to a player with an InterSK of less than -1.0.

On the pitching side, the picture is again less clear. For high-end pitchers, we can draw the same conclusions as we did for hitters (negative skews are better), but for players $15 and lower, the curve is essentially flat. InterSK is not as helpful for low-valued pitchers.

As for the Intra-Projection Standard Deviation (IntraSD), for now, we will only analyze batters. Pitchers can at most be four-category contributors, with starters lacking saves and relievers lacking wins (for the most part).

For hitters, the graph shows a negative downward slope for players valued $10 or greater. Multi-category contributors (a low IntraSD) have a lower risk of value attrition. This seems intuitive. According to the trendline, adding half a point of standard deviation lowers the player’s expected production by about $5.

For lower valued hitters ($10 or less), more diversity in categorical makeup is more desirable. Adding half a point of standard deviation adds $5 to the expected value result. In essence, if the investment is low – you want to increase your categorical risk,  at least on an individual player basis.

Conclusion

Ron Shandler started the foray into fantasy baseball risk analysis with his BABS methodology. The next challenge for us will be to pursue a greater understanding of risk and volatility through its quantification. The goal is to define it, make it more measurable, and most importantly – to create an actionable framework.

Inter-Projection volatility is not a be-all-end-all set of risk metrics. It is but a mere first step into this challenging journey of risk quantification in fantasy baseball. As both an actuary and baseball analyst, I look forward to the next steps of this voyage.

There will be more discussion and research on projection volatility in the coming weeks. For now, we would like to hear from you. What early findings have you discovered from ATC? What questions do you have about projection volatility? Let us know in the comments below! Best of luck to you in the 2021 fantasy baseball season.

–

A special thanks to Patrick Davitt of Baseball HQ Radio for reviewing this article in advance, and for providing helpful editing notes.