2023 Projection Systems Comparison – A Game Theory Approach

Introduction

In 2018, I introduced a game theory approach for comparing baseball projection systems. Proudly, the original article was nominated for Baseball Article of the Year by the Fantasy Sports Writers Association (FSWA). Today, I am proud to release the same in-depth research for the sixth consecutive year!

This is not a typical statistical analysis. There won’t be any Chi-squared tests, nor will I calculate Type I or Type II errors. Forget about mean squared errors or any hypothesis testing.

My methodology does not incorporate a statistical model. Instead, it looks to determine the profitability potential of each projection system by simulating what would have happened in a fantasy auction draft. It games the projections.

What do I mean by this?

Let’s think about what happens in a real fantasy [pun intended] baseball draft auction.

Suppose that Jared Cross himself (or anyone exclusively using the Steamer projections) walks into a rotisserie auction prior to the 2023 baseball season. Let’s say that Jared decides to participate in an NFBC auction league. Mr. Cross would take his projections and run a valuation method through them to obtain auction prices. He would generate a list that looked something like this …

Steamer Projected Values: Aaron Judge 41, Ronald Acuna Jr. 40, Kyle Tucker 40, Vladimir Guerrero Jr. 39, Jacob deGrom 39, Fernando Tatis Jr. 35, Juan Soto 34, … , Alex Vesia 1, Ranger Suarez 1, Jared Walsh 1, etc.

In addition to the raw projected values generated by the Steamer system, Jared would then establish a price point that he is willing to pay for each player. There might be a premium that he is willing to pay for the top valued players, and a discount that he expects to save on lower cost players. He may be willing to bid up to $42 on Kyle Tucker (valued at $40), but would only pay $1 for a $4 Manuel Margot.

Players would now fall into two simple categories:

  1. Players that are too expensive to buy. [Auction cost > Price Point]
  2. Players that can be purchased. [Price Point >= Auction Cost]

#1 is easy to deal with – Jared won’t purchase them. If the cost to buy someone during the auction exceeds what he is willing to pay, then he will simply pass on the player and will cease to bid. Typically, some 65-80% of all auctioned players will fall under this category.

For #2 – We can break this down a little further:

  • Players that Jared will purchase [exactly 1 out of 15, or 6.7% of all auctioned players].
  • Players that Jared will not purchase [somewhere between 13-28% of all auctioned players].

Jared cannot draft/buy all players that fall under his acceptable price point. He only has room for 23 players on his roster. Hopefully, and if he drafts successfully, he will end up as close as he can to the 23 players that will earn him the largest profits in the aggregate.

One other key point to note – There will be a number of players in the pool who will have a similar auction price & price point. If Cross did his valuations correctly, he should be indifferent/apathetic to choosing any specific player – all things being equal otherwise.

Measuring Success

Now that we have set the stage for the 2023 NFBC auction, let’s continue with measuring the success of Jared Cross’s Steamer projections.

Let’s say that Derek Carty (or someone exclusively using THE BAT X projections) is one of the other teams competing in this auction It is late in the game and both Derek & Jared need an outfielder.

Let’s look at the $2-4 outfielders who are still available to be purchased in the auction:

Available OFs
Player Steamer THE BAT X Market Value 2023 Earnings Profit
Wil Myers 9 3 4 -4 -8
Lourdes Gurriel 6 8 4 12 8
Joc Pederson 12 9 3 1 -2
Chris Taylor -2 8 2 4 2
Jose Siri 6 3 2 7 5
Enrique Hernandez 0 5 2 -1 -3
Charlie Blackmon 10 5 2 0 -2

For 2023, Lourdes Gurriel and Jose Siri turned out to be the most accretive players among this group of outfielders. A team purchasing either would have amassed a $5+ profit. However, if a team had purchased Wil Myers, they would have realized an $8 loss.

Question: Which fantasy team is in the best position here – Jared’s or Derek’s?

The answer is Derek’s team.

Above are the “BUY” and “PASS” decisions generated by Steamer and THE BAT X for the above low-valued outfielders. For this specific example, we have assumed that a projection system will “purchase” a player if the projected value exceeds the market value.

Below are the associated game theory profitability metrics:

OF Profitability
Steamer THE BAT X
# Players to Buy 5 6
# Profitable Players 2 3
# Unprofitable Players 3 3
Total Gains 13 15
Total Losses -12 -7
Total Profit 1 8
% Profitable 40% 50%
% Unprofitable 60% 50%
Avg Gain Per Profitable Player 6.5 5.0
Avg Loss Per Unprofitable Player -4.0 -2.3
Average Profit Per Player 0.2 1.3

All six of Derek’s potential players are similarly priced. He will end up buying one of six players, likely randomly. If he happens to buy any of Gurriel/Taylor/Siri, he will profit. Should he happen to purchase one of Pederson/Hernandez/Blackmon, he will realize a loss. On average, he stands to make a $1.3 profit.

Jared, on the other hand, would only be inclined to buy one of five players. As three turned out to result in a loss, he has just a 40% chance of ending up profitable with this roster slot. On average, Jared stands to only make a 20-cent profit.

Looking at the low-valued outfielders alone, anyone employing THE BAT X projections instead of Steamer would have had a better chance of picking a profitable player and would have earned a larger expected profit. For this limited example, one would rather have been using Derek’s projections than Jared’s.

Effectiveness over Accuracy

Rather than testing projection accuracy, the game theory approach assesses effectiveness.

Accuracy does not always tell the full story.

To illustrate this, consider the following two hypothetical player projections from two different projection systems:

Player Example
Player A Player B
Razzball Projection $32 $20
ATC Projection $12 $16
Market Price / AAV $2 $18

For Player A, the projection systems are far apart, yet both models price the player far above the market. For Player B, the projection systems are close to one another. Razzball is a few dollars over the market price, and ATC is a few dollars below.

What is important to note is that Razzball will have the propensity to purchase both Players A & B, while ATC will only purchase Player A. What is even more crucial to understand is that the price that Razzball will pay for Player A will be identical to the price that ATC will pay – at $2.

The market price sets the price.

A projection system only decides whether or not to pay the price.

Continuing with this fictitious example, suppose the final accumulated values of Player A and B turned out to be $32 and $2, respectively.

Statistical Test: Absolute Errors
Player A Player B Total
2023 Final Value $32 $2
Razzball Projection $32 $20
ATC Projection $12 $16
Razzball Absolute Error $0 $18 $18
ATC Absolute Error $20 $14 $34

Considering the absolute value of errors (one such statistical test), Razzball was the more accurate projection system.

Razzball was spot on for Player A, whereas ATC was way off in its prediction. As for Player B, both systems were both highly inaccurate. On the whole, Razzball generated a lower aggregate absolute error. In this statistical test of accuracy, Razzball would be declared the winner.

Now, let’s consider the profitability of each projection system from a fantasy perspective. There are three key elements to understand here:

  1. Since Razzball’s projection for both Players A & B was above the market’s cost demands – Razzball would have had the propensity to purchase shares for both Players A & B. In contrast, only ATC’s projection for Player A was above the market’s asking price. ATC would end up with shares of Player A, and decline to Purchase Player B.
  2. No matter what value the projection system computes – the purchase price of a player is always the same – and is set by the market. In a fantasy baseball auction, a player is purchased if you spend $1 more than the next highest bid without regard to projections.
  3. No matter what a player was projected for before the season starts, the profit (or loss) earned for him during the season is the same. It doesn’t matter if your projection system showed a preseason expected profit of $10, $20 or -$15. If you purchase a player on your fantasy team, you will earn the actual realized profit.

Consider these final earned player values for 2023:

Projection System Profitability
Player A Player B Total
2023 Final Value $32 $2 $34
Razzball Spend $2 $18 $20
ATC Spend $2 N/A $2
Razzball Final Player Values $32 $2 $34
ATC Final Player Values $32 N/A $32
Razzball Net Profit $30 ($16) $14
ATC Net Profit $30 N/A $30

Despite Razzball’s spot on prediction of Player A – the net profit accumulated by each projection was the same. ATC’s projection of $12 was above the market, and so it produced a “BUY” signal for player A. Similarly, Razzball also produced a “BUY” signal. It didn’t matter that ATC wasn’t all that accurate on the player evaluation – it only mattered that it was above the market. In the end, the total fantasy profit realized was the same.

However, for Player B – as ATC produced a “PASS” signal, it avoided purchasing a value draining commodity. Razzball succumbed to the landmine, and ceded $16 of value along the way. Despite the overall favorable statistical accuracy of Razzball to ATC, in practice, it produced a net profit of $14 to ATC’s $30.

You were better off having ATC’s projection in-hand, rather than having Razzball’s figures in the above example.

To me, this is truly the heart of it all. This game theory comparison approach provides a method to compare the historical effectiveness of the projection systems.

The Projection Systems

Below are the projection systems that I have analyzed for 2023:

2023 Baseball Projection Systems
Projection System Creator
ATC Ariel Cohen
THE BAT / THE BAT X Derek Carty
Razzball Rudy Gamble
Steamer Jared Cross
ZiPS DC Dan Szymborski

Just as last year, I will be comparing the ATC, THE BAT / THE BAT X, Steamer and the ZiPS DC projection systems, which were all available on FanGraphs. In addition, I am also including Rudy Gamble’s Razzball projections. The two systems that I will no longer include / analyze are Mike Podhorder’s Pod projections and Dan Szymborski’s ZiPS projections.

For the Pod projections – as of last year, Mike no longer produces the set. As for ZiPS – we have previously demonstrated that for fantasy purposes, the ZiPS DC version is superior to the ZiPS projections alone. Unlike ZiPS DC, realistic pro-rated playing time estimates are not fully incorporated into the ZiPS architecture, making them less suitable for this study.

Methodology

The game theory methodology of comparison is identical to last year. Once again, here is the procedure:

1) Start with the raw projections data (AB, H, HR, IP, K, etc.). For this analysis, I have assembled each projection system’s stats as of the day prior to Opening Day 2023.

2) Produce a projected value for each player, by system. For this valuation, I use my own auction calculator, which follows a Z-Score methodology (similar to the FanGraphs auction calculator). So that I can best compare projected values to “market,” I use the NFBC main event settings (15 teams, mixed AL/NL, $260 budget and positions, standard 5×5 scoring). I also assume that players were eligible only at their original 2023 positions + any positions that they were expected to gain in the first 2 weeks of the season.

3) Adjust the projected player values to obtain a Price Point for each player. For this, I have assumed the following:

Projection Price Points
Projected Price Price Point
< $1 Do Not Buy
$1 to $4 $1
$5 to $9 $3 Discount
$10 to $14 $2 Discount
$15 to $19 $1 Discount
$20 to $27 At Cost
$28 to $35 $1 Premium
$36 to $40 $2 Premium
> $40 $3 Premium

For example, if Steamer projects a player for $17 – I assume that the maximum that it would pay for the player is $16. If it projected a player for $42 – I assume that it would pay up to $45. Any player below replacement will not be purchased in this exercise.

This attempts to simulate what happens in real life fantasy draft/auctions. Managers typically purchase players at the top for a premium. In the mid-rounds, players are purchased roughly at their projected cost. Towards the endgame, players are only purchased for discounts.

4) Obtain an Auction Price. I use an average auction value (AAV) for each player. For this, I am using actual NFBC AAVs for auctions in the month of March.

5) Compute the rotisserie player values for this season. This will represent what a player was worth in 2023. It is computed using the same methodology as above in #2.

Note that for all of the above, I have let the Z-Score method determine the inherent Hitter/Pitcher split of the total auction dollars. This will differ from the NFBC AAVs, which is typically pitcher heavy (and was about 63/37 this past year).

6) Players were then “purchased” for a system if their Price Point was higher than the player’s AAV.

Terminology – I identify a player as “purchased” as long as they appear to be a bargain for the given system.

I then tracked the number of players purchased who were profitable, the number of players purchased who were unprofitable, and their respective gains and losses.

2023 Results

First, let’s look at the number of players that each system would “buy.” To get a sense of where the projection systems purchase their players – displayed are the number of players that would be bought by each system, for the top N cumulative players, ranked by AAV.

Let’s start as always by looking at players in the top 50. When I started the game theory projections comparisons back in 2018, we typically witnessed approximately 3 to 6 players purchased by each projection system. However, in the past three years, something has changed. Projections have been finding more potential bargains up top. Since 2021, we have seen systems regularly purchase anywhere from 5-15 players in the top few rounds. ATC, which is typically on the low side of things, showcased a 20% purchase rate this year.

Last year, I had previously thought that this sudden rise could be attributed to the market’s post-COVID uncertainty. Sure, that certainly was a part of it, but I believe that the baseball drafting environment was the larger factor.

For the past few seasons, scarce category statistics such as saves and steals had exhibited larger than normal market premiums. Bargain hitters tended to be the ones lacking a speed element (examples: Kyle Schwarber, Matt Olson, etc.). Closers took on a larger portion of the pitching spend – with market premiums sometimes larger than $7 over projected values. More money at the top flowed into the scarce roto categories, which opened up more bargains according to projections.

I had remarked last year that this phenomenon would not likely continue. Yes, the 2023 figures say otherwise, but from early drafting results – 2024 appears to finally be a reversal of the trend. Saves and steals are still a premium early on in drafts – but nowhere near the levels of the past few seasons.

As far as the entire player pool (top 450 depth) is concerned, the total frequency of purchases by each system has not differed greatly from year to year. Projection systems tend to mimic their own historical levels, and hover at about a 30% purchase rate. Drafters who rely heavily on projections have the knowledge that they operate in a reliable manner.

This year, it was THE BAT and THE BAT X which had the most purchases up and down the player pool. This is typical for Derek Carty’s systems. As usual, it was ATC that purchased the fewest players once again.

Onto profitability …

In the below:

GREEN colored figures represent the more successful projection results. RED colored figures represent less successful results. The “All players” column displays the figure for purchasing every player. “All Players” essentially represents the market.

First, what needs to be pointed out is that frequency of success or “hit rates” are extremely important. The magnitude of success (or failure) may change wildly for projections year over year – but the hit rates tend to be fairly stable. It is always more important to have more favorable outcomes in the more controllable [less volatile] aspect of a model.

On top of that, remember that the magnitude of success (or failure) of a player is the same regardless of the underlying projection. If two projections give a specific player a “BUY” signal – both projections realize the same profit for the player. If two projections give a specific player a “PASS” signal – both projections also realize the same profit for the player, which is $0. The idea is always to have more correct “BUY” signals, and more correct “PASS” signals.

For 2023 – out of the most expensive 50 players of the auction, a record 13 turned a profit (26%). This result comes after last year’s record low figure of 6 (12%). Typically, we see 16-20% of the top players turn a profit. Profitable players inside the top 50 this season included Ronald Acuna, Matt Olson, Freddie Freeman, Mookie Betts, Gerrit Cole and Francisco Lindor.

The market was fantastic at identifying top talent this season; it was the best we have ever seen it.

ATC and Steamer identified three out of the thirteen profitable players. THE BAT X and ZiPS DC identified two, and Razzball / THE BAT identified just one.

In terms of success rates, ATC had the highest percentage of their “purchased” players returning a profit within the top 50 – an excellent rate of 30%. ATC was the only system to beat the market’s 26% hit rate. ZiPS DC was the next best at an excellent 25% clip.

In the 50-100 range, Razzball and Steamer took the prize, each with a 27% success rate. For the 1-100 cumulative range, Steamer (as seen above), was the most successful overall.

Lower down in the 250-450 range, it was ATC and Razzball which were the dominant two systems as seen by the slew of green on the above two charts. ZiPS DC brought up the rear.

For the entire player pool, Razzball finished with the highest hit rate of all projection systems, edging out ATC by a few parts of one percent.

Steamer was the projection system with the lowest overall hit rate for the player pool – largely deficient after pick 250. Steamer exhibited very different success rates between the top and bottom of the curve.

As for an overall winner for 2023 hit rates, ATC appears to be the lead projection system, performing well in almost all player depths. I would split the runner-up trophy and hand it to Steamer for their success at the top, and to Razzball for their success at the bottom.

Now for unprofitable players …

Now let’s look at players purchased for a loss. We expect that most of the top players to be unprofitable. The rate table shown is the compliment of the profitable one above (all percentages will sum to 100%). The added information here is the quantity of failures.

ATC purchased the fewest number of failures in 2023 overall with only 76 busts. Razzball, as we have seen above – had the lowest bust rate. Razzball was especially good up top at avoiding value drains, which is the most crucial portion of the draft for failures.

ZiPS DC did a fairly good job at avoiding disaster this past season. Within the top 150 players, it only triggered 27 landmines, fewest of all systems. THE BAT and THE BAT X, however, performed poorly in this category for 2023.

Now onto the magnitude of player acquisition …

For the magnitude of gains, it greatly helps to first see how profitability has changed over time. Below is how the “All Players” column has looked over the past six seasons. [For this particular chart I have replaced the “All Players” terminology with “ADP.”]

It does not take a statistician to see that 2020 was a sheer aberration. The COVID shortened year should be fully ignored. The short season resulted in an abnormally high gain rate per hit. Since the ending player values were more widely distributed in a smaller sample size – the highs were higher, and the lows were lower. The above chart tackles the highs alone.

Other than the highest player depths, the gains per profitable player have stabilized over the past few seasons. Do note that there is a downwards sloping trend across all years for all players – a sign that the market is gradually getting “smarter.” Pricing has been approaching the future realized values more closely, i.e., it has been harder for the market to find immense gains.

At the very top (50) – last year was an outsized result for the market. The average $10.0 gain per profitable player achieved a historic level. This year, the 50-100 range had an all-time low of $5.1 average gain per player. For the moment, let’s attribute these two anomalies to sheer randomness arising from a small sample size.

Now onto the 2023 gains …

Clearly, in terms of pure dollars – THE BAT X ended up seizing more gains than any other system this year. On a per player basis, THE BAT X also held up nicely as evidenced by the sea of green above.

Let’s drill down a bit more into where the gains came from. Below in an incremental view of gains per player by ADP range:

THE BAT X was excellent in the top 50 players, as well as after pick 200. Derek Carty’s other system, THE BAT, had a near $20 gain per player at the top – however, that corresponds to just ONE player (Matt Olson).

As far as pockets of success for gains, Razzball is an interesting system to look at. It outperformed the other projections in a few select ranges – most notably in the 250-300 range. That ~$17 of profit per player within the range is fantastic. However, the 300+ range for Razzball was underwhelming, dropping them towards the bottom for the overall player pool.

ATC had a poor showing this year in the category – with few outstanding pockets of players. The only range that ATC was especially good at in 2023 was the very bottom – where it had a $9+ gain per player, for its 13 identified bargains.

As an aside, in deciding whether you should use projections or follow the market – do take the time to identify how the projection systems compare to the “All Players” results. In 2023, both prior to the 50th player and after pick #200 – no matter which projection you observe, every system had exhibited larger gains than the market. The same was true for last year. However, in the middle of the draft, the “All Players” perspective does outperform certain projections at times.

The way that I interpret this result is to trust projections at the top and to seek out some diamonds in the rough straight from projections in the endgame. It is wise to spend your research time for fantasy drafts on players in the middle. Overall, you are still far better off using projections rather than trusting the drafting market blindly.

Onto losses …

For the unprofitable players, as always, an important adjustment has been made to the figures. All 2023 final values have been capped at -$5. That is, we will not let a player’s obtained value in 2023 fall below the threshold of -$5. A player who was injured all season, or who was clearly droppable, should not be penalized with an overly negative final valuation, which would skew results. I have previously written about the concept of capped values more in depth here.

Using visual inspection, ATC is the clear winner for being best at avoiding large unprofitable players. ATC purchased the least amount of dollars of loss and was outstanding on a per player purchased basis as well. ZiPS DC was excellent on a pure dollars basis, while THE BAT and THE BAT X excelled on a per-player basis. Steamer and Razzball finished towards the bottom in 2023.

To note, ATC has consistently been at the top, or a close 2nd in this category each and every year. One of the key strengths of projections aggregation is the reduction of parameter risk, which has an enormous ability to limit losses. Avoiding pitfalls and value traps in an underrated component of fantasy baseball success.

Onto total profitability …

Now comes the part where we put it all together, as we look at the total profitability by projection system. All of the dollars gained are added up, and all of the dollars lost are subtracted out. It is the total summary of system profitability. These will be the most important charts of this analysis.

But before we look at the individual projections, let’s first check in once again to see how profitability on the whole has fared over the past few seasons.

Once again, we should completely ignore the 2020 short season. As we mentioned above, the overall level of profitability of 2020 is not comparable to other seasons. This is entirely a function of sample size. A top-X drafted player is far less likely to end the season as a top-X player in a third of a season.

The market had its best year for profitability yet. In every single player depth above, using ADP/AAV alone in 2023 had ceded the least amount of loss to date.

It is important to note that the market profitability figures above have nothing to do with projections directly. The market is now simply doing a better job of calculating future costs.

For comparison, below is what the average projection has done over the past few seasons:

While market profitability is independent of projections, the opposite is not true. The profitability of projections in this study is directly related to the market, as it uses the actual AAVs in conjunction with their calculated strike prices. For projections to gain profitability using this methodology, the market must miscalculate future player value. That is to say, if the market has gotten better over time, we should see a deterioration in projection profitability within this exercise.

In 2022, projections were decidedly stronger than the market. For 2023, you were still far better off using projections, but the gap has dramatically closed. I am curious to see where 2024 will end up.

Now let’s dive into 2023’s profitability by projection system:

What immediately jumps out to me is that no projection system in 2023 turned a profit on the full player pool. Typically, we get one or two projections that turn a profit for the year – usually in the 10 cent to a dollar range. Again, as we just witnessed, the market was more efficient than it had ever been in 2023.

Next, it is clear that ATC had the best overall results in terms of profitability this past season. The mitigating of losses at the very top of the player pool carried them throughout.

Before crowning ATC the victory of profitability, let’s dive more into the specific player ranges by looking at the incremental chart:

ATC’s performance at the top was outstanding this year. It is interesting to see that some systems beat the “All Players” profitability, while others fared worse. Last year, every system performed better than the market at the very top.

For all other player depths, ATC was about mid-pack, but still always outperformed the market. That and a strong showing in the top 50 gave it the best overall profitability of 2023.

For the other systems, success wavered all throughout the player pool. THE BAT was excellent from 50 to 100, but was rather poor comparatively from 250-300. Steamer was worse than the market on top, but in the 250-300 range it had a number of tremendous finds … leading to an average $9.4 profit! The consistency award goes to ATC for being above average almost everywhere.

All in all, ATC finished with the highest overall figures and the most consistency – making it the winner for profitability in 2023. I would give THE BAT X the nod for 2nd place, with a number of outstanding pockets of profitability, and the second best overall figures.

2023 Winner

Overall, I would give the nod to ATC this season as the most effective projection set of 2023.

ATC was the strongest profit generator, and was very consistent across all player depths. ATC also demonstrated superior hit rates at almost every single point on the curve. It seemed to provide users with the best chances at profitability, plus the smallest realized losses across the board.

Multi-Year Results

Now let’s take a look at the multi-year profitability results.

Please note that a few of these projections were not analyzed for all of the years of this study. For these projections, the averages are compiled from only of a limited number of seasons.

THE BAT X and ZiPS DC were new additions back in 2021, so the averages only contain three years worth of data. Razzball’s history is from 2019-2023 (5 years). All other projections have a full six years of data points.

Here is the cumulative chart by player depth:

Here is the incremental chart by player depth:

Over the past six seasons, we can visually inspect that ATC and THE BAT X have offered the most projected profit on a per player purchased basis. The two systems look fairly similar up and down the curve on a cumulative basis, with ATC having the longer historical track record, and the most year-to-year consistency.

ATC has outperformed all other systems cumulatively at every depth. ATC has had the slight edge over THE BAT X in the top 50 range, as well as post 300 – and is above average for the remainder. Most notably, ATC is the only projection system to turn a profit on average for the entire player pool over the six year analysis period.

THE BAT has also performed well to date, and in some middle ADP ranges was in fact the best projection!

Razzball has been mediocre for the top of the player curve but has found numerous gems over the years in the low-cost players. It has been one of the most profitable systems projections after player 250. ZiPS DC on the other hand has been average at the top, but has performed poorly for lower valued players.

Steamer has been more volatile than some of the other projections. In some years it has been one of the best systems, and in some it has been towards the back half. Globally, its cumulative curve is fairly close to that of Razzball’s.

Finally, note that each projection system (at almost every level) has beaten the “All Players” perspective over time. Again – using projection systems is clearly the way to draft, as opposed to simply going by ADP/AAV alone.

Assorted Notes & Method Limitations

A few assorted notes, method limitations and other findings – in no particular order:

  • ZiPS DC did not always project saves. In previous years where saves were absent, I simply substituted in the Steamer saves.
  • As mentioned in the past, playing time estimates are vitally important to a projection system, and are directly factored into this method. Systems which have poor playing time figures, but good rate stats per playing time are penalized in this analysis; it is the raw counting stats which are used to evaluate player values.
  • THE BAT and THE BAT X made a modification last season to incorporate ATC playing time. Previously, they had been using the FanGraphs Depth Charts figures. ATC now differs from Derek Carty’s projections by rates alone.
  • The dollar values in this study were generated by a Z-Score methodology. Users of the Standings Gain Points (SGP) might see slightly different results.
  • This methodology only looks at the final player values vs. the projected player values. If a system would have instead projected Matt Olson for ~60 stolen bases as opposed to ~40 homeruns with the same initial projected dollar value … this analysis would still appear to enumerate the very same results. Olson would have been accretive to fantasy baseball teams but for very different reasons, which is not counted in this methodology. I may not dub this as a bug of the methodology (a valuable player is a valuable player), but it certainly isn’t a feature of it either.
  • In this analysis, the categorization of a “Profitable Player” vs. an “Unprofitable Player” is defined by whether the final accumulated roto value exceeded the initial draft price. However, if a player purchased for $35 returned $34 of value, I would hardly call that a failure. Not only should there be a market pricing curve (as I already have), but there should also be a success curve. A $5 loss on a $40 player should be categorized as a win, whereas a $1 gain on a $2 player could be construed as a loss. This distinction only applies to categorizing players as “gain” and “losses.” The total profit though, remains unchanged.
  • Suppose that you follow a certain projection system. For Player A – it (its strike price) exceeds the market value by a wide margin, whereas for Player B it exceeds the market value by a small margin. If you draft many fantasy leagues in real life, you are more likely to have Player A on your roster than Player B. However, this analysis considers the likelihood of having either player to be identical. This is a limitation to this method. In a further study, I might experiment with additionally assigning a weight/factor to each player’s purchase ability by size of value to market spread.

For a further explanations of the game theory methodology (and for comparison), please refer to the past articles: 2022 2021 2020 2019 2018

Once again, I still hope that you find this game theory method of evaluating projection systems to be different, yet insightful. No one method of projection comparison is perfect and without limitations. There are other more statistically based methods that are certainly more than valid. In choosing the projection system(s) to incorporate into your fantasy preparation, this article should be one additional point of reference.

I will reiterate once again that the test of effectiveness is a better measure of success than accuracy. Whether you thought that Google stock was worth $50 or $100 at a point in time, had you purchased Google shares for $10 – you would have indeed made lots and lots of money.

Wishing everyone a fantastic drafting season!

Please comment below if you have any thoughts on either my method or to the conclusions drawn. Perhaps you have an interesting variant of the effectiveness testing, or spot an interesting shortcoming. Let me know!





Ariel is the 2019 FSWA Baseball Writer of the Year. Ariel is also the winner of the 2020 FSWA Baseball Article of the Year award. He is the creator of the ATC (Average Total Cost) Projection System. Ariel was ranked by FantasyPros as the #1 fantasy baseball expert in 2019. His ATC Projections were ranked as the #1 most accurate projection system over the past three years (2019-2021). Ariel also writes for CBS Sports, SportsLine, RotoBaller, and is the host of the Beat the Shift Podcast (@Beat_Shift_Pod). Ariel is a member of the inaugural Tout Wars Draft & Hold league, a member of the inaugural Mixed LABR Auction league and plays high stakes contests in the NFBC. Ariel is the 2020 Tout Wars Head to Head League Champion. Ariel Cohen is a fellow of the Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA). He is a Vice President of Risk Management for a large international insurance and reinsurance company. Follow Ariel on Twitter at @ATCNY.

14 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
rlothmanmember
3 months ago

I was just looking back at your prior iteration of this analysis as I start on draft prep for this year, and it feels like Christmas morning to have your updated analysis drop today! This is an incredibly rich and insightful comparison, and you clearly articulate the accuracy vs. effectiveness distinction and illustrate your approach in a way that is easily understandable.

I think many baseball fans who look at projections primarily for evaluating player performance in real life (i.e., across all facets of the game as it is actually played) fail to distinguish between IRL accuracy and fantasy effectiveness in translating raw projections to successful price points to inform draft strategy. Your work has helped me to take a much more nuanced approach to this translation. Thanks for all you do!

I’m not sure how valuable it would be, but I would be curious how a similar comparison would play out when isolating particular statistical categories. Could this approach reveal areas in which particular systems systematically over- or under-project certain types of value? Are there discernible themes as to which types of players different systems tend to hit on or miss on in different bands?