Projection Accuracy: Early March Plate Appearances

I’ve always wanted to do a projection analysis, especially at different time points. I had it started in 2020 and everything fell apart with the shortened season. I’m starting off simply today by looking at at-bat projections from March 1st with the Wisdom of the Crowds prevailing.

The reason I chose March 1st was that The Great Fantasy Baseball Invitational (TGFBI) started drafted that day. I pulled all 14 projections that morning. I contacted the paid providers and all but one agreed to have their name associated with the results. They are:

Projections

  • Steamer (FanGraphs)
  • ZIPS
  • DepthCharts (FanGraphs)
  • The Bat
  • The Bat X
  • Davenport
  • ATC (FanGraphs)
  • Pod (Mike Podhorzer)
  • Masterball (Todd Zola)
  • PECOTA (Baseball Prospectus)
  • RotoWire
  • CBS
  • Razzball (Steamer)
  • Paywall #1

To create a list of players to compare for accuracy, I took the TGFBI ADP (players in demand at that time), selected the top 450 players (30-man roster, 15 teams in TGFBI), and used the hitters. Generally, all the players were projected with the following exceptions. Rotowire and Pods didn’t have a Josh Rojas projection while Pods also didn’t have projections for Jazz Chisholm Jr. and Mike Brosseau. Additionally, CBS was missing several projections. I am blamed for part of it because I forgot to pull designated hitters and they didn’t project as many outfielders. Finally, I just removed the projection for Yasiel Puig.

To determine accuracy, I calculated the Root Mean Square Error (RMSE) for four different sets of values. RMSE is “measure of how far from the regression line data points are” and the smaller a value the better.

  • All hitters in the subset (no CBS).
  • All hitters with the three Pod and Rotowire missed.
  • Removed the three along with several who missed quite a bit of time (Lewis, Mondesi, Trout, Ozuna, Hicks, Rendon).
  • Every hitter that I pulled from CBS.

Finally, I created two additional projections. One is the average of the above projections. For the second one, I averaged the results from the best two non-aggerate projections, a found its rank (not included in the one with just the three removed). With all the theatrics out of the way, here are the results.

All Hitters (no CBS)
Projection RMSE
ATC 150.3
Razzball 151.5
Top-2 151.9
Average 152.5
Davenport 153.9
Steamer 155.3
Paywall #1 155.3
ZiPS 155.7
Mastersball 157.0
PECOTA 158.5
Rotowire 158.8
DepthCharts 161.0
Pods 161.2
Bat 169.1
BatX 169.1

 

Hitters with Three Zeros Removed (no CBS)
Projection RMSE
ATC 149.0
Razzball 150.1
Average 151.2
Davenport 152.2
Paywall #1 153.5
Steamer 153.7
Mastersball 155.7
ZiPS 155.9
Pods 156.3
Rotowire 156.4
PECOTA 157.4
DepthCharts 159.8
BatX 168.1
Bat 168.1

 

Hitters with Major Injuries Removed (no CBS)
Projection RMSE
Top-2 138.8
ATC 138.9
Razzball 140.7
Average 141.0
Davenport 142.1
Steamer 142.3
Paywall #1 144.7
Pods 146.2
Mastersball 146.5
Rotowire 147.2
PECOTA 147.7
DepthCharts 148.8
ZiPS 148.9
Bat 157.5
BatX 157.5

 

Hitters with Available CBS
Projection RMSE
ATC 141.0
Top -2 142.7
Razzball 143.3
Average 143.4
Steamer 145.3
Davenport 145.6
ZiPS 146.4
Paywall #1 146.6
PECOTA 147.8
Rotowire 147.8
Pods 148.1
Mastersball 148.5
DepthCharts 151.3
CBS` 160.0
Bat 160.2
BatX 160.2

 

There is a lot to digest here.

  • ATC projections come out on top but that should not be a surprise since they are an optimized combination of several projections.
  • The boys over at Razzball performed the best of the independents with Davenport projections holding their own.
  • The Wisdom of the Crowds performed as I expected by crushing all but a few projections.
  • For the rest, I am backing off from making too many judgments. It is just at-bats with many more projections to compare. The BAT and CBS obviously have some work to catch up, but the others are all just incrementally worse than the one listed before it.

And that is it for today. I know it’s just a small taste and sorry.

What I really like is some feedback on the process before I move any further along. If anyone thinks some hitters should be added or removed, here are the hitters I analyzed. Thanks for any suggestions.





Jeff, one of the authors of the fantasy baseball guide,The Process, writes for RotoGraphs, The Hardball Times, Rotowire, Baseball America, and BaseballHQ. He has been nominated for two SABR Analytics Research Award for Contemporary Analysis and won it in 2013 in tandem with Bill Petti. He has won four FSWA Awards including on for his Mining the News series. He's won Tout Wars three times, LABR twice, and got his first NFBC Main Event win in 2021. Follow him on Twitter @jeffwzimmerman.

19 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
themchughmember
2 years ago

I think Carty has said that the Bat and Batx don’t project playing time, they just use Depth Charts/Steamer from Fangraphs.

rustydudemember
2 years ago
Reply to  themchugh

Considering Depth Charts’ lack of success at projecting playing time (see comment below), it makes sense that Bat/Batx ended up at the bottom of the list.