Projection Accuracy: Late March Pitcher Rate Stats

by
November 17, 2021

I’m finally done crunching numbers (part 1, 2, 3, 4, 5, 6, 7, and 8) with late-season pitcher rate stats being the last to go. It’s so monotonous and painstaking, but I’ve learned a few things going through it all.  The Average rises to the top … again with several other projections popping to the top.

First, here are the projections analyzed.

  • Steamer (FanGraphs)
  • ZIPS
  • DepthCharts (FanGraphs)
  • THE BAT
  • Davenport
  • ATC (FanGraphs)
  • ZIELE (Fantasy Pros)*
  • Pod (Mike Podhorzer)
  • Masterball (Todd Zola)
  • PECOTA (Baseball Prospectus)
  • RotoWire
  • Razzball (Steamer)
  • Paywall #1
  • Average of the above projections

To create a list of players to compare for accuracy, I took the NFBC ADP (players in demand at that time) and selected all the pitchers in the top-450 drafted players (30-man roster, 15 teams in the Main Event) in ten or more leagues. Then I removed all the pitchers who never threw an inning.

To determine accuracy, I calculated the Root Mean Square Error (RMSE) for two different sets of values. RMSE is a “measure of how far from the regression line data points are” and the smaller a value the better. Additionally, I included the actual and league average rates for reference.

As with the previous pitcher dataset, some pitchers didn’t have a projection. In all, eight pitchers were missing from one or more of three different projections.

Projection: Players missing
Pods: 6
THE BAT: 2
Razzball: 1

For this reason, I created two comparisons. One with all the projection systems but fewer pitchers and one with all the pitchers but fewer projection systems. Here are the results for the rate stats.

 

Late March ERA Projection Accuracy: All Projections
Projection RMSE Projection Avg Value
Steamer 1.320 RotoWire 3.69
DepthCharts 1.321 PECOTA 3.78
ZIELE 1.324 Paywall #1 3.82
ATC 1.330 Pods 3.82
THE BAT 1.343 Mastersball 3.97
ZiPS 1.351 Average 3.98
Razzball 1.357 THE BAT 3.98
Davenport 1.357 ATC 3.99
Average 1.378 ZIELE 4.03
Paywall #1 1.379 ZiPS 4.07
PECOTA 1.417 Actual 4.07
Mastersball 1.428 DepthCharts 4.10
RotoWire 1.529 Razzball 4.13
Pods 1.538 Steamer 4.13
Davenport 4.17

 

Late March ERA Projection Accuracy: All Pitchers
Projection RMSE Projection Avg Value
THE BAT 1.285 PECOTA 3.77
ZIELE 1.305 Paywall #1 3.82
DepthCharts 1.308 Mastersball 3.96
Steamer 1.311 Average 3.97
ATC 1.313 ATC 3.99
ZiPS 1.333 THE BAT 3.99
Davenport 1.340 ZIELE 4.02
Paywall #1 1.358 Actual 4.06
Average 1.364 ZiPS 4.06
PECOTA 1.398 DepthCharts 4.10
Mastersball 1.406 Steamer 4.13
Davenport 4.16

 

Late March ERA Projection Accuracy: Adjusted to Actual League Value
All Projections Adj RMSE All Pitchers Adj RMSE
Average 1.213 Average 1.229
THE BAT 1.286 DepthCharts 1.319
ZIELE 1.308 ZIELE 1.319
Pods 1.309 Steamer 1.320
DepthCharts 1.312 ATC 1.325
Steamer 1.313 THE BAT 1.341
Razzball 1.313 ZiPS 1.343
ATC 1.314 Davenport 1.346
ZiPS 1.337 Paywall #1 1.355
Davenport 1.337 PECOTA 1.387
Paywall #1 1.344 Mastersball 1.416
PECOTA 1.379
Mastersball 1.410
RotoWire 1.447

 

Late March WHIP Projection Accuracy: All Projections
Projection RMSE Projection Avg Value
THE BAT 0.206 RotoWire 1.199
Pods 0.212 Paywall #1 1.235
ATC 0.212 Actual 1.235
ZIELE 0.215 Pods 1.250
DepthCharts 0.216 ATC 1.252
ZiPS 0.218 Davenport 1.256
Davenport 0.220 THE BAT 1.257
Steamer 0.221 Mastersball 1.258
PECOTA 0.221 Average 1.261
Razzball 0.223 ZIELE 1.269
Paywall #1 0.223 ZiPS 1.270
Mastersball 0.230 PECOTA 1.272
Average 0.248 DepthCharts 1.283
RotoWire 0.259 Steamer 1.295
Razzball 1.298

 

Late March WHIP Projection Accuracy: All Pitchers
Projection RMSE Projection Avg Value
ATC 0.216 Paywall #1 1.236
ZIELE 0.219 Actual 1.240
DepthCharts 0.219 ATC 1.254
ZiPS 0.222 THE BAT 1.258
Davenport 0.222 Davenport 1.261
Steamer 0.223 Mastersball 1.261
PECOTA 0.225 Average 1.263
THE BAT 0.226 ZIELE 1.271
Paywall #1 0.229 ZiPS 1.273
Mastersball 0.235 PECOTA 1.273
Average 0.250 DepthCharts 1.285
Steamer 1.296

 

Late March WHIP Projection Accuracy: Adjusted to Actual League Value
All Projections Adj RMSE All Pitchers Adj RMSE
Average 0.196 Average 0.202
THE BAT 0.205 DepthCharts 0.215
DepthCharts 0.211 ATC 0.216
ATC 0.212 ZIELE 0.217
Pods 0.212 Steamer 0.218
Steamer 0.213 ZiPS 0.219
ZIELE 0.213 Davenport 0.220
Razzball 0.215 PECOTA 0.222
ZiPS 0.215 THE BAT 0.227
PECOTA 0.218 Paywall #1 0.228
Davenport 0.219 Mastersball 0.232
Paywall #1 0.224
Mastersball 0.229
RotoWire 0.258

 

Late March K/IP Projection Accuracy: All Projections
Projection RMSE Projection Avg Value
ATC 0.173 THE BAT 1.039
Razzball 0.173 Davenport 1.041
ZIELE 0.173 Steamer 1.052
Steamer 0.174 Razzball 1.054
PECOTA 0.175 ZIELE 1.061
DepthCharts 0.176 Pods 1.063
Pods 0.177 Average 1.064
THE BAT 0.181 ATC 1.064
Davenport 0.185 DepthCharts 1.065
Mastersball 0.187 Paywall #1 1.068
ZiPS 0.189 Mastersball 1.075
Paywall #1 0.189 ZiPS 1.080
Average 0.196 Actual 1.080
RotoWire 0.196 PECOTA 1.083
RotoWire 1.087

 

Late March K/IP Projection Accuracy: All Pitchers
Projection RMSE Projection Avg Value
Steamer 0.191 THE BAT 1.039
ATC 0.194 Davenport 1.041
DepthCharts 0.196 Steamer 1.051
PECOTA 0.197 ZIELE 1.061
ZIELE 0.197 Average 1.063
Mastersball 0.208 ATC 1.064
Davenport 0.209 DepthCharts 1.064
ZiPS 0.211 Paywall #1 1.067
Paywall #1 0.211 Actual 1.072
Average 0.215 Mastersball 1.074
THE BAT 0.218 ZiPS 1.079
PECOTA 1.082

 

Late March K/IP Projection Accuracy: Adjusted to Actual League Value
All Projections Adj RMSE All Pitchers Adj RMSE
Average 0.161 Average 0.178
Razzball 0.171 Steamer 0.191
Steamer 0.172 ATC 0.194
ATC 0.173 PECOTA 0.196
ZIELE 0.173 DepthCharts 0.197
PECOTA 0.176 ZIELE 0.197
DepthCharts 0.176 Davenport 0.207
Pods 0.177 Mastersball 0.208
THE BAT 0.177 ZiPS 0.210
Davenport 0.182 Paywall #1 0.210
Mastersball 0.188 THE BAT 0.216
ZiPS 0.189
Paywall #1 0.189
RotoWire 0.196

 

Late March W/IP Projection Accuracy: All Projections
Projection RMSE Projection Avg Value
DepthCharts 0.0287 Mastersball 0.0571
Pods 0.0288 THE BAT 0.0582
Davenport 0.0291 Steamer 0.0586
Average 0.0297 ATC 0.0603
ZiPS 0.0300 Davenport 0.0608
ZIELE 0.0301 Paywall #1 0.0610
ATC 0.0304 PECOTA 0.0612
Razzball 0.0305 ZIELE 0.0613
Paywall #1 0.0305 Average 0.0617
Steamer 0.0307 Actual 0.0621
THE BAT 0.0308 RotoWire 0.0625
PECOTA 0.0309 Razzball 0.0632
RotoWire 0.0326 DepthCharts 0.0641
Mastersball 0.0342 Pods 0.0643
ZiPS 0.0687

 

Late March W/IP Projection Accuracy: All Pitchers
Projection RMSE Projection Avg Value
DepthCharts 0.0295 Mastersball 0.0563
Davenport 0.0302 THE BAT 0.0578
Average 0.0306 Steamer 0.0581
ZiPS 0.0308 ATC 0.0600
ZIELE 0.0309 Davenport 0.0606
Steamer 0.0311 Paywall #1 0.0609
THE BAT 0.0317 ZIELE 0.0610
ATC 0.0317 PECOTA 0.0612
PECOTA 0.0317 Average 0.0613
Paywall #1 0.0326 Actual 0.0618
Mastersball 0.0351 DepthCharts 0.0639
ZiPS 0.0688

 

Late March W/IP Projection Accuracy: Adjusted to Actual League Value
All Projections Adj RMSE All Pitchers Adj RMSE
Average 0.0274 Average 0.0283
DepthCharts 0.0287 DepthCharts 0.0295
Pods 0.0288 ZiPS 0.0298
ZiPS 0.0290 Davenport 0.0303
Davenport 0.0292 ZIELE 0.0311
ZIELE 0.0303 Steamer 0.0312
ATC 0.0305 PECOTA 0.0317
Razzball 0.0305 ATC 0.0318
Steamer 0.0307 THE BAT 0.0319
Paywall #1 0.0307 Paywall #1 0.0329
THE BAT 0.0309 Mastersball 0.0355
PECOTA 0.0310
RotoWire 0.0326
Mastersball 0.0348

Notes

  • Once adjusting to the actual league value, the average projection handily wins. It has been this case with all the analyses.
  • THE BAT performed decently for WHIP and ERA.
  • I’m amazed at the variation of ERA (range of 0.4) and WHIP (range of 0.10) between the projection. Again, it is a stat I want to track as the season gets closer.

I am going to piece together one more article on how I would recommend approaching projection accuracy going into the 2021 season. I’ll likely finish it after Thanksgiving.  As for now, it’s back to Mining the News and trying to get The Process out.

 

* ATC, Clay Davenport, Depth Charts, Harper Wallbanger, Luke Gloeckner, Mike Podhorzer, CBS Sports, ESPN, RotoChamp, Razzball, numberFire, Steamer, THE BAT, and ZiPS.





 

Jeff, one of the authors of the fantasy baseball guide,The Process, writes for RotoGraphs, The Hardball Times, Rotowire, Baseball America, and BaseballHQ. He has been nominated for two SABR Analytics Research Award for Contemporary Analysis and won it in 2013 in tandem with Bill Petti. He has won four FSWA Awards including on for his Mining the News series. He's won Tout Wars three times, LABR twice, and got his first NFBC Main Event win in 2021. Follow him on Twitter @jeffwzimmerman.

1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
chc77milb
2 years ago

fangraphs steamer is inaccurate.. the formula is messed up and not correct.. there is a major problem with how it calculates projection.. this formula is inaccurate and has major flaw..
This is the major flaw.. fangraphs steamer calculates every level a prospect performs.. if the pitcher prospect is at CPX then High A then AA.. and pitches 3 inning in CPX has 0 era and 7k with 70.0% k%.. pitches 37.1 inning Low A has 3.62 era and 42k with27.5% K%.. pitches 35 inning with 5.66 era and 40k and 26.3% K%.. Fangraphs steamer calculates the 3 levels equally which strews the Projection incorrectly because the CPX level was very high numbers but was only 3 innings.. but those 3 inning’s big stat numbers cause the fangraphs steamer projection to be much higher than it should because the formula in the fangraphs steamer is wrong causing incorrect projections

-2