Projection Accuracy: Late March Pitcher Rate Stats
I’m finally done crunching numbers (part 1, 2, 3, 4, 5, 6, 7, and 8) with late-season pitcher rate stats being the last to go. It’s so monotonous and painstaking, but I’ve learned a few things going through it all. The Average rises to the top … again with several other projections popping to the top.
First, here are the projections analyzed.
- Steamer (FanGraphs)
- ZIPS
- DepthCharts (FanGraphs)
- THE BAT
- Davenport
- ATC (FanGraphs)
- ZIELE (Fantasy Pros)*
- Pod (Mike Podhorzer)
- Masterball (Todd Zola)
- PECOTA (Baseball Prospectus)
- RotoWire
- Razzball (Steamer)
- Paywall #1
- Average of the above projections
To create a list of players to compare for accuracy, I took the NFBC ADP (players in demand at that time) and selected all the pitchers in the top-450 drafted players (30-man roster, 15 teams in the Main Event) in ten or more leagues. Then I removed all the pitchers who never threw an inning.
To determine accuracy, I calculated the Root Mean Square Error (RMSE) for two different sets of values. RMSE is a “measure of how far from the regression line data points are” and the smaller a value the better. Additionally, I included the actual and league average rates for reference.
As with the previous pitcher dataset, some pitchers didn’t have a projection. In all, eight pitchers were missing from one or more of three different projections.
Projection: Players missing
Pods: 6
THE BAT: 2
Razzball: 1
For this reason, I created two comparisons. One with all the projection systems but fewer pitchers and one with all the pitchers but fewer projection systems. Here are the results for the rate stats.
Projection | RMSE | Projection | Avg Value |
---|---|---|---|
Steamer | 1.320 | RotoWire | 3.69 |
DepthCharts | 1.321 | PECOTA | 3.78 |
ZIELE | 1.324 | Paywall #1 | 3.82 |
ATC | 1.330 | Pods | 3.82 |
THE BAT | 1.343 | Mastersball | 3.97 |
ZiPS | 1.351 | Average | 3.98 |
Razzball | 1.357 | THE BAT | 3.98 |
Davenport | 1.357 | ATC | 3.99 |
Average | 1.378 | ZIELE | 4.03 |
Paywall #1 | 1.379 | ZiPS | 4.07 |
PECOTA | 1.417 | Actual | 4.07 |
Mastersball | 1.428 | DepthCharts | 4.10 |
RotoWire | 1.529 | Razzball | 4.13 |
Pods | 1.538 | Steamer | 4.13 |
Davenport | 4.17 |
Projection | RMSE | Projection | Avg Value |
---|---|---|---|
THE BAT | 1.285 | PECOTA | 3.77 |
ZIELE | 1.305 | Paywall #1 | 3.82 |
DepthCharts | 1.308 | Mastersball | 3.96 |
Steamer | 1.311 | Average | 3.97 |
ATC | 1.313 | ATC | 3.99 |
ZiPS | 1.333 | THE BAT | 3.99 |
Davenport | 1.340 | ZIELE | 4.02 |
Paywall #1 | 1.358 | Actual | 4.06 |
Average | 1.364 | ZiPS | 4.06 |
PECOTA | 1.398 | DepthCharts | 4.10 |
Mastersball | 1.406 | Steamer | 4.13 |
Davenport | 4.16 |
All Projections | Adj RMSE | All Pitchers | Adj RMSE |
---|---|---|---|
Average | 1.213 | Average | 1.229 |
THE BAT | 1.286 | DepthCharts | 1.319 |
ZIELE | 1.308 | ZIELE | 1.319 |
Pods | 1.309 | Steamer | 1.320 |
DepthCharts | 1.312 | ATC | 1.325 |
Steamer | 1.313 | THE BAT | 1.341 |
Razzball | 1.313 | ZiPS | 1.343 |
ATC | 1.314 | Davenport | 1.346 |
ZiPS | 1.337 | Paywall #1 | 1.355 |
Davenport | 1.337 | PECOTA | 1.387 |
Paywall #1 | 1.344 | Mastersball | 1.416 |
PECOTA | 1.379 | ||
Mastersball | 1.410 | ||
RotoWire | 1.447 |
Projection | RMSE | Projection | Avg Value |
---|---|---|---|
THE BAT | 0.206 | RotoWire | 1.199 |
Pods | 0.212 | Paywall #1 | 1.235 |
ATC | 0.212 | Actual | 1.235 |
ZIELE | 0.215 | Pods | 1.250 |
DepthCharts | 0.216 | ATC | 1.252 |
ZiPS | 0.218 | Davenport | 1.256 |
Davenport | 0.220 | THE BAT | 1.257 |
Steamer | 0.221 | Mastersball | 1.258 |
PECOTA | 0.221 | Average | 1.261 |
Razzball | 0.223 | ZIELE | 1.269 |
Paywall #1 | 0.223 | ZiPS | 1.270 |
Mastersball | 0.230 | PECOTA | 1.272 |
Average | 0.248 | DepthCharts | 1.283 |
RotoWire | 0.259 | Steamer | 1.295 |
Razzball | 1.298 |
Projection | RMSE | Projection | Avg Value |
---|---|---|---|
ATC | 0.216 | Paywall #1 | 1.236 |
ZIELE | 0.219 | Actual | 1.240 |
DepthCharts | 0.219 | ATC | 1.254 |
ZiPS | 0.222 | THE BAT | 1.258 |
Davenport | 0.222 | Davenport | 1.261 |
Steamer | 0.223 | Mastersball | 1.261 |
PECOTA | 0.225 | Average | 1.263 |
THE BAT | 0.226 | ZIELE | 1.271 |
Paywall #1 | 0.229 | ZiPS | 1.273 |
Mastersball | 0.235 | PECOTA | 1.273 |
Average | 0.250 | DepthCharts | 1.285 |
Steamer | 1.296 |
All Projections | Adj RMSE | All Pitchers | Adj RMSE |
---|---|---|---|
Average | 0.196 | Average | 0.202 |
THE BAT | 0.205 | DepthCharts | 0.215 |
DepthCharts | 0.211 | ATC | 0.216 |
ATC | 0.212 | ZIELE | 0.217 |
Pods | 0.212 | Steamer | 0.218 |
Steamer | 0.213 | ZiPS | 0.219 |
ZIELE | 0.213 | Davenport | 0.220 |
Razzball | 0.215 | PECOTA | 0.222 |
ZiPS | 0.215 | THE BAT | 0.227 |
PECOTA | 0.218 | Paywall #1 | 0.228 |
Davenport | 0.219 | Mastersball | 0.232 |
Paywall #1 | 0.224 | ||
Mastersball | 0.229 | ||
RotoWire | 0.258 |
Projection | RMSE | Projection | Avg Value |
---|---|---|---|
ATC | 0.173 | THE BAT | 1.039 |
Razzball | 0.173 | Davenport | 1.041 |
ZIELE | 0.173 | Steamer | 1.052 |
Steamer | 0.174 | Razzball | 1.054 |
PECOTA | 0.175 | ZIELE | 1.061 |
DepthCharts | 0.176 | Pods | 1.063 |
Pods | 0.177 | Average | 1.064 |
THE BAT | 0.181 | ATC | 1.064 |
Davenport | 0.185 | DepthCharts | 1.065 |
Mastersball | 0.187 | Paywall #1 | 1.068 |
ZiPS | 0.189 | Mastersball | 1.075 |
Paywall #1 | 0.189 | ZiPS | 1.080 |
Average | 0.196 | Actual | 1.080 |
RotoWire | 0.196 | PECOTA | 1.083 |
RotoWire | 1.087 |
Projection | RMSE | Projection | Avg Value |
---|---|---|---|
Steamer | 0.191 | THE BAT | 1.039 |
ATC | 0.194 | Davenport | 1.041 |
DepthCharts | 0.196 | Steamer | 1.051 |
PECOTA | 0.197 | ZIELE | 1.061 |
ZIELE | 0.197 | Average | 1.063 |
Mastersball | 0.208 | ATC | 1.064 |
Davenport | 0.209 | DepthCharts | 1.064 |
ZiPS | 0.211 | Paywall #1 | 1.067 |
Paywall #1 | 0.211 | Actual | 1.072 |
Average | 0.215 | Mastersball | 1.074 |
THE BAT | 0.218 | ZiPS | 1.079 |
PECOTA | 1.082 |
All Projections | Adj RMSE | All Pitchers | Adj RMSE |
---|---|---|---|
Average | 0.161 | Average | 0.178 |
Razzball | 0.171 | Steamer | 0.191 |
Steamer | 0.172 | ATC | 0.194 |
ATC | 0.173 | PECOTA | 0.196 |
ZIELE | 0.173 | DepthCharts | 0.197 |
PECOTA | 0.176 | ZIELE | 0.197 |
DepthCharts | 0.176 | Davenport | 0.207 |
Pods | 0.177 | Mastersball | 0.208 |
THE BAT | 0.177 | ZiPS | 0.210 |
Davenport | 0.182 | Paywall #1 | 0.210 |
Mastersball | 0.188 | THE BAT | 0.216 |
ZiPS | 0.189 | ||
Paywall #1 | 0.189 | ||
RotoWire | 0.196 |
Projection | RMSE | Projection | Avg Value |
---|---|---|---|
DepthCharts | 0.0287 | Mastersball | 0.0571 |
Pods | 0.0288 | THE BAT | 0.0582 |
Davenport | 0.0291 | Steamer | 0.0586 |
Average | 0.0297 | ATC | 0.0603 |
ZiPS | 0.0300 | Davenport | 0.0608 |
ZIELE | 0.0301 | Paywall #1 | 0.0610 |
ATC | 0.0304 | PECOTA | 0.0612 |
Razzball | 0.0305 | ZIELE | 0.0613 |
Paywall #1 | 0.0305 | Average | 0.0617 |
Steamer | 0.0307 | Actual | 0.0621 |
THE BAT | 0.0308 | RotoWire | 0.0625 |
PECOTA | 0.0309 | Razzball | 0.0632 |
RotoWire | 0.0326 | DepthCharts | 0.0641 |
Mastersball | 0.0342 | Pods | 0.0643 |
ZiPS | 0.0687 |
Projection | RMSE | Projection | Avg Value |
---|---|---|---|
DepthCharts | 0.0295 | Mastersball | 0.0563 |
Davenport | 0.0302 | THE BAT | 0.0578 |
Average | 0.0306 | Steamer | 0.0581 |
ZiPS | 0.0308 | ATC | 0.0600 |
ZIELE | 0.0309 | Davenport | 0.0606 |
Steamer | 0.0311 | Paywall #1 | 0.0609 |
THE BAT | 0.0317 | ZIELE | 0.0610 |
ATC | 0.0317 | PECOTA | 0.0612 |
PECOTA | 0.0317 | Average | 0.0613 |
Paywall #1 | 0.0326 | Actual | 0.0618 |
Mastersball | 0.0351 | DepthCharts | 0.0639 |
ZiPS | 0.0688 |
All Projections | Adj RMSE | All Pitchers | Adj RMSE |
---|---|---|---|
Average | 0.0274 | Average | 0.0283 |
DepthCharts | 0.0287 | DepthCharts | 0.0295 |
Pods | 0.0288 | ZiPS | 0.0298 |
ZiPS | 0.0290 | Davenport | 0.0303 |
Davenport | 0.0292 | ZIELE | 0.0311 |
ZIELE | 0.0303 | Steamer | 0.0312 |
ATC | 0.0305 | PECOTA | 0.0317 |
Razzball | 0.0305 | ATC | 0.0318 |
Steamer | 0.0307 | THE BAT | 0.0319 |
Paywall #1 | 0.0307 | Paywall #1 | 0.0329 |
THE BAT | 0.0309 | Mastersball | 0.0355 |
PECOTA | 0.0310 | ||
RotoWire | 0.0326 | ||
Mastersball | 0.0348 |
Notes
- Once adjusting to the actual league value, the average projection handily wins. It has been this case with all the analyses.
- THE BAT performed decently for WHIP and ERA.
- I’m amazed at the variation of ERA (range of 0.4) and WHIP (range of 0.10) between the projection. Again, it is a stat I want to track as the season gets closer.
I am going to piece together one more article on how I would recommend approaching projection accuracy going into the 2021 season. I’ll likely finish it after Thanksgiving. As for now, it’s back to Mining the News and trying to get The Process out.
* ATC, Clay Davenport, Depth Charts, Harper Wallbanger, Luke Gloeckner, Mike Podhorzer, CBS Sports, ESPN, RotoChamp, Razzball, numberFire, Steamer, THE BAT, and ZiPS.
Jeff, one of the authors of the fantasy baseball guide,The Process, writes for RotoGraphs, The Hardball Times, Rotowire, Baseball America, and BaseballHQ. He has been nominated for two SABR Analytics Research Award for Contemporary Analysis and won it in 2013 in tandem with Bill Petti. He has won four FSWA Awards including on for his Mining the News series. He's won Tout Wars three times, LABR twice, and got his first NFBC Main Event win in 2021. Follow him on Twitter @jeffwzimmerman.
fangraphs steamer is inaccurate.. the formula is messed up and not correct.. there is a major problem with how it calculates projection.. this formula is inaccurate and has major flaw..
This is the major flaw.. fangraphs steamer calculates every level a prospect performs.. if the pitcher prospect is at CPX then High A then AA.. and pitches 3 inning in CPX has 0 era and 7k with 70.0% k%.. pitches 37.1 inning Low A has 3.62 era and 42k with27.5% K%.. pitches 35 inning with 5.66 era and 40k and 26.3% K%.. Fangraphs steamer calculates the 3 levels equally which strews the Projection incorrectly because the CPX level was very high numbers but was only 3 innings.. but those 3 inning’s big stat numbers cause the fangraphs steamer projection to be much higher than it should because the formula in the fangraphs steamer is wrong causing incorrect projections