Projection Accuracy: Early March Pitcher Rate Stats

by
November 11, 2021

I completed the counting stat analysis on early March pitcher counting stats after finishing the hitter projection comparisons (part 1, 2, 3, 4, and 5).  it is time for the pitchers to take center stage. For the first article, I’ll measure the accuracy of counting stats from early March. The results were mixed this time with the aggregators having a decent showing.

First, here are the projections analyzed.

  • Steamer (FanGraphs)
  • ZIPS
  • DepthCharts (FanGraphs)
  • THE BAT
  • Davenport
  • ATC (FanGraphs)
  • Pod (Mike Podhorzer)
  • Masterball (Todd Zola)
  • PECOTA (Baseball Prospectus)
  • RotoWire
  • Razzball (Steamer)
  • Paywall #1
  • Average of the above projections

To create a list of players to compare for accuracy, I took the TGFBI ADP (players in demand at that time) and selected all the pitchers in the top-450 drafted players (30-man roster, 15 teams in the Main Event) in ten or more leagues. Then I removed all the pitchers who never threw an inning.

To determine accuracy, I calculated the Root Mean Square Error (RMSE) for two different sets of values. RMSE is a “measure of how far from the regression line data points are” and the smaller a value the better. Additionally, I included the actual and league average rates for reference.

I ran into a bit of an issue as 16 pitchers were missing from one or more of three different projections.

Projection: Players missing
Pods: 7
The Bat: 6
Rotowire: 5

For this reason, I created two comparisons. One with all the projection systems but fewer pitchers and one with all the pitchers but fewer projection systems. Here are the results with rate stats. I know the Win on is kind of dumb, I want to know if any system was just way off.

 

Early March ERA Projection Accuracy: All Projections
Projection RMSE Projection Avg Proj
THE BAT 1.27 Rotowire 3.67
Razzball 1.28 PECOTA 3.74
Steamer 1.28 Pods 3.80
DepthCharts 1.28 Paywall #1 3.81
ATC 1.28 Mastersball 3.93
Average 1.28 Average 3.97
Zips 1.30 ATC 4.06
Pods 1.31 Zips 4.06
Mastersball 1.34 Davenport 4.08
Paywall #1 1.34 DepthCharts 4.09
Davenport 1.34 Actual 4.10
Rotowire 1.40 Razzball 4.11
PECOTA 1.41 Steamer 4.12
THE BAT 4.20

 

Early March ERA Projection Accuracy: All Pitchers
Projection RMSE Projection Avg Proj
Razzball 1.33 PECOTA 3.76
Steamer 1.34 Paywall #1 3.82
ATC 1.34 Mastersball 3.95
DepthCharts 1.34 Average 3.99
Average 1.35 ATC 4.07
Zips 1.36 Zips 4.07
Paywall #1 1.40 DepthCharts 4.10
Mastersball 1.41 Davenport 4.12
Davenport 1.41 Razzball 4.12
PECOTA 1.48 Steamer 4.13
Actual 4.16

 

Early March ERA Projection Accuracy: Adjusted to League Scoring
All Projections RMSE All Pitchers RMSE
Average 1.18 Average 1.25
THE BAT 1.27 Razzball 1.34
Pods 1.28 ATC 1.34
Razzball 1.28 Steamer 1.34
ATC 1.28 DepthCharts 1.34
Steamer 1.28 Zips 1.37
DepthCharts 1.28 Paywall #1 1.37
Zips 1.31 Mastersball 1.40
Paywall #1 1.31 Davenport 1.41
Mastersball 1.33 PECOTA 1.44
Rotowire 1.35
Davenport 1.35
PECOTA 1.37

 

Early March WHIP Projection Accuracy: All Projections
Projection RMSE Projection Avg Proj
Pods 0.207 Rotowire 1.194
ATC 0.210 Paywall #1 1.232
Average 0.210 Actual 1.237
THE BAT 0.212 Pods 1.245
DepthCharts 0.212 Mastersball 1.255
Steamer 0.214 Average 1.266
Razzball 0.217 ATC 1.266
Zips 0.217 PECOTA 1.268
Paywall #1 0.218 Zips 1.271
PECOTA 0.221 DepthCharts 1.281
Mastersball 0.227 THE BAT 1.291
Davenport 0.228 Davenport 1.293
Rotowire 0.231 Steamer 1.293
Razzball 1.297

 

Early March WHIP Projection Accuracy: All Pitchers
Projection RMSE Projection Avg Proj
ATC 0.214 Paywall #1 1.235
Average 0.214 Actual 1.249
DepthCharts 0.216 Mastersball 1.259
Steamer 0.218 Average 1.268
Razzball 0.219 ATC 1.269
Zips 0.221 PECOTA 1.270
Paywall #1 0.221 Zips 1.274
PECOTA 0.228 DepthCharts 1.284
Davenport 0.230 Steamer 1.296
Mastersball 0.230 Davenport 1.298
Razzball 1.300

 

Early March WHIP Projection Accuracy: Adjusted to League Scoring
All Projections RMSE All Pitchers RMSE
Average 0.193 Average 0.204
THE BAT 0.205 ATC 0.214
Steamer 0.208 Steamer 0.214
Pods 0.208 DepthCharts 0.214
DepthCharts 0.208 Razzball 0.215
ATC 0.209 Zips 0.220
Razzball 0.209 Paywall #1 0.222
Zips 0.215 Davenport 0.224
PECOTA 0.219 PECOTA 0.228
Paywall #1 0.219 Mastersball 0.230
Davenport 0.221
Mastersball 0.226
Rotowire 0.228

 

Early March K/IP Projection Accuracy: All Projections
Projection RMSE Projection Avg Proj
Razzball 0.187 Davenport 1.016
Steamer 0.189 THE BAT 1.019
ATC 0.191 ATC 1.053
Average 0.191 Steamer 1.056
PECOTA 0.192 Razzball 1.058
DepthCharts 0.193 Average 1.063
Pods 0.196 Pods 1.068
THE BAT 0.201 DepthCharts 1.070
Mastersball 0.202 Paywall #1 1.074
Rotowire 0.205 Actual 1.076
Paywall #1 0.205 Mastersball 1.079
Zips 0.206 Zips 1.084
Davenport 0.211 PECOTA 1.089
Rotowire 1.089

 

Early March K/IP Projection Accuracy: All Pitchers
Projection RMSE Projection Avg Proj
Razzball 0.186 Davenport 1.010
ATC 0.186 ATC 1.049
Average 0.187 Steamer 1.056
Steamer 0.188 Razzball 1.059
DepthCharts 0.189 Average 1.059
PECOTA 0.191 DepthCharts 1.068
Mastersball 0.199 Actual 1.068
Paywall #1 0.201 Paywall #1 1.070
Zips 0.203 Mastersball 1.073
Davenport 0.208 Zips 1.079
PECOTA 1.083

 

Early March K/IP Projection Accuracy: Adjusted to League Scoring
All Projections RMSE All Pitchers RMSE
Average 0.177 Average 0.173
Razzball 0.187 ATC 0.186
Steamer 0.188 Razzball 0.187
ATC 0.190 Steamer 0.188
PECOTA 0.192 DepthCharts 0.190
DepthCharts 0.193 PECOTA 0.191
THE BAT 0.194 Mastersball 0.199
Pods 0.197 Paywall #1 0.201
Mastersball 0.203 Davenport 0.202
Rotowire 0.204 Zips 0.203
Davenport 0.205
Zips 0.206
Paywall #1 0.206

 

Early March W/IP Projection Accuracy: All Projections
Projection RMSE Projection Avg Proj
DepthCharts 0.0292 Mastersball 0.0574
Pods 0.0294 THE BAT 0.0582
Davenport 0.0296 Steamer 0.0592
Average 0.0296 ATC 0.0600
Zips 0.0298 Davenport 0.0606
ATC 0.0300 Actual 0.0607
Steamer 0.0306 PECOTA 0.0614
Paywall #1 0.0308 Average 0.0618
Razzball 0.0312 Paywall #1 0.0622
THE BAT 0.0315 Rotowire 0.0626
PECOTA 0.0320 Razzball 0.0634
Rotowire 0.0322 DepthCharts 0.0638
Mastersball 0.0340 Pods 0.0649
Zips 0.0684

 

Early March W/IP Projection Accuracy: All Pitchers
Projection RMSE Projection Avg Proj
DepthCharts 0.0299 Mastersball 0.0561
Average 0.0303 Steamer 0.0584
Davenport 0.0304 ATC 0.0594
Zips 0.0308 Davenport 0.0597
ATC 0.0309 Actual 0.0600
Steamer 0.0314 PECOTA 0.0607
Paywall #1 0.0315 Average 0.0612
Razzball 0.0319 Paywall #1 0.0619
PECOTA 0.0327 Razzball 0.0626
Mastersball 0.0349 DepthCharts 0.0635
Zips 0.0684

 

Early March W/IP Projection Accuracy: Adjusted to League Scoring
All Projections RMSE All Pitchers RMSE
Average 0.177 Average 0.173
Razzball 0.187 ATC 0.186
Steamer 0.188 Razzball 0.187
ATC 0.190 Steamer 0.188
PECOTA 0.192 DepthCharts 0.190
DepthCharts 0.193 PECOTA 0.191
THE BAT 0.194 Mastersball 0.199
Pods 0.197 Paywall #1 0.201
Mastersball 0.203 Davenport 0.202
Rotowire 0.204 Zips 0.203
Davenport 0.205
Zips 0.206
Paywall #1 0.206

Some thoughts on the results.

  • It’s tough to find a single winner here. The aggregators (ATC, Depth Charts, and the average) per near the top as they always have. Beyond those three Steamer, Razzball (use Steamer), Pods, and THE BAT place in the top half. Again, I’m like to recommend a select-crowd strategy to create a better aggregate projection.
  • The range of each of the projection values is larger than I expected with ERA going from 3.67 to 4.20. That’s over half a run. As I’ve stated before, this value range bugs me and it’s one item I want to take into account going forward. Once more than a couple of projections are available, I’ll start looking into these differences.

That’s it for now with just two looks to go next week. It’s time to go Mine the News.





 

Jeff, one of the authors of the fantasy baseball guide,The Process, writes for RotoGraphs, The Hardball Times, Rotowire, Baseball America, and BaseballHQ. He has been nominated for two SABR Analytics Research Award for Contemporary Analysis and won it in 2013 in tandem with Bill Petti. He has won four FSWA Awards including on for his Mining the News series. He's won Tout Wars three times, LABR twice, and got his first NFBC Main Event win in 2021. Follow him on Twitter @jeffwzimmerman.

1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
RotoholicMember since 2016
3 years ago

This is fantastic. I really love the inclusion of the Adjusted to League Scoring sections.

A major takeaway from this is: Keep it Simple. The average finished first in every single “Adjusted to League Scoring” category, and it wasn’t close. We can try to be fancy by taking the best components from each system, weighting them, and using a hybrid, like ATC. But this clearly didn’t work better than a straight average. I think the reason it doesn’t work is because we’re taking a small signal and amplifying it too much which creates noise. These systems change every year, and there are lots of variables. What works one year not only might not work the next year, but the systems themselves might have changed anyway. I do wonder, though, if we should remove or underweight projects with high collinearity. It probably doesn’t make sense to use both Steamer and Razzball’s ERA projection, for example. Or maybe weight each half as much as the others. And Depth Charts is just ZiPS and Steamer combined, I believe. So we can probably ignore that one.

I think ATC is probably still the best for playing time since there are far fewer variables with it and the people good at projecting playing time (Rudy, Podhorzer, Mastersball, etc), and the skills involved in doing it, don’t really change much. So it’s probably easy to predict who will be good at it next year.

Regarding who is best at projecting league average, I think this could just be luck rather than skill. So I don’t think it needs to be adjusted much, if at all. Unless there is one system that is consistently able to be among the best at projecting league average. I’m inclined to ignore it entirely unless there’s some reason not to. Maybe I’ll read about that in The Process.

3