Projection Accuracy: Late March Pitcher Counting Stats

I’m on the home stretch with most of the comparisons (part 1, 2, 3, 4, 5, 6, and 7) already done. Today, the counting stats for the late-season pitcher projections taking center stage. The boys over at Razzball dominated most of the results with the aggregators coming in near the top … again (might be a theme).

First, here are the projections analyzed.

  • Steamer (FanGraphs)
  • ZIPS
  • DepthCharts (FanGraphs)
  • THE BAT
  • Davenport
  • ATC (FanGraphs)
  • ZIELE (Fantasy Pros)*
  • Pod (Mike Podhorzer)
  • Masterball (Todd Zola)
  • PECOTA (Baseball Prospectus)
  • RotoWire
  • Razzball (Steamer)
  • Paywall #1
  • Average of the above projections

To create a list of players to compare for accuracy, I took the NFBC ADP (players in demand at that time) and selected all the pitchers in the top-450 drafted players (30-man roster, 15 teams in the Main Event) in ten or more leagues. Then I removed all the pitchers who never threw an inning.

As for predicting Saves, two projections (Zips and THE BAT) don’t project Saves so I removed them. Also, I limited the player pool down to any pitcher from the initial subset who accumulated one Save or was projected for one Save.

To determine accuracy, I calculated the Root Mean Square Error (RMSE) for two different sets of values. RMSE is a “measure of how far from the regression line data points are” and the smaller a value the better. Additionally, I included the actual and league average rates for reference.

As with the previous pitcher dataset, some pitchers didn’t have a projection. In all, eight pitchers were missing from one or more of three different projections.

Projection: Players missing
Pods: 6
The Bat: 2
Razzball: 1

For this reason, I created two comparisons. One with all the projection systems but fewer pitchers and one with all the pitchers but fewer projection systems. Here are the results for the counting stats. The Win on is dumb, I want to know if any system was just way off.

 

Late March Innings Pitched Projection Accuracy: All Projections
Projection RMSE Projection Average
Razzball 45.5 Actual 103.2
Pods 48.9 Razzball 114.4
Average 50.7 PECOTA 116.6
Zeile 50.7 Zips 116.9
PECOTA 50.7 Pods 120.9
ATC 50.8 THE BAT 122.6
DepthCharts 51.7 ATC 122.7
Paywall #1 52.3 DepthCharts 122.9
THE BAT 53.4 Zeile 123.1
Zips 53.7 Average 123.4
Steamer 54.0 Paywall #1 126.0
Rotowire 55.6 Mastersball 126.9
Davenport 55.9 Davenport 127.4
Mastersball 56.7 Steamer 128.8
Rotowire 131.3

 

Late March Innings Pitched Projection Accuracy: All Pitchers
Projection RMSE Projection Average
Average 50.1 Actual 101.1
Zeile 50.2 PECOTA 114.3
PECOTA 50.5 Zips 115.4
ATC 50.7 ATC 120.0
DepthCharts 51.1 DepthCharts 120.0
Paywall #1 51.9 THE BAT 120.3
Steamer 53.2 Zeile 120.4
THE BAT 53.2 Average 120.7
Zips 53.4 Paywall #1 122.9
Davenport 55.3 Mastersball 123.8
Mastersball 55.9 Davenport 124.4
Steamer 125.8

 

Late March Strikeout Projection Accuracy: All Projections
Projection RMSE Projection Average
Razzball 44.6 Actual 108.4
ATC 48.5 Razzball 114.4
Average 48.6 PECOTA 116.6
Pods 48.6 Zips 116.9
Zeile 48.7 Pods 120.9
DepthCharts 49.5 THE BAT 122.6
Paywall #1 49.7 ATC 122.7
PECOTA 50.0 DepthCharts 122.9
THE BAT 50.7 Zeile 123.1
Steamer 51.4 Average 123.4
Rotowire 52.3 Paywall #1 126.0
Zips 52.6 Mastersball 126.9
Davenport 52.8 Davenport 127.4
Mastersball 53.7 Steamer 128.8
Rotowire 131.3

 

Late March Strikeout Projection Accuracy: All Pitchers
Projection RMSE Projection Average
Average 48.1 Actual 106.1
Zeile 48.3 PECOTA 114.3
ATC 48.5 Zips 115.4
DepthCharts 49.0 ATC 120.0
Paywall #1 49.7 DepthCharts 120.0
PECOTA 49.9 THE BAT 120.3
Steamer 50.8 Zeile 120.4
THE BAT 50.8 Average 120.7
Zips 52.3 Paywall #1 122.9
Davenport 52.4 Mastersball 123.8
Mastersball 53.0 Davenport 124.4
Steamer 125.8

 

Late March Win Projection Accuracy: All Projections
Projection RMSE Projection Average
Razzball 3.7 Actual 6.6
Average 3.8 PECOTA 6.9
ATC 3.8 Razzball 7.2
PECOTA 3.8 THE BAT 7.2
THE BAT 3.8 ATC 7.3
Zeile 3.9 Mastersball 7.3
DepthCharts 3.9 Zeile 7.4
Pods 3.9 Average 7.5
Steamer 3.9 Zips 7.5
Paywall #1 4.0 Paywall #1 7.5
Davenport 4.0 Steamer 7.6
Rotowire 4.2 DepthCharts 7.6
Zips 4.2 Pods 7.6
Mastersball 4.3 Davenport 7.7
Rotowire 8.0

 

Late March Win Projection Accuracy: All Pitchers
Projection RMSE Projection Average
Average 3.8 Actual 6.5
ATC 3.8 PECOTA 6.8
PECOTA 3.8 THE BAT 7.1
THE BAT 3.8 Mastersball 7.1
Zeile 3.8 ATC 7.1
DepthCharts 3.8 Zeile 7.3
Steamer 3.9 Average 7.3
Paywall #1 3.9 Paywall #1 7.3
Davenport 4.0 Steamer 7.4
Zips 4.2 DepthCharts 7.4
Mastersball 4.3 Zips 7.4
Davenport 7.5

 

Late March Save Projection Accuracy: All Projections
Projection RMSE Projection Average
Paywall #1 8.9 Razzball 10.5
Average 9.1 Actual 14.1
Zeile 9.2 ATC 14.2
Razzball 9.4 Steamer 14.2
Davenport 9.5 DepthCharts 14.2
ATC 9.6 Zeile 14.5
Steamer 9.9 Pods 14.7
DepthCharts 9.9 Paywall #1 14.7
Mastersball 10.0 Average 15.0
Pods 10.1 Rotowire 16.2
Rotowire 10.3 PECOTA 16.3
PECOTA 10.9 Davenport 16.4
Mastersball 18.2

 

Late March Save Projection Accuracy: All Pitchers
Projection RMSE Projection Average
Paywall #1 9.0 Actual 12.4
Average 9.2 Steamer 12.5
Zeile 9.3 DepthCharts 12.5
Davenport 9.6 ATC 12.7
ATC 9.9 Zeile 12.8
Steamer 10.0 Paywall #1 12.9
DepthCharts 10.0 Average 13.2
Mastersball 10.2 Davenport 14.4
PECOTA 11.2 PECOTA 14.7
Mastersball 16.0

Notes

  • Continuing a theme from the early season pitcher projections, Razzball absolutely crushed the innings and strikeout projections. The only time the RMSE has seen such a large difference is when the league-adjusted Average is compared to the other rate stats. Just by focusing on innings, Pods, PECOTA, and the aggregators also held up.
  • All the projections over projected the number of innings thrown which is not a surprise based on pitcher injuries. ZiPS, PECOTA, and Razzball all were on their own tier closer to the eventual average. Some other projections overshot by 15 more innings than those three.
  • With Saves, the Paywall projection comes out of nowhere to edge out the Average. With Saves, I want to pull in as many sources and see what each one thinks. This is more of a case of not missing out on a possible role change. I am a little worried about the herd effect driving some of these Save totals.

Just one more analysis to go and I might write something up to tie it all together so I can compare all the results.

 

* ATC, Clay Davenport, Depth Charts, Harper Wallbanger, Luke Gloeckner, Mike Podhorzer, CBS Sports, ESPN, RotoChamp, Razzball, numberFire, Steamer, THE BAT, and ZiPS.





Jeff, one of the authors of the fantasy baseball guide,The Process, writes for RotoGraphs, The Hardball Times, Rotowire, Baseball America, and BaseballHQ. He has been nominated for two SABR Analytics Research Award for Contemporary Analysis and won it in 2013 in tandem with Bill Petti. He has won four FSWA Awards including on for his Mining the News series. He's won Tout Wars three times, LABR twice, and got his first NFBC Main Event win in 2021. Follow him on Twitter @jeffwzimmerman.

1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
biomanekcsmember
2 years ago

I appreciate your hard work, Jeff. It has to be a herculean task to crunch the data on the different analysis. I eagerly await your final summation of all this work.