Projection Accuracy: Late March Pitcher Counting Stats

I’m on the home stretch with most of the comparisons (part 1, 2, 3, 4, 5, 6, and 7) already done. Today, the counting stats for the late-season pitcher projections taking center stage. The boys over at Razzball dominated most of the results with the aggregators coming in near the top … again (might be a theme).

First, here are the projections analyzed.

• Steamer (FanGraphs)
• ZIPS
• DepthCharts (FanGraphs)
• THE BAT
• Davenport
• ATC (FanGraphs)
• ZIELE (Fantasy Pros)*
• Pod (Mike Podhorzer)
• Masterball (Todd Zola)
• PECOTA (Baseball Prospectus)
• RotoWire
• Razzball (Steamer)
• Paywall #1
• Average of the above projections

To create a list of players to compare for accuracy, I took the NFBC ADP (players in demand at that time) and selected all the pitchers in the top-450 drafted players (30-man roster, 15 teams in the Main Event) in ten or more leagues. Then I removed all the pitchers who never threw an inning.

As for predicting Saves, two projections (Zips and THE BAT) don’t project Saves so I removed them. Also, I limited the player pool down to any pitcher from the initial subset who accumulated one Save or was projected for one Save.

To determine accuracy, I calculated the Root Mean Square Error (RMSE) for two different sets of values. RMSE is a “measure of how far from the regression line data points are” and the smaller a value the better. Additionally, I included the actual and league average rates for reference.

As with the previous pitcher dataset, some pitchers didn’t have a projection. In all, eight pitchers were missing from one or more of three different projections.

Projection: Players missing
Pods: 6
The Bat: 2
Razzball: 1

For this reason, I created two comparisons. One with all the projection systems but fewer pitchers and one with all the pitchers but fewer projection systems. Here are the results for the counting stats. The Win on is dumb, I want to know if any system was just way off.

Notes

• Continuing a theme from the early season pitcher projections, Razzball absolutely crushed the innings and strikeout projections. The only time the RMSE has seen such a large difference is when the league-adjusted Average is compared to the other rate stats. Just by focusing on innings, Pods, PECOTA, and the aggregators also held up.
• All the projections over projected the number of innings thrown which is not a surprise based on pitcher injuries. ZiPS, PECOTA, and Razzball all were on their own tier closer to the eventual average. Some other projections overshot by 15 more innings than those three.
• With Saves, the Paywall projection comes out of nowhere to edge out the Average. With Saves, I want to pull in as many sources and see what each one thinks. This is more of a case of not missing out on a possible role change. I am a little worried about the herd effect driving some of these Save totals.

Just one more analysis to go and I might write something up to tie it all together so I can compare all the results.

* ATC, Clay Davenport, Depth Charts, Harper Wallbanger, Luke Gloeckner, Mike Podhorzer, CBS Sports, ESPN, RotoChamp, Razzball, numberFire, Steamer, THE BAT, and ZiPS.