Why Picking Your Projection System Matters
Prior to each baseball season, usually in January or February, I put together a massive spreadsheet that rates players in every format I play in. Placing a value on a given player is actually not that hard, assuming you have a decent projection of what that player will do over the course of the next season. Valuing “Albert Pujols” may not always be easy, but if I tell you I have a 1B who will put up 31 HR with a .285/.359/.516 line, that is something you can probably work with.
The issue is that depending on what system you pick, you could end up with some very different values.
Just as an example, I picked five players at random from the ZiPS projections released to date and compared their total points in an ottoneu league in three different projection systems: ZiPS, CAIRO, and Bill James Projections. A table of the results are below:
| Player | ZiPS | CAIRO | BJ |
|---|---|---|---|
| Erick Aybar | 624.1 | 602.1 | 570.6 |
| Albert Pujols | 928.3 | 978.1 | 1247.8 |
| Ryan Zimmerman | 822.9 | 763.7 | 943.1 |
| Pablo Sandoval | 684.2 | 762.0 | 870.0 |
| Yoenis Cespedes | 760.2 | 734.3 | 950.5 |
A couple things should jump out at you. First, the Bill James projections are roughly 150 points higher, on average, than the other two, and this is despite being LOWER for Aybar. James projects more than 300 more points from Pujols than does ZiPS. Last year, 319.5 fewer points would have moved me from 1st to 3rd in the FanGraphs Staff League.
Next, ZiPS and CAIRO are within 78 points of each other for all five players, and within 50 points of each other for three of the five. James is only within 50 points of the other two in one case – a 31.5 point gap between James and CAIRO on Aybar.
The fact is, if you are using Bill James Projections as your primary method of valuing players, you are likely expecting bigger numbers from your players than other owners. Those projections have one player over 1300 pts (Miguel Cabrera), three more over 1200 (Pujols, Mike Trout, Prince Fielder), and 13 more over 1000. CAIRO, by contrast, has no one even over 1200 and only six players (Cabrera, Joey Votto, Trout, Ryan Braun, Fielder, and robinson Cano) over 1000. That’s a big difference.
And it isn’t just hitters. Nineteen starting pitchers are projected to crack 1000 pts by the Bill James Projections; CAIRO has only 11. As a result, a pitcher who posts 200 IP, 200 H, 20 HR, 50 BB, 5 HBP, and 200 K, thereby scoring 949 points, would be the 19th ranked pitcher in CAIRO’s world and the 31st ranked pitcher in Bill James’s.
So, there are a couple things to keep in mind and a suggestion:
1) When you are talking to another owner and saying you expect 35 HR or 1,000 pts or whatever out of a given hitter, and he only projects 25 HR or 800 points, keep in mind the difference may have more to do with a projection system than anything else. One of you may be using ZiPS while the other uses Bill James.
2) Even if you agree on the stats, that does’t mean you agree on value. As with the 200 IP SP described above, one of you could be talking about a top-20 starter while the other is looking at a mid-tier guy.
And the suggestion:
Use multiple systems. I take the time each year to put together a mixed projection using a number of systems, but if you aren’t going to do that (and I don’t blame you for not wanting to do that), I recommend looking at multiple projection systems and building your own opinions based not simply on believing that ZiPS or CAIRO or James or anyone else is best, but on recognizing that each system will have its strengths and its flaws.
A long-time fantasy baseball veteran and one of the creators of ottoneu, Chad Young's is the Managing Editor for RotoGraphs, and can be heard on the Keep or Kut Podcast. You can follow him on Bluesky @chadyoung.bsky.social.
Do you guys ever go back and “grade” the projection systems to see which was the most accurate? I attempted to do that back in 2010 but it was too time consuming. I’m sure my method of scoring wasn’t the best either.
Gordon Beckham (Bill James) 2010
http://fan-exchange.com/mlb/predictionshitter.asp?playerid=beckhgo01&yearid=2010
White Sox (Bill James) 2010
http://fan-exchange.com/mlb/teamhittingpredictions.asp?userid=156&franchid=CHW&yearid=2010
White Sox (Bill James) 2010 Projection Score
http://fan-exchange.com/mlb/teamhittingpredscore.asp?franchid=CHW&yearid=2010&userid=156
The number you see in the Difference row and the AVG column is the score. The closer to 0 the more accurate. There is no weighting between stats though so if you are off by 1 AB or 1 HR it’s scored the same.
Tom Tango has done this – search the archives at The Book blog.
His tests are too general. wOBA is not a fantasy stat, except, roughly, in Ottoneu, and that’s what he tested. wOBA is very noisy, too, since overprojecting strikeouts and home runs washes out. A test of the 5×5 stats would be much more valuable.
There was a better test of 5×5 stuff in fangraphs community research about a year ago.
Here’s Tango’s study that mcbrown cited:
http://www.insidethebook.com/ee/index.php/site/article/testing_the_2007_2010_forecasting_systems_official_results/
The obvious solution: average multiple projections, with weights if you so desire.
You can, although that’s a lot of work for a full list of players. I think this is mostly just a nice way of saying that the Bill James team has a tendency to slap a best case scenario on almost every player. So it’s more about knowing the structure and style of the projection systems you’re dealing with. Back when I made mine they were league-based, so every win for one pitcher had to be balanced by a loss from someone else, 95% of a team’s runs had to be matched by the team’s RBI total, etc. That resulted in my projections being extremely conservative given that I couldn’t accurately anticipate increases or decreases in playing time.
If you’re using projections for fantasy baseball purposes then individual player regression models generally make the most sense because they produce the most likely results even if they’re bad at anticipating breakouts or decline. Frankly, middle ground results like that are also a benefit because they’re more similar to the ones your competition is working from. Going into a draft with projections that differ strongly ends up skewing who you target, as drafts are as much about recognizing who your opponents are getting and for how much as correctly assessing player value. If every speedster is being undervalued in the abstract sense, you end up with a team of them, which obviously makes their practical value much less.
Even better make your own projection! it’s easy with the right y2y correlations (http://www.beyondtheboxscore.com/2011/9/1/2393318/what-hitting-metrics-are-consistent-year-to-year) and the right assumptions. If you want help doing this, im sure the experts at fangrpahs/rotographs can do that. Otherwise feel free to email me at rotobanter@gmail.com – i wont email our projection formula or anything overwhelming like that, but i’ll emaill the steps and stats to start on (i.e. AB & PA; BB% & K%; BIP Data/hit trajectory; HR/FB; discipline stats,park factors, etc. etc. etc.)
Could you link in or reference an article on valuating players based on the projection? League specifications are certainly going to drive this, but it seems the valuation process can have subtle differences.
value above replacement is a great place to start: http://www.fangraphs.com/fantasy/index.php/basebal-fantasy-value-above-replacement/
Thanks!
For fantasy purposes, value is going to be extremely league-dependent. Not only do you have to consider the roster and scoring settings, but in a keeper league you have to factor in those kept, and then value changes during the course of a draft based on the current composition of your team and what players are left. Becoming familiar with value over replacement is indeed a good place to start, and the more you play with that the more comfortable you should be calculating specific values for your league as well as estimating adjusted values on the fly during your draft. Some people do all that with spreadsheets, but I don’t like spending the draft nose-deep in a laptop.
Any fangraphs writers willing to endorse any combination of projection systems?
The average of 3-5 systems is perfectly fine. The Wisdom of Crowds.
My system 🙂 But seriously, I like Steamer the best for pitcher projections. Last year, I noticed I tended to agree with their set the most for individual pitchers. I also like their process since they take additional variables into account, such as fastball velocity.
Is there a valuation formula that anyone would recommend? Ideally, I would like to take the average of a number of reputable projections then come up with a valuation.
Anyone know what’s happened to Last Player Picked? I’ve always used their price guide you can customize to your league’s categories. Seems like the site is gone…..
LPP was resurrected for 2014 on the draftbuddy.com site.
Is there any truth to the thought that the value between the systems might be very similar?
What I mean is if you were to rank all the 1B options using each of the 3 projection systems independently and then compare the rankings, you may find that the order is very close. If that is the case then all 3 systems would have value. You might think your tam will score more points than it will, but you would also be overestimating the totals of your opponents.
The area it gets sticky in is when you try to choose between the highest ranked 1B left and the highest ranked 3B left.
How does one reconcile the fact that BJ seems to be overly optimistic on both hitters AND pitchers? IE, if his system anticipates that batters are producing more runs and pitchers are allowing fewer … how does this universe add up?
One reconciles this by not using BJ’s projections.
Or by using BJ projection as the optimistic ceiling.
I generally look to ZiPS for low/middle end projections and BJ for the high end. Gives me a sort of spread around their projections. If I’m unsure I’ll consult Steamer or PECOTA.
It is surprising to me that so many people argue for combining systems. I know as a creator of a projection system, my outliers are different for a reason and at the end of the season they are just as likely to be right as wrong. But when you have a good outlier, you can get that player as a discount. By combining systems, you don’t have outliers you can get at a discount.
It would be an interesting study to see how correct the various systems are in their outliers.
I’ve done this in the past using some simple z-score analysis; basically, measuring likelihoods for each z-score bin for each projection system. This is one of the few applications where I get Bill James’s projections to not have a 0 coefficient.
Yeah. For fantasy owners, the value from a projection system is from the outliers. If you average them all together, suddenly you have the same projections as everyone else, safe and comfortable. That won’t help you win your league. It’s the projections that make you go the extra dollar or drop out of the bidding early that are valuable, and they disappear when averaging the systems.
Disagree completely. For fantasy owners, the main value of a projection system is taking emotion out of player evaluation. Averaging several systems makes it less likely you’re going to latch onto a particular outlier that conforms to your personal bias (“I’m feeling a breakout for Joe Schmoe this year and ZiPS agrees!”).
To mcbrown,
The point is more about how some systems have strengths. For example, I know based on my results that my system is exceptional at projecting batting average and it is intuitive to me because I have a unique system for doing it.
Other systems view it more as a fluctuating little part of OPS rather than an all important fantasy stat and just use some weighted mean of a few years to get it.
Similarly, Shandler puts a lot more care into his SB projections than non-fantasy systems. Oliver is built around good MLEs and is best for young players. Etc.
When you average you lose the individual strengths of those systems.
Agree with Chad on the point that each system will have strengths and weaknesses and it is important for owners to know those. But averaging each system is a really bad idea for reasons previous posters have pointed out. Here is a helpful link to learn the basics about projection systems:
http://www.fangraphs.com/library/index.php/the-projection-rundown-the-basics-on-marcels-zips-cairo-oliver-and-the-rest/
It is also important to consider what projection systems are using for their run environment. There is a great article here explaining the details of this thought process:
http://www.hardballtimes.com/main/fantasy/article/the-crazy-bill-james-projections/
The point is, each projection system needs to be taken into context. If we are expecting Albert Pujols to put up the exact points projected by James in this article, we may be disappointed. But if we value him with the James system and know he is worth incrementally more than the next best 1b, it doesn’t really matter what the ‘total points’ projected is. The point is to find value.
Finally, if readers would like to learn more about projection system accuracy, check out these links (although Tango link above is more recent I’m aware of):
Most recent study: http://www.insidethebook.com/ee/index.php/site/article/testing_the_2007_2010_forecasting_systems_official_results/
BP (2007): http://www.baseballprospectus.com/unfiltered/?p=564#34764
Chone (2006): http://lanaheimangelfan.blogspot.com/2006/12/pecota.html
One thing to note is when the Pecota (BP) system was at it’s best Nate Silver was at the helm and he is no longer associated with the system.
If you’re choosing one set of projections, the actual projection doesn’t matter as much as how the players rank by those projections. In the example of the Bill James’ Pujols projection, Pujols is projected for 1247.8 points. What’s important is how that point total compares to the replacement level first baseman in the Bill James projections.
If all of James’ projections are high, then the replacement level first baseman will be high also, so the difference between Pujols and the replacement level could be the same as any other projection system.
Whether you use points in an Ottoneau league or z-scores, the important thing is the difference between that player and replacement level.