The ATC Projection System
We are pleased to announce a new projection system, “ATC”, is available on FanGraphs. It can be found on the projection pages and is also available in the auction calculator.
Today we are introducing to the public, the Average Total Cost Projection System (ATC Projection System). The system gets its name from the fact that it “averages” many projection systems together.
The ATC system does not simply take a straight average of all the projection systems. Instead, each system accounts for a different weight for each statistic for which it projects (The weights are based on historical past performance). For example, System A might be given a 20% weight for batter homeruns. but just a 5% weight for pitcher strikeouts. System B might have a 10% weight for HRs, but just a 2% weight for Ks. And so on. The ATC system incorporates ZiPS, Steamer, FanGraphs FANS and other freely available projections, plus prior MLB statistics over the past 3 seasons. There are, of course, some manual edits made along the way, mostly for playing time projections, as we get closer to Opening Day.
A good way to understand what the ATC Projection System does – is to picture what Nate Silver does with his presidential election forecasting at www.fivethirtyeight.com. Nate collects polls data, adjusts them, grades them, weights them intelligently, combines them, and then calibrates them. If one polling firm has been more predictive than others in the past – it will receive a larger weight of the aggregation. Nate then adds in some manual tweaks and changes, and he has created a great model which is far more predictive of any one model by itself. Nate Silver does little or no polling of his own … yet his predictive model is the best in the industry. ATC operates in a similar manner, but for baseball.
The Average Total Cost System was developed by Ariel Cohen before the 2011 MLB season for the purposes of winning fantasy baseball leagues. Ariel and his fantasy partner, Reuven Guy, have used the ATC system projections in their home leagues ever since, winning half a dozen league titles. In 2015, they finished 2nd in the national Doubt Wars Mixed Auction league, and in 2016 they finished in 1st place in their NFBC New York Auction League.
Ariel Cohen is a fellow of the Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA). He is a Vice President of Risk Management for a large international insurance and reinsurance company.
You should join the $2,500 league; there are two openings.
Is there any evidence that these projections perform better than the Depth Charts projections? Is there any way to see the ATC Projection System’s pre-season projections for previous years?
I was actually going to add to this… Have the system weights been back-tested such that they warrant the over-weighting/under-weighting?
I typically will use a simple rate average of each category because the level of precision gets to the point where $1 either way in an auction isn’t going to matter that much in the calculation.
I may misunderstand the way the weights are applied, but back-testing seems unnecessary since the weights are determined by back-testing the source forecasts. In theory, this system would be less accurate than the best forecast for a single category, but more accurate than any across all forecasts.
I could imagine a mashup version of the projections that only takes the top performing projection system for each category (i.e. Depth Charts for ABs, Steamer for K%, ZIPS for HR, etc.)
Depends on how confident you were in evaluating the projections I guess. Without knowing the secret sauce in ATC, maybe that’s what happens – if Steamer is great at K%, maybe it gets 100% weight.
Right, but you can backtest an algorithm that is based on the past, as long as you don’t let it look ahead. Simply run the algo to determine what the weights would have been in March 2016 and see how it performed that season.
Using this system, we have finished Top 3 in about 90% of our leagues since 2011.
Your performance doesn’t mean ATC is “effective”. If you’re in a weak league and a $20 player is available for $16, you’ll bid $16 regardless of what projection system you use.
I’m not questioning if ATC isn’t the best predictor ex-post. But does the weighting have any material difference versus a simple average of systems? If ATC comes up with a value of a player of $20.14 and a simple average comes up with $19.87, is it really going to matter in an auction where the bid can fluctuate +/- $3 dollars vs. most projections?
So, ATC is supposed to work as well as Nate Silver’s Presidential Elections Forecast? No thanks.
Nate Silver said a few days before the election that Trump had a 1-in-3 chance of winning. Those were the best odds I saw anywhere.
I’m with Kevin. With binary outcomes (Presidential election), being less-wrong isn’t the same as being right.
This is just about the most ridiculous comment I may have come across on FG. Whether it rains or not is a binary outcome. Would you like your weather forecast to say “it’s not going to rain today” whether there’s a 1% or 49% chance of rain? Probabilities are used to express degrees of confidence of something happening.
Let’s put it in terms we can all relate to as baseball fans: Nate gave Trump the same chance of winning the election as Josh Donaldson had of getting a hit in 2016 (.284 AVG last year). Whereas most other models were giving him less than a pitcher’s chance of getting a hit (1-10%). Another recent sporting event with similar odds: the Warriors were heavy 70% favorites to win the NBA title against the Cavs. And we know how that turned out.
And if you had read 538 with any regularity, Nate and Harry repeatedly warned that Trump was only a standard polling error away from winning, and also had the electoral college advantage.
I’m no statistician, but I can understand basic probability. Thanks for the lesson though.
I’ve read 538 since inception. I respect Nate & his methodology – it’s proven better than individual polls. But there’s no way to spin that Nate was right on Trump. Less wrong, yes.
This isn’t the same as Donaldson getting a hit. With baseball, we have a sample size of hundreds/thousands for a player, and millions for all of baseball. We know the limits for a baseball player.
Do you think if we ran the election a thousand times, Hilary wins 65-70%? That’s what it means to say Nate was “right”. I don’t believe that’s the case.
Now we’re getting somewhere. It sounds to me like you are saying that 65-70% was still too high confidence for HRC, which certainly is a valid argument. I am looking forward to seeing what changes Nate will make for the next round.
Here’s how I think about it: Nate did his best to build a model and explain how it worked, and has tweaked it since he first started it years ago. He was getting bashed for being too unequivocal and not expressing MORE confidence in HRC, when it turns out the opposite may have been true. On the other hand, he was using the same data that’s available to everyone else, and was wayy closer to the mark than other models. Put another way: for the 2020 election, would you trust some pundit that correctly predicted a Trump win… or are you going to look at what Nate’s model looks like?
I think distilling it down to “right or wrong” is not the way to look at it. You wouldn’t say that Steamer is “wrong.” Sure it might miss on a player (and be less accurate about certain types of players), but overall it’s usually the best (or one of the best). Just like Nate will miss some states (and in this case, a certain subset of states). But he was still by far the closest, seeing as other models were basically predicting a HRC win with an extremely high degree of confidence. I wouldn’t say Nate was right or wrong; he was the most accurate though.
“. . .for the 2020 election, would you trust some pundit that correctly predicted a Trump win… or are you going to look at what Nate’s model looks like?”
Yep, without providing any info on how weightings are done or how ATC works, we have no way to determine whether ATC makes sense or if it is “garbage in, garbage out.”
So if a dog ate out of a bowl that said Trump and that was the pundit’s methodology, you would go the route that since the dog was correct, its better at predicting future outcomes?
I guess I get why people are downvoting you, but I’ll speak up and say that was funny, you got an upvote from me.
Here I thought “ATC” stood for Ariel T. Cohen
This looks very promising. What makes Nate Silver’s models so great has to do with how he weights information. While he doesn’t let the world in on everything, there’s a lot written on the approaches he takes as well as on the underlying theory. Are you planning to shed any light on your methods?
Does Ariel’s middle name begin with T?
Yes! ?
J’ACCUSE!!!!!!!!!!!!!!!!!
I have a weird league and may be getting atypical results, but it looks like when I use ATC in the calculator, when a player has multi-position eligibility ATC uses the position adjustment for the player’s less-valuable position?
For example, it adds $7.4 to Kris Bryant as a 3B instead of the $13 for him as an OF. The calculator doesn’t seem to do that with the depth chart settings.
Is that a mistake, is there a reason behind it, or am I misunderstanding something?
The calculator uses a fixed position priority currently for multiple position players. This may be a problem depending on how “weird” your league is and shouldn’t be the result of any projection system. C, SS, 2B, 3B, OF, 1B, is the current ordering.
Is there any way to factor position eligibility into the projections? Players are going to benefit from having multiple position eligibility in almost all leagues.
Interesting. I’m in an 11 team NL-only league (5 outfielders) with no bench.
On the Depth Charts the values go C ($15.5), 3B ($13.4), OF ($12.2), SS ($12), 1B ($10.5), 2B ($9.8).
On the ATC the values go C ($16.5), OF ($13), 2B ($12.6), 1B ($11.5), SS ($11.2), 3B ($7.4).
So does that mean that for whatever reason, ATC likes the depth of the NL-only 3B pool much much more than Steamer/Zips? and 2B a lot less?
At the risk of looking this gift horse Auction Calculator in the mouth, are there any plans on incorporating a dynamic position priority based on the positional values generated?
Well, I’m not entirely sure how it might be done dynamically, the issue is that every time you move a player from one pool to another, you have to iterate again, and that becomes taxing. Also, you can get stuck in an iteration loop, but there are ways around that.
I’ve added a way you can statically set the position priority, so you can set things up how you like and experiment to see how much things change. For most leagues, the order is not going to matter much.
Um, this is awesome. Thank you for making this change! It really does help
if the auction calculator defaults a player to his weakest position off that list there’s seems to be a problem. 1B is no longer the deepest position. Just look what the calculator spits out.
One critique of existing projection systems is that they miss on the ‘tails’ of the bell curve. Because they regress stats to historical norms, they get more correct in total, but miss on key breakouts, etc.
I would expect ATC to exacerbate that issue. By taking averages/weighting, you’re reducing the variance between the projections. The end result may be slightly greater accuracy for the majority of projected players, but lesser predictive power at finding players at the top & bottom.
That doesn’t invalidate this method – as stated, existing projection systems don’t do a good job at finding breakouts from the norm. But if you’re trying to win a league, you do need to find overperformance somewhere.
ATC should actually help to identify undervalued players.
First of all, it does a better job of getting player projections right. If System A projects SBs better than the other systems do, and System B projects HRs better – you get a far better projection by taking a higher weight of SBs from A, and a higher weight of HRs from B.
So for players who are power-speed combo players – ATC should be much better in total than any one system alone. For roto leagues – The high valued & undervalued players are ones that over-perform in many categories.
Sure, system A will do a better job at getting the speed breakouts right. And sure, System B will do a better job at getting the power breakouts right. But ATC will do a better job in getting the 5 category players right. And the largest value breakouts will come from combo/multiple category players.
Thanks for reply. I think the point I’m perhaps inelegantly trying to convey is that projections as a whole haven’t solved the problem of the ‘leap’ players.
How extreme does the weighting get? My assumption is that you’re weighting based on your confidence in the projection’s ability to project a certain stat (retroactive testing). I’m curious how strong your confidence gets. Without giving away your secret sauce, can you relay the maximum weighting for a single category?
It makes a big difference if max weighting is 30%, or 50%, or 100%, right? Your point is that the best categorical projection system will receive the highest weight, thus producing a better overall projection. But unless you’re using 100% weight, you’re also reducing the impact of that outlier (correct) projection by pulling in other (worse) projections and weighting those. Presumably you aren’t confident enough to go 100% in one category so I’m curious as to the upper bound.
Just spitballing, but for finding breakouts, I wonder if looking for the outliers in each projection system separately would be a better path.
To answer your question, the max weighting is about one-third.
If there was such a projection system out there that could find a $30 player that is going in the market for $3 – Please point me to that system. Rarely will you find one single system that shows a projection so different from the rest.
What you realistically want to happen, is to find the $30 player which is going in the market for $15, that you project for $20. This will signal a “buy” [And conversely, you hope to identify the $5 player which in the market is going for $25, and that you project for $20, which signals, “don’t buy”]
To me, if you are just a few dollars more in projected value for “breakout players” than the market – you will end up rostering a number of breakout players on your fantasy teams.
ATC helps to do that. Some players that ended up on my fantasy rosters last year this way were – Jonathan Villar, Khris Davis, Mookie Betts, etc. They were rostered on my teams, simply because ATC added some modest $2-3 of value to their market prices.
If you just look for an “outlier” projection by itself – You will find many players that don’t pan out. I would rather be correct by $2-3 on average – and that’s where/how ATC works.
Good discussion. I get your approach and respect it for what it is.
You’re assuming bc System A has projected a certain category more accurately in the past, it will continue to do so. This is pretty unlikely, unless the out-performance is huge.
ATC sounds like curve-fitting to me; I would have to see evidence of performance before using it.
Will the ATC projections be on the player pages along with Steamer, ZiPS, Depth Charts, Fans? That would be quite nice!
Question about how keepers work with this system. Lets say my league had a whole lot of shortstops as keepers. When I plug these players in will the value of remaining shortstops go up relative to other players due to the new lack of depth?
if half the leauge has a SS kept is it a lack of depth? Those teams don’t need another SS.
Still seems like it matters. For example lets say shortstops 1-5 are all pretty similar but there’s a big dropoff at SS-6. If 2-5 are kept as keepers, SS-1 should become a bit more valuable.
Based on the way replacement level calculations work, no, it doesn’t matter. If there aren’t good SS’s available, just draft other positions…
Would love to see past years – hard to know whether they’re worth using without being able to compare to the other systems.
Any chance of getting a wRC+ output from ATC?
Just as devils advocate – this is used more towards fantasy (it was on RotoGraphs and the article mentioned fantasy a lot). wRC+ isn’t really useful in fantasy like wOBA (or even RC). You don’t really care WHERE the hits are being done at, just that they are being done. So even if it were a Rockies player vs a Giants player, the Rockies wOBA of .350 > Giants wOBA of .330 even though they may have the same wRC+ (just examples). You want the .350 wOBA regardless of park.
I agree that wRC+ would be nice though and seems easy to implement.
” Nate Silver does little or no polling of his own … yet his predictive model is the best in the industry.”
Source?
If you’re going to synthesize other people’s work, at least have the courtesy to disclose all of your sources and your weightings.
This is very cool. Would love to know who falls into the category of “other freely available projections”. I know of other subscription projections but not as familiar with proven free sources besides what’s on the page. Confidence in the fan projections are fading.
Also if a player doesn’t have 3 years of MLB experience how is that factored?
Very interested to take a look and appreciate having a new source to use.
looks cool. any way to add position to the export and projections?
1. Thanks
2. Will these be added to the player pages?
3. Can you disclose what component projection systems you’re using? EG, I am trying to figure out how the ATC projection for Matt Carpenter is higher than each of the Zips, Steamer, and Fans projections. And it’s not simply an issue of taking the most optimistic projection from each on a component basis; ATC sees a higher ISO than any of those does. And it’s not a just a trailing-performance issue either; you seem to be projecting Carpenter for the highest AVG of his career.
4. Tying in to trailing performance – for Appleman: any chance of bringing back the CHONE projections? They serve as a useful reference point for evaluations like this. I know Tango isn’t doing them anymore, but others are.
5. See point #1
In case it was unclear, this was not meant to be a Carpenter-specific question. He’s just an example. Votto and Jake Lamb seem to be in the same general bucket. Gregory Polanco and Brandon Belt, although perhaps to lesser degrees. Etc.
“The ATC system incorporates ZiPS, Steamer, FanGraphs FANS and other freely available projections, plus prior MLB statistics over the past 3 seasons.”
Does that mean no Pod Projections? Boo.
I used to do this years ago, and it was a lot of extra work for marginal gains. Each system has their strengths and it does help, though. Would you share any specifics about what you use from each system, even if not the exact weighting? One thing I remember from when I did them was PECOTA was always way, way off in projecting Wins from relief pitchers. That always stuck in my head even though it has almost no bearing on overall projections.
Maybe I’m missing something (if so please help me learn) but I’m concerned about the projections being spat out by ATC. Overall, it seems to over project Plate Appearances for fringe players.
For example, Jesse Winker… Depth Charts projections say 42 PA, Steamer 64, ZIPS 509 (we all know why)… ATC? It weighted averages them and says 354! I’d imagine most of us would bet our house that Winker won’t get over 300 PAs this year in lieu of an “act of god.”
Sooooo am I using ATC projections wrong? Should I not try to use ATC projection of counting stats the same way I use Depth Chart projections?
So am I just salty on Winker or do I not understand how to use ATC? Thank you for any feedback.
I definitely would not use playing time projections that include data from ZiPS. The Padres have a terrible rotation but ZiPS projects the pitchers in their system for a combined 478 Games Started. Obviously these are not useful for projecting playing time.
That does sound high for Winker. However, most of the underlying projection systems in the ATC projection model have him tabulated for significant playing time (> 150 PAs). Steamer happens to be on the low end for this fringe player.
I’m sure that Winker isn’t the only player who might look very high or low compared to the Depth Charts projections. But, that’s a good thing.