Humanity vs. the Robots: Who Will Win?

While writing my recent articles about how to use projections to find undervalued players based on market ADPs, I was struck by a follow-up question: when ranking players, which is better – ADP, or projections?

As we strive to value players more objectively, probing this question is valuable. It will help calibrate how much weight to assign consensus rankings versus your favorite projection systems. If we’re lucky, it might point us to some things that the market or the robots identify particularly well relative to the other. When we can marry the strengths of objective, data-driven processes with the advantages of human behavior and judgment, we are better off for it.

And this is a meaty question; theoretically each method has its strengths. Average draft position is a ranking derived from the wisdom of thousands of fantasy owners. It should be valuable in this respect. But we fantasy owners are subject to our own pernicious little human biases when it comes to player evaluation. Are we better off just letting the robots do the work for us?

Methodology

To answer this question, I used year-to-date performance through last week in three different scoring settings: standard 5×5 roto, ESPN points and Yahoo points (note, for brevity, I will refer to these settings henceforth as roto, ESPN, and Yahoo, respectively). As I mentioned I would in Part 1 of my Asset Valuation series, I used 2026 pre-season ATC projections for both hitters and pitchers and set the Auction Calculator to a standard, 12-team, 23-man roster setting. My ADP data is sourced from FantasyPros because they provide ADP data for a variety of sources in the industry.

With the data in hand, my first step was straightforward. For each scoring setting and various flavors of ADP, I measured the correlations of projections and ADP with year-to-date player value. I used Spearman correlations, which measure the directional relationship between two rank variables. To accommodate this measure, turned my three variables into ranks, as opposed to their raw values which, for ADP would be the actual ADP value (1.3, or 23.7, or 44.2, etc.). For projections and YTD values, this would be the dollar value output from the Auction Calculator.

For the statistically inclined, this is in contrast to a Pearson correlation, which measures the relationship between two continuous variables. I felt Spearman correlations were a better fit for this question, given that it is seeking to determine which method of ranking is superior. Reasonable minds could disagree though, and I ran the Pearson correlations anyway; the results were broadly similar, so I will stick with presenting the Spearman results from this point forward.

As a note, I chose to compare Yahoo values with Yahoo ADP. This is because of Shohei Ohtani. Only Yahoo ADP features both Ohtanis, so it doesn’t make sense to correlate other ADP ranks with Yahoo settings when those settings will only be available on the Yahoo platform. For design consistency, I used ESPN ADP with ESPN points, and compared the overall industry average ADP for roto.

This tells us what we wanted to find out: correlations for the projection-based ranks are all higher to varying degrees. So far, the projections seem to be in the lead. But will these gaps hold up to greater scrutiny?

Verification

Realizing that the differences in these correlations were not that large, I sought a more rigorous way to determine whether we could have confidence in the result. Enter Steiger’s Z-Test.

I will spare you the gory details here but you can think of this test to be like a t-test that compares the results of two separate correlations that share a variable. The end result is a measure of confidence that the two correlations are not actually the same. There are a few additional, exceedingly technical caveats involving normal distribution assumptions that my data do not meet. If I ever sought to publish this as a more formal academic study, I would sort these out. But for the purposes of this analysis, as an overall reasonableness check, I believe the tests still add value.

Anyway, back to this planet – all three correlations passed the test:

Here, each column with r() is a correlation. You will notice the first two from my earlier table, and the third is correlating each respective rank value against each other. Unsurprisingly, they are quite correlated. The next column, n, is the number in the sample. Yahoo has one more because, as mentioned, it splits Ohtani.

Yahoo and Roto were significant to the p<0.001 threshold, while ESPN was significant only to the p<0.05 threshold. This makes sense, because the correlations for ESPN were the closest together. Based on this, and my technical caveat about distributions above, I think the ESPN-specific significance may not hold up given a more robust empirical treatment. So, perhaps use some caution when thinking about these results for ESPN, but not so much that it invalidates the overall analysis.

Now that I had confidence that projected ranks were in fact more predictive of YTD value than ADP based ranks, I wanted to find out where in the draft this edge is occurring. Is it a small, persistent benefit throughout the draft? Are projections better early on? Late? To this end, I looked at the scatterplots of each rank variable against YTD value, fitted with a smoothed line:

In each case, it seems that neither rank method is especially superior in the early rounds. Then, roughly around pick 150 in each, the lines diverge and indicate slightly higher values for projected ranks until much later in the draft.

To map this out more explicitly, I broke the ranking groups into several buckets in 75-slot increments up to 525. Within each bucket, I removed all overlapping players. Then, with the remaining players, I marked whether they were unique to the ADP rank or the projection rank, and tallied the average YTD values of each group. As I moved through each bucket, I had to make sure that players who had already shown up in an earlier bucket were not in subsequent ones to avoid double counting them.

To explain the last bit mechanically, and because it tripped me up the most when designing the analysis, consider the example of Zack Wheeler. He rates as the 43rd highest projected roto player, but is the 125th player off the board in ADP. As such, he would show up in both datasets of non-overlapping players for the top 50 and 51-150 buckets. His value is already being tallied and compared against ADP-preferred players in the top 50 bucket because he was rated stronger by projections. He should then not be included in the next bucket because he was already factored into a projection rank calculation in the top 50 bucket. That he would also be in the 51-150 bucket because is ADP would put him there, which was the lower of the two ranking methods for him. Therefore, it isn’t appropriate to give that ranking method the boost of his good performance to date, or to have him counted for projections twice.

In summary, slicing up the player pool in this manner will allow us to isolate the players driving differences in value for a given rank. Without further ado, I present below the average value of players unique to each bucket for each ranking.

A quick note on the composition: these buckets are not all symmetrical because the number of players preferred by projections is lower than the number of players preferred by ADP for each setting. This might seem unintuitive at first, and it did indeed briefly melt my brain, but consider the following simplified example:

• System 1 ranks Players A, B, C, and D in the following order: 1, 2, 3, and 4
• System 2 ranks Players A, B, C, and D in the following order: 4, 1, 2, and 3

Here, only one player is preferred by System 1, while three are preferred by System 2, demonstrating that it’s possible for an asymmetrical distribution of the rank preference sums.

Circling back to the table, the story is basically consistent with what we saw in the scatterplots: neither system is clearly advantaged in the first 150 or so picks (except ADP for ESPN, interestingly). Meanwhile the average value of players ranked higher by projections in the back end of drafts is typically better (or, perhaps more accurately, less bad) than the average value of players ranked higher by ADP.

These results alone are interesting, but there is some still some more digging to be done. I also wanted to figure out whether there was a specific type of player driving these gaps, so I looked into the types of players that ADP and projections were favoring. I put the players from the first two and last five buckets above into one new bucket each (top 150 and 151+), filtered for players with greater than $0 of YTD value, and created position groupings.

As a note, I did not reconfigure which players were unique based on new buckets; there would be too much overlap in the respective pools to derive useful analysis. The idea here was to be able to identify the type of player that each rank methodology might find more readily, and keeping the distinctions noted above helps achieve this goal. For players ranked in the top 150 of each method:

It seems that projections are identifying good relievers more often relative to ADP, given that ADP is not identifying any. Now, on to the back end of the rankings boards:

The pattern here is similar. Projections are again more frequently favoring positive-value relievers in each setting.

To me, this makes sense. It’s not unusual to find clusters of relievers with decent projections but unclear roles at the back end of drafts. These players are going to be ignored by the market; usually they don’t have a ton of name value and there are a number of well-known veteran hitters clogging the same area of the ranks. Why would you draft Adrian Morejon when Nolan Arenado is sitting there on the board? He could bounce back on a new team (or so you tell yourself, and then proceed to cut him for a reliever after opening weekend).

Projection Strengths

To tell this story a little clearer, I wanted to verify whether projections actually were hitting on a number of good relievers, or if the slice above picked up many lower-valued, but still positive relievers while the real value was driven by hits at other positions. To do this, I looked at which position groups are driving the relatively higher late-round values for projection-favored players:

The results are somewhat inconsistent among positively performing players, but when considering the full pool it does appear that relievers are driving the bus. They have the highest (least bad) rankings for any position group across each setting. To find out which relievers are delivering the most impact, I looked at the top performing relievers (again favored by projections) beyond rank 150 in each setting:

This is a pretty good list; each player above shows up in the overall top-10 performing relievers for their respective setting. Further, a few even had positive projected value. This helps us reasonably state (without relying too heavily on hindsight) that they could have been a feasible dart throw late in drafts. So, if you’re spelunking for relievers at the back end of drafts, you might want to look at where the projections are diverging from ADP.

ADP Strengths

It should be evident by now that using projections to rank players in the draft shouldn’t serve as some type of panacea to fantasy success. The best we can say is that, when comparing projections against ADP, it looks like the main benefit is the ability to find low-ranked relievers. This isn’t nothing; if you feel confident in your late-draft relievers, you can probably skip drafting Mason Miller and spend that pick on a hitter or starting pitcher. To touch on another theme from last week’s articles, this is the concept of optionality in action.

And after a closer look, I think leveraging ADP in your rankings could have a few strengths too. Below is another breakdown table, except instead of splitting by position type, the only bucket is for players aged 25 or younger.

Among the top 150 ranks, there is a very limited sample of players younger than 25 who are preferred by projections, and where there are some, ADP is uniformly doing a better job of preferring them. This closely mirrors the pattern for relievers demonstrated earlier. To look at the specific players whom ADP hit on:

No one on this list is especially shocking. For comparison, here is the full list of who projections who projections preferred:

This list is, uh, not quite the same. Based on these data, if the projections are materially higher on a very young player than ADP is, you might want to steer clear.

Conclusion

This was admittedly a long journey, but I think we’ve found a few nuggets that, if not made of gold, are at least made of bronze. Projections seem more apt at identifying relievers at the back end of ranking boards. Meanwhile, at the front end, it doesn’t seem worth trusting young players whom projections materially prefer relative to the ADP market. These are real takeaways that can be used come draft time.

The only thing I regret is that this is all based on just half of a season’s worth of data. Ideally, I’d be able to look at a larger sample to discern whether this pattern holds consistently or if any others worth noting pop up. To that end, though – stay tuned.