Prospect Valuation Follow-Up

Last week, I studied how well Baseball America’s prospect rankings translated into eventual fantasy performance.

The results were fairly conclusive; after adjusting to floor negative-value seasons, hitters performed better than pitchers, and there was clearer stratification in hitter value across different ranking buckets than their counterparts.

That analysis included a significant amount of data going back to BA’s original lists in 1990, which has its pluses and minuses. On the plus side, the more the merrier. Sample sizes are bigger and we can more confidently assert statistical significance with the right tests. But on the other hand, 1990 was a long time ago (a phrase typed with millennial tears hitting the keyboard). Macro trends in prospect evaluation and baseball writ large have changed immensely.

Surely these factors would influence how prospect lists predict fantasy value, right?

…

Right?

TL;DR: It seems not.

One caveat I will note before proceeding further: I acknowledge that fantasy performance is explicitly not a goal associated with the construction of prospect rankings. The claim I’m making here is a narrow one; if I assert that prospect lists aren’t improving over time, this is emphatically not intended as a criticism of the lists. It could very well be that the lists do indeed improve in what they’re trying to measure, i.e. real world player value. Given asymmetry involved (namely, defensive value) between fantasy and real world production, it would be irresponsible for me to assert anything regarding their performance in any respect other than the one I’m specifically discussing here.

With that said, the first way I looked into this borrowed the years used in last week’s analysis (1990-2013) to see whether rankings correlated with eventual full career fantasy value.

First off, all the correlations are directionally intuitive, which is good. This tells us that there is some link overall between rankings and eventual performance. We knew this based on looking at the buckets from last week’s analysis, but it’s good to have it otherwise confirmed. Second, though, the correlations are not especially strong. This makes sense given the uncertainty involved with prospect prognostication.

I split the data in two ways, once in half and once by thirds. The only meaningful takeaway to my eye is that the earliest partition of pitchers performed particularly poorly. Hitters in the middle third of the sample were likewise depressed, though to a lesser extent. Overall, though, this reads to me that predictiveness of a prospect rank is fairly steady over time.

A lot has happened since 2013 though, especially with regard to pitcher usage. I had to change my target measure because taking career value will undoubtedly bias the results against players in later lists who have had fewer seasons to accrue fantasy value. To that end, I extended the analysis through 2019 lists and changed the target measure from career non-negative fantasy to non-negative fantasy value across their first seven seasons post-list appearance. This measure isn’t perfect. For example, for a player who appears on multiple lists, the period of time used to calculate those seven years of value is different. This could bias the performance of lower ranks, because a player who starts low and climbs the ranks will have more years of MLB playing time to count toward their value calculation associated with their higher ranks.

But, in the spirit of my methodological choice to include all rankings for a prospect in any year (not just their best or latest one), I believe this is the best way to account for a truncated period over which to accrue value in the latest bucket. The other best method I can think of would be to take a player’s first n number of non-negative seasons, instead of their career. While this would help, it could still introduce truncation bias to the extent that players in the latest bucket would still have fewer chances to accrue value within whatever n number of seasons is chosen. This is particularly toxic to an analysis which aims to ascertain how the rankings are performing over time. You could balance that by taking a small n hoping you’ll catch a meaningful sample in your gap between the latest list you choose and the latest MLB season. You can’t go too low though because you’ll run into a situation where there are relatively few positive seasons to count at all.

With these caveats in mind, the method I chose may depress the correlations overall but should not specifically impact any temporal bucket. Anyway – here are the results:

These are quite similar to the results noted above, down to the lower pitcher correlations in the earliest sample. I broke it down even further by half-decade:

Still nothing to write home about, other than that we seem to have isolated the problem area for pitchers: 1990 to 1994. This is useful information which I will discuss in a follow-up article. There is some more variation than we were seeing before, but this is to be expected the further we slice up the data. In addition, 2015-19 is performing somewhat worse relative to the overall history, which flies against the notion that these lists should be improving over time.

Now, the above correlations only look at the relationship between prospect rank and dollar value, which can be influenced by the quality of players in any given prospect cycle. Better prospects will most likely generate more dollars; not every year is going to have A-Rod or Mike Trout come up and be generationally awesome right away. To this end, I also ranked players in each list by their eventual values based on the two measures described above and conducted correlations of those with their prospect rank, which tells us how well the lists predict the rank order of eventual performers, not the dollar amount. The results were broadly similar – positive and weak (roughly 0.2 to 0.3 coefficients) with the lowest correlations for pitchers in early lists.

If you at home have any suggestions for how else to measure the overall performance of these ranks as it relates to eventual fantasy value, I’m all ears. But at this point, to this analyst, it seems that performance has stayed pretty steady over time. As it pertains to the eventual use in any dynasty rankings, this is good! It means that more data can be leveraged in the sample without as much fear that shifts in prospect evaluation over time are affecting the rankings in aggregate. But there are still some nuggets to unearth by digging a little deeper, which I promise to do in a follow-up article.