Prospect Valuation Follow-Up Pt. 2

Intro

On Tuesday, I followed up my study from last week on how prospect lists translate to fantasy auction value. Last week’s research primarily identified key stratifications in career and peak value based on list placement and whether the player is primarily a pitcher or a hitter. This week, I found that the predictiveness of ranking lists relative to fantasy auction value doesn’t seem to be changing much over time.

But before crossing “prospect lists” off of my dynasty ranking model checklist, I wanted to combine these two analyses to look at how or if the ranking buckets hold when broken into more granular year groupings. This is important to my final rankings because I want whatever bucket groupings I use to be representative enough to capture a reasonable picture of the forward-looking value of a given prospect.

So, how do the buckets shake out? Well, read on.

Pitchers

Earlier in the week, I identified that the highest bucket of pitching ranks in the earliest list years was not especially correlated with overall value. Looking at the corresponding seven-year  value metric I derived in Tuesday’s article, this is confirmed:

With a median value of $0 in all three buckets, the earliest prospect lists were functionally useless with regard to readily identifying pitchers which returned fantasy value. To track down the offenders:

As expected, other than Pedro, this is a pretty underwhelming list. Meanwhile, 10 years later, we see a relative golden age for pitching prospects within the top bucket:

This is a superb list. There are two first-ballot Hall-of-Famers and a number of additional guys who have a reasonable case. Even in the lower end of the bucket, we see players who carved out decent careers as relievers.

Other than those two windows, though, the performance of pitchers within the top bucket has been very consistent. Based on how clearly these two buckets of time mirror each other, I’m inclined to believe they represent random talent spikes rather than any substantive statement about the lists themselves.

Calling back to the year sample split measured by career value, the break is not quite as clean:

This is intuitive though; in the dual split, the earlier bucket is dominated by the poorer performing 1990-1994 segment. Likewise for the earlier bucket in the three-way split. On the flip side, the outstanding run of pitching prospects from the late aughts is driving the high median values for the later year groupings.

With this in mind, I looked at whether the rank buckets measuring career value maintained statistical significance from each other after additionally breaking them down by year:

Remember, the above tests depicts whether or not there is at least one statistically significant difference between the median career value in the buckets for each year split. The tests below verify that each bucket is statistically distinct from one another.

The above two images show the results within the dual (1990 to 2001 vs. 2002 to 2013) year sample split, while the three below show the results for the triple year sample splits.

It turns out, despite the relative lack of differentiation in the earlier year buckets, they did. Given on this constellation of factors, I feel pretty good about treating pitchers based on these buckets in the dynasty model. There is still the specter of ever-changing usage patterns looming over the most recent years of these data, but I think this is mitigated somewhat by the fact that fantasy auction value each year is scaled to that year’s environment.

In other words, while it might be true that the best starting pitching prospects these days are only barely expected to qualify for the ERA title, if everyone is in that range of innings, the best players will still return the highest auction values. So, to the extent that the prospect lists are adequately identifying pitcher skill (we have no reason to believe they aren’t), we should expect that skill to likewise translate somewhat smoothly into fantasy value. This is, of course, something to closely monitor over time as more data accrue. But for now, I think that this works.

Hitters

Hitters, meanwhile, saw relatively stabler correlations over time. This tracks with what one might assume. But we all know what assuming can do, so I took a look at how the buckets behaved:

Well, well, well! This is significantly muddier. For one, there doesn’t seem to be meaningful distinction in the value of the top two buckets from 1995-2004. For another, the 11 to 25 bucket returned roughly three times as much value as the three to 10 bucket in the 2005-2009 sample. Likewise, there is not much separating rank three through 50 in 2010-2014, and for many of the years, there doesn’t seem to be much differentiation in the bottom two buckets. This is overall consistent with the career value measure:

What to do? I think this is reason enough to look into different hitter buckets. For the purposes of this exercise, I don’t think you necessarily need statistically significant breaks along every single axis; if you cut the data enough ways eventually your sample sizes will get small enough that you will inevitably manufacture problems. But given how cleanly the pitcher buckets hold over time I wanted to see if there was a way to split the hitters that stayed distinct without relying on 24 years of rankings data. I tried a few and the grouping below seemed the most promising based on median career value:

I liked this one for a number of reasons. First, choosing three buckets is symmetrical with the pitcher dataset. To hearken back to the credo I espoused in my debut article, I think parsimony is important. If you can achieve a similar, conceptually valid modeling outcome with fewer steps, you should. In addition, the values are fairly consistent. This again is similar to the story with pitchers. I will acknowledge that using the adjusted seven-year value metric, the picture is not quite as rosy:

This is worth keeping an eye on, but isn’t something I think sinks the ship. For one, the sample sizes in the earliest bucket are naturally much smaller than those in the pitcher version of this chart. For another, the second bucket, with larger sample sizes, varies much more reasonably around the mean value. The latest bucket is clearly consistent overall.

With this in mind, I ran the statistical tests on the rank buckets for career value split out over different year groupings:

Consider the first bar passed. Now, to see if the individual buckets within these groupings are statistically distinct:

As with their pitcher counterparts, the above two images show the results within the dual year sample split, while the three below show the results for the triple split.

All but one pairing (Top 3 vs. 4 to 30 for 1990 to 1997) tests out. That p-value too is a whisper away from marginal significance (p=0.107). I think this is a problem with sample size – the median values shown above are very distinct, but there are only 15 hitters who ranked in the top three for those eight years. To me, these buckets pass the smell test.

Conclusion

So there you have it. Prospect rankings (at least, Baseball America’s) have generally been consistent in their ability to predict fantasy success over time; in addition, poking around the data more revealed that the typical dollar value of certain bucket rankings has likewise been fairly consistent, subject to some reasonable variation.

This exercise also proved valuable in illuminating that the pitcher buckets I unearthed last week were reliable over time, while the hitters weren’t and required a bit of reconfiguring. Pitchers, durable? Who would have thought.

Jokes aside, the end result is, in my opinion, a more robust treatment of prospect data for the dynasty model than what I had previously conducted. Stay tuned for next week when I dive into the next piece of the puzzle: aging curves.