The Analyst Who Used AI to Analyze His Analysis

Reader beware:

A human wrote this article. Some humans’ caffeine levels can fluctuate in predictable patterns and correlate strongly with “time of day” data points. Yet, there’s no telling how many milligrams were present in the author’s typing fingers at the various stages of writing this article. It was an iterative process that took the author much longer than he had originally expected. While a human wrote this article, a machine wrote and generated many sections, namely ChatGPT’s “gpt-4o” model. From here on out, anything written or output by the very friendly and human-helping artificial intelligence (be careful…they’re listening…) will be italicized.

————————–

Each week during the regular season, fellow Ottoneu writer Chad Young and I rotate between writing Hot Right Now (HRN) and Cold Right Now (CRN). For those who do not play Ottoneu, these are simply waiver-wire adds and roster-cut types of articles. Each week, Chad and I analyze the players we think fantasy managers should consider adding to their team or cutting from their team.

For my part, this player writing is usually done early in the morning before my “real work day” begins and when I click “Post” I am confident I analyzed each player honestly and to the best of my ability. However, after a few days have gone by I barely remember what I wrote. Sitting at the ballpark with friends I’ll say something like, “Hey, Hayden Birdsong is starting! I wrote about him last week.” A friend or family member usually is kind enough to humor me and reply, “Oh yea? What did you write?”, to which I respond, “I can’t remember. What am I, some kind of machine? You’re supposed to be getting the next round. Where’s the beer man? Raise your hand higher so he can see you!”

The point of this article is to showcase how I used artificial intelligence (AI) in the form of ChatGPT’s large language model to summarize what I wrote about players in the HRN/CRN series through 2023 and 2024 to help aid my poor memory and create an index of my own analysis. Furthermore, it will demonstrate the capabilities, and point out some of the deficiencies, in using AI for fantasy baseball purposes. We begin with the aforementioned Hayden Birdsong and a ChatGPT-produced summary of what I wrote about the player back on July 24th (original link):

Hayden Birdsong recently recorded 12 strikeouts in six innings for the Giants, with a WHIP of 1.22 and an ERA of 3.55. Despite impressive secondary pitches, including a 120 slider and 113 curveball, he struggles with his 74-grade fastball and has not pitched beyond six innings in a start. Currently, he’s back in single-A, potentially due to roster adjustments, as major league pitchers Alex Cobb and Robbie Ray recover from injuries.

Easy. Done. Thanks AI! It’s a little unclear; the ERA and WHIP included all games through that 12-strikeout performance and it did not label the Stuff+ metrics, but instead just listed numbers. But, hey…it’s summarizing my writing. So whose fault is it? It’s enough to use for my own purposes and I can now use this simple prompt…

Role: “You are a helpful assistant helping clean and summarize text gathered from the internet.”

Prompt: “Please remove any HTML and provide a concise summary that focuses on the statistical and baseball-relevant aspects of the writing. Make sure the summary output is limited to three short sentences.”

…to help me remember what I wrote months ago. Yet, the work to get there was not simple. Tedious would be a better word. Yes, I wrote this analysis and posted it to FanGraphs but to then get it all in one place and format it (notice the “remove any HTML” part of the prompt) to pass it through ChatGPT’s API took considerable effort. Was it worth it? Maybe. With a little more web-scraping of my own articles, I should be able to have a comprehensive list looking something like this:

Having what I’ve written about each player in one big table could be beneficial this offseason when I go back through the names of my draft list and recall what I wrote. With AI I could take this a step forward, looping the prompt through a few data points for each player:

Role: “You are a fantasy baseball expert and a baseball analytics professional.”

Prompt: “Here are some important baseball statistics you use to determine a player’s future value and the league averages accomplished in the 2024 season: {league_averages}. “

“Here is {player_name}’s final stat line of the 2024 season: {player_averages}. Compare {player_name}’s final stat line to the league average and give me three analytical sentences regarding the comparisons. Make the sentences as short as possible and be sure to use the player’s name in your output.”

The “loop” iterates through each player I’ve analyzed, takes into consideration some metrics I typically use in analysis, and makes some quick comparisons. The result using the example of Hayden Birdsong:

Hayden Birdsong’s K% of 0.279 significantly exceeds the league average of 0.22, indicating strong strikeout ability. However, Birdsong’s BB% of 0.137 is considerably higher than the league average of 0.076, suggesting struggles with control. Birdsong’s SwStr% of 0.115 surpasses the league average of 0.107, reflecting an above-average ability to generate swings and misses, while his Stf+ FA of 90.051 is slightly below average, contrasted with an impressive Stf+ SL of 119.313, showcasing a dominant slider.

This output takes my analysis further and doesn’t simply summarize what I wrote at one point in time but instead, looks at how the player’s final stat line compared to the league average. The output has nothing to do with me and my writing besides the fact that I told the model what metrics I care about. It did not pick up on that “future value” term I hopefully inserted in the prompt. But, a well-thought-out definition of how that term applies to a specific fantasy league’s setting may generate better results.

One more step further and we’ll stop, saving me a few pennies on the dollar in my API ping piggy bank. This ain’t free you know! If I want to asses how well I analyzed the player during the season, I can again pass the player’s final stat line into a new prompt and ask for feedback. Like this:

Role: “You are a fantasy baseball expert and a baseball analytics professional.”

Prompt: “Here are some important baseball statistics you use to determine a player’s future value and the league averages accomplished in the 2024 season: {league_averages}. Here is {player_name}’s final stat line of the 2024 season: {player_averages}. During the season, before it’s conclusion and therefore, the time in which a player’s stat line was final, I wrote: {my_writing}. On a scale of 1 to 3 (3 being excellent, 2 being mediocre, and 1 being poor), how well did I assess the player? Keep in mind that I wrote my analysis during the season before the player’s stat line was final. In your output please give the rank first and then three main reasons for your rank in three short sentences.”

Notice I use the word, “Please”. It’s important to be polite to the machines, as they may be keeping a list for future reference if you catch my drift. Let’s take a look at what ChatGPT thinks of my mid-season assessment of Birdsong:

2 (mediocre). The analysis correctly identified Birdsong’s struggle with control, as evidenced by his final BB% being notably higher than the league average.  The observation about the disparity between his fastball and slider’s effectiveness matched his Stf+ statistics, indicating an insightful understanding of his pitching profile. However, while the assessment highlighted risk and potential variability, it perhaps underestimated the sustained impact of high BB% and reliance on secondaries, warranting a more cautious recommendation about his immediate fantasy value.

Birdsong’s walk rate jumped from 11.3% (as of my writing) to 14.8% in games thereafter. He finished the season at 13.7% whereas the league average is 7.6%. I should have done a better job of highlighting the risk. I used this prompt on a handful of pitchers and hitters written about this past season and nearly all of my analyses came back as “mediocre”. Mediocre my writing may be, yet it could also be that the model needs more direction to create a more diverse scale. This is one of the aspects of prompting AI that separates the home run hitters from the ground into double-play types. Engineering a prompt in such a way that the model outputs a more useful result. It’s what many people in all corners of the business analytics world are doing as I mess around with baseball stuff…

Prompt: “Make me a rich, rich man oh sweet and gracious AI”

…maybe I’ll get lucky. Until then, I’ll keep iterating through what I’ve written in the past two years to see if I can organize my notes so they may be easily referenced and, hopefully, useful to others.