I had been meaning to do this analysis for a while now, but since Lucas Giolito was coming off a no-hitter, I thought that there was never a better time to test out the Pitch Predictor! If you’re unfamiliar, check out a previous post where I detail the project/API. As a quick refresher, I built a model that took an in-game situation as the inputs and output the predicted pitch type for the next pitch!
For this exercise, Lucas Giolito was used and the model was built on his 2019 pitch log. According to FanGraphs, Giolito threw his Fastball 55% of the time, the Changeup 26% of the time, the Slider 15%, and the Curveball 4% of the time. With the inputs, the model was able to correctly predict Lucas Giolito’s next pitch at an accuracy of 58%! 58% on four pitch types is better than randomly guessing, but there is an argument to be had if it would outperform a Major League Baseball player’s ability to guess the next pitch in a sequence – which we will get to later in this post.
With my Pitch Predictor API, ESPN Gamecast (for confirming pitch types), and Green Man Brewing Trickster IPA, I was all set to test out my model to the real 2020 version of Giolito! Here is how the model performed by inning (accuracy is # of correctly predicted pitches/# of pitches thrown):
* 0% on 7 pitches
The overall accuracy of the model was 41%, so why did it do such a poor job at predicting Giolito on Monday night’s game? Long story short, it’s could be a variety of factors! Right off the bat (heh), the pitch usage differed from his 2019 and 2020 usage rates. He threw the FB about 6% less than his 2020 averages, he threw the Changeup 4% more than his 2020 averages, and the Slider 3% more than his 2020 averages. A model that is built off higher Fastball usage and less Changeups and Sliders will not perform as strongly when the real life version of Giolito mixes up his pitch choices (which is a weakness of the model). Watching last night’s game, it was easy to see the difference in strategy that Giolito took. In the 1st and 5th innings, Giolito threw four consecutive Sliders to Nelson Cruz – which is something the model did not predict.
While we are still talking about the performance of the model on last night’s game, it’s important to understand how to evaluate a model’s performance. Accuracy is a popular evaluation metric, but is not necessarily the right metric to look at. Sure, the model performed at 41% last night, but how did it do correctly predicting a Fastball when Giolito threw a Fastball, or a Changeup when he threw a Changeup, or a Slider when he threw a Slider? This is known as Sensitivity – which is basically the model’s ability to correctly predict a class (pitch type) when that class actually occurs. A Fastball was predicted 83% of the time it was actually thrown, the Changeup was predicted 23% of the time it was actually thrown, and the Slider was predicted 20% of the time it was actually thrown. This means that the model was very strong when it came to predicting Fastballs and quite poor when predicting Changeups and Sliders.
This exercise proved to be a fun test of my model to a real world situation. The results were not surprising – the model was primarily built off of pitching log data and did not include much about the batters in the at-bat. If I were to go back and tweak the model, I would want to include data that took into account the batter (perhaps their stats at the time of the at-bat, or historical stats when applicable). Perhaps this would capture Giolito’s approach to pitching to Nelson Cruz!
To wrap this up, I still think this could be a valuable tool. Do I think it would outperform Nelson Cruz’s, or another MLB hitter, ability to predict the next pitch in an at-bat sequence? No, not necessarily. I do think it would be able to catch some nuances that a player simply cannot remember since they are cramming so much game prep information into their minds before and during games. With all of this said, I’d like to know if you would use the model or would you trust your own scouting abilities? Leave a message/reply and maybe we can put your skills to the test against the Pitch Predictor!
Comments