How I built a pitch-by-pitch baseball simulator

#analytics #data #datascience

How I Spent 6 Months Building a Pitch-by-Pitch Baseball Simulator

Some lessons about never trusting anything and how beating the books is the only true test

Written by Jesse, NegativeEV. Last updated 28 July 2026.

I think it started 10 years ago. I built my first lineup in FanDuel for $2 and hit submit. Then I lost.

I couldn't let it go. I was so fascinated to find out why I lost. Why didn't I see the obvious flaws in these players that I picked that caused them to play so poorly? Why did others pick better players? It took a long time for me to learn that it was mostly luck and variance at work, but that was only after I saw there was very clearly something there to predict.

My first simulators were very, very crude. I'd get some projections from RotoGrinder and pick out the floor, ceiling, and actual projection. Then I used those to generate a skewed normal distribution for each player, at which point I'd randomly grab a single value from the curve. That was his simulated score for the day. I'd do that for every player on the slate, and I would do it 10,000 times for the entire slate. From there it was a matter of solving for the optimal lineup given the simulated scores.

This produced some interesting results. This was pre-AI. I didn't know what a Monte Carlo simulation was, but that's what I was doing. A friend of mine with a much higher risk tolerance than me ended up using it and winning three separate $100,000 prizes. Crazy, right?

Eventually, of course, the market caught up and those wins turned into ties turned into second and third-place finishes. Others had built something better.

The one thing that always drove me crazy with the crude simulator was the lack of correlation. Each player was "predicted" individually. So a quarterback could pass for 500 yards and each of his receivers could have less than 50. Made no sense. My friend had to a complicated post-process where he removed lineups like this.

At this point it was obvious that the holy grail is a pure play-by-play simulator. But this seemed impossible. It seemed technically impossible and physically impossible: I only had one computer. But I couldn't let it go. It was too good of a challenge, and it became all-engrossing.

I started with baseball. I don't know why. As it turns out, baseball is far and away the hardest sport to predict. But it also has the most data for training. So I banged my head against the wall over and over again, and I built it.

I started with such insane ambition it's amazing I made it to the end. Well, actually I quit, twice. The first time is most memorable and was the most instructive. Remember my goal in building this was academic and, dare I say, ego-driven. So before even building the simulator, I built an extensive back testing suite that compared my simulations against over 60 days of historical odds data.

That was always marker, odds. Anyone can say the built a "realistic" simulator. But the true test of the thing was always going to be if its results were more predictive than the market. And not even that I was going to use to gamble. It's just, that was the standard that, over and above, this thing had actual use.

So on one of my earliest back tests I stared in disbelief at the numbers that printed out on the terminal. No way, I thought. +15% ROI. I think I even got up from my chair and did two fist pumps.

But I'm a skeptical software developer. I'd built sprawling, critical systems for large companies as part of my day job. I settled back into my chair and went to work testing my back tester. Oof.

Data leakage! I couldn't believe it. I had let post-pitch data into my training set. Of course it was going to be profitable. I hung my head in shame. I fixed the issue and stared listlessly at the resultant -22% ROI on the screen after the next test.

I stopped working on it for a couple months. But then I got the itch again.

You have to understand that I'd been going at this with a "no compromises" attitude. I wanted to make the best simulator on earth. That was my directive. Even though I had no access to any real compute (just my personal one), I made all architecture and design decisions as if I had a google-sized swarm. Back tests took hours. The fan from my computer would keep my kids up at night.

One time my simulator showed profit over 10 days of back tests. I did a mini fist pump. I expanded it out to 15 days. Negative again. And so it went. For months, I kid you not.

Okay Part 2 will have to be another post.