I spent four months tuning a custom weather ensemble. It was worse than guessing. The fix was not a better ensemble. It was admitting someone already built the right thing and giving it away for free.
What I built and why it failed
The original weather bot counted forecast members. It pulled raw output from four systems: GFS, AIGEFS, ECMWF IFS, and AIFS. Up to 164 individual simulations per contract. The logic was simple. If at least three of four systems agreed on direction, the bot traded. If they disagreed, it sat out.
That sounds reasonable. It was not.
I ran 112 settled trades through the system and scored the model with a Brier score. The model scored 0.2858. Predicting the historical base rate, with no model at all, scores 0.2439. Lower is better. My model was worse than making no prediction.
The problem was not direction. Direction was right about 60 percent of the time. The problem was confidence. The model spread its probabilities 2.1 to 4.0 times too narrow. It was certain when it should have been uncertain. In prediction markets, confidence sizes your bets. A model that is too confident trades too big on the wrong calls. The confident wrong calls cost more than the confident right ones made.
There was also a systematic temperature bias at the gridpoint level, peaking around seven degrees Fahrenheit. The model leaned warm in a way that was not in the data. It was in the model.
What I should have done first
Before building anything, I should have checked whether the thing I was building already existed in better form.
NOAA publishes the National Blend of Models. It blends dozens of forecast systems and applies statistical post-processing no individual model can match. It produces calibrated, bias-corrected, station-level probabilistic temperature guidance. For exactly the stations Kalshi settles on. For free.
The NBM already does what I was trying to do by hand. It corrects the biases I was measuring. It produces uncertainty ranges I was approximating with member counts. It runs at individual station level, not gridpoint level, which eliminates the seven-degree bias problem entirely.
I spent months tuning a member-counting approach when the right answer was a public product I could have downloaded at any time.
Why member counting was the mistake
Counting ensemble members is seductive. It feels like rigor. You are comparing 164 independent forecasts. That has to be better than one.
It is not. The members are not independent in the way that matters. They share systematic biases. If GFS runs warm, all 31 GFS members run warm. If ECMWF has a marine layer problem at LAX, all 51 ECMWF members have the same problem. Agreement across members of the same model tells you the model is internally consistent. It does not tell you the model is right.
The NBM solves this at the source. It does not count members. It uses statistical post-processing to calibrate each model against observed outcomes at each station. The output is not a vote. It is a calibrated probability distribution with honest uncertainty ranges.
The difference between "164 forecasts agree" and "one calibrated probability distribution" is the difference between confidence and calibration. My ensemble was confident. The NBM is calibrated. In prediction markets, calibrated is what makes money. Confident is what loses it.
What the rebuild looks like
The v2.5 weather bot runs on NBM as the primary source. NBM weight is 0.75. Five raw models sit behind it at a combined 0.25 weight as a disagreement check: GFS, AIGEFS, ECMWF IFS, AIFS, and HRRR. They used to be the model. Now they are a sanity check on the model.
The bot scans 291 weather series across 20 cities. The old scanner asked for category "Climate" when the real value was "Climate and Weather." It matched nothing. A fallback scanned the raw catalogue and every log line looked normal while the bot saw 39 to 63 of 291 series. That is fixed.
NBM coverage begins at forecast hour 24. Same-day markets are inside that window. Rather than fall back to the old model that lost money, the bot skips same-day markets entirely. It does not price what it cannot price.
This costs candidates. A trade that does not get opened is a trade that does not generate a return. That is the point. The most important thing a trading system can do is refuse to act when the conditions it trusts are not present.
Chicago settles on Midway, not O'Hare. Houston on Hobby, not Bush. San Antonio's ticker reads SATX but the climate product is issued as SAT. Every station is pulled from Kalshi's own metadata and pinned in a test suite that runs 426 tests across 33 files.
Where this stands
The weather bot is rebuilt and undergoing validation. There are zero settled trades under the v2.5 model. The last trade under the old model settled August 7. v2.5 shipped August 9. Nothing has been measured.
Forecast quality can be assessed in roughly two weeks once daily verification runs accumulate. Whether the strategy makes money needs 100 or more completed trades, which is four to six months at current rates.
I am not going to tell you the new model performs because I do not know. I will publish what I find when I find it.
The lesson that transfers
The lesson is not about weather models. It is about checking whether the thing you are building already exists in more rigorous form before you build it.
I assumed the right approach was more data, more models, more members. The right approach was less data, better processed, by an organization that has been doing this longer than I have. NOAA's National Blend of Models is not a secret. It is published. It is free. It is better than what I built.
The instinct that cost me four months was the builder's instinct. I wanted to build it myself. I wanted to understand it from the ground up. That instinct is valuable. It is also expensive when the public good already exists.
The fix was not a cleverer model. It was the humility to use one that was already done.
Predict & Profit is a $75 one-time source-code package. Two Kalshi bots, a dashboard, and the full Python source. Discount code HOME15. The public results ledger is at predictandprofit.io/results. This is software, not financial advice. Prediction-market trading involves real risk of loss.
Top comments (1)
This is the lesson that shows up in a lot of AI systems: the model was not the bottleneck, the feedback loop was. Better data boundaries, clearer labels, and a simpler evaluation path often beat swapping in a larger model.