I Audited My Trading Bot Using a Brier Score. It Had No Skill.

I ran a weather trading bot on Kalshi for four months. 112 settled trades. Then I tore it apart with a proper scoring rule. The result was worse than doing nothing.

A Brier score of 0.2858 against a base-rate baseline of 0.2439. My model was subtracting information from reality.

What a Brier score tells you that win rate does not

Win rate said the model was directionally right about 60 percent of the time. That sounds fine. But a Brier score decomposes into calibration and refinement. It showed the model's probabilities were spread 2.1 to 4.0 times too narrow.

In plain terms: the bot was confident when it should have been uncertain. And in prediction markets, confidence sizes your positions. A model that is too sure trades too big. Direction can be right 60 percent of the time and you can still lose money.

The second mechanism was a systematic temperature bias at the gridpoint level, peaking around seven degrees Fahrenheit. The model leaned warm in a way that was not in the data. A larger position on a biased forecast destroys whatever edge the directional signal might have provided.

The fix was not a cleverer model

NOAA's National Blend of Models already publishes calibrated, bias-corrected, station-level probabilistic temperature guidance for exactly the stations Kalshi settles on. For free. I was hand-rolling a worse version of a public good.

The bot was counting ensemble members across raw forecast systems. The NBM blends dozens of models and applies statistical post-processing no individual system can match. The uncertainty ranges come baked in. The bias correction runs at individual station level.

I spent months tuning a member-counting approach when the right answer was published by a government agency I could have downloaded at any time.

The test suite that caught everything the old one missed

The rebuild ships with 426 tests across 33 files. The old suite was smaller and missed a lot.

The scanner had been looking for Kalshi category "Climate" when the real value was "Climate and Weather." It matched nothing. A fallback scanned the raw catalogue, and every log line looked normal while the bot saw 39 to 63 of 291 available series. The count changed between runs, so entire cities blinked in and out for four months.

Chicago settles on Midway, not O'Hare. Houston on Hobby, not Bush. Every station code is now pulled from Kalshi's own metadata and pinned in a test.

Three money bugs shared the same shape: the daily-loss kill switch reads settled P&L, so a mis-recorded trade silently moved the limit. Partial fills never settled. Weather P&L was booked gross of fees. The econ bot never recorded a position it closed. All three are fixed, and the fixes touched the ledger.

Why this matters if you are building a model

Brier-score your model before you trade it. The baseline is always predicting the historical base rate. If your model scores worse than that, it is subtracting information and you would be better off flat. Directional accuracy is not enough. Confidence calibration is what determines position sizing, and position sizing determines whether you survive.

The rebuild is on NOAA's NBM now. Same-day markets are skipped because NBM coverage starts at forecast hour 24. Rather than fall back to a model that lost money, the bot declines. That costs candidates. The most important thing a trading system can do is refuse to act when the conditions it trusts are not present.

Zero settled trades under the new model. The last trade settled August 7. The new version shipped August 9. Forecast verification needs about two weeks. Strategy evaluation needs 100 or more trades, which is four to six months. I will publish what I find.

---

The full audit, including the bug stories and the NBM rebuild, is at stfarm.substack.com.