DEV Community: Steve Farmer

The Fix Was Not a Cleverer Model

Steve Farmer — Thu, 13 Aug 2026 00:01:00 +0000

I spent four months tuning a custom weather ensemble. It was worse than guessing. The fix was not a better ensemble. It was admitting someone already built the right thing and giving it away for free.

What I built and why it failed

The original weather bot counted forecast members. It pulled raw output from four systems: GFS, AIGEFS, ECMWF IFS, and AIFS. Up to 164 individual simulations per contract. The logic was simple. If at least three of four systems agreed on direction, the bot traded. If they disagreed, it sat out.

That sounds reasonable. It was not.

I ran 112 settled trades through the system and scored the model with a Brier score. The model scored 0.2858. Predicting the historical base rate, with no model at all, scores 0.2439. Lower is better. My model was worse than making no prediction.

The problem was not direction. Direction was right about 60 percent of the time. The problem was confidence. The model spread its probabilities 2.1 to 4.0 times too narrow. It was certain when it should have been uncertain. In prediction markets, confidence sizes your bets. A model that is too confident trades too big on the wrong calls. The confident wrong calls cost more than the confident right ones made.

There was also a systematic temperature bias at the gridpoint level, peaking around seven degrees Fahrenheit. The model leaned warm in a way that was not in the data. It was in the model.

What I should have done first

Before building anything, I should have checked whether the thing I was building already existed in better form.

NOAA publishes the National Blend of Models. It blends dozens of forecast systems and applies statistical post-processing no individual model can match. It produces calibrated, bias-corrected, station-level probabilistic temperature guidance. For exactly the stations Kalshi settles on. For free.

The NBM already does what I was trying to do by hand. It corrects the biases I was measuring. It produces uncertainty ranges I was approximating with member counts. It runs at individual station level, not gridpoint level, which eliminates the seven-degree bias problem entirely.

I spent months tuning a member-counting approach when the right answer was a public product I could have downloaded at any time.

Why member counting was the mistake

Counting ensemble members is seductive. It feels like rigor. You are comparing 164 independent forecasts. That has to be better than one.

It is not. The members are not independent in the way that matters. They share systematic biases. If GFS runs warm, all 31 GFS members run warm. If ECMWF has a marine layer problem at LAX, all 51 ECMWF members have the same problem. Agreement across members of the same model tells you the model is internally consistent. It does not tell you the model is right.

The NBM solves this at the source. It does not count members. It uses statistical post-processing to calibrate each model against observed outcomes at each station. The output is not a vote. It is a calibrated probability distribution with honest uncertainty ranges.

The difference between "164 forecasts agree" and "one calibrated probability distribution" is the difference between confidence and calibration. My ensemble was confident. The NBM is calibrated. In prediction markets, calibrated is what makes money. Confident is what loses it.

What the rebuild looks like

The v2.5 weather bot runs on NBM as the primary source. NBM weight is 0.75. Five raw models sit behind it at a combined 0.25 weight as a disagreement check: GFS, AIGEFS, ECMWF IFS, AIFS, and HRRR. They used to be the model. Now they are a sanity check on the model.

The bot scans 291 weather series across 20 cities. The old scanner asked for category "Climate" when the real value was "Climate and Weather." It matched nothing. A fallback scanned the raw catalogue and every log line looked normal while the bot saw 39 to 63 of 291 series. That is fixed.

NBM coverage begins at forecast hour 24. Same-day markets are inside that window. Rather than fall back to the old model that lost money, the bot skips same-day markets entirely. It does not price what it cannot price.

This costs candidates. A trade that does not get opened is a trade that does not generate a return. That is the point. The most important thing a trading system can do is refuse to act when the conditions it trusts are not present.

Chicago settles on Midway, not O'Hare. Houston on Hobby, not Bush. San Antonio's ticker reads SATX but the climate product is issued as SAT. Every station is pulled from Kalshi's own metadata and pinned in a test suite that runs 426 tests across 33 files.

Where this stands

The weather bot is rebuilt and undergoing validation. There are zero settled trades under the v2.5 model. The last trade under the old model settled August 7. v2.5 shipped August 9. Nothing has been measured.

Forecast quality can be assessed in roughly two weeks once daily verification runs accumulate. Whether the strategy makes money needs 100 or more completed trades, which is four to six months at current rates.

I am not going to tell you the new model performs because I do not know. I will publish what I find when I find it.

The lesson that transfers

The lesson is not about weather models. It is about checking whether the thing you are building already exists in more rigorous form before you build it.

I assumed the right approach was more data, more models, more members. The right approach was less data, better processed, by an organization that has been doing this longer than I have. NOAA's National Blend of Models is not a secret. It is published. It is free. It is better than what I built.

The instinct that cost me four months was the builder's instinct. I wanted to build it myself. I wanted to understand it from the ground up. That instinct is valuable. It is also expensive when the public good already exists.

The fix was not a cleverer model. It was the humility to use one that was already done.

Predict & Profit is a $75 one-time source-code package. Two Kalshi bots, a dashboard, and the full Python source. Discount code HOME15. The public results ledger is at predictandprofit.io/results. This is software, not financial advice. Prediction-market trading involves real risk of loss.

I Audited My Trading Bot Using a Brier Score. It Had No Skill.

Steve Farmer — Wed, 12 Aug 2026 00:26:08 +0000

I Audited My Trading Bot Using a Brier Score. It Had No Skill.

I ran a weather trading bot on Kalshi for four months. 112 settled trades. Then I tore it apart with a proper scoring rule. The result was worse than doing nothing.

A Brier score of 0.2858 against a base-rate baseline of 0.2439. My model was subtracting information from reality.

What a Brier score tells you that win rate does not

Win rate said the model was directionally right about 60 percent of the time. That sounds fine. But a Brier score decomposes into calibration and refinement. It showed the model's probabilities were spread 2.1 to 4.0 times too narrow.

In plain terms: the bot was confident when it should have been uncertain. And in prediction markets, confidence sizes your positions. A model that is too sure trades too big. Direction can be right 60 percent of the time and you can still lose money.

The second mechanism was a systematic temperature bias at the gridpoint level, peaking around seven degrees Fahrenheit. The model leaned warm in a way that was not in the data. A larger position on a biased forecast destroys whatever edge the directional signal might have provided.

The fix was not a cleverer model

NOAA's National Blend of Models already publishes calibrated, bias-corrected, station-level probabilistic temperature guidance for exactly the stations Kalshi settles on. For free. I was hand-rolling a worse version of a public good.

The bot was counting ensemble members across raw forecast systems. The NBM blends dozens of models and applies statistical post-processing no individual system can match. The uncertainty ranges come baked in. The bias correction runs at individual station level.

I spent months tuning a member-counting approach when the right answer was published by a government agency I could have downloaded at any time.

The test suite that caught everything the old one missed

The rebuild ships with 426 tests across 33 files. The old suite was smaller and missed a lot.

The scanner had been looking for Kalshi category "Climate" when the real value was "Climate and Weather." It matched nothing. A fallback scanned the raw catalogue, and every log line looked normal while the bot saw 39 to 63 of 291 available series. The count changed between runs, so entire cities blinked in and out for four months.

Chicago settles on Midway, not O'Hare. Houston on Hobby, not Bush. Every station code is now pulled from Kalshi's own metadata and pinned in a test.

Three money bugs shared the same shape: the daily-loss kill switch reads settled P&L, so a mis-recorded trade silently moved the limit. Partial fills never settled. Weather P&L was booked gross of fees. The econ bot never recorded a position it closed. All three are fixed, and the fixes touched the ledger.

Why this matters if you are building a model

Brier-score your model before you trade it. The baseline is always predicting the historical base rate. If your model scores worse than that, it is subtracting information and you would be better off flat. Directional accuracy is not enough. Confidence calibration is what determines position sizing, and position sizing determines whether you survive.

The rebuild is on NOAA's NBM now. Same-day markets are skipped because NBM coverage starts at forecast hour 24. Rather than fall back to a model that lost money, the bot declines. That costs candidates. The most important thing a trading system can do is refuse to act when the conditions it trusts are not present.

Zero settled trades under the new model. The last trade settled August 7. The new version shipped August 9. Forecast verification needs about two weeks. Strategy evaluation needs 100 or more trades, which is four to six months. I will publish what I find.

The full audit, including the bug stories and the NBM rebuild, is at stfarm.substack.com.

We Turned Down Google's AI Weather Model

Steve Farmer — Fri, 07 Aug 2026 02:01:44 +0000

Google released a weather model earlier this year that beats the best forecast in the world 97% of the time. Free. Clean API. Drop-in compatible with what we already run.

Every data engineering instinct I have said: adopt this immediately.

I spent one afternoon testing it instead. We declined to ship it.

What we actually measure

Kalshi temperature contracts settle on one number: the official daily high temperature at a specific airport, reported by NOAA. Not the mean. Not the overnight low. The daily high.

Our entire edge, if we have one, lives or dies on the accuracy of one number, at one station, for one 24-hour window.

The test

I pulled WeatherNext 2 forecasts for 16 Kalshi settlement stations over two weeks and compared against NOAA observed highs. Then against the models we already run.

Model	Mean Error (°F)
GFS	+0.3
ECMWF IFS	-0.2
ICON	-0.4
GEM	-0.1
WeatherNext 2	-5.2

Minus 5.2 degrees Fahrenheit, cold, on the exact temperature type our contracts settle on.

Kalshi temperature contracts span 2-3 degree ranges. A 5.2°F cold bias does not introduce noise. It systematically points to the wrong contract on almost every trade.

Why this happens

WeatherNext 2 is benchmarked primarily on 500hPa geopotential height, wind fields, and mean 2-meter temperature. Daily maximum 2-meter temperature is a different problem. It depends on the planetary boundary layer, local surface energy balance, urban heat island effects, and the specific observing conditions at individual ASOS stations.

WeatherNext was built for global forecast skill. We need local daily-high accuracy.

The nuance

WeatherNext's spread — its estimate of its own uncertainty — is roughly twice as well calibrated as the model we run today. That part is genuinely valuable and stays on the roadmap.

We're not anti-AI. We're anti-assumption.

The lesson

The industry ships the shiny thing and calls it innovation. What almost nobody does is measure the shiny thing against the specific problem they are actually solving, find the gap, and say no.

Anyone can adopt a model. Knowing when not to is the actual skill.

I built this system after 30 years as a data engineer. The forecast is easy. The honesty is hard.

Most weather traders check one app. My bot reads 164 forecasts.

Steve Farmer — Mon, 01 Jun 2026 01:06:46 +0000

I am a data engineer 30 years in and I trade weather contracts on Kalshi as a hobby. For a while I did it the way everyone does. Check a forecast app pick a side and hope. Then it hit me that the market price is just that same single forecast everyone else looked at.

When Kalshi prices a temperature contract it is pricing whatever the average trader believes after glancing at one app. A weather ensemble does NOT work that way. The GFS runs 31 simulations from slightly different starting conditions. ECMWF runs 51. The newer AI ensembles NOAA AIGEFS and ECMWF AIFS add more. My bot pulls all four at once up to 164 independent forecasts for the same city and day.

Then it counts. If 118 of 164 members clear the threshold that is a 72% model read. If the market is at 40 cents that gap is the trade. It only fires when at least 3 of the 4 forecast families agree. Most scans produce zero trades. That is the design working NOT a bug.

It all runs on my RackNerd VPS on Ubuntu, logs every decision to SQLite, and the data sources are free.

What's everyone else here building? I'd like to hear what bots you run and where they fight you the most. For me the boring parts, fills and accounting, have been way more trouble than the model ever was.