Google released a weather model earlier this year that beats the best forecast in the world 97% of the time. Free. Clean API. Drop-in compatible with what we already run.
Every data engineering instinct I have said: adopt this immediately.
I spent one afternoon testing it instead. We declined to ship it.
What we actually measure
Kalshi temperature contracts settle on one number: the official daily high temperature at a specific airport, reported by NOAA. Not the mean. Not the overnight low. The daily high.
Our entire edge, if we have one, lives or dies on the accuracy of one number, at one station, for one 24-hour window.
The test
I pulled WeatherNext 2 forecasts for 16 Kalshi settlement stations over two weeks and compared against NOAA observed highs. Then against the models we already run.
| Model | Mean Error (°F) |
|---|---|
| GFS | +0.3 |
| ECMWF IFS | -0.2 |
| ICON | -0.4 |
| GEM | -0.1 |
| WeatherNext 2 | -5.2 |
Minus 5.2 degrees Fahrenheit, cold, on the exact temperature type our contracts settle on.
Kalshi temperature contracts span 2-3 degree ranges. A 5.2°F cold bias does not introduce noise. It systematically points to the wrong contract on almost every trade.
Why this happens
WeatherNext 2 is benchmarked primarily on 500hPa geopotential height, wind fields, and mean 2-meter temperature. Daily maximum 2-meter temperature is a different problem. It depends on the planetary boundary layer, local surface energy balance, urban heat island effects, and the specific observing conditions at individual ASOS stations.
WeatherNext was built for global forecast skill. We need local daily-high accuracy.
The nuance
WeatherNext's spread — its estimate of its own uncertainty — is roughly twice as well calibrated as the model we run today. That part is genuinely valuable and stays on the roadmap.
We're not anti-AI. We're anti-assumption.
The lesson
The industry ships the shiny thing and calls it innovation. What almost nobody does is measure the shiny thing against the specific problem they are actually solving, find the gap, and say no.
Anyone can adopt a model. Knowing when not to is the actual skill.
I built this system after 30 years as a data engineer. The forecast is easy. The honesty is hard.
Top comments (0)