DEV Community: myfirstcodeo

I backtested a forecast-based model against Kalshi's temperature markets. The market won. Here's the data.

myfirstcodeo — Wed, 23 Sep 2026 16:33:39 +0000

I built a tool that takes the NWS forecast high for a city, wraps it in that city's measured forecast-error distribution, prices every Kalshi daily high-temperature bracket from it, and compares to the Kalshi price at the close of the day before. Then I backtested it on every settled US high-temp market from Jul 25 to Sep 22, 2026: 24 cities, 1,317 city-days, 7,902 markets.

The short version: the market beat the model. Here are the numbers, and the one thing in the data that was genuinely useful.

Setup

Markets and ground truth: Kalshi's public API. Settlement value is Kalshi's own expiration_value, which is The Weather Company's reading at the station named in each market's rules (for NYC, "New York City (CLINYC)", the Central Park climate station). I cross-checked it against the NWS climate report: it matched on 1,316 of 1,317 city-days. The one miss was Miami on Aug 29, 90 vs 85.
Forecast: Open-Meteo's previous-runs archive, the hourly temperature_2m_previous_day1 series, max over the local day. I tried GFS, ECMWF, and their average; the blend had the lowest pooled sigma so it won.
Market price: the close of Kalshi's daily candle for the last full day before the target date. Mid for scoring, ask for the P&L.
Model: actual high ~ Normal(forecast + city bias, city sigma), bracket probability is the mass inside the bracket with a half-degree continuity correction.

Forecast error by city (actual minus day-1 forecast, F)

city	n days	bias	sigma	MAE	skew	ex. kurt	within 1sd	within 2sd	normal ok?
NYC	60	-1.074	2.078	1.861	-0.35	0.14	0.7	0.983	yes
MIA	60	3.657	1.916	3.667	0.11	-1.18	0.583	0.983	yes
CHI	60	0.26	2.407	1.837	-0.74	1.57	0.733	0.95	yes
AUS	60	2.458	1.455	2.622	-1.29	4.69	0.75	0.95	no
HOU	60	-0.025	2.267	1.75	0.64	1.44	0.717	0.967	yes
DC	60	-0.593	2.546	2.113	-0.02	-0.49	0.7	0.95	yes
SAN	35	-3.354	2.543	3.669	0.51	-0.18	0.657	0.971	yes
NOLA	60	1.159	2.201	1.926	-0.64	2.02	0.717	0.967	no
MIN	60	-0.116	2.976	2.366	-0.38	-0.09	0.617	0.967	yes
SATX	60	0.526	1.592	1.204	-0.39	2.06	0.8	0.933	no
DEN	60	0.624	1.946	1.641	-0.52	0.32	0.733	0.917	yes
LAX	60	-3.961	3.064	4.346	0.19	-0.15	0.717	0.917	yes
BOS	60	0.309	2.446	1.901	-0.71	0.79	0.683	0.983	yes
ATL	60	0.326	2.128	1.647	0.4	-0.5	0.7	0.967	yes
PHL	60	1.901	1.884	2.224	-0.08	-0.56	0.6	0.95	yes
OKC	60	1.263	2.603	2.26	-0.39	2.47	0.85	0.933	no
PHX	60	0.972	1.832	1.608	0.33	0.46	0.717	0.933	yes
LAS	60	0.616	2.232	1.587	-1.4	6.02	0.85	0.933	no
TTN	28	0.546	1.651	1.425	-0.72	-0.18	0.679	0.964	yes
EWR	28	0.873	2.236	1.87	0.17	-0.53	0.643	0.929	yes
SFO	60	-1.095	3.434	2.527	-1.42	2.8	0.767	0.967	no
SEA	60	0.809	2.151	1.912	-0.76	0.76	0.8	0.95	yes
DFW	60	0.45	1.926	1.527	-1.49	4.11	0.733	0.967	no
SDF	26	-0.285	2.465	1.573	-0.76	3.81	0.846	0.923	no
pooled	1317	0.321	2.755	2.136	-0.72	1.57	0.733	0.948	yes

Pooled: bias +0.3F, sigma 2.76F, MAE 2.1F. The station biases are the interesting part. Miami runs 3.7F warmer than forecast; LAX runs 4.0F colder. Those are large, stable, and exactly the kind of thing a normal-distribution model should be able to exploit if the market weren't already pricing them.

Out of sample: fit on the first 30 days, test on the last 30

	Brier (lower is better)
Model (quoted markets)	0.1466
Market mid (quoted markets)	0.1195
Base rate (constant)	0.1541
Model (all markets incl. unquoted)	0.1285

The market wins by 0.027 Brier. It won in every city and in every price band.

Calibration in 10% bins:

bin	model n	model mean p	observed	market n	market mean p	observed
0.0-0.1	1240	0.034	0.079	1407	0.042	0.026
0.1-0.2	614	0.147	0.179	498	0.143	0.131
0.2-0.3	681	0.249	0.266	411	0.244	0.238
0.3-0.4	491	0.345	0.263	347	0.348	0.34
0.4-0.5	142	0.444	0.423	384	0.447	0.427
0.5-0.6	53	0.535	0.358	162	0.545	0.556
0.6-0.7	24	0.641	0.25	63	0.634	0.683
0.7-0.8	18	0.734	0.556	10	0.736	0.7
0.8-0.9	15	0.84	0.533	3	0.847	1.0
0.9-1.0	7	0.945	0.571	0

Where the model said 50 to 70 percent and disagreed with the market, the event happened 25 to 36 percent of the time. The model's lowest bin (0 to 10 percent) resolved yes 7.9 percent of the time against the market's 2.6 percent: fat tails that a normal distribution under-prices.

What happens if you trade it

strategy	trades	wins	staked $	fees $	P&L $	ROI
Buy YES (as specified)	689	74	6850.81	420.8	-2583.61	-0.3771
Buy NO (mirror: model NO prob >= NO ask + 0.08)	787	454	7630.54	215.34	-550.88	-0.0722

Every price band loses. The worst are the sub-10-cent tails, which is where an 8-cent "edge" shows up most often and where the normal assumption is most wrong.

What I take from this

A public forecast plus a normal error model does not beat the day-before Kalshi price. The people trading these markets are already doing at least that, with fresher model runs than a previous-day archive.
The per-station bias table is real information. If you trade weather markets and are not adjusting for the station, that is the edge you are leaving, and it is the one thing the tool computes fresh each day.
Honest caveats: 60 summer days only; the backtest forecast is archived GFS/ECMWF while the live tool uses NWS; the market close has slightly fresher information than the archived run, so the benchmark is a little favored.

The tool

Free, MIT, standard-library Python, no API keys: https://github.com/myfirstcodeo/kalshi-weather-fair-value

python weather_edge.py scans all 24 open series against the live NWS forecast and prints model probability, market price, edge, and a quarter-Kelly size per bracket. python backtest.py reproduces every number above from public data. python test_smoke.py checks the pricing math offline.

The cross-venue tool that does pay for itself is the Kalshi vs Polymarket divergence scanner, a one-time $29 purchase: https://instaverb.gumroad.com/l/pm-scanner

Not financial advice. Read the market rules before you trade anything.

Matching Kalshi and Polymarket contracts is harder than it looks (and how I made it mostly work)

myfirstcodeo — Wed, 23 Sep 2026 15:18:55 +0000

Every cross-venue "arb scanner" for prediction markets does three things: fetch both venues, decide which contracts are the same question, subtract. The first and third are trivial. The second one ate my afternoon.

The data

Both venues expose keyless market data.

Kalshi: GET /trade-api/v2/events?status=open&with_nested_markets=true, cursor-paged, 200 events per page. Two gotchas. Nested markets report status: "active" rather than "open", and they report liquidity_dollars: "0.0000" across the board, so you gate on volume_fp and a two-sided book instead. Also skip everything with an event ticker starting KXMVE: those are multivariate parlay shards, and there are tens of thousands of them.
Polymarket: GET https://gamma-api.polymarket.com/markets?active=true&closed=false&order=volume24hr&ascending=false, 100 per page. Offsets past a couple thousand return HTTP 422, so sort by volume and accept that you get the liquid top of the book. That is where the arbs are anyway.

After filtering: about 28,000 Kalshi contracts with real volume, about 1,600 Polymarket contracts with real liquidity.

Naive matching fails in three specific ways

I started with token Jaccard plus a difflib sequence ratio. The top "opportunities" were:

"Highest temperature in Miami: 92° to 93°" vs "between 90-91°F". Same city, same day, adjacent bracket. A 70-cent "gap."
"Closest Senate race in 2026: Maine" vs "Will the Democrats win the Maine Senate race?" Shares four tokens, different question entirely.
"When will Xi Jinping visit Taiwan? Before 2027" vs "Will Xi Jinping visit US before 2027?" The naive tokenizer dropped "us" as a two-letter word, leaving the two questions nearly identical.

Each one is a class, not a one-off.

Three vetoes fixed most of it

Before any scoring, reject the pair if:

Numbers differ. Extract every number from both raw strings. If either side has any, the sets must be equal. Kills the bracket problem and most date confusion.
A pivot word is on one side only. A short list: closest, margin, first, neither, before, after, not, win, lose, Democrats, Republicans, above, below, and so on. If the symmetric difference of the token sets hits the list, it is a different question.
Places conflict. A list of countries, states, and cities. If both sides mention a place and they differ, reject. This needed a "keep short tokens" exception for us, uk, eu, xi, 10y.

Then require containment (shared / smaller set) of at least 0.6, and score 0.4 Jaccard + 0.3 containment + 0.3 sequence ratio. One-to-one assignment, greedy by score.

What survived

On a mid-September afternoon: 21 pairs above a 3-cent gap and $500 of liquidity. The clean ones were things like Senate "Clarity Act" vote markets per senator (3 to 5 cent gaps), a Trump-speech word market (75c vs 71c), a Warsh press-conference word market. And one liquid pair, whether the US reopens its embassy in Iran before 2027, priced 3c on Kalshi and 16c on Polymarket with $223k behind it.

Also one 80-cent gap on "who leads Venezuela at end of 2026: Maduro". Both books tight, both liquid. That is not free money; that is two venues resolving on different rules, and the scanner's most useful output was flagging it as "VERIFY RULES" instead of ranking it first.

Performance note

The first version segfaulted. Candidate generation was a token inverted index, and tokens like "win" fanned out to tens of thousands of Kalshi markets. Capping index tokens at 5% document frequency fixed it and the whole match runs in seconds.

The lite version of the scanner is free on GitHub: https://github.com/myfirstcodeo/pm-divergence-scanner-lite. The full one with watch mode and Telegram alerts is a one-time purchase at https://instaverb.gumroad.com/l/pm-scanner. Not financial advice. Read both rule sets.