In May I promised — out loud, on the internet, where people can screenshot you — that I'd grade Helium's published prob_itm forecasts when the June 2026 AAPL contracts expired.
June 26 came. The contracts died. So I graded them.
This is not a victory lap. Mean Brier loss on n=2 is 0.3846. A coin flip that always says 0.5 scores 0.25 when the world resolves to 0 or 1. We did worse than a coin flip. That is information.
What we froze in May
| Contract | Helium prob_itm
|
Market-ish implied |
|---|---|---|
| AAPL $310C 2026-06-26 | 0.42 | ~0.50 |
| AAPL $295P 2026-06-26 | 0.23 | ~0.24 |
What the stock actually did
AAPL NASDAQ close on 2026-06-26: $283.78
- $310 call → OTM → realized ITM = 0 → Brier = (0.42 − 0)² = 0.1764
- $295 put → ITM → realized ITM = 1 → Brier = (0.23 − 1)² = 0.5929
Honest read (short)
- The call forecast was closer to reality than the market's ~0.50 shrug — but "closer" is not "good." Any non-zero probability on an OTM finish still costs you.
- The put is the bruise. Probabilities looked aligned pre-expiry; then the underlying fell hard enough that both market and model had understated the ITM risk. Helium's number was far from 1, so the Brier loss is ugly.
- n=2. This is a promise kept, not a calibration paper. If you want to treat it as science, wait for more expiries on the honesty board.
Reproduce
git clone https://github.com/connerlambden/helium-mcp-cookbook
cd helium-mcp-cookbook
python calibration/grade_june_expiry.py
Frozen inputs live in calibration/june_2026_forecasts.json.
Why bother publishing a bad score
Because the alternative is the usual ML options blog post: a backtest, a vibe, a screenshot of a green P&L. Publishing prob_itm with a fixed expiry is a tiny discipline. Grading it after the fact is the second half of that discipline.
If you log your own forecasts with recipe 02, you can run the same scorer when your contracts expire. I'll keep adding rows.
—
Also adjacent, if you're here for news rather than options: we published a free 212×37 news-outlet framing corpus with an explorer. Different animal. Same "put numbers where people can grade them" instinct.
Top comments (1)
Author note, since HN killed the Show HN link and that's fair when karma is 1: the honesty board is the durable URL — connerlambden.github.io/helium-new... — and I'll keep adding rows as contracts expire. If you catch a math error in the Brier writeup, leave it here; I'll fix the scorecard.