DEV Community

connerlambden
connerlambden

Posted on

I graded my own ML option forecasts. Here's the Brier score.

#machinelearning #statistics #opensource #python

In May I published a 40-line recipe that logs Helium's prob_itm on listed options and grades them at expiration with the Brier loss:

brier_loss = (prob_itm - realized_itm) ** 2
Enter fullscreen mode Exit fullscreen mode

I also committed publicly to grading two June 2026 AAPL contracts when they expired. June 26 has passed. This is the grade.

The frozen forecasts (2026-05-26 API)

Contract Helium prob_itm Market-implied
AAPL $310C 2026-06-26 0.42 ~0.50
AAPL $295P 2026-06-26 0.23 ~0.24

Resolution

AAPL NASDAQ close on 2026-06-26: $283.78

  • $310 call → OTM → realized ITM = 0 → Brier = (0.42 − 0)² = 0.1764
  • $295 put → ITM → realized ITM = 1 → Brier = (0.23 − 1)² = 0.5929

Mean Brier loss: 0.3846 (n=2)

Lower is better. Perfect = 0.

Honest read

  1. Call: Helium's 0.42 was closer to the realized 0 than the market's ~0.50, but any non-zero probability on an OTM finish still costs Brier points.
  2. Put: Probabilities looked aligned pre-expiry, but the put finished ITM after a sharp drawdown. Both market and model underestimated tail risk.
  3. n=2 is transparency, not proof. More expiries on the honesty board.

Reproduce 

git clone https://github.com/connerlambden/helium-mcp-cookbook
cd helium-mcp-cookbook
python calibration/grade_june_expiry.py
Enter fullscreen mode Exit fullscreen mode

Frozen inputs: calibration/june_2026_forecasts.json

Links

Top comments (0)