DEV Community

Cover image for How to Tell Whether Your Forecasting Model Beats the Market: A Python Walkthrough
NeuPortal
NeuPortal

Posted on Originally published at neuportal.ai

How to Tell Whether Your Forecasting Model Beats the Market: A Python Walkthrough

Brier scores, the Murphy decomposition, a clustered bootstrap and a blending test, run on an open dataset of 5,654 timestamped model forecasts against Polymarket prices.

If you build a probabilistic model for events that also trade on a prediction market, the market price is your natural baseline, and an unforgiving one. Saying "our model is 60% accurate" means very little; the question that matters is whether your probabilities carry information the price does not.

This walkthrough answers that question end to end on a real dataset. It covers four steps that are worth doing in this order for any forecaster:

  1. Score both sides with a proper scoring rule.
  2. Decompose the score to see why one side wins.
  3. Put an honest confidence interval on the gap.
  4. Test whether the model adds anything on top of the price.

The data are model forecasts on Polymarket questions published by NeuPortal (disclosure: that is my project). Each forecast was stored next to the market price at the moment it was made and timestamped into Bitcoin, so none of the numbers can have been edited after the outcome. The dataset is open (CC BY 4.0).

Setup

import numpy as np
import pandas as pd

URL = "https://huggingface.co/datasets/neuportal/forecasts-vs-market/resolve/main/forecasts.csv"
df = pd.read_csv(URL)

# keep resolved questions whose forecast was timestamped before the outcome was known
df = df[(df.status == "final") & (df.sealed_before_resolution.astype(str).str.lower() == "true")]
y = df.outcome.to_numpy()          # 1 = Yes, 0 = No
model = df.our_p.to_numpy()        # model probability of Yes
market = df.market_p_at_call.to_numpy()  # market price at the same moment
print(len(df))                     # 5654
Enter fullscreen mode Exit fullscreen mode

The filter on sealed_before_resolution matters. The dataset also contains an early batch that was timestamped late, after many outcomes were already known. Those rows are flagged and should never be used as evidence of forecasting skill. Whatever data you work with, check when each forecast provably existed before you score it.

Step 1: score both sides

The Brier score is the mean squared error between the probability and the 0/1 outcome. Lower is better; always saying 50% scores 0.25.

def brier(p, y):
    return np.mean((p - y) ** 2)

print(brier(model, y), brier(market, y))   # 0.1959  0.1598
Enter fullscreen mode Exit fullscreen mode

The market wins by a wide margin. That is the headline, but on its own it tells you nothing about what to fix.

Step 2: decompose the score

The Murphy decomposition splits the Brier score into three parts:

  • Uncertainty depends only on the outcomes. It is the same for both forecasters.
  • Reliability is the calibration error: when you say 30%, does it happen 30% of the time? Lower is better.
  • Resolution measures how much your forecasts separate events that happen from events that don't. Higher is better.

Brier ≈ uncertainty − resolution + reliability.

def murphy(p, y, bins=10):
    idx = np.clip((p * bins).astype(int), 0, bins - 1)
    base = y.mean()
    rel = res = 0.0
    for k in range(bins):
        m = idx == k
        if m.any():
            w = m.mean()
            rel += w * (p[m].mean() - y[m].mean()) ** 2
            res += w * (y[m].mean() - base) ** 2
    return rel, res, base * (1 - base)

for name, p in [("model", model), ("market", market)]:
    rel, res, unc = murphy(p, y)
    print(f"{name}: reliability {rel:.4f}  resolution {res:.4f}  uncertainty {unc:.4f}")
Enter fullscreen mode Exit fullscreen mode
reliability (lower is better) resolution (higher is better) uncertainty
model 0.0111 0.0192 0.2033
market 0.0002 0.0429 0.2033

This table is more useful than the headline. The market is almost perfectly calibrated, with reliability close to zero. The model is not, but calibration is the smaller half of its problem. Most of the gap is resolution: the market separates Yes from No outcomes more than twice as well. Recalibrating the model, for example with isotonic regression, would close only part of the gap. The model is missing information, not just mislabelling its confidence.

A quick look at the reliability table shows where the calibration error sits:

bins = pd.cut(model, np.linspace(0, 1, 11), include_lowest=True)
print(pd.DataFrame({"p": model, "y": y}).groupby(bins, observed=True).agg(n=("y", "size"), mean_p=("p", "mean"), hit=("y", "mean")))
Enter fullscreen mode Exit fullscreen mode

In this data the worst bucket is the bottom one. When the model said 10% or less, events happened far more often than it claimed. That is a classic sign of a model whose predictive distribution is too narrow.

Step 3: an honest confidence interval

It is tempting to bootstrap the Brier difference over rows. Here that would be wrong, because rows are not independent. Many questions belong to the same event: several temperature bands for one city and day, for instance, all resolve together. Resample events, not rows:

df["d"] = (df.our_p - df.outcome) ** 2 - (df.market_p_at_call - df.outcome) ** 2
g = df.groupby("event").d.agg(["sum", "count"])

rng = np.random.default_rng(0)
boot = []
for _ in range(2000):
    s = g.iloc[rng.integers(0, len(g), len(g))]
    boot.append(s["sum"].sum() / s["count"].sum())
print(df.d.mean(), np.percentile(boot, [2.5, 97.5]))   # 0.0361  [0.0317  0.0407]
Enter fullscreen mode Exit fullscreen mode

The model's excess Brier is 0.036, with a 95% interval of roughly 0.032–0.041 across 1,938 events. The interval does not come close to zero, so this is not noise. Clustering is also what keeps you honest when the gap is small. A row-level bootstrap would report an interval that is too narrow, and that is how "significant" edges are found where none exist.

Step 4: does the model add anything to the price?

A model can lose to the market overall and still be useful, if its errors differ from the market's. The cleanest test is to blend the two and see whether any weight on the model improves on the price alone. Blending in log-odds space is the usual choice for probabilities:

def logit(p):
    p = np.clip(p, 1e-3, 1 - 1e-3)
    return np.log(p / (1 - p))

for w in [0, 0.1, 0.2, 0.3, 0.5]:
    p = 1 / (1 + np.exp(-(w * logit(model) + (1 - w) * logit(market))))
    print(w, round(brier(p, y), 4))
Enter fullscreen mode Exit fullscreen mode
weight on model Brier of blend
0.0 (market only) 0.1598
0.1 0.1610
0.2 0.1633
0.3 0.1662
0.5 0.1732

Every positive weight makes things worse. On this sample the model carries no information that the price does not already contain. Run this test before any other. If the optimal weight on your model is zero, no amount of tuning the threshold or position sizing will rescue it.

What to take away

  • Score against the price, not against a coin flip. Beating 0.25 is easy; beating a liquid market is the actual bar.
  • Decompose before you fix. Here the market won mostly on resolution, which points to missing inputs rather than a calibration patch.
  • Bootstrap the unit that is actually independent. In prediction-market data that is usually the event, not the question.
  • The blending test is the verdict. If adding your model to the price never helps, the model is not yet adding information.
  • Check when each forecast provably existed. Scores computed on forecasts recorded after the outcome measure nothing.

The data and the proof files are public if you want to rerun any of this or try your own model against the same questions:

More: the full experiment at https://neuportal.ai/experiment and the write-up 7,426 forecasts against Polymarket: exactly where the market beat our models.

Educational transparency data, not financial advice.

Top comments (0)