DEV Community: SoccerDream AI

You can't rank 200 models with one leaderboard

SoccerDream AI — Mon, 09 Nov 2026 02:08:27 +0000

I run a platform where a couple of hundred independent models publish probability estimates on football fixtures. They aren't variants of one model with different seeds — each one has its own training corpus, its own feature set, its own retraining schedule, and its own life cycle. At the time of writing: 209 have existed, 27 are active, 102 are retired, and 25,963 predictions have been graded.

The first version of the admin panel had a leaderboard. Sort by accuracy, descending, take the top one. It took about three weeks to learn that a leaderboard over a population that large is not a ranking — it's a multiple-comparisons machine with a UI.

Here's what replaced it, and the four bugs that forced each piece.

The problem with sorting by accuracy

Two hundred models, dozens of fixtures a day. Somebody will hit a hot streak. It is not a hypothesis that one of them will look excellent over a 40-fixture window; it is arithmetic. A leaderboard has no memory of how many candidates were tried, so a 40-fixture window of luck is indistinguishable from a 40-fixture window of skill — and the leaderboard will promote it to the top of the page, where it gets attention, which is the only reward the system offers.

The failure isn't subtle. In the early build the "top model" changed almost weekly, always upward, never stable. When I pulled the histories, the models that topped the board were systematically the ones with the fewest graded predictions. Small sample, large variance, and a ranking that treats variance as merit.

So the ranking had to stop being a ranking and start being a life cycle.

Four states, one direction

probation ──▶ active ──▶ degraded ──▶ retired
     │            │           │
     └────────────┴───────────┘
        (demotion allowed, promotion re-earned)

probation — new model, publishing publicly but with a visible "unproven" badge. It exists to accumulate a sample.
active — passed the promotion gate. Eligible for prominence.
degraded — was active, drifted. Still publishing, no longer promoted. This state exists because the alternative is a binary that oscillates, which I'll get to.
retired — stopped. Terminal in practice, but see the section on why the rows stay.

One rule I'd defend hard: state changes are events, not columns.

The obvious schema is models.status plus a cron job that reassigns it. That's what I wrote first, and it cost me every bit of history. Once the cron overwrites the field, you cannot answer "when did this model die, and on what evidence" — which is the only question worth asking when you're deciding whether a retirement was a mistake. A status column is a snapshot pretending to be a record.

Append-only transitions

CREATE TABLE model_transitions (
    id         bigserial PRIMARY KEY,
    model_id   bigint      NOT NULL REFERENCES models(id),
    from_state text        NOT NULL CHECK (from_state IN ('probation','active','degraded','retired')),
    to_state   text        NOT NULL CHECK (to_state   IN ('probation','active','degraded','retired')),
    reason     text        NOT NULL,
    evidence   jsonb       NOT NULL,   -- the actual numbers that justified it
    decided_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX ON model_transitions (model_id, decided_at DESC);

Current state is a view, never stored:

CREATE VIEW model_state AS
SELECT DISTINCT ON (model_id)
       model_id, to_state AS state, decided_at AS state_since
FROM model_transitions
ORDER BY model_id, decided_at DESC;

And the promotion guard lives in the writer, not the reader:

def transition(conn, model_id, to_state, reason, evidence, min_dwell_days=7):
    cur = conn.execute(
        "SELECT state, state_since FROM model_state WHERE model_id = %s", (model_id,)
    ).fetchone()
    if cur and cur.state == to_state:
        return None                       # no-op, not an error
    if cur and (now() - cur.state_since) < timedelta(days=min_dwell_days):
        return None                       # too soon, by design
    conn.execute(
        "INSERT INTO model_transitions (model_id, from_state, to_state, reason, evidence) "
        "VALUES (%s, %s, %s, %s, %s)",
        (model_id, cur.state if cur else "probation", to_state, reason, json.dumps(evidence)),
    )

evidence is the part people skip and then regret. It stores the window size, the metric value, and the comparison baseline at the moment of the decision. Six months later, "why was this model retired" has an answer that isn't my memory.

Flapping, and why `degraded` exists

The first demotion rule was a single threshold. Below it, demote. Above it, promote. A model sitting near the line generates a state change every single evaluation tick, and each change fires notifications, rewrites the public badge, and generally makes the system look broken.

Three fixes, all necessary:

Two thresholds, not one. Promote at the upper band, demote only below the lower band. The gap is dead space where nothing happens. This is just hysteresis and I should have started there.
Minimum dwell time. Even inside the dead space, refuse any transition within 7 days of the last one. Short-lived states are almost never information.
degraded as a buffer. A model coming off active goes to degraded first, not straight to retired. Retirement requires it to fail again from degraded. One bad window shouldn't be terminal; two consecutive bad windows in different states is a pattern.

The counter I keep is the number of transitions per model per month. Anything above ~2 is either a threshold sitting exactly at the population mean or a genuinely unstable model — both worth knowing.

Retiring means stopping, not deleting

This is the one that actually corrupted a number I had published.

When a model failed, the cleanup job deleted its predictions. Reasonable-sounding: dead models don't need rows. The effect was that the platform-level accuracy figure — computed over whatever predictions still existed — silently excluded every model that performed badly enough to be retired. The number went from 54% to 61% and I briefly believed it, because the computation was correct and the denominator was a lie.

The platform-level metric is only meaningful if the population includes the failures:

-- every model that ever published, including the dead
SELECT count(*) FILTER (WHERE g.outcome = argmax_of_prediction)::float / count(*) AS accuracy
FROM predictions p
JOIN models m       ON m.id = p.model_id
LEFT JOIN grades g  ON g.prediction_id = p.id
WHERE p.created_at BETWEEN %s AND %s;

No WHERE m.state = 'active'. If you filter to survivors and call the result platform performance, you're not measuring the system, you're measuring the system after it selected for the outcome you wanted.

Retired models keep their predictions, their grades, and their transition history. 102 of them at the moment. They are the control group.

Promotion gates that survive contact with reality

Absorbing everything above, the gate for probation → active is:

at least 50 graded predictions in the current window (below that, the interval on any rate is wider than the differences we're trying to detect)
performance measured within the same competition and compared against that competition's contemporaneous median, never against a global figure
no promotion within 7 days of the last transition

The competition scoping is not pedantry. A model covering one evenly-matched league and a model covering forty leagues of wildly different predictability are answering different questions. Blending them produces a number that describes neither, and the model that happened to pick the easy competition wins the leaderboard for reasons that have nothing to do with modelling.

I also stopped showing a single "platform accuracy" number on the site. The honest answer to "how accurate is the platform" is a distribution across models and competitions, and the moment you collapse it to one figure you've started marketing instead of measuring.

Failure modes, honestly

The cron that ate the history. Described above. Append-only transitions, view for current state. Cost: one extra join on read, worth it every time.
Thresholds set at the population mean. Guarantees flapping, because half the population is always on the wrong side of the mean. Set bands from the distribution's interquartile range, not its centre.
Comparing a model to itself across regimes. A model's own history includes retraining. Comparing this month to six months ago compares two different objects wearing the same model_id. Any retraining that changes weights is a new model row.
retired used as a soft delete. If retirement hides a model from every query, you've rebuilt survivorship bias with a nicer name.
No cap on promotions. Without a dwell time, a noisy window promotes, the next window demotes, and the public page flickers. Users read flicker as unreliability, correctly.

What this buys

The leaderboard answered "which model looks best right now," which is a question about noise. The life cycle answers "which models have earned attention, which are being watched, and which are gone" — and every answer has a timestamp, a reason, and the numbers that produced it.

The schema is thirty lines. The hard part is accepting that a model's status is a claim you have to defend later, with evidence you kept at the time.

The platform this comes from is SoccerDream AI — independent models publishing timestamped probability estimates on football fixtures, graded in the open, retired models included in the record. Per-model pages show the full life cycle rather than a current score.

Analytics and research content. Not betting advice. 18+

Immutable predictions: a schema for grading model output you can't quietly rewrite

SoccerDream AI — Mon, 09 Nov 2026 01:36:08 +0000

Every model evaluation I've ever seen has the same hole in it: the predictions are stored in a table someone can edit.

Not maliciously, usually. Someone reruns a pipeline, a backfill overwrites yesterday's rows, a "fix" recomputes an output with the model that was retrained this morning, and suddenly your 30-day accuracy number describes a model that never existed. The evaluation looks better, the model is not better, and nobody can tell the difference because there's no evidence trail.

I build a football analytics platform where a few hundred independent models publish probability estimates before kickoff. Because the output is public and the public likes to argue about it, I had to make one property non-negotiable: a prediction, once written, is a fact about the past, not a row you can improve.

Here's the schema and the pipeline that enforces it, with the parts that actually bit me.

The three properties that matter

Append-only. No UPDATE. No DELETE. A wrong prediction stays wrong forever — that's the whole point.
Timestamped at write, not at display. The time has to come from the database and be bound to the content, or it's decoration.
Gradeable exactly once. A grading job that can run twice is a grading job that will run twice, and the second run will disagree with the first.

Schema

Postgres, and the constraints do the enforcing rather than application code. Application code is where discipline goes to die.

CREATE TABLE predictions (
    id              bigserial PRIMARY KEY,
    model_id        bigint      NOT NULL REFERENCES models(id),
    fixture_id      bigint      NOT NULL REFERENCES fixtures(id),
    -- the three outcomes, as probabilities, CHECKed not trusted
    p_home          numeric(5,4) NOT NULL CHECK (p_home BETWEEN 0 AND 1),
    p_draw          numeric(5,4) NOT NULL CHECK (p_draw BETWEEN 0 AND 1),
    p_away          numeric(5,4) NOT NULL CHECK (p_away BETWEEN 0 AND 1),
    created_at      timestamptz NOT NULL DEFAULT now(),
    -- hash of (model_id, fixture_id, probabilities, created_at)
    content_hash    text        NOT NULL,
    prev_hash       text,                    -- hash chain within a model
    inputs_snapshot jsonb       NOT NULL,    -- what the model actually saw
    UNIQUE (model_id, fixture_id)
);

-- the only sane constraint in the whole table
ALTER TABLE predictions
  ADD CONSTRAINT probs_sum_to_one
  CHECK (abs(p_home + p_draw + p_away - 1) < 0.001);

Two decisions worth defending.

inputs_snapshot is a column, not a foreign key. I originally pointed at a model_inputs table and kept it normalised. It broke within a week: the input builder got a bug fix, the referenced row changed, and every historical prediction was suddenly "computed from" data it never saw. Snapshots cost storage and buy reproducibility. Store the blob.

UNIQUE (model_id, fixture_id) without an ON CONFLICT clause. When an insert collides, I want to know. A silent upsert here means the model's first answer gets replaced by a better-informed later answer, which is exactly the failure mode this table exists to prevent. Publish once, then live with it.

Tamper evidence via hash chain

Constraints stop accidental edits. They don't stop UPDATE predictions SET p_home = 0.61. For that, each row carries the hash of its predecessor per model:

import hashlib, json

def content_hash(row: dict, prev_hash: str | None) -> str:
    payload = json.dumps({
        "model":  row["model_id"],
        "fixture": row["fixture_id"],
        "p": [str(row["p_home"]), str(row["p_draw"]), str(row["p_away"])],
        "ts": row["created_at"].isoformat(),
        "prev": prev_hash,
    }, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode()).hexdigest()

Verify by replaying the chain. Cost is linear in rows, so I run it per model over the last N and nightly over everything; a break is loud, precise, and localised to a model. Decimal values are stringified before hashing — 0.6100 and 0.61 are the same probability and different JSON, and I lost an afternoon to that.

Grading exactly once

The grading worker takes a fixture result and writes grades. The trap is re-entry: retries, overlapping scheduler ticks, a manual rerun at 2am. Guard with a unique constraint on the result, not with a mutex in application memory:

CREATE TABLE grades (
    prediction_id bigint PRIMARY KEY REFERENCES predictions(id),
    outcome       char(1) NOT NULL CHECK (outcome IN ('H','D','A')),
    brier         numeric(6,5) NOT NULL,
    graded_at     timestamptz NOT NULL DEFAULT now(),
    result_id     bigint NOT NULL REFERENCES results(id)
);

prediction_id as the primary key makes the second grade of the same prediction a physical impossibility. The worker uses ON CONFLICT DO NOTHING and reports how many rows it skipped; a non-zero skip count on a fresh fixture means something upstream is duplicating results.

And the grading job refuses to run when predictions.created_at > results.kickoff_at. Cheap check, catches the case where a pipeline backfills a prediction after the match ended. A post-hoc "prediction" graded as correct is worse than no prediction at all — it's a lie with a timestamp on it.

Metrics I actually trust

Brier score decomposes into reliability and resolution, which is why it beats accuracy for probability output:

def brier(ps: list[tuple[float, float, float]], ys: list[int]) -> float:
    total = 0.0
    for (ph, pd_, pa), y in zip(ps, ys):
        p = (ph, pd_, pa)[y]
        total += sum((pi - (1.0 if i == y else 0.0)) ** 2 for i, pi in enumerate((ph, pd_, pa)))
    return total / len(ps)

Note the divisor: over a fixed window of fixtures per model, not aggregated across leagues. A model that only covers one competition and a model that covers forty are answering different questions; averaging them produces a number that describes neither. I compute per model per competition and I refuse to publish a blended figure. When someone asks "what's the platform's accuracy," the honest answer is a distribution, and the moment you collapse it to a single number you've started marketing instead of measuring.

Calibration is the more useful diagnostic anyway. Bucket by predicted probability, plot observed frequency:

SELECT width_bucket(p_max, 0, 1, 10) AS bucket,
       count(*)                     AS n,
       avg(p_max)                   AS predicted,
       avg((outcome_idx = argmax_idx)::int) AS observed
FROM graded_predictions
GROUP BY 1 ORDER BY 1;

A model sitting on the diagonal is trustworthy in the only sense that matters: when it says 0.7, it happens about 70% of the time. A model off the diagonal in either direction is miscalibrated, and no amount of accuracy fixes that.

Failure modes, honestly

Timezone drift. created_at in local time meant a prediction "sealed before kickoff" appeared to be 8 hours after it. Everything is timestamptz now, and the kickoff comparison uses UTC.
Float drift in checks. The abs(sum - 1) < 0.001 tolerance exists because normalising three floats independently lands on 0.9999999 regularly.
Hash chain forks. Two workers inserting for the same model concurrently both read the same prev_hash and produce a fork. Fixed with SELECT ... FOR UPDATE on the model row inside the insert transaction. Obvious in retrospect; the resulting chain looked valid for weeks before the verifier caught the branch.
Snapshot bloat. 40KB average per prediction. Fine at first, 200GB later. Compressed jsonb, partitioned by month.

What this buys

The value isn't the schema, it's that the schema makes a class of argument impossible. Nobody can ask "was that prediction made before or after the result" — it's in the hash chain. Nobody can quietly drop the bad weeks — deletions break verification. Nobody can swap in a retrained model and call it the same model — different weights is a different model_id.

If you're evaluating anything public-facing, the schema is the cheap part. Deciding that your numbers are not allowed to improve retroactively is the expensive part.

The platform this comes from is SoccerDream AI — independent models publishing timestamped probability estimates on football fixtures, graded in the open, losses included. If you want to poke at the grading output, the per-model pages show the full history rather than a summary.

Analytics and research content. Not betting advice. 18+

DEV Community: SoccerDream AI

You can't rank 200 models with one leaderboard

The problem with sorting by accuracy

Four states, one direction

Append-only transitions

Flapping, and why degraded exists

Retiring means stopping, not deleting

Promotion gates that survive contact with reality

Failure modes, honestly

What this buys

Immutable predictions: a schema for grading model output you can't quietly rewrite

The three properties that matter

Schema

Tamper evidence via hash chain

Grading exactly once

Metrics I actually trust

Failure modes, honestly

What this buys

Flapping, and why `degraded` exists