DEV Community

Vaibhav
Vaibhav

Posted on

Feature Stores and Point-in-Time Correctness: The Bug That Silently Ruins ML Models

#mlops #machinelearning #dataengineering #insurtech

Your model scored 0.87 AUC in the notebook and is mediocre in production. Nine times out of ten the algorithm is fine — the culprit is training-serving skew or a point-in-time leak. A feature store is the discipline that kills both, and in regulated domains like insurance it's the difference between a model you can defend and one you can't.

Training-serving skew

You compute avg_claim_amount_12m one way in a training SQL query and re-implement it in the scoring service. They agree 95% of the time. The other 5% — claim reopenings, currency conversion, the boundary of the 12-month window — is where the model silently degrades. A feature store gives you one definition, computed once, served identically to training and inference. Write the logic once; the skew disappears because there's only one implementation.

The part everyone gets wrong: point-in-time correctness

Each training row is a decision made at a moment in the past. Its features must reflect what was known at that moment — not now.

Join today's avg_claim_amount_12m onto a claim from six months ago and you've leaked the future into the past. Offline metrics look fantastic; production collapses. This is label leakage via non-point-in-time features, and it's endemic in insurance because so many features are time-windowed aggregates.

A proper feature store enforces an as-of join: for each training row, retrieve the feature value as it was at that row's timestamp.

# conceptually: point-in-time correct training set
training_df = store.get_historical_features(
    entity_df=labels,           # has entity_id + event_timestamp
    features=[
        "claims:avg_claim_amount_12m",
        "policy:tenure_days",
    ],
)  # each row gets feature values as-of its own event_timestamp
Enter fullscreen mode Exit fullscreen mode

Doing this by hand in SQL is possible but one wrong join invalidates the whole model.

Why insurance makes it worse

  • Long horizons. Liability claims develop over years; point-in-time errors compound over a long tail.
  • Regulatory scrutiny. "The feature was computed slightly differently in training and production" is not an answer you want to give a regulator. One versioned definition = one auditable story.
  • Slow ground truth. You may not know if a reserve was right for years — you can't afford skew on top of already-noisy labels.

Minimum viable feature store

You don't need a heavyweight platform to start:

  1. A registry — one name, one owner, one definition, one version per feature.
  2. Point-in-time retrieval for training (as-of joins, enforced).
  3. A serving path that reuses the same definition — no re-implementation.

Start with the five features your first model needs; grow from there.

Full write-up with the insurance-specific angle:

Feature Stores for Insurance ML: The Point-in-Time Correctness Problem →

From an ongoing series on the data foundation under insurance AI by IntelliBooks.

How's your team handling as-of joins — a real feature store, or hand-rolled SQL and hope?

Top comments (0)