Feature Stores and the Problem They Solve

#ai #llm #machinelearning #programming

A feature store exists because the same feature gets computed twice — once in a batch job for training and once in a service for scoring — and the two definitions drift apart. This page shows one feature breaking, in eight lines, and the join that stops it.

One feature, two definitions

The feature is avg_order_value_30d. It appears in a fraud model. The training set is built in the warehouse:

-- training pipeline, run nightly over history
SELECT
  o.order_id,
  o.customer_id,
  o.created_at,
  AVG(h.amount) OVER (
    PARTITION BY o.customer_id
    ORDER BY o.created_at
    RANGE BETWEEN INTERVAL '30 days' PRECEDING AND CURRENT ROW
  ) AS avg_order_value_30d
FROM orders o
JOIN orders h ON h.customer_id = o.customer_id

And the serving path computes it in the application:

# scoring service, called on each incoming order
total = redis.get("aov:sum:" + customer_id)   # rolling 30d sum, updated
count = redis.get("aov:cnt:" + customer_id)   # after each order settles
avg_order_value_30d = float(total) / max(int(count), 1)

The two disagree in three ways, none of which produces an error.

The current row. The SQL window says CURRENT ROW, so the training value includes the order being scored. The Redis counter is updated after settlement, so the serving value excludes it. On a customer with two prior orders, that is a 33% difference in the denominator.
The window boundary. RANGE ... 30 days PRECEDING is measured from the order timestamp. The Redis counter is expired by a TTL relative to when the key was written. These are the same window only when nothing is late.
Failed and refunded orders. The warehouse table has them, filtered or not depending on who wrote the query. The Redis counter has whatever the settlement webhook decided. Nobody wrote down which is correct.

Why the model silently gets worse

The model was fitted on the first distribution and is scored on the second. It does not crash, it does not log anything, and its offline evaluation — which is run against the training pipeline — continues to look exactly as good as it did on the day it shipped.

What degrades is the live decision quality, and the degradation is concentrated on the rows where the two definitions differ most: new customers with few orders, where the inclusion or exclusion of one order changes the feature by a third. Those are usually the highest-risk rows, so the skew does its worst damage exactly where the model matters most.

The general diagnostic is to log the served feature vector on every request and compare its distribution to the training distribution. Anything that disagrees on more than a rounding error is a bug, and it is the same comparison drift monitoring makes — with the difference that this discrepancy was there on day one rather than arriving later.

Point-in-time correctness

The deeper requirement, and the one that gives feature stores their reason to exist, is this: to build a training row for an event at time T, every feature must hold the value it held at T, not the value it holds now.

Naively joining a feature table to a label table on the entity id gets this wrong in the most damaging possible way. The feature table holds current values, so the training row for a fraud attempt in March gets the customer’s August risk profile — which was shaped by the fraud. The model learns to read the consequence of the label. This is target leakage through a stale join and it produces spectacular offline scores.

WRONG: current-value join

  labels                        features (current)
  event_id  cust  t_event       cust  aov   risk_band
  1         A     2026-03-04    A     412   HIGH     <- became HIGH in June
  2         B     2026-03-09    B      88   LOW

RIGHT: as-of join against a feature history

  feature history
  cust  valid_from    aov   risk_band
  A     2026-01-01    120   LOW
  A     2026-06-11    412   HIGH
  B     2026-02-02     88   LOW

  event 1 at 2026-03-04 -> the row valid at that moment
                        -> aov 120, risk_band LOW

Building this requires the feature history to exist, which is a decision that has to be made before you need it. A table that is updated in place cannot be reconstructed, and no library recovers what was overwritten. It is the same argument as versioning the dataset: the cheap moment to start keeping history is before anybody needs it, and there is no expensive moment that works instead. The three-window layout in the churn page is what this join is used to build.

The as-of join, in code

For datasets that fit in memory, pandas does this directly. The semantics are exactly right: for each event, take the most recent feature row at or before the event time, within each entity.

import pandas as pd

labels = labels.sort_values("t_event")
history = history.sort_values("valid_from")

train = pd.merge_asof(
    labels,
    history,
    left_on="t_event",
    right_on="valid_from",
    by="customer_id",
    direction="backward",        # most recent value at or before t_event
    allow_exact_matches=True,
)

Two arguments carry the correctness. direction="backward" is what forbids the future. by="customer_id" is what keeps one customer’s history out of another’s row. Both frames must be sorted on the join key or the result is wrong rather than an error.

Set allow_exact_matches=False when the feature is computed from the same event as the label — that is the equivalent of the CURRENT ROW problem above, and it is the single most common source of a suspiciously good model.

The equivalent in SQL is a lateral join or a window function selecting the last feature row before the event timestamp. It is slow and correct; write it once, materialise the result, and do not re-derive it per experiment.

What a feature store actually provides

Capability	Description
one definition	A feature is defined once, in one place, and both the training pipeline and the serving path read that definition. This is the whole product; everything else is plumbing around it.
offline store	The historical values, with validity timestamps, queryable point-in-time. Usually a warehouse table or a set of Parquet files with a metadata layer.
online store	The current values for one entity, retrievable in single-digit milliseconds. Usually Redis, DynamoDB or similar. Populated from the same computation as the offline store, which is the guarantee that matters.
point-in-time retrieval	Given a set of (entity, timestamp) pairs, return the feature values as of each timestamp. The as-of join above, generalised and made fast.
reuse and discovery	The fifth team to need days-since-last-order finds it rather than writing a sixth slightly different version. Real value at organisational scale and no value at all below it.
monitoring	Distribution statistics per feature, computed once and used both for drift alerts and for the train-serve comparison described above.

When you do not need one

Feature stores are frequently adopted by teams whose problem they do not solve, and the cost is a substantial piece of infrastructure with its own failure modes sitting in the middle of the model pipeline.

Batch scoring only. If predictions are computed nightly by the same job that computes the features, there is no second code path and therefore no skew. A table and a scheduler is the correct architecture, and it is not a lesser one.
One or two models, one team. The reuse and discovery benefits need many teams to exist at all. With one team, the shared definition can be a shared Python module, imported by both paths. That is a feature store with a smaller name and a hundredth of the operational burden.
Features that come with the request. If the model scores on fields present in the incoming payload — amount, country, device, item category — there is nothing to look up and nothing to keep in sync.
You do not have feature history yet. A feature store cannot reconstruct the past from a mutable table. If your source data is overwritten in place, the first project is an event log or slowly-changing-dimension tables; the store is the second.

The minimum viable version of all of this is one Python function per feature, imported by both the training job and the serving path, plus a history table with valid_from, plus one test that asserts the two paths produce the same value for a fixed input. That test is the thing that catches the bug at the top of this page, and it does not require any product at all.