TL;DR — Feature stores solved the storage problem — point-in-time joins, online/offline sync, schema registries — but they rarely version the transformation logic itself. When the same feature name is computed by two different pipelines, you get silent train/serve skew that no schema check will ever catch. The fix is treating transforms as versioned, testable functions, not just columns in a table.
Every feature store pitch starts the same way: training-serving skew, point-in-time joins, online/offline consistency. These are real problems, and modern feature stores have gotten reasonably good at solving them. But there's a failure mode sitting one layer below storage that almost nobody talks about, and it's the one actually taking down production models: two pipelines compute a feature with the same name, in slightly different ways, and both pass every schema check you have.
The feature store solved the wrong layer
A feature store's marketing surface is a table. You register a feature group, it has a schema, it has a timestamp column, and it gets materialized to both an offline store for training and an online store for low-latency serving. That's the product. But the thing that actually determines model behavior isn't the table — it's the function that produced the values in it.
Consider avg_transaction_amount_7d, a feature that shows up in roughly every fraud and recommendation system ever built. The batch version, computed nightly in a warehouse job, windows over calendar days and treats a missing window as null. The streaming version, computed in a low-latency event processor for serving, windows over a rolling 7*24-hour period and treats a missing window as zero. Same feature name. Same declared type: float. Same nullable flag: false, because someone backfilled the nulls to zero at some point and forgot to tell the batch job.
Your schema registry sees two floats. Your feature store sees one logical feature with two sync'd materializations. Your model sees a distribution shift it was never trained on, and it quietly degrades. Nothing in the stack flags this, because nothing in the stack is checking the function — only the shape of its output.
Skew isn't a sync problem, it's a semantics problem
The feature store industry solved sync. Dual-write pipelines, change-data-capture into online stores, point-in-time correct backfills — this is mature tooling now. What it didn't solve is the much harder problem: guaranteeing that the batch transform and the streaming transform are the same function, not just two functions that happen to produce values with the same name and type.
This matters more as feature logic gets more complex. A rolling average is forgiving — the failure mode is a slow drift. A feature built on a sequence of joins, a categorical encoding with a fitted vocabulary, or a windowed aggregation with specific null-handling is not forgiving at all. Small semantic divergences compound, and because the divergence is in logic rather than schema, your validation layer — which checks types, ranges, and nullability — has nothing to catch it on.
The uncomfortable truth is that most teams don't actually have one feature pipeline with two execution targets. They have two independently maintained codebases that are supposed to agree, maintained by whoever had time to port the Spark job to Flink, or vice versa, six months after the original was written.
Treat the transform as the artifact, not the table
The fix is a shift in what gets versioned. A feature store should be a registry of functions with a single canonical implementation, compiled or interpreted into both batch and streaming execution engines — not a registry of tables that two separate codebases are trusted to agree on.
This looks less like a data warehouse feature and more like a package manager entry. A feature definition has a version, a hash of its logic, a declared contract for null-handling and windowing semantics, and a test suite that runs representative inputs through both execution paths and diffs the output. If the batch and streaming implementations diverge by more than a defined tolerance, that's a build failure, not a silent model regression discovered three weeks later in a retro.
Some of the newer feature engines are moving this direction — defining a transform once in a dataframe-like API and compiling it to both a batch job and a streaming operator. That's the right instinct. It treats the transformation as the source of truth and the execution engine as an implementation detail, which is exactly backwards from how most feature stores are architected today, where the table is the source of truth and the transform is an implementation detail scattered across two repos.
What changes when agents start calling features directly
This problem is about to get worse, not better. As agentic systems start pulling features directly — an agent calling a feature store as a tool to get a user's recent behavior summary rather than going through a model's fixed input schema — the consumers of a feature multiply and decouple from any single training pipeline. A feature that was previously consumed by exactly one model, with one well-understood tolerance for semantic drift, is now consumed by an arbitrary set of agents making arbitrary decisions based on its value.
In that world, an undocumented semantic change to a feature — swapping null-for-zero, changing a window boundary, re-fitting a categorical encoder — doesn't just degrade one model's AUC quietly. It changes the behavior of every downstream agent that treats the feature as ground truth, with no shared test suite catching the regression because none of those consumers were part of the original training pipeline's validation loop.
The contract that matters here isn't "this endpoint returns a float." It's "this float is computed by this exact logic, versioned, and guaranteed identical regardless of whether you're hitting it from a training job, a real-time inference path, or an agent's tool call." That guarantee has to be enforced at the function level, because enforcing it at the storage level only tells you the shape stayed the same — never that the meaning did.
The practical takeaway
If you're building or auditing a feature platform, stop asking whether your offline and online stores are in sync. Ask whether your offline and online transforms are the same compiled artifact, or two hand-maintained implementations that happen to agree today. Ask whether a change to feature logic triggers a diff test across execution engines, or just a schema migration. Ask whether a new consumer of a feature — a new model, a new agent — inherits the same semantic guarantees as the first one, or is quietly trusting a float that means something slightly different than they assume.
Storage was never the hard part. The hard part was always agreeing, provably, on what a number means — and that's a property of the function, not the table it lands in.
Top comments (0)