TL;DR — Most teams adopt feature stores for low-latency online serving, but that was never the hard problem they were designed to solve. The actual job is point-in-time correctness — preventing future data from leaking into training labels — and most implementations quietly skip it. LLM-derived features make this worse by adding an unversioned leakage vector nobody is tracking.
Ask a team why they adopted a feature store and you'll hear some version of: "we needed consistent features between training and serving, and low-latency lookups for the online model." That's true, but it's the easy 20% of the problem. The hard 80% — the reason feature stores exist as a distinct architectural pattern instead of just "a database with a cache in front of it" — is point-in-time correctness. Most teams never actually build that part. They build a cache and call it a feature store.
The leak nobody notices until prod
Point-in-time correctness means: when you compute a feature for a training example, you use only the data that would have been available at that example's event timestamp — not data from the future relative to that row. This sounds obvious. It is violated constantly, and it is violated in ways that make your offline metrics look better, not worse, which is exactly why it survives review.
Take a classic fraud feature: avg_transaction_amount_30d. If you compute this by joining against a transactions table using a plain GROUP BY customer_id without constraining the window to "before this specific transaction's timestamp," you've just let transactions from after the labeled event influence the feature. The model learns a pattern that includes information from the future. Offline AUC looks great. In production, that future data doesn't exist yet at inference time, and the feature distribution the model actually sees is different from the one it trained on. Nobody calls this what it is — a leakage bug — because it doesn't throw an error. It shows up months later as "unexplained model degradation," gets blamed on drift, and triggers a retraining cycle that doesn't fix anything because the leak is still there.
Why this gets skipped
Point-in-time joins are expensive to build correctly and boring to test. You need an offline store that supports as-of joins against a timestamped event log, not a data warehouse table that only has "current state." Most teams don't have that log. They have slowly-changing dimension tables, nightly snapshots, or worse, a production database replica with no history at all. Building the infrastructure to reconstruct "what did this feature look like at exactly this moment" is a real engineering investment, and it produces no visible product value on its own — it just prevents a bug that hasn't happened yet. That's a hard sell in a roadmap review, so it gets deprioritized in favor of the part that's easy to demo: a Redis-backed online store with sub-10ms lookups.
The result is an industry full of systems labeled "feature store" that are really just a shared cache with a schema registry attached. They solve the training/serving skew problem for feature definitions — same transformation code runs in both paths — but they don't solve the training/serving skew problem for feature timing, which is the more dangerous one because it's invisible in code review.
The LLM-era version of the same bug
This problem just got a second life. Teams are now storing LLM-derived signals as features: an embedding of a support ticket, a sentiment score from a classifier prompt, a "risk category" tag generated by an LLM call over unstructured text. These get written into the feature store next to your numeric aggregates, with the same entity key and timestamp, and everyone assumes the same guarantees apply.
They don't, because these features carry a dependency the schema doesn't capture: which model version and which prompt produced this value. A numeric aggregate like average transaction amount is stable in definition — the SQL either changed or it didn't, and you can version the SQL. An LLM-derived feature's meaning can shift silently when the underlying model is updated, when a prompt template gets a one-word edit, or when a provider changes default sampling behavior. None of that shows up as a schema change. Your feature store will happily tell you ticket_sentiment_score has been "the same feature" for a year, when it was actually computed by three different model versions with three different notions of sentiment.
This is point-in-time leakage's cousin: point-in-version leakage. A model trained on ticket_sentiment_score computed by model version A will not behave correctly when that feature is silently regenerated by model version B for serving, and vice versa for backfills used in retraining. If your feature store's provenance metadata stops at "timestamp and entity key," you have no way to detect this, and no way to explain a performance drop when
Top comments (0)