DEV Community: Lukas Brunner

Detecting Silent Model Failure: Drift Monitoring That Actually Works

Lukas Brunner — Thu, 21 May 2026 06:52:38 +0000

TL;DR: Most drift monitoring setups alert on the wrong thing. Feature distribution drift is cheap to compute and almost always misleading. Prediction drift plus a delayed ground-truth feedback loop catches the failures that actually cost money. Here is the setup I use at Yokoy.

A model that returns HTTP 200 with a plausible-looking float is the worst kind of broken. No exception, no pager, no Slack message. The metric only moves three weeks later when finance reviews the numbers.

I have spent the last two years rebuilding the monitoring story for our expense classification models. What follows is what I kept after throwing out the rest.

The mistake I keep seeing

Teams instrument input feature drift first because it is the easiest thing to compute. Pull yesterday's feature values, pull today's, run a KS test on each column, alert when p < 0.05.

This generates noise. A lot of noise.

Features drift constantly for reasons that have nothing to do with model quality. A new customer onboards, the merchant category distribution shifts, you get a Slack ping at 03:00 for something that does not matter. After two weeks of this, on-call mutes the channel. After four weeks, the channel is deleted.

The problem is not the test. The problem is that input drift is a weak proxy for what you actually care about: did model performance degrade.

What to monitor instead

Three signals, ranked by cost and value.

Signal	Compute cost	Latency to detect	False positive rate
Input feature drift	Low	Hours	High
Prediction distribution drift	Low	Hours	Medium
Performance vs delayed labels	Medium	Days to weeks	Low

Prediction drift is the underrated one. If your model started returning a different distribution of outputs without you shipping new weights, something upstream broke. Could be feature pipeline. Could be a provider returning malformed embeddings. Could be a real population shift. All of these are worth investigating.

The detection logic is short:

from scipy.stats import wasserstein_distance
import numpy as np

def prediction_drift_score(reference: np.ndarray, current: np.ndarray) -> float:
    return wasserstein_distance(reference, current)

# reference = predictions from the validation window when the model was promoted
# current = predictions from the last 24h of production traffic
# alert when score exceeds the 99th percentile of bootstrapped baseline scores

Wasserstein over KS for prediction monitoring. KS is hypersensitive to large samples and you will have large samples in production. With 500k predictions per day, KS rejects the null hypothesis for differences nobody cares about.

The feedback loop is non-negotiable

For expense classification, ground truth arrives when a human approves or corrects the prediction. Median latency is four days. P95 is three weeks.

We log every prediction with a join key and write it to a Parquet table partitioned by date. When labels arrive, a nightly Kubeflow pipeline joins them and computes per-segment performance: accuracy per merchant category, per country, per customer tier.

The per-segment view is what surfaces the failures. Aggregate accuracy stays at 94% while accuracy on a specific Swiss VAT category collapses to 71%. The aggregate view would never have caught it.

# Simplified pipeline component spec
- name: compute-segmented-metrics
  inputs:
    predictions_table: gs://yokoy-ml/predictions/dt={{date}}
    labels_table: gs://yokoy-ml/labels/dt={{date}}
  outputs:
    metrics_table: gs://yokoy-ml/metrics/dt={{date}}
  segments:
    - merchant_category
    - country
    - customer_tier
  resource_request:
    cpu: 4
    memory: 16Gi

The cost: roughly 12 minutes of compute per day on our volume. The value: every regression we caught in the last 18 months was caught here, not by drift monitoring.

Where input drift still earns its place

I have not fully abandoned input drift. It is useful as a debugging tool after the fact. When per-segment accuracy drops, the first question is which features moved. Having the historical drift scores already computed means the investigation starts with a query instead of a backfill.

So compute it, store it, do not alert on it.

A note on LLM-based features

We added an LLM-derived feature last year for invoice text classification, routed through a gateway in front of multiple providers (Bifrost handles this for us, though others like LiteLLM or Portkey cover the same ground). The drift profile changed immediately. Provider model updates, even minor ones, shift the feature distribution in ways you cannot see from your side.

Lesson: pin the provider model version explicitly. Treat a provider model change as a feature pipeline change. Re-run the validation set. This sounds obvious until the day a default model alias updates and you find out from the metrics.

Trade-offs and Limitations

Per-segment monitoring has a cardinality problem. With three segments of 50, 30, and 5 values you get 7500 cells. Most are empty or have too few samples for meaningful metrics. We use a minimum sample threshold of 100 per cell per day and accept that long-tail segments take longer to detect issues in.

Delayed labels mean delayed detection. For models where the label takes weeks, you need a complementary fast signal. Prediction drift fills part of that gap but it is a leading indicator, not a measurement.

Wasserstein distance has no native interpretation in production units. You bootstrap a baseline and alert on deviation from it. This works but it is not as crisp as "accuracy dropped 3 points."

Storing every prediction with features for joinability is expensive. We compress aggressively and tier old partitions to cold storage after 90 days. Plan the storage cost before you build it, not after.

Detecting Silent Model Failure: Drift Monitoring That Actually Works

Lukas Brunner — Wed, 20 May 2026 06:55:39 +0000

I have spent the last two years rebuilding the monitoring story for our expense classification models. What follows is what I kept after throwing out the rest.

The mistake I keep seeing

Teams instrument input feature drift first because it is the easiest thing to compute. Pull yesterday's feature values, pull today's, run a KS test on each column, alert when p < 0.05.

This generates noise. A lot of noise.

The problem is not the test. The problem is that input drift is a weak proxy for what you actually care about: did model performance degrade.

What to monitor instead

Three signals, ranked by cost and value.

Signal	Compute cost	Latency to detect	False positive rate
Input feature drift	Low	Hours	High
Prediction distribution drift	Low	Hours	Medium
Performance vs delayed labels	Medium	Days to weeks	Low

The detection logic is short:

from scipy.stats import wasserstein_distance
import numpy as np

def prediction_drift_score(reference: np.ndarray, current: np.ndarray) -> float:
    return wasserstein_distance(reference, current)

# reference = predictions from the validation window when the model was promoted
# current = predictions from the last 24h of production traffic
# alert when score exceeds the 99th percentile of bootstrapped baseline scores

The feedback loop is non-negotiable

For expense classification, ground truth arrives when a human approves or corrects the prediction. Median latency is four days. P95 is three weeks.

The per-segment view is what surfaces the failures. Aggregate accuracy stays at 94% while accuracy on a specific Swiss VAT category collapses to 71%. The aggregate view would never have caught it.

# Simplified pipeline component spec
- name: compute-segmented-metrics
  inputs:
    predictions_table: gs://yokoy-ml/predictions/dt={{date}}
    labels_table: gs://yokoy-ml/labels/dt={{date}}
  outputs:
    metrics_table: gs://yokoy-ml/metrics/dt={{date}}
  segments:
    - merchant_category
    - country
    - customer_tier
  resource_request:
    cpu: 4
    memory: 16Gi

The cost: roughly 12 minutes of compute per day on our volume. The value: every regression we caught in the last 18 months was caught here, not by drift monitoring.

Where input drift still earns its place

So compute it, store it, do not alert on it.

A note on LLM-based features

Trade-offs and Limitations

Wasserstein distance has no native interpretation in production units. You bootstrap a baseline and alert on deviation from it. This works but it is not as crisp as "accuracy dropped 3 points."

The Rise of Inference Optimization: The Real LLM Infra Trend Shaping 2026

Lukas Brunner — Sun, 19 Apr 2026 20:43:01 +0000

The large language model space is moving fast, but one trend is quietly defining the next phase of AI: inference optimization. While headlines focus on bigger models and benchmark wins, the real innovation is happening behind the scenes. Teams are no longer asking how to build smarter models. They are asking how to run them efficiently, cheaply, and at scale.

If you care about LLM infrastructure, this shift matters more than the next model release.

Why Inference Optimization Is Taking Over

Training a model is expensive, but it is a one time cost. Inference is forever. Every user query, every API call, every generated token adds to ongoing compute costs. For companies deploying LLMs in production, inference quickly becomes the dominant expense.

This is why optimization is now the priority. Reducing latency, lowering cost per token, and improving throughput directly impacts margins and user experience. A model that is slightly less capable but twice as fast is often the better business decision.

Key Techniques Driving This Trend

Model Quantization

Quantization reduces the precision of model weights, which significantly lowers memory usage and speeds up inference. Moving from 16 bit to 8 bit or even 4 bit precision can unlock major performance gains with minimal quality loss.

This is especially important for edge deployments and cost sensitive applications.

Smart Routing and Model Cascades

Not every query needs a top tier model. Smart routing systems analyze incoming requests and decide which model should handle them. Simple queries go to smaller, cheaper models. Complex ones are escalated.

This approach, often called model cascading, reduces overall costs without sacrificing quality where it matters.

KV Cache Optimization

Key value caching is critical for speeding up long conversations. By reusing previously computed attention states, systems avoid recomputing tokens from scratch.

Efficient cache management can dramatically reduce latency, especially in chat based applications where context grows over time.

Speculative Decoding

Speculative decoding is gaining traction as a way to accelerate generation. A smaller model generates candidate tokens, and a larger model verifies them. If the guess is correct, the system skips expensive computation.

This technique can improve throughput without compromising output quality.

The Tradeoffs You Cannot Ignore

Optimization is not free. Every gain comes with a tradeoff.

Aggressive quantization can degrade output quality
Routing systems can introduce inconsistency
Caching strategies can create stale or repetitive responses

The challenge is finding the right balance for your use case. There is no universal setup. What works for a consumer chatbot may fail in a high accuracy enterprise workflow.

Why This Trend Matters for Builders

For developers and companies, inference optimization is no longer optional. It is a competitive advantage.

Lower costs mean you can serve more users. Faster responses improve engagement. Efficient systems unlock new product experiences that were previously too expensive to run.

In short, infrastructure decisions are now product decisions.

Final Thoughts

The future of LLMs will not be defined by who has the biggest model. It will be defined by who can run models the smartest way.

Inference optimization is where that battle is happening right now. If you are building in this space, this is the layer you cannot afford to ignore.

Focus less on chasing model hype and more on mastering the systems that make those models usable at scale. That is where the real leverage is.

DEV Community: Lukas Brunner

Detecting Silent Model Failure: Drift Monitoring That Actually Works

The mistake I keep seeing

What to monitor instead

The feedback loop is non-negotiable

Where input drift still earns its place

A note on LLM-based features

Trade-offs and Limitations

Further Reading

Detecting Silent Model Failure: Drift Monitoring That Actually Works

The mistake I keep seeing

What to monitor instead

The feedback loop is non-negotiable

Where input drift still earns its place

A note on LLM-based features

Trade-offs and Limitations

Further Reading

The Rise of Inference Optimization: The Real LLM Infra Trend Shaping 2026

Why Inference Optimization Is Taking Over

Key Techniques Driving This Trend

Model Quantization

Smart Routing and Model Cascades

KV Cache Optimization

Speculative Decoding

The Tradeoffs You Cannot Ignore

Why This Trend Matters for Builders

Final Thoughts