DEV Community: Marcus Chen

We shipped a model on a 2-point eval win. It was noise.

Marcus Chen — Tue, 02 Jun 2026 06:33:10 +0000

TL;DR: We promoted a fine-tuned 7B because it beat the incumbent by 2.1 points on our internal eval. Two weeks later we added bootstrap confidence intervals to the harness and found the gain sat well inside the noise band. The model was not better. We just had no way to tell.

The win that wasn't

Our eval suite at Nexus Labs is 840 prompts. Enterprise agent tasks. Each one is scored pass/fail by an exact-match check against a known-good structured output, so every result is a 1 or a 0.

The fine-tuned candidate scored 73.4%. The incumbent scored 71.3%. A 2.1-point lift on a suite that size felt real, so we shipped it to staging and started the rollout paperwork.

It was not real. Or rather, we had zero evidence either way, which is worse, because we acted like we did.

Why a single number lies

An eval run is a sample, not a measurement. Run the same 840 prompts against the same model with any sampling at temperature above 0 and you get a different number. Even at temperature 0, batching order and kernel nondeterminism in vLLM move it.

The math is not subtle. For a pass rate around 0.73 over n=840, the binomial standard error is sqrt(p(1-p)/n), which is about 1.53 points. The standard error of the difference between two such rates is roughly 2.1 points.

So our 2.1-point gap was about one standard error wide. A coin flip dressed up As a result.

Bootstrap instead of hand-waving

The fix is cheap. We resample the per-prompt results and look at the distribution of the difference. Because both models ran the same prompts, we pair them, which cuts the variance compared to treating the two runs as independent.

import numpy as np

# per-prompt correctness, 1/0, aligned by prompt id
old = np.load("old_correct.npy")   # shape (840,)
new = np.load("new_correct.npy")

def paired_bootstrap(a, b, iters=10000, seed=0):
    rng = np.random.default_rng(seed)
    n = len(a)
    diffs = np.empty(iters)
    for i in range(iters):
        idx = rng.integers(0, n, n)
        diffs[i] = b[idx].mean() - a[idx].mean()
    lo, hi = np.percentile(diffs, [2.5, 97.5])
    return diffs.mean(), lo, hi

mean, lo, hi = paired_bootstrap(old, new)
print(f"delta={mean:.3f}  95% CI=[{lo:.3f}, {hi:.3f}]")
# delta=0.021  95% CI=[-0.004, 0.046]

The 95% interval runs from -0.4 points to +4.6 points. It crosses zero. We could not rule out that the new model was slightly worse.

What the numbers actually said

Metric	Incumbent 7B	Fine-tuned 7B
Pass rate	71.3%	73.4%
Paired delta	baseline	+2.1 pts
95% CI on delta	baseline	[-0.4, +4.6] pts
Significant at p<0.05?	baseline	no

Reading the table is the whole point. The headline delta is positive. The interval that contains it includes outcomes where we regressed. You do not ship on that.

What changed in our process

Three rules now gate any model promotion on my team.

First, no promotion without a paired bootstrap CI that excludes zero, or a McNemar test under p<0.05. The raw delta is not allowed in the PR description on its own anymore.

Second, every candidate runs the eval three times. If the three pass rates spread by more than a point at temperature 0, the harness is nondeterministic and we fix that before trusting any comparison. We caught a vLLM max_tokens truncation bug this way that was silently failing 11 long-output prompts on some runs.

Third, when we compare a self-hosted candidate against a hosted reference like gpt-4o-mini, we route both through one gateway so the request shape, retries, and timeouts are identical. We use Bifrost (https://github.com/maximhq/bifrost) for that, since it exposes every provider behind one OpenAI-compatible endpoint and the eval code stops caring who serves the tokens. Same harness, different backend. That removes a confound I used to ignore.

The cost of all this is one extra function and roughly 2x more eval compute. Against the cost of shipping a regression to an enterprise customer, that is nothing.

The deeper problem

840 prompts sounds like a lot. For detecting a 5-point difference, it is fine. For detecting a 2-point difference at 95% confidence, you need closer to 3,000 prompts, and for 1 point you need over 9,000. Most internal evals are too small to resolve the differences people argue about in standups.

So we also report the minimum detectable effect for our suite. Right now ours is about 4.5 points. Anything smaller, we say out loud that we cannot measure it, and we either grow the suite or stop pretending the comparison means something.

Trade-offs and Limitations

Bootstrap CIs assume your prompts are a representative sample of production. They are usually not. A tight interval on a biased suite is confidently wrong, and no amount of resampling fixes the sample.

The paired approach needs aligned per-prompt results, so you have to log at the prompt level, not the aggregate. That is more storage and more plumbing.

And significance is not importance. A real 0.3-point gain can be statistically solid and operationally meaningless. The test tells you the difference exists, not that you should care.

Provider drift broke our regression evals. We pinned versions through Bifrost.

Marcus Chen — Mon, 01 Jun 2026 16:03:19 +0000

TL;DR: Our nightly agent regression suite dropped 4 points on a tool-calling metric with zero code or prompt changes. The cause was a provider silently rotating the model behind a floating alias. We moved eval traffic through Bifrost, pinned exact model strings per provider, and added Prometheus per-model latency so the next drift shows up as a graph instead of a Slack mystery.

I lead the fine-tuning and eval team at Nexus Labs. Series B, enterprise agent automation. We run a nightly suite of about 2,400 adversarial test cases against whatever models our agents call in production. The suite is the contract. If it moves, something changed.

On a Tuesday in April it moved. Tool-call accuracy went from 0.91 to 0.87 overnight. No deploy. No prompt edit. Git was clean.

The model under you is not stable

We were calling a floating alias on a hosted provider. The kind that maps to "the current version" and gets repointed when the vendor ships an update. Our eval harness recorded the alias string, not the resolved version. So the harness thought it was testing the same thing two nights running. It wasn't.

That is the part people skip. You can pin your seed, your temperature, your prompt template, your sampling params. The weights still move under you. A 4-point swing on a contract metric is the difference between shipping and not, and we spent a day and a half bisecting our own code for a bug that lived in someone else's deploy.
The fix is boring. Pin the exact version. Make the gateway enforce it. Alert when the resolved model string changes.

Why a gateway and not just a constant in our config

We already had model names in a config file. The problem is enforcement and visibility, not storage. We needed three things at the call layer: the exact model string sent on every request, a metric tagged by resolved model, and failover so a provider 500 mid-suite does not kill a 90-minute run.

We put Bifrost (https://github.com/maximhq/bifrost) in front. It is a Go gateway, OpenAI-compatible, so our eval client changed by one base URL. Provider and model become an explicit provider/model string in the request, no floating aliases unless we opt into one.

# bifrost config -- explicit versions, no floating aliases
providers:
  openai:
    keys:
      - value: env.OPENAI_KEY
  anthropic:
    keys:
      - value: env.ANTHROPIC_KEY

# eval client now sends fully-qualified model strings:
#   anthropic/claude-sonnet-4-6
#   openai/gpt-4o-mini-2024-07-18
# a dated string cannot be silently rotated under us

The request side stays explicit:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini-2024-07-18",
    "messages": [{"role": "user", "content": "..."}]
  }'

Native Prometheus metrics gave us latency and request counts labeled by model. When the dated string stops resolving because the vendor retired it, the suite fails loud on a 4xx instead of quietly testing a substitute. That is the behavior I want. Fail visible, not silent.

Failover that does not corrupt the eval

A subtler trap: automatic failover is great for production and dangerous for evals. If provider A times out and Bifrost retries on provider B, your eval row now reflects a different model than the column header says. So we scope it. Production keys get fallbacks. The eval virtual key gets retries on the same model only, no cross-provider fallback. Same gateway, two policies.

That distinction matters more than the drift fix itself. A gateway that just works by silently routing around failures is exactly the thing that poisoned our data in the first place.

Honest comparison

We looked at LiteLLM and Portkey before landing here.

Concern	Bifrost	LiteLLM	Portkey
OpenAI-compatible single API	Yes	Yes	Yes
Self-host, no vendor cloud	Yes (Go binary/Docker)	Yes (Python)	Gateway OSS; control plane leans hosted
Per-model Prometheus metrics	Native	Via callbacks/config	Hosted dashboards
Maturity / ecosystem	Newer, fewer integrations	Largest provider list, most battle-tested	Polished hosted UX
Config surface	Web UI + JSON	Python-config heavy	Hosted-first

LiteLLM has the wider provider coverage and far more StackOverflow answers when something breaks at 2am. If you live in Python and want the longest integration list, it is the safe pick. Portkey hosted observability is genuinely nicer out of the box than wiring your own Grafana. Bifrost won for us because it is a single Go process we run ourselves, the OpenAI-compatible surface meant a one-line client change, and the Prometheus labels were exactly the cardinality we wanted without a callback plugin. Different teams, different answer.

Trade-offs and Limitations

A gateway does not detect drift on its own. It records the exact model string; you still have to alert on changes and pin dated versions. If you keep calling floating aliases through Bifrost, you have added a hop and solved nothing.
It's another process in the path. For our eval traffic that is fine, sub-millisecond overhead against multi-second LLM calls. For ultra-latency-sensitive serving you would benchmark it yourself.

And it is younger software. We hit one config-reload quirk early. LiteLLM longer track record is a real argument if you cannot afford to debug a gateway.

Dated model strings also age out. When a provider retires gpt-4o-mini-2024-07-18, our suite breaks loudly and we re-baseline on purpose. That is the point, but it is maintenance, not magic.

The model is the easy part. The thing that moved under us was the infrastructure around it, and the only defense is making every change observable.

Aggregate eval scores hid a 14-point regression in one user segment

Marcus Chen — Mon, 01 Jun 2026 06:32:22 +0000

TL;DR: Our agent eval suite reported 87% pass rate before and after a fine-tune. The aggregate didn't move. One customer segment dropped from 91% to 77% and we shipped it anyway. The fix was stratifying every eval run by segment and gating on the worst slice, not the mean.

I lead the fine-tuning and eval team at Nexus Labs. We build agent automation for enterprise customers. Roughly 40 of them in production, each with their own document formats, tool schemas, and edge cases.

Here's the thing about a single accuracy number. It's an average, and averages lie by construction.

What happened

We fine-tuned a Qwen2.5-7B agent on a fresh batch of tool-calling traces. Standard LoRA run in TRL, nothing exotic. Our eval suite had 1,200 cases. Pass rate before: 87.1%. After: 87.4%. Within noise. We shipped.

Four days later one customer filed a ticket. Their automation was failing on multi-step refund flows. We pulled their slice out of the eval set. 47 cases. The old model passed 43. The new one passed 36. A 14-point drop, completely invisible in the aggregate because that segment was 4% of the total set and the rest had improved slightly.

The new traces over-represented a different customer's invoice format. The model got better at invoices and worse at refunds. The mean stayed flat. Classic.

Stratify everything

The change was small in code and large in discipline. Every eval case now carries a segment tag. The harness reports per-segment pass rates, and CI gates on the minimum slice, not the mean.

# eval_config.yaml
gating:
  metric: pass_rate
  aggregate: min_segment   # not "mean"
  threshold: 0.85
  min_cases_per_segment: 20

segments:
  - refund_flow
  - invoice_parse
  - contract_review
  - escalation_routing

The min_cases_per_segment field matters. A slice with 6 cases swings 16 points if one flips. We flag any segment under 20 cases as low-confidence and don't gate on it, but we still print it. Silent truncation is how you end up trusting a number that's really three coin flips.

Here's the reporting we wired into the run output:

segment            n     before   after    delta
refund_flow        47    0.915    0.766    -0.149  ❌
invoice_parse      210   0.838    0.910    +0.072
contract_review    156   0.885    0.891    +0.006
escalation_route   89    0.831    0.843    +0.011
---
mean (weighted)    1200  0.871    0.874    +0.003

That -0.149 would have blocked the deploy. The weighted mean would have waved it through. Same data, different verdict.

Where the segments come from

You can't tag what you don't capture. We log every production agent call with the customer ID attached, then sample stratified by customer to build eval sets. Our gateway sits in front of the provider calls and writes structured logs we can replay, so building a new slice is a query, not a data-collection project. We run that through Bifrost, which gives us per-request logging we pull into the eval pipeline. Other teams use a sidecar or their own proxy. The point is the customer dimension has to survive into the log, or you can't reconstruct the slice later.

One detail that bit us: we were sampling uniformly at random for the eval set. Big customers dominated. Small customers with weird formats had 5 cases each and got rounded into noise. Stratified sampling with a floor per segment fixed the representation problem before the gating could even help.

Why the mean is the wrong default

A mean assumes every case is interchangeable. In a multi-tenant product they're not. A 14-point regression for one customer is a churn risk even if 39 other customers improved. The business doesn't experience the average. Each customer experiences their own slice.

This is the same reason a single benchmark number tells you almost nothing. MMLU at 0.81 doesn't tell you the model fell apart on the 3% of questions your users actually ask. You have to cut the data along the dimensions that matter to the people paying you.

Comparison

Gating strategy	Catches per-segment regression	False-block rate	Setup cost
Weighted mean	No	Low	Trivial
Unweighted mean	Sometimes	Medium	Trivial
Min segment (floor on n)	Yes	Medium	Moderate
Per-segment + manual review	Yes	Low	High

We run min-segment in CI and route any blocked deploy to a 10-minute human review. The false blocks are real. A small slice flips, CI goes red, and it turns out to be a flaky case. We accept that cost. Shipping a 14-point regression to a paying customer costs more than a few false alarms.

Trade-offs and Limitations

Min-segment gating is noisier than the mean. With 40 segments, the probability that at least one drops by chance on any given run is high, so you will get blocked deploys that aren't real regressions. The min_cases_per_segment floor helps but doesn't eliminate it.

It also doesn't scale to thousands of segments without becoming a triage burden. At some point you cluster segments into families and gate on those instead of every individual customer.

And it tells you a slice regressed, not why. You still need to read the failing traces. The harness points at the wound. It doesn't diagnose it.

Last thing: stratified eval is only as good as your segment definitions. If you pick the wrong dimension to cut on, you'll get clean-looking slices that hide the real variance. We got customer-segment right and missed document-length entirely for two months.

Serving 40 LoRA adapters on one base model: the throughput we got

Marcus Chen — Fri, 29 May 2026 06:32:33 +0000

TL;DR: We fine-tune one LoRA adapter per enterprise customer on top of a single Llama 3.1 8B base. Running them as 40 separate deployments would have cost roughly $24k/month in mostly-idle GPU. Multi-LoRA serving in vLLM put all 40 on two A100s. Numbers and the parts that broke below.

At Nexus Labs we run the fine-tuning and eval team for agent automation. Each enterprise customer gets its own adapter because each has a different tool schema and a different house style for responses. Right now that's 40 customers in production. Rank-16 LoRA, about 42MB per adapter on disk, trained with PEFT and TRL on their own trace data.

The obvious setup is one model server per customer. That's 40 copies of an 8B base. In bf16 the base is around 16GB of weights before KV cache. Forty of those does not fit on anything we can afford, and most customers send fewer than 5 requests a minute. So you're paying for a GPU to sit at 3% utilization. We priced it at about $24k/month across the fleet on reserved A100s. No.

Multi-LoRA: one base, many adapters

vLLM (we're on 0.6.3) loads the base weights once and applies adapter deltas at request time. You turn it on with --enable-lora and register adapters by name. The base sits in GPU memory once. Each adapter is a few MB, so dozens fit in the same box.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --max-loras 8 \
  --max-lora-rank 16 \
  --max-cpu-loras 64 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

A request picks its adapter through the model field:

curl http://localhost:8000/v1/chat/completions \
  -d '{"model": "customer_acme_v3", "messages": [...]}'

--max-loras 8 is the number of distinct adapters that can be active in a single batch on the GPU. --max-cpu-loras 64 is the CPU-side pool that adapters get swapped in from. When a 9th distinct adapter shows up in a batch, vLLM evicts the least-recently-used one back to CPU. That swap costs us 30 to 50ms measured at p50. Swapping from disk instead of the CPU pool is much worse, so size the CPU pool to your real customer count.

The numbers

Two A100 80GB, base loaded once per box, adapters shared. Load tested at 600 req/min across the 40 adapters with a Poisson arrival mix weighted by real customer traffic.

Metric	40 separate deployments	Multi-LoRA, 2x A100
GPUs needed	~40 (or heavy quant + packing)	2
Base weights in memory	40 copies	2 copies
Adapter memory	n/a	~1.7GB total resident
Idle cost / month	~$24k	~$1.2k
p50 latency (256 tok)	410ms	470ms
Cold adapter swap (CPU pool)	n/a	30-50ms
Aggregate throughput	bounded by idle waste	~3,100 tok/s/box

The latency tax is real but small. About 60ms at p50 from the grouped GEMM the multi-LoRA kernel runs when a batch contains several different adapters. For our agent workloads, where a single tool-call turn is 100 to 400 output tokens, that's noise next to the network round trip.

Eval gating, because outputs are not identical

I do not roll out a serving change without an eval gate. Multi-LoRA does not produce bit-identical output to a standalone fine-tuned model. The batched LoRA kernel accumulates differently than the single-adapter path. Greedy decode matched on our set. Sampled decode diverged within tolerance, which is expected, but I wanted it measured, not assumed.

So before cutover we ran each customer's adversarial eval set, 200 tool-call prompts apiece, scoring exact match on tool name plus a JSON-normalized arg comparison. Gate: no regression above 0.5% versus the standalone deployment. Two adapters tripped it. Both turned out to be rank mismatches in how they were exported, not a serving bug. Fixed the export, re-ran, shipped.

In front of the vLLM box we run Bifrost (https://github.com/maximhq/bifrost) as the gateway. It gives us one OpenAI-compatible endpoint, and if the self-hosted box saturates or drops, it falls back to a hosted provider running the generic adapter so a customer gets a degraded answer instead of a 503. It's one gateway option among several; we picked it for the failover behavior.

Trade-offs and Limitations

Eviction thrash. --max-loras 8 means bursty traffic across more than 8 distinct customers in the same window causes constant swapping. If your concurrency exceeds your active-adapter slots, you pay the 30-50ms swap on a chunk of requests. Watch your eviction rate, not just latency.
Uniform rank. Mixing rank 8 and rank 64 adapters wastes the padded buffer, which is sized to the max. We standardized on rank 16 across all customers. If one needs more capacity, it doesn't belong in this pool.
Throughput per adapter drops when many distinct adapters land in one batch, because the kernel does a grouped GEMM instead of one dense matmul. Few adapters per batch, near-dense speed. Many, you lose some.
One base, one tokenizer. Every adapter has to share the same base model and tokenizer. A customer who needs a different base (say a 70B) gets its own deployment. No way around it.
Numerical drift means you own an eval set. If you don't have per-customer regression tests, you can't safely make this swap. The infra savings assume you can prove output parity.

The model was the easy part here. Two A100s instead of forty came down to knowing how many adapters are actually hot at once and sizing the slots to that, then proving the outputs didn't move.

Shadow-testing a fine-tuned 8B against gpt-4o-mini through Bifrost

Marcus Chen — Thu, 28 May 2026 16:03:41 +0000

TL;DR: We fine-tuned a Llama 3.1 8B for invoice line-item extraction. Before flipping production over, we mirrored 14 days of live traffic to both the fine-tune and gpt-4o-mini using Bifrost's load balancing, then diffed outputs offline. The 8B won on accuracy by 3.2 points and cut per-call cost by 71%. The interesting bug: 4% of "wins" were the fine-tune hallucinating a field the base model correctly left null.

Our team at Nexus Labs ships an agent that pulls structured fields out of supplier invoices. The previous version hit gpt-4o-mini for every call. Bill was getting unfun.

I'm not a fan of swapping production models based on benchmark numbers. MT-Bench scores tell you very little about whether your specific eight-field extraction prompt works on the long tail of malformed PDFs that your customers actually send. So we shadow-tested.

The setup

We needed three things wired together:

Mirror live production traffic to a second model without affecting the primary response
Log both responses with a shared request ID
Replay an offline judge over the diffs

We were already running Bifrost in front of OpenAI for spend visibility. Turns out the load balancing config lets you weight providers across a single virtual model name, and the per-request log includes the full input and output payload. That covered the first two.

A trimmed slice of the config we used:

providers:
  primary_extractor:
    weight: 1.0
    model: openai/gpt-4o-mini
  shadow_extractor:
    weight: 1.0
    model: vllm/llama-3.1-8b-extract-v4
    shadow: true

The shadow: true flag is implemented via a custom plugin. The Bifrost README documents the plugin architecture but does not ship a built-in mirror mode. Our plugin sends the shadow request async and discards the response from the client path. Both log records share a trace ID so downstream comparison is a join, not a search.

What we found in 14 days

Fourteen days, 218,400 production requests, mirrored to both targets. The numbers:

Metric	gpt-4o-mini	Fine-tuned 8B
Field-level accuracy (judge)	94.1%	97.3%
Latency p50	480ms	190ms
Latency p99	1.8s	410ms
Cost per 1k requests	$0.42	$0.12
Hallucinated field rate	0.3%	1.1%

The accuracy win is real. The cost win is real. The latency win is mostly because we run the 8B on a single H100 with vLLM continuous batching and there is no network egress.

The hallucination rate is the part that almost killed the migration. The fine-tune confidently filled in vendor_tax_id on 1.1% of invoices where the field genuinely did not exist. The base model returned null. Our judge initially scored the hallucinations as correct because the format was valid. That's a separate post.

What the judge missed

We were using gpt-4o as the offline judge. It graded outputs against the ground-truth JSON. The grader rewarded any non-null field that matched the schema, which meant a plausible-sounding made-up tax ID got partial credit.

We swapped to a stricter judge that compared field-by-field against a held-out human-labeled set of 2,400 invoices. The fine-tune still won, but the margin shrank to 1.8 points. Worth the migration. Not worth the marketing pitch our PM wanted.

Why Bifrost vs LiteLLM or Portkey

I've used all three. Honest comparison:

LiteLLM is fine if all you want is the proxy layer. Easier to drop into a Python script. The plugin story is weaker, so you'd be writing more glue for the mirror behavior we needed.
Portkey has nicer observability dashboards out of the box, and its guardrails feature is more mature than what Bifrost ships today. If your priority is policy enforcement on user-facing chat traffic, look there first.
Bifrost won for us because the Go core handles the request volume without GIL-related weirdness, the plugin hooks let us implement the shadow flag without forking, and the virtual keys model already matched how we track team budgets. The semantic caching feature was not relevant here. Extraction prompts are too input-specific.

I'd switch tomorrow if Portkey shipped a documented mirror primitive and a Go core. They haven't.

Trade-offs and Limitations

The shadow approach doubles your inference cost during the test window. We ran for 14 days, which felt long. Five would have been enough for the distribution, but extraction has weekly seasonality (Mondays look different) so we wanted two full cycles.

vLLM on a single H100 fits our throughput. If your shadow target is a 70B model you'd need cluster routing, and Bifrost's clustering is enterprise-only. The README is explicit about that. Plan accordingly.

The judge problem cost us a week of confusion. Run your judge against a small human-labeled set first. If it agrees with humans below 90%, the judge is the bottleneck, not the model.

One last thing. Shadow traffic with the same trace ID means your APM tool sees double the spans. Filter those out at the collector or your dashboards lie.

Continuous batching wrecked our p99 latency. Here's the trace.

Marcus Chen — Thu, 28 May 2026 06:33:12 +0000

TL;DR: We turned on vLLM continuous batching for a throughput win and watched p99 latency 8x in the wrong direction. Long prefills were stalling decodes in the same forward pass. Chunked prefill and a tuned max_num_batched_tokens got the SLO back at the cost of ~11% of the throughput gain.

We run Llama 3.3 70B as the routing brain for our agent platform at Nexus Labs. ~14 internal services hit it. SLO is 2s p99 for the single-turn routing call.

Last month we flipped on vLLM 0.7's continuous batching to push more requests through our 4xH100 box. p50 dropped from 340ms to 190ms. We were happy for about 36 hours.

Then the latency dashboard turned red.

What we actually saw

p99 went from 1.2s to 9.8s on the routing endpoint. p50 was still good. p99.9 was unprintable.

The first alert came off our routing service's p99 panel. We checked the upstream load balancer. Healthy. Then the model server CPU and GPU. Healthy by every coarse metric. GPU utilization was 81%, not saturated. KV cache hit rate held at 67%. The Prometheus exporter from vLLM showed something stranger: vllm:time_per_output_token_seconds had widened from 32ms to 380ms during peak. The model itself wasn't slow. The scheduler was making everyone wait.

Long requests with 4k+ token prefills were eating decode slots. Short single-turn routing calls were starving behind them. The forward pass would dedicate ~60ms to a prefill chunk for one user's request, and 23 in-flight decode streams would block on it.

That's the contract of naive continuous batching. Prefill and decode share one forward pass. A big prefill stops everyone.

The fix

vLLM ships chunked prefill. It splits a large prefill into ~512-token chunks and interleaves them with decode steps. The tradeoff: total throughput per long request goes down. In exchange, decode never stalls for more than one chunk worth of time.

The other knob is max_num_batched_tokens. Set too high and you reintroduce the stall. Set too low and you starve throughput. We landed at 4096 for our workload after a sweep.

# vllm config that ended up in prod
model: meta-llama/Llama-3.3-70B-Instruct
tensor_parallel_size: 4
max_model_len: 8192
enable_chunked_prefill: true
max_num_batched_tokens: 4096
max_num_seqs: 96
gpu_memory_utilization: 0.92
swap_space: 16

Before and after

Metric	No batching	Naive CB	+ chunked prefill
p50 latency	340ms	190ms	215ms
p99 latency	1.2s	9.8s	1.4s
p99.9 latency	2.1s	27s	3.1s
Tokens/sec (cluster)	2,650	4,820	4,310
Cost/1M output	$0.74	$0.41	$0.46

We paid back ~11% of the throughput win. We bought back the SLO. Cheap trade.

Things that didn't help

We tried priority lanes where small requests jump the queue. It cut p99 to 5.2s but cratered p99 for the long requests instead of solving the underlying scheduling problem. Routing them to separate replicas would have worked, but doubled our GPU footprint. Not worth it for our traffic mix.

We tried bumping max_num_seqs to 256 thinking more concurrent decodes would amortize prefills. It made things worse. KV pressure spiked, eviction churn ate compute.

We tried separating ingress by content length at the gateway layer. Under 1k tokens to one pool, the rest to another. Worked on paper. In practice the small pool got 92% of traffic and we ran out of headroom there. Bin packing prompts isn't free either.

We added a circuit breaker upstream that sheds to a hosted provider when our internal p99 crosses 3s. We pipe everything through Bifrost so the failover is one config change instead of a deploy. It catches the edge cases when prefill-heavy traffic spikes faster than autoscaling reacts.

Trade-offs and Limitations

Chunked prefill is not free. For workloads with very long prompts and short decodes (think doc-QA over 32k context), per-request latency goes up by 15-25%. If that's your hot path, you'd want to split traffic by class and run two pools with different configs.

max_num_batched_tokens is workload-specific. The number we landed on is wrong for someone with a different prompt distribution. There's no shortcut. You run the sweep.

Continuous batching also makes p99 noisier across deployments. A neighbor service pushing a new feature with 8k prompts can hurt yours. The isolation story at the vLLM layer is real but not airtight. We file this under "things our k8s admission controller now checks."

What the eval suite said

The boring point. None of this showed up in offline eval. Eval measured correctness on a fixed batch size. Production measures tail latency under realistic prompt mix. If you only have the first one, you'll ship the dashboard regression we shipped.

We added a load-shape replay step to our deployment pipeline two weeks ago. It replays a sampled 5-minute window of real traffic shape against the candidate. Catches this class of regression before it touches real users.

Virtual keys per tenant: ditching our custom LLM billing layer

Marcus Chen — Wed, 27 May 2026 16:02:19 +0000

TL;DR: We had 11,247 lines of Python middleware handling per-tenant LLM cost attribution, rate limiting, and provider failover. Replaced about 60% of it with Bifrost's virtual keys and governance features. Some honest gaps remain, which is why this is a writeup and not a sales pitch.

The setup we inherited

Nexus Labs runs enterprise agent automation. Each customer gets isolated workloads. Each workload makes between 200 and 50,000 LLM calls per day across OpenAI, Anthropic, Bedrock, and Vertex.

When I joined, we had a Python middleware doing four things at once: API key rotation per provider, per-tenant rate limits in Redis, cost attribution via request tagging, and fallback logic when a provider returned 429s.

11,247 lines of Python. Three engineers had touched it. Two had left. One of them had encoded their team-internal pricing assumptions inline. Every model deprecation became a sprint.

What we actually needed

Three things, in priority order:

Per-customer spend caps that don't require a deploy to update.
Provider failover that survives Anthropic going down for 23 minutes (it did, last March).
Cost data we don't have to reconstruct from CloudWatch logs.

I evaluated three gateways before picking one. Here is the comparison after running each through a 2-week eval against our actual traffic shape.

Feature	Bifrost	LiteLLM	Portkey
Per-tenant virtual keys with budgets	Native	Plugin/config	Native
Self-host without external deps	Yes	Yes	Limited
OpenAI-compatible API for all providers	Yes	Yes	Yes
Built-in Prometheus metrics	Yes	Yes (newer)	Hosted preferred
Semantic caching	Yes	Yes	Yes
MCP gateway	Yes	No	Limited
Built-in web UI for config	Yes	Limited	Cloud-first

LiteLLM was the real contender. Larger community, more battle-tested in production for some workload shapes. Where it lost for us: setting up hierarchical budgets across customer to team to workload tiers required more YAML wrangling than we wanted, and the failover behavior on streaming requests was less predictable under our tests.

Portkey was strong on dashboards. We didn't want a hosted dependency for our cost control path.

What changed

The piece that surprised me most was the virtual keys model. From the docs (governance/virtual-keys), every tenant gets a virtual key. The key carries the budget cap, rate limit, allowed providers, and allowed models. Our orchestrator stopped caring about provider routing entirely.

Config that replaced 4,200 lines of Python:

virtual_keys:
  - id: vk_acme_prod
    customer_id: acme_corp
    budget:
      max_per_month_usd: 12000
      reset_duration: monthly
    rate_limit:
      requests_per_minute: 600
    allowed_providers:
      - openai
      - anthropic
      - bedrock
    fallbacks:
      - provider: openai
        model: gpt-4o
      - provider: anthropic
        model: claude-sonnet-4-6
      - provider: bedrock
        model: anthropic.claude-sonnet-4-6

Our orchestrator now does one thing: pick a virtual key based on tenant. Send the request. Done.

The numbers

Before:

11,247 LOC in gateway_middleware/
p95 added latency from middleware: 47ms
Mean time to add a new model: 2 days (testing, rollout, monitoring)

After 4 months:

4,108 LOC remaining (mostly business logic we still need)
p95 added latency from Bifrost in front: 8ms
Mean time to add a new model: under an hour

The latency number was the biggest surprise. Bifrost is Go. Our middleware was Python doing synchronous Redis calls. We knew that was a problem. Solving it wasn't on the roadmap.

Trade-offs and Limitations

This isn't free.

Migration was harder than the docs suggest. Our cost attribution data didn't map cleanly. We had legacy fields like team_internal_billing_code baked into every log. Mapping these to virtual key metadata took a full sprint, and the team still grumbles about it.

Semantic caching is risky for our workload. Our agents call LLMs with tool results embedded in prompts. Two prompts that look 92% similar can require very different responses. We disabled semantic caching for the agent path. Enabled it only for our content generation path, where we saw a 31% hit rate.

MCP gateway integration is newer than the rest. We use it for filesystem access from a customer-facing automation agent. Works fine. But debugging when a tool call fails requires more log digging than the rest of the platform.

No native cost-anomaly alerting yet. Budget caps work. But "this customer's usage spiked 3x in 2 hours" is still wired up via Prometheus alerts and PagerDuty by hand. Portkey has this in their hosted product. If real-time anomaly alerts are your top requirement, weight that.

What I'd tell a peer team

If you have one provider and one customer, you don't need this. Use the provider's SDK.

If you have 3+ providers, multiple customer tiers, and someone on your team has written class CostTrackingMiddleware more than once, evaluate. Spin up the Docker container (quickstart). Point staging traffic at it for a week. Look at the metrics. Decide.

The model is the easy part. Cost attribution is the part that wakes you up at 2am when a customer's bill is wrong.

LLM-as-judge variance broke our DPO training signal for 3 weeks

Marcus Chen — Wed, 27 May 2026 06:31:57 +0000

TL;DR: Our DPO pipeline used a single LLM as the preference judge. Training reward climbed every run. Production accuracy fell 4 points. The judge was flipping its own labels 28% of the time at temperature 0.

The setup

Nexus Labs ships agents that book travel, file expenses, process insurance claims. Eight engineers on my fine-tuning team. We run DPO on Qwen2.5-32B, target latency under 800ms p95 on a single H100.

Our preference data pipeline:

2,400 prompts sampled from production traces per cycle
4 completions per prompt from the current checkpoint
GPT-4o-mini grades pairwise preferences against a 6-axis rubric
TRL DPO, 3 epochs, lr 5e-7, beta 0.1

Standard recipe. Worked fine for two months.

What we saw

Week 9. Training loss curves looked clean. Reward margins grew run over run. Held-out eval reward climbed 0.62 → 0.71. Internal dashboards were green.

Then product filed tickets. Latency was fine. Tool use accuracy on our production traffic mirror was down 4 points against the pre-DPO baseline. The thing we shipped to make the agent better made it worse.

We trusted offline eval. We were wrong.

The investigation

I rebuilt the judge call as a deterministic test. Same prompt, same two completions, GPT-4o-mini at temperature 0. Fired the API 50 times in a row.

The judge flipped its preference 14 of 50 times. 28% self-disagreement on a single pair.

That number alone should have killed the project. We had built a training signal on top of a weighted coin.

Ran the test across 200 prompt pairs. Median self-disagreement was 19%. The tail was worse. 8% of pairs had over 40% flip rates, and those pairs were exactly the ambiguous multi-step agent traces we cared about most.

What was actually happening

DPO gradients care about margin. When labels are noisy, the model still gets a gradient, but the direction is garbage. Over thousands of pairs you converge on whatever spurious feature the judge weights at temperature 0. Which, surprise, is not what end users want.

Our offline reward went up because the model learned the judge's quirks. Production accuracy dropped because the quirks weren't the task.

The fix

Three changes.

# preference_judging.yaml
judges:
  - provider: anthropic
    model: claude-sonnet-4-6
  - provider: openai
    model: gpt-4o-2024-11-20
  - provider: google
    model: gemini-2.5-pro
consensus:
  min_agree: 2
  drop_pair_if_split: true
sampling:
  judges_per_pair: 3
  rotate_completion_order: true

Three judges, 2-of-3 majority. Drop the pair if split. We lose 18% of pairs. Acceptable.
Rotate completion order per judge. Position bias was ~7% on its own. Sonnet was closer to 2%, GPT-4o-mini was the worst offender.
Bootstrap CIs on the eval set. Report reward with a 95% interval, not a point estimate. Half of our prior "improvement" was inside the noise floor.

The judge fleet routes through Bifrost (https://github.com/maximhq/bifrost). One OpenAI-compatible endpoint, automatic fallback when a provider degrades, per-judge token accounting in one place. We were already running three providers for app traffic, so the judge pool was a config change.

Numbers after the fix

| Metric | Single judge | 3-judge consensus |
|---|---|
| Judge self-consistency | 72% | 94% |
| Production tool-use accuracy | -4.0 pts | +2.1 pts |
| Training pairs retained | 100% | 82% |
| Cost per 10k pairs (USD) | $11 | $34 |
| Eval-to-prod Spearman correlation | 0.31 | 0.78 |

Cost tripled. The signal went from misleading to useful. We take that trade every cycle.

Trade-offs and limitations

This isn't free and it isn't a silver bullet.

Judge cost. 3x judges plus pair retries. Budget for it before you propose this to a director.
Consensus isn't truth. Three judges can agree on the wrong thing. We still sample 5% of pairs for human review weekly. That review process has caught two systematic biases all three LLM judges shared. Probably trained on overlapping data.
Latency. Preference labeling is no longer a same-afternoon job. Two-day turnaround on a full cycle now. Plan the data pipeline schedule around it.
Bad rubric, no rescue. If your scoring criteria don't match what users care about, ensembling judges won't save you. We rewrote the rubric twice during this work.
Position bias varies by model. Don't assume. Measure.

The deeper point. Most teams I talk to treat the judge as an oracle and the model as the unknown. It's backwards. The model converges on whatever target you point it at. If the target wobbles, the model wobbles with it, and you won't see it in your reward curve.

We spent three weeks training a model to imitate a noisy judge. The model worked. That was the bug.

Token-level eval harness for tool-calling agents: what we wired up

Marcus Chen — Tue, 26 May 2026 16:03:35 +0000

TL;DR: We replaced our "did the agent finish the task" pass/fail eval with a token-level harness that scores tool selection, argument shape, and recovery behavior separately. Pass rate went from a single 73% number to four signals that actually tell us what broke. Bifrost sits in front as the provider switch so the same eval runs against four models without rewriting the harness.

At Nexus Labs we run agent automation for enterprise workflows. Twelve people on the team, around 40 tool definitions across the production agents, mix of GPT-4.1, Claude Sonnet 4.6, and a fine-tuned Qwen3 32B we serve ourselves on vLLM.

Last quarter our eval suite told us the new agent build was "72% passing." Shipped it. Two customers reported the agent was silently picking the wrong tool and confabulating success. Pass rate didn't catch it because the final assistant message looked fine.

So we rebuilt the harness.

The four signals

End-to-end pass/fail is one number that hides everything. We split it.

Signal	What it measures	Failure mode it catches
Tool selection accuracy	Did the agent pick the right tool at step N	Picks `search_db` when it should call `query_api`
Argument F1	Token-level F1 on tool arguments vs gold	Right tool, wrong filter or off-by-one date
Recovery rate	After a tool returns an error, does the next step make sense	Loops the same failing call three times
Trajectory length delta	Steps taken vs minimum needed	Wanders for 11 steps on a 3-step task

None of these are novel on their own. The point is having all four on every run, per-model, per-tool. When our 72% number dropped to 68% on the new build, the breakdown showed argument F1 collapsed on date-range tools while selection stayed flat. That's a tokenizer regression on the fine-tune, not a reasoning regression. Different fix.

The eval loop

We needed to run the same suite against four models without writing four clients. Bifrost handles that. One OpenAI-compatible endpoint, swap the model string.

eval_targets:
  - name: gpt-4-1
    model: openai/gpt-4.1
  - name: sonnet-4-6
    model: anthropic/claude-sonnet-4-6
  - name: qwen3-internal
    model: ollama/qwen3-32b-tools-v4
  - name: cerebras-llama
    model: cerebras/llama-3.3-70b

gateway:
  base_url: http://bifrost:8080/v1
  headers:
    x-bf-virtual-key: ${EVAL_VK}

The virtual key matters. We give the eval harness its own budget cap through Bifrost's governance so a runaway nightly run can't burn $4K on Anthropic before anyone notices. Last month it did exactly that, capped at $200, dropped the rest of the requests. Email at 3am instead of a Slack thread the next morning.

Semantic caching off for eval runs. Obvious reason: cached responses defeat the point. Bifrost lets you disable it per-request via header, docs here.

Argument F1, in code

The non-obvious signal is argument F1. Most harnesses do exact-match on the JSON, which is brittle ("2026-05-26" vs "May 26, 2026" both call the right API but exact-match scores zero).

def arg_f1(predicted: dict, gold: dict) -> float:
    pred_tokens = tokenize_args(predicted)
    gold_tokens = tokenize_args(gold)
    if not pred_tokens or not gold_tokens:
        return 0.0
    tp = len(pred_tokens & gold_tokens)
    if tp == 0:
        return 0.0
    precision = tp / len(pred_tokens)
    recall = tp / len(gold_tokens)
    return 2 * precision * recall / (precision + recall)

tokenize_args flattens nested JSON and normalizes dates, IDs, and known enums. It's 80 lines. We diff against gold per-key and weight required keys higher than optional ones.

This caught the Qwen regression. Selection accuracy was 91%, argument F1 dropped from 0.84 to 0.61 in one fine-tune iteration. Turned out the tokenizer was splitting ISO dates differently after we added a new SFT batch.

Why Bifrost vs LiteLLM or Portkey

Honest comparison. We tried all three.

	Bifrost	LiteLLM	Portkey
Provider count	23+	More (50+)	~40
Self-hosted free tier	Yes	Yes	Limited
Built-in virtual keys with budget caps	Yes	Plugin/proxy config	Yes
Native Prometheus metrics	Yes	Via callback	Hosted-first
Latency overhead (our measurement, p50)	~1ms	~3-4ms	n/a (hosted)

LiteLLM has more providers and a larger community. If you need a niche provider that's the safer bet. Portkey's hosted UX is more polished if you don't want to run anything. We picked Bifrost because the Prometheus integration is native (we already run Prometheus + Grafana) and the overhead was the lowest in our test. Your tradeoffs may differ.

Trade-offs and Limitations

Token-level argument F1 needs gold labels. We hand-labeled 1,200 trajectories. That's not free. If your agent universe is huge and changing weekly, this approach gets expensive.

Recovery rate is the noisiest signal. It needs a judge model to score whether the next step "makes sense" given the error, and judge models disagree with humans about 8% of the time in our spot checks. We use it as a trend indicator, not a gate.

Adding a gateway adds a hop. ~1ms in our setup, but if your eval is running 50K trajectories overnight, that's still real wall-clock time. We accept it because the centralized rate limiting and budget caps are worth more than the millisecond.

Bifrost's MCP gateway is enterprise-only. We use the open-source build, so for MCP tool routing we still wire that ourselves outside the gateway.

Prefix caching in vLLM under multi-tenant agent traffic

Marcus Chen — Tue, 26 May 2026 06:35:20 +0000

TL;DR: We turned on vLLM's prefix cache for our agent workloads at Nexus Labs and watched TTFT drop from 480ms to 110ms on one tenant and stay exactly the same on another. The split wasn't about traffic volume. It was about how each team templated their system prompts.

The setup

Our fine-tuning team serves 14 enterprise agents through a shared inference cluster. Four H100 nodes, vLLM 0.6.x, Qwen2.5-32B as the workhorse model. Traffic is bursty. One customer's nightly workflow can hit 8k requests in twenty minutes while another trickles through 30 calls an hour.

Before turning on prefix caching, average TTFT across the cluster sat at 410ms p50, 1.2s p95. Cost wasn't the urgent problem. Latency was, because agents loop. A 400ms TTFT on a 12-step plan turns into 4.8 seconds of dead time before the user sees anything.

What the cache actually does

vLLM's prefix cache keeps KV blocks for tokens it has already processed. If a new request shares a prefix with something in the cache, those blocks get reused instead of recomputed. The unit is a block (16 tokens by default), so caching is greedy at block boundaries.

If your system prompt is 1,024 tokens and identical across requests, you skip prefill for 1,024 tokens. At Qwen2.5-32B prefill speeds, that's roughly 90 to 110ms saved per call on our hardware.

Where it worked

Tenant A's agent uses a fixed system prompt assembled at deploy time. Same 1,847 tokens for every request, byte-for-byte. After we flipped enable_prefix_caching=True:

TTFT p50: 480ms → 110ms
TTFT p95: 1.4s → 280ms
GPU prefill compute dropped by 38%

Their hit rate ran around 94% steady-state. The 6% misses were cold starts after pod restarts.

Where it didn't

Tenant B's agent rebuilds its system prompt every call. They inject the current timestamp, a session UUID, and a hash of recent tool outputs into the first 200 tokens. Looked stable on paper. In practice, every request had a unique prefix starting at token 47.

vLLM caches at block granularity. One differing token in the first block invalidates everything after it. Tenant B's hit rate: 0.3%.

We didn't catch this in staging because our staging traffic replays canned prompts. The diff between tenants only showed up under real traffic.

The fix for Tenant B

I talked their team into pushing the volatile fields to the end of the prompt. Took two hours of refactoring on their side. After:

TTFT p50: 510ms → 145ms
Hit rate: 0.3% → 87%

Then they asked why nobody mentioned this in the vLLM docs. The docs do mention it. Nobody reads docs when defaults already look fine on the neighboring tenant.

Config

# vllm serve flags we landed on
--model Qwen/Qwen2.5-32B-Instruct
--enable-prefix-caching
--block-size 16
--gpu-memory-utilization 0.92
--max-num-seqs 256
--swap-space 16
--preemption-mode recompute

--preemption-mode recompute matters under memory pressure. We tried swap and watched the cache thrash when bursts hit. Recompute throws cache blocks away cleanly instead of evicting them to CPU and back.

Comparison

Workload	Prompt structure	Hit rate	TTFT p50 before	TTFT p50 after
Tenant A (fixed)	Static 1,847-token prefix	94%	480ms	110ms
Tenant B (before fix)	Volatile fields at token 47	0.3%	510ms	505ms
Tenant B (after fix)	Volatile fields moved to tail	87%	510ms	145ms
Internal eval pipeline	Per-eval unique prompts	4%	390ms	380ms

The eval pipeline column is honest. Prefix caching does nothing for workloads where every prompt is genuinely unique. We left it on anyway because the overhead is negligible.

For routing across providers when we burst beyond self-hosted capacity, we run a small gateway in front (Bifrost is what we landed on, but the principle works with any of them). The local cache only helps for traffic that lands back on our own node, not the failover path.

Trade-offs and limitations

The cache costs GPU memory. We reserved roughly 14% of HBM for cached blocks at our max-num-seqs setting. That's tokens we can't use for batch concurrency. Worth it for us because TTFT mattered more than throughput. Not worth it if you're optimizing for tokens-per-second on offline batch.

Cache invalidation is binary at block boundaries. A one-token change at position 0 kills the whole prefix. No fuzzy matching. Semantic-caching products exist for that, but they're a different beast. They cache responses, not KV state, and the failure modes differ.

The cache is per-node. We have four nodes behind a round-robin LB, so the same prompt hits a cold cache 75% of the time on first contact. We considered sticky routing by prompt hash. Decided the complexity wasn't worth a 200ms improvement on first-contact latency. Maybe later.

The model is the easy part. Knowing where your tokens go is the hard part.

We Audited Our Agent Tool-Call Traces. Half Our Eval Data Was Garbage.

Marcus Chen — Mon, 25 May 2026 16:03:44 +0000

TL;DR: We pulled 41,000 production agent traces at Nexus Labs to build a fine-tuning dataset. After a manual audit of 1,200 of them, ~48% were unusable: tool calls that "succeeded" but returned wrong data, retries masking provider failures, and silent fallbacks that changed which model answered. Putting Bifrost in front of the agent fleet fixed the trace problem more than any sampling strategy we tried.

We run an enterprise agent product. Sales-ops automations mostly. Each user task ends up as a chain of 8-40 tool calls across a planner model, a worker model, and roughly 12 internal tools.

For the last quarter my team has been building a fine-tune dataset from real traces. The plan was straightforward. Pull successful task completions. Filter by user thumbs-up. Use the trace as the training signal.

It did not work.

What "successful" actually meant in our traces

The first audit pass was 1,200 traces, two engineers, three weeks. We tagged each trace as "clean", "noisy", or "corrupted".

Category	% of traces	What it meant
Clean	52%	Tool calls returned correct data, model picked the right next step
Noisy	31%	Right answer eventually, but with hidden retries, fallback to a different model, or stale cache hits
Corrupted	17%	Trace claimed success, output was wrong. User had not noticed yet.

The noisy category is the one that broke me. We had been treating these as gold-standard data. A trace where the planner called crm_lookup, got a 500, retried twice, then succeeded on a fallback Anthropic key while the original trace span still pointed at OpenAI gpt-4o. The training pair we would have generated: "given this user input, output this tool call sequence." But the sequence was the result of three providers and two model versions stitched together. No reproducibility.

Worse: nothing in our trace told us which model actually produced the final answer. We had a model field. It logged whichever provider was configured at request start.

Why we ended up putting a gateway in front of everything

We tried two things first. Both partial fixes.

The first was logging at the application layer. Wrap every provider call, log model, latency, retry count, fallback path. This works until you have four services calling four SDKs with four retry policies. Our Python service used the official openai client. Our Go service used a hand-rolled HTTP client. The TypeScript planner used Vercel AI SDK. Three different definitions of "retry".

The second was forcing all traffic through LiteLLM. It got us to a unified call surface but the observability was thin for our needs, and the failover behaviour was harder to reason about under load. Not a knock on LiteLLM, it just was not the shape we wanted.

We migrated the fleet behind Bifrost about five months ago. Two reasons specific to our problem:

The Automatic Fallbacks config makes the fallback chain a first-class object. When a request fails over from Anthropic to Bedrock, that is in the response metadata. Not in three different log lines you have to join.
Native Prometheus metrics (observability docs) meant bifrost_requests_total is tagged by the actual provider that served the request, not the one we asked for.

Here is a chunk of the config that mattered for trace cleanup:

providers:
  openai:
    keys:
      - value: env.OPENAI_API_KEY_1
        weight: 0.7
      - value: env.OPENAI_API_KEY_2
        weight: 0.3
  anthropic:
    keys:
      - value: env.ANTHROPIC_API_KEY

fallbacks:
  - model: openai/gpt-4o
    fallback_to:
      - anthropic/claude-sonnet-4-6
      - openai/gpt-4o-mini

logging:
  include_fallback_chain: true
  include_provider_actual: true

The two include_* flags meant every trace span we emitted downstream had a deterministic answer to "who served this token". Our corrupted-trace rate on the next 5,000 sampled dropped from 17% to under 3%.

What the audit actually changed about our fine-tuning

We stopped using user thumbs-up as the primary filter. Thumbs-up correlates with "user got what they wanted eventually", not "the model made the right call". Now the filter is:

Single-provider, single-model trace (no fallback fired)
No retry on any tool call
Tool call result schemas validated post-hoc against a recorded ground truth
Span timing within 1.5x median for that task class

That filter throws away about 71% of our raw traces. Painful. But the 29% that survives is data we can actually train on.

Trade-offs and limitations

Honest take on what this did not solve.

Bifrost is not a debugger. It tells you which provider served the request and whether a fallback fired. It does not tell you whether the tool result was correct. We still need the post-hoc schema validation pass.
Semantic caching (docs) made the corruption worse before it got better. Cache hits looked like fresh model calls in our old logging. We had to explicitly tag cached responses in the trace pipeline. Once tagged, fine, but the default was confusing.
LiteLLM has a larger provider list at the long-tail. If you need niche providers, check both before committing.
Portkey's prompt management UI is nicer. We do prompt management elsewhere so it did not matter for us. If you want one tool for both, Portkey is worth a look.
The MCP gateway feature (docs) is interesting but we have not put it in production. Cannot vouch for it yet.

The model is the easy part. The infrastructure around the trace is where your eval dataset lives or dies.

Why Your LLM Eval Harness Is Lying to You (And How to Fix It)

Marcus Chen — Fri, 22 May 2026 06:32:58 +0000

TL;DR: Most eval harnesses I see in production are measuring the wrong thing. They report 87% pass rate on a static suite that hasn't been touched in four months, while the model silently regresses on the queries that actually matter. Here's how we restructured ours at Nexus Labs after a bad week in February.

We shipped a fine-tuned Llama 3.1 70B variant in late January. Eval score: 91.2 on our internal suite. Two weeks later, support tickets spiked. Customers running multi-step agent workflows were getting truncated tool calls roughly 12% of the time. Our eval suite caught zero of these.

The suite wasn't broken. It was answering a question nobody had asked in months.

The static-suite trap

Here's the pattern I keep seeing. Team builds an eval set of 500 examples around the time they ship v1. Each example gets a reference answer and a string-match or embedding-similarity check. The suite becomes the source of truth. CI gates on it. Dashboards graph it. Nobody questions it.

But your traffic distribution shifts. New customer onboards with a different query pattern. A prompt change upstream alters tool-call frequency. The suite still passes because the suite hasn't moved.

We pulled three months of production traces and binned them by intent cluster. The original eval suite covered four of the eleven clusters that showed up in real traffic. The four it covered were the easiest ones.

What we actually changed

Three things. None of them clever.

1. Replay-based eval, refreshed weekly. We sample 2,000 real production traces per week, strip PII, and run them through the candidate model. We compare structured outputs (tool calls, JSON fields) against the production response using exact match on tool name plus a learned judge for arguments. Free-form text gets a pairwise preference check against the current production model using a separate judge model.

2. Cluster-stratified sampling. Embed every trace with text-embedding-3-large, cluster with HDBSCAN, sample proportionally. This stops the eval from being dominated by the one chatty customer who sends 40% of traffic.

3. Adversarial slices owned by humans. Our support team flags any ticket that traces back to a model failure. Those traces get added to a permanent adversarial set. That set grows. It never shrinks. Currently sitting at 847 examples and climbing.

eval_config:
  replay:
    sample_size: 2000
    window_days: 7
    strip_pii: true
    cluster_method: hdbscan
    min_cluster_size: 15
  judges:
    structured: exact_match_with_arg_judge
    freeform: pairwise_preference
    judge_model: claude-sonnet-4-6
  adversarial:
    path: ./evals/adversarial_permanent.jsonl
    weight: 3.0
  gates:
    regression_threshold: 0.02
    adversarial_floor: 0.85

The weight: 3.0 on adversarial is deliberate. Those examples represent real customer pain. A 1% regression on adversarial costs us more than a 1% regression on the easy cases.

Routing the eval traffic

Running 2,000 traces against a candidate model plus a judge model plus the production baseline gets expensive fast. We were burning $400/week on judge calls alone before we got serious about caching and routing.

Two things helped. First, semantic caching on the judge prompts. The same trace evaluated twice against the same model pair should not cost twice. Second, we route across providers based on per-token cost for the judge role specifically. We use Bifrost (https://github.com/maximhq/bifrost) for this because it gives us one OpenAI-compatible endpoint and lets us shift judge traffic between Anthropic and Google without touching the eval code. LiteLLM works similarly if that's already in your stack.

Cost dropped to $140/week. Same coverage.

Comparison: what we tried

Approach	Coverage of real traffic	Maintenance cost	Catches silent regressions
Static curated suite	Low (drifts fast)	Low	Rarely
Pure replay	High	Medium	Sometimes (misses rare-but-critical)
Replay + cluster sampling + adversarial	High	Medium-high	Yes
LLM-judge-only with no replay	Medium	Low	Inconsistent

Trade-offs and limitations

Replay-based eval has real problems and I don't want to undersell them.

Judge models are not ground truth. Pairwise preference between two model outputs is noisy. We run each comparison three times with temperature 0.3 and take majority vote. Even then, agreement with human raters sits around 78% on our adversarial slice. Useful, not authoritative.

PII stripping is fragile. We use a regex stack plus a small NER model. We still find leakage occasionally during audits. If your domain has strict data handling rules, you may need synthetic replays instead of real ones, which loses some of the distributional fidelity that makes this work.

Replay assumes today's traffic looks like tomorrow's. For a stable product, fine. For one shipping new features weekly, you're always one release behind.

And the adversarial set has a selection bias. We only add examples that humans flagged. Failures nobody noticed don't make it in. We try to compensate by manually sampling 50 random traces per week for human review, but we're not closing the loop completely.

What hasn't worked

Tried benchmark suites like MT-Bench and HELM as our primary gate. Useless for our domain. They measure general capability. We don't ship general capability. We ship agent reliability on a narrow task surface.

Tried a single LLM-as-judge with one rubric. Too much variance. Rubric drift between runs was higher than the signal we were trying to measure.

DEV Community: Marcus Chen

We shipped a model on a 2-point eval win. It was noise.

The win that wasn't

Why a single number lies

Bootstrap instead of hand-waving

What the numbers actually said

What changed in our process

The deeper problem

Trade-offs and Limitations

Further Reading

Provider drift broke our regression evals. We pinned versions through Bifrost.

The model under you is not stable

Why a gateway and not just a constant in our config

Failover that does not corrupt the eval

Honest comparison

Trade-offs and Limitations

Further Reading

Aggregate eval scores hid a 14-point regression in one user segment

What happened

Stratify everything

Where the segments come from

Why the mean is the wrong default

Comparison

Trade-offs and Limitations

Further Reading

Serving 40 LoRA adapters on one base model: the throughput we got

Multi-LoRA: one base, many adapters

The numbers

Eval gating, because outputs are not identical

Trade-offs and Limitations

Further Reading

Shadow-testing a fine-tuned 8B against gpt-4o-mini through Bifrost

The setup

What we found in 14 days

What the judge missed

Why Bifrost vs LiteLLM or Portkey

Trade-offs and Limitations

Further Reading

Continuous batching wrecked our p99 latency. Here's the trace.

What we actually saw

The fix

Before and after

Things that didn't help

Trade-offs and Limitations

What the eval suite said

Further Reading

Virtual keys per tenant: ditching our custom LLM billing layer

The setup we inherited

What we actually needed

What changed

The numbers

Trade-offs and Limitations

What I'd tell a peer team

Further Reading

LLM-as-judge variance broke our DPO training signal for 3 weeks

The setup

What we saw

The investigation

What was actually happening

The fix

Numbers after the fix

Trade-offs and limitations

Further Reading

Token-level eval harness for tool-calling agents: what we wired up

The four signals

The eval loop

Argument F1, in code

Why Bifrost vs LiteLLM or Portkey

Trade-offs and Limitations

Further Reading

Prefix caching in vLLM under multi-tenant agent traffic

The setup

What the cache actually does

Where it worked

Where it didn't

The fix for Tenant B

Config

Comparison

Trade-offs and limitations

Further reading