DEV Community: Richard Atkins

The real economics of a production LLM pipeline: resumability, cost-aware routing, and measuring when local beats the API

Richard Atkins — Sat, 18 Jul 2026 15:11:28 +0000

The gif, and the honest headline

Here's a multi-stage LLM pipeline being killed mid-run and started again:

It picks up exactly where it stopped. No duplicate model calls, no re-spend. That's the engineering payoff. The headline I want to be honest about up front is the economics one, because it runs against the usual pitch:

This is not "run it locally and save a fortune." At a real content pipeline's scale, batched cloud inference is genuinely cheap. The valuable thing is a reproducible way to measure the local-vs-cloud crossover, and the answer of when local does and doesn't pay off.

Here's the real number, from an instrumented production run: one weekly run of this pipeline moves 641,671 input and 77,409 output tokens across nine local model stages, plus 134 locally-generated images. Run it on batched cloud models instead and that's about $0.88, under a dollar a week. So the interesting question was never "how much does local save." At this scale, cloud is already pocket change. It's when that stops being true, and how you would know.

The pains everyone hits on week two

The first version of any LLM pipeline is a for loop over a list. It works in the demo. Then production happens:

It dies at item 800 of 1000 and you do not want to pay for the first 800 again.
The same run, re-triggered, re-does work and double-writes downstream.
Near-duplicate inputs get processed twice, because nothing deduplicates them.
You have no idea what it cost, because nobody recorded the tokens.
A rate limit or a credit ceiling hits and the run dies dirty, mid-item.

None of these is hard on its own. Together they're the difference between a script and a pipeline. resumable-llm-pipeline is the small, framework-light reference that wires them together, extracted (patterns only) from a real weekly content pipeline at Cedar & Bloom. It runs offline on the stdlib alone via a deterministic mock router, so you can see all of it in five minutes.

The build

Two pieces carry the weight.

ProgressStore, for resumability. A crash-safe {item_id -> {status, ...}} written with a temp-file-and-rename atomic swap after every item. Each stage filters to unfinished items and reads the previous stage's output. That decoupling is the resumability: kill the process anywhere and re-running recomputes nothing that already finished. The test suite proves it in numbers. Across a killed run and its resume, total model calls equal the number of kept items exactly. No item is processed twice.

ModelRouter, for cost-aware routing. complete(prompt, tier) and embed(text) behind one uniform response that carries token counts. Two tiers (cheap and quality) map to a backend (mock, local ollama, or hosted) by one config line. Because both local and hosted return (text, tokens_in, tokens_out), cost and metrics are backend-agnostic, and swapping local for hosted to re-run the benchmark is a one-liner. A Budget(max_tokens, max_cost) guard checks the paid stage as it runs; on breach it checkpoints and exits cleanly. Resume is the retry.

Routing between a cheap and an expensive model is not a new idea.
RouteLLM
(paper) showed a learned router can cut cost by around 85% on some benchmarks while holding quality. This router is the boring, explicit version: you assign each stage a tier by hand. The goal is not to be clever, it is to make the cost of that choice measurable.

The measurement: the tokens were being thrown away

Here's the detail that started this. The production pipeline it generalises calls local models via Ollama, and Ollama's response includes prompt_eval_count and eval_count, the exact input and output token counts, in the final chunk. The original code threw them away. It did real token work every week and recorded none of it.

So the router captures those counts on every call (and usage.input_tokens / usage.output_tokens on the hosted path), and a fail-open metrics module writes one JSON line per call: tokens, latency, backend, model. Fail-open matters. The thing that measures the pipeline must never be the thing that breaks it. rlp report rolls it up.

Here's the actual per-stage scorecard from one instrumented weekly run (qwen2.5:14b is the cheap tier, qwen2.5:72b the quality tier):

stage          model            calls   tok_in   tok_out
novelty        qwen2.5:14b        273   417647    12860
rewrite        qwen2.5:72b         67   112516    48673
image_prompt   qwen2.5:14b         67    52193    11094
enrich         qwen2.5:14b         67    43759     3289
cluster        qwen2.5:14b          4    10264      700
timelines      qwen2.5:14b         10     2568       60
companies      qwen2.5:72b         20     1445       661
edition_intro  qwen2.5:14b          1     1279        72
image_gen      FLUX.1-schnell     134        0        0   (images — timing only)
--------------------------------------------------------------
TOTAL                                    641671    77409

Until this run, every one of those token counts was None. The pipeline did the work weekly and recorded nothing. Notice novelty on its own: 273 calls, 417k input tokens, 65% of the entire run's input. Hold that thought, it matters for the cost.

The benchmark: measure the crossover, don't guess it

rlp benchmark reads the tokens the run actually recorded, maps each tier to its hosted equivalent, projects to an annual workload, and compares hosted cost (list price, and with the Batch API's 50% discount) against a fixed local cost. Fed the real weekly run, with the cheap tier priced as claude-haiku-4-5 ($1/$5 per 1M) and the quality tier as claude-sonnet-5 ($3/$15 per 1M):

Measured (weekly run): 641,671 input + 77,409 output tokens across 9 stages + 134 images.
Modeled:  hosted prices as above, Batch 50% off, hosted image ~$0.003/img, local $560/yr amortised (hardware capex + power), 52 weekly runs/yr.

| line                         | per run | per year (x52) |
|------------------------------|---------|----------------|
| hosted text (list)           | $1.75   | $91            |
| hosted text (batch, 50% off) | $0.88   | $46            |
| hosted images (modeled)      | $0.40   | $21            |
| local (fixed, amortised)     | —       | $560           |

Break-even is about 640 weekly runs per year, roughly 12x the real cadence. Below that, batched cloud wins on cash.

Add the monthly evergreen run (measured: 47,596 in / 33,920 out, so $0.47 a run at list, $0.24 batched, roughly $3 to $6 a year) and the whole Cedar & Bloom LLM workload lands around $50 to $70 a year on batched cloud (about $110 at list). Against a $560/yr amortised local box, cloud is 8 to 12 times cheaper on cash at the real volume.

The discipline that makes this trustworthy, and the whole reason the tool exists: token counts are measured; hosted prices, the per-image cloud rate, and the local yearly cost are modeled and labelled. "The hardware's a sunk cost" is not a fair comparison. An honest break-even needs the amortised capex line, which is why it's in there.

The counter-intuitive bit: the cost isn't where the tokens are

Look again at that weekly scorecard through a cost lens:

stage	share of input tokens	share of hosted cost
`novelty` (cheap tier)	65%	~28%
`rewrite` (quality tier)	18%	~61%

novelty moves two-thirds of the tokens but stays cheap, because it runs on the cheap model and emits almost nothing. rewrite is the reverse: modest token volume, but its output tokens on the quality model at $15/1M dominate the bill. The money is in output tokens on the quality tier, not where the token volume is. Knuth's old warning that "premature optimization is the root of all evil" has a neat LLM corollary: don't guess which stage is expensive. The tokens pile up in novelty; the money leaves through rewrite. You only see that by measuring, and it's what tells you which stage to move to cloud first if you ever wanted a hybrid.

One genuinely measured cloud data point

Everything above prices local tokens as if they'd run on the cloud. But one stage in the wider Cedar & Bloom stack already is hosted: SteadyPath's quiz runs on Anthropic Haiku, and instrumenting it captured 4 real calls, 1,490 input and 1,861 output tokens, actual usage from the API, not a projection. Same metrics module, same JSONL, local and hosted side by side. That's the point of the backend-agnostic router. The moment you move a stage to the cloud, its real cost lands in the same ledger as the local estimate.

The honest conclusion the data supports: at Cedar & Bloom's real scale, batched cloud would cost tens of dollars a year, and a weekly run is under a dollar. Local's edge is control, privacy, and no rate limits, and it only wins on cash at roughly 10 times the volume, or once you bolt on a token-hungry agentic layer. The number that tells you which regime you're in is one you can now measure instead of argue about.

Take the pattern

git clone https://github.com/uxdw/resumable-llm-pipeline && cd resumable-llm-pipeline
python -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]"
rlp reset && rlp run && rlp run       # watch the 2nd run resume with 0 model calls
rlp benchmark                          # your own local-vs-cloud crossover

Point the router at your models, feed the benchmark your real per-stage tokens and verified prices, and read your own break-even. Measure your crossover. Don't inherit someone else's headline.

(Companion piece: Stop shipping AI Agents you can't measure: evals observability from scratch, on evals and observability as a CI gate. This pipeline's metrics core is the same telemetry idea, pointed at cost.)

Built and measured at Cedar & Bloom. MIT-licensed. Written by Richard Atkins.

Stop shipping AI agents you can't measure: evals + observability from scratch

Richard Atkins — Sat, 18 Jul 2026 15:00:00 +0000

The demo, in one screen

Here's an agent's eval scorecard on a green build:

metric                  rate  baseline    delta
-----------------------------------------------
task_success          95.00%    95.00%   +0.00%
tool_correctness     100.00%   100.00%   +0.00%
schema_validity      100.00%   100.00%   +0.00%
groundedness         100.00%   100.00%   +0.00%
PASS: no regressions vs baseline.

Now I make one change. I stop the agent from actually reading the documents it cites, so it answers from search snippets alone. That's one environment variable, AES_DISABLE_FETCH=1. Re-run the exact same eval:

metric                  rate  baseline    delta
-----------------------------------------------
task_success          75.00%    95.00%  -20.00%  <-- REGRESSION
tool_correctness      15.00%   100.00%  -85.00%  <-- REGRESSION
schema_validity      100.00%   100.00%   +0.00%
groundedness          15.00%   100.00%  -85.00%  <-- REGRESSION
FAIL: 3 metric(s) regressed vs baseline: task_success, tool_correctness, groundedness

The command exits non-zero. CI goes red, and the PR is blocked. One behavioural regression, caught by three independent metrics, before it ever reached a user.

Would your agent have caught that? For most teams I talk to, the honest answer is no. Not because they don't care, but because the tooling that makes it easy is all paid SaaS, and the DIY version never quite becomes a priority. This post is that DIY version, built once so you can copy it.

The gap

Everyone can build an agent now. Almost no one can prove it still works after the next prompt tweak, model bump, or refactor. The tools that help (Braintrust, LangSmith, DeepEval) are mostly paid platforms you bolt on later. For the deep version of why evals matter and how to design domain-specific ones, Hamel Husain's Your AI Product Needs Evals is the canonical read. This post is the smallest running skeleton of that idea, wired into CI.

There's no clean, vendor-neutral, framework-light reference that shows the whole loop: a working agent, traced, evaluated, and guardrailed, with evals running as CI so a regression fails the build like a broken unit test. So I built one. The repo, agent-eval-starter, runs end-to-end with zero API keys and zero network on a deterministic mock provider, so you can clone it and see all of the above in under five minutes.

The build

Four pieces, each deliberately boring:

A real agent loop using a JSON-action protocol rather than a vendor's native tool-calling. The agent emits {"action": "...", ...} and the harness parses it. That keeps it portable across providers and trivially mockable, with no framework lock-in.
OpenTelemetry tracing. The same spans that draw the trace tree roll up into a cost, latency, and token scorecard, and export to any OTel backend. It isn't a bespoke tracer: the OpenTelemetry GenAI semantic conventions are standardising exactly this shape of LLM span, so you're not locked into my format.
An eval harness that scores a committed, labelled dataset on four pass-rate metrics and compares against a committed baseline.json.
Guardrails: a deterministic banned-phrase post-filter, plus an optional LLM-as-verifier review pass.

The four metrics, each a pass-rate over the dataset:

task_success: did it answer correctly, or correctly abstain on the unanswerable cases? (Abstention counts as a correct answer.)
tool_correctness: did it use tools honestly, citing only documents it actually fetched?
schema_validity: is the final answer well-formed? (pydantic-validated)
groundedness: the hallucination guard. Is every claim backed by fetched text?

The insight worth stealing

The single most useful thing I've learned shipping this stuff:

Prompt-level guardrails ("never say X", "always cite your sources") are unreliable. A deterministic post-filter is the dependable fallback, and an LLM-as-verifier is the second layer, not the first.

The model will, eventually, ignore the instruction. So the banned-phrase check runs in code, after generation, every time. It cannot be talked out of it. The LLM-as-verifier pass then catches the fuzzier failures a deterministic filter can't express. Two layers,
cheap one first. It's why the demo works: groundedness isn't a line in the system prompt asking nicely, it's a computed check over what the agent actually fetched.

The numbers (and their caveats)

Everything above is measured on the repo's committed synthetic 20-case suite, not a production workload. I'm flagging that on purpose. The credibility of an eval story is the honesty about what got measured.

Baseline pass-rates: task_success 95%, tool_correctness 100%, schema_validity 100%, groundedness 100%.
One seeded regression (disabling document fetch) drops 3 of 4 metrics, the suite exits non-zero, and CI posts the scorecard on the PR.
Cost and latency per run: about $0 on the mock provider (no network). Point it at claude or ollama and the OpenTelemetry roll-up reports tokens, cost, and p95 latency from real calls, so accuracy and spend are tracked over time from the same spans.

None of this proves the agent is correct. As Edsger Dijkstra put it, testing "can be used to show the presence of bugs, but never to show their absence." What evals give you is the other thing production needs: a tripwire. The moment a change makes the agent measurably worse, the build stops.

So the point isn't "95% is good." It's that the 95% is a number you can defend, version, and regress against. A gate, not a vibe.

Take the pattern

git clone https://github.com/uxdw/agent-eval-starter && cd agent-eval-starter
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"
agent-eval eval          # score the suite -> scorecard
make demo                # green, then a seeded regression turns it red

Swap the corpus and dataset for your domain, point the provider at your model, and commit a new baseline. The pattern is the product; the example is just a runnable seed. Make your agent's accuracy a CI gate.

(Companion piece: The real economics of a Production LLM
pipeline - resumability cost aware routing & measuring, on resumability, cost-aware routing, and measuring when local beats the API.)

Built and measured at Cedar & Bloom. MIT-licensed. Written by Richard Atkins.