The gif, and the honest headline
Here's a multi-stage LLM pipeline being killed mid-run and started again:
It picks up exactly where it stopped. No duplicate model calls, no re-spend. That's the engineering payoff. The headline I want to be honest about up front is the economics one, because it runs against the usual pitch:
This is not "run it locally and save a fortune." At a real content pipeline's scale, batched cloud inference is genuinely cheap. The valuable thing is a reproducible way to measure the local-vs-cloud crossover, and the answer of when local does and doesn't pay off.
Here's the real number, from an instrumented production run: one weekly run of this pipeline moves 641,671 input and 77,409 output tokens across nine local model stages, plus 134 locally-generated images. Run it on batched cloud models instead and that's about $0.88, under a dollar a week. So the interesting question was never "how much does local save." At this scale, cloud is already pocket change. It's when that stops being true, and how you would know.
The pains everyone hits on week two
The first version of any LLM pipeline is a for loop over a list. It works in the demo. Then production happens:
- It dies at item 800 of 1000 and you do not want to pay for the first 800 again.
- The same run, re-triggered, re-does work and double-writes downstream.
- Near-duplicate inputs get processed twice, because nothing deduplicates them.
- You have no idea what it cost, because nobody recorded the tokens.
- A rate limit or a credit ceiling hits and the run dies dirty, mid-item.
None of these is hard on its own. Together they're the difference between a script and a pipeline. resumable-llm-pipeline is the small, framework-light reference that wires them together, extracted (patterns only) from a real weekly content pipeline at Cedar & Bloom. It runs offline on the stdlib alone via a deterministic mock router, so you can see all of it in five minutes.
The build
Two pieces carry the weight.
ProgressStore, for resumability. A crash-safe {item_id -> {status, ...}} written with a temp-file-and-rename atomic swap after every item. Each stage filters to unfinished items and reads the previous stage's output. That decoupling is the resumability: kill the process anywhere and re-running recomputes nothing that already finished. The test suite proves it in numbers. Across a killed run and its resume, total model calls equal the number of kept items exactly. No item is processed twice.
ModelRouter, for cost-aware routing. complete(prompt, tier) and embed(text) behind one uniform response that carries token counts. Two tiers (cheap and quality) map to a backend (mock, local ollama, or hosted) by one config line. Because both local and hosted return (text, tokens_in, tokens_out), cost and metrics are backend-agnostic, and swapping local for hosted to re-run the benchmark is a one-liner. A Budget(max_tokens, max_cost) guard checks the paid stage as it runs; on breach it checkpoints and exits cleanly. Resume is the retry.
Routing between a cheap and an expensive model is not a new idea.
RouteLLM
(paper) showed a learned router can cut cost by around 85% on some benchmarks while holding quality. This router is the boring, explicit version: you assign each stage a tier by hand. The goal is not to be clever, it is to make the cost of that choice measurable.
The measurement: the tokens were being thrown away
Here's the detail that started this. The production pipeline it generalises calls local models via Ollama, and Ollama's response includes prompt_eval_count and eval_count, the exact input and output token counts, in the final chunk. The original code threw them away. It did real token work every week and recorded none of it.
So the router captures those counts on every call (and usage.input_tokens / usage.output_tokens on the hosted path), and a fail-open metrics module writes one JSON line per call: tokens, latency, backend, model. Fail-open matters. The thing that measures the pipeline must never be the thing that breaks it. rlp report rolls it up.
Here's the actual per-stage scorecard from one instrumented weekly run (qwen2.5:14b is the cheap tier, qwen2.5:72b the quality tier):
stage model calls tok_in tok_out
novelty qwen2.5:14b 273 417647 12860
rewrite qwen2.5:72b 67 112516 48673
image_prompt qwen2.5:14b 67 52193 11094
enrich qwen2.5:14b 67 43759 3289
cluster qwen2.5:14b 4 10264 700
timelines qwen2.5:14b 10 2568 60
companies qwen2.5:72b 20 1445 661
edition_intro qwen2.5:14b 1 1279 72
image_gen FLUX.1-schnell 134 0 0 (images — timing only)
--------------------------------------------------------------
TOTAL 641671 77409
Until this run, every one of those token counts was None. The pipeline did the work weekly and recorded nothing. Notice novelty on its own: 273 calls, 417k input tokens, 65% of the entire run's input. Hold that thought, it matters for the cost.
The benchmark: measure the crossover, don't guess it
rlp benchmark reads the tokens the run actually recorded, maps each tier to its hosted equivalent, projects to an annual workload, and compares hosted cost (list price, and with the Batch API's 50% discount) against a fixed local cost. Fed the real weekly run, with the cheap tier priced as claude-haiku-4-5 ($1/$5 per 1M) and the quality tier as claude-sonnet-5 ($3/$15 per 1M):
Measured (weekly run): 641,671 input + 77,409 output tokens across 9 stages + 134 images.
Modeled: hosted prices as above, Batch 50% off, hosted image ~$0.003/img, local $560/yr amortised (hardware capex + power), 52 weekly runs/yr.
| line | per run | per year (x52) |
|------------------------------|---------|----------------|
| hosted text (list) | $1.75 | $91 |
| hosted text (batch, 50% off) | $0.88 | $46 |
| hosted images (modeled) | $0.40 | $21 |
| local (fixed, amortised) | — | $560 |
Break-even is about 640 weekly runs per year, roughly 12x the real cadence. Below that, batched cloud wins on cash.
Add the monthly evergreen run (measured: 47,596 in / 33,920 out, so $0.47 a run at list, $0.24 batched, roughly $3 to $6 a year) and the whole Cedar & Bloom LLM workload lands around $50 to $70 a year on batched cloud (about $110 at list). Against a $560/yr amortised local box, cloud is 8 to 12 times cheaper on cash at the real volume.
The discipline that makes this trustworthy, and the whole reason the tool exists: token counts are measured; hosted prices, the per-image cloud rate, and the local yearly cost are modeled and labelled. "The hardware's a sunk cost" is not a fair comparison. An honest break-even needs the amortised capex line, which is why it's in there.
The counter-intuitive bit: the cost isn't where the tokens are
Look again at that weekly scorecard through a cost lens:
| stage | share of input tokens | share of hosted cost |
|---|---|---|
novelty (cheap tier) |
65% | ~28% |
rewrite (quality tier) |
18% | ~61% |
novelty moves two-thirds of the tokens but stays cheap, because it runs on the cheap model and emits almost nothing. rewrite is the reverse: modest token volume, but its output tokens on the quality model at $15/1M dominate the bill. The money is in output tokens on the quality tier, not where the token volume is. Knuth's old warning that "premature optimization is the root of all evil" has a neat LLM corollary: don't guess which stage is expensive. The tokens pile up in novelty; the money leaves through rewrite. You only see that by measuring, and it's what tells you which stage to move to cloud first if you ever wanted a hybrid.
One genuinely measured cloud data point
Everything above prices local tokens as if they'd run on the cloud. But one stage in the wider Cedar & Bloom stack already is hosted: SteadyPath's quiz runs on Anthropic Haiku, and instrumenting it captured 4 real calls, 1,490 input and 1,861 output tokens, actual usage from the API, not a projection. Same metrics module, same JSONL, local and hosted side by side. That's the point of the backend-agnostic router. The moment you move a stage to the cloud, its real cost lands in the same ledger as the local estimate.
The honest conclusion the data supports: at Cedar & Bloom's real scale, batched cloud would cost tens of dollars a year, and a weekly run is under a dollar. Local's edge is control, privacy, and no rate limits, and it only wins on cash at roughly 10 times the volume, or once you bolt on a token-hungry agentic layer. The number that tells you which regime you're in is one you can now measure instead of argue about.
Take the pattern
git clone https://github.com/uxdw/resumable-llm-pipeline && cd resumable-llm-pipeline
python -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]"
rlp reset && rlp run && rlp run # watch the 2nd run resume with 0 model calls
rlp benchmark # your own local-vs-cloud crossover
Point the router at your models, feed the benchmark your real per-stage tokens and verified prices, and read your own break-even. Measure your crossover. Don't inherit someone else's headline.
(Companion piece: Stop shipping AI Agents you can't measure: evals observability from scratch, on evals and observability as a CI gate. This pipeline's metrics core is the same telemetry idea, pointed at cost.)
Built and measured at Cedar & Bloom. MIT-licensed. Written by Richard Atkins.
Top comments (0)