I was working on cost optimization for an LLM-based document translation
pipeline.
At that point, the LLM translation flow was still very direct: ...
For further actions, you may consider blocking this person and/or reporting abuse
Interesting take on prompt batching—costs can definitely spiral if you're not careful with token counts and model pricing tiers. In my work with GPU-accelerated inference, I've seen how batch sizes affect both latency and resource utilization. Sometimes smaller batches with tighter token limits yield better cost/performance trade-offs, especially when using models with strict context windows.
Great write-up. This matches what I've seen in agent-style LLM systems: batching can reduce request count while still increasing total cost once retries, partial failures, and larger-context pricing kick in. The real metric isn't calls per job, it's cost per successful task with acceptable latency and debuggability. Fewer API calls can be a very misleading KPI.
The detail that jumped out at me is that 100 of your 107 calls were retries — batching wasn't the villain, the all-or-nothing fallback was. One missing key nuking 19 good translations is exactly the kind of thing that stays invisible until you reconcile call counts against the dashboard like you did.
We hit a near-identical trap batching extraction jobs. Strict
json_schemacut our miss rate a lot (so good call there), but the change that helped most was making the fallback surgical — only re-request the missing keys instead of replaying the whole batch, since even with strict schemas you still drop the occasional item on long outputs.Two things I'm curious about: did you find a batch size where the tail-degradation basically disappears for nano? We landed around 8–10 items, not 20. And are you bounding batches by estimated output tokens rather than segment count? Translation length varies enough that a fixed count of 20 can quietly push past the model's reliable range even when the segment count looks safe.
I haven’t done a proper batch-size sweep for nano model yet. On the file I tested, batch size 20 worked after strict schema + retrying only the missing IDs: 140 segments, 7 calls, 0 fallback. But I wouldn’t treat 20 as a generally "safe" number.
I still need to benchmark this with larger and more complex files, and your point about bounding batches by estimated output tokens may be useful there. In this run the batching was still count-based (
PROMPT_BATCH_SIZE=20). I addedmax_tokensestimation and split-on-length as a safety net after the request, but I’m not yet packing batches by estimated output tokens before sending them.20 short segments and 20 long segments are very different workloads, so that’s probably one of the next things I’d benchmark.
Thanks for the insight 🙌🏻
This is a good reminder that batching is a workflow decision, not a default optimization. If the batch hides failure modes or forces extra context into every request, the cost curve can move the wrong way.
The useful pattern is to measure per task: what can be cached, what needs fresh reasoning, and what should run as a deterministic terminal step instead.
That’s a good way to frame it. I also started with the assumption that fewer requests would mean lower cost, but the benchmark showed that the workflow around the batch matters just as much 😅
Yes, exactly. Batching is only cheaper when the unit of work stays clean.
Once the batch creates extra parsing, retries, larger prompts, or harder debugging, the “fewer requests” metric stops being the real cost model. I like benchmarking it as a workflow, not as an API-call count.
I hit the same wall batching translation segments. The token cost of dragging shared context into every batch quietly beat the per-call savings, and it only showed once I logged cost per segment instead of per request. Measuring the wrong unit hid it for a week.
Batching prompts sounds efficient in theory but falls apart when you factor in context window waste. Each batch item shares the system prompt tokens, so you're paying for that overhead N times whether you batch or not. The real savings come from reducing round trips at the orchestration layer, not cramming more into a single call. For agent workloads with branching logic, sequential calls with proper caching end up cheaper than trying to batch everything upfront.
The retry economics are the part that bit us with batching too. One segment failing or coming back malformed means you re-send the whole batch, so a 5 percent per-segment failure rate quietly turns into a much higher re-billed-token rate once you batch 20 of them together. We only saw it after stamping a correlation id per logical document and measuring cost per document instead of per API call, the per-call number looked great while the per-document number got worse. Batching is a real win when the failure rate is low and a cost trap when it is not, and most of us do not measure the failure rate at the batch granularity until the bill says so.
The detail that the missing IDs clustered near the tail is the useful tell. That's the output-token budget running out mid-response, not the model "forgetting." One lever that pairs nicely with your max_tokens + split fix: group segments by length before batching instead of taking them in document order. A batch that mixes a two-word heading with a 200-word paragraph spends its budget on the long one and drops the short ones at the end, even though they were trivial.
Great write-up! This is a common trap — batching helps throughput but can blow up token costs if individual prompts get padded. One thing that helped me: set a max_tokens per request and group prompts by expected response length. Thanks for sharing the real numbers!