DEV Community

lizer yang
lizer yang

Posted on Originally published at smartgate.network

Build a Low-Cost AI Backend Architecture: Count, Cache, Meter

#cheap #ibm #llm #mcp

Short answer: a low-cost AI backend is not a cheaper model list. It is four decisions taken in
one order: price the call before you shop, with a cost series built from the route you actually
used; cache and batch the work that repeats, because a repeated prompt prefix and an offline batch
are the two cheapest tokens you will ever buy; route each call to the cheapest model that passes
its own quality gate; and self-host only past a break-even computed from your own utilisation.
Metering is the substrate underneath all three - and the guardrails that cap work get there first.

Key takeaways

  • Price the call, not the model. A table of prices per million tokens cannot tell you what your traffic costs. The mix of input to output, and the savings your own request path produced, can.
  • Cache and batch what repeats. Providers bill a cache read at a fraction of fresh input, and an asynchronous batch endpoint well below the interactive rate. Both prices reward moving work off the interactive path.
  • Route on cost per successful task. A route that is cheap per token but fails and gets retried costs more than the model you were avoiding.
  • Self-host break-even is arithmetic, and utilisation decides the answer. The same rented GPU is a bargain at 40% utilisation and an expensive lesson at 4%.
  • Guardrails are the caps that arrive before the invoice. An output limit, an iteration limit and a monthly budget per team are cheaper than the incident they prevent.
  • Meter where you enforce. A number you cannot read per call cannot be routed, cached or defended later.

A cheap LLM API is a price list; your bill is a series

Every hunt for a cheap LLM API starts as a table of prices per million tokens, and that table
answers a question nobody asked. Your bill is the sum, over calls, of input tokens at the route's
input price plus output tokens at the route's output price, minus whatever your own request path
avoided before anything was sent. Two of those terms belong to the model; two belong to your
traffic and your pipeline. Until all four sit in one series, "cheap" is a headline rather than a
measurement.

# lib/dashboard/reports-trend-metrics.ts — source lines 124–127 (buildTokenCostSeries)
function buildTokenCostSeries(
  dailyTokens: ReportsTrendDataSlice["dailyTokens"],
  costSavings: ReportsTrendDataSlice["costSavings"],
) {
Enter fullscreen mode Exit fullscreen mode 
# lib/dashboard/reports-trend-metrics.ts — source lines 147–163 (buildTokenCostSeries)
  for (const row of costSavings) {
    const prev = byDate.get(row.date);
    if (prev) {
      prev.compress = row.compress;
      prev.fetch = row.fetch;
      prev.search = row.search;
    } else {
      byDate.set(row.date, {
        date: row.date,
        output: 0,
        compress: row.compress,
        fetch: row.fetch,
        search: row.search,
      });
    }
  }
  return [...byDate.values()].sort((a, b) => a.date.localeCompare(b.date));
Enter fullscreen mode Exit fullscreen mode

Read the function as those four terms meeting on one axis. Token totals and savings are joined on
the same date key, so a single chart shows spend and the savings that offset it. A date present in
only one of the two inputs is still emitted, because a day with compression savings and no model
spend is a real day. And the result comes back sorted, so no consumer has to sort it again.

Three consequences an engineer can act on. Count with the encoder the model family actually uses -
tiktoken for OpenAI-family encodings - or the series and the invoice will disagree at the edges.
Keep the price table versioned, because a per-token price quoted from one version does not
reproduce under another, and a savings number without its price version is a claim rather than a
measurement. And check your input-to-output ratio before you shop: chat and agent workloads are
output-heavy, so a cheap-looking model whose output price matches the expensive one is not cheaper
for you at all.

What an IBM RAG and agentic AI professional certificate leaves out

An IBM RAG and agentic AI professional certificate track teaches the parts that demo well:
chunking, embeddings, a vector store, an agent loop over tools. Nothing in the syllabus bills you
for them, and the distance between a capstone notebook and a production backend is almost entirely
a cost question. Four things change on the way. Every chunk is embedded once and re-embedded
whenever the source document changes. A vector index carries a monthly bill and a memory ceiling.
Retrieval adds tokens to every prompt it feeds. And each extra agent step re-reads a context that
has grown since the previous step.

# lib/smartgate/trace.ts — source lines 80–87 (buildTraceView / summary fields)
    summary: {
      requestId: str(summaryRaw.request_id),
      traceKind,
      status: summaryRaw.status === "error" ? "error" : "success",
      totalDurationMs: num(summaryRaw.total_duration_ms),
      totalTokens: num(summaryRaw.total_tokens),
      spanCount: num(summaryRaw.span_count, spansRaw.length),
      route: summaryRaw.route != null ? str(summaryRaw.route) : null,
Enter fullscreen mode Exit fullscreen mode

The trace summary is the bridge between the course and the invoice, and it carries the numbers a
cost model needs with nothing decorative attached: total tokens and total duration for the run,
span count for how many model calls the run took, the route that served it, and the agent platform
that asked. Two habits follow from that shape. Record the route and the platform on the trace
itself rather than only in configuration, so a change in spend can be traced back to a change in
routing. And attribute cost per trace rather than per HTTP request: a single agent run can be a
dozen billable calls, and a per-request average buries the run that costs forty times its
neighbours.

LLM guardrails that cap spend, not just output

LLM guardrails are usually discussed as output policy - moderation, injection defence, PII
filtering. The guardrails that decide your infrastructure bill are the ones that cap work before
it is purchased, and every one of them is cheap to add:

  • A maximum output token count per request. Output is the expensive half of almost every price table, and it is the half a model chooses for itself when nobody caps it.
  • A ceiling per request. One pasted document should not be able to spend a team's day.
  • An iteration cap on agent loops. A loop with no ceiling is not an architecture, it is a spend event with a status endpoint.
  • A monthly budget per team, with headroom. Refuse when the remainder crosses a threshold rather than at zero, so the caller gets a warning instead of an outage.
  • A retry budget. Two retries of a 4,000-token prompt cost more than the model upgrade you postponed.

Two asymmetries decide the implementation. The read that informs - how much is left - should fail
open, so a monitoring outage never takes serving down with it. The counter that enforces must have
one writer, or you will eventually discover that the number you bill is the sum of three
optimistic paths. And the quota belongs in the same store as the counter, so the figure on the
dashboard and the figure the gate applies cannot drift apart. The per-plan mechanics - explicit
limit, plan default, unlimited escape hatch - are in
per-team token quotas.

MCP prompts are cache keys: reuse the template, not the round trip

MCP prompts are the protocol's reusable unit: a server publishes a template, a client lists what
is available and pulls it, and the same text reaches many calls. Reuse is where cost is decided,
because the part of a prompt that never changes is the part a provider can cache. Prompt caching
prices a cache read at a fraction of fresh input - on the Claude API a cache write costs 1.25 times
base input and a read about 0.1 times base, with a five-minute default TTL - which turns prompt
layout into a budget decision: stable instructions first, variable data last, never interleaved. A
single reordered word inside the cached prefix invalidates it and the next call pays full price for
the whole prefix, which is a cost bug that looks like nothing in a diff.

Reuse also needs a monthly reading to prove it paid off, and that reading has to come from counters
you already trust rather than from a second pipeline written for the report.

# dashboard-calibration/dashboard_calibration/verify_redis.py — source lines 22–41 (build_monthly_u_from_seed_stats)
def build_monthly_u_from_seed_stats(seed_stats: dict[str, Any]) -> dict[str, int]:
    monthly = seed_stats.get("monthly_u")
    if isinstance(monthly, dict) and monthly:
        return {str(k): int(v) for k, v in monthly.items()}

    by_date = seed_stats.get("daily_u_by_date")
    if isinstance(by_date, dict) and by_date:
        out: dict[str, int] = {}
        for day_key, amount in by_date.items():
            month = str(day_key)[:7]
            out[month] = out.get(month, 0) + int(amount)
        return out

    # Legacy: entire window treated as current month only.
    total = int(seed_stats.get("u_month_total", 0))
    if total > 0:
        from datetime import date

        return {date.today().strftime("%Y-%m"): total}
    return {}
Enter fullscreen mode Exit fullscreen mode

The function is the boring half of that reading, and the boring half is where the accounting bugs
live. An explicit monthly map wins when one exists; otherwise day keys are grouped by their first
seven characters, which is the YYYY-MM prefix; and a legacy total is treated as the current
month. Two invariants keep it honest: derive the month from the key prefix so a period rollover
needs no cleanup job, and make the fold idempotent - re-running it must overwrite, never
accumulate. The same discipline is what makes an avoided-token claim auditable: counts per call,
aggregated by day, folded into the month your invoice already uses.
MCP resources, prompts and sampling are the three
server-side primitives a client can reach; only prompts repeat on a schedule.

What agentic RAG costs when the loop runs more than once

Plain RAG spends a predictable amount per answer: one retrieval, one prompt, one generation.
Agentic RAG hands the number of steps to the model, and the bill becomes a function of the loop.
The arithmetic is short. Cost is the sum over steps of context tokens at that step times the input
price, plus output tokens at that step times the output price. Because every appended observation
grows the transcript, step n costs more than step one, and the total grows faster than the step
count. The transcript is the multiplier, not the question.

That shape decides which levers are worth pulling, and none of them is "pick a cheaper model".
Retrieve once into a deduplicated context instead of retrieving before every step. Summarise the
scratchpad instead of appending to it. Put the static instructions at the front where a cache can
hold them. Give planning and routing to the smallest model that can make the decision, and reserve
the largest for the final answer. Cap the iterations, because three passes that solve the task beat
ten that also solve it. If you want the mechanics before you optimise them, start from
what agentic RAG is and then count the steps your own traces show.

An agentic RAG survey read as a cost catalogue

The agentic RAG survey literature reads as a catalogue of ways to spend more per answer. The
reference taxonomy - Agentic Retrieval-Augmented Generation: A Survey on Agentic
RAG is the version the others cite - sorts the systems by how
much autonomy they take: a single agent that plans and reflects, a router that chooses among
retrievers, and multi-agent pipelines that delegate to specialists. Every axis buys accuracy with
model calls, and the families differ mainly in how many. Read from the cost side, the useful
question is not which pattern is best but which pattern's extra call saves more than it costs. A
corrective pass that prevents a wrong answer the user would otherwise re-ask is worth its tokens;
a second planner that restates the first planner's output is not.

Three practices make that comparison possible on your own traffic. Put the pattern name on the
trace, in the agent_platform and route fields above, so cost per resolved task can be compared
between patterns instead of argued about. Send a canary share of traffic down the cheaper pattern
and compare success rather than latency. And measure the whole answer, retries included, because a
pattern's failure rate is part of its price. A survey maps the design space; only your traces say
which corner of it you can afford.

MCP router and model router: route per call, not per team

An MCP router and a model router sound like one product and are two decisions. An MCP router
decides which server a tool call lands on. A model router decides which model answers a given
prompt. Both are routing decisions, both are reversible, and both are only auditable if the choice
is written down at the moment of the call - which is what this summary line is for.

# lib/smartgate/audit-logs.ts — source lines 72–93 (buildSummary)
function buildSummary(
  auditTool: string,
  params: Record<string, unknown>,
  tokenUsed: number | null,
  success: boolean,
): string {
  const parts: string[] = [];
  const route = params.route as string | undefined;
  if (route) parts.push(`route=${route}`);
  if (params.url && typeof params.url === "string") {
    parts.push(params.url.length > 48 ? `${params.url.slice(0, 48)}…` : params.url);
  }
  if (params.query && typeof params.query === "string") {
    parts.push(`q=${params.query}`);
  }
  if (tokenUsed != null && tokenUsed > 0) {
    parts.push(`${tokenUsed.toLocaleString()} tokens`);
  }
  if (!success) parts.push("failed");
  if (parts.length === 0) return auditTool || "audit";
  return parts.join(" · ");
}
Enter fullscreen mode Exit fullscreen mode

The audit line above is that record: the route, a trimmed URL or query, the token count when one is
known, and a failure marker. It is the minimum needed to answer the two questions routing creates -
which route served the traffic, and which route produced the failures - and it is the per-call
input to the cost series from the first section. Two rules keep routing honest. Route on cost per
successful task rather than cost per token, because a model five times cheaper that fails a third
of calls and gets retried is the more expensive one. And make the decision in the gateway, which is
the only component that sees every call whatever client made it: an
MCP gateway is where one route table serves every host, and where a route
your plan does not cover can be refused instead of billed.

MCP sampling: who pays for the tokens the server asks for

MCP sampling inverts the direction of the call: instead of a client asking a server to do
something, the server asks the client's model for a completion. Architecturally that is elegant - a
server can use language reasoning without holding model credentials. Financially it hands the bill
to someone else, because the tokens a sampled completion produces are paid by whoever owns the
client's model. So the design question is not whether to allow sampling; it is which side of the
boundary should own the spend.

The same three habits bound the request from either side. Ask for the smallest completion that
does the job, and set an explicit maximum token count on every sampling request so a retry loop
cannot escalate the bill. Prefer deterministic local steps - parsing, filtering, arithmetic - over
asking a model for something a function can decide. And log sampling calls next to the calls you
started, so one trace tells the whole story. On the client or gateway side those habits mirror:
cap tokens per sampling request, cap requests per server, and refuse rather than queue when a team
is out of budget.

When the same tokens can be billed to two parties, the meter stops being bookkeeping and becomes
the contract. That is why skip reasons and idempotency keys belong in the charge path rather than
in a report, the same discipline a queue-and-webhook billing chain needs -
AI Automation for SaaS Operations covers that
half.

Where the money goes: four levers and what each one costs you

Lever What it changes What you give up When it breaks even
Prompt and prefix caching bills the repeated part of a prompt at a fraction of the fresh-input price prompt layout discipline; any edit inside the cached prefix invalidates it the second call that repeats a prefix, which for an assistant with a system prompt is usually the first afternoon
Batching an asynchronous batch endpoint prices work well below the interactive rate turnaround measured in hours rather than seconds any job nobody is waiting for: embeddings, backfills, evaluations, nightly summaries
Model routing output tokens land on the cheapest route that clears a quality gate an evaluation harness, a fallback chain and a canary share of traffic when the cheaper route passes your own evals; a route that fails a third of calls is not cheaper
Self-hosting a fixed hourly cost for a GPU replaces a price per token operations, upgrades, idle capacity only above the break-even rate computed from your own utilisation

Self-host break-even is arithmetic, and utilisation is what the arithmetic is unforgiving about.
Assume a rented GPU at $2 per hour, a server that sustains 1,200 output tokens per second, and 40%
utilisation: it produces 1,200 x 3,600 x 0.4 = 1.728M output tokens per hour, so those two dollars
buy 1.728M output tokens - about $1.16 per million output tokens, before operations and on-call
enter the picture. If your routed API price for comparable quality sits below that figure,
self-hosting is a hobby. Push utilisation to the other extreme, 4% on an idle box waiting for a
nightly job, and the same formula returns $11.60 per million: the version of the calculation most
teams meet after the hardware arrives. Compare per output token, because output is the half of the
bill that scales with generation and the half a self-hosted server actually produces - and treat
the result as a floor, since it prices the GPU and not the model upgrade, the outage or the
engineer who keeps it running.

How to get started

  1. Record the route and the token count on every call. One audit line per call carrying the route that served it, the token count when one is known, and a failure marker. A week of that data beats any benchmark you can download.
  2. Build the cost series before changing any model. Spend and savings on one date axis, split by route, so "cheap" becomes a comparison you can read rather than a claim you can argue.
  3. Cache the stable part of every prompt, and batch what can wait. Stable prefix first, variable tail last; embedding backfills, evaluations and summaries belong on the asynchronous endpoint rather than inside a user's request.
  4. Route per call behind a quality gate. Cheapest route that passes, one defined fallback, and a canary share kept on the old route until the new one proves itself on success rate.
  5. Cap before you optimise. A maximum output count per request, an iteration limit on loops, and a monthly budget per team with enough headroom to warn before it refuses.
  6. Then run the break-even with your own numbers. Your utilisation, your route prices, your quality bar - Optimize AI Agent Execution Cost is the attribution discipline this rests on, and the gateway primitives documentation lists the count, cap and meter primitives that make steps three to five configuration rather than a project.

The free tier exposes the count, the cap and the meter as gateway primitives (2M tokens a month,
the smart_* tool surface including smart_budget_guard, no card): start
free. The pricing
page and the versioned table behind it are what the series
above should be reconciled against, and talk to sales
covers contract token pools and HMAC-based team budgets on Enterprise.

FAQ

Is a cheap LLM API just the one with the lowest price per token?
No, because your bill is a mix. Output tokens usually dominate a chat or agent workload, so the
model with the lowest input price can be the expensive one. The comparison that settles it is your
own series: cost per day split by route, with the savings your pipeline produced counted against
it.

Does prompt caching matter if my prompts change on every call?
The stable part of a prompt is usually bigger than people assume: system instructions, tool
definitions, retrieved policy text, few-shot examples. Cache those and keep the variable tail last.
A cache read is priced at a fraction of fresh input, and the TTL is short - five minutes on the
Claude API - so the win lands on traffic that repeats inside the window, not on traffic that is
unique.

How do I compute self-host break-even?
Divide the hourly cost of the server by the output tokens it serves in an hour: hourly cost divided
by (tokens per second x 3,600 x utilisation) is cost per token, and multiplying by a million gives
the figure you can compare with an API price. Then subtract operations and see whether the routed
price is above or below it. Utilisation, not capability, is the number that decides.

Where does money leak in an agentic RAG pipeline?
Three places, in order of size: loop iterations that re-read a growing transcript, retrieval that
adds context to every step instead of once, and re-embedding documents that did not change. All
three are visible in a trace that carries span count and token count, and none of them is visible
in a per-request average.

Is batching always worth the discount?
Only for work nobody is waiting for. Asynchronous batch endpoints are priced well below the
interactive rate and return results on a turnaround measured in hours, so embeddings, backfills,
evaluations and nightly summaries fit; an interactive chat turn does not.

What should I meter first if I only have a week?
Route and token count per call. Those two fields make every other decision measurable - which model
actually served the traffic, what a resolved task costs, and whether the cheaper route is still
cheaper after retries. Everything else in this article is arithmetic on top of that pair.

Limitations and what this does not do

  • This is a cost architecture, not an optimiser. Nothing here compresses a prompt or picks a model for you; it makes those decisions measurable and reversible.
  • Caching depends on a prefix that stays identical. Any edit inside the cached region - a reordered sentence, a timestamp, a per-user name - turns a cache read back into a full-price write, silently.
  • Batch discounts trade turnaround for price. A job with a deadline belongs on the interactive endpoint, whatever the discount says.
  • Routing needs an evaluation harness. Without one, "cheap" is a guess that fails in production rather than in a spreadsheet.
  • Break-even arithmetic is a floor, not a decision. It prices the GPU and ignores operations, model upgrades, on-call, and the cost of running three model generations behind.
  • The lever you cannot meter is the one that keeps surprising you. A per-call record is what separates cost engineering from a monthly argument; reconstructing the numbers later from logs is the reporting-only variant of the same discipline, which is where a FinOps practice starts rather than ends.
  • Prices move. Every figure here is mechanics rather than a quote: a number is only reproducible next to the version of the price table that produced it.

Sources

Method note

The prose was rewritten against a fresh measured keyword plan for this page: eight sections, every
one of them drawn from the project's own demand evidence rather than from a code symbol. Four
sections quote a code block; the other four are written from external sources, because the symbol
matcher's verdict for those sections was abstain or no-slice - the honest outcome of a whole-name
rule, and the reason no code sits under them. Every fenced block was cut directly out of the slice
body returned by the SmartGate slice API and re-asserted byte-for-byte as a substring of that body
before publication; the first line inside each fence records the file and the exact source lines.
Two of the four blocks are quoted as line windows rather than whole bodies, and the provenance
table below records every range.

Slice provenance

# SERP keyword Symbol File Source lines How it was pinned sha256(12)
1 cheap llm api buildTokenCostSeries lib/dashboard/reports-trend-metrics.ts 124–127, 147–163 rule A L2 → slot-proof d265de9f668c
2 ibm rag and agentic ai professional certificate buildTraceView lib/smartgate/trace.ts 80–87 rule A L2 → slot-proof 49028130dba9
3 mcp prompts build_monthly_u_from_seed_stats dashboard-calibration/dashboard_calibration/verify_redis.py 22–41 rule A L2 → slot-proof 9b4fecef236f
4 mcp router buildSummary lib/smartgate/audit-logs.ts 72–93 rule A L2 → slot-proof 009eace0bd53

Every fenced block above was cut from the slice body and re-asserted against it byte-for-byte before
publication. 4 of 8 sections pinned, 1 abstentions, 3 misses.

Top comments (0)