DEV Community

SmartGate profile image lizer yang
lizer yang for SmartGate

Posted on Originally published at smartgate.network

Token Optimization Techniques for AI Apps: Caps and Cost Control

#llm #windsurf #ai #mcp

Short answer: Token optimization for AI apps is four disciplines in one request path: measure tokens before you spend them, shape and compress the payload before the model sees it, cache and route so the same bytes are never paid for twice, and cap the loop with quotas and output limits. SmartGate implements all four in gateway code, and this page shows the mechanisms and the numbers behind them.

Key takeaways

  • Measure with the model's own tokenizer. A count that comes back as an integer is something a quota can compare; a character estimate is not.
  • Shape before you cap. Compression and deduplication reduce what the model reads, which is cheaper than refusing the request after it has been built.
  • One URL per client. A hosted MCP endpoint replaces a private key per tool, and the host config is a short JSON file the gateway generates.
  • Cache the repeats, not the decisions. A 45-second window on a budget snapshot is fine; a cached authorisation decision is not.
  • Count what you saved. Two audit fields turn "we compressed something" into a number a finance team can read.
  • Cap last, but cap. Per-minute rate limits, monthly token quotas and output limits are the backstop when shaping, caching and routing have not been enough.

The four levers: shape, cache, route, cap

Every technique on this page belongs to one of four levers, and they pay off in that order.
Shaping is decided at the call site: fewer bytes leave your process. Caching removes a call
entirely instead of making it cheaper. Routing changes which model answers, and at what
price. Capping is the backstop that holds when the first three miss.

Most teams start with the cap, which is the most expensive place to begin. A monthly quota
tells you the bill is too high; it does not tell you which of the three cheaper levers would
have avoided it.

Lever What it changes Concrete control Where it runs
Shape the bytes that reach the model compression, deduplication, compact tool output, structured data instead of prose the tool layer
Cache how often the same bytes are paid for twice response cache with an explicit TTL; cached budget snapshots the gateway edge
Route which model or provider answers, and at what tier dynamic routing, fallbacks, tiering by task class the gateway policy
Cap the ceiling per call, per key, per team output limits, monthly token quotas, per-minute rate limits the plan catalog and the budget guard

The four levers fail differently, which is why the order matters. Shaping changes what the
model reads. Caching changes whether it reads at all. Routing changes the price of the call
that remains. Capping decides what happens when the other three are not enough. An agent loop
that appends its own transcript every turn defeats shaping and caching at the same time, so
measurement comes first: you cannot tell which lever is missing until you can see the number.

Why LLM token efficiency is a gateway metric

Token efficiency looks like a prompt-writing habit and behaves like a traffic problem. The
cost of one agent turn is not the user's question; it is the question plus every tool
definition the host sent, every result the loop accumulated, and every retry. A 2026
measurement of an identical ten-turn agent conversation counted 2,400 tokens per turn with
no MCP servers attached, 18,700 with three, and 31,200 with five
(MCP's dirty secret).
A separate Q1 2026 dataset of 22 teams put MCP input tokens at 41–58% of total
coding-agent spend (MCP gateway economics).

Nothing in the prompt changed between those runs. What changed was the tool surface, which is
why the count has to be taken where the traffic is, not where the prose is written. And the
count itself has a subtlety that decides whether anyone keeps taking it.

get_token_length is that measurement path: text goes in, an integer comes out, and the
caller decides what to do with it. Two details make it worth reading. First, two tokenizer
families sit behind one call and the function picks the matching one, so a team that switches
host models does not have to switch its budget arithmetic. Second, it calls
tokenizer.tokenize and adds the special-token count separately instead of calling the
tokenizer's own __call__, because the latter warns and does more work on inputs longer than
the model's maximum length — which is exactly the shape of input a fetch tool produces. A
measurement that costs a warning on every large document is a measurement nobody runs on the
hot path, and an unmeasured path is the one that produces the surprise invoice.

# backend/smartgate/modules/context_gate/algorithm.py — source lines 987–1003 (token count)
def get_token_length(
        self,
        text: str,
        add_special_tokens: bool = True,
        use_oai_tokenizer: bool = False,
    ):
        if use_oai_tokenizer:
            return len(self.oai_tokenizer.encode(text))
        else:
            # tokenize + special token count avoids tokenizer.__call__ on megabyte
            # texts (transformers warns when len > model_max_length).
            n = len(self.tokenizer.tokenize(text))
            if add_special_tokens and hasattr(
                self.tokenizer, "num_special_tokens_to_add"
            ):
                n += self.tokenizer.num_special_tokens_to_add(pair=False)
            return n
Enter fullscreen mode Exit fullscreen mode

The special tokens are the detail teams drop first, and dropping them is worth a constant
error per request: invisible once, significant over a month of agent traffic. Keeping the
count separate from the decision is the other half of the design. The function returns an
integer; the quota check, the dashboard and the audit row all consume the same number later,
so measurement never has to be reimplemented per caller.

Windsurf MCP: one URL where a key per tool used to be

The practical objection to a gateway is configuration sprawl: every host wants its own file,
its own key and its own transport setting. The fix is to make the host config a generated
artifact rather than a document each developer transcribes by hand, and to point every host
at one endpoint.

The excerpt below is the builder for one host, and it is short on purpose. The entry type is
streamable-http rather than a command, so nothing is spawned locally; serverUrl is the
single MCP URL; and the authorisation header is written from a placeholder, not from a
live key, so a config committed by accident does not carry a credential. The platform
constant in the last line is not decoration: it is what lets the request be attributed to the
host that sent it when the activity log is read back.

# lib/connect/mcp-config-templates.ts — source lines 99–116 (host config)
function buildWindsurfMcpConfigJson(
  mcpUrl: string,
  apiKeyPlaceholder: string = API_KEY_PLACEHOLDER,
): string {
  return JSON.stringify(
    {
      mcpServers: {
        smartgate: {
          type: "streamable-http",
          serverUrl: mcpUrl,
          headers: buildMcpAuthHeaders(apiKeyPlaceholder, PLATFORM_AGENT_ID.Windsurf),
        },
      },
    },
    null,
    2,
  );
}
Enter fullscreen mode Exit fullscreen mode

Two consequences for the token bill. First, the transport decision stops being a per-developer
choice: if the host speaks Streamable HTTP, the same endpoint serves Cursor, Claude Desktop,
Windsurf and a CI job, so a compression or caching change reaches every client at once instead
of landing in one editor. Second, the tool surface stays in one place, which is the only way
to keep it small. Five servers configured by hand add five tool lists to every request; one
endpoint can expose a bounded set and route the rest server-side. The gateway generates the
snippet for the host you connect, and the
connect page lists the current hosts.

Prompt compression: shrink the payload before it is sent

Compression is the lever with the best ratio of effort to tokens, and it is also the one that
breaks quality most quietly. It splits into two families.

Mechanical compression removes bytes that carried no information: duplicated passages across
retrieved documents, navigation chrome around the paragraph you wanted, repeated whitespace and
boilerplate, and tool output serialised as prose when a table would do. This family is lossless
in practice and safe to enable by default, which is why it should be the first thing a team
turns on.

Learned compression rewrites the prompt itself. LLMLingua's answer is a budget controller
that sets how much may be cut and a token-level compression pass that keeps the remaining
tokens interdependent, so a compressed prompt is not a bag of keywords
(LLMLingua, paper). The budget
controller is the part most implementations skip, and skipping it is how compression becomes a
silent quality regression: without a stated budget, the compressor cuts until it fits instead
of until the answer survives.

Three rules keep compression honest. Keep the instruction block out of the compressed region;
compress retrieved evidence, not the question. Log the ratio per call rather than per week, so
a regression is attributable. And decide the loss budget per task class, because a summarisation
task tolerates more cutting than a code edit. The code-sensitive parts of the same problem —
which tokens survive when context is tight — are in
context window management techniques.

Caching: stop paying twice for the same bytes

An agent stack wants three different caches, and conflating them is how caching gets a bad
reputation.

Response caching stores a completed model answer under a key built from the exact prompt,
model and parameters. It is only correct when the key is exact: a cache that answers a
"similar" prompt is a correctness bug with a good hit rate.

Tool-result caching stores what a fetch or retrieval returned for a URL or query inside a
short window. This is where the largest savings usually sit, because agent loops re-read pages
they already read three steps earlier.

Decision caching stores the answer to a cheap, frequently asked question, such as whether a
team is inside its budget. SmartGate caches that snapshot for 45 seconds by default: the
TTL is an environment override, and a value that is non-finite or not positive falls back to 45
rather than becoming an infinite window. The trade-off is explicit — a burst of tool calls
shares one budget check, and the cap can be up to 45 seconds stale. That is the right trade for
an allowance check and the wrong one for an authorisation decision, which is why
tool-call authorisation is evaluated per request instead.

Two disciplines make the difference between a cache and a leak. Every entry needs a TTL, not
just an eviction policy, because a stale entry that never expires is indistinguishable from a
wrong one. And every cached response that reaches a user needs a key that includes the tenant:
a cache shared across teams will eventually serve one team's data to another.

Cloudflare AI gateway: caching, routing, fallbacks

Cloudflare AI Gateway is a hosted control point in front of model providers, and its feature
list reads like the four levers: response caching, rate limiting, guardrails, dynamic routing
with fallbacks, per-request logging, and token and cost analytics
(AI Gateway). Dynamic routing is the routing
lever made concrete: rules combine provider, model, quota and condition, and a request that
fails or exceeds its condition falls through to the next target rather than failing the turn
(dynamic routing).

Read the scope carefully before assuming it solves a tool-traffic problem. An AI gateway
governs model calls: the completion your app asks for. Most of the token amplification
measured above happens one layer earlier, in the tool calls that fill the context before
the completion is requested, and those are governed by an MCP gateway instead. The two layers
compose: route and cache the completion, and compress, authorize and meter the tools that feed
it. If the distinction is new, the
AI gateway versus API gateway
is the shorter read.

MCP server on AWS: where the loop runs and who pays

Hosting an MCP server on AWS is a solved problem with several answers: a container behind a
load balancer, a Lambda function for infrequent tool calls, or a managed gateway. Amazon
Bedrock AgentCore Gateway is the managed version of that last shape — one secure entry point
for agentic traffic that routes to tools, to other agents and to models, with MCP servers
declared as gateway targets and authentication, policy enforcement and observability
consolidated at the endpoint (gateway targets,
gateway overview).

The token-relevant point is what hosting does not change. Where the server runs affects
latency, cold starts, egress and your operational surface, but the tokens a call consumes are
decided by the payload the server returns and by how often the host re-sends its tool
definitions. A server moved from a laptop to a managed gateway produces the same context,
unless the move also puts compression, caching and metering in the request path. So choose the
runtime on deployment grounds, and make the accounting a property of the request path rather
than of the host. The wider deployment question — what the gateway layer should own versus
what the application keeps — is covered in
enterprise AI gateway architecture.

MCP server testing: assert the token counts, not just the answers

Tool tests usually assert that the right content came back. For a token-cost problem that is
half a test suite, because a server can return the correct document in the most expensive
possible shape. The assertions worth writing are the ones a cost regression would fail:

  • Long input, stable count. A very large text should produce a token count within a small tolerance of the reference, so a tokenizer swap or a truncation bug shows up as a test failure instead of a bill.
  • Chunking by tokens, not characters. A splitter that measures characters fragments differently across languages and markdown, and the fragments are what get compressed.
  • Explicit degradation. An unknown model name should degrade in a defined way rather than throwing on the hot path; the accounting for it should be visibly zero rather than missing.
  • Degenerate input. A single-token input and an empty result are the inputs that break ratio arithmetic.

The savings fields are the ones that turn compression from a claim into a number, and the
excerpt below shows how they are attached. The function reads the raw and compressed token
counts out of a tool result and adds them to the audit parameters — but only when the raw
count is above zero, and it returns the parameters untouched when the result is not the
dictionary shape it expects. That guard is the whole point: a failed compression must not be
able to write a phantom saving, because the savings aggregate is what a bill is computed from.

# backend/smartgate/core/audit_params.py — source lines 4–15 (savings fields)
def enrich_compress_audit_params(
    params: dict,
    result_data: object,
) -> dict:
    """Add raw_tokens / compressed_tokens for savings aggregation (spec §4.1)."""
    if not isinstance(result_data, dict):
        return params
    origin = int(result_data.get("origin_tokens") or 0)
    compressed = int(result_data.get("compressed_tokens") or 0)
    if origin > 0:
        params = {**params, "raw_tokens": origin, "compressed_tokens": compressed}
    return params
Enter fullscreen mode Exit fullscreen mode

The same pattern generalises to any metered tool: keep the raw count, keep the count after
processing, and attach both to the audit record for the call. Two integers per call is what
makes a monthly report reconcilable, and it is what lets a compression change be evaluated on
evidence instead of on a feeling about response quality.

MCP stdio versus remote transports

The Model Context Protocol defines exactly two standard transports: stdio, where the client
launches the server as a subprocess and exchanges JSON-RPC messages over standard input and
output, and Streamable HTTP for remote servers. The specification's guidance is that clients
should support stdio whenever possible
(transports), and
for a developer tool that is the right default: no port, no certificate, no auth handshake.

Token-wise the choice is narrower than it looks. A stdio server still declares its tools to the
host, and the host still serialises those definitions into the context of every turn — the
amplification measured in the first section happens with local servers too. What stdio changes
is operations, not payload: the process is yours, the data stays on the machine, and there is
nothing to authorise. What a remote endpoint changes is the opposite, and that is where the
levers live: a hosted endpoint can compress a response before it returns, cache a fetch, and
charge the call to a team. The practical rule is to pick the transport for the deployment you
actually have, then make sure the token controls sit in the layer both of them call. The
revision history matters here, because the
remote transport changed once already.

RAG vs agentic RAG: what a loop adds to the bill

Classic retrieval-augmented generation retrieves once and sends the top passages with the
question: one retrieval, one completion, a predictable payload. Agentic RAG lets the model
choose its retrievals, so the number of hops becomes a runtime decision — and every hop
re-sends the context accumulated so far, including the tool output of the previous hop
(agentic RAG survey).

The bill therefore scales with hops rather than with documents. Two retrievals on one page cost
more than one retrieval across two pages, because the second call carries the first result in
its context. This is where the four levers stop being independent: compressing a retrieved
passage pays twice, once when the passage is fetched and again on every later hop that includes
it. The classic-versus-agentic trade-off and where to stop looping are covered in
RAG versus agentic RAG; the compression side is above.

For budgeting, the useful move is to treat a hop as a metered unit. Give the loop a hop ceiling
and a per-hop token allowance, then compare the two after the fact. A loop that hits its hop
ceiling before its token allowance is reasoning; a loop that hits the token allowance first is
re-reading.

Output limits: quotas, rate limits and per-call caps

Capping has three levels, and a complete setup uses all of them. Per-call output limits bound
the most expensive surprise, because generated tokens are billed to your account and a runaway
generation is invisible until it is finished. Per-minute rate limits bound the burst: they
decide how fast a loop can spend. Monthly quotas bound the month and are the number a team
actually plans against.

SmartGate's catalog makes the levels explicit rather than hiding them in a single number. The
monthly token limits step 2M, 20M, 100M and 200M across the plans, and the rates step with them:

Plan Monthly token limit Team requests/min MCP requests/min per key Team ceiling
Free 2M 60 120 120
Pro 20M 120 300 600
Teams 100M 300 600 3000
Enterprise 200M 600 1200 9999

Read as a budget, the interesting column is the last two: a per-key limit is what stops one
misconfigured client, and the team ceiling is what stops ten correct clients from arriving at
once. The implementation details — how a quota is enforced per team, what happens at the
boundary, and how the decision reaches the tool call — are in
enforcing a token quota per team.

How SmartGate compares

The gateway category splits by what sits in the request path. SmartGate is narrower than "AI
gateway": it governs tool traffic, and its billing participates only after it has saved you
something.

What it governs How you run it What you pay
SmartGate MCP tool traffic: fetch, compression, dedup, budgets, pipelines Hosted gateway; one MCP URL per client Free: 2M tokens/month and all seven tools. Pro from $18/month, first month $5; the share starts after $15 of measured savings and is capped at $18/month, so the total stays under $36/month (pricing)
Cloudflare AI Gateway Model traffic: caching, rate limiting, guardrails, dynamic routing with fallbacks Hosted gateway in front of model providers Usage-based, from your provider relationship (AI Gateway)
Amazon Bedrock AgentCore Gateway Agentic traffic to tools, agents and models, with MCP servers as targets Managed AWS gateway Priced with the rest of the service you deploy (overview)
Local compression servers (for example mcp-context-guard) Context compression inside one developer's editor A stdio server you install and run Open source; you own the runtime (PyPI)

Two honest readings of that table. If your only problem is that one long document keeps filling
an editor's context, a local compression server is a reasonable start: it is free and it does
not touch your network. If the problem is enforcement — who may call what, how much, under
which budget, with what audit trail — compression alone does not answer it, and a gateway does.
The last line of the table is also the one to read twice: a locally installed server is
convenient and unmetered, which is another way of saying nobody can tell you what it saved.

The seven tools themselves are documented in the
MCP tools reference, and the per-request accounting that makes
the savings share measurable is in the
execution-cost breakdown.

How to get started

  1. Connect one client. Generate the config for Cursor, Claude Desktop, Windsurf or your own host, paste it in, and confirm the seven tools appear in the tool list. This is the step that replaces a key per tool with one URL.
  2. Turn on shaping before capping. Run a fetch, a retrieval and a dedup over a workflow you already own — competitor pages, documentation, issue threads — and watch the compressed against raw token counts on the usage report. If the ratio does not move, the payload was already small and the cap is not your problem.
  3. Then enforce. Wrap the loop with a budget check and a record call, set the team's monthly cap, and give long research-shaped work a pipeline call so four steps become one audited request instead of four.

Start on the free plan — 2M tokens/month, all seven tools, no card — with
start free, then check the caps against your
own workload on the pricing page.

FAQ

What counts as token optimization here?
Four levers over one request path: measure with the model's own tokenizer, shape the payload with
compression and deduplication, cache and route so repeated bytes are not paid for twice, and cap
the loop with output limits, rate limits and monthly quotas.

Does SmartGate replace my model or my gateway?
No. Your agent still talks to its host model, and an AI gateway still governs the completion.
SmartGate sits on the tool traffic that fills the context before that completion is requested,
and it measures, compresses, authorizes and meters those calls.

Why cache a budget check for 45 seconds instead of always reading it live?
Because the check sits on the path of every tool call and the underlying snapshot rarely changes
within a burst. The 45-second window is an environment override, non-positive values fall back to
it, and the trade-off is stated plainly: up to 45 seconds of staleness on the allowance, in
exchange for one check per burst instead of one per call. Authorisation decisions are not cached
this way.

How do I know compression saved anything?
The audit row for the call carries the raw token count and the compressed one, and the savings
fields are only written when the raw count is above zero. If a call reports no raw count, no
saving is claimed for it, which is what keeps the aggregate reconcilable.

Do stdio servers avoid the token amplification?
They do not. A local server's tool definitions are still serialised into every turn. stdio buys
you locality, no port and no auth handshake; it does not shrink the context by itself.

Do I need an MCP client to use these controls?
The tools are MCP-native and the same modules are reachable over REST, so a CI job or a back-end
service can get the same counts, the same quota decision and the same audit rows without an agent
in the loop.

Limitations and what this does not do

  • Compression is lossy. Splitting and compressing a long payload will drop content you wanted. Test the ratio on your own corpus before you rely on it, and keep the instruction block out of the compressed region.
  • The decision cache trades freshness for cost. A 45-second window means the allowance reading can be up to 45 seconds behind. That is a deliberate trade for an allowance and the wrong one for an authorisation.
  • The numbers here are measurements, not guarantees. The per-turn amplification figures come from a specific agent, a specific host and a specific set of servers; your tool surface decides yours.
  • A gateway is not a sandbox. Governing which tools may be called does not make an unsafe tool safe. Pair it with least-privilege keys, role checks and a review of what your agent is allowed to reach.

Sources

Method note

The code excerpts in this article are not transcribed. Each block was cut directly out of the
slice body returned by the SmartGate slice API and re-asserted byte-for-byte as a substring of
that body before publication; the first line inside every fence records the file and the exact
source lines it came from. Symbols were pinned by whole-name containment (rule A level 2) and
confirmed by a server-side proof call before any of them entered the text.

Three of the eight planned sections resolved to a symbol that could be pinned, and those are the
three excerpts below. The remaining five are written from the protocol specification, the
provider documentation and the product's own plan catalog, which is the house rule: a section
that cannot be pinned is sourced, never invented. Repository-relative paths are shown as they
are in the source; internal hosts, credentials and private addresses were stripped before any of
it reached this document.

Slice provenance

# SERP keyword Symbol File Source lines How it was pinned sha256(12)
1 llm token efficiency PromptCompressor.get_token_length backend/smartgate/modules/context_gate/algorithm.py 987–1003 rule A L2 → slot-proof 1ed8b361a397
2 windsurf mcp buildWindsurfMcpConfigJson lib/connect/mcp-config-templates.ts 99–116 rule A L2 → slot-proof 82aabcf99dd1
3 mcp server testing enrich_compress_audit_params backend/smartgate/core/audit_params.py 4–15 rule A L2 → slot-proof 952380769807

Every fenced block above was cut from the slice body and re-asserted against it byte-for-byte
before publication. 3 of 8 sections pinned, 0 abstentions,
5 misses.

Top comments (0)