DEV Community

Cover image for LLM Gateway & FinOps Architecture for B2B SaaS: Semantic Caching, Routing & Tenant Quotas
Muhammad Ali Mughal
Muhammad Ali Mughal

Posted on

LLM Gateway & FinOps Architecture for B2B SaaS: Semantic Caching, Routing & Tenant Quotas

#ai #architecture #saas #devops

The finance lead forwards you the model provider invoice with one line highlighted. Spend tripled in a month. Revenue did not. When you dig in, a single tenant on the starter plan is running a nightly script that sends the same 40-page contract through your "summarize" endpoint 600 times, every request goes to the most expensive model because that was the default in the SDK call, and nothing in your stack knew which tenant any of those tokens belonged to.

I wrote this with help from AI tools and checked every config, code path and claim myself. I run Loop Origin, where we build custom web applications for B2B SaaS teams, and this is the final part of the series: the gateway layer that turns LLM calls from an unbounded cost into a metered, budgeted product feature.

📚 Series Roadmap — Production AI & SaaS Systems Engineering

What is an LLM gateway and how does it control AI costs in a B2B SaaS?

Direct answer: An LLM gateway is one internal service that every model call passes through. It identifies the tenant and feature, checks a token budget before the call, serves exact or semantic cache hits, routes each task to the cheapest model that meets its quality bar, and records actual token usage afterwards so costs land on the right tenant's ledger.

Why per-service SDK calls stop scaling

In most SaaS codebases, the first AI feature calls the provider SDK directly from a controller. The second feature copies that code. By the fifth, you have five places that pick a model, five retry policies, no shared cache, and API keys in three services. Nobody can answer the two questions finance will ask: which tenant spent this and what would it cost on a cheaper model.

A gateway centralizes four decisions that should never be made inside product code:

  1. Who is paying? Tenant, plan and feature on every request.
  2. May they spend this? Budget and rate checks before tokens are bought.
  3. Have we answered this before? Exact and semantic caching.
  4. Which model is good enough? Routing by task class, with fallbacks.

The architecture 

 Laravel API ──┐
 Node workers ─┼─► LLM Gateway ──► quota check (Redis) ──► cache (exact → semantic)
 MCP agents  ──┘       │                                        │ miss
                       │                                        ▼
                       │                             router: task class → model tier
                       │                                        │
                       └──── usage events (Part 4 meter) ◄── provider A / B / self-hosted
Enter fullscreen mode Exit fullscreen mode

The gateway can be an open-source proxy you configure or a small service you write. Either way, product code sends a task, not a model name: { tenant_id, feature: "contract_summary", input }.

Step 1: Budgets before tokens

Reserve an estimated token cost before the call, then settle the real number afterwards. A Lua script keeps the check-and-reserve atomic in Redis or Valkey:

-- KEYS[1] = budget:{tenant}:{yyyy-mm}   ARGV[1] = estimate   ARGV[2] = monthly limit
local used = tonumber(redis.call('GET', KEYS[1]) or '0')
if used + tonumber(ARGV[1]) > tonumber(ARGV[2]) then
  return -1
end
redis.call('INCRBY', KEYS[1], ARGV[1])
redis.call('EXPIRE', KEYS[1], 40 * 24 * 3600)
return used + tonumber(ARGV[1])
Enter fullscreen mode Exit fullscreen mode 
// gateway/budget.ts
export async function reserve(tenant: Tenant, estimate: number) {
  const key = `budget:${tenant.id}:${monthKey()}`;
  const res = await redis.eval(RESERVE_LUA, 1, key, estimate, tenant.plan.monthlyTokens);
  if (res === -1) throw new QuotaExceeded(tenant.id, tenant.plan.name);
}

export async function settle(tenant: Tenant, estimate: number, actual: number) {
  await redis.incrby(`budget:${tenant.id}:${monthKey()}`, actual - estimate);
  await meter.emit({ tenant: tenant.id, metric: "llm_tokens", value: actual }); // Part 4 pipeline
}
Enter fullscreen mode Exit fullscreen mode

Estimate input tokens with the provider's tokenizer and add the request's max_tokens cap for output. Overestimating slightly is fine because settlement corrects it; what matters is that the 600-request nightly loop hits a wall instead of the invoice.

Step 2: Cache in two layers

Exact cache first: hash the normalized request (model tier, system prompt version, input, temperature) and look it up. It is cheap and has zero quality risk. In my experience it catches more than teams expect, because batch jobs and retries resend identical inputs.

Semantic cache second: embed the input, search a vector index scoped to the same tenant and feature, and return the stored answer only above a strict similarity threshold.

const hit = await vectors.search({
  namespace: `${tenant.id}:${feature}:${promptVersion}`,
  vector: await embed(input),
  topK: 1,
});
if (hit && hit.score >= FEATURE_THRESHOLDS[feature]) return hit.payload; // e.g. 0.95
Enter fullscreen mode Exit fullscreen mode

Three rules keep semantic caching from becoming a data-leak or wrong-answer machine:

  • Always namespace by tenant. A cross-tenant cache hit returns one customer's data to another.
  • Include the prompt version. Changing the system prompt must invalidate old answers.
  • Only cache stable tasks. FAQ answers and classification, yes. Anything personalized or time-sensitive, no.

Polished brass valve handles on copper pipes, representing per-tenant budget controls at the LLM gateway, AI-generated
AI-generated image.

Step 3: Route by task class, not by habit

Most SaaS AI features are not equally hard. Classify them once and map each class to a tier:

Task class Example feature Model tier Fallback Cacheable
Extraction / classification Tag a support ticket Small, fast model Second small model Exact + semantic
Summarization Contract summary Mid-tier model Small model with shorter output Exact
Reasoning / generation Draft a proposal Frontier model Mid-tier model + warning flag Rarely
Agentic tool use (MCP) Multi-step workflow Frontier model with tool support Halt and ask human No
Embeddings Search, semantic cache Embedding model Queue and retry N/A 
# gateway/routes.yaml
contract_summary:
  primary: { provider: provider_a, model: mid-tier, max_tokens: 800 }
  fallback: { provider: provider_b, model: small, max_tokens: 500 }
  timeout_ms: 20000
  plans:
    starter: { tier_override: small }
ticket_classify:
  primary: { provider: self_hosted, model: small }
  fallback: { provider: provider_a, model: small }
Enter fullscreen mode Exit fullscreen mode

The plans override is where FinOps meets pricing: starter tenants get the cheaper tier, enterprise tenants get the better one, and the difference is visible on your pricing page instead of hidden in your margin.

Validate a cheaper route before switching: replay a sample of real (anonymized) inputs through both tiers and score the outputs with your own rubric. Route changes without an eval are guesses.

Step 4: Fallbacks that do not double the bill

Retries are where LLM costs quietly multiply. As Part 3 covered for caches, a retry storm turns one outage into many. In the gateway:

  • Retry only on timeouts and 429/5xx, never on 4xx validation errors.
  • Use a circuit breaker per provider so a failing provider is skipped for a cool-down window.
  • Count tokens from every attempt against the tenant budget. A failed call that streamed half an answer still cost money.

Step 5: Make cost observable

Reuse the OpenTelemetry setup from Part 5. Each gateway call becomes a span with low-cardinality attributes:

span.setAttributes({
  "llm.feature": feature,
  "llm.route": route.name,
  "llm.model_tier": route.tier,
  "llm.cache": cacheResult,          // "exact" | "semantic" | "miss"
  "llm.tokens.input": usage.input,
  "llm.tokens.output": usage.output,
  "tenant.plan": tenant.plan.name,   // plan, not tenant ID, to keep metrics cheap
});
Enter fullscreen mode Exit fullscreen mode

Generate metrics from these spans for cost per feature, cache hit rate and fallback rate. Keep the tenant ID in the usage events and the billing ledger, not in metric labels, so dashboards stay affordable.

Before and after

Metric Direct SDK calls Behind the gateway
Who spent it Unknown Tenant + feature on every call
Runaway script Runs all night Blocked at the monthly budget
Repeated inputs Paid every time Exact cache hit
Model choice Whatever the SDK default was Per task class and plan
Provider outage Feature down Fallback route + circuit breaker
Billing Absorbed in margin Metered and invoiceable (Part 4)

Diagnostic checklist

  1. Does every model call in the codebase go through the gateway? Search for direct SDK imports.
  2. Does each request carry tenant, plan and feature?
  3. Is the budget check atomic and run before the provider call?
  4. Are tokens from failed and retried attempts counted?
  5. Is the semantic cache namespaced by tenant, feature and prompt version?
  6. Does every route have a timeout, a fallback and a circuit breaker?
  7. Was each cheaper route validated with an eval on real inputs?
  8. Do usage events reach the metering pipeline and reconcile with the provider invoice?
  9. Are prompts and outputs kept out of span attributes and logs?
  10. Is there an alert when any tenant passes 80% of budget?

FAQ

Should I build a gateway or use an open-source one?

Start with an existing open-source LLM proxy if it supports your providers, budgets and logging. Write your own thin service when you need tenant-aware quotas tied to your billing data, which most B2B SaaS products eventually do.

Does a gateway add latency?

A budget check in Redis and an exact cache lookup are usually a few milliseconds. Semantic cache adds an embedding call, so only enable it for features where hit rates justify it.

What similarity threshold should semantic caching use?

There is no universal number. Start strict, around 0.95 cosine similarity, log near-misses, and review them by hand before loosening it per feature.

How do I estimate tokens before the call?

Count input tokens with the provider's tokenizer library and add the max_tokens cap for output. Settle with the real usage numbers the provider returns.

Can I pass LLM costs through to customers?

Yes, if your metering is accurate. Emit settled token usage into the same pipeline as your other usage events and price it per plan, with clear limits shown in the product.

How does this fit with MCP agents from Part 1?

Agents make many calls per task, so they need the gateway most. Give each agent run its own budget reservation and stop the run, with a human approval gate, when it exceeds it.

That closes Production AI & SaaS Systems Engineering. Across six parts we sandboxed MCP agents, migrated databases without downtime, protected caches from stampedes, metered usage for billing, made traces affordable and, here, put a budget and a router in front of every model call. The common thread: put one deliberate control point in front of every expensive or risky operation, measure what passes through it, and let product code ask for outcomes instead of hard-coding infrastructure choices. Questions about your own setup are welcome in the comments.

Follow & Connect

Top comments (0)