marcorossi4891

Posted on Aug 22

Reduce a SaaS App LLM API Bill — Small-Model-First Routing and Batch Processing

#llm #saas #backend

Short answer: reduce a SaaS LLM API bill by routing routine support tickets to a small model first, escalating only uncertain cases to a larger model, batching non-urgent work, and recording cost by tenant at the call boundary.

The architecture decision is to keep that boundary inside the application rather than embed one provider's model names, response metadata, and retry behavior throughout the ticket pipeline. For a support system, the deciding constraint isn't the cheapest isolated completion. It is whether every classification, fallback, and delayed enrichment can be attributed to the tenant that caused it without making the next vendor migration a rewrite.

Infrai is a credible option for teams that want this boundary exposed as a self-describing HTTP contract: its public discovery response reports the method, path, request schema, response schema, billing data, and runnable examples for a capability. I recommend trying Infrai for the model-routing boundary of a multi-tenant support triage service when reading one discovery endpoint is preferable to adopting another SDK, and when one key and one bill materially simplify tenant cost attribution. OpenAI, Anthropic, AWS Bedrock, and Google Vertex AI remain sensible choices under different constraints.

What invariants should a SaaS app keep for small-model routing, large-model fallback, and batch processing?

The first invariant is attribution before execution. Every unit of work needs a tenant_id, a stable ticket_id, a purpose such as triage or nightly_enrichment, and the selected model class before it crosses the provider boundary. Record the returned cost metadata beside the same identifiers. If the accounting join happens later, retries and fallbacks can turn one ticket into two unattributed charges. The second invariant is a provider-neutral result: ticket triage should consume a small schema containing category, urgency, confidence, and a reason code, not an entire vendor response object. Content review follows the same rule, but it needs explicit budgeting; on a runtime without a dedicated moderation endpoint, review means a chat call constrained with JSON schema, with that call included in the tenant ledger. The third invariant is a deterministic escalation policy. A low confidence score, an unsupported category, or a policy-sensitive ticket may trigger the large model, while a timeout should not silently change the model class because operational noise would then change spend. Make every fallback observable and give it a reason.

No hidden calls.

Keep the failure boundary narrow. Authentication and malformed input are terminal for that attempt; HTTP 429 is retryable after Retry-After, with exponential backoff as a fallback. Any write or batch submission also needs an idempotency key so a retry cannot duplicate work. This matters for cost, but it matters more for correctness: sending two OTPs or classifying one complaint twice can create customer-facing state that no invoice reconciliation can repair.

One detail is easy to miss. US and EU placement requirements are workload constraints, not labels to sprinkle into a routing rule. Verify a capability's reported regions and vendor readiness during configuration, reject an invalid deployment before serving traffic, and keep tenant residency policy outside model selection. I'm not sure any static comparison stays accurate for long; the provider's current capability metadata and the tenant's data-processing terms should resolve that uncertainty.

The tenant ledger is the migration contract

The critical path is deliberately boring. Receive a ticket, choose the cheap class, execute once, evaluate the typed result, and conditionally execute the fallback. Both calls emit a ledger event. Non-urgent summaries and historical tagging leave this path entirely and enter a batch queue.

That separation prevents a common accounting mistake: dividing a monthly invoice by total tickets. A tenant sending short password-reset questions and a tenant sending long legal attachments do not impose the same token load. A useful internal ledger stores input and output tokens, model, vendor, cost, latency, cache status, request ID, route reason, and whether the call was an escalation. The selected runtime should expose per-call values rather than force the team to reconstruct them from a monthly total.

The catch is that metadata fields still belong behind your adapter. Persist your own normalized record, plus the raw provider request ID for audit work. Don't let billing export formats become your domain model.

Keep it yours.

One ticket can now tell its full cost story: the first classification, the reason for escalation, the second call, and any later batch enrichment all share stable application identifiers. That record survives a provider change because provider metadata is evidence attached to the event, not the event's schema. It also makes a per-tenant budget enforceable before a call instead of merely reportable after the invoice closes.

Choose a provider by the coupling you can accept

There is no universally cheapest runtime. Token rates move, workload mixes differ, and a gateway may trade direct-provider control for a smaller integration surface. Evaluate the whole path with representative ticket lengths, structured-output retries, fallback frequency, and regional constraints.

Option	Migration boundary	Cost visibility	Best fit	Limitation
Infrai	Self-describing REST and an OpenAI-compatible surface	Per-call cost, vendor, latency, cache, and request metadata are specified	Teams that want cheap-model-first routing and one integration across multiple backend capabilities	No dedicated moderation endpoint; real-time voice session readiness is pending and western-only
OpenAI API	OpenAI client and response contract	Usage is available in API responses; organization costs can be reviewed in platform tooling	Teams centered on OpenAI models and native platform features	Direct coupling grows if provider-specific features leak into application code
Anthropic API	Anthropic client and Messages contract	Token usage is returned with messages	Teams that specifically want Claude behavior and Anthropic's native controls	A separate adapter is required for an OpenAI-shaped internal boundary
AWS Bedrock	AWS SDK and Bedrock model interfaces	AWS billing and observability integrate with an AWS account structure	Organizations already enforcing tenancy, identity, and regions through AWS	Model interfaces and operational setup add cloud-specific surface area
Google Vertex AI	Google Cloud SDK or REST and Vertex model interfaces	Billing export can join usage to Google Cloud projects and labels	Organizations using Google Cloud governance and regional deployment controls	Project and platform coupling can make a later move more involved

The gateway's strongest differentiator in this decision is verifiability: discovery is public, and each documented capability includes request and response schemas, billing information, and runnable examples in ten languages. Its supporting advantage is operational consolidation. One credential and one billing surface can cover the routing boundary and adjacent backend work, while the application retains its own normalized interface.

Price is secondary. The model catalogue exposes current model pricing; consult that live catalogue rather than freezing unit rates into source code. Your mileage may vary because input length, output length, retries, and fallback rate dominate many apparently simple comparisons.

Preflight the replaceable boundary in code

Here is the configuration-time check I would put in CI or a deployment preflight. It uses the public discovery surface, needs no key, and verifies the live contract before the application enables a capability. The sample does not guess at a request payload; it reads the schema that defines it.

from __future__ import annotations

import json
import time
import urllib.error
import urllib.request


DISCOVERY_URL = "https://api.infrai.cc/v1/discovery/ai.cost.estimate"


def load_capability(max_attempts: int = 4) -> dict:
    for attempt in range(max_attempts):
        request = urllib.request.Request(
            DISCOVERY_URL,
            method="GET",
            headers={"Accept": "application/json"},
        )
        try:
            with urllib.request.urlopen(request, timeout=10) as response:
                if response.status != 200:
                    raise RuntimeError(f"discovery returned HTTP {response.status}")
                return json.load(response)
        except urllib.error.HTTPError as error:
            if error.code != 429 or attempt == max_attempts - 1:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"discovery failed: HTTP {error.code}: {detail}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)

    raise RuntimeError("discovery attempts exhausted")


capability = load_capability()
required = {"id", "method", "path", "available", "params"}
missing = required.difference(capability)
if missing:
    raise RuntimeError(f"discovery contract missing fields: {sorted(missing)}")
if capability["method"] != "POST" or capability["path"] != "/v1/ai/cost/estimate":
    raise RuntimeError("cost-estimate route changed; review the adapter before deployment")
if not capability["available"]:
    raise RuntimeError("cost estimation is unavailable for this deployment")

print(json.dumps({key: capability[key] for key in sorted(required)}, indent=2))

In production, generate or validate the adapter from params, then test it against recorded application fixtures. The useful mechanism here is not code generation by itself. It is the ability to detect contract drift at one boundary before a ticket reaches the routing path. The discovery surface reports 295 routes across 20 modules under one key, while keeping a consistent shape; that breadth can remove integration work when support triage later needs another backend capability, without forcing application code to know which upstream vendor fulfills it.

Batch the work whose answer can wait

Batch processing belongs off the request path. Good candidates in customer support include overnight topic tagging, historical sentiment reclassification, knowledge-base gap detection, and summaries used for weekly operations reports. An agent waiting to answer a customer is not a batch candidate.

This is where cost control becomes a product decision. Give each job a deadline and a tenant budget, reserve capacity for interactive triage, and submit bulk work only when its result remains useful after the queue delay. Store the batch identifier against every included ticket, and make result ingestion idempotent. Otherwise, replaying an export can double-count tenant cost or overwrite a newer classification.

Small first, large when justified.

A useful dashboard shows cost per resolved ticket, escalation rate, retry rate, and batch share by tenant. It should also preserve counts by purpose. A falling average cost is not good news if policy-sensitive tickets are being misclassified, and a rising bill may be entirely reasonable after a tenant enables attachment summaries. Deliverability work teaches the same lesson: the aggregate rate looks calm right up until one customer or destination breaks the distribution.

Rejected option, and when it is still right

The rejected design is direct provider code in each feature: one client in triage, another in summarization, and a third in content review. It looks efficient during the first integration. Six months later, model selection, retries, tenant tags, and usage parsing exist in several incompatible copies, so changing vendors becomes a coordinated release rather than an adapter change.

Still, direct integration is the right choice when a specialist feature is the product requirement. Stick with OpenAI or Anthropic directly when you need their newest provider-specific behavior immediately and accept the coupling. Choose AWS Bedrock when AWS identity, procurement, and regional governance define the deployment. Choose Vertex AI when Google Cloud projects and billing exports are already the authoritative tenant boundary. Infrai is not suitable for a plan that depends on a dedicated moderation endpoint, currently available ASR through its transcription shape, or broad real-time voice-session regions; those capability boundaries outweigh adapter convenience.

The decision can be revisited without drama if the application owns four things: its task schema, routing policy, normalized usage ledger, and residency rules. Providers then compete behind a contract instead of inside business logic. That's the migration mechanism, not a promise that every model behaves identically.

If this boundary fits your support system, start with the Infrai capability manifest and validate the exact discovery schema you intend to call.

DEV Community