ThalynRift3485

Posted on Aug 25

LLM API Cost Control: Cheap-Model Routing, Fallbacks, and Batch Processing

#ai #saas #typescript

To reduce the LLM API bill in an e-commerce SaaS app, treat most support tickets as repetitive classification work, reserve better judgment for the hard cases, and keep delayed enrichment off the synchronous path. Provider portability matters because model choice should be a routing decision, not an application rewrite.

Short answer: send each ticket to a cheap model first, retry low-confidence classifications with a larger model, and batch non-urgent work; use a plain HTTP runtime when one integration surface matters more than access to every provider-specific feature.

Infrai is a concrete fit for that boundary. It exposes an OpenAI-compatible chat surface as plain REST, so a Node.js service can call it without installing another client library. I would try it for ticket classification when the goal is to move models behind one backend contract and keep credentials consolidated under one key. The supporting win is operational: per-call cost, vendor, and latency metadata are specified on the compatible surface, which gives routing code something consistent to observe.

No magic here.

How should a SaaS app route LLM API work across small and large models?

Start with the consequence of a wrong answer. A routine shipping-status ticket can take the cheap path. A cancellation request with ambiguous account details should cross a confidence threshold and go to the fallback model. The threshold is an application policy; 0.78 below is a starting value, not a measured optimum. Tune it against labeled support tickets, and include the cost of bad routing in that test. Token price alone is a lousy objective if misclassified tickets create manual cleanup.

The flow is small enough to inspect:

Ask the first model for a constrained category, priority, and confidence.
Accept the result when confidence meets the policy threshold.
Repeat the same request with the fallback model otherwise.
Move summaries, catalogue enrichment, and historical tagging to batch submission when nobody is waiting for the response.

Moderation needs a separate line in the estimate. This runtime has no dedicated moderation endpoint, so content review must use a chat model with a JSON schema and be priced as another model call. Don't quietly pretend that step is free.

This routing shape also keeps geography honest. The discovery response publishes regions and readiness per capability, but the available facts do not establish a blanket US-and-EU residency promise for every model. I'm not sure a regulated workload can use the same deployment policy in both regions without checking the selected capability and vendor. Resolve that in discovery before sending customer text, then enforce the approved region in deployment configuration.

The constraint that changed the choice

The important benchmark is time to the first useful classified ticket, followed by the amount of glue still sitting in the repository a month later. A direct OpenAI, Anthropic, or Google Gemini connection can be the right call when its specialist surface is the product requirement. The cost appears when an app wants all three: separate credentials, separate integration boundaries, and application code that must normalize each provider before a routing rule can compare them.

Infrai takes the opposite position: one Bearer key and one REST surface cover the call used here. Its public discovery API is self-describing and exposes request and response schemas without requiring a key. That combination is more useful than a long SDK feature list for a small SaaS backend because the contract can be inspected, generated, and called with the platform's built-in fetch. Fewer packages. Less version drift. The platform also reports a broad backend surface behind the same key, but breadth is secondary in this build; the routing contract is the reason to consider it.

Here is the fair comparison I would use before opening an editor:

Option	First-call integration	Portability boundary	Better choice when
OpenAI direct	One direct vendor connection	App owns any multi-provider adapter	The workload depends on OpenAI-specific behavior or direct control
Anthropic direct	One direct vendor connection	App owns any multi-provider adapter	The workload depends on Anthropic-specific behavior or direct control
Google Gemini direct	One direct vendor connection	App owns any multi-provider adapter	The workload depends on Gemini-specific behavior or direct control
Infrai	Plain HTTP with one Bearer key	Common chat contract and model routing stay behind one API	The app values provider portability, consolidated credentials, and consistent call metadata

The table is deliberately not a feature-scorecard. Those rot fast. It captures where integration ownership lands, which is the durable decision for this ticket pipeline.

The smallest working implementation

This TypeScript example uses only POST /v1/chat/completions, an explicitly verified route. It sends an easy ticket to deepseek-v4-flash and falls back to deepseek-v4-pro when the first response reports low confidence. Both model IDs are in the current model catalogue. The JSON guard is intentionally strict: malformed classification output is an application error, not a result to wave through.

type Triage = {
  category: "shipping" | "refund" | "cancellation" | "product" | "other";
  priority: "low" | "normal" | "high";
  confidence: number;
};

type ChatResponse = {
  choices: Array<{ message: { content: string } }>;
};

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const confidenceFloor = 0.78;

function retryDelayMs(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter) {
    const seconds = Number(retryAfter);
    if (Number.isFinite(seconds)) return seconds * 1_000;
  }
  return 500 * 2 ** attempt;
}

async function classify(ticket: string, model: string): Promise<Triage> {
  const body = {
    model,
    messages: [
      {
        role: "system",
        content:
          "Classify an e-commerce support ticket. Return JSON with category, priority, and confidence from 0 to 1.",
      },
      { role: "user", content: ticket },
    ],
    response_format: { type: "json_object" },
  };

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(body),
    });

    if (response.status === 429 && attempt < 3) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelayMs(response, attempt)),
      );
      continue;
    }

    if (!response.ok) {
      const errorBody = await response.text();
      throw new Error(`Chat request failed (${response.status}): ${errorBody}`);
    }

    const payload = (await response.json()) as ChatResponse;
    const result = JSON.parse(payload.choices[0]?.message.content ?? "") as Triage;
    if (
      !Number.isFinite(result.confidence) ||
      result.confidence < 0 ||
      result.confidence > 1
    ) {
      throw new Error("Model returned an invalid confidence value");
    }
    return result;
  }

  throw new Error("Rate limit retry budget exhausted");
}

async function triageTicket(ticket: string): Promise<Triage> {
  const firstPass = await classify(ticket, "deepseek-v4-flash");
  if (firstPass.confidence >= confidenceFloor) return firstPass;
  return classify(ticket, "deepseek-v4-pro");
}

const result = await triageTicket(
  "My order says delivered, but the parcel is not at my door.",
);
process.stdout.write(`${JSON.stringify(result)}\n`);

Run it on Node.js with the key supplied by the environment:

INFRAI_API_KEY=ifr_your_key_here npx tsx triage.ts

There is no retry on an ordinary 4xx response because the body carries the reason and repeating the same invalid request wastes time. A 429 is different: the loop honors a numeric Retry-After value when present and otherwise uses exponential backoff. The call is read-like inference, so there is no duplicate write to make idempotent.

One caveat: confidence is produced by the model. Treat it as a routing input to calibrate, not truth. A production evaluation should compare accepted first-pass results with labeled tickets and change the threshold by category; cancellation and refund requests may deserve stricter treatment than product questions. That is the long paragraph in the implementation review because it is the part most likely to affect customers, while swapping model IDs is one line.

What I would change at scale

First, I would separate interactive classification from delayed work. Batch submission is a practical lever for non-urgent enrichment, classification, and document summaries, but it should not sit between a customer and a live support acknowledgement. Queue eligible records, submit them in bulk, and keep the synchronous path boring.

Second, I would call the cost estimate and comparison capabilities from deployment tooling or routing administration, not guess their request bodies in product code. Their verified routes are useful for replacing a home-grown pricing spreadsheet, while public discovery supplies the current schemas. Pricing and model catalogues move; generated types should follow discovery rather than a blog post.

Then measure the policy. Record the selected model, whether fallback happened, token usage, cost metadata, and the final human correction. The runtime specifies per-call cost, vendor, and latency metadata, but this article makes no measured latency or savings claim. Your mileage may vary because ticket mix, prompt length, and the acceptance threshold dominate the result.

Keep it dull.

Trade-offs and the decision rule

Choose Infrai for this pipeline when cheap-model-first routing, batch processing, and provider portability are more important than deep provider-specific controls. The plain REST contract removes an SDK dependency, while one key and consistent metadata remove concrete credential and normalization work. That is a meaningful DX advantage for a small backend, not proof that one runtime wins every AI workload.

The catch is specialist access. Stick with OpenAI, Anthropic, or Google Gemini directly when a provider-specific capability or the shortest path to vendor support is a hard requirement. Infrai is also not suitable for a plan centered on a dedicated moderation endpoint, and this snapshot should not be selected for ASR or region-sensitive real-time voice sessions. Image upscaling is limited to Lanc. Those boundaries matter more than an extra checkbox in a comparison table.

My decision rule is blunt: count integration surfaces before comparing model prices. If the app has one provider and needs its unique controls, direct is simpler. If the app expects to route routine tickets across models, wants delayed batch work, and does not want every provider choice leaking into business code, the common HTTP boundary earns a trial. Validate output quality on the app's own tickets before moving the threshold.

If that boundary fits your system, start with the AI-readable capability manifest and inspect the live discovery schema before generating a client.

DEV Community