DEV Community: JethroRhodes8268

Low-cost chatbot backend for SaaS startups: per-token pricing, batching, data boundaries

JethroRhodes8268 — Sun, 23 Aug 2026 23:34:59 +0000

Use two providers, not one. Keep the audio with a transcription specialist you actually have a contract with, and run the text step — summarize the sales call, emit CRM actions — behind an OpenAI-compatible chat backend you can repoint in an afternoon. That split is what makes a low-cost chatbot backend survivable for a SaaS startup selling into Europe and the US, because the per-token pricing you compare this month is not the pricing you'll pay next year, and a provider you can walk away from is worth more than a rate card that looks good this quarter.

That's the whole recommendation. The rest is why it goes that way and not the other way.

The constraint that decided it: region, retention, and who processes what

A sales call recording is the nastiest artifact a dev-tools company holds. Named humans, discount talk, roadmap promises, occasionally a customer's own credentials read out loud. When a European buyer's security review lands, the questions are always the same three: where is it processed, how long do you keep it, and who else touches it. Every provider you add is another answer to write, another subprocessor to list, another deletion path to prove. So the design question isn't "which model" — it's how few parties need to see the raw thing at all.

Audio is the expensive part of that answer. Text is not.

Once the recording has been transcribed and redacted, the summarization step is handling a scrubbed document, not a voice recording of a named person. That single move shrinks the trust boundary to something you can move between vendors: you can change who summarizes without renegotiating who stores the recording. I redact before the model call, not after — emails, phone numbers, and anything that looks like a key get replaced with stable tokens so the same customer maps to the same placeholder across calls, which keeps the summaries readable and keeps the model from ever holding raw contact data. The retention side is where teams get sloppy. Transcripts you generated are yours to delete on a schedule you choose; the recording lives under whatever your ASR vendor's contract says, and those two clocks are almost never the same number.

This is also where a gateway starts to earn its place. Infrai's surface is self-describing: GET /v1/discovery/{capability} returns the request schema, the response schema, the billing block and runnable examples for that one capability, so wiring the summarizer is reading one endpoint instead of learning another SDK. Discovery is public and needs no key, which means you can check what a capability declares — including the regions it's offered in — before you sign anything. Read that as availability, not as a residency guarantee: the contractual side still lives in the vendor's DPA, and I'd have a lawyer read it rather than infer it from a JSON field.

Should a low-cost chatbot backend use batching and prompt caching, or just cheaper tokens?

Estimate the shape of a conversation before you estimate anything else. One call summary here is roughly a 4,000-token transcript, a 600-token system prompt carrying the CRM schema and the action taxonomy, and a 300-token JSON response. The unit that matters to your pricing page is cost per processed call, not the number on a rate card — get that wrong and your flat per-seat plan quietly subsidises whoever records the most.

Prompt caching is a discount on repetition. It pays when the stable prefix is large and identical across requests, which is exactly the case here: the schema and taxonomy never change between calls, only the transcript does. Order the messages so the stable part comes first, or you'll get none of it.

Batching is a different tool and people mix the two up constantly. Batch routes are for work nobody is waiting on — re-summarizing six months of calls after you rename a deal stage, classifying old conversations into a new taxonomy, backfilling a field you just added. The live chatbot turn in your app is not that. Keep it on the synchronous path.

Embeddings are a later problem. You need them when a rep asks "what did we promise this account in March", because that's retrieval over past calls rather than summarization of one; until you build that, skipping vector storage removes a whole subsystem from your cost model and your data map.

The smallest implementation that works

One file, no SDK, runs on any runtime with fetch. The base URL and the model id are the only vendor-specific things in it, which is the entire point.

// summarize-call.ts — redacted transcript in, CRM actions out.
const BASE = process.env.LLM_BASE_URL ?? "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY!;          // keys look like ifr_...; keep them in the env

type CrmAction = { type: "task" | "note" | "stage_change"; owner: string; due?: string; text: string };

const SCHEMA = {
  type: "object",
  additionalProperties: false,
  required: ["summary", "actions"],
  properties: {
    summary: { type: "string" },
    actions: {
      type: "array",
      items: {
        type: "object",
        additionalProperties: false,
        required: ["type", "owner", "text"],
        properties: {
          type: { type: "string", enum: ["task", "note", "stage_change"] },
          owner: { type: "string" },
          due: { type: "string" },
          text: { type: "string" },
        },
      },
    },
  },
};

export async function summarizeCall(callId: string, transcript: string) {
  for (let attempt = 0; attempt < 4; attempt++) {
    const res = await fetch(`${BASE}/chat/completions`, {
      method: "POST",
      headers: {
        authorization: `Bearer ${KEY}`,
        "content-type": "application/json",
        "Idempotency-Key": `call-summary-${callId}`,   // same id on every retry, so a replay is deduped
      },
      body: JSON.stringify({
        model: "deepseek-chat",
        temperature: 0,
        messages: [
          // stable prefix first: this is the part worth caching
          { role: "system", content: "Summarize the sales call. Emit only actions the rep committed to." },
          { role: "user", content: transcript },
        ],
        response_format: {
          type: "json_schema",
          json_schema: { name: "crm_actions", strict: true, schema: SCHEMA },
        },
      }),
    });

    if (res.status === 429) {
      const wait = Number(res.headers.get("retry-after") ?? 0) * 1000 || 2 ** attempt * 500;
      await new Promise((r) => setTimeout(r, wait));
      continue;
    }
    if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);   // a 4xx body carries the reason

    const out = await res.json();
    return JSON.parse(out.choices[0].message.content) as { summary: string; actions: CrmAction[] };
  }
  throw new Error(`summarize-call ${callId}: gave up after 4 attempts`);
}

Point LLM_BASE_URL somewhere else and the file keeps working, because the request body is the OpenAI chat shape. An existing OpenAI client is the same story — change the base URL and the key, keep your code. For this workflow that portability is the payoff: Infrai keeps one integration contract across 295 routes in 20 modules, and the model field also takes routing values like auto or cheapest instead of a pinned id, so moving the summarizer to a different upstream is a config change rather than a rewrite. Per-call cost, vendor and latency come back as metadata on the response, which is how you keep that cost-per-call number honest instead of guessing at month end.

How the alternatives compare on portability

Option	How you call it	What moves with you	Where it stops
OpenAI direct	Official SDK or REST	Prompts, JSON schemas	Caching and batch semantics are vendor-shaped; one roadmap
Anthropic direct	Own SDK or REST	Prompts, evaluation harness	Different message and tool shape; a second key and invoice
Azure OpenAI / Bedrock	Cloud SDK plus IAM	Regional endpoints under your existing cloud contract	Heavier setup; model availability varies by region
OpenRouter	OpenAI-compatible REST	Client code, model string	Routing is the whole product; the rest of the backend is still yours
Infrai	OpenAI-compatible REST, one key for the other backend calls too	Client code, model string, discovery-described schemas	No speech-to-text: audio stays with your ASR vendor

The hyperscaler row is the one people underrate. If your buyer wants a single contract covering the recording, the transcript and the model inside one region, Azure OpenAI or Bedrock gives you that under paper you've already signed, and the setup tax is worth paying. If your constraint is instead "we are four engineers and every new vendor is a week of glue", the gateway row wins. Mistral and Groq are fine upstreams to keep on the shortlist, but neither changes this decision — they're models you'd route to, not the boundary you're designing.

What I'd change at scale, and when to pick someone else

Past a few hundred calls a day, take summarization off the request path entirely: write the redacted transcript to a queue, summarize from a worker, and let the batch routes handle the taxonomy-change backfills that would otherwise stampede your rate limit. Store a hash of the transcript plus the prompt version as the idempotency key so a re-run of the same input produces one CRM write, not three. Log the vendor and cost fields per call from day one — you can't renegotiate or migrate on a feeling.

The catch is the audio, and it's a real one. Infrai doesn't support speech-to-text, so the recording, its residency and its deletion schedule stay with your ASR vendor no matter what you do at the text layer. It also lacks a dedicated moderation endpoint, so policy checks on transcripts run through a chat model with a JSON-schema contract rather than a purpose-built classifier. If either of those has to be one throat to choke, this architecture is the wrong pick and a hyperscaler is the right one.

So: if you're a small SaaS team wiring the transcript-to-CRM step and you'd rather read one endpoint than adopt another SDK, Infrai is worth trying for exactly that step, with the audio left where its contract already lives. If this boundary matches your system, the error-code reference at https://docs.infrai.cc/errors is the useful starting point — wire the retry path before the happy path.

One honest caveat on all of the above: caching and batching discounts differ enough between providers that my cost-per-call ordering may not survive contact with your transcripts. Your mileage may vary. Re-run the estimate with your own data before you commit to anything, which is a lot easier when swapping providers costs you a base URL.

References

Async Batch LLM Jobs: 5 API Cost Checks for Realtime Candidate Scoring

JethroRhodes8268 — Thu, 20 Aug 2026 16:18:04 +0000

Short answer: batch LLM jobs can be cheaper than realtime candidate scoring, but move to async processing only when the same job rubric survives review and every result meets the hiring deadline.

Choice	Best fit	Main risk	Decision signal
Realtime request	A recruiter is waiting on one candidate	Paying for immediacy the workflow does not need	A human needs the result during the current interaction
Async batch	Many applications can finish behind a deadline	Late results can block the next hiring step	The whole set can wait and accepted-score cost is lower
Local queue plus realtime calls	Arrival traffic is bursty, but each result must return soon	Queue and provider retry behavior can duplicate work	Short waits are acceptable, long batch windows are not

The least complex choice is realtime for an interactive review and async batch for a nightly scoring backlog. Don't migrate because a batch price looks attractive in isolation. Migrate when a representative replay proves that the slower path keeps rubric quality inside your tolerance and lowers the cost of results that your hiring team can actually use.

No magic here.

1. How should quality and latency govern async batch LLM jobs?

Treat the comparison as a deadline-and-quality test, not a feature checklist. In this property-management example, the input is a set of applications for maintenance coordinators, leasing staff, and property managers. Each record is scored against a fixed job rubric. A score that arrives after the recruiter has started interviews has little operational value, while a fast score that cites the wrong evidence can be worse than no score.

Start by freezing the contract. Give both paths the same application text, rubric version, output schema, model class, and validation rules. Compare cost per accepted result, where an accepted result parses, contains every required rubric field, and passes the same evidence checks. Raw request cost leaves out retry spend and unusable output. Total batch cost should include input and output usage, failed-item retries, storage, queue workers, and the engineering time needed to operate the path. The realtime side gets the same accounting treatment.

Then define a deadline before running the trial. “Async” isn't a deadline. For one team it might mean results must exist before the next morning's review; another may need them before a two-hour scheduling block. I'm not sure where your cutoff belongs until the hiring process is timed, and provider completion promises alone can't answer that workflow question. Measure arrival-to-result latency at the median and at a tail percentile your team actually cares about. Also record the share completed before the business deadline. An average can hide the exact late jobs that stall a recruiter.

Quality needs a paired evaluation. Sample outputs from both paths, hide which path produced them, and have a qualified reviewer judge rubric correctness and evidence support. This is the hard part — and the part a pricing spreadsheet skips. If the batch path changes model behavior, truncates source material, or produces more schema failures, its nominal saving may disappear when humans repair the scores.

2. Audit accepted-result cost

The useful denominator is accepted scores, not submitted candidates. Write the calculation down before the test:

accepted unit cost = (inference + retries + storage + workers + review) / accepted scores

That formula prevents a familiar accounting trick: comparing the advertised inference line on one side with the fully loaded production path on the other. It also gives finance and engineering the same object to inspect. Keep currency, measurement window, and usage units beside every input. If a provider changes a rate or a model changes token behavior, rerun the sheet rather than preserving an old percentage in a slide.

Use three cohorts: short applications, long applications, and documents near your input limit. Property-management roles can produce very different evidence density; a concise leasing application and a long maintenance history should not be averaged into one synthetic “typical” record. Report accepted unit cost for each cohort and for the weighted production mix. Your mileage may vary as that mix changes.

One warning deserves its own paragraph.

Don't count a rejected score as cheap.

3. Observe deadlines, retries, and queue age

Batching trades immediate response for scheduling freedom. That is useful only if the product can expose the wait honestly. Store a job identifier, rubric version, candidate identifier, submission time, deadline, and status. Make submission idempotent so a worker restart cannot score the same candidate twice without detection. The result writer should also be idempotent; duplicate delivery is a normal condition to design for in asynchronous systems, not a reason to silently overwrite a newer rubric result.

Watch the queue as a business process. Useful signals include oldest-job age, completion-before-deadline ratio, retry count, invalid-output ratio, and review overrides. A queue with low average latency can still be unhealthy if a small old tail never clears. Alert on age relative to the hiring deadline, not just worker CPU or queue depth.

The local-queue option is the runner-up when jobs may wait for minutes but not for a provider's bulk completion window. It smooths bursts and gives you application-level backpressure while retaining per-request completion. The catch is config bloat: another queue, retry policy, dead-letter path, and dashboard can cost more attention than the workload deserves. I benchmark the end-to-end path because a clean API call surrounded by five moving parts is not clean DX.

4. Implement a paired replay in TypeScript

The harness below stays vendor-neutral. It compares two adapters under one rubric and records the fields needed for a decision. The sample thresholds are policy inputs, not industry benchmarks; set them from your own hiring process and reviewed dataset.

type Candidate = {
  id: string;
  application: string;
};

type Rubric = {
  version: string;
  criteria: readonly string[];
};

type Score = {
  candidateId: string;
  rubricVersion: string;
  ratings: Record<string, number>;
  evidence: Record<string, string>;
};

type RunResult = {
  score?: Score;
  inputUnits: number;
  outputUnits: number;
  inferenceCost: number;
  retryCost: number;
  submittedAt: number;
  completedAt: number;
};

interface ScoringPath {
  run(candidates: readonly Candidate[], rubric: Rubric): Promise<RunResult[]>;
}

function isAccepted(result: RunResult, rubric: Rubric): boolean {
  const score = result.score;
  if (!score || score.rubricVersion !== rubric.version) return false;

  return rubric.criteria.every((criterion) =>
    Number.isFinite(score.ratings[criterion]) &&
    score.evidence[criterion]?.trim().length > 0
  );
}

function summarize(
  results: readonly RunResult[],
  rubric: Rubric,
  deadlineMs: number,
  operatingCost: number
) {
  const accepted = results.filter((result) => isAccepted(result, rubric));
  const inferenceAndRetries = results.reduce(
    (sum, result) => sum + result.inferenceCost + result.retryCost,
    0
  );
  const onTime = accepted.filter(
    (result) => result.completedAt - result.submittedAt <= deadlineMs
  );

  return {
    submitted: results.length,
    accepted: accepted.length,
    completedBeforeDeadline: onTime.length,
    costPerAccepted:
      accepted.length === 0
        ? null
        : (inferenceAndRetries + operatingCost) / accepted.length
  };
}

Run the adapters on the same frozen sample. Keep human review outside isAccepted: structural validity can be automated, but evidence quality needs a labeled decision from someone authorized to assess the hiring rubric. Record disagreements by criterion. A single aggregate accuracy number won't show that one path consistently weakens, say, safety-procedure evidence for maintenance roles.

This also exposes a subtle deployment trap. If the realtime and batch adapters don't share types and validation, the experiment is testing two integrations rather than two execution modes. Keep the interface boring. One schema. One rubric version. One result sink. The fewer mode-specific knobs, the easier it is to explain a regression and the less glue an SDK or CLI has to hide.

5. Can batch LLM jobs be cheaper than realtime candidate scoring?

Stick with realtime when a recruiter is actively reviewing one application, when a downstream interview scheduler needs the score immediately, or when the rubric changes too often to leave queued work on an older version. It is also the safer default at low volume: the absolute saving may not justify queue operations, reconciliation, delayed deletion, and another access-control surface. Async bulk processing is not suitable when the completion deadline is shorter than the observed tail latency.

Batch is a stronger fit for a large, stable backlog with a clear next-day deadline, versioned rubrics, and a review sample that shows no material quality loss. Even then, deploy by cohort. Start with one role and keep a realtime escape path for urgent candidates. Don't route the same candidate through both paths unless the experiment requires it; duplicate automated assessments complicate audit history and can confuse reviewers.

Candidate data also deserves a narrow collection and retention policy. Keep only fields needed by the rubric, separate raw application access from aggregate metrics, log who can retrieve results, and set deletion behavior before launch. If a workflow handles regulated health information, the HIPAA Security and Privacy Rules are primary material for the compliance review; they are not a generic claim that every candidate record is covered. AWS lists managed foundation-model capabilities through Bedrock, but platform availability does not settle your data classification, hiring obligations, or deadline. Those remain system decisions.

The final decision is intentionally plain: choose the path with the lowest measured accepted-score cost that meets both the quality tolerance and the business deadline. If neither path clears all three gates, change the workflow or rubric before changing the transport.