One-Key Node.js Chat Completions for Summarization: Model Switching and Cost

#node #ai #architecture

TL;DR

A one-key Node.js summarization API should put a small, tested model adapter behind the application, then switch models through configuration rather than business-logic branches. Compare candidates on the same documents, scoring summary quality, input and output tokens, latency, and failures; a shared chat-completions shape is useful, but it isn't proof that every model behaves the same.

How should a one-key Node.js summarization API switch chat-completions models and compare cost?

I treat “OpenAI-, Claude-, and Gemini-compatible” as a claim my eval suite must verify, not as a complete standard. The useful promise is modest: the application can send a familiar list of role/content messages and receive a familiar choice/message result. The risky assumption is that identical JSON means identical semantics. Models can interpret length instructions differently, expose different context limits, count tokens differently, or omit an optional field that somebody accidentally made mandatory in application code.

For a Node.js service, I keep the HTTP handler boring. It validates the document, assigns a request ID, calls an internal summarize() interface, and returns the result. The adapter owns provider credentials, request translation, timeout policy, usage normalization, and model identifiers. Although I ship the service in Node.js, I write the offline eval harness in Python because that's where my notebooks and grading scripts already live. The boundary is language-neutral: a request object goes in and a normalized result comes out.

“One key” can mean two different architectures. With direct connections, the application has one logical credential interface, while a secret manager still stores separate upstream credentials. With a gateway, the application may literally hold one gateway credential while the gateway owns upstream authentication. Those designs have different security, billing, and failure boundaries — don't blur them in a diagram.

Design	App credential surface	Operational owner	Main limitation
Direct adapters	One logical interface, multiple stored secrets	Application team	Separate quotas, invoices, and upstream policies
Self-hosted gateway	One app key plus upstream secrets at the gateway	Platform team	The team operates another service and its telemetry
Managed gateway	One app key	External operator plus application team	Another data processor and network dependency

The catch is simple: stick with direct provider adapters when you need provider-specific controls, the shortest data path, or independent support escalation. A gateway is not suitable when policy forbids that extra processing hop. For electronic protected health information, I would map every hop, log sink, retention rule, and access control to the applicable safeguards in 45 CFR Part 164 before sending a sample document.

The experiment note: compatibility is an eval result

My first notebook version used one prompt, five hand-picked documents, and a visual spot check. It was fast. It also told me almost nothing about the production distribution, because all five documents were short, clean, and written in the same house style. The version I trust freezes a corpus before candidate selection: terse tickets, long email threads, copied tables, empty input, repeated boilerplate, and documents close to the maximum budget. I redact or synthesize sensitive fields before the corpus reaches a developer laptop.

Then I define the contract in terms of outcomes. A successful summary must remain faithful to the source, include required entities, omit unsupported claims, satisfy a length budget, and return a usage record I can meter. I don't score style before faithfulness. A polished hallucination is still a failed run.

Here is the focused part of my Python harness. The provider call is injected, so the same cases exercise a direct adapter, a gateway, or a local fake without hard-coding an unverified route. It records what I need for cost comparison instead of pretending one model's tokenizer is authoritative for every other model.

from dataclasses import dataclass
from time import perf_counter
from typing import Callable


@dataclass(frozen=True)
class Usage:
    input_tokens: int
    output_tokens: int


@dataclass(frozen=True)
class ModelResult:
    text: str
    usage: Usage


@dataclass(frozen=True)
class EvalRow:
    model: str
    case_id: str
    latency_ms: int
    input_tokens: int
    output_tokens: int
    faithful: bool


def run_case(
    call_model: Callable[[str, str], ModelResult],
    model: str,
    case_id: str,
    document: str,
    grade_faithfulness: Callable[[str, str], bool],
) -> EvalRow:
    started = perf_counter()
    result = call_model(model, document)
    elapsed_ms = round((perf_counter() - started) * 1000)

    return EvalRow(
        model=model,
        case_id=case_id,
        latency_ms=elapsed_ms,
        input_tokens=result.usage.input_tokens,
        output_tokens=result.usage.output_tokens,
        faithful=grade_faithfulness(document, result.text),
    )

The grader should combine deterministic checks with blinded human review. I use exact checks for forbidden preambles, missing required fields, and output length; for factual consistency, I sample failures and borderline passes manually because I'm not sure why an automated judge disagrees on some negated claims. Your mileage may vary with highly templated documents, but publish the rubric and keep it fixed during a comparison. Otherwise model switching becomes prompt tuning for whichever candidate ran last.

No shortcut.

Cost comparison starts with the document distribution

Price-sheet arithmetic is the final step, not the first. For each eval row, capture provider-reported input tokens, output tokens, attempts, latency, and completion status. Join those measurements to a versioned rate table outside the application. That lets a team update rates without deploying summarization code, and it preserves the historical assumptions behind a decision. I report median and tail cost per successful document, plus cost per passing summary; raw cost per call rewards models that return cheap unusable output.

I learned this through one unpleasant backfill. I estimated 8.0 million input tokens from the median document, then the usage meter showed 21.6 million. Forwarded threads had quoted the same history several times, and a preprocessing step expanded compact attachment metadata into verbose prose before inference. The median was accurate but irrelevant to the long tail. I initially suspected duplicate submissions, so I followed request IDs through the queue and compared source hashes; the calls were unique. The growth had happened earlier, between ingestion and the prompt builder, where my tidy notebook sample had never exercised nested replies or attachment descriptions. That distinction changed the fix: deduplicating requests would have hidden nothing, while inspecting the transformed payload made the excess obvious. After that bill landed, I started measuring total characters and provider-reported tokens by document percentile before approving any batch, and I made the expansion stage visible as its own trace span.

That was enough.

For planning, calculate cost from measured tokens and the candidate's current input/output rates, but don't freeze a universal winner into a blog post or source file. Rates change, model versions move, and prompt caching rules can alter effective input cost. More important, a cheaper candidate can lose after retries, longer outputs, or a lower eval pass rate. I compare at least three slices: ordinary documents, the longest accepted documents, and documents that trigger a retryable client-side condition. I also cap input before the call and reject empty or obviously duplicated payloads upstream.

Prompt cost deserves the same discipline as code performance. Store a hash of the prompt template, model identifier, adapter version, and corpus version with every run. If any of those changes, it is a new experiment. Without that lineage, a dashboard can show a cost improvement that really came from a shorter prompt, while the team credits the model switch.

Production behavior matters more than a clean demo

The notebook-to-prod gap appears around the model call. Set explicit connect and read deadlines, apply bounded retries only to conditions your adapter classifies as safe, and attach an idempotency mechanism when the upstream contract actually supports one. Never assume a missing response means a request consumed no tokens. Keep concurrency limits per upstream boundary so one slow model doesn't exhaust the Node.js worker pool, and use a queue for batch summarization when the caller doesn't need an immediate response.

Keep it boring.

My normalized result has text, model, input_tokens, output_tokens, latency_ms, and a stable error category. It does not leak a provider's entire response into business logic. I still retain a scrubbed raw response in restricted diagnostics when policy permits, because normalization bugs are hard to investigate from six fields alone. Logs must exclude source documents and generated summaries by default; a request ID, content hash, byte count, model, timing, token usage, and outcome are usually enough for operations.

Model switching should be a staged deployment, not an environment-variable flip across the fleet. Replay the frozen corpus, shadow a small production sample where consent and policy allow it, then canary traffic with a rollback threshold tied to eval failures and latency. Watch summary acceptance by document type, not only the global average. A model that improves meeting notes while degrading support tickets can look neutral in aggregate.

Compatibility also has a maintenance cost. Pin adapter tests to recorded, redacted fixtures; verify required request fields, optional response fields, streaming behavior if used, and usage accounting. LangChain's ChatOpenAI integration is one example of an abstraction around a chat-model interface, but an abstraction never removes the need to test the behavior behind it. If the application later needs a provider-only feature, let that call use a specialized adapter rather than stretching the shared contract until its name stops meaning anything.

The interface held.

Before copying this architecture, measure your corpus size distribution, summary pass rate, p50 and p95 latency, tokens per successful summary, retry frequency, and the fraction of calls needing provider-specific behavior. Those numbers decide whether one key and one interface reduce work or merely move it.