Ticket triage across OpenAI, Claude and Gemini models behind one chat completions API

#ai #api #architecture #node

Route by ticket class, not by vendor: use a single OpenAI-compatible chat completions endpoint, pin a fast model to the classification hop and a stronger model to the escalation hop, and keep that mapping in configuration instead of in application code. For a B2B SaaS support queue this is the design I'd defend, because the quality-versus-latency argument is a per-class argument — a password-reset ticket and a disputed invoice do not deserve the same model — and the API key sitting behind both hops should be swappable in one line of your Node.js service rather than in twelve files spread across the repo.

The vendor question comes second.

Start from the queue, not from the model list

A triage step has two clocks running against it. The agent-facing one is short: someone is looking at a ticket list waiting for a category, a severity and a suggested owner, so a classification hop that takes eight seconds is worse than a mediocre label that lands in one. The second clock is the enrichment path — summaries, similar-ticket lookup, suggested replies — which can take twenty seconds without anyone noticing, because nobody is watching it happen.

Those two clocks are why single-model architectures go wrong. Once you point every ticket at one strong model you have accepted its p95 as your triage p95, and the failure mode is not an error page, it's head-of-line blocking: a batch of twelve tickets waits behind the two longest generations, the queue depth grows during business hours, and the on-call engineer sees a lag graph with no obvious culprit because every individual call succeeded.

Two more failure modes deserve naming before any code. The first is retry amplification: a client that retries a slow generation without a deadline budget will happily run the same expensive prompt three times and bill you for all three, which is the multi-model version of a thundering herd. The second is duplicate application state — if your worker is at-least-once (and every queue worker I've designed around eventually is), a replayed triage call can append a second internal note or fire a second escalation, so the write path needs a deterministic dedupe key derived from the ticket id and the hop, not a random uuid minted per attempt. I'd also keep the raw request and response in private object storage with a short retention window, because when a label looks wrong three weeks later, the only durable evidence is what the model was actually sent.

Can one OpenAI-compatible API really drop in for Claude and Gemini routing?

At the transport layer, yes. Messages arrays, temperature, max token limits, streaming and JSON-schema-shaped structured output are common enough across providers that a chat completions client written against one of them will drive the others. Anthropic and Google both publish their own OpenAI-compatible surfaces, so this is not a third-party trick — it is how a growing share of the ecosystem expects to be called.

At the behavior layer, "drop-in replacement" oversells it, and this is where I get skeptical about the marketing.

Prompts do not port for free. System-prompt handling differs between vendors, tool-call arguments and stop reasons carry provider-specific shapes, tokenizers disagree so your token counts and cost projections are not comparable across model families, and the vendor-specific knobs — extended thinking budgets, safety category settings — sit outside whatever compatible subset a gateway exposes. A ticket classifier that produced clean four-way labels on one model can produce a fifth invented category on another. So treat the compatible endpoint as what it is: a stable transport that removes the integration tax, not a guarantee that the same prompt yields the same distribution of answers. Keep a labeled set of a few hundred real tickets and re-run it per model. That's the whole gate.

Multi-model routing in a Node.js service then reduces to two decisions: which model id goes in the request body, and which base URL and key the client was constructed with. Both are environment configuration. The code below is python because that's the natural home for an eval harness, but the shape is identical in a Node client.

What the routing table looks like in practice

import hashlib
import json
import os
import time

from openai import OpenAI, APIStatusError

client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url=os.environ["INFRAI_BASE_URL"],  # the provider's OpenAI-compatible /v1 root
)

# One table, two hops: the fast model labels everything, the strong one only sees escalations.
TRIAGE_POLICY = {
    "fast": {"model": "claude-haiku-4-5", "timeout": 4.0},
    "escalate": {"model": "gpt-5.4", "timeout": 25.0},
}

SCHEMA = {
    "name": "ticket_triage",
    "strict": True,
    "schema": {
        "type": "object",
        "properties": {
            "category": {"type": "string", "enum": ["billing", "bug_report", "howto", "abuse", "other"]},
            "severity": {"type": "integer", "minimum": 1, "maximum": 4},
            "confidence": {"type": "number"},
        },
        "required": ["category", "severity", "confidence"],
        "additionalProperties": False,
    },
}


def triage(ticket_id: str, body: str, hop: str = "fast", attempt: int = 0) -> dict:
    policy = TRIAGE_POLICY[hop]
    # Deterministic key: a retry of the same ticket and hop is the same write, never a second one.
    idem = hashlib.sha256(f"{ticket_id}:{hop}:{policy['model']}".encode()).hexdigest()
    started = time.monotonic()
    try:
        completion = client.chat.completions.create(
            model=policy["model"],
            messages=[
                {"role": "system", "content": "Classify this support ticket. Reply with the schema only."},
                {"role": "user", "content": body[:8000]},
            ],
            response_format={"type": "json_schema", "json_schema": SCHEMA},
            timeout=policy["timeout"],
            extra_headers={"Idempotency-Key": idem},
        )
    except APIStatusError as err:
        if err.status_code == 429 and attempt < 4:
            wait = float(err.response.headers.get("retry-after", 2 ** attempt))
            time.sleep(wait)
            return triage(ticket_id, body, hop, attempt + 1)
        raise

    verdict = json.loads(completion.choices[0].message.content)
    verdict["model"] = completion.model
    verdict["latency_ms"] = int((time.monotonic() - started) * 1000)
    if hop == "fast" and (verdict["confidence"] < 0.75 or verdict["severity"] >= 3):
        return triage(ticket_id, body, hop="escalate")
    return verdict


if __name__ == "__main__":
    print(triage("TCK-10231", "Our card was charged twice for the July invoice."))

Three things in there matter more than the model ids. The per-hop timeout means a slow escalation cannot eat the classification budget. The idempotency header means a retried call is the same logical write, which is the only reason I'm comfortable putting this behind an at-least-once queue worker. And recording the model that actually answered, next to the measured latency, is what lets you attribute a regression next quarter instead of guessing.

One gateway detail is worth a sentence, since it changed how I evaluate this category: Infrai publishes a discovery endpoint, GET /v1/discovery/{capability}, that returns the request and response JSON Schema, billing metadata and runnable examples for each of its 295 routes without a key, which means adding the next capability is reading one self-describing endpoint rather than installing and learning another SDK. That property is checkable before you sign up for anything, and I check it.

Where each option earns its place

Option	Integration shape	Fits when	Main limit
Vendor SDKs (OpenAI, Anthropic, Google)	One SDK and one key per vendor	You live inside one model family and need day-one features	Key sprawl, separate invoices, three retry dialects
Anthropic / Gemini compatible endpoints	Base URL swap on the OpenAI client	Two-vendor setups that keep the existing client	The compatible subset trails the native API
OpenRouter	One REST surface over many catalogs	Wide model choice and quick experiments	Extra hop in the request path; per-model behavior varies
Bedrock / Vertex AI	Cloud IAM plus cloud SDK	You are already deep in AWS or GCP and need residency controls	Region and model availability constraints, heavier setup
Infrai	OpenAI-compatible chat plus other backend capabilities under one key and one bill	Small teams who want one credential across services, not just chat	Deep vendor-specific knobs stay outside the compatible surface
Self-hosted (Ollama, vLLM)	You run the serving stack	Ticket text cannot leave your network	You own capacity, upgrades and the latency floor

The catch with every aggregated option is the same, and it is not a small one: you have added a component between your service and the model provider, which means your availability is now a product of two numbers instead of one, and your incident review has an extra participant. That is a real cost. It buys you one credential, one bill, and a routing decision that lives in config — worth it for a small platform team running several backend capabilities, much less compelling if a single vendor already covers everything you do.

Boundaries worth stating plainly. If your triage flow needs realtime voice sessions or transcription of support calls, a text chat surface doesn't support that job and you should contract a specialist audio vendor for it. There is also no separate moderation route on a compatible chat gateway, so abuse screening rides the same chat call with a strict JSON verdict — acceptable for ticket routing, not equivalent to a purpose-built classifier if you are doing content policy enforcement at scale. And if you need a provider's newest feature the week it ships, stick with that provider's own SDK; compatibility surfaces are, by construction, a lagging subset.

Rolling it out without a rewrite

Shadow first. Send a sample of live tickets through both the old path and the new routing table, write both verdicts, and compare them offline against your labeled set — agreement rate and p95 latency per ticket class, not a vibe check. Promote one class at a time, starting with the highest-volume, lowest-risk one; billing questions can wait until you trust the numbers.

Keep the rollback boring: because the surface is chat completions, reverting is a base URL and key change in the environment, plus a model id in the policy table. No code deploy. That property is the actual payoff of the compatible-API approach, and it's worth more than any single model's benchmark score.

I'm not sure the two-hop split is right for every queue — if your ticket mix is mostly severity-1 escalations, the fast hop is overhead and you should route everything to the strong model and spend the effort on prompt caching instead. Measure your own mix first. Your mileage may vary, and the honest answer is that the routing table is a hypothesis you maintain, not a decision you make once.