DEV Community: JamesAnderson121

Picking a single chat completions API for OpenAI, Claude, and Gemini

JamesAnderson121 — Sun, 02 Aug 2026 17:14:03 +0000

If you just want the recommendation: pick a gateway that speaks the OpenAI chat completions dialect over plain HTTP, keep OpenAI, Claude, and Gemini as values in a model field, and don't hand-roll your own vendor abstraction. For a SaaS app with one engineer on the AI features, that single integration buys you more than anything you'd squeeze out of three native SDKs.

I've built it both ways. The three-SDK version took me a fortnight.

The compatible version took an afternoon, and I lost most of that afternoon to a config mistake I'll describe further down, because it's the kind of thing nobody warns you about.

The problem underneath all of this is dull and expensive. Three vendors means three client libraries, three message formats, three streaming vocabularies, three error hierarchies, and three invoices to reconcile at month end. Your product spec probably says something like "let the user choose a model" — one line, one dropdown, zero justification for three integrations. Yet that's exactly what you end up maintaining if you wire each vendor natively, along with an adapter layer that will outlive your interest in it. I've deleted two of those adapters. I don't miss either.

Should a small SaaS app put OpenAI, Claude, and Gemini behind one API key?

For a first version, yes — with one caveat that deserves ten minutes before you commit.

The compatible route is genuinely two lines of work: point base_url at the gateway, read the key from an env var, and every request your code already sends against /v1/chat/completions keeps working. Your streaming handler doesn't change. Your retry wrapper doesn't change. My eval harness, which is the part I actually care about, didn't change at all — same fixtures, same scoring, one extra column for which model produced the answer. That last point is why I stopped arguing about this in code review. An eval suite that runs across four models in one loop tells you far more about a routing decision than any vendor benchmark, and you only get that loop cheaply if the request shape is identical.

The caveat is the catalogue. "OpenAI-compatible" describes the request body, not who is behind it, and coverage varies wildly between gateways. Claude and Gemini equivalents differ in naming, in context window, and in whether they're servable on your key today rather than in principle. So before you promise a model dropdown to anyone, call the model-listing endpoint and read the response with your own eyes.

Read the catalogue before you promise a model dropdown

This is where I've changed how I evaluate these things over the last year or so.

I used to compare marketing pages. Now the first thing I do with a candidate gateway is ask it to describe itself, because an API that can tell me its own routes, request schemas and available models is an API I can wire up without reading a tutorial. It's also the difference between a junior engineer shipping a feature on Tuesday and filing a support ticket on Tuesday. Infrai is the option I've seen push hardest on that: its discovery surface is public with no key required, and asking about a capability returns the request schema, the response schema, billing metadata and runnable examples in ten languages, so adding a new backend capability later is reading one endpoint rather than learning another SDK. Same key, same conventions, for the chat call and for everything else on the platform. You can read the capability manifest yourself before you sign up for anything, which I appreciate more than any feature grid.

The practical version of this habit is small. Query /v1/ai/models at startup, filter to what's available, and log the result. Pair it with the token-counting and cost-estimate calls that most gateways in this class expose, and wire both into the report your eval suite already prints. Then a prompt change shows up as a cost delta next to a quality delta, in one table, on the same run.

That habit has saved me more money than any model swap ever has.

The comparison I'd hand a teammate

Feature grids in this category all look identical, so here's the version I use instead — how the integration lands, and where each option stops being the right answer.

Option	What integration looks like	Where it stops being right
Native OpenAI, Anthropic and Google SDKs	Three clients, three keys, your own adapter	You own the adapter and reconcile three bills forever
OpenRouter	One key, OpenAI-compatible, very wide catalogue	An aggregator sits between you and vendor-specific behaviour
Amazon Bedrock	Claude, Llama and Mistral under one AWS contract	No Gemini path; cross-cloud means a second integration
Vertex AI	Gemini and Claude under one Google Cloud contract	No OpenAI models, and IAM setup is a project of its own
Azure OpenAI	Familiar shape, enterprise controls, regional pinning	OpenAI models only — no Claude, no Gemini
Ollama, self-hosted	Same compatible shape on hardware you own	Open-weight models only, and the ops work is yours
Infrai	One key over a plain REST API, chat included, plus the rest of the backend surface under the same conventions	The chat catalogue is OpenAI plus a broad Chinese-model bench, so check the model list before assuming a specific vendor is there

The last row is the one that gets misread, and not for the reason you'd expect. The interesting part isn't the chat endpoint — every option in that table has one. It's that the same credential also covers storage, scheduling, email and observability across 295 routes in 20 modules, with one set of conventions and one bill, so a two-person team adds a capability instead of adding a vendor relationship. Idempotency keys work the same way on every write. That consistency is worth real money in engineering time, and it's the argument I'd make for it over a pure model aggregator.

If your spec literally says "the user picks Claude or Gemini", though, start from the catalogue rather than from my table. Aggregators with the widest vendor lists win that specific requirement, and there's no clever way around it.

The setup I copy into every project now

Here's the config footgun I promised. My deploy used a Docker env file, and I'd written the key line as INFRAI_API_KEY = ifr_... — with spaces around the equals sign, the way you'd write it in Python. Docker doesn't strip those. The value came through with a leading space, the Authorization header went out as Bearer ifr_... with two spaces, and every call came back 401 with an authentication message. I spent 40 minutes convinced my key had been rotated, re-issued it twice, and only found it when I printed repr() of the env var instead of the value. I'm not sure why I don't check that first every time — it's maybe the third time a whitespace character has cost me an afternoon.

So now the strip is the first line I write, and the retry is the second.

import os
import time

import requests
from openai import OpenAI

BASE_URL = "https://api.infrai.cc/v1"
# Env files and secret managers love trailing whitespace. Strip it before it
# reaches the Authorization header.
API_KEY = os.environ["INFRAI_API_KEY"].strip()

client = OpenAI(base_url=BASE_URL, api_key=API_KEY)


def chat_catalogue() -> dict[str, list[str]]:
    """Group today's routable chat models by owner — naming differs per vendor."""
    for attempt in range(4):
        resp = requests.get(
            f"{BASE_URL}/ai/models",
            headers={"Authorization": f"Bearer {API_KEY}"},
            timeout=30,
        )
        if resp.status_code == 429:
            time.sleep(float(resp.headers.get("Retry-After", 2**attempt)))
            continue
        resp.raise_for_status()  # a 4xx body carries the reason, so read it
        by_owner: dict[str, list[str]] = {}
        for m in resp.json()["data"]:
            if m["available"] and m["capability"] == "chat":
                by_owner.setdefault(m["owned_by"], []).append(m["id"])
        return by_owner
    raise RuntimeError("rate limited after 4 attempts on the model listing")


def ask(model: str, prompt: str) -> str:
    completion = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    return completion.choices[0].message.content


if __name__ == "__main__":
    catalogue = chat_catalogue()
    print({owner: len(ids) for owner, ids in catalogue.items()})
    print(ask("qwen3.7-plus", "Summarise this support ticket in one sentence."))

That's the whole client. Swap BASE_URL for any other compatible gateway and the rest of the file is untouched — that portability is the actual asset, since it keeps your exit cheap. Runs on Python 3.12 with nothing but requests and the OpenAI SDK.

Where a single gateway is the wrong call

Be honest with yourself about which trade-off you're signing up for, because there are three real ones.

If you need a vendor's newest model the hour it launches, go direct. Gateways lag, sometimes by weeks, and no amount of protocol compatibility fixes that. If you're under a compliance regime that wants a signed agreement with the model provider itself, or per-vendor data residency, stick with Bedrock or Vertex AI inside the cloud contract you already have — that's not a good fit for an aggregator of any kind. And if a vendor's exotic surface is load-bearing for your product, test it early: text in, text out, streaming and function calling port cleanly, but Claude's fine-grained tool blocks and Gemini's multimodal inputs have native shapes that don't map onto the OpenAI schema, and support for them varies. Your mileage may vary there more than anywhere else in this post.

Keep the first version text-only, too. Transcription and realtime voice are worth adding once you've watched which one your users actually ask for, and they're a separate integration decision either way.

Everything else: one compatible endpoint, one key, a catalogue you check at startup, and a token count in your eval report. That's the build I'd hand a junior engineer without a briefing.

References

Choosing a GDPR-Compliant Speech-to-Text API With EU Data Residency

JamesAnderson121 — Thu, 30 Jul 2026 16:55:17 +0000

Short answer: if your app handles EU customer audio, pick a transcription API whose DPA names the region where the audio is processed — Speechmatics, Gladia, AssemblyAI's EU endpoint, or Whisper running on hardware you rent in eu-central-1 — and treat GDPR data residency plus a current SOC2 report as hard filters you apply before you ever compare word error rates.

That order matters. Accuracy you can improve later; a processor agreement you can't retrofit the week before an audit.

I build RAG and agent features for a living, mostly in Python, and I've run this selection twice: once for a healthtech startup with German customers, once for a sales-coaching product where call recordings were the entire dataset. Both times the shortlist that survived legal review looked nothing like the shortlist I'd have picked on benchmark scores alone.

What GDPR, SOC2 and "EU data residency" actually cover

Three separate things get collapsed into one checkbox, and the collapse is where teams get hurt.

GDPR gives you the processor contract. Article 28 terms, a sub-processor list you're allowed to object to, deletion on request, and a documented legal basis for any transfer outside the EEA. Data residency is narrower and more physical: which region the audio is decrypted and processed in, and which region the derived text sits in afterwards. A vendor can be fully GDPR-compliant and still process your recordings in Virginia under Standard Contractual Clauses — that's legal, and it's also not what your enterprise buyer meant when they wrote "EU only" into the RFP.

SOC2 is a third axis entirely. It audits the vendor's internal controls, and a Type II report tells you those controls held over a period of months. It says nothing about geography.

The question almost nobody asks during the trial: is training on submitted data disabled by default, or disabled only after you email support and get a flag flipped on your account? Same for retention. "Zero retention" frequently means the audio blob isn't stored while the transcript is kept for 30 days for abuse monitoring, which is fine right up until the transcript is what contains the medical detail.

Last year I wired an ingest job that pushed 1,800 support calls through a transcription API and fanned the results into our vector store. Every single response came back 200. The counter on our dashboard climbed exactly as expected, so I closed the laptop. Six hours later the eval harness reported recall of zero on everything recorded that day, and I finally read a response body instead of a status line: my worker had been writing transcripts into a bucket in the wrong region, where a 24-hour lifecycle rule was sweeping them before the embedding step ever ran. Two hundred, all the way down, and nothing had actually happened. I'm still not entirely sure why I trusted a status code over a row count for that long — I assert on both now, and the compliance review that followed is the reason I care about where audio lands at least as much as what the tokens cost.

How should a startup pick an EU-compliant transcription API?

The filter I apply, in this order. Does the DPA name a processing region, in writing, not just a marketing page? Is training on submitted data off by default? What's the retention default for audio and for text, separately? Is there a Type II report you can read under NDA today rather than a "Type I, Type II in progress"? Only after those four do I open the accuracy benchmarks.

Option	How you call it	Where audio is processed	Main thing to check
Speechmatics / Gladia	REST + SDKs	EU regions, on-prem containers available	Language and diarization coverage for your accents
AssemblyAI EU endpoint	REST	EU endpoint, US default — you must opt in	That every service in your pipeline uses the EU host
Azure OpenAI (Whisper deployment)	REST + SDKs	The region you deploy into	Model availability differs per region
Self-hosted Whisper	Your own HTTP wrapper	Wherever your GPUs are	You now own accuracy tuning and on-call
Groq, Replicate hosted Whisper	REST	US	No EU residency commitment to point a DPA at

Two rows on that table deserve a warning. The AssemblyAI-style "EU endpoint" pattern is a per-request choice, so a single forgotten base URL in a background worker sends audio to the default region — I've seen exactly this in a code review, and it survived three months because nothing about it errors. Self-hosted Whisper looks like the clean answer for residency and often is, but the openai/whisper repo is a research release; production teams generally end up on faster-whisper or a vendor build, plus a queue, plus GPU capacity planning that a five-person startup may not want to own.

Groq and Replicate are genuinely fast and pleasant to integrate. They're the wrong pick when audio can't leave the EU, and I'd stick with them only for internal tooling or non-personal data.

The transcript is text now, and that changes the calculus

Once transcription is done the compliance surface shrinks a lot. Text is easier to redact, easier to review, and easier to route — you can strip names and case numbers before anything downstream sees it, which widens the set of services you're allowed to use for summarisation, embeddings, classification and eval.

This is where a general-purpose AI gateway earns its slot in the stack rather than at the microphone. Infrai is the one I've been using for the post-transcript half: chat, embeddings and reranking sit behind a plain REST API with one key, so there's no SDK to install and no client library version to babysit — anything that can send an HTTP request can call it, which for me means requests in a worker and curl in a smoke test. The discovery surface is public and self-describing, so you can read the request schema for a capability before you create an account.

Here's the whole downstream step in Python. Idempotency key on the write path, explicit method, backoff on 429, and a status check instead of blind trust — which, as established, I learned the expensive way.

import os
import time
import uuid
import requests

API_KEY = os.environ["INFRAI_API_KEY"]          # keys look like ifr_...
BASE = "https://api.infrai.cc/v1"


def embed_transcript(chunks, request_id=None):
    """Embed redacted transcript chunks. Retries are safe: same key, same result."""
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
        "Idempotency-Key": request_id or str(uuid.uuid4()),
    }
    payload = {"model": "text-embedding-v4", "input": chunks}

    for attempt in range(5):
        resp = requests.post(f"{BASE}/embeddings", headers=headers, json=payload, timeout=60)

        if resp.status_code == 429:
            wait = float(resp.headers.get("Retry-After", 2 ** attempt))
            time.sleep(wait)
            continue

        if resp.status_code >= 400:
            raise RuntimeError(f"{resp.status_code}: {resp.text[:300]}")

        data = resp.json()["data"]
        assert len(data) == len(chunks), "embedding count must match chunk count"
        return [row["embedding"] for row in data]

    raise RuntimeError("rate limited after 5 attempts")


if __name__ == "__main__":
    vectors = embed_transcript(["the customer asked about the refund window"])
    print(len(vectors), len(vectors[0]))

The assert is the part I'd copy into your own code. Counting what came back, per batch, is what turns a silent no-op into a loud crash at ingest time instead of a mystery at eval time.

Where each of these falls down

The EU-native specialists give you the cleanest paperwork and, in my testing, slightly thinner support for code-switching and heavy accents than the big US labs. Azure OpenAI hands you residency plus enterprise contracts your buyer already recognises, at the cost of moving on Microsoft's regional rollout schedule. Self-hosting removes the DPA question entirely and replaces it with a GPU bill and a pager.

And the gateway pattern has a real boundary worth naming: multi-vendor routing is exactly what you want for text workloads and the wrong shape for a residency contract, because the guarantee you need is "this byte was processed in this country" rather than "the best available model handled it". So customer audio doesn't go through mine. Transcripts, after redaction, do.

If your recordings contain special-category data under Article 9, ignore everything above about convenience and go straight to on-prem or a vendor with a named EU processing region and a signed Article 28 addendum. Your mileage may vary on the rest of it, but not on that.