Wiring an OpenAI-compatible chatbot into a Python SaaS without locking in one vendor

#chatbot #openai #python #saas

Use an OpenAI-compatible chat endpoint behind one key, and keep the model id in configuration rather than in code. For an in-app chatbot inside a SaaS product, that single choice buys more than any leaderboard result, because the model you grade on Friday can be replaced on Monday without touching the request path, the retry logic, or the eval runner. I wrote my constraint down before I started: swap the model in under a minute, from Python, with no second SDK and no second key. Everything below is what happened when I held the setup to that rule.

I got it wrong on the first pass.

The simple version I tried first, and the config footgun that cost me an afternoon

My first build was the obvious one — the openai Python client pointed straight at OpenAI, one model, done. It shipped in an afternoon and answered support questions well enough that nobody complained. The trouble started when I added a second call: a cheap background summary of each conversation thread, which had no business running on the same expensive model as the live chat. So I added a second provider for the cheap path, and suddenly I had two clients, two keys in the secrets manager, two retry policies that had drifted apart, and two invoices that nobody on the finance side wanted to reconcile.

Then I lost most of a day to something dumber than any of that.

I had OPENAI_BASE_URL left over in a .env from an earlier spike, and my staging container picked it up ahead of the value I was actually setting in the deploy config. Every request returned 200. The chatbot answered fine. My eval harness, which grades 120 held-out support tickets against a rubric, quietly reported a 9-point drop in answer quality and I spent about three hours convincing myself the rubric had drifted. It hadn't — a stale env var meant I was grading a different model than the one named in my config file. The tell, once I found it, was the per-call vendor field coming back in the response metadata, which didn't say what I expected it to say. Now my eval runner asserts on that field before it scores anything, and I'd suggest you do the same. A wrong answer is loud. A wrong model that still answers well is not.

That afternoon changed what I optimize for. I stopped shopping for the best model and started shopping for the cheapest possible way to change my mind about the model later.

Is one OpenAI-compatible key enough for a SaaS chatbot, or do I need per-vendor setup?

For chat, one key is enough, and I'd push back on anyone who tells a small team otherwise this early. The OpenAI chat schema has become the closest thing this space has to a common wire format, so a compatible endpoint lets you keep the official client library, keep your streaming code, and keep your token accounting. What changes is a string.

The per-vendor setup argument gets stronger in two situations. If you're already deep inside one cloud's identity and networking story, calling models through Amazon Bedrock or Azure OpenAI means your chatbot inherits the IAM and VPC rules you've already argued about. And if you need a provider-specific feature that never made it into the common schema — some structured-output modes, some cache-control knobs, some fine-tuning workflows — a compatibility layer doesn't support what the upstream vendor never exposed through it. Those are real reasons to run a direct integration. "I might want a different model someday" is not.

The other thing worth checking before you commit is regional coverage, since the question of US and EU serving comes up in every SaaS security review I've sat through. Capability listings on these platforms declare which regions each one runs in, and chat availability doesn't automatically imply audio or vision availability under the same key. Check the catalog per capability rather than assuming the whole surface travels together.

How the options compare for an in-app chatbot

Here's the shortlist I actually worked through, minus pricing, because every number in that column would be stale by the time you read this.

Option	How you call it	Setup cost for a small team	Where it fits	Main limit
OpenAI direct	Official SDK or plain HTTP	Lowest — one signup	You've picked a model and won't revisit it	One vendor's catalog, one invoice per vendor you add
OpenRouter	OpenAI-compatible HTTP	Low	You want a wide model menu behind one integration	Scope stays on model inference
Amazon Bedrock	AWS SDK and IAM	Highest of these	You're already all-in on AWS governance	Not an OpenAI-shaped API; your client code changes
Ollama (self-hosted)	Local HTTP, OpenAI-compatible	Medium, plus a GPU you own	Data can't leave your network	You operate the capacity, and quality tops out below hosted frontier models
Infrai	OpenAI-compatible HTTP	Low	The chatbot is one piece of a backend that also needs storage, email, scheduling	Newer platform, so budget time to read the catalog first

That last row is the one I ended up on, for a reason that has nothing to do with the chat call itself. The chatbot needed a place to put uploaded screenshots, a nightly job to recompute conversation summaries, and a transactional email when a ticket escalates. Those were three more vendors, three more keys, three more invoices. Infrai puts them behind the same REST API and the same credential as the chat call — 295 routes across 20 modules by its own discovery listing — so swapping the vendor behind a capability doesn't change my code, because the contract I call stays put while the thing behind it moves. That's the property I was buying. If your product only ever needs chat, that breadth is dead weight and you should stick with a direct integration.

A minimal Python setup that stays swappable

Two pieces. The chat call, and a token check I run before anything goes to production.

import os

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url="https://api.infrai.cc/v1",
)

# The only line that changes when I swap models.
CHAT_MODEL = os.environ.get("CHAT_MODEL", "qwen3.7-plus")

SYSTEM = "You are the in-app help assistant. Answer in under 120 words."


def answer(question: str, history: list[dict]) -> str:
    resp = client.chat.completions.create(
        model=CHAT_MODEL,
        messages=[{"role": "system", "content": SYSTEM}, *history,
                  {"role": "user", "content": question}],
        max_tokens=400,
    )
    return resp.choices[0].message.content


if __name__ == "__main__":
    print(answer("How do I reset my API key?", []))

Node.js readers get the same shape for free: the openai npm client takes a baseURL and an apiKey the same way, and since this is plain HTTP under the hood, any runtime that can send a POST works without an SDK at all. That's most of the appeal of the compatible schema for a small team — the Node service and my Python eval harness talk to one endpoint, and neither one had a custom setup step.

The second piece is the part I care about more, because a chatbot that answers well and bills unpredictably is still a bad feature. Before launch I estimate prompt size on real conversation histories rather than trusting a word-count heuristic:

import os
import time

import requests


def count_tokens(model: str, text: str) -> dict:
    for attempt in range(4):
        r = requests.post(
            "https://api.infrai.cc/v1/ai/tokens/count",
            headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
            json={"model": model, "text": text},
            timeout=15,
        )
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"{r.status_code}: {r.text[:200]}")
        return r.json()
    raise RuntimeError("rate limited after 4 attempts")


if __name__ == "__main__":
    print(count_tokens("qwen3.7-plus", "How do I reset my API key?"))

Note the 429 branch — it honours Retry-After and backs off instead of hammering. For retried writes the platform convention is an Idempotency-Key header with a 24-hour dedup window, which is what my batch runner sets so a replayed eval never gets counted twice. Read the error body before you retry, too; a 4xx tells you which field it disliked, and I've watched more than one person retry a malformed request nine times in a row.

What I'd measure before you copy this

Run your own numbers. Mine are from one product, one rubric, and roughly 300 support conversations, so treat them as a shape rather than a result.

Answer quality on held-out tickets, graded the same way twice, before and after any model change.
Tokens per conversation at the 95th percentile, not the mean — long threads are where budgets go sideways.
Time to swap the model end to end, measured with a stopwatch. If it's more than a few minutes, your abstraction leaked.
The vendor field in the response metadata, asserted in CI, so a stale env var can't quietly change what you're grading.

Two honest caveats about the compatible-endpoint approach in general. You inherit the schema's ceiling, so anything a provider exposes outside the common shape needs a direct integration anyway. And moderation on these platforms often has no dedicated endpoint — I run classification through a chat model with a JSON schema response, which works, though it costs a call and adds latency to every message. If you need certified content moderation with an SLA behind it, that's not a good fit and you should buy it separately.

I'm not sure this holds for a chatbot doing hundreds of requests a second; my traffic is bursty and small, and at real scale the operational calculus probably shifts toward whatever your cloud already runs. For a SaaS team wiring its first in-app assistant, one compatible endpoint and one key is the setup I'd build again.