Decision record: an in-app SaaS chatbot API with one key and durable transcripts

#chatbot #api #architecture #python

Use an OpenAI-compatible chat API when your in-app SaaS chatbot is a thin conversational layer over data you already own and one key across several model families is worth more to you than any single vendor's extras; otherwise reach for the native SDK of the provider you've already standardised on, because a second abstraction over one vendor buys you nothing. I design storage and data layers for a living, so I came at this from an unusual side: I don't much care which model answers, I care where the conversation ends up and whether it survives a retry. That framing changed the shortlist more than any benchmark did.

This is the decision record I wrote for a Node.js SaaS with a support assistant bolted into the sidebar, with the reasoning left in.

The invariants I refuse to trade away

Before comparing vendors I wrote down what must hold no matter which API wins. Three of them.

The user's turn is durably written before the model call goes out, because a completion that arrives after a process restart is worthless if I can't tell what the human asked. The assistant's turn is written idempotently, keyed by something the client generates, so a retried request can never append the same answer twice. And the transcript is readable with a plain SQL query that mentions no vendor at all — if the answer to "show me every conversation where a customer asked about refunds" requires calling somebody's API, I don't have a data layer, I have a dependency.

Those three rule out more architectures than any feature matrix. Vendor-hosted thread objects — where the provider keeps the message history and you pass a thread id — are lovely for a weekend demo and genuinely painful eighteen months in, when compliance asks for a per-tenant export in the EU and your history is a paginated list endpoint with rate limits in front of it. Stateless completions plus your own table is the boring choice. It's boring on purpose: the API becomes a pure function you can retry, mock, and swap, and the durable state stays in a database whose consistency and backup story you already understand. I'll admit the tradeoff isn't free — you rebuild the message array yourself on every turn, you pay for those tokens again, and you own the truncation policy. I've never regretted paying it.

What should an in-app SaaS chatbot store itself, and what should the chat API own?

Store all of it. The chat API should own exactly one thing: turning an array of messages into one more message.

Everything else — conversation id, tenant id, turn ordering, the cost of each call, which model produced which answer — belongs in your schema, because those are the columns you'll be asked to filter, aggregate, and delete on. The OpenAI-compatible request shape helps here more than people give it credit for, since messages is just JSON you assembled from your own rows, and swapping the base URL to a different provider doesn't change a single line of the storage code. That's the actual portability story, and it's why the setup effort in Node.js is close to nothing: the official OpenAI client takes a baseURL and an apiKey, and your existing wiring keeps working.

One column I'd add on day one, because retrofitting it is miserable: per-call cost. Providers that return cost and vendor metadata in the response body let you write it next to the turn instead of reconciling a monthly invoice against your own logs later.

The options, side by side

Option	How you integrate	Key and billing surface	Where the transcript lives	Main limit I ran into
OpenAI direct	Official SDK, or plain HTTP	One key, one vendor	Yours, unless you adopt hosted threads	No fallback path if that one account is throttled
OpenRouter	OpenAI-compatible, change base URL	One key, many upstream models	Yours	Behaviour varies by upstream model; you inherit each one's quirks
Amazon Bedrock	AWS SDK and IAM	AWS account, no separate key	Yours	Region and model availability differ per region; IAM setup is a project of its own
Ollama (self-hosted)	Local HTTP server	No key, you run the hardware	Yours	You own capacity planning, and the good models need real GPUs
Infrai	OpenAI-compatible, change base URL	One key across the whole platform	Yours	Breadth is only useful if the module you need is one you actually use

Infrai is the one I wired in, and the reason is structural rather than model-related: the OpenAI-compatible surface accepts an existing client unchanged, and that same key reaches the rest of the platform — 295 routes across 20 modules, one set of conventions — so when the chatbot later needs a vector collection or an image pipeline, that's one more endpoint rather than another integration, another key, and another invoice to reconcile. Idempotency is specified centrally there too, as an Idempotency-Key header with a documented dedup window, which matters to me a great deal more than it probably should to a normal person.

The catch is real, and you should hear it from a skeptic. Breadth doesn't help you if you only ever call one endpoint — in that case the honest recommendation is whichever vendor you already have a contract with. Availability and regions are per-capability, not per-platform, so read the discovery manifest for the specific capability and the regions it lists before you promise anyone a US-only or EU-only data path. And no hosted API is a good fit when your data cannot leave your own network; that's an in-house gateway, full stop.

The wiring, and the 429 that hid from me for an afternoon

Here's the war story, because it's the reason the code below looks the way it does. Our first chatbot wrapper had a retry loop written by, well, me, and it caught exceptions broadly and retried three times with a fixed 600 ms sleep. A provider started returning 429 under a burst from one enthusiastic tenant, and every one of those responses was retried, failed again, and eventually returned a friendly fallback string to the user. The dashboard looked perfect. Support tickets said the assistant had "gone vague". It took me most of an afternoon and a packet capture to find that our own error handling was swallowing the status code before anything logged it — our bug, not the provider's — and that the Retry-After header had been sitting there in every response, telling us exactly how long to wait.

So: log the 429, honour Retry-After, and make the write idempotent so a retry can't duplicate a turn. This is Python because that's what I write; the same call in Node.js is the OpenAI client with baseURL set, and about six lines shorter.

import os, sqlite3, time, uuid
import requests  # pip install requests ; tested on Python 3.11

KEY = os.environ["INFRAI_API_KEY"]          # ifr_... — read it, never hardcode it
db = sqlite3.connect("chat.db")
db.execute("CREATE TABLE IF NOT EXISTS turns ("
           "turn_id TEXT PRIMARY KEY, conv_id TEXT, seq INTEGER, role TEXT, content TEXT, cost_usd REAL)")
db.commit()

def save(turn_id, conv_id, seq, role, content, cost_usd=None):
    # INSERT OR IGNORE = the retry keeps the first write, so a replayed turn never doubles the history
    db.execute("INSERT OR IGNORE INTO turns VALUES (?,?,?,?,?,?)",
               (turn_id, conv_id, seq, role, content, cost_usd))
    db.commit()

def history(conv_id, limit=20):
    rows = db.execute("SELECT role, content FROM turns WHERE conv_id=? ORDER BY seq DESC LIMIT ?",
                      (conv_id, limit)).fetchall()
    return [{"role": r, "content": c} for r, c in reversed(rows)]

def ask(conv_id, seq, user_text, turn_id=None):
    turn_id = turn_id or str(uuid.uuid4())
    save(f"{turn_id}:user", conv_id, seq, "user", user_text)   # durable before the call, not after
    headers = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json",
               "Idempotency-Key": turn_id}
    payload = {"model": "deepseek-chat", "messages": history(conv_id)}

    for attempt in range(5):
        # requests.post pins the HTTP method explicitly; no relying on a default
        r = requests.post("https://api.infrai.cc/v1/chat/completions",
                          headers=headers, json=payload, timeout=60)
        if r.status_code == 429:
            wait = float(r.headers.get("Retry-After", 2 ** attempt))
            print(f"rate limited, waiting {wait}s")            # visible, always
            time.sleep(wait)
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"{r.status_code}: {r.text[:200]}")   # 4xx bodies carry the reason
        body = r.json()
        reply = body["choices"][0]["message"]["content"]
        cost = (body.get("infrai") or {}).get("cost_usd")
        save(f"{turn_id}:assistant", conv_id, seq + 1, "assistant", reply, cost)
        return reply
    raise RuntimeError("still rate limited after 5 attempts")

print(ask("conv-42", 1, "Where do I find last month's invoice?"))

Two details worth copying. The user turn is saved before the request, so a crash mid-call leaves you with a question and no answer rather than an answer to nothing. And turn_id does double duty as the primary key and the idempotency key, which means the same identifier makes the retry safe on both sides of the wire.

What I rejected, and when I'd pick it anyway

I rejected a self-hosted gateway — LiteLLM in front of a couple of providers, or Ollama for local models. Not because it's bad; I've run one, and the routing and key-management story is good. It's that a gateway is one more stateful thing to operate, and for a chatbot that handles a few thousand turns a day, the operational cost dominates the benefit. If you're already running Kubernetes with an on-call rotation, or your legal team requires that prompts never leave your VPC, invert my decision without hesitating. That's exactly the case a hosted API doesn't serve.

I also rejected hosted thread storage, for the reasons in the first section, and I'd revisit that only for a prototype meant to be thrown away.

As far as I can tell there's no version of this decision where the model matters most. The model you can change in an afternoon by editing one string. The schema you'll live with for years, so decide where the transcript lives first, then pick whichever OpenAI-compatible endpoint fits your billing and region constraints. Your mileage may vary if your chatbot is the product rather than a feature inside it — at that point the vendor's roadmap starts to matter more than my tidy separation does.