We almost put the OpenAI SDK inside the agent runtime.
It looked like the short path. The runtime already runs a turn: load the published agent, load the chat, maybe search knowledge, maybe load memory, call a model, save the assistant message. Installing @anthropic-ai/sdk or the OpenAI client right there would have made the first demo work in an afternoon.
We did not do that. This post is about why, what we built instead, and how that choice helps a multi-tenant agent platform when the traffic is no longer a laptop.
The audience I have in mind is a senior backend or platform engineer who has already shipped one LLM feature and is now asking where the keys, the retries, and the bill should live.
What we were building
QuickStart AI is a production-oriented platform. Organizations run workspaces. A workspace is the tenant. Every customer row is scoped by workspace_id.
Inside a workspace we already had the product pieces a chat agent needs:
- Agents. Instructions and model settings. Publish is immutable. Rollback points at an older published version.
- Conversations. Ordered messages in Postgres. Writes use an idempotency key so a client retry does not append the same turn twice.
- Knowledge. Ingest plus retrieval. RAG goes through OpenSearch.
- Memory. Facts about an end user, stored in Postgres.
- Runtime. The service that executes one turn.
The public door is an API gateway on port 3000. It checks the JWT, checks workspace membership, and applies a Redis rate limit. It then proxies to internal services. A browser and a dashboard never call an LLM vendor or a scrape vendor themselves.
The stack around this is a Node.js and TypeScript monorepo, Postgres, Redis (Valkey in some environments), OpenSearch, and SQS-shaped queues (ElasticMQ locally). Internal services listen on their own ports.
The numbers we wrote down, so the resilience work had a target, are:
- 10,000+ organizations, 50,000+ users, 100,000+ agents
- About 10,000 HTTP requests per second sustained, and 20,000 at peak, on the gateway and API. That is not 10,000 LLM calls per second.
- Millions of messages per day, and 10,000+ concurrent chats
- Later, a separate voice service for about 5,000 concurrent inbound calls and 5,000 outbound calls. Voice is out of scope for this post.
Identity, workspace RBAC, the gateway, agents, conversations, a Redis membership cache, knowledge, memory, and the runtime turn path were already shipped. The open question was outbound: how every service talks to paid vendors without copying keys and failure policy into each deployable.
We wrote the decision down in ADR 0010 for the model gateway and ADR 0011 for web fetch.
What we needed the design to do
A turn is not only "call the model." Some paths also fetch a page or search the web, so knowledge ingest and future tools can acquire content. Those are different vendors, different bills, and different failure shapes. The requirement was the same for both:
- Vendor API keys live in as few deployables as possible.
- Timeout, retry, and "do not double-charge" rules are written once.
- A slow vendor cannot pin every replica until the process is useless.
- We can prove, later, which workspace spent tokens or pages.
- Product services keep a stable internal contract when we swap a provider.
If any one of those is missing, the platform still demos. It does not stay cheap or operable at the scale above.
Three ways to call a model
We treated this as a real fork, not a style preference. The same fork applies to paid web APIs.
Option A — install the vendor SDK in every service
Runtime imports the SDK. A tools service imports it. An eval worker imports it. Each process holds a key and picks its own timeout.
| Problem | Why it hurts at our scale |
|---|---|
| Secrets surface area | The API key is in N deployables. Rotation and audit are N operations. |
| Duplicated policy | Each service chooses its own timeout, retry count, and logs. Those choices drift. |
| Streaming and retry | After SSE tokens are already on the wire, a blind retry can duplicate what the user sees, or charge twice. |
| Thundering herd | Many replicas times many retries becomes vendor 429s and a runaway bill. |
| Compliance story | It is hard to say "all model traffic is logged and capped in one place." |
This option is fine for one script. It is a poor fit once more than one service can spend vendor money.
Option B — a shared npm package
A package such as @quickstart/llm-client puts the HTTP or SDK call in one module. Callers stay DRY. We still rejected it as the final shape.
| Problem | Why a package was not enough |
|---|---|
| Policy still runs in every process | A global in-flight limit needs shared state, which for us means Redis. You end up rebuilding half a service inside a library. |
| Circuit state per replica | If the package does not talk to Redis, each pod has its own breaker. Protection is uneven. |
| Deploy coupling | A new retry rule or a new provider means redeploying every consumer. |
| Operational ownership | On-call still asks which service version ran the bad retry loop. |
| Billing hooks | One caller forgets to pass workspace_id or to log tokens. The ledger is already wrong. |
A shared package fixes copy-paste. It does not give you one process that owns the vendor.
Option C — dedicated internal services
This is what we shipped.
- model-gateway, port 3005. Every chat completion and every stream to an LLM vendor goes here.
- web-fetch-service, port 3007. Scrape, search, and async crawl go here. Direct HTTP and Olostep sit behind it as providers.
Product services speak internal HTTP. They do not import vendor SDKs.
| Criterion | SDK in every service | Shared npm package | Dedicated model-gateway + web-fetch |
|---|---|---|---|
| Secret sprawl | High | Medium | Low |
| Retry and stream policy stays the same | Low | Medium | High |
| Global in-flight cap and budget | Hard | Hard unless every caller talks to Redis | Redis inside the service |
| Deploy when vendor policy changes | Many services | Many services | One service |
| Scale chat and LLM concurrency apart | Coupled | Coupled | Decoupled |
| Test vendor failures | Per service | Package tests | Fake adapters plus orchestrator tests |
Why dedicated services fit this product
We picked option C for five reasons that map onto the platform we already had.
One place for credentials and policy. Timeouts, retries, circuits, caps, and usage logs sit next to the adapter that speaks to the vendor. Key rotation is one secret on one deployable.
A stable contract for product services. Runtime, future tools, and ingest workers call internal HTTP. Swapping a model vendor is PROVIDER_PROFILES_JSON on the gateway, not a runtime release.
Separate scale knobs. More runtime replicas means more concurrent chats. More model-gateway replicas means more LLM concurrency. More web-fetch replicas and workers means more page acquisition. Keys and policy stay in the two outbound services.
Cost checked where the unit is known. Web fetch denies a workspace that is over its daily unit budget in Redis, before the vendor call. Model usage is logged in one shape: tokens, latency, workspace_id. There is no billing ledger on the request path yet. The facts the ledger will need are already there.
Vendor keys stay off the public gateway. Staff and end users present a JWT to port 3000. model-gateway and web-fetch trust a separate internal bearer token. In production those ports sit on a private network, with Kubernetes network policy planned as roadmap item P8.
Knowledge and memory stay separate services. They have their own failure rules. This post is about the vendor outbound pattern: model calls and web acquisition. RAG and memory are neighbors, not the same control plane.
How a request moves
A user message hits the gateway. The gateway checks the JWT, checks that the person belongs to the workspace, and applies the public rate limit. It does not call OpenAI.
The runtime on port 3006 runs the turn. When it needs a model, it sends POST /internal/complete (or /internal/stream) to model-gateway with Authorization: Bearer <MODEL_GATEWAY_INTERNAL_TOKEN>. The body carries workspace_id, request_id, a model profile, and the messages. The token is an internal secret, at least 32 characters. It is not the end user's JWT.
model-gateway picks an adapter, runs policy, and only then calls OpenAI, Anthropic, OpenRouter, or another configured provider.
Web acquisition is the parallel path. Knowledge ingest and future tools call POST /internal/scrape, /internal/search, or /internal/crawl on web-fetch. That service calls direct HTTP or Olostep.
[ Browser / API client ]
|
v
+---------------------+
| API Gateway :3000 |
| JWT, membership, |
| Redis rate limit |
+----------+----------+
|
+---------------------+---------------------+
| | |
v v v
+---------------+ +----------------+ +------------------+
| runtime :3006 | | knowledge :3009| | memory :3010 |
| (turn loop) | | (RAG API) | | (facts API) |
+-------+-------+ +----------------+ +------------------+
|
| POST /internal/complete
v
+---------------------+
| model-gateway :3005 |
| adapters + policy |
+----------+----------+
|
v
[ OpenAI / Anthropic / OpenRouter / ... ]
knowledge ingest, tools --> POST /internal/scrape|search|crawl
|
v
+---------------------+
| web-fetch :3007 |
| adapters + policy |
+----------+----------+
|
v
[ direct HTTP / Olostep / ... ]
Public clients stop at port 3000. Product services do not hold vendor keys. Only model-gateway and web-fetch talk to paid APIs.
Inside model-gateway the split is simple, and it is the split we repeat in web-fetch:
Runtime --> Orchestrator (timeout, retry, breaker, cap, logs)
|
v
Adapter (one vendor's HTTP or SSE shape)
|
v
Vendor
Adapters talk to vendors. Orchestrators talk to failure. That sentence is the whole design.
Model gateway: one door for models
model-gateway is the control plane for LLM calls. Runtime does not know whether the profile is OpenRouter or Anthropic.
The internal routes we use are:
-
POST /internal/completefor a full response -
POST /internal/streamfor SSE:tokenevents, thendone -
POST /internal/decidefor an optional routing decision
Auth on all of them is the internal bearer token.
Adapters implement a narrow interface, ChatProvider, with complete() and stream(). The drivers we have are fake (for CI), openai_compatible, openrouter, and anthropic. A registry, built by createProviderRegistry, maps a profile id from config to a driver instance. A profile is a named provider config, for example openrouter-prod. The driver is the implementation type.
Adapters throw ProviderError with retryable: true or false, and an optional HTTP status. They map the vendor's HTTP and SSE. They do not decide "retry three times." That split matters. The adapter knows that 429 and 400 are different. The orchestrator knows we only allow one retry, and that a stream which already sent tokens must not be retried.
The orchestrator (CompletionOrchestrator) wraps every call. Timeouts, the retry loop, the circuit breaker, the Redis in-flight cap, fallback, and model.usage logs live here.
What this gives the product:
- We change vendor by config, without a runtime deploy.
- A request may name
fallback_model. If the primary profile fails and the stream has not started, we try that fallback once. - Every successful or attempted call can emit a structured
model.usagelog: tokens, latency,workspace_id. That is the input for a later billing feature (Feature 26). The ledger itself is not on this path. - Workspace bring-your-own-key is reserved for the secrets work (Feature 16). If a request says the credential source is the workspace, the gateway returns 400 today. We would rather fail closed than store customer keys halfway.
Web fetch: the same shape, a different bill
We did not put scrape inside model-gateway. The vendors differ. The billing unit differs: pages and search queries, not tokens. The policy differs: try cheap direct HTTP first, and use Olostep when we need JavaScript rendering or when direct fetch is not enough. Crawl is also asynchronous. It needs a Postgres job row and a worker. Those are bad reasons to share one orchestrator with chat completions.
web-fetch is the control plane for acquisition.
-
POST /internal/scrapeis synchronous -
POST /internal/searchis synchronous, with a cap onmax_results -
POST /internal/crawlreturns a job id (HTTP 202 style) -
GET /internal/jobs/:jobIdreturns status
Auth is WEB_FETCH_INTERNAL_TOKEN. A legacy alias is still accepted.
The provider registry has direct_http, olostep, and fake. Before the orchestrator runs, decideAcquisitionProviders picks an ordered list of profile ids. For scrape, direct HTTP profiles come first, then Olostep. Search uses Olostep or the fake driver. render_js goes down the Olostep path.
AcquisitionOrchestrator then applies caps, the workspace budget, the breaker, the profile chain, and retries.
What this gives the product:
- Knowledge ingest (Feature 11) and future tools call one service. We do not put the Olostep SDK in five places.
- Each workspace has a daily unit budget in Redis. Over budget means 429, and the vendor is not called.
- Usage facts record
action,provider,units, andestimated_cost_micros. - Crawl inserts a row as
queued. A worker claims it withFOR UPDATE SKIP LOCKED, so two replicas can poll without taking the same job.
URL ingest from the knowledge service into web-fetch is still a follow-up. The service exists. That wire is not the whole story yet. I say that again in the limitations, because it is easy to over-claim.
How the resilience works, and why we did not take the simpler version
Each pattern below is something we run inside these two services. I include the simpler idea we rejected, because that is the part that is easy to skip when you are only trying to make the call succeed.
Adapters plus a registry
Vendor code implements a small interface. Config names a profile. The registry returns the driver for that profile id.
New vendor means a new adapter file and a config row. Runtime does not fork. In CI, the fake driver can simulate a 429, a hang, a failure before any token, and a failure after tokens, with no network.
The SDK already speaks the vendor's shape. Our product speaks one shape: a completion body, or a scrape body. The adapter is the anti-corruption layer between those two shapes. If the adapter also owns retry policy, every new vendor copies a slightly different loop. We kept that loop out.
One orchestrator per domain
CompletionOrchestrator and AcquisitionOrchestrator wrap the outbound call. Callers stay thin. Runtime does not parse OpenRouter error bodies.
LLM streaming rules are not the same as a scrape profile chain, so we did not force one giant policy class. The primitives match. The policy does not.
Without this layer, if (retry) grows inside each adapter. Six months later, two providers retry differently and nobody remembers which one double-charged a stream. On-call for "the model vendor is down" is one service, one log shape, not a search across every consumer's version.
A deadline on every vendor call
Vendor TCP can hang. At 10,000 concurrent chats, one hung connection holds work in the Node process until the event loop lags and other chats stall.
model-gateway races the provider call against a timer (UPSTREAM_TIMEOUT, about 60 seconds for a non-streaming complete). Web fetch uses a per-action timeout around 30 seconds, as we wrote in ADR 0011. On the model path a timeout is retryable, inside the one-retry rule below. Vendor SDK defaults differ, and some are long enough to exhaust the process before they fire.
At most one extra attempt, and only when the error says so
model-gateway allows a maximum of two attempts when the error is retryable: ProviderError.retryable, a timeout, or an abort. Web fetch allows up to two attempts per profile, and only when the error is retryable and no billable unit has been recorded yet.
We do not retry until success. Each retry costs money and adds to turn latency. A 429 plus a retry storm turns a vendor rate limit into an outage we caused.
The retryable flag stays on the adapter, because HTTP meaning belongs next to the vendor. The attempt budget stays on the orchestrator, because "only once, and never after tokens" is a product rule.
If the stream already started, stop
This rule is model-gateway only, and it is the one I care about most.
For stream, if any token was already sent to the client, we do not retry and we do not switch to the fallback model. The attempt loop breaks.
A retry after partial SSE duplicates content. The user sees two answers stuck together, and the vendor may bill both attempts. Every caller that owns an SDK has to remember this rule. In model-gateway it exists once.
One fallback model, not a chain of five
If the primary profile fails policy and the stream has not started, we try fallback_model once. The point is degradation: a cheaper or more available model instead of a hard error to the user. The bound is deliberate. A chain of five models looks resilient and spends like an incident.
Web fetch is different on purpose. decideAcquisitionProviders returns an ordered list of profile ids, for example direct HTTP then Olostep. The orchestrator walks that list in executeScrape until one succeeds or the list is done. Direct fetch is cheap. Olostep is for JavaScript rendering or for when the cheap path failed. That is multi-provider routing for acquisition, not a primary and secondary model id. We did not pretend those are the same feature.
A circuit breaker for a sustained outage
Retry helps a blip. A breaker helps a vendor that is down for minutes.
We share a CircuitBreaker implementation. After N failures the state is open. New calls fail fast with CIRCUIT_OPEN instead of waiting on another timeout. After a cooldown the breaker goes half-open. A success closes it. Each orchestrator keeps a breaker per profile id.
When the vendor is down, a fast 503 means replicas are not stacked on timeouts, and the vendor is not hit by a synchronized retry from every pod.
That state is inside the process today, not in Redis. Two pods can disagree for a while. Per-replica breakers still help. They are not a cluster-wide view.
A fleet cap in Redis before the vendor call
A library-level semaphore counts in-flight calls inside one process. The moment we add a second replica, the vendor sees twice the concurrency. That is the opposite of a cap.
Before model-gateway or web-fetch calls a vendor, we increment a Redis key (model-gateway:inflight or web-fetch:inflight). If the count is over the max, we return 429 with RATE_LIMITED and we do not call the vendor. The increment is released when the call finishes.
This is a bulkhead. Model slowness cannot occupy unbounded work. It is also the start of stopping one tenant from filling every vendor connection. The global cap exists. Per-workspace dollar or concurrency caps on the model path do not. Those are later runtime and Redis work.
This is also why a shared package was not enough. A fleet cap has to live in one service plus Redis.
Budget where the unit is the bill
The public gateway already rate-limits HTTP with Redis. That protects the front door. It does not know what a scrape costs.
web-fetch keeps a per-workspace daily unit budget in Redis and rejects with 429 before the vendor call. Acquisition is billed per unit. We cannot depend on every future caller remembering to check. One workspace in a crawl loop should stop when the budget is gone, not when the invoice arrives.
Reject work that cannot succeed
Some requests are wrong before any vendor should see them. Examples: bring-your-own-key with credential_source: workspace while secrets are not shipped (400), a bad URL, a bad workspace id, a missing request_id.
The rule is simple. Do not pay a vendor for a request we already know will fail. Callers also get a clear 4xx instead of a 5xx that looks like an outage.
One error language
Vendors return a mess of bodies and status codes. We map that to platform codes: UPSTREAM_UNAVAILABLE, UPSTREAM_TIMEOUT, RATE_LIMITED, VALIDATION_FAILED, UNAUTHORIZED. Runtime and the gateway branch on those codes. Logs do not store raw vendor bodies. ADR 0010 is the failure-policy contract for the model path.
This is what lets product services stay ignorant of vendor semantics. They handle a small set of outcomes. They do not special-case one provider's JSON.
Usage logs now, ledger later
model.usage and the acquisition usage facts carry workspace_id, request_id, tokens or units, latency_ms, and the provider profile. Billing must not run inside the hot path of a chat turn. If the fields arrive only when the ledger exists, the months before that cannot be audited. One service keeps the field names the same.
Long work does not hold the HTTP request
Crawl returns a job id and finishes in a worker. Knowledge ingest uses a queue and a worker in its own service. Same idea, different process: the request thread must not wait for N pages or for embeddings. Clients disconnect. Timeouts fire. The door stays blocked.
Workers scale by claiming jobs. For crawl, SKIP LOCKED is what makes two replicas safe. For knowledge, more consumers read the SQS-shaped queue. model-gateway's philosophy is "do not block the door with an unbounded vendor wait." Crawl applies that to a job that is long by nature.
Idempotency sits on the turn
model-gateway is not the idempotency store for chat. Conversations already take an idempotency key on messages and turns. A client retry must not double-append and must not double-call the model for the same turn key.
I mention it because vendor retry policy cannot fix a duplicate turn that our own API accepted twice. Resilience is end to end. The vendor layer is one cut. The conversation write is another. We documented them as related, not as the same mechanism.
How internal calls stay different from public calls
We did not want "internal" to mean "no identity, just a private port." The boundary is explicit.
| Concern | Public path | Internal path |
|---|---|---|
| Identity | JWT sub becomes the staff actorId
|
Service bearer token |
| Tenant |
workspace_id in the URL, plus a membership check |
workspace_id on the body or route, from a trusted caller |
| Exposure | Internet, through the gateway | Private network, or localhost ports in development |
| Stability | Versioned public API | Can change with the contracts package |
Runtime's ModelGatewayClient posts to {MODEL_GATEWAY_URL}/internal/complete with the internal token and a JSON body from @quickstart/contracts.
We chose internal HTTP, not "only gRPC" and not a shared SDK, for three practical reasons. A network hop is something we can later lock with Kubernetes network policy. Another language can call the same contract without importing our TypeScript package. And a hop forces a timeout and an error envelope at the boundary. An in-process function call makes it too easy to forget both.
The public gateway still does not replace per-service checks. Membership and permissions are enforced again downstream. A valid JWT at port 3000 is not a blank permission inside every service.
How the pieces scale
| Component | What we add | Shared state |
|---|---|---|
| API gateway | More replicas | Redis rate limit |
| Runtime | More replicas | None in the process. Conversations live in Postgres |
| model-gateway | More replicas | Redis in-flight cap. Breakers are per pod |
| web-fetch | More replicas and more workers | Redis budget and in-flight cap. Crawl jobs in Postgres |
| Knowledge worker | More consumers | SQS-shaped queue |
Chat capacity and LLM concurrency are separate dials. Adding runtime pods does not, by itself, multiply vendor connections, because the cap lives in Redis behind model-gateway. Adding model-gateway pods does not copy the API key into the runtime.
Breakers are the exception. They are per pod until we share that state. Doubling model-gateway replicas does not double the in-flight cap. During an outage, each pod can still let a few calls through until its own breaker opens.
Cost, keys, and the on-call page
Cost. One extra model attempt, and none after the stream starts, is the spend limit on retries. Web fetch spends the cheap direct fetch first and stops the workspace at the Redis budget. Token and unit logs are what we will use for chargeback. They are not the invoice.
Keys and compliance. Vendor credentials are deploy secrets on model-gateway and web-fetch only. Service-to-service calls use internal tokens. Staff JWTs stay on public routes. Internal ports are meant to sit on a private network in production.
Operations. Fake providers let CI exercise 429s, hangs, and partial streams without a vendor account. web-fetch has circuit tests in its suite. When OpenRouter is down, there is one service to open, not a list of every repo that imported a client.
What we would do next
Three items are already implied by the limits above. They are not shipped.
- Put circuit state in a shared store so every pod agrees that a profile is open.
- Add per-workspace caps on LLM calls. Today the model path only has a global in-flight cap.
- Store workspace keys through the secrets work, then let model-gateway use them. Until that exists, those requests stay a 400.
What we still don't do
I want this list in the post so the architecture diagram is not read as a finished platform.
- Circuit breaker state is per process, not cluster-wide.
- model-gateway does not enforce per-workspace dollar caps. It enforces a global in-flight cap. Per-tenant caps are planned elsewhere.
- There is no billing ledger on the request path. We write usage logs only.
- Knowledge embeddings use a lighter pattern. They do not go through the full provider registry and orchestrator described here.
- Knowledge calling web-fetch for URL ingest is a roadmap follow-up.
- The public gateway does not replace per-service RBAC. Membership and permissions are still enforced downstream.
Memory reads can degrade when memory is down. A knowledge-base turn can fail when retrieval was required and RAG is not healthy. Those are different features with different failure choices. They are not the outbound policy in this post. I include them only so "resilience" is not mistaken for "every dependency fails open."
Closing
The hard part was not the first successful LLM call. It was keeping keys, retries, streams, and spend in one place while chat traffic and vendor traffic grow on different curves.
An SDK in every service spreads the key and the mistakes. A shared package shares the code and still leaves policy inside every process. model-gateway and web-fetch are where vendor credentials and vendor failure live.
Adapters translate one vendor. Orchestrators decide what failure is allowed to cost. The rest of the platform calls a contract.
Top comments (0)