A reference architecture for running one AI assistant platform for many organisations, inside a data-residency boundary
This document describes a deployment pattern for an AI assistant platform: a system that answers questions over an organisation's documents, calls its internal tools, and takes actions on its behalf. It is written for architects and platform engineers who have to decide where such a system runs and have to defend that decision in a security review.
It is the first of three documents. Each covers one deployment pattern with the same structure: a one-picture architecture, the request flow, the components, use cases, design considerations, what crosses the boundary, and when to choose it. The design is independent of any product. The technologies named (Kong, Keycloak, Argo CD, Vault, pgvector, vLLM) are common choices and can be replaced without changing the design.
Terms used throughout. The provider is whoever builds and operates the platform: a vendor, or an internal platform team. The customer is the organisation whose users and data the platform serves: an external customer, or a business unit of your own company. A tenant is one customer on a shared platform.
Prerequisites. A cloud account with two regions inside the required residency geography, a Kubernetes service, managed Postgres and object storage, a key management service, and access to at least one model provider with a region inside the geography. A team that already runs Kubernetes in production.
Out of scope. Model quality, prompt design and the assistant's features. This series covers only where and how the platform runs.
The three patterns at a glance
Three questions decide the pattern. Ask them in this order.
- Where does the data live?
- Who operates the system?
- What crosses the boundary between the provider and the customer?
| Pattern | Data lives in | Operated by | Crosses the boundary |
|---|---|---|---|
| 1. Shared service (this document) | the provider's account, inside a residency boundary | the provider | every user's traffic, inside the boundary |
| 2. Control/data split | the customer's cloud account | the customer runs the data plane; the provider runs a control plane | signed software in, optional health metadata out |
| 3. Isolated site | the customer's building | the customer, everything | two signed files in, one optional file out |
Most organisations are served well by Pattern 1. Patterns 2 and 3 exist for the ones whose rules make it impossible.
Architecture
What makes it shared. Ten to fifteen tenants run in one Kubernetes cell, share one database cluster, and draw on the same model capacity. What makes it safe to share. Tenant identity is taken only from a credential the platform resolved itself, and every layer below re-checks it. Who runs it. The provider operates everything except the tenant's own identity provider and the tenant's internal systems.
Request flow
The numbered hops follow one chat message through the detailed topology at the end of this document.
| Hop | Path | Protocol | Who opens it | Authentication |
|---|---|---|---|---|
| ① | Client → CDN edge | HTTPS/443 | client | opaque session cookie, personal token or API key |
| ② | Edge → anycast entry point | HTTPS/443 | edge | secret origin header, checked at the gateway |
| ③ | Anycast entry → network load balancer | TCP/443 | anycast | none; TLS pass-through |
| ④ | Load balancer → API gateway | TCP/443 | load balancer | the caller's credential is resolved here |
| ⑤ | Gateway → identity broker | HTTPS/443 | gateway | mesh identity; login only, not on the chat path |
| ⑥ | Gateway → Chat Gateway | HTTP/8080 over mTLS | gateway | mesh identity |
| ⑦ | Chat Gateway → Orchestrator | gRPC over mTLS | Chat Gateway | mesh identity |
| ⑧ | Orchestrator → tools and actions | gRPC / HTTPS over mTLS | Orchestrator | mTLS; the sandbox holds no credentials |
| ⑨ | Orchestrator → data | Postgres, Redis, S3 | Orchestrator | cloud IAM role per tenant; per-tenant Redis ACL |
| ⑩ | Orchestrator → model router → model providers | HTTPS/443, private endpoints | Orchestrator | cloud IAM |
| ⑪ | Egress gateway → NAT → customer system | HTTPS/443 from static addresses | Orchestrator, via egress | per-tenant credential from Vault |
The answer streams back over the connection the client opened. If it drops, the client resumes from the last sequence number it received, for as long as the turn buffer holds it (about five minutes).
Components
| Component | Role | Common choice |
|---|---|---|
| CDN edge | TLS termination, web application firewall, DDoS protection, static cache; adds the secret origin header | CloudFront, Cloudflare |
| Anycast entry point | One stable address per residency boundary; traffic dial for failover | AWS Global Accelerator |
| API gateway | Resolves the caller, strips client-sent tenant headers, injects its own, rate limits | Kong |
| Identity broker | One realm per tenant, federated to the tenant's identity provider | Keycloak |
| Chat Gateway | Server-sent events streaming, heartbeat, resume from the turn buffer | custom service |
| Orchestrator | Durable step runtime: every inference and tool call is a resumable step | Temporal, custom |
| Tools and actions | Code sandbox, action gateway, tool connectors, per-user credential vault, inbound webhooks, metering | gVisor or Firecracker, MCP servers |
| Data | Postgres with pgvector, object storage per tenant prefix, Redis turn buffer, one KMS key per tenant | Aurora, S3, ElastiCache, KMS |
| Model router | One interface over several model providers, each pinned to a region | custom, LiteLLM |
| Egress gateway and NAT | Allowlisted outbound calls from static addresses in both regions | Envoy, NAT gateway |
| Delivery | GitOps for everything, signed images verified at admission, promotion gate | Argo CD, cosign, Kyverno |
| Observability and audit | Independent telemetry per region, synthetic canaries, one immutable audit trail | OpenTelemetry, Grafana, S3 Object Lock |
Use cases
A vendor serving many organisations from one platform
The classic SaaS case. The provider wants one codebase, one deployment, and one on-call rotation. Customers want a product that works on day one and accept that their content is stored in the provider's account, inside an agreed region.
An internal platform team serving many business units
The same design works inside one company. Business units are tenants, the platform team is the provider, and the residency boundary is the company's own cloud region. The isolation layers matter just as much: finance should not see legal's documents because they share a cluster.
A regulated customer who needs residency but not isolation
A customer who must keep data in a specific country can be served by Pattern 1 as long as the provider runs a full regional pair inside that country. Residency is a property of where the boundary is drawn, not of who operates the system.
Design considerations
Security and privacy
Only one way in. The edge adds a rotating secret header that the gateway requires. An attacker who finds the origin's address gains nothing, because the origin refuses anything that did not come through the edge and its firewall. Admins get a separate edge path with stricter rules and an optional IP allowlist.
The gateway resolves the caller; it does not trust a bearer token. Browser sessions are opaque secrets whose hash is stored server-side. CLI tokens and API keys are resolved the same way. The identity provider's JWT is verified once, at login, and never used as a request credential again. Nothing at rest is replayable.
Tenant identity comes only from the platform. The gateway strips any tenant header the client sent and injects its own, derived from the resolved principal. Every layer below re-checks it.
Isolation in eight layers. Tenants share a cell, so isolation is layered, and a failure in one layer is caught by the next.
| Layer | Per-tenant control |
|---|---|
| Identity | Own realm; tenant claim issued by the broker |
| Gateway | Own consumer, rate limit and quota |
| Cluster | Namespace with default-deny NetworkPolicy, scoped RBAC, ResourceQuota |
| Mesh | Authorization policy that re-checks the tenant claim on every hop |
| Cloud IAM | Own IAM role; no long-lived keys |
| Data | Policy compiled into every query, row-level security beneath it, own object prefix, own Redis ACL, own KMS key |
| Egress | Own destination allowlist |
| Model | Own token budget and context limit |
Tenancy has two levels: an organisation and the workspaces inside it. Workspaces share the tenant's namespace and are separated by policy.
Authorization is decided once. One policy decision point evaluates tenant and user policy and compiles it into every query. There is no "load then filter" in application code. Row-level security sits beneath it as defence in depth, not as the policy.
Tools and actions are gated. The sandbox holds no platform credentials and has default-deny egress with an allowlist of package registries. Email, messaging and writes go through an action gateway with per-tenant policy, an approval step for irreversible actions, and an audit record for each action. Tool connectors run against a per-tenant allowlist. Per-user tokens are sealed in a credential vault the model and the sandbox never see. Inbound webhooks enter through the edge with signature verification.
Each model provider is its own boundary. The router pins each provider to a region inside the residency boundary, uses private endpoints, and keeps invocation logging off. Residency and logging are recorded per provider, never "per platform".
Keys and exit. Each tenant has one multi-region KMS key, with an optional bring-your-own-key grant. At exit the key is disabled immediately and deleted after the KMS waiting period, which makes the tenant's data unreadable. Chat content is never written to logs; gateways, collectors and log shippers strip bodies.
Reliability
A second region runs as a warm standby at minimum scale: ingress, identity, the cell with its tools layer, data replicas, its own model capacity, its own NAT addresses and its own telemetry. Failover is deliberately not automatic. A person approves, then an orchestrator promotes the database, scales the cell and gateway, and moves the anycast traffic dial.
Design targets, to be validated in failover testing: recovery time about 15 minutes; recovery point of seconds for the database and within 15 minutes for object storage; the Redis turn buffer is lost, so chats in progress restart.
The identity broker is the single login dependency for every tenant. If it is down, nobody signs in. Its state lives in a global database so it survives a regional failover, but not its own outage.
Operations
Everything is delivered by GitOps, routes included. Nothing is hand-applied, self-heal is on, and promotion to production goes through a gate. Images are signed and the signature is verified at admission.
Each region has an independent telemetry stack. Synthetic canaries probe from outside the platform through the real edge, so operators see what users see. One audit trail collects cloud API calls, cluster and gateway audit, every agent action with its tenant and user, and every break-glass session, normalised to one schema and written to immutable storage. Every line carries a credential id, never the secret.
Cost
The shared cell is the main saving: one cluster, one database cluster and one model quota serve many tenants. The main costs are the warm standby, which runs all the time, and the edge plus anycast plus gateway plus mesh stack, which is more components than any one alone. Each residency geography needs its own full regional pair, so the cost scales with the number of geographies, not the number of tenants.
Performance
The chat tier scales on active streaming connections, not CPU, because idle connections use almost none. Model capacity is shared per account per region and controlled by per-tenant token budgets. Tool output is capped so one call cannot consume the context budget.
Design alternatives
- DNS-only failover instead of an anycast entry. Cheaper, but depends on resolver caches honouring TTLs. The traffic dial switches new connections in seconds.
- An application load balancer instead of a layer-4 load balancer. Would terminate TLS twice and duplicate the routing the gateway already does.
- A database per tenant. Removes the shared cluster key and blast radius at the cost of many more clusters. Right for a premium or regulated tier, not the default.
- Trusting the identity provider's JWT on every request. Simpler, but a long-lived bearer token at rest is replayable.
- A managed sandbox service. Less to operate, but it is a third party that sees content, and it must appear in the boundary table.
- Automatic failover. Faster, until it fires on a false alarm.
What crosses the boundary
Fill this table in for your own design before a security review asks for it.
| Crossing | Direction | What travels | Can the customer switch it off? |
|---|---|---|---|
| User traffic → edge | customer → provider | requests and streamed answers | no, it is the product; the customer controls accounts and the admin allowlist |
| Identity broker ↔ tenant IdP | both, sign-in only | assertions and claims, no business data | yes; disabling federation stops sign-in |
| Model router → model providers | provider → model providers | prompts with retrieved content | no; per-provider residency and logging terms are recorded |
| Sandbox and tool servers | provider → sandbox and allowlisted tools | code, files, tool inputs | yes, per tool |
| Action gateway → channels | provider → email, messaging, systems | the agent's actions | yes, per action type and tenant |
| Inbound webhooks | third parties → edge → gateway | signed callbacks | yes, by not enabling the channel |
| Egress → customer systems | provider → customer, from static addresses | connector calls and results | yes; remove the firewall rule or disable the connector |
| Per-tenant SIEM export | provider → customer | that tenant's audit events | yes, opt-in |
| Bring-your-own-key grant | customer → provider | permission to use the customer's key | yes; revoking makes that tenant's data unreadable |
Trade-offs
- Tenants share a cluster and a database cluster key. One cell's blast radius spans its tenants.
- Model quota is shared per account per region, controlled by per-tenant budgets.
- The identity broker is the single login dependency.
- Failover is human-approved and takes about 15 minutes as a design target. Chats in progress restart.
- Each residency geography needs its own regional pair.
When to choose this pattern
Choose the shared service when the customer wants the provider to run everything, accepts content stored in the provider's account inside an agreed residency boundary, and does not need an air gap. Put the effort into the eight isolation layers and the audit trail.
Choose Pattern 2 if the customer's documents, index and keys must stay in their own account under a key they can revoke. Choose Pattern 3 if nothing may leave their building, not even a prompt.
Deployment outline
Build from the inside out, so that each layer is verified before the next depends on it.
- Foundation. Two regions, VPCs, the Kubernetes cell in each, the global database with its replica, object storage with cross-region replication, one KMS key per tenant.
- Identity and gateway. The identity broker with one realm per tenant, federated to each tenant's identity provider. The API gateway with opaque session resolution, per-tenant consumers and the tenant-header rule.
- Edge. The CDN distribution with WAF and the secret origin header, the anycast entry point with the Region B dial at zero, the separate admin distribution.
- Application. Chat Gateway, Orchestrator, retrieval and connector services, the service mesh with mTLS and the tenant re-check policy, the admission controller that generates namespaces, quotas and network policies per tenant.
- Tools and actions. The sandbox with default-deny egress, the action gateway, tool connectors, the per-user credential vault, webhook intake, metering.
- Model router and egress. Providers pinned to regions, private endpoints, invocation logging off; the egress gateway and NAT with static addresses in both regions.
- Delivery and observability. GitOps for every component, image signing and admission verification, the promotion gate; telemetry per region, synthetic canaries, the audit pipeline into immutable storage.
- Failover. The DR orchestrator and runbook, then a full failover test that measures the recovery figures and replaces the design targets with results.
Detailed topology
The full design in thirteen bands, in the order a request travels. The hop numbers match the request flow table above. Open the figure at full size to read the component notes.
What's next
This is part 1 of a three-part series. The other parts are also linked in the series box at the top of this article.
- Pattern 2, the control/data split (customer cloud): the customer runs the whole data plane in their own cloud account.
- Pattern 3, the isolated site (on-premises and air-gapped): everything runs in the customer's building, with models on local GPUs.
Top comments (0)