DEV Community: Rajesh Yadav

How to deploy an AI assistant: Pattern 2, the control/data split (customer cloud)

Rajesh Yadav — Tue, 03 Nov 2026 05:21:35 +0000

A reference architecture for running the data plane of an AI assistant in the customer's own cloud account, with the provider keeping only a control plane

This document describes the second of three deployment patterns for an AI assistant platform. It is written for architects and platform engineers whose customer, or whose own organisation, has said: "our documents, our search index and our encryption key stay in our cloud account, under a key we can revoke without asking anyone."

The structure is the same as Pattern 1: a one-picture architecture, the request flow, the components, use cases, design considerations, what crosses the boundary, and when to choose it. The design is independent of any product; the technologies named can be replaced.

Terms used throughout. The provider is whoever builds and operates the platform software: a vendor, or an internal platform team. The customer is the organisation whose users and data the platform serves. In this pattern the customer also operates most of the system.

Prerequisites. On the customer's side: a cloud account, a Kubernetes service, managed Postgres and object storage, a key management service, and access to a cloud model provider in the chosen region. On the provider's side: a build pipeline that can sign artifacts and an OCI registry. The customer needs a team that can run Kubernetes and its supporting services in production.

The three patterns at a glance

Pattern	Data lives in	Operated by	Crosses the boundary
1. Shared service	the provider's account, inside a residency boundary	the provider	every user's traffic, inside the boundary
2. Control/data split (this document)	the customer's cloud account	the customer runs the data plane; the provider runs a control plane	signed software in, optional health metadata out
3. Isolated site	the customer's building	the customer, everything	two signed files in, one optional file out

The three questions still decide: where does the data live, who operates it, what crosses the boundary. For this pattern the third question is the one to get right, and the next section shows why.

The design to avoid: split by component

The obvious way to meet the customer's rule is to split by component: move the pieces that store customer content into the customer's account and keep the rest.

The customer runs retrieval over their own vector store, a connector module that reaches their systems from inside their network, a cache, their documents bucket, Vault and a KMS key they can revoke. The provider reaches these over one private link with mutual TLS. Software is pull-only. Stored data never leaves the customer's account.

Now trace one chat message and mark every arrow where customer content crosses an account boundary with ◆.

#	Crossing	Why it exists
①	The user's question travels through the provider's edge, gateway and Chat Gateway	The entry point is the provider's
②	Retrieved passages come back over the link	The Orchestrator is the provider's
③	Results from the customer's systems come back over the link	Same reason
④	The full prompt goes to the model in the provider's account	The model is the provider's
⑤	The streamed answer sits in the provider's Redis for about five minutes	The Chat Gateway is the provider's

Five crossings per question. None of them is a breach: each is encrypted, disclosed, inside the residency boundary and gone within minutes. But the design cannot support the sentence the customer wants: "your data never reaches our account." Stored data does not; data in flight does, five times.

Two rules follow, and they apply to any architecture:

Mark every arrow where content crosses a boundary. If you cannot draw it, you do not know it.
"Defensible" is the floor. The useful question is whether the customer can verify the promise without trusting the provider.

Architecture

The fix is to split by plane, not by component.

The control plane is everything that is the same for every customer and carries no content: building and signing software, publishing it, issuing licences, collecting health metadata, taking support requests. The provider keeps this.
The data plane is everything that touches a user, a document, a system or a prompt: entry point, gateway, sign-in, chat, Orchestrator, tools and actions, retrieval, connectors, data, keys, and the model call. The customer runs all of it, in their account.

Three things change. There is no connectivity link, because there is nothing for the provider to reach, and the provider has no inbound path at all. The model call stays home: the Orchestrator calls the customer's own cloud model providers in the customer's account, under the customer's contract. There is nothing to crypto-shred at exit, because the provider holds no configuration, sessions or buffer. The ◆ marker appears nowhere.

Request flow

Hop	Path	Protocol	Who opens it	Authentication
①	Client → customer edge	HTTPS/443	client	opaque session cookie or token, resolved at the gateway
②	Edge → API gateway	HTTPS/443	load balancer	n/a (network hop)
③	Gateway → identity broker	HTTPS/443	gateway	login and signing keys only; not on the chat path
④	Gateway → Chat Gateway	HTTP/8080 over mTLS	gateway	mesh identity
⑤	Chat Gateway → Orchestrator	gRPC over mTLS	Chat Gateway	mesh identity
⑥	Orchestrator → tools and actions	gRPC / HTTPS over mTLS	Orchestrator	mTLS; the sandbox holds no credentials; per-user tokens from the vault
⑦	Orchestrator → retrieval → pgvector	gRPC; then Postgres/5432	Orchestrator, then retrieval	mTLS; database role from the customer's Vault
⑧	Orchestrator → model router → customer's model providers	HTTPS/443, private endpoint	Orchestrator	the customer's cloud identity
⑨	Orchestrator → connector hub → customer systems	gRPC; then HTTPS	Orchestrator, then connector	mTLS; connector credentials from Vault
⑩	Streamed answer → Chat Gateway → user	SSE over HTTPS	already open	same session as ①

Every hop stays inside the customer's boundary. The provider's column in the detailed topology has no live-request arrows at all.

Components

Component	Runs in	Operated by	Common choice
Customer edge: domain, certificate, WAF, load balancer	customer account	customer	cloud load balancer and WAF
API gateway, identity broker, identity provider	customer account	customer	Kong, Keycloak, the customer's IdP
Chat Gateway, Orchestrator, retrieval, connector hub, Redis, mesh	customer's Kubernetes cluster	customer	EKS, AKS or GKE; Istio or Linkerd
Tools and actions: sandbox, action gateway, tool connectors, per-user credential vault, metering	customer account	customer	gVisor or Firecracker, MCP servers
Postgres with pgvector, documents bucket, Vault, KMS key	customer account	customer	managed Postgres, object storage, Vault
Model router and model accounts	customer account, customer's contract	customer	Bedrock, Azure OpenAI, Vertex AI
Delivery: Argo CD, release gate, Kyverno, optional registry mirror	customer account	customer	Argo CD, cosign, Kyverno
Observability, egress proxy, audit, SIEM	customer account	customer	OpenTelemetry stack
Release pipeline, signing, OCI registry, signed licence	provider	provider	GitHub Actions, cosign, Harbor or a cloud registry
Health dashboard (opt-in heartbeat), support intake	provider	provider	Grafana

Use cases

A customer whose data must stay in their own cloud account

The defining case. Documents, index and keys in the customer's account, encrypted under a key the customer can revoke, with no content in the provider's infrastructure. The customer can operate Kubernetes and its supporting services, or has a managed-service team that can.

A provider who must support several clouds

Because the customer runs the data plane, the provider's software has to run on whichever cloud the customer uses. The design keeps every cloud-specific dependency behind an interface: the model router, the identity binding for the registry pull, and the key management.

A regulated business unit inside one company

An internal platform team is the provider; a business unit with its own cloud account and its own compliance regime is the customer. The pull-only delivery and the egress proxy log give the business unit's auditors proof without trusting the platform team.

Design considerations

Security and privacy

Delivery without access. The provider never pushes anything. Its pipeline signs every chart and image, attaches an SBOM and a vulnerability-scan attestation, and publishes to an OCI registry. In the customer's cluster, Argo CD polls the registry over the customer's own outbound HTTPS and applies new versions; a release gate lets the customer review the SBOM and attestation first; Kyverno verifies the signature with a bundled, offline trust root before any pod starts. The customer holds every admin credential.

For cross-cloud registry access, two methods work: federate the cluster's workload identity into a pull-only role at the provider, or issue a per-customer pull-only robot token the customer keeps in their own Vault. Both are scoped to that customer's repositories and can only pull.

Identity inside the boundary. The customer runs the identity broker, and the identity provider sits inside their boundary. The IdP token is verified once at login; after that the browser holds an opaque, hash-stored session. CLI tokens and API keys resolve the same way. No sign-in assertion crosses to the provider.

Tools and actions in the customer's account. The sandbox is self-hosted in the customer's account with no platform credentials and default-deny egress; a managed sandbox service would be a third party that sees content. The action gateway applies the customer's policy and approval steps. Tool connectors run against an allowlist the customer sets. Per-user tokens are sealed in a credential vault with a key id. Metering writes to a local store with limits per organisation and workspace.

The model router points at the customer's own providers. Several providers behind one interface, each in the customer's account, pinned to a region the customer chooses, reached over private endpoints, with invocation logging as the customer's own setting. The prompt never touches the provider's account. The nuance to state plainly: the model provider still sees prompts, under the customer's contract rather than the vendor's.

Data and authorization. Postgres with pgvector, the documents bucket, Redis and Vault live in the customer's account. Authorization is one decision point: policy compiled into every query, with row-level security beneath it. Encryption uses a customer-owned KMS key. Revoke it and everything stops and becomes unreadable, including to provider support.

Reliability

High availability and disaster recovery are the customer's: multi-zone, a second region per their policy, immutable backups. The provider states no recovery targets here because there is no provider data plane to fail over. The provider commits only to registry availability, and an optional local registry mirror removes even that runtime dependency.

Operations

Telemetry is local by default: Fluent Bit, OpenTelemetry, Loki, Jaeger, VictoriaMetrics, a local Grafana. Cloud and cluster audit logs stay in the account. The egress proxy log is the customer's proof: it shows exactly two external names ever reached for the provider (the registry and, if opted in, the heartbeat), plus the customer's own model, identity and system endpoints.

Support works two ways, both optional: a redacted support bundle the customer reads before sending, or a break-glass session the customer opens, approves, time-boxes and records in their own audit log. The break-glass session is the one path that could ever show content to the provider, and the diagram says so.

Cost

The customer pays for the whole data plane: cluster, databases, model consumption under their own contract, a second region if they want one. Managed cloud services soften the operating cost but do not remove it. The provider's cost is small and fixed: a pipeline, a registry, a dashboard.

Performance

Model capacity, quota and regional placement are the customer's. The provider cannot add quota or fix an outage at the customer's model provider. Everything else scales as in Pattern 1, inside the customer's cluster.

Design alternatives

Split by component (the first design above). Fewer things for the customer to run, but five content crossings per question.
A managed sandbox service. Less to operate, but a third party that sees content.
Push-based delivery with a provider credential into the customer's cluster. Simpler for the provider, but gives it an inbound path, which this pattern exists to remove.
Automatic heartbeat. Useful for support, but it must stay opt-in and payload-free, or the egress proxy log stops being proof.

What crosses the boundary

Crossing	Direction	What travels	Who opens it	Can the customer switch it off?
Software pull	customer → provider registry; software flows in	signed charts, images, licence	the customer's Argo CD and nodes	yes; the running version keeps working, and a mirror removes the runtime dependency
Heartbeat	customer → provider	versions, health, error counts; fixed schema, no payloads	the customer's agent	yes; opt-in
Support bundle	customer → provider, manual	redacted diagnostics the customer read first	the customer	yes; never required
Break-glass session	customer opens; provider joins	whatever the customer shows; nothing retained	the customer	yes; per incident, time-boxed, recorded
Model call	customer → their own model provider	prompt with retrieved passages	the customer's Orchestrator	it is the product; but it stays under the customer's account, contract and region

Compare this with the first design's five content crossings per question. That comparison is the whole argument for splitting by plane.

Trade-offs

The customer runs the whole data plane. This is close to the operating burden of an on-premises deployment, eased by managed cloud services.
Support works half-blind: heartbeat and redacted bundles by default.
Several model providers to support, with differences in models, limits and behaviour.
No provider recovery targets. The provider commits only to registry availability.
Version skew: N and N-1 are supported; a customer who stalls their release gate drifts out of the window.
Not zero third parties. The customer's model provider still processes prompts.

When to choose this pattern

Choose the control/data split when the customer's rule is "our documents, index and keys in our account, under a key we can revoke, and no content in your infrastructure", and they can operate Kubernetes and its supporting services. If they want the provider to run everything, Pattern 1. If not even a prompt may leave their building, Pattern 3.

Deployment outline

The provider prepares; the customer deploys.

Provider: release pipeline. Build, sign every chart and image, attach the SBOM and scan attestation, publish to the OCI registry, issue the signed licence and a pull-only credential per customer.
Customer: foundation. Cloud account, network, Kubernetes cluster, managed Postgres with pgvector, object storage, Vault, a customer-owned KMS key, the model accounts with private endpoints.
Customer: delivery path. Argo CD pointed at the provider's registry over outbound HTTPS, the release gate, Kyverno with the bundled trust root, the optional registry mirror. Verify that the egress proxy log shows only the registry.
Customer: identity and edge. The identity broker inside the boundary, federated to the customer's identity provider; the customer's domain, certificate, WAF and load balancer; the API gateway.
Customer: application and tools. Argo CD applies the signed charts: Chat Gateway, Orchestrator, retrieval, connectors, mesh, the tools-and-actions layer, the model router configured for the customer's providers.
Customer: observability and support. Local telemetry, audit logs to the customer's SIEM, the opt-in heartbeat if wanted, the break-glass procedure written into the customer's runbook.
Customer: resilience. High availability, a second region if policy requires it, immutable backups, and a test that revoking the KMS key stops everything as intended.

Detailed topology

The full design in ten customer bands plus the provider's two control-plane bands. The hop numbers match the request flow table above. Open the figure at full size to read the component notes.

What's next

This is part 2 of a three-part series. The other parts are also linked in the series box at the top of this article.

Pattern 1, the shared service (SaaS): the provider runs everything inside a residency boundary.
Pattern 3, the isolated site (on-premises and air-gapped): everything runs in the customer's building, with models on local GPUs.

How to deploy an AI assistant: Pattern 3, the isolated site (on-premises and air-gapped)

Rajesh Yadav — Tue, 03 Nov 2026 05:12:53 +0000

A reference architecture for running the whole AI assistant platform, models included, inside the customer's own facility

This document describes the third of three deployment patterns for an AI assistant platform. It is written for architects and platform engineers whose customer, or whose own organisation, has said: "No prompt leaves this building. Not to a vendor, not to a cloud model provider, not to anyone. Our network has no route to the internet."

Defence, sovereign functions, regulated manufacturing, some banks and hospitals. Fewer organisations need this than the first two patterns, but the ones who ask for it will check.

The structure is the same as Patterns 1 and 2. The design is independent of any product; the technologies named can be replaced.

Terms used throughout. The provider is whoever builds and signs the platform software: a vendor, or an internal platform team. The customer is the organisation whose facility the platform runs in. In this pattern the customer operates everything.

Prerequisites. A facility with its own network, GPU servers sized for peak inference, a Kubernetes distribution that can be installed offline, an internal certificate authority and time source, an approved process for carrying media in and out, and a team that can run Kubernetes, databases, Vault and GPUs without vendor access. A second data center if site-level recovery is required.

The three patterns at a glance

Pattern	Data lives in	Operated by	Crosses the boundary
1. Shared service	the provider's account, inside a residency boundary	the provider	every user's traffic, inside the boundary
2. Control/data split	the customer's cloud account	the customer runs the data plane; the provider runs a control plane	signed software in, optional health metadata out
3. Isolated site (this document)	the customer's building	the customer, everything	two signed files in, one optional file out

For this pattern the three questions have short answers. The data lives in the building. The customer operates everything. Nothing crosses on an ordinary day. The rest of this document is what it takes to make those sentences true, and what it costs.

Architecture

Four things to notice. There is no provider column. The provider builds, signs and prepares bundles and licences, and runs nothing in or near the customer. The models are local. A model router in front of vLLM or Triton on the customer's own GPUs replaces every cloud model call. The air gap is drawn as a band, not a line. The bottom of the diagram is the complete list of what ever crosses, with direction and condition. The ◆ marker appears nowhere, because there is no arrow to put it on.

A connected on-premises variant uses the same stack with one difference: bundles are pulled over a single approved outbound HTTPS channel instead of physical media.

Request flow

Hop	Path	Protocol	Who opens it	Authentication
①	Global traffic manager → load balancer	HTTPS/443; internal DNS steers to the active site	client	opaque session cookie or token, resolved at the gateway
②	Load balancer → OpenShift Router	HTTPS/443	load balancer	n/a (network hop)
③	Router → API gateway	HTTPS/443	router	n/a (network hop)
④	Gateway → identity broker	HTTPS/443	gateway	login and signing keys only; not on the chat path
⑤	Gateway → Chat Gateway	HTTP/8080 over mTLS	gateway	mesh identity
⑥	Chat Gateway → Orchestrator	gRPC over mTLS	Chat Gateway	mesh identity
⑦	Orchestrator → tools and actions	gRPC / HTTPS over mTLS	Orchestrator	mTLS; the sandbox holds no credentials; per-user tokens from the vault
⑧	Orchestrator → retrieval → pgvector	gRPC; then Postgres/5432	Orchestrator, then retrieval	mTLS; database role from Vault; embeddings from a local model
⑨	Orchestrator → model router → local model servers	gRPC / HTTP over mTLS, inside the cluster	Orchestrator	mesh identity; admission control per user
⑩	Orchestrator → connector hub → internal systems	gRPC; then HTTPS	Orchestrator, then connector	mTLS; connector credentials from Vault
⑪	Streamed answer → Chat Gateway → user	SSE over HTTPS	already open	same session as ①

The table is almost identical to Pattern 2's. The application does not change between patterns; the ground it stands on does.

Components

Component	Role	Common choice
Global traffic manager, load balancers, router	Internal entry path; steers users to the active site	F5 GTM, NGINX, OpenShift Router
Internal CA	Certificates for every service, renewed locally	cert-manager with an internal CA
API gateway, identity broker, directory	Opaque session resolution; IdP token verified once at login	Kong, Keycloak, Active Directory or LDAP
Cluster: Chat Gateway, Orchestrator, retrieval, connector hub, Redis, mesh	The application, single-customer, mTLS between every service	OpenShift, Istio or Linkerd
Tools and actions	Sandbox with no egress, action gateway, MCP connectors to internal tools, per-user credential vault, internal webhooks, metering	gVisor or Firecracker, MCP servers
Model router, model servers, embedding model	Several local models behind one interface; request queue and admission control	vLLM, Triton
GPU nodes	Inference capacity, N+1	customer-owned
Data	Postgres with pgvector, object storage, offline immutable backups	CloudNativePG, MinIO, Velero
Secrets and keys	Vault with HSM auto-unseal; local key management	Vault, HSM
Delivery	Scan station, checksum and signature verification with an offline trust root, local registry, GitOps, admission verification	cosign, Harbor, Argo CD, Kyverno
Observability and audit	All local; SIEM export; self-diagnostics	Fluent Bit, OpenTelemetry, Loki, Jaeger, VictoriaMetrics, Grafana
Licensing and time	Signed licence file verified locally; internal NTP	custom, NTP
DR data center	Warm standby: load balancer, minimum-scale cluster, GPU nodes, replicas	same stack

Use cases

A facility with no route to the internet

Defence and sovereign environments, where the network is physically isolated and every transfer in or out is a controlled event on approved media. The pattern as drawn.

A regulated site that permits one outbound channel

Manufacturing, utilities, some financial institutions. The connected variant: the same stack, with software bundles pulled over one approved outbound HTTPS channel and nothing else leaving. Everything on the diagram stays as drawn except the physical media.

An organisation that cannot accept any cloud model provider

Even under their own contract and in their own region, some organisations cannot send prompts to a cloud model provider. Pattern 2 does not help them; this pattern does, at the cost of a local model class and local GPUs.

Design considerations

Security and privacy

Delivery across an air gap. The software pull of Pattern 2 is replaced by a bundle: images, the Helm chart, model weights and an SBOM, signed, written to approved physical media, and carried in through the customer's channel. Inside, it passes a customer-run scan and quarantine station, then checksum and signature verification against an offline trust root that shipped with the first installation. There is no call to a transparency log because there is nothing to call. The trust root is rotated through the same channel, with the old key signing the new one. From the local registry onward, the cluster behaves exactly as in Pattern 2: Argo CD pulls, Kyverno verifies before any pod starts, nothing is hand-applied.

Identity inside the building. The identity broker and the directory both sit inside the facility. The IdP token is verified once at login; the browser then holds an opaque session stored as a hash. CLI tokens and API keys resolve the same way. Certificates come from the internal CA and expiry is monitored, because there is no public CA to fall back on.

Tools and actions with nowhere to go. The sandbox has default-deny egress and takes packages only from the local registry mirror; there is no package index to reach, and this constraint belongs in the product documentation. Tool connectors run against internal tool servers only. The action gateway applies policy and approval steps for irreversible actions. Per-user tokens are sealed in a credential vault. Inbound webhooks come only from internal systems, through the same entry path, with signatures verified. Metering writes to a local store with per-department limits.

Models on the customer's GPUs. A model router fronts several local model servers behind one interface, with a request queue and admission control per user. A local embedding model serves retrieval on the same GPU pool. Nothing calls out. Model updates arrive as weights in the bundle, verified like everything else.

Data, keys and Vault. Authorization is one decision point: policy compiled into every query, row-level security beneath. Keys live in a local HSM or software KMS. Backups are offline and immutable, on separate media, because replication copies corruption and ransomware faithfully. Vault unseal uses HSM auto-unseal where possible; with Shamir shares, Vault stays sealed after a power loss until three of five key holders are present, which puts people in the recovery path.

Reliability

A DR data center runs as a warm standby: a load balancer with a standby VIP, a minimum-scale cluster, GPU nodes for degraded-capacity inference, and replicas of object storage, Postgres, Vault and the registry. Failover is the customer's runbook with human approval; the global traffic manager steers users to the surviving site.

Design targets, to be validated in the customer's own failover testing: recovery time about one hour; recovery point of seconds to minutes for Postgres and minutes for object storage; reduced GPU capacity until the primary site is restored. A customer with one data center gets local high availability and offline backups, and no site-level DR.

GPU capacity is the hard ceiling. There is no cloud to burst into. Size for peak, queue beyond it, and build N+1 so one failing node does not stop inference.

Operations

Everything is local: Fluent Bit, OpenTelemetry, Loki, Jaeger, VictoriaMetrics, a local Grafana, feeding the customer's SIEM together with cluster and infrastructure audit logs. Nothing is collected centrally by the provider, ever. There is no heartbeat because there is no line for it.

Support has to be designed for this: built-in self-diagnostics with a one-click health report, a redacted support bundle the customer reads before it leaves and sends only if they choose to, and an on-site provider engineer or supervised session only if the customer allows. Remote sessions exist only in the connected variant.

Patch latency is the operating cost that most needs saying early. Security fixes arrive at the bundle cadence, monthly as standard with emergency bundles for critical CVEs, and each needs a physical transfer, a scan and a verification.

A signed licence file is verified locally with no network check; expiry leads to a grace period and in-app warnings, never a silent stop. Licence and certificate checks depend on a correct internal NTP source, so clock drift is monitored as a first-class alert.

Cost

The customer pays for everything: GPUs, servers, storage, a second site, and the people to run them. Nothing is managed. This is the heaviest operating burden of the three patterns by a wide margin, and the only one with capital hardware cost.

Performance

Performance is bounded by local GPU capacity and model class. Only models that can be delivered as weights and served on this hardware are available; frontier hosted models are out of scope by definition. The request queue and admission control make the ceiling visible to users instead of letting requests fail.

Design alternatives

The connected on-premises variant. One approved outbound HTTPS channel for bundles. Removes physical media and shortens patch latency, at the cost of one line out of the facility.
Shamir unseal instead of an HSM. Cheaper, but puts people in the recovery path.
One data center. Removes the second site's cost and gives up site-level DR; local HA and offline backups remain.
Pattern 2 with a private model endpoint. For an organisation that can accept a cloud model provider under its own contract, Pattern 2 is lighter and offers more model choice.

What crosses the air gap

Crossing	Direction	What travels	Who carries it	Can the customer refuse it?
Offline bundle	provider → customer, physical media	signed images, Helm chart, model weights, SBOM	the customer's approved channel; scanned and verified before the registry	yes; the running version keeps working
Licence renewal file	provider → customer, same channel	signed file: new expiry and entitlements	same as above	yes; expiry means a grace period and warnings, never a silent stop
Support bundle (optional)	customer → provider, manual	redacted diagnostics the customer read first	the customer	yes; never required; the alternative is an engineer on site

That is the whole table. No heartbeat, no software pull, no model call, no sign-in assertion.

Trade-offs

The customer runs everything, including GPUs and a second site.
Model class is fixed by hardware: local open-weight models only.
Patch latency at the bundle cadence.
Support works blind: bundles the customer chooses to send, or a visit.
People in the recovery path unless an HSM is used for unseal.
No provider commitments on recovery. Every figure is the customer's to validate.

When to choose this pattern

Choose the isolated site when the rule is "no prompt leaves this building, and there is no route out", when the organisation can operate Kubernetes, GPUs, databases and a second site, and when a local model class meets the need. If a cloud model provider under their own contract is acceptable, Pattern 2 is lighter. If they want the provider to run it, Pattern 1.

Deployment outline

Everything arrives as a bundle and is installed by the customer.

First bundle. Images, Helm chart, model weights, SBOM, the offline trust root and the signed licence, on approved media. Scan, verify the checksum and signatures, load into the local registry.
Foundation. The Kubernetes cluster, GPU nodes, Postgres with pgvector, object storage, Vault with HSM auto-unseal, the internal certificate authority and time source.
Delivery path. Argo CD pointed at the local registry, Kyverno with the offline trust root, the scan and quarantine station as a standing procedure.
Identity and entry path. The identity broker federated to the internal directory, the global traffic manager, load balancers, router and API gateway with internal certificates.
Application, tools and models. Argo CD applies the signed charts: Chat Gateway, Orchestrator, retrieval, connectors, mesh, the tools-and-actions layer, the model router and model servers with the delivered weights.
Observability, licensing, support. Local telemetry into the SIEM, the licence file loaded and verified, self-diagnostics and the support-bundle procedure agreed with the customer.
Second site and backups. The DR data center with replicas and a minimum-scale cluster, offline immutable backups, then a failover test that replaces the design targets with measured results.

Detailed topology

The full design in twelve bands. The hop numbers match the request flow table above. Open the figure at full size to read the component notes.

Choosing between the three patterns

	Pattern 1: shared service	Pattern 2: control/data split	Pattern 3: isolated site
Where the data lives	provider's account, inside a residency boundary	customer's cloud account	customer's building
Who operates	provider	customer (data plane), provider (control plane)	customer, everything
Model	provider's model accounts, pinned to the region	customer's own cloud model providers	local models on the customer's GPUs
Software delivery	provider deploys	customer pulls signed software	signed bundle on physical media
Support	provider sees its own system	redacted bundle, or break-glass on the customer's terms	redacted bundle or on-site visit
Operating burden on the customer	low	heavy	heaviest
What crosses the boundary	every user's traffic, inside the boundary	software pull, opt-in heartbeat, support; model call to the customer's own provider	two signed files in, one optional file out

Three patterns, one application, the same three questions. If you can answer them for your own system, with every crossing drawn and every figure labelled for what it is, you can answer the questions a security review will ask.

What's next

This is part 3 of a three-part series. The other parts are also linked in the series box at the top of this article.

Pattern 1, the shared service (SaaS): the provider runs everything inside a residency boundary.
Pattern 2, the control/data split (customer cloud): the customer runs the whole data plane in their own cloud account.

How to deploy an AI assistant: Pattern 1, the shared service (SaaS)

Rajesh Yadav — Sat, 17 Oct 2026 14:03:52 +0000

A reference architecture for running one AI assistant platform for many organisations, inside a data-residency boundary

This document describes a deployment pattern for an AI assistant platform: a system that answers questions over an organisation's documents, calls its internal tools, and takes actions on its behalf. It is written for architects and platform engineers who have to decide where such a system runs and have to defend that decision in a security review.

It is the first of three documents. Each covers one deployment pattern with the same structure: a one-picture architecture, the request flow, the components, use cases, design considerations, what crosses the boundary, and when to choose it. The design is independent of any product. The technologies named (Kong, Keycloak, Argo CD, Vault, pgvector, vLLM) are common choices and can be replaced without changing the design.

Terms used throughout. The provider is whoever builds and operates the platform: a vendor, or an internal platform team. The customer is the organisation whose users and data the platform serves: an external customer, or a business unit of your own company. A tenant is one customer on a shared platform.

Prerequisites. A cloud account with two regions inside the required residency geography, a Kubernetes service, managed Postgres and object storage, a key management service, and access to at least one model provider with a region inside the geography. A team that already runs Kubernetes in production.

Out of scope. Model quality, prompt design and the assistant's features. This series covers only where and how the platform runs.

The three patterns at a glance

Three questions decide the pattern. Ask them in this order.

Where does the data live?
Who operates the system?
What crosses the boundary between the provider and the customer?

Pattern	Data lives in	Operated by	Crosses the boundary
1. Shared service (this document)	the provider's account, inside a residency boundary	the provider	every user's traffic, inside the boundary
2. Control/data split	the customer's cloud account	the customer runs the data plane; the provider runs a control plane	signed software in, optional health metadata out
3. Isolated site	the customer's building	the customer, everything	two signed files in, one optional file out

Most organisations are served well by Pattern 1. Patterns 2 and 3 exist for the ones whose rules make it impossible.

Architecture

What makes it shared. Ten to fifteen tenants run in one Kubernetes cell, share one database cluster, and draw on the same model capacity. What makes it safe to share. Tenant identity is taken only from a credential the platform resolved itself, and every layer below re-checks it. Who runs it. The provider operates everything except the tenant's own identity provider and the tenant's internal systems.

Request flow

The numbered hops follow one chat message through the detailed topology at the end of this document.

Hop	Path	Protocol	Who opens it	Authentication
①	Client → CDN edge	HTTPS/443	client	opaque session cookie, personal token or API key
②	Edge → anycast entry point	HTTPS/443	edge	secret origin header, checked at the gateway
③	Anycast entry → network load balancer	TCP/443	anycast	none; TLS pass-through
④	Load balancer → API gateway	TCP/443	load balancer	the caller's credential is resolved here
⑤	Gateway → identity broker	HTTPS/443	gateway	mesh identity; login only, not on the chat path
⑥	Gateway → Chat Gateway	HTTP/8080 over mTLS	gateway	mesh identity
⑦	Chat Gateway → Orchestrator	gRPC over mTLS	Chat Gateway	mesh identity
⑧	Orchestrator → tools and actions	gRPC / HTTPS over mTLS	Orchestrator	mTLS; the sandbox holds no credentials
⑨	Orchestrator → data	Postgres, Redis, S3	Orchestrator	cloud IAM role per tenant; per-tenant Redis ACL
⑩	Orchestrator → model router → model providers	HTTPS/443, private endpoints	Orchestrator	cloud IAM
⑪	Egress gateway → NAT → customer system	HTTPS/443 from static addresses	Orchestrator, via egress	per-tenant credential from Vault

The answer streams back over the connection the client opened. If it drops, the client resumes from the last sequence number it received, for as long as the turn buffer holds it (about five minutes).

Components

Component	Role	Common choice
CDN edge	TLS termination, web application firewall, DDoS protection, static cache; adds the secret origin header	CloudFront, Cloudflare
Anycast entry point	One stable address per residency boundary; traffic dial for failover	AWS Global Accelerator
API gateway	Resolves the caller, strips client-sent tenant headers, injects its own, rate limits	Kong
Identity broker	One realm per tenant, federated to the tenant's identity provider	Keycloak
Chat Gateway	Server-sent events streaming, heartbeat, resume from the turn buffer	custom service
Orchestrator	Durable step runtime: every inference and tool call is a resumable step	Temporal, custom
Tools and actions	Code sandbox, action gateway, tool connectors, per-user credential vault, inbound webhooks, metering	gVisor or Firecracker, MCP servers
Data	Postgres with pgvector, object storage per tenant prefix, Redis turn buffer, one KMS key per tenant	Aurora, S3, ElastiCache, KMS
Model router	One interface over several model providers, each pinned to a region	custom, LiteLLM
Egress gateway and NAT	Allowlisted outbound calls from static addresses in both regions	Envoy, NAT gateway
Delivery	GitOps for everything, signed images verified at admission, promotion gate	Argo CD, cosign, Kyverno
Observability and audit	Independent telemetry per region, synthetic canaries, one immutable audit trail	OpenTelemetry, Grafana, S3 Object Lock

Use cases

A vendor serving many organisations from one platform

The classic SaaS case. The provider wants one codebase, one deployment, and one on-call rotation. Customers want a product that works on day one and accept that their content is stored in the provider's account, inside an agreed region.

An internal platform team serving many business units

The same design works inside one company. Business units are tenants, the platform team is the provider, and the residency boundary is the company's own cloud region. The isolation layers matter just as much: finance should not see legal's documents because they share a cluster.

A regulated customer who needs residency but not isolation

A customer who must keep data in a specific country can be served by Pattern 1 as long as the provider runs a full regional pair inside that country. Residency is a property of where the boundary is drawn, not of who operates the system.

Design considerations

Security and privacy

Only one way in. The edge adds a rotating secret header that the gateway requires. An attacker who finds the origin's address gains nothing, because the origin refuses anything that did not come through the edge and its firewall. Admins get a separate edge path with stricter rules and an optional IP allowlist.

The gateway resolves the caller; it does not trust a bearer token. Browser sessions are opaque secrets whose hash is stored server-side. CLI tokens and API keys are resolved the same way. The identity provider's JWT is verified once, at login, and never used as a request credential again. Nothing at rest is replayable.

Tenant identity comes only from the platform. The gateway strips any tenant header the client sent and injects its own, derived from the resolved principal. Every layer below re-checks it.

Isolation in eight layers. Tenants share a cell, so isolation is layered, and a failure in one layer is caught by the next.

Layer	Per-tenant control
Identity	Own realm; tenant claim issued by the broker
Gateway	Own consumer, rate limit and quota
Cluster	Namespace with default-deny NetworkPolicy, scoped RBAC, ResourceQuota
Mesh	Authorization policy that re-checks the tenant claim on every hop
Cloud IAM	Own IAM role; no long-lived keys
Data	Policy compiled into every query, row-level security beneath it, own object prefix, own Redis ACL, own KMS key
Egress	Own destination allowlist
Model	Own token budget and context limit

Tenancy has two levels: an organisation and the workspaces inside it. Workspaces share the tenant's namespace and are separated by policy.

Authorization is decided once. One policy decision point evaluates tenant and user policy and compiles it into every query. There is no "load then filter" in application code. Row-level security sits beneath it as defence in depth, not as the policy.

Tools and actions are gated. The sandbox holds no platform credentials and has default-deny egress with an allowlist of package registries. Email, messaging and writes go through an action gateway with per-tenant policy, an approval step for irreversible actions, and an audit record for each action. Tool connectors run against a per-tenant allowlist. Per-user tokens are sealed in a credential vault the model and the sandbox never see. Inbound webhooks enter through the edge with signature verification.

Each model provider is its own boundary. The router pins each provider to a region inside the residency boundary, uses private endpoints, and keeps invocation logging off. Residency and logging are recorded per provider, never "per platform".

Keys and exit. Each tenant has one multi-region KMS key, with an optional bring-your-own-key grant. At exit the key is disabled immediately and deleted after the KMS waiting period, which makes the tenant's data unreadable. Chat content is never written to logs; gateways, collectors and log shippers strip bodies.

Reliability

A second region runs as a warm standby at minimum scale: ingress, identity, the cell with its tools layer, data replicas, its own model capacity, its own NAT addresses and its own telemetry. Failover is deliberately not automatic. A person approves, then an orchestrator promotes the database, scales the cell and gateway, and moves the anycast traffic dial.

Design targets, to be validated in failover testing: recovery time about 15 minutes; recovery point of seconds for the database and within 15 minutes for object storage; the Redis turn buffer is lost, so chats in progress restart.

The identity broker is the single login dependency for every tenant. If it is down, nobody signs in. Its state lives in a global database so it survives a regional failover, but not its own outage.

Operations

Everything is delivered by GitOps, routes included. Nothing is hand-applied, self-heal is on, and promotion to production goes through a gate. Images are signed and the signature is verified at admission.

Each region has an independent telemetry stack. Synthetic canaries probe from outside the platform through the real edge, so operators see what users see. One audit trail collects cloud API calls, cluster and gateway audit, every agent action with its tenant and user, and every break-glass session, normalised to one schema and written to immutable storage. Every line carries a credential id, never the secret.

Cost

The shared cell is the main saving: one cluster, one database cluster and one model quota serve many tenants. The main costs are the warm standby, which runs all the time, and the edge plus anycast plus gateway plus mesh stack, which is more components than any one alone. Each residency geography needs its own full regional pair, so the cost scales with the number of geographies, not the number of tenants.

Performance

The chat tier scales on active streaming connections, not CPU, because idle connections use almost none. Model capacity is shared per account per region and controlled by per-tenant token budgets. Tool output is capped so one call cannot consume the context budget.

Design alternatives

DNS-only failover instead of an anycast entry. Cheaper, but depends on resolver caches honouring TTLs. The traffic dial switches new connections in seconds.
An application load balancer instead of a layer-4 load balancer. Would terminate TLS twice and duplicate the routing the gateway already does.
A database per tenant. Removes the shared cluster key and blast radius at the cost of many more clusters. Right for a premium or regulated tier, not the default.
Trusting the identity provider's JWT on every request. Simpler, but a long-lived bearer token at rest is replayable.
A managed sandbox service. Less to operate, but it is a third party that sees content, and it must appear in the boundary table.
Automatic failover. Faster, until it fires on a false alarm.

What crosses the boundary

Fill this table in for your own design before a security review asks for it.

Crossing	Direction	What travels	Can the customer switch it off?
User traffic → edge	customer → provider	requests and streamed answers	no, it is the product; the customer controls accounts and the admin allowlist
Identity broker ↔ tenant IdP	both, sign-in only	assertions and claims, no business data	yes; disabling federation stops sign-in
Model router → model providers	provider → model providers	prompts with retrieved content	no; per-provider residency and logging terms are recorded
Sandbox and tool servers	provider → sandbox and allowlisted tools	code, files, tool inputs	yes, per tool
Action gateway → channels	provider → email, messaging, systems	the agent's actions	yes, per action type and tenant
Inbound webhooks	third parties → edge → gateway	signed callbacks	yes, by not enabling the channel
Egress → customer systems	provider → customer, from static addresses	connector calls and results	yes; remove the firewall rule or disable the connector
Per-tenant SIEM export	provider → customer	that tenant's audit events	yes, opt-in
Bring-your-own-key grant	customer → provider	permission to use the customer's key	yes; revoking makes that tenant's data unreadable

Trade-offs

Tenants share a cluster and a database cluster key. One cell's blast radius spans its tenants.
Model quota is shared per account per region, controlled by per-tenant budgets.
The identity broker is the single login dependency.
Failover is human-approved and takes about 15 minutes as a design target. Chats in progress restart.
Each residency geography needs its own regional pair.

When to choose this pattern

Choose the shared service when the customer wants the provider to run everything, accepts content stored in the provider's account inside an agreed residency boundary, and does not need an air gap. Put the effort into the eight isolation layers and the audit trail.

Choose Pattern 2 if the customer's documents, index and keys must stay in their own account under a key they can revoke. Choose Pattern 3 if nothing may leave their building, not even a prompt.

Deployment outline

Build from the inside out, so that each layer is verified before the next depends on it.

Foundation. Two regions, VPCs, the Kubernetes cell in each, the global database with its replica, object storage with cross-region replication, one KMS key per tenant.
Identity and gateway. The identity broker with one realm per tenant, federated to each tenant's identity provider. The API gateway with opaque session resolution, per-tenant consumers and the tenant-header rule.
Edge. The CDN distribution with WAF and the secret origin header, the anycast entry point with the Region B dial at zero, the separate admin distribution.
Application. Chat Gateway, Orchestrator, retrieval and connector services, the service mesh with mTLS and the tenant re-check policy, the admission controller that generates namespaces, quotas and network policies per tenant.
Tools and actions. The sandbox with default-deny egress, the action gateway, tool connectors, the per-user credential vault, webhook intake, metering.
Model router and egress. Providers pinned to regions, private endpoints, invocation logging off; the egress gateway and NAT with static addresses in both regions.
Delivery and observability. GitOps for every component, image signing and admission verification, the promotion gate; telemetry per region, synthetic canaries, the audit pipeline into immutable storage.
Failover. The DR orchestrator and runbook, then a full failover test that measures the recovery figures and replaces the design targets with results.

Detailed topology

The full design in thirteen bands, in the order a request travels. The hop numbers match the request flow table above. Open the figure at full size to read the component notes.

What's next

This is part 1 of a three-part series. The other parts are also linked in the series box at the top of this article.

Pattern 2, the control/data split (customer cloud): the customer runs the whole data plane in their own cloud account.
Pattern 3, the isolated site (on-premises and air-gapped): everything runs in the customer's building, with models on local GPUs.