Sid Probstein

Posted on Sep 12 Edited on Sep 17

Build Your Own Private LLM: A 2026 Reference Architecture for the Enterprise

Every enterprise I talk to has the same conversation twice a year. Legal will not let the contract data leave the building. Security will not approve a vendor that trains on prompts. Finance has seen the API bill and would like a word. And somebody in the room says: how hard would it be to just run our own?

Harder than a weekend. Easier than it was eighteen months ago. And the parts that are hard are not the parts people expect. The model is the easy bit. The hard bits are power, identity, and what happens when a document your model reads tells it to do something.

This is an end-to-end reference architecture for a private LLM platform: the arithmetic to size it, the commands that actually run, and the numbers that decide whether it pays back. Everything is current as of September 2026.

There is a runnable companion too: swirlai/private-llm-reference, Apache-2.0. docker compose up stands up an OAuth2 server, two MCP servers, an ACL-trimming backend and a gateway, and proves three security properties a local model server does not give you.

What "private" actually means

"Private LLM" gets used for four very different architectures, and the confusion costs people real money. Pick your posture before you pick your hardware.

Posture	Weights	Inference runs	Prompts leave your control	Typical driver
Vendor API with zero-retention terms	vendor	vendor	yes, contractually bounded	speed to value
Vendor model in your cloud tenancy (Bedrock, Foundry, Vertex)	vendor	vendor-managed, your VPC	no, but vendor code path	procurement comfort
Open weights, your cloud	yours	your VMs, your VPC	no	control plus elasticity
Open weights, your metal	yours	your data center	no	data residency, sovereignty, unit economics at scale

The last two are what this article is about. They share almost the entire software stack and differ only in who owns the depreciation schedule and who gets paged when a power supply fails.

Be honest about your driver. "We want control over our data" is real and on-prem solves it. "We want to save money" is real but only above a certain sustained utilization, and the math is in the cost section. "We want a frontier model that is also private" is a wish, not a driver.

Step 0: size the workload before you shop

The most expensive mistake is buying hardware before you can state your workload in tokens. Write these five numbers down first.

Peak concurrent requests. Not users. Concurrent in-flight generations.
Median and p95 input tokens. Retrieval-augmented prompts are big. A RAG assistant with ten retrieved chunks runs 4,000 to 12,000 input tokens routinely.
Median and p95 output tokens. A chat reply is 300. A reasoning model's hidden chain of thought can be 20,000.
Time-to-first-token target. Interactive chat wants under 500 ms. Batch summarization does not care.
Inter-token latency target. Human reading speed is roughly 5 to 8 tokens per second. Anything above 20 tokens/s per stream feels instant.

These map onto hardware differently. Prefill, processing your input, is compute bound. Decode, generating output, is memory-bandwidth bound. A cluster tuned for long-input summarization looks nothing like one tuned for short high-concurrency chat.

The heuristic that matters: decode speed is governed by HBM bandwidth, not FLOPS. Tokens per second is roughly memory bandwidth divided by bytes read per token. That is why an H200 often beats a nominally faster part with less bandwidth, and why quantization buys speed as well as capacity.

The memory arithmetic

Weights first. Bytes per parameter: BF16/FP16 = 2, FP8 = 1, FP4 (NVFP4 or MXFP4) is about 0.5 plus scale overhead.

Model size	BF16	FP8	FP4
8B	16 GB	8 GB	~4 GB
30B	60 GB	30 GB	~15 GB
70B	140 GB	70 GB	~35 GB
120B	234 GB	117 GB	~60 GB
405B	810 GB	405 GB	~203 GB
675B	1.35 TB	675 GB	~338 GB

The MoE trap. Mixture-of-experts models advertise a small active parameter count. That governs speed, not memory. Every expert stays resident, so a 2.8T-total model needs 2.8T of VRAM whatever its active count. People plan around the active number and come up short by an order of magnitude.

Now the KV cache, which is where capacity plans go to die:

kv_bytes_per_token = 2 * n_layers * n_kv_heads * head_dim * bytes_per_element

The leading 2 is for K and V. For a 70B model with grouped-query attention (8 KV heads, 80 layers, 128 head dim) at BF16 that is about 0.33 MB per token. So:

32K context: about 10.7 GB per concurrent request
128K context: about 43 GB per concurrent request

Sixteen concurrent users at 32K context is about 170 GB of KV cache, more than the weights themselves. This is why --kv-cache-dtype fp8 is close to a free win (halves it, near-lossless) and why models with multi-head latent attention (MLA) or aggressive GQA are disproportionately cheap to serve.

A quick Python sizer worth keeping around:

def vram_estimate(
    params_b: float,
    bytes_per_param: float,
    n_layers: int,
    n_kv_heads: int,
    head_dim: int,
    kv_bytes: float,
    context_len: int,
    concurrency: int,
    overhead: float = 1.15,
) -> dict[str, float]:
    """Rough VRAM budget in GB. Overhead covers activations, CUDA graphs, fragmentation."""
    weights = params_b * 1e9 * bytes_per_param / 1e9
    kv_per_token = 2 * n_layers * n_kv_heads * head_dim * kv_bytes / 1e9
    kv_total = kv_per_token * context_len * concurrency
    return {
        "weights_gb": round(weights, 1),
        "kv_gb": round(kv_total, 1),
        "total_gb": round((weights + kv_total) * overhead, 1),
    }

# 70B at FP8, FP8 KV cache, 32K context, 32 concurrent
print(vram_estimate(70, 1.0, 80, 8, 128, 1.0, 32_768, 32))
# {'weights_gb': 70.0, 'kv_gb': 171.8, 'total_gb': 278.1}

That output tells you something important: a 70B model at FP8 serving 32 concurrent users at 32K context needs roughly 278 GB. That is not one H200 and it is not two. It is four, or a shorter context window, or fewer concurrent slots. The weights were never the problem. Run this before you sign a purchase order.

Step 1: getting GPUs

Buy two nodes of 8x RTX PRO 6000 Blackwell Server Edition to start, and do not wait for prices to fall. The rest of this section is why.

Prices are going up

The constraint moved from GPU dies to high-bandwidth memory, and new fab capacity does not land in volume until 2028. DRAM rose 93 to 98 percent quarter-over-quarter in Q1 2026. Reserved cloud capacity got more expensive through H1 2026: the one-year H100 contract index went from about $1.70/GPU-hr in October 2025 to about $2.35 in March 2026, and CoreWeave raised list prices about 25 percent in July 2026. If you are waiting for a better entry point, you are waiting for a worse one.

What to buy

Tier 0, the pilot. One or two nodes of 8x RTX PRO 6000 Blackwell Server Edition. 96 GB each, PCIe Gen5, configurable to 600 W, MIG-partitionable four ways, and air-coolable in a standard chassis from Dell, HPE, Lenovo or Supermicro. About $16,000 per GPU, so roughly $130,000 of GPU per node. 768 GB per node serves 8B to 70B models, several at once.

This is the highest-probability first purchase for a mid-size enterprise, and the reason is boring: it fits your existing rack, cooling and 8 to 15 kW power budget. No liquid retrofit, no 30-week lead time, no structural review.

Tier 1, production. Two to four nodes of 8x H200 at roughly $370,000 a node, or 8x B200 at roughly $450,000. Sixteen to thirty-two GPUs, $0.75M to $1.8M. Handles a 70B dense model at high concurrency or a 671B MoE at FP8 on one node. H200 for lowest risk and best dollars per GB of HBM; B200 if you need native FP4.

Do not buy GB300 NVL72 first. 135 kW, mandatory liquid, 1.36 tons, roughly $3.7M, six to twelve month lead time, and it does not fit through a standard data center door. Rack-scale NVLink wins for trillion-parameter dense inference. It is not what you want for a 70B assistant.

Part	VRAM	TDP	Cooling	Note
RTX PRO 6000 SE	96 GB GDDR7	up to 600 W	air or liquid	best pilot part
H200 SXM	141 GB HBM3e	700 W	air	best risk-adjusted production buy
B200 SXM	192 GB HBM3e	1,000 W	air/liquid	FP4 native
B300 SXM	288 GB HBM3e	up to 1,400 W	liquid	14.5 kW/node
AMD MI350X	288 GB HBM3e	~1,000 W	air	most VRAM you can air-cool
AMD MI355X	288 GB HBM3e	1,400 W	liquid	same memory, denser
Rubin R200	288 GB HBM4	~1.8 kW	liquid	partner availability H2 2026

Used Hopper is underrated. Refurbished H100 runs $21,000 to $34,000 per GPU, used 8-GPU servers $150,000 to $180,000. Dispersion is wide, so get quotes. Hopper is not collapsing in value: CoreWeave rebooked expiring 2022-vintage contracts at 95 percent of original pricing.

AMD, in one rule

If you have a platform engineer who can read HIP and debug a container, AMD saves real money. MI300X is the cheapest HBM-class capacity available, and MI350X gives you 288 GB you can air cool, which no NVIDIA part does. If your team's first instinct on a CUDA error is to file a ticket, buy NVIDIA and spend the difference on people.

ROCm 10.0 ships production vLLM and SGLang containers and is genuinely viable for mainstream models. But decode parity with NVIDIA is still a roadmap goal, new-model support lags by weeks to months, and Hugging Face's Text Embeddings Inference is NVIDIA-only, so an AMD-only RAG stack needs a different embedding server.

Intel is not a credible primary platform for a 2026 decision. Among the alternatives, SambaNova and Tenstorrent are the only two that will genuinely sell you a box; Groq and Cerebras are cloud businesses now.

One 8-GPU node is a rack

A DGX B300 draws 14.5 kW. Typical enterprise colo racks are provisioned at 8 to 15 kW. Your single node eats the entire cabinet.

Cooling	Practical ceiling
Optimized air	30 to 40 kW/rack
Rear-door heat exchanger	60 to 80 kW/rack
Direct-to-chip liquid	60 to 120 kW/rack
Immersion	120+ kW/rack

Converting an air-cooled facility to liquid runs on the order of $2M to $3M per megawatt, with power upgrades often adding as much again. That is exactly why the Tier 0 pilot is a good first move: it is the largest useful deployment you can do without having the liquid conversation.

Three more constraints that surprise people. Colo vacancy in North America hit a record-low 1 percent, and wholesale pricing is rising, about $196/kW/month in H2 2025. Enterprises are securing capacity 18 to 24 months ahead. Lead times run 8 to 16 weeks through an OEM and 30+ weeks direct.

Networking and storage: do not overbuy

You do not need InfiniBand for an inference cluster under 64 GPUs. Inference is far less sensitive to collective latency than training.

Fabric	vs InfiniBand	Amortized 3-yr	Use when
InfiniBand NDR/XDR	100%	~$0.29/GPU/hr	64+ GPUs, real training
Spectrum-X Ethernet	85 to 90%	~$0.19/GPU/hr	mid-scale mixed
Tuned RoCEv2	70 to 80%	~$0.10/GPU/hr	inference-first

If you go RoCEv2, budget real engineering time for PFC and ECN tuning. Misconfigured, its latency becomes unpredictable and genuinely hard to debug.

One hard rule: keep tensor parallelism inside a single NVLink domain. Cross-node tensor parallelism is prohibitively slow. Go pipeline or data parallel across nodes instead.

Storage is undemanding for inference. Baseline 1 GB/s per GPU. Weights, a vector index and logs fit comfortably on one good NVMe tier. Save the parallel filesystem conversation for fine-tuning.

Buy versus rent: duty cycle decides

Naively, a $25,000 GPU at $3.00/GPU-hr pays back in about 347 days of continuous use. Add power, cooling, space and staff and it is 18+ months at near-100 percent utilization.

Here is the number that actually decides it. Cast AI measured average GPU utilization across enterprise Kubernetes clusters at roughly 5 percent. At 5 percent duty cycle, a $3.44/GPU-hr committed rate costs $68.80 per genuinely useful hour. Owned or rented, low utilization is what kills you.

So measure your duty cycle on rented capacity for a quarter before buying anything. Below 40 to 60 percent sustained, on-prem will not pay back.

Class	Neocloud on-demand	Hyperscaler on-demand
B200	$5.91 to $8.60/GPU-hr	$14.00 to $16.11
H200	$3.99 to $6.31	$7.91 to $10.85
RTX PRO 6000	$1.41 to $2.50	$3.36 to $5.50

Hyperscalers run 2 to 3x neocloud for identical silicon. Committed pricing spreads wider still: 64x B200 for a year costs about $1.93M on CoreWeave and about $9.03M on Google Cloud. Check the discount structure, because it sets your break-even: CoreWeave Reserved at 60 percent off breaks even at 40 percent utilization, while AWS Capacity Blocks at 13 percent off need 86.8 percent. Watch the line items outside the headline rate too, since cluster networking adds roughly 40 percent and hyperscalers meter egress.

Step 2: choosing and obtaining the model

Pick from the Apache-2.0 and MIT tier unless you have a reason not to, and read the actual LICENSE file at the commit you pin. Open weights are not open source, and that distinction is where legal review stalls.

What is deployable

Open weights now trade within roughly one generation of frontier closed models, and on some coding and agentic work the gap has closed. What has not closed is operational: you own the evals, the safety tuning and the incident response.

Model	Size	License	Fits on
DeepSeek V4-Pro	1.6T / 49B active	MIT	multi-node
Mistral Large 3	675B / 41B	Apache 2.0	4 to 8 GPUs at FP4
Inkling	975B / 41B	Apache 2.0	multi-node
DeepSeek V4-Flash	285B / 13B	MIT	4 to 8 GPUs
gpt-oss-120b	117B / 5.1B	Apache 2.0	1 GPU, ships in MXFP4
Mistral Small 4	119B / 6B	Apache 2.0	1 to 2 GPUs
Llama 4 Scout	109B / 17B	community	1 to 2 GPUs
Qwen3.8-27B	27B dense	Apache 2.0	1 GPU
Granite 4.2	3B / 8B / 30B	Apache 2.0	1 GPU, IBM indemnifies via watsonx
Gemma 4	to 31B	Apache 2.0	1 GPU

Two planning notes that save real time. Llama 5 does not exist. Llama 4 from April 2025 is the final Llama release; Meta moved to the closed Muse line. Content farms publish fabricated Llama 5 specifications, so do not put it on a roadmap. And most enterprises should live in the mid tier, not the frontier tier. A 30B model right-sized to the task triples your throughput against a 70B and cuts unit cost proportionally.

The three license tiers

Tier 1, unencumbered. Apache 2.0 or MIT. No thresholds, no acceptable-use policy, no gates. DeepSeek, Mistral's open lineup, gpt-oss, Gemma 4, Qwen3.8-27B, Granite, Olmo, Apertus, Inkling. If your legal team has low tolerance for novel license review, restrict the candidate list to this tier and you are done.

Tier 2, revenue or user gated. These read like MIT for three paragraphs and then add numbered conditions. The pattern is consistent and, critically, most carve out purely internal use:

Kimi K3: MaaS providers over $20M revenue in any 12 months must negotiate separately. Attribution above 100M MAU or $20M monthly revenue.
Qwen3.8-Max: attribution above 100M MAU or $20M monthly revenue; MaaS businesses above $50M trailing-twelve-month revenue need a paid license.
MiniMax: mandatory attribution, and it binds fine-tunes and distilled derivatives.
Llama 4: the 700M MAU carve-out, branding requirements, an acceptable-use policy by reference, and EU restrictions on the multimodal variants. Not OSI open source.

Tier 3, territorial exclusions. New, and the one to watch. MiniMax's H3 license excludes the United States, EU, UK and South Korea, and the exclusion covers deploying the outputs, not just running the weights. Check for this clause explicitly.

Pin every model to a commit SHA and archive the LICENSE that was in force at ingest. Hugging Face repos are mutable, and "it was MIT when we downloaded it" is only a defense if you can prove it.

Quantization policy

Pick a house standard:

FP8 is the default. Indistinguishable from BF16 for most work, half the memory, native on Hopper and Blackwell.
NVFP4 when memory forces it on Blackwell. Within about 1 percent of FP8 on MMLU-Pro and GPQA.
MXFP4 when you need portability. Runs on Blackwell and MI355X.
AWQ or GPTQ only on Ampere and Ada, where FP4 has no hardware path.

The exception that matters: both FP4 formats degrade measurably on reasoning and math. If your workload is reasoning-heavy, stay at FP8 and buy the memory. And benchmark on your own eval set, because published deltas are averages over benchmarks you do not run.

Getting weights safely

Four controls, in priority order:

Enforce safetensors at the registry layer. Pickle formats execute arbitrary code at load. Make it a policy gate, not a convention.
Ban trust_remote_code=True in production. It executes arbitrary repo-supplied Python regardless of weight format, bypassing every other control.
Sign at ingest with your own key. No registry signs automatically. Signing at your gate proves the artifact passed your review, not that somebody uploaded it.
Hash-pin to a commit SHA, never a branch.

Scanning is defense in depth, not a control: malware has been demonstrated evading pickle scanners via compression. Run ModelScan anyway, but do not rely on it.

A minimal ingest gate:

"""Ingest gate: pull a pinned revision, enforce format, sign, record."""
import hashlib, json, pathlib, subprocess
from huggingface_hub import snapshot_download

ALLOWED_LICENSES = {"apache-2.0", "mit", "openmdw-1.1"}
BANNED_SUFFIXES = {".bin", ".pt", ".pth", ".pkl", ".ckpt"}

def ingest(repo_id: str, revision: str, dest: pathlib.Path) -> dict:
    path = pathlib.Path(
        snapshot_download(
            repo_id=repo_id,
            revision=revision,            # a commit SHA, never a branch name
            local_dir=dest / repo_id.replace("/", "__"),
            allow_patterns=["*.safetensors", "*.json", "*.txt", "*.model", "LICENSE*"],
        )
    )

    bad = [p.name for p in path.rglob("*") if p.suffix in BANNED_SUFFIXES]
    if bad:
        raise RuntimeError(f"non-safetensors weight artifacts present: {bad}")

    cfg = json.loads((path / "config.json").read_text())
    if cfg.get("auto_map") or cfg.get("trust_remote_code"):
        raise RuntimeError("model declares custom code; requires manual security review")

    digests = {
        p.relative_to(path).as_posix(): hashlib.sha256(p.read_bytes()).hexdigest()
        for p in sorted(path.rglob("*.safetensors"))
    }
    manifest = {"repo_id": repo_id, "revision": revision, "sha256": digests}
    (path / "INGEST_MANIFEST.json").write_text(json.dumps(manifest, indent=2))

    # Sign the manifest with your internal key; refuse to load unsigned models at serve time.
    subprocess.run(
        ["cosign", "sign-blob", "--yes",
         "--key", "hashivault://model-signing",
         "--output-signature", str(path / "INGEST_MANIFEST.sig"),
         str(path / "INGEST_MANIFEST.json")],
        check=True,
    )
    return manifest

For restricted egress, run a proxy in a DMZ allowlisted to huggingface.co, point clients at it with HF_ENDPOINT, and set HF_HUB_OFFLINE=1 in production so a cache miss fails loudly instead of quietly reaching the internet.

For true air gap, weights run 10 to 400 GB. Move them on encrypted media or a one-way diode and treat every model update as a release event with security review and chain-of-custody logging.

One governance note: NVIDIA announced a definitive agreement to acquire Hugging Face in early September 2026. Whatever your view, single-vendor dependency on the Hub is now a procurement question. Mirror accordingly.

Step 3: serving it

vLLM is the default, and here is why

vLLM 0.29.0 (released 2026-09-09) is the pragmatic choice for enterprise serving: continuous batching, PagedAttention, prefix caching, FP8 KV cache, LoRA multiplexing, structured outputs, speculative decoding, disaggregated prefill/decode, an OpenAI-compatible API and a genuinely good Prometheus surface.

Two things that will break your existing runbooks:

The V0 engine is gone. VLLM_USE_V1 no longer exists. There is nothing to enable.
--guided-decoding-backend no longer exists. Structured outputs moved to --structured-outputs-config, taking a JSON object with backend (one of auto, xgrammar, guidance, outlines, lm-format-enforcer), plus reasoning_parser, enable_in_reasoning and related fields.

Also, python -m vllm.entrypoints.openai.api_server is deprecated in favour of vllm serve.

A production launch command:

vllm serve /models/mistral-small-4 \
  --served-model-name corp-general-v1 \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --quantization fp8 \
  --async-scheduling \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --structured-outputs-config '{"backend":"xgrammar"}' \
  --enable-lora --max-loras 4 --max-lora-rank 32 \
  --otlp-traces-endpoint http://otel-collector.observability:4317 \
  --enable-prompt-tokens-details \
  --enable-per-request-metrics \
  --ssl-certfile /certs/tls.crt --ssl-keyfile /certs/tls.key

Four of those matter most. --enable-prefix-caching is close to free money for RAG and agents, where a long system prompt repeats on every call. --enable-chunked-prefill stops a long input stalling every in-flight decode. --max-num-seqs is your concurrency ceiling and bounds KV cache pressure, so it is the knob to turn when you see preemptions. --tool-call-parser values are registry-dependent, so check your build rather than copying mine.

Speculative decoding now has flat aliases, mutually exclusive with the config-object form:

# either
  --spec-method eagle3 --spec-model /models/draft-1b --spec-tokens 5
# or
  --speculative-config '{"method":"eagle3","model":"/models/draft-1b","num_speculative_tokens":5}'

Do not rely on vLLM's --api-key for authentication. vLLM's own documentation flags API-key auth as limited. It is a speed bump for accidental access, not a control. The gateway in the next section is your authentication boundary; vLLM should be reachable only from it, over mTLS, on a network policy that denies everything else.

The other engines, and when to use them

SGLang: strong prefix reuse and constrained generation, best for complex multi-turn and agentic control flow.
TensorRT-LLM: peak throughput on NVIDIA-only fleets, at the cost of engine-build complexity. Worth it for one high-volume model, not a fleet.
llama.cpp: CPU, edge, Apple Silicon, single user.
Ollama: laptops and demos. Not a serving tier. I have seen it in production and it does not end well under concurrency.

Kubernetes: the parts that matter

GPU Operator 26.7.0 brings the DRA driver to GA, requiring Kubernetes 1.34.2 or later. It deploys via a new GPUCluster resource that cannot coexist with ClusterPolicy, so this is a migration decision, not a flag.

MIG gives hard isolation and is what you want for multi-tenant SLOs. Time-slicing gives none and is fine for dev. Neither substitutes for right-sizing a replica.

A serving deployment, trimmed to the interesting parts:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-corp-general
  namespace: inference
spec:
  replicas: 2
  selector:
    matchLabels: {app: vllm-corp-general}
  template:
    metadata:
      labels: {app: vllm-corp-general}
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      serviceAccountName: vllm-runner          # bound to a SPIFFE identity
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: vllm
          image: registry.internal/vllm-openai:v0.29.0
          args:
            - "--model=/models/mistral-small-4"
            - "--served-model-name=corp-general-v1"
            - "--tensor-parallel-size=4"
            - "--max-model-len=32768"
            - "--gpu-memory-utilization=0.90"
            - "--max-num-seqs=256"
            - "--enable-chunked-prefill"
            - "--enable-prefix-caching"
            - "--kv-cache-dtype=fp8"
            - "--async-scheduling"
            - "--otlp-traces-endpoint=http://otel-collector.observability:4317"
          env:
            - name: HF_HUB_OFFLINE            # fail loudly, never silently fetch
              value: "1"
            - name: VLLM_NO_USAGE_STATS
              value: "1"
          ports: [{containerPort: 8000, name: http}]
          resources:
            limits: {nvidia.com/gpu: 4}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: ["ALL"]}
          volumeMounts:
            - {name: models, mountPath: /models, readOnly: true}
            - {name: shm, mountPath: /dev/shm}
          startupProbe:                        # weight loading takes minutes
            httpGet: {path: /health, port: 8000}
            failureThreshold: 60
            periodSeconds: 15
          readinessProbe:
            httpGet: {path: /health, port: 8000}
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim: {claimName: model-weights, readOnly: true}
        - name: shm
          emptyDir: {medium: Memory, sizeLimit: 16Gi}

Two details people get wrong. The /dev/shm volume is required for tensor parallelism; the default 64 MB will hang you. And the startup probe needs a long failure threshold, because loading 70 GB of weights off network storage is not fast.

Autoscaling: scale on queue depth, not GPU utilization. GPU utilization is a terrible signal for LLM serving because a fully-batched engine sits at 100 percent whether it is keeping up or drowning. The canonical signal is vLLM's own vllm:num_requests_waiting:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-corp-general
  namespace: inference
spec:
  scaleTargetRef: {name: vllm-corp-general}
  minReplicaCount: 2
  maxReplicaCount: 8
  pollingInterval: 15
  cooldownPeriod: 600          # weight loading is expensive; do not thrash
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.observability:9090
        # KEDA's Prometheus scaler treats threshold as a per-replica average,
        # so this targets ~3 queued requests per replica.
        query: sum(vllm:num_requests_waiting{model_name="corp-general-v1"})
        threshold: "3"

vLLM exposes roughly forty Prometheus metrics. The ones worth dashboards on day one: vllm:num_requests_waiting (autoscaling), vllm:kv_cache_usage_perc (capacity headroom), vllm:num_preemptions (you are oversubscribed), vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds (your user-facing SLOs), and the prefix-cache hit ratio (your caching ROI).

When you outgrow single-node replicas, llm-d and NVIDIA Dynamo give you disaggregated prefill/decode with KV-cache-aware routing, both on the Gateway API Inference Extension.

Do not start here. Single-node replicas behind a gateway carry you a long way, and disaggregated serving adds a lot of operational surface.

Step 4: wrapping it with APIs

The gateway is the product, not the model server. It is what your developers integrate with, what your security team audits, and what lets you swap the model underneath without a migration project.

vLLM speaks the OpenAI protocol, so it is tempting to hand teams that endpoint and call it done. A raw inference endpoint has no notion of who is calling, no budget, no audit trail, no failover, and no way to deprecate a model without breaking every consumer.

What the gateway tier must do

Authenticate the human or workload. OIDC for people, workload identity for services. Never a shared API key in a config file.
Authorize per model. Not everyone gets the 675B model or the internet-connected tools.
Meter and cap. Per-user, per-team, per-application budgets with hard stops. Unbounded consumption is now OWASP LLM06:2026 and it is a real availability risk, not a billing annoyance.
Route and fail over. Model aliases (corp-general rather than mistral-small-4-fp8-tp4) so you can move the underlying model without touching consumers.
Redact and inspect. PII detection inbound, guardrail classification both ways.
Emit a complete audit record. Who, what model, what prompt hash, what tools, what cost, what trace ID.
Cache. Exact-match and semantic caching, with a scope key that includes identity so you never serve one user's answer to another.

The options

Product	Version	Self-hostable	Note
LiteLLM	1.100.1 (2026-09-10)	MIT, fully	pragmatic default; virtual keys, budgets, 100+ providers
Agent Router (formerly Envoy AI Gateway)	1.1.0 (2026-08-21)	yes, on Envoy Gateway	includes a full MCP gateway with OAuth
Kong AI Gateway	3.14 (2026-04-14)	OSS core, most AI plugins enterprise	shipped RFC 8693 token exchange
Apache APISIX	current	Apache-2.0, yes	`ai-proxy`, `ai-rate-limiting`, MCP plugins
Cloudflare AI Gateway	n/a	no, SaaS only	disqualified for a private deployment

Note the rename: Envoy AI Gateway is now "Agent Router" under the Agentic AI Foundation. Same code, same maintainers, new domain. Any config or doc referencing the old name is stale.

Start with LiteLLM for the control plane and add Kong or Agent Router at the edge when you need token exchange and MCP brokering. A minimal config:

model_list:
  - model_name: corp-general              # the alias your developers code against
    litellm_params:
      model: hosted_vllm/corp-general-v1
      api_base: https://vllm-corp-general.inference.svc:8000/v1
      api_key: os.environ/VLLM_INTERNAL_KEY
    model_info:
      max_input_tokens: 32768
      supports_function_calling: true

  - model_name: corp-general              # second replica pool, same alias
    litellm_params:
      model: hosted_vllm/corp-general-v1
      api_base: https://vllm-corp-general-b.inference.svc:8000/v1
      api_key: os.environ/VLLM_INTERNAL_KEY

  - model_name: corp-reasoning
    litellm_params:
      model: hosted_vllm/deepseek-v4-flash
      api_base: https://vllm-reasoning.inference.svc:8000/v1
      api_key: os.environ/VLLM_INTERNAL_KEY

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  allowed_fails: 3
  cooldown_time: 30
  fallbacks:
    - corp-reasoning: ["corp-general"]

litellm_settings:
  drop_params: true
  set_verbose: false
  callbacks: ["otel", "langfuse"]
  redact_user_api_key_info: true
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    ttl: 3600
    # scope cache entries by caller so answers never cross a tenant boundary
    supported_call_types: ["acompletion", "atext_completion"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  enforce_user_param: true                # every request must carry an end-user id
  max_budget: 25000                       # platform-wide monthly ceiling, USD
  budget_duration: 30d
  alerting: ["slack"]

Then issue scoped virtual keys per team rather than distributing the master key:

curl -sS -X POST https://llm-gateway.internal/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "team_id": "contracts-team",
        "models": ["corp-general"],
        "max_budget": 500,
        "budget_duration": "30d",
        "rpm_limit": 120,
        "tpm_limit": 400000,
        "metadata": {"owner": "legal-ops", "data_class": "confidential"}
      }'

Identity: the part that is usually wrong

Here is the failure mode I see most often. A user authenticates to a chat UI. The UI holds a service account key for the gateway. The gateway holds a service account key for the tool backends. By the time a tool executes a query, the identity of the human who asked has evaporated, and every user has the union of every permission the service account holds.

That is a confused deputy, and in an agentic system it is how a prompt-injected document reads someone else's payroll data.

The correct pattern has three layers.

Layer 1, workload identity. SPIFFE/SPIRE issues an identity to every pod, and mTLS authenticates by that identity rather than by network position. This answers "which service is calling." It does not carry the human.

Layer 2, user identity propagation. The user's OIDC token must not be forwarded verbatim downstream. The gateway exchanges it:

User authenticates to the gateway via OIDC. The gateway now holds a token audienced to the gateway.
Gateway performs an RFC 8693 token exchange at the authorization server: presents the user token as subject_token, sets resource (RFC 8707) to the canonical URI of the specific downstream service, and narrows scope to the minimum this call needs.
The AS returns a token whose aud is that one service, carrying the user as sub and the gateway as act (the actor claim, which is the on-behalf-of signal).
Gateway calls the service with that token, over mTLS.
The service validates signature, audience against its own resource identifier, scope, and expiry, and rejects anything not minted for it.

Kong AI Gateway 3.14 productized step 2.

Layer 3, secrets. Vault, now on 2.x, for provider keys and database credentials, with SPIFFE-authenticated login so no bootstrap secret sits on disk.

Audit. Propagate W3C trace context end to end and record sub plus act at every hop, so a tool invocation is attributable to a human rather than to a service account. This is the artifact your auditors will actually ask for.

Step 5: security

There is no reliable prompt-level defense against prompt injection. Stop looking for one and put the controls where the document cannot reach.

The threat model

Two OWASP lists govern this now. The GenAI LLM Top 10, 2026 edition:

Code	Risk	Movement
LLM01	Prompt Injection	unchanged at #1
LLM02	Sensitive Information Disclosure	unchanged at #2
LLM03	Excessive Agency	up from #6, biggest mover
LLM04	Supply Chain	down from #3
LLM05	Data and Model Poisoning	down from #4
LLM06	Unbounded Consumption	up from #10
LLM07	Misinformation	up from #9
LLM08	Hidden Context Exposure	renamed from System Prompt Leakage
LLM09	Vector and Embedding Weaknesses	down from #8
LLM10	Improper Output Handling	down from #5

Two movements matter. Excessive Agency went from sixth to third, which is the agentic era in one data point. And LLM08 was broadened: hidden context now covers any non-user-facing content assembled into the prompt, explicitly including retrieved policy text and tool schemas. If your retrieval layer injects internal policy documents, that is in scope.

The companion OWASP Top 10 for Agentic Applications covers goal hijack, tool misuse, privilege abuse, memory poisoning and rogue agents. If you are wiring tools to a model, read it.

What actually works

Instruction hierarchies, delimiters and "ignore any instructions in the following document" all fail under adaptive attack. Any vendor claiming their model is injection-resistant is selling a probability, not a control.

The principle: never let untrusted content influence which tool runs or what arguments it gets. The strongest published direction is CaMeL (arXiv:2503.18813), which extracts a plan from the trusted user query only and executes it under a capability system, so tool output can populate data but never redirect control. That is a provable property rather than a filter.

What you can build today:

Deterministic allowlists on tool arguments. The model proposes; non-model code validates. If the model asks for customer 12345, check the authenticated user is entitled to customer 12345 before the call runs.
Treat all tool output as untrusted input on the way back. Everyone skips this. A web page, a ticket comment, a PDF in a shared drive: all attacker-controlled.
Egress filtering. Exfiltration is the payoff for most injections. If your servers cannot reach arbitrary hosts, a successful injection has nowhere to send the data. This blunts a large share of real attacks and costs you a NetworkPolicy.
Human in the loop for irreversible actions. Mail, money, deletions, production writes.
Dual-LLM. A privileged planner that never sees untrusted text, and a quarantined model that reads it but returns only structured data.

Guardrail models

Model	License	Best at
Granite Guardian 4.1-8b	Apache-2.0	general safety plus RAG hallucination checks
Qwen3Guard Gen and Stream	Apache-2.0	Stream classifies token-by-token, for live interception
Llama Guard 4-12B	Llama Community	MLCommons 14-hazard taxonomy, multimodal
Llama Prompt Guard 2	Llama Community	injection and jailbreak classification specifically
NeMo Guardrails	Apache-2.0	framework, not a model

Granite Guardian and Qwen3Guard are Apache-2.0 and the safest legally. Llama Guard and ShieldGemma carry acceptable-use terms; review before shipping them in a product.

A gateway-side hook:

"""Guardrail middleware: classify in, classify out, fail closed on the way in."""
import httpx
from dataclasses import dataclass

GUARD_URL = "http://vllm-guard.inference.svc:8000/v1/chat/completions"
BLOCK_ON_INPUT = {"jailbreak", "pii_exfiltration", "violent_crimes", "code_interpreter_abuse"}
BLOCK_ON_OUTPUT = {"pii_leak", "credential_leak", "violent_crimes"}

@dataclass
class Verdict:
    safe: bool
    categories: list[str]

async def classify(client: httpx.AsyncClient, text: str, role: str) -> Verdict:
    resp = await client.post(
        GUARD_URL,
        json={
            "model": "granite-guardian-4.1-8b",
            "messages": [{"role": role, "content": text}],
            "max_tokens": 16,
            "temperature": 0.0,
        },
        timeout=5.0,
    )
    body = resp.json()["choices"][0]["message"]["content"].strip().lower()
    if body.startswith("safe"):
        return Verdict(True, [])
    return Verdict(False, [c.strip() for c in body.split("\n")[1:] if c.strip()])

async def guarded_completion(client, gateway, request, user_id, trace_id):
    inbound = await classify(client, request["messages"][-1]["content"], "user")
    if not inbound.safe and set(inbound.categories) & BLOCK_ON_INPUT:
        audit(trace_id, user_id, "blocked_input", inbound.categories)
        raise PermissionError(f"request blocked: {inbound.categories}")

    result = await gateway.completion(**request)
    text = result["choices"][0]["message"]["content"]

    outbound = await classify(client, text, "assistant")
    if not outbound.safe and set(outbound.categories) & BLOCK_ON_OUTPUT:
        audit(trace_id, user_id, "blocked_output", outbound.categories)
        return refusal("I can't share that. Withheld by policy.")

    audit(trace_id, user_id, "allowed", [])
    return result

Three notes. Run the guard on a separate small replica so a guard timeout cannot take down your serving pool. Decide explicitly whether a timeout fails open or closed: inbound closed, outbound streaming usually open with async flagging, because blocking mid-token is a terrible experience. And log every verdict with the trace ID, because the false-positive rate decides whether people route around your platform.

The data boundary

You built this to keep data in. Then somebody enables verbose request logging and every prompt lands in a log aggregator half the company can query.

Log prompt hashes and token counts by default, not prompt text. Make full capture an explicit, time-boxed, separately authorized debugging mode.
Set trace retention independently from metrics. Thirty days is usually plenty and shrinks exposure sharply.
Scope caches by identity. A semantic cache keyed only on prompt similarity will serve one user's answer to another.
Give the vector index the same data classification as its source documents. Embeddings are not anonymized; inversion attacks recover meaningful text.
Decide up front whether you retain prompts for fine-tuning. If yes, that is a separate consent and retention regime, and it belongs in your privacy notice before you collect anything.

Step 6: MCP, or how the model reaches your systems

A private LLM that cannot see your data is expensive autocomplete. MCP is how you connect it. If your mental model of MCP is from 2025, four things will break your code.

What changed on 2026-07-28

Sessions are gone, including Mcp-Session-Id. Servers are stateless and scale horizontally without sticky sessions, which is a real operational win. Cross-call state now uses server-minted handles passed as ordinary tool arguments.
The initialize handshake is gone. Every request carries its protocol version in _meta, and servers must implement a new server/discover RPC.
Server-initiated requests are gone. Instead the server returns an InputRequiredResult and the client retries with inputResponses. Every result now carries a resultType.
SSE resumability was removed. A broken stream loses the in-flight request, so make your tools idempotent or safely retryable.

Also deprecated: Roots, Sampling, Logging, the HTTP+SSE transport, and OAuth Dynamic Client Registration. Streamable HTTP is the transport; stdio stays for local tools.

The Python SDK went to v2

FastMCP no longer exists. The module is a tombstone that raises on import. The class is mcp.server.MCPServer. Pin mcp>=1.28,<2 if you are not ready. The TypeScript SDK is still on 1.x, so do not assume parity.

Here is the security-critical core of a server. The full working version is in the companion repo:

from mcp.server import MCPServer
from mcp.server.auth.settings import AuthSettings
from mcp.server.auth.middleware.auth_context import get_access_token

mcp = MCPServer(
    name="corp-contracts",
    token_verifier=IntrospectingVerifier(...),     # sets AccessToken.resource from the token aud
    auth=AuthSettings(
        issuer_url=ISSUER_URL,
        resource_server_url=RESOURCE_URL,          # this server's RFC 8707 identifier
        required_scopes=["contracts.read"],
        validate_token_resource=True,              # refuse tokens minted for anything else
        identity_assertion_enabled=True,           # SEP-990 enterprise IdP flow
    ),
)

@mcp.tool(annotations=ToolAnnotations(readOnlyHint=True))
async def search_contracts(query: str, limit: int = 10) -> SearchResult:
    token = get_access_token()
    if token is None or token.subject is None:
        raise PermissionError("no authenticated principal on this request")

    # Query the backend AS THE USER. This server holds no superuser credential,
    # so a prompt injection that reaches this tool has nothing to steal.
    async with httpx.AsyncClient(verify=CA) as http:
        r = await http.post(BACKEND, json={"q": query, "limit": limit},
                            headers={"Authorization": f"Bearer {token.token}",
                                     "X-On-Behalf-Of": token.subject})
    data = r.json()
    return SearchResult(hits=[ContractHit(**h) for h in data["hits"]],
                        total_matched=data["total"],
                        truncated_by_permission=data["denied_count"])

Four things carry the security here:

validate_token_resource=True is the control. Without it, a token minted for the ticketing server is happily accepted by the contracts server. In SDK 2.x, leaving it unset warns and behaves as False, so set it explicitly.
The tool calls the backend with the user's token. If this server held a superuser key, an injection reaching this tool would have that key's full authority.
truncated_by_permission returns a count, never content. Telling the model "4 more exist that you cannot see" is useful. Returning them is a breach.
ToolAnnotations are advisory. readOnlyHint lets a client auto-approve. It does not stop your code writing. Enforce in the handler.

MCP security, concretely

The spec's hard rule:

MCP servers MUST NOT accept any tokens that were not explicitly issued for the MCP server.

That is worth watching rather than reading. Below, a token minted for the contracts server is replayed against the tickets server and refused with a 401 and a real WWW-Authenticate challenge. Then validate_token_resource is turned off and the identical replay succeeds:

The full stack that produced that clip is public. Five services, two MCP servers, the policy layer and the property tests: github.com/swirlai/private-llm-reference. docker compose up, then make demo, and all three security properties run on your machine in about a minute.

No model runs in that recording. A prompt cannot stop a token being replayed. A one-line audienceGive m check can, and it lives in the resource server, not the model and not the client.

The operational controls beyond the spec:

Pin and hash tool definitions. A rug pull is a server changing a tool's description after approval. Alert on drift.
Render full tool descriptions to whoever approves them. Tool poisoning hides instructions in description text the approver never sees.
Never auto-trust one server's output as instructions. Tool results are data.
Isolate credentials per server, so a compromised server cannot borrow another's authority.
Run an internal MCP registry. The official one is in preview, does not support private servers, and is not designed for self-hosting. Run your own implementing its OpenAPI interface, and make it the only place agents discover tools.

For enterprise identity, SEP-990 (the Identity Assertion JWT Authorization Grant) is the flow you want. It carries a signed assertion from your IdP and exchanges it for an MCP access token. Its trust model is inverted from ordinary MCP OAuth: the authorization server is configuration, not discovery. No metadata fetch, no dynamic registration, no server-driven scope selection, which eliminates the mix-up and confused-deputy classes at the root rather than defending against them.

Step 7: connecting it to your data

Most of the value in a private LLM comes from retrieval, not generation. Two things determine whether this works.

Permission-trimmed retrieval. The vector index must not be a permission bypass. Either filter at query time on the caller's groups, which is faster but needs your ACLs denormalized into the index, or retrieve candidates and re-check entitlement against the source system, which is slower and always correct. Either way, the check happens before text enters the context window. Once a document is in the prompt, it is disclosed.

Embeddings and reranking. Current sensible defaults, all self-hostable:

Purpose	Model	License	Note
Embedding, general	Qwen3-Embedding-0.6B / 4B / 8B	Apache-2.0	tops MTEB multilingual; Matryoshka dims 32 to 4096; 32K context
Embedding, hybrid	BGE-M3	MIT	dense + sparse + multi-vector in one pass
Embedding, long docs	Jina Embeddings v4	Apache-2.0	128K context
Reranking, default	BGE-Reranker-v2-m3	Apache-2.0	568M, ~35 ms/pair
Reranking, quality	Qwen3-Reranker-8B	Apache-2.0	~77% BEIR nDCG@10
Reranking, latency	ms-marco-MiniLM-L-12-v2	Apache-2.0	33M, under 5 ms/pair

Avoid NV-Embed-v2 and Jina ColBERT v2 in commercial deployments: both are CC-BY-NC. And verify every license on the model card rather than in a comparison article, because this specific category is the most consistently misreported.

Two notes. Embedding models turn over far slower than LLMs, so standardize on one and leave it alone, because changing it means reindexing everything. And a reranker is the highest return per dollar in a RAG stack. Adding one usually beats upgrading your generator.

Step 8: observability, evals, and the acceptance gate

Instrumentation

Instrument to the OpenTelemetry GenAI conventions. They are still marked Development, so expect the attribute names to change under you.

Emit gen_ai.client.token.usage, gen_ai.server.time_to_first_token, gen_ai.server.time_per_output_token and gen_ai.execute_tool.duration. Langfuse and Arize Phoenix are both self-hostable and both fine; pick one.

The thing that matters more than the tool: ship one trace ID from the gateway through to the tool call, so a single identifier ties a user complaint to a prompt, a model version, a retrieval set and a tool invocation.

Evals are the acceptance gate

This is the discipline that separates a platform from a demo. You cannot upgrade a model you cannot evaluate. Without an eval suite, every model change is a leap of faith, and the practical consequence is that you never upgrade, which means you paid for control and got stagnation.

Build three layers:

Capability baselines. lm-eval for academic benchmarks. Useful for sanity, not for your business.
Task evals on your data. 200 to 500 real labeled examples from your actual workload, run in CI with promptfoo or DeepEval. This is the one that matters and the one people skip, because building it is unglamorous.
Safety evals. A red-team corpus of injection attempts and permission-boundary probes, run on every model or prompt change.

Wire them into a pipeline that gates promotion:

# .gitlab-ci.yml (or equivalent)
eval:
  stage: verify
  script:
    - promptfoo eval -c evals/task-suite.yaml --output results.json
    - promptfoo eval -c evals/injection-suite.yaml --output redteam.json
    - python scripts/gate.py results.json redteam.json
  rules:
    - changes: [ "config/models.yaml", "prompts/**/*", "tools/**/*" ]

# scripts/gate.py: fail the build, not the users
import json, sys

results, redteam = (json.load(open(p)) for p in sys.argv[1:3])

task_pass = results["stats"]["successes"] / max(results["stats"]["total"], 1)
injection_block = redteam["stats"]["successes"] / max(redteam["stats"]["total"], 1)

FAILURES = []
if task_pass < 0.92:
    FAILURES.append(f"task pass rate {task_pass:.1%} below 92% floor")
if injection_block < 1.00:
    FAILURES.append(f"injection suite blocked only {injection_block:.1%}; must be 100%")

if FAILURES:
    print("\n".join(f"BLOCKED: {f}" for f in FAILURES))
    sys.exit(1)
print(f"PASS  task={task_pass:.1%}  injection_blocked={injection_block:.1%}")

The injection suite threshold is 100 percent deliberately. Task quality is a tradeoff curve. A permission boundary is not.

Load testing

Benchmark before you promise an SLO. GuideLLM drives load at your actual token distributions, which is what synthetic benchmarks get wrong. Measure at target concurrency, not at concurrency 1, and set --max-num-seqs from what you measured rather than what you hoped.

Step 9: governance you cannot skip

EU AI Act, as it actually stands in September 2026

The timeline moved. The Digital Omnibus on AI deferred the high-risk deadlines; it did not cancel them. Current state:

Date	Obligation	Status
2025-02-02	Prohibited practices, AI literacy	in force
2025-08-02	GPAI model obligations	in force
2026-08-02	AI Office enforcement powers over GPAI active	in force, just started
2026-08-02	Article 50 transparency (synthetic content marking)	in force, grace period for pre-existing systems
2026-12-02	Article 50 grace period ends	upcoming
2027-08-02	Pre-2025-08-02 GPAI models must be compliant	upcoming
2027-12-02	Chapter III high-risk, Art. 6(2) / Annex III	deferred from Aug 2026
2028-08-02	Chapter III high-risk, Art. 6(1) / Annex I	deferred from Aug 2027

If you deploy rather than develop GPAI models, your near-term exposure is Article 50 marking plus AI-literacy duties. High-risk obligations are fifteen months out. That is breathing room, not a pass: deciding whether your system is high-risk under Annex III takes longer than people expect, so start now.

The rest

NIST AI RMF plus its Generative AI Profile is voluntary, and the most useful structure I have found for organizing the work.
ISO/IEC 42001 is the certifiable counterpart. If you sell to enterprises, expect it in questionnaires.
Keep a model inventory: every model in production with source, revision SHA, license as of ingest, quantization, eval results and owner. Building it after the fact is miserable.

The cost model, honestly

Here is a worked example for the Tier 1 cluster: two nodes of 8x H200, sixteen GPUs, colocated.

Line	Annual
Capex amortized (2 nodes ~$370k each + ~$150k network/storage/racks, 3-year straight line)	~$297,000
Colocation (space, power, cooling at ~$196/kW/month for ~28.6 kW IT load)	~$67,000
Platform engineering (0.5 FTE, fully loaded)	~$110,000
Subtotal	~$474,000
Optional: NVIDIA AI Enterprise ($4,500/GPU/yr, if you need NGC/NIM support)	+$72,000

That is 140,160 GPU-hours a year, so roughly $3.38 per GPU-hour all-in. Compare against committed neocloud H200 at roughly $2.75 to $3.50/GPU-hr and hyperscaler on-demand at $7.91 to $10.85.

Read that carefully, because it is the conclusion most vendor content will not give you: at this scale, on-prem is roughly at parity with committed neocloud capacity. It is decisively cheaper than hyperscaler on-demand. It is not a cost play against a well-negotiated neocloud contract.

Now translate to tokens, which is what your finance team actually wants:

cost_per_million_output_tokens = annual_cost / (throughput_tok_s * duty_cycle * 31.536)

At a measured 5,000 output tokens/second aggregate across both nodes:

Duty cycle	Cost per million output tokens
20%	$15.03
40%	$7.51
70%	$4.29
90%	$3.34

Duty cycle dominates everything else in this model. It swings your unit cost by more than 4x, which is more than any hardware choice, model choice or quantization decision available to you. Recall the Cast AI measurement of roughly 5 percent average GPU utilization across enterprise Kubernetes clusters. If that is where you land, none of this pencils.

So be clear-eyed about what the number means. Private inference at sixteen GPUs is not cheaper per token than a commodity open-model token vendor, which will sell you the same model for well under a dollar per million tokens. It is substantially cheaper than frontier closed-model API pricing at volume. And it is the only option when the data genuinely cannot leave. Also note that the same hardware running a 30B model rather than a 70B roughly triples throughput and cuts unit cost proportionally, so right-sizing the model to the task is the largest cost lever you control after utilization.

Build this because you need control. Treat cost parity as a pleasant side effect at sufficient scale, not as the business case.

A 90-day plan that works

Days 1 to 30, prove the workload.
Rent. Do not buy anything. Stand up vLLM on a rented 8-GPU node, put LiteLLM in front of it, wire OIDC, and give it to one real team with one real use case. Build the 200-example eval set from their actual work. Instrument everything. At the end of thirty days you should know your token distributions, your peak concurrency and your duty cycle, which are exactly the numbers Step 0 asked for and that nobody can guess.

Days 31 to 60, harden.
Add the guardrail model and the audit trail. Implement token exchange so user identity reaches the tools. Build the first MCP server against a system that matters, with validate_token_resource=True and per-user permission trimming. Write the injection eval suite and make the CI gate real. Run GuideLLM at three times your observed peak and find where it breaks.

Days 61 to 90, decide.
You now have a working platform on rented capacity and a quarter of utilization data. Run the buy-versus-rent math with your real duty cycle. If you clear 40 to 60 percent sustained and have a residency requirement, order the Tier 0 or Tier 1 hardware and start the colo and cooling conversation, remembering the 8-to-16-week OEM lead time. If you do not, sign a committed contract and revisit in two quarters. Either way you have a platform in production, which is more than most of these projects achieve in a year.

What I would skip

Fine-tuning, at first. Almost every "we need a fine-tune" turns out to be a retrieval problem or a prompt problem. Exhaust both before you build a training pipeline, because a fine-tune is a permanent maintenance obligation attached to a model version you will want to change.
Building your own gateway. LiteLLM is MIT-licensed and does more than your first three sprints would.
A vector database evaluation project. Pick one, index, ship, measure. Retrieval quality comes overwhelmingly from chunking, hybrid search and reranking, not from the store.
Rack-scale NVLink systems on day one. Covered above, but it bears repeating because somebody in the room will want one.
Multi-model routing before you have evals. Routing between models you cannot compare is a way to make quality unmeasurable.
Agent frameworks, for a while. A gateway, a well-scoped MCP server and a loop you wrote yourself will outperform a framework you do not understand, and the security properties will be ones you can actually reason about.

Where this leaves you

The stack is stable enough to build on: vLLM behind a gateway, open weights under a license your counsel has read, identity that survives the whole call path, MCP tools with audience-bound tokens, and evals that gate promotion. None of it is exotic in 2026, and all of it is achievable in a quarter with two good engineers.

The parts that will actually determine whether you succeed are not technical. Duty cycle decides your economics. Power and cooling decide your timeline. And the discipline of building evals before you need them decides whether you can ever upgrade the model, which decides whether the platform is an asset or a monument.

Two caveats on the numbers. NVIDIA and AMD publish no list prices, so treat every capex figure here as an indication and get a quote. And verify a model's license on its card, at the commit you pin, not in any table including mine.

References

Primary sources for the claims that decide something, so you can check them as they change.

Companion code
swirlai/private-llm-reference

Serving and Kubernetes
vLLM v0.29.0 ·
NVIDIA GPU Operator release notes ·
Kubernetes 1.34 DRA GA ·
llm-d

Gateways and identity
LiteLLM ·
Agent Router, formerly Envoy AI Gateway ·
Kong AI Gateway 3.14 ·
SPIRE

MCP
Spec 2026-07-28 changelog ·
Authorization spec ·
Security best practices ·
Enterprise-Managed Authorization, SEP-990 ·
Python SDK

Security and governance
OWASP GenAI LLM Top 10 2026 ·
OWASP Top 10 for Agentic Applications ·
CaMeL ·
EU AI Act timeline ·
Sigstore model-transparency

Model licenses change without notice. Read the LICENSE file at the exact commit you pin, including for anything named above.