DEV Community: Sid Probstein

Build Your Own Private LLM: A 2026 Reference Architecture for the Enterprise

Sid Probstein — Sat, 12 Sep 2026 13:32:50 +0000

Every enterprise I talk to has the same conversation twice a year. Legal will not let the contract data leave the building. Security will not approve a vendor that trains on prompts. Finance has seen the API bill and would like a word. And somebody in the room says: how hard would it be to just run our own?

Harder than a weekend. Easier than it was eighteen months ago. And the parts that are hard are not the parts people expect. The model is the easy bit. The hard bits are power, identity, and what happens when a document your model reads tells it to do something.

This is an end-to-end reference architecture for a private LLM platform: the arithmetic to size it, the commands that actually run, and the numbers that decide whether it pays back. Everything is current as of September 2026.

There is a runnable companion too: swirlai/private-llm-reference, Apache-2.0. docker compose up stands up an OAuth2 server, two MCP servers, an ACL-trimming backend and a gateway, and proves three security properties a local model server does not give you.

What "private" actually means

"Private LLM" gets used for four very different architectures, and the confusion costs people real money. Pick your posture before you pick your hardware.

Posture	Weights	Inference runs	Prompts leave your control	Typical driver
Vendor API with zero-retention terms	vendor	vendor	yes, contractually bounded	speed to value
Vendor model in your cloud tenancy (Bedrock, Foundry, Vertex)	vendor	vendor-managed, your VPC	no, but vendor code path	procurement comfort
Open weights, your cloud	yours	your VMs, your VPC	no	control plus elasticity
Open weights, your metal	yours	your data center	no	data residency, sovereignty, unit economics at scale

The last two are what this article is about. They share almost the entire software stack and differ only in who owns the depreciation schedule and who gets paged when a power supply fails.

Be honest about your driver. "We want control over our data" is real and on-prem solves it. "We want to save money" is real but only above a certain sustained utilization, and the math is in the cost section. "We want a frontier model that is also private" is a wish, not a driver.

Step 0: size the workload before you shop

The most expensive mistake is buying hardware before you can state your workload in tokens. Write these five numbers down first.

Peak concurrent requests. Not users. Concurrent in-flight generations.
Median and p95 input tokens. Retrieval-augmented prompts are big. A RAG assistant with ten retrieved chunks runs 4,000 to 12,000 input tokens routinely.
Median and p95 output tokens. A chat reply is 300. A reasoning model's hidden chain of thought can be 20,000.
Time-to-first-token target. Interactive chat wants under 500 ms. Batch summarization does not care.
Inter-token latency target. Human reading speed is roughly 5 to 8 tokens per second. Anything above 20 tokens/s per stream feels instant.

These map onto hardware differently. Prefill, processing your input, is compute bound. Decode, generating output, is memory-bandwidth bound. A cluster tuned for long-input summarization looks nothing like one tuned for short high-concurrency chat.

The heuristic that matters: decode speed is governed by HBM bandwidth, not FLOPS. Tokens per second is roughly memory bandwidth divided by bytes read per token. That is why an H200 often beats a nominally faster part with less bandwidth, and why quantization buys speed as well as capacity.

The memory arithmetic

Weights first. Bytes per parameter: BF16/FP16 = 2, FP8 = 1, FP4 (NVFP4 or MXFP4) is about 0.5 plus scale overhead.

Model size	BF16	FP8	FP4
8B	16 GB	8 GB	~4 GB
30B	60 GB	30 GB	~15 GB
70B	140 GB	70 GB	~35 GB
120B	234 GB	117 GB	~60 GB
405B	810 GB	405 GB	~203 GB
675B	1.35 TB	675 GB	~338 GB

The MoE trap. Mixture-of-experts models advertise a small active parameter count. That governs speed, not memory. Every expert stays resident, so a 2.8T-total model needs 2.8T of VRAM whatever its active count. People plan around the active number and come up short by an order of magnitude.

Now the KV cache, which is where capacity plans go to die:

kv_bytes_per_token = 2 * n_layers * n_kv_heads * head_dim * bytes_per_element

The leading 2 is for K and V. For a 70B model with grouped-query attention (8 KV heads, 80 layers, 128 head dim) at BF16 that is about 0.33 MB per token. So:

32K context: about 10.7 GB per concurrent request
128K context: about 43 GB per concurrent request

Sixteen concurrent users at 32K context is about 170 GB of KV cache, more than the weights themselves. This is why --kv-cache-dtype fp8 is close to a free win (halves it, near-lossless) and why models with multi-head latent attention (MLA) or aggressive GQA are disproportionately cheap to serve.

A quick Python sizer worth keeping around:

def vram_estimate(
    params_b: float,
    bytes_per_param: float,
    n_layers: int,
    n_kv_heads: int,
    head_dim: int,
    kv_bytes: float,
    context_len: int,
    concurrency: int,
    overhead: float = 1.15,
) -> dict[str, float]:
    """Rough VRAM budget in GB. Overhead covers activations, CUDA graphs, fragmentation."""
    weights = params_b * 1e9 * bytes_per_param / 1e9
    kv_per_token = 2 * n_layers * n_kv_heads * head_dim * kv_bytes / 1e9
    kv_total = kv_per_token * context_len * concurrency
    return {
        "weights_gb": round(weights, 1),
        "kv_gb": round(kv_total, 1),
        "total_gb": round((weights + kv_total) * overhead, 1),
    }

# 70B at FP8, FP8 KV cache, 32K context, 32 concurrent
print(vram_estimate(70, 1.0, 80, 8, 128, 1.0, 32_768, 32))
# {'weights_gb': 70.0, 'kv_gb': 171.8, 'total_gb': 278.1}

That output tells you something important: a 70B model at FP8 serving 32 concurrent users at 32K context needs roughly 278 GB. That is not one H200 and it is not two. It is four, or a shorter context window, or fewer concurrent slots. The weights were never the problem. Run this before you sign a purchase order.

Step 1: getting GPUs

Buy two nodes of 8x RTX PRO 6000 Blackwell Server Edition to start, and do not wait for prices to fall. The rest of this section is why.

Prices are going up

The constraint moved from GPU dies to high-bandwidth memory, and new fab capacity does not land in volume until 2028. DRAM rose 93 to 98 percent quarter-over-quarter in Q1 2026. Reserved cloud capacity got more expensive through H1 2026: the one-year H100 contract index went from about $1.70/GPU-hr in October 2025 to about $2.35 in March 2026, and CoreWeave raised list prices about 25 percent in July 2026. If you are waiting for a better entry point, you are waiting for a worse one.

What to buy

Tier 0, the pilot. One or two nodes of 8x RTX PRO 6000 Blackwell Server Edition. 96 GB each, PCIe Gen5, configurable to 600 W, MIG-partitionable four ways, and air-coolable in a standard chassis from Dell, HPE, Lenovo or Supermicro. About $16,000 per GPU, so roughly $130,000 of GPU per node. 768 GB per node serves 8B to 70B models, several at once.

This is the highest-probability first purchase for a mid-size enterprise, and the reason is boring: it fits your existing rack, cooling and 8 to 15 kW power budget. No liquid retrofit, no 30-week lead time, no structural review.

Tier 1, production. Two to four nodes of 8x H200 at roughly $370,000 a node, or 8x B200 at roughly $450,000. Sixteen to thirty-two GPUs, $0.75M to $1.8M. Handles a 70B dense model at high concurrency or a 671B MoE at FP8 on one node. H200 for lowest risk and best dollars per GB of HBM; B200 if you need native FP4.

Do not buy GB300 NVL72 first. 135 kW, mandatory liquid, 1.36 tons, roughly $3.7M, six to twelve month lead time, and it does not fit through a standard data center door. Rack-scale NVLink wins for trillion-parameter dense inference. It is not what you want for a 70B assistant.

Part	VRAM	TDP	Cooling	Note
RTX PRO 6000 SE	96 GB GDDR7	up to 600 W	air or liquid	best pilot part
H200 SXM	141 GB HBM3e	700 W	air	best risk-adjusted production buy
B200 SXM	192 GB HBM3e	1,000 W	air/liquid	FP4 native
B300 SXM	288 GB HBM3e	up to 1,400 W	liquid	14.5 kW/node
AMD MI350X	288 GB HBM3e	~1,000 W	air	most VRAM you can air-cool
AMD MI355X	288 GB HBM3e	1,400 W	liquid	same memory, denser
Rubin R200	288 GB HBM4	~1.8 kW	liquid	partner availability H2 2026

Used Hopper is underrated. Refurbished H100 runs $21,000 to $34,000 per GPU, used 8-GPU servers $150,000 to $180,000. Dispersion is wide, so get quotes. Hopper is not collapsing in value: CoreWeave rebooked expiring 2022-vintage contracts at 95 percent of original pricing.

AMD, in one rule

If you have a platform engineer who can read HIP and debug a container, AMD saves real money. MI300X is the cheapest HBM-class capacity available, and MI350X gives you 288 GB you can air cool, which no NVIDIA part does. If your team's first instinct on a CUDA error is to file a ticket, buy NVIDIA and spend the difference on people.

ROCm 10.0 ships production vLLM and SGLang containers and is genuinely viable for mainstream models. But decode parity with NVIDIA is still a roadmap goal, new-model support lags by weeks to months, and Hugging Face's Text Embeddings Inference is NVIDIA-only, so an AMD-only RAG stack needs a different embedding server.

Intel is not a credible primary platform for a 2026 decision. Among the alternatives, SambaNova and Tenstorrent are the only two that will genuinely sell you a box; Groq and Cerebras are cloud businesses now.

One 8-GPU node is a rack

A DGX B300 draws 14.5 kW. Typical enterprise colo racks are provisioned at 8 to 15 kW. Your single node eats the entire cabinet.

Cooling	Practical ceiling
Optimized air	30 to 40 kW/rack
Rear-door heat exchanger	60 to 80 kW/rack
Direct-to-chip liquid	60 to 120 kW/rack
Immersion	120+ kW/rack

Converting an air-cooled facility to liquid runs on the order of $2M to $3M per megawatt, with power upgrades often adding as much again. That is exactly why the Tier 0 pilot is a good first move: it is the largest useful deployment you can do without having the liquid conversation.

Three more constraints that surprise people. Colo vacancy in North America hit a record-low 1 percent, and wholesale pricing is rising, about $196/kW/month in H2 2025. Enterprises are securing capacity 18 to 24 months ahead. Lead times run 8 to 16 weeks through an OEM and 30+ weeks direct.

Networking and storage: do not overbuy

You do not need InfiniBand for an inference cluster under 64 GPUs. Inference is far less sensitive to collective latency than training.

Fabric	vs InfiniBand	Amortized 3-yr	Use when
InfiniBand NDR/XDR	100%	~$0.29/GPU/hr	64+ GPUs, real training
Spectrum-X Ethernet	85 to 90%	~$0.19/GPU/hr	mid-scale mixed
Tuned RoCEv2	70 to 80%	~$0.10/GPU/hr	inference-first

If you go RoCEv2, budget real engineering time for PFC and ECN tuning. Misconfigured, its latency becomes unpredictable and genuinely hard to debug.

One hard rule: keep tensor parallelism inside a single NVLink domain. Cross-node tensor parallelism is prohibitively slow. Go pipeline or data parallel across nodes instead.

Storage is undemanding for inference. Baseline 1 GB/s per GPU. Weights, a vector index and logs fit comfortably on one good NVMe tier. Save the parallel filesystem conversation for fine-tuning.

Buy versus rent: duty cycle decides

Naively, a $25,000 GPU at $3.00/GPU-hr pays back in about 347 days of continuous use. Add power, cooling, space and staff and it is 18+ months at near-100 percent utilization.

Here is the number that actually decides it. Cast AI measured average GPU utilization across enterprise Kubernetes clusters at roughly 5 percent. At 5 percent duty cycle, a $3.44/GPU-hr committed rate costs $68.80 per genuinely useful hour. Owned or rented, low utilization is what kills you.

So measure your duty cycle on rented capacity for a quarter before buying anything. Below 40 to 60 percent sustained, on-prem will not pay back.

Class	Neocloud on-demand	Hyperscaler on-demand
B200	$5.91 to $8.60/GPU-hr	$14.00 to $16.11
H200	$3.99 to $6.31	$7.91 to $10.85
RTX PRO 6000	$1.41 to $2.50	$3.36 to $5.50

Hyperscalers run 2 to 3x neocloud for identical silicon. Committed pricing spreads wider still: 64x B200 for a year costs about $1.93M on CoreWeave and about $9.03M on Google Cloud. Check the discount structure, because it sets your break-even: CoreWeave Reserved at 60 percent off breaks even at 40 percent utilization, while AWS Capacity Blocks at 13 percent off need 86.8 percent. Watch the line items outside the headline rate too, since cluster networking adds roughly 40 percent and hyperscalers meter egress.

Step 2: choosing and obtaining the model

Pick from the Apache-2.0 and MIT tier unless you have a reason not to, and read the actual LICENSE file at the commit you pin. Open weights are not open source, and that distinction is where legal review stalls.

What is deployable

Open weights now trade within roughly one generation of frontier closed models, and on some coding and agentic work the gap has closed. What has not closed is operational: you own the evals, the safety tuning and the incident response.

Model	Size	License	Fits on
DeepSeek V4-Pro	1.6T / 49B active	MIT	multi-node
Mistral Large 3	675B / 41B	Apache 2.0	4 to 8 GPUs at FP4
Inkling	975B / 41B	Apache 2.0	multi-node
DeepSeek V4-Flash	285B / 13B	MIT	4 to 8 GPUs
gpt-oss-120b	117B / 5.1B	Apache 2.0	1 GPU, ships in MXFP4
Mistral Small 4	119B / 6B	Apache 2.0	1 to 2 GPUs
Llama 4 Scout	109B / 17B	community	1 to 2 GPUs
Qwen3.8-27B	27B dense	Apache 2.0	1 GPU
Granite 4.2	3B / 8B / 30B	Apache 2.0	1 GPU, IBM indemnifies via watsonx
Gemma 4	to 31B	Apache 2.0	1 GPU

Two planning notes that save real time. Llama 5 does not exist. Llama 4 from April 2025 is the final Llama release; Meta moved to the closed Muse line. Content farms publish fabricated Llama 5 specifications, so do not put it on a roadmap. And most enterprises should live in the mid tier, not the frontier tier. A 30B model right-sized to the task triples your throughput against a 70B and cuts unit cost proportionally.

The three license tiers

Tier 1, unencumbered. Apache 2.0 or MIT. No thresholds, no acceptable-use policy, no gates. DeepSeek, Mistral's open lineup, gpt-oss, Gemma 4, Qwen3.8-27B, Granite, Olmo, Apertus, Inkling. If your legal team has low tolerance for novel license review, restrict the candidate list to this tier and you are done.

Tier 2, revenue or user gated. These read like MIT for three paragraphs and then add numbered conditions. The pattern is consistent and, critically, most carve out purely internal use:

Kimi K3: MaaS providers over $20M revenue in any 12 months must negotiate separately. Attribution above 100M MAU or $20M monthly revenue.
Qwen3.8-Max: attribution above 100M MAU or $20M monthly revenue; MaaS businesses above $50M trailing-twelve-month revenue need a paid license.
MiniMax: mandatory attribution, and it binds fine-tunes and distilled derivatives.
Llama 4: the 700M MAU carve-out, branding requirements, an acceptable-use policy by reference, and EU restrictions on the multimodal variants. Not OSI open source.

Tier 3, territorial exclusions. New, and the one to watch. MiniMax's H3 license excludes the United States, EU, UK and South Korea, and the exclusion covers deploying the outputs, not just running the weights. Check for this clause explicitly.

Pin every model to a commit SHA and archive the LICENSE that was in force at ingest. Hugging Face repos are mutable, and "it was MIT when we downloaded it" is only a defense if you can prove it.

Quantization policy

Pick a house standard:

FP8 is the default. Indistinguishable from BF16 for most work, half the memory, native on Hopper and Blackwell.
NVFP4 when memory forces it on Blackwell. Within about 1 percent of FP8 on MMLU-Pro and GPQA.
MXFP4 when you need portability. Runs on Blackwell and MI355X.
AWQ or GPTQ only on Ampere and Ada, where FP4 has no hardware path.

The exception that matters: both FP4 formats degrade measurably on reasoning and math. If your workload is reasoning-heavy, stay at FP8 and buy the memory. And benchmark on your own eval set, because published deltas are averages over benchmarks you do not run.

Getting weights safely

Four controls, in priority order:

Enforce safetensors at the registry layer. Pickle formats execute arbitrary code at load. Make it a policy gate, not a convention.
Ban trust_remote_code=True in production. It executes arbitrary repo-supplied Python regardless of weight format, bypassing every other control.
Sign at ingest with your own key. No registry signs automatically. Signing at your gate proves the artifact passed your review, not that somebody uploaded it.
Hash-pin to a commit SHA, never a branch.

Scanning is defense in depth, not a control: malware has been demonstrated evading pickle scanners via compression. Run ModelScan anyway, but do not rely on it.

A minimal ingest gate:

"""Ingest gate: pull a pinned revision, enforce format, sign, record."""
import hashlib, json, pathlib, subprocess
from huggingface_hub import snapshot_download

ALLOWED_LICENSES = {"apache-2.0", "mit", "openmdw-1.1"}
BANNED_SUFFIXES = {".bin", ".pt", ".pth", ".pkl", ".ckpt"}

def ingest(repo_id: str, revision: str, dest: pathlib.Path) -> dict:
    path = pathlib.Path(
        snapshot_download(
            repo_id=repo_id,
            revision=revision,            # a commit SHA, never a branch name
            local_dir=dest / repo_id.replace("/", "__"),
            allow_patterns=["*.safetensors", "*.json", "*.txt", "*.model", "LICENSE*"],
        )
    )

    bad = [p.name for p in path.rglob("*") if p.suffix in BANNED_SUFFIXES]
    if bad:
        raise RuntimeError(f"non-safetensors weight artifacts present: {bad}")

    cfg = json.loads((path / "config.json").read_text())
    if cfg.get("auto_map") or cfg.get("trust_remote_code"):
        raise RuntimeError("model declares custom code; requires manual security review")

    digests = {
        p.relative_to(path).as_posix(): hashlib.sha256(p.read_bytes()).hexdigest()
        for p in sorted(path.rglob("*.safetensors"))
    }
    manifest = {"repo_id": repo_id, "revision": revision, "sha256": digests}
    (path / "INGEST_MANIFEST.json").write_text(json.dumps(manifest, indent=2))

    # Sign the manifest with your internal key; refuse to load unsigned models at serve time.
    subprocess.run(
        ["cosign", "sign-blob", "--yes",
         "--key", "hashivault://model-signing",
         "--output-signature", str(path / "INGEST_MANIFEST.sig"),
         str(path / "INGEST_MANIFEST.json")],
        check=True,
    )
    return manifest

For restricted egress, run a proxy in a DMZ allowlisted to huggingface.co, point clients at it with HF_ENDPOINT, and set HF_HUB_OFFLINE=1 in production so a cache miss fails loudly instead of quietly reaching the internet.

For true air gap, weights run 10 to 400 GB. Move them on encrypted media or a one-way diode and treat every model update as a release event with security review and chain-of-custody logging.

One governance note: NVIDIA announced a definitive agreement to acquire Hugging Face in early September 2026. Whatever your view, single-vendor dependency on the Hub is now a procurement question. Mirror accordingly.

Step 3: serving it

vLLM is the default, and here is why

vLLM 0.29.0 (released 2026-09-09) is the pragmatic choice for enterprise serving: continuous batching, PagedAttention, prefix caching, FP8 KV cache, LoRA multiplexing, structured outputs, speculative decoding, disaggregated prefill/decode, an OpenAI-compatible API and a genuinely good Prometheus surface.

Two things that will break your existing runbooks:

The V0 engine is gone. VLLM_USE_V1 no longer exists. There is nothing to enable.
--guided-decoding-backend no longer exists. Structured outputs moved to --structured-outputs-config, taking a JSON object with backend (one of auto, xgrammar, guidance, outlines, lm-format-enforcer), plus reasoning_parser, enable_in_reasoning and related fields.

Also, python -m vllm.entrypoints.openai.api_server is deprecated in favour of vllm serve.

A production launch command:

vllm serve /models/mistral-small-4 \
  --served-model-name corp-general-v1 \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --quantization fp8 \
  --async-scheduling \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --structured-outputs-config '{"backend":"xgrammar"}' \
  --enable-lora --max-loras 4 --max-lora-rank 32 \
  --otlp-traces-endpoint http://otel-collector.observability:4317 \
  --enable-prompt-tokens-details \
  --enable-per-request-metrics \
  --ssl-certfile /certs/tls.crt --ssl-keyfile /certs/tls.key

Four of those matter most. --enable-prefix-caching is close to free money for RAG and agents, where a long system prompt repeats on every call. --enable-chunked-prefill stops a long input stalling every in-flight decode. --max-num-seqs is your concurrency ceiling and bounds KV cache pressure, so it is the knob to turn when you see preemptions. --tool-call-parser values are registry-dependent, so check your build rather than copying mine.

Speculative decoding now has flat aliases, mutually exclusive with the config-object form:

# either
  --spec-method eagle3 --spec-model /models/draft-1b --spec-tokens 5
# or
  --speculative-config '{"method":"eagle3","model":"/models/draft-1b","num_speculative_tokens":5}'

Do not rely on vLLM's --api-key for authentication. vLLM's own documentation flags API-key auth as limited. It is a speed bump for accidental access, not a control. The gateway in the next section is your authentication boundary; vLLM should be reachable only from it, over mTLS, on a network policy that denies everything else.

The other engines, and when to use them

SGLang: strong prefix reuse and constrained generation, best for complex multi-turn and agentic control flow.
TensorRT-LLM: peak throughput on NVIDIA-only fleets, at the cost of engine-build complexity. Worth it for one high-volume model, not a fleet.
llama.cpp: CPU, edge, Apple Silicon, single user.
Ollama: laptops and demos. Not a serving tier. I have seen it in production and it does not end well under concurrency.

Kubernetes: the parts that matter

GPU Operator 26.7.0 brings the DRA driver to GA, requiring Kubernetes 1.34.2 or later. It deploys via a new GPUCluster resource that cannot coexist with ClusterPolicy, so this is a migration decision, not a flag.

MIG gives hard isolation and is what you want for multi-tenant SLOs. Time-slicing gives none and is fine for dev. Neither substitutes for right-sizing a replica.

A serving deployment, trimmed to the interesting parts:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-corp-general
  namespace: inference
spec:
  replicas: 2
  selector:
    matchLabels: {app: vllm-corp-general}
  template:
    metadata:
      labels: {app: vllm-corp-general}
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      serviceAccountName: vllm-runner          # bound to a SPIFFE identity
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: vllm
          image: registry.internal/vllm-openai:v0.29.0
          args:
            - "--model=/models/mistral-small-4"
            - "--served-model-name=corp-general-v1"
            - "--tensor-parallel-size=4"
            - "--max-model-len=32768"
            - "--gpu-memory-utilization=0.90"
            - "--max-num-seqs=256"
            - "--enable-chunked-prefill"
            - "--enable-prefix-caching"
            - "--kv-cache-dtype=fp8"
            - "--async-scheduling"
            - "--otlp-traces-endpoint=http://otel-collector.observability:4317"
          env:
            - name: HF_HUB_OFFLINE            # fail loudly, never silently fetch
              value: "1"
            - name: VLLM_NO_USAGE_STATS
              value: "1"
          ports: [{containerPort: 8000, name: http}]
          resources:
            limits: {nvidia.com/gpu: 4}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: ["ALL"]}
          volumeMounts:
            - {name: models, mountPath: /models, readOnly: true}
            - {name: shm, mountPath: /dev/shm}
          startupProbe:                        # weight loading takes minutes
            httpGet: {path: /health, port: 8000}
            failureThreshold: 60
            periodSeconds: 15
          readinessProbe:
            httpGet: {path: /health, port: 8000}
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim: {claimName: model-weights, readOnly: true}
        - name: shm
          emptyDir: {medium: Memory, sizeLimit: 16Gi}

Two details people get wrong. The /dev/shm volume is required for tensor parallelism; the default 64 MB will hang you. And the startup probe needs a long failure threshold, because loading 70 GB of weights off network storage is not fast.

Autoscaling: scale on queue depth, not GPU utilization. GPU utilization is a terrible signal for LLM serving because a fully-batched engine sits at 100 percent whether it is keeping up or drowning. The canonical signal is vLLM's own vllm:num_requests_waiting:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-corp-general
  namespace: inference
spec:
  scaleTargetRef: {name: vllm-corp-general}
  minReplicaCount: 2
  maxReplicaCount: 8
  pollingInterval: 15
  cooldownPeriod: 600          # weight loading is expensive; do not thrash
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.observability:9090
        # KEDA's Prometheus scaler treats threshold as a per-replica average,
        # so this targets ~3 queued requests per replica.
        query: sum(vllm:num_requests_waiting{model_name="corp-general-v1"})
        threshold: "3"

vLLM exposes roughly forty Prometheus metrics. The ones worth dashboards on day one: vllm:num_requests_waiting (autoscaling), vllm:kv_cache_usage_perc (capacity headroom), vllm:num_preemptions (you are oversubscribed), vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds (your user-facing SLOs), and the prefix-cache hit ratio (your caching ROI).

When you outgrow single-node replicas, llm-d and NVIDIA Dynamo give you disaggregated prefill/decode with KV-cache-aware routing, both on the Gateway API Inference Extension.

Do not start here. Single-node replicas behind a gateway carry you a long way, and disaggregated serving adds a lot of operational surface.

Step 4: wrapping it with APIs

The gateway is the product, not the model server. It is what your developers integrate with, what your security team audits, and what lets you swap the model underneath without a migration project.

vLLM speaks the OpenAI protocol, so it is tempting to hand teams that endpoint and call it done. A raw inference endpoint has no notion of who is calling, no budget, no audit trail, no failover, and no way to deprecate a model without breaking every consumer.

What the gateway tier must do

Authenticate the human or workload. OIDC for people, workload identity for services. Never a shared API key in a config file.
Authorize per model. Not everyone gets the 675B model or the internet-connected tools.
Meter and cap. Per-user, per-team, per-application budgets with hard stops. Unbounded consumption is now OWASP LLM06:2026 and it is a real availability risk, not a billing annoyance.
Route and fail over. Model aliases (corp-general rather than mistral-small-4-fp8-tp4) so you can move the underlying model without touching consumers.
Redact and inspect. PII detection inbound, guardrail classification both ways.
Emit a complete audit record. Who, what model, what prompt hash, what tools, what cost, what trace ID.
Cache. Exact-match and semantic caching, with a scope key that includes identity so you never serve one user's answer to another.

The options

Product	Version	Self-hostable	Note
LiteLLM	1.100.1 (2026-09-10)	MIT, fully	pragmatic default; virtual keys, budgets, 100+ providers
Agent Router (formerly Envoy AI Gateway)	1.1.0 (2026-08-21)	yes, on Envoy Gateway	includes a full MCP gateway with OAuth
Kong AI Gateway	3.14 (2026-04-14)	OSS core, most AI plugins enterprise	shipped RFC 8693 token exchange
Apache APISIX	current	Apache-2.0, yes	`ai-proxy`, `ai-rate-limiting`, MCP plugins
Cloudflare AI Gateway	n/a	no, SaaS only	disqualified for a private deployment

Note the rename: Envoy AI Gateway is now "Agent Router" under the Agentic AI Foundation. Same code, same maintainers, new domain. Any config or doc referencing the old name is stale.

Start with LiteLLM for the control plane and add Kong or Agent Router at the edge when you need token exchange and MCP brokering. A minimal config:

model_list:
  - model_name: corp-general              # the alias your developers code against
    litellm_params:
      model: hosted_vllm/corp-general-v1
      api_base: https://vllm-corp-general.inference.svc:8000/v1
      api_key: os.environ/VLLM_INTERNAL_KEY
    model_info:
      max_input_tokens: 32768
      supports_function_calling: true

  - model_name: corp-general              # second replica pool, same alias
    litellm_params:
      model: hosted_vllm/corp-general-v1
      api_base: https://vllm-corp-general-b.inference.svc:8000/v1
      api_key: os.environ/VLLM_INTERNAL_KEY

  - model_name: corp-reasoning
    litellm_params:
      model: hosted_vllm/deepseek-v4-flash
      api_base: https://vllm-reasoning.inference.svc:8000/v1
      api_key: os.environ/VLLM_INTERNAL_KEY

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  allowed_fails: 3
  cooldown_time: 30
  fallbacks:
    - corp-reasoning: ["corp-general"]

litellm_settings:
  drop_params: true
  set_verbose: false
  callbacks: ["otel", "langfuse"]
  redact_user_api_key_info: true
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    ttl: 3600
    # scope cache entries by caller so answers never cross a tenant boundary
    supported_call_types: ["acompletion", "atext_completion"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  enforce_user_param: true                # every request must carry an end-user id
  max_budget: 25000                       # platform-wide monthly ceiling, USD
  budget_duration: 30d
  alerting: ["slack"]

Then issue scoped virtual keys per team rather than distributing the master key:

curl -sS -X POST https://llm-gateway.internal/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "team_id": "contracts-team",
        "models": ["corp-general"],
        "max_budget": 500,
        "budget_duration": "30d",
        "rpm_limit": 120,
        "tpm_limit": 400000,
        "metadata": {"owner": "legal-ops", "data_class": "confidential"}
      }'

Identity: the part that is usually wrong

Here is the failure mode I see most often. A user authenticates to a chat UI. The UI holds a service account key for the gateway. The gateway holds a service account key for the tool backends. By the time a tool executes a query, the identity of the human who asked has evaporated, and every user has the union of every permission the service account holds.

That is a confused deputy, and in an agentic system it is how a prompt-injected document reads someone else's payroll data.

The correct pattern has three layers.

Layer 1, workload identity. SPIFFE/SPIRE issues an identity to every pod, and mTLS authenticates by that identity rather than by network position. This answers "which service is calling." It does not carry the human.

Layer 2, user identity propagation. The user's OIDC token must not be forwarded verbatim downstream. The gateway exchanges it:

User authenticates to the gateway via OIDC. The gateway now holds a token audienced to the gateway.
Gateway performs an RFC 8693 token exchange at the authorization server: presents the user token as subject_token, sets resource (RFC 8707) to the canonical URI of the specific downstream service, and narrows scope to the minimum this call needs.
The AS returns a token whose aud is that one service, carrying the user as sub and the gateway as act (the actor claim, which is the on-behalf-of signal).
Gateway calls the service with that token, over mTLS.
The service validates signature, audience against its own resource identifier, scope, and expiry, and rejects anything not minted for it.

Kong AI Gateway 3.14 productized step 2.

Layer 3, secrets. Vault, now on 2.x, for provider keys and database credentials, with SPIFFE-authenticated login so no bootstrap secret sits on disk.

Audit. Propagate W3C trace context end to end and record sub plus act at every hop, so a tool invocation is attributable to a human rather than to a service account. This is the artifact your auditors will actually ask for.

Step 5: security

There is no reliable prompt-level defense against prompt injection. Stop looking for one and put the controls where the document cannot reach.

The threat model

Two OWASP lists govern this now. The GenAI LLM Top 10, 2026 edition:

Code	Risk	Movement
LLM01	Prompt Injection	unchanged at #1
LLM02	Sensitive Information Disclosure	unchanged at #2
LLM03	Excessive Agency	up from #6, biggest mover
LLM04	Supply Chain	down from #3
LLM05	Data and Model Poisoning	down from #4
LLM06	Unbounded Consumption	up from #10
LLM07	Misinformation	up from #9
LLM08	Hidden Context Exposure	renamed from System Prompt Leakage
LLM09	Vector and Embedding Weaknesses	down from #8
LLM10	Improper Output Handling	down from #5

Two movements matter. Excessive Agency went from sixth to third, which is the agentic era in one data point. And LLM08 was broadened: hidden context now covers any non-user-facing content assembled into the prompt, explicitly including retrieved policy text and tool schemas. If your retrieval layer injects internal policy documents, that is in scope.

The companion OWASP Top 10 for Agentic Applications covers goal hijack, tool misuse, privilege abuse, memory poisoning and rogue agents. If you are wiring tools to a model, read it.

What actually works

Instruction hierarchies, delimiters and "ignore any instructions in the following document" all fail under adaptive attack. Any vendor claiming their model is injection-resistant is selling a probability, not a control.

The principle: never let untrusted content influence which tool runs or what arguments it gets. The strongest published direction is CaMeL (arXiv:2503.18813), which extracts a plan from the trusted user query only and executes it under a capability system, so tool output can populate data but never redirect control. That is a provable property rather than a filter.

What you can build today:

Deterministic allowlists on tool arguments. The model proposes; non-model code validates. If the model asks for customer 12345, check the authenticated user is entitled to customer 12345 before the call runs.
Treat all tool output as untrusted input on the way back. Everyone skips this. A web page, a ticket comment, a PDF in a shared drive: all attacker-controlled.
Egress filtering. Exfiltration is the payoff for most injections. If your servers cannot reach arbitrary hosts, a successful injection has nowhere to send the data. This blunts a large share of real attacks and costs you a NetworkPolicy.
Human in the loop for irreversible actions. Mail, money, deletions, production writes.
Dual-LLM. A privileged planner that never sees untrusted text, and a quarantined model that reads it but returns only structured data.

Guardrail models

Model	License	Best at
Granite Guardian 4.1-8b	Apache-2.0	general safety plus RAG hallucination checks
Qwen3Guard Gen and Stream	Apache-2.0	Stream classifies token-by-token, for live interception
Llama Guard 4-12B	Llama Community	MLCommons 14-hazard taxonomy, multimodal
Llama Prompt Guard 2	Llama Community	injection and jailbreak classification specifically
NeMo Guardrails	Apache-2.0	framework, not a model

Granite Guardian and Qwen3Guard are Apache-2.0 and the safest legally. Llama Guard and ShieldGemma carry acceptable-use terms; review before shipping them in a product.

A gateway-side hook:

"""Guardrail middleware: classify in, classify out, fail closed on the way in."""
import httpx
from dataclasses import dataclass

GUARD_URL = "http://vllm-guard.inference.svc:8000/v1/chat/completions"
BLOCK_ON_INPUT = {"jailbreak", "pii_exfiltration", "violent_crimes", "code_interpreter_abuse"}
BLOCK_ON_OUTPUT = {"pii_leak", "credential_leak", "violent_crimes"}

@dataclass
class Verdict:
    safe: bool
    categories: list[str]

async def classify(client: httpx.AsyncClient, text: str, role: str) -> Verdict:
    resp = await client.post(
        GUARD_URL,
        json={
            "model": "granite-guardian-4.1-8b",
            "messages": [{"role": role, "content": text}],
            "max_tokens": 16,
            "temperature": 0.0,
        },
        timeout=5.0,
    )
    body = resp.json()["choices"][0]["message"]["content"].strip().lower()
    if body.startswith("safe"):
        return Verdict(True, [])
    return Verdict(False, [c.strip() for c in body.split("\n")[1:] if c.strip()])

async def guarded_completion(client, gateway, request, user_id, trace_id):
    inbound = await classify(client, request["messages"][-1]["content"], "user")
    if not inbound.safe and set(inbound.categories) & BLOCK_ON_INPUT:
        audit(trace_id, user_id, "blocked_input", inbound.categories)
        raise PermissionError(f"request blocked: {inbound.categories}")

    result = await gateway.completion(**request)
    text = result["choices"][0]["message"]["content"]

    outbound = await classify(client, text, "assistant")
    if not outbound.safe and set(outbound.categories) & BLOCK_ON_OUTPUT:
        audit(trace_id, user_id, "blocked_output", outbound.categories)
        return refusal("I can't share that. Withheld by policy.")

    audit(trace_id, user_id, "allowed", [])
    return result

Three notes. Run the guard on a separate small replica so a guard timeout cannot take down your serving pool. Decide explicitly whether a timeout fails open or closed: inbound closed, outbound streaming usually open with async flagging, because blocking mid-token is a terrible experience. And log every verdict with the trace ID, because the false-positive rate decides whether people route around your platform.

The data boundary

You built this to keep data in. Then somebody enables verbose request logging and every prompt lands in a log aggregator half the company can query.

Log prompt hashes and token counts by default, not prompt text. Make full capture an explicit, time-boxed, separately authorized debugging mode.
Set trace retention independently from metrics. Thirty days is usually plenty and shrinks exposure sharply.
Scope caches by identity. A semantic cache keyed only on prompt similarity will serve one user's answer to another.
Give the vector index the same data classification as its source documents. Embeddings are not anonymized; inversion attacks recover meaningful text.
Decide up front whether you retain prompts for fine-tuning. If yes, that is a separate consent and retention regime, and it belongs in your privacy notice before you collect anything.

Step 6: MCP, or how the model reaches your systems

A private LLM that cannot see your data is expensive autocomplete. MCP is how you connect it. If your mental model of MCP is from 2025, four things will break your code.

What changed on 2026-07-28

Sessions are gone, including Mcp-Session-Id. Servers are stateless and scale horizontally without sticky sessions, which is a real operational win. Cross-call state now uses server-minted handles passed as ordinary tool arguments.
The initialize handshake is gone. Every request carries its protocol version in _meta, and servers must implement a new server/discover RPC.
Server-initiated requests are gone. Instead the server returns an InputRequiredResult and the client retries with inputResponses. Every result now carries a resultType.
SSE resumability was removed. A broken stream loses the in-flight request, so make your tools idempotent or safely retryable.

Also deprecated: Roots, Sampling, Logging, the HTTP+SSE transport, and OAuth Dynamic Client Registration. Streamable HTTP is the transport; stdio stays for local tools.

The Python SDK went to v2

FastMCP no longer exists. The module is a tombstone that raises on import. The class is mcp.server.MCPServer. Pin mcp>=1.28,<2 if you are not ready. The TypeScript SDK is still on 1.x, so do not assume parity.

Here is the security-critical core of a server. The full working version is in the companion repo:

from mcp.server import MCPServer
from mcp.server.auth.settings import AuthSettings
from mcp.server.auth.middleware.auth_context import get_access_token

mcp = MCPServer(
    name="corp-contracts",
    token_verifier=IntrospectingVerifier(...),     # sets AccessToken.resource from the token aud
    auth=AuthSettings(
        issuer_url=ISSUER_URL,
        resource_server_url=RESOURCE_URL,          # this server's RFC 8707 identifier
        required_scopes=["contracts.read"],
        validate_token_resource=True,              # refuse tokens minted for anything else
        identity_assertion_enabled=True,           # SEP-990 enterprise IdP flow
    ),
)

@mcp.tool(annotations=ToolAnnotations(readOnlyHint=True))
async def search_contracts(query: str, limit: int = 10) -> SearchResult:
    token = get_access_token()
    if token is None or token.subject is None:
        raise PermissionError("no authenticated principal on this request")

    # Query the backend AS THE USER. This server holds no superuser credential,
    # so a prompt injection that reaches this tool has nothing to steal.
    async with httpx.AsyncClient(verify=CA) as http:
        r = await http.post(BACKEND, json={"q": query, "limit": limit},
                            headers={"Authorization": f"Bearer {token.token}",
                                     "X-On-Behalf-Of": token.subject})
    data = r.json()
    return SearchResult(hits=[ContractHit(**h) for h in data["hits"]],
                        total_matched=data["total"],
                        truncated_by_permission=data["denied_count"])

Four things carry the security here:

validate_token_resource=True is the control. Without it, a token minted for the ticketing server is happily accepted by the contracts server. In SDK 2.x, leaving it unset warns and behaves as False, so set it explicitly.
The tool calls the backend with the user's token. If this server held a superuser key, an injection reaching this tool would have that key's full authority.
truncated_by_permission returns a count, never content. Telling the model "4 more exist that you cannot see" is useful. Returning them is a breach.
ToolAnnotations are advisory. readOnlyHint lets a client auto-approve. It does not stop your code writing. Enforce in the handler.

MCP security, concretely

The spec's hard rule:

MCP servers MUST NOT accept any tokens that were not explicitly issued for the MCP server.

That is worth watching rather than reading. Below, a token minted for the contracts server is replayed against the tickets server and refused with a 401 and a real WWW-Authenticate challenge. Then validate_token_resource is turned off and the identical replay succeeds:

The full stack that produced that clip is public. Five services, two MCP servers, the policy layer and the property tests: github.com/swirlai/private-llm-reference. docker compose up, then make demo, and all three security properties run on your machine in about a minute.

No model runs in that recording. A prompt cannot stop a token being replayed. A one-line audienceGive m check can, and it lives in the resource server, not the model and not the client.

The operational controls beyond the spec:

Pin and hash tool definitions. A rug pull is a server changing a tool's description after approval. Alert on drift.
Render full tool descriptions to whoever approves them. Tool poisoning hides instructions in description text the approver never sees.
Never auto-trust one server's output as instructions. Tool results are data.
Isolate credentials per server, so a compromised server cannot borrow another's authority.
Run an internal MCP registry. The official one is in preview, does not support private servers, and is not designed for self-hosting. Run your own implementing its OpenAPI interface, and make it the only place agents discover tools.

For enterprise identity, SEP-990 (the Identity Assertion JWT Authorization Grant) is the flow you want. It carries a signed assertion from your IdP and exchanges it for an MCP access token. Its trust model is inverted from ordinary MCP OAuth: the authorization server is configuration, not discovery. No metadata fetch, no dynamic registration, no server-driven scope selection, which eliminates the mix-up and confused-deputy classes at the root rather than defending against them.

Step 7: connecting it to your data

Most of the value in a private LLM comes from retrieval, not generation. Two things determine whether this works.

Permission-trimmed retrieval. The vector index must not be a permission bypass. Either filter at query time on the caller's groups, which is faster but needs your ACLs denormalized into the index, or retrieve candidates and re-check entitlement against the source system, which is slower and always correct. Either way, the check happens before text enters the context window. Once a document is in the prompt, it is disclosed.

Embeddings and reranking. Current sensible defaults, all self-hostable:

Purpose	Model	License	Note
Embedding, general	Qwen3-Embedding-0.6B / 4B / 8B	Apache-2.0	tops MTEB multilingual; Matryoshka dims 32 to 4096; 32K context
Embedding, hybrid	BGE-M3	MIT	dense + sparse + multi-vector in one pass
Embedding, long docs	Jina Embeddings v4	Apache-2.0	128K context
Reranking, default	BGE-Reranker-v2-m3	Apache-2.0	568M, ~35 ms/pair
Reranking, quality	Qwen3-Reranker-8B	Apache-2.0	~77% BEIR nDCG@10
Reranking, latency	ms-marco-MiniLM-L-12-v2	Apache-2.0	33M, under 5 ms/pair

Avoid NV-Embed-v2 and Jina ColBERT v2 in commercial deployments: both are CC-BY-NC. And verify every license on the model card rather than in a comparison article, because this specific category is the most consistently misreported.

Two notes. Embedding models turn over far slower than LLMs, so standardize on one and leave it alone, because changing it means reindexing everything. And a reranker is the highest return per dollar in a RAG stack. Adding one usually beats upgrading your generator.

Step 8: observability, evals, and the acceptance gate

Instrumentation

Instrument to the OpenTelemetry GenAI conventions. They are still marked Development, so expect the attribute names to change under you.

Emit gen_ai.client.token.usage, gen_ai.server.time_to_first_token, gen_ai.server.time_per_output_token and gen_ai.execute_tool.duration. Langfuse and Arize Phoenix are both self-hostable and both fine; pick one.

The thing that matters more than the tool: ship one trace ID from the gateway through to the tool call, so a single identifier ties a user complaint to a prompt, a model version, a retrieval set and a tool invocation.

Evals are the acceptance gate

This is the discipline that separates a platform from a demo. You cannot upgrade a model you cannot evaluate. Without an eval suite, every model change is a leap of faith, and the practical consequence is that you never upgrade, which means you paid for control and got stagnation.

Build three layers:

Capability baselines. lm-eval for academic benchmarks. Useful for sanity, not for your business.
Task evals on your data. 200 to 500 real labeled examples from your actual workload, run in CI with promptfoo or DeepEval. This is the one that matters and the one people skip, because building it is unglamorous.
Safety evals. A red-team corpus of injection attempts and permission-boundary probes, run on every model or prompt change.

Wire them into a pipeline that gates promotion:

# .gitlab-ci.yml (or equivalent)
eval:
  stage: verify
  script:
    - promptfoo eval -c evals/task-suite.yaml --output results.json
    - promptfoo eval -c evals/injection-suite.yaml --output redteam.json
    - python scripts/gate.py results.json redteam.json
  rules:
    - changes: [ "config/models.yaml", "prompts/**/*", "tools/**/*" ]

# scripts/gate.py: fail the build, not the users
import json, sys

results, redteam = (json.load(open(p)) for p in sys.argv[1:3])

task_pass = results["stats"]["successes"] / max(results["stats"]["total"], 1)
injection_block = redteam["stats"]["successes"] / max(redteam["stats"]["total"], 1)

FAILURES = []
if task_pass < 0.92:
    FAILURES.append(f"task pass rate {task_pass:.1%} below 92% floor")
if injection_block < 1.00:
    FAILURES.append(f"injection suite blocked only {injection_block:.1%}; must be 100%")

if FAILURES:
    print("\n".join(f"BLOCKED: {f}" for f in FAILURES))
    sys.exit(1)
print(f"PASS  task={task_pass:.1%}  injection_blocked={injection_block:.1%}")

The injection suite threshold is 100 percent deliberately. Task quality is a tradeoff curve. A permission boundary is not.

Load testing

Benchmark before you promise an SLO. GuideLLM drives load at your actual token distributions, which is what synthetic benchmarks get wrong. Measure at target concurrency, not at concurrency 1, and set --max-num-seqs from what you measured rather than what you hoped.

Step 9: governance you cannot skip

EU AI Act, as it actually stands in September 2026

The timeline moved. The Digital Omnibus on AI deferred the high-risk deadlines; it did not cancel them. Current state:

Date	Obligation	Status
2025-02-02	Prohibited practices, AI literacy	in force
2025-08-02	GPAI model obligations	in force
2026-08-02	AI Office enforcement powers over GPAI active	in force, just started
2026-08-02	Article 50 transparency (synthetic content marking)	in force, grace period for pre-existing systems
2026-12-02	Article 50 grace period ends	upcoming
2027-08-02	Pre-2025-08-02 GPAI models must be compliant	upcoming
2027-12-02	Chapter III high-risk, Art. 6(2) / Annex III	deferred from Aug 2026
2028-08-02	Chapter III high-risk, Art. 6(1) / Annex I	deferred from Aug 2027

If you deploy rather than develop GPAI models, your near-term exposure is Article 50 marking plus AI-literacy duties. High-risk obligations are fifteen months out. That is breathing room, not a pass: deciding whether your system is high-risk under Annex III takes longer than people expect, so start now.

The rest

NIST AI RMF plus its Generative AI Profile is voluntary, and the most useful structure I have found for organizing the work.
ISO/IEC 42001 is the certifiable counterpart. If you sell to enterprises, expect it in questionnaires.
Keep a model inventory: every model in production with source, revision SHA, license as of ingest, quantization, eval results and owner. Building it after the fact is miserable.

The cost model, honestly

Here is a worked example for the Tier 1 cluster: two nodes of 8x H200, sixteen GPUs, colocated.

Line	Annual
Capex amortized (2 nodes ~$370k each + ~$150k network/storage/racks, 3-year straight line)	~$297,000
Colocation (space, power, cooling at ~$196/kW/month for ~28.6 kW IT load)	~$67,000
Platform engineering (0.5 FTE, fully loaded)	~$110,000
Subtotal	~$474,000
Optional: NVIDIA AI Enterprise ($4,500/GPU/yr, if you need NGC/NIM support)	+$72,000

That is 140,160 GPU-hours a year, so roughly $3.38 per GPU-hour all-in. Compare against committed neocloud H200 at roughly $2.75 to $3.50/GPU-hr and hyperscaler on-demand at $7.91 to $10.85.

Read that carefully, because it is the conclusion most vendor content will not give you: at this scale, on-prem is roughly at parity with committed neocloud capacity. It is decisively cheaper than hyperscaler on-demand. It is not a cost play against a well-negotiated neocloud contract.

Now translate to tokens, which is what your finance team actually wants:

cost_per_million_output_tokens = annual_cost / (throughput_tok_s * duty_cycle * 31.536)

At a measured 5,000 output tokens/second aggregate across both nodes:

Duty cycle	Cost per million output tokens
20%	$15.03
40%	$7.51
70%	$4.29
90%	$3.34

Duty cycle dominates everything else in this model. It swings your unit cost by more than 4x, which is more than any hardware choice, model choice or quantization decision available to you. Recall the Cast AI measurement of roughly 5 percent average GPU utilization across enterprise Kubernetes clusters. If that is where you land, none of this pencils.

So be clear-eyed about what the number means. Private inference at sixteen GPUs is not cheaper per token than a commodity open-model token vendor, which will sell you the same model for well under a dollar per million tokens. It is substantially cheaper than frontier closed-model API pricing at volume. And it is the only option when the data genuinely cannot leave. Also note that the same hardware running a 30B model rather than a 70B roughly triples throughput and cuts unit cost proportionally, so right-sizing the model to the task is the largest cost lever you control after utilization.

Build this because you need control. Treat cost parity as a pleasant side effect at sufficient scale, not as the business case.

A 90-day plan that works

Days 1 to 30, prove the workload.
Rent. Do not buy anything. Stand up vLLM on a rented 8-GPU node, put LiteLLM in front of it, wire OIDC, and give it to one real team with one real use case. Build the 200-example eval set from their actual work. Instrument everything. At the end of thirty days you should know your token distributions, your peak concurrency and your duty cycle, which are exactly the numbers Step 0 asked for and that nobody can guess.

Days 31 to 60, harden.
Add the guardrail model and the audit trail. Implement token exchange so user identity reaches the tools. Build the first MCP server against a system that matters, with validate_token_resource=True and per-user permission trimming. Write the injection eval suite and make the CI gate real. Run GuideLLM at three times your observed peak and find where it breaks.

Days 61 to 90, decide.
You now have a working platform on rented capacity and a quarter of utilization data. Run the buy-versus-rent math with your real duty cycle. If you clear 40 to 60 percent sustained and have a residency requirement, order the Tier 0 or Tier 1 hardware and start the colo and cooling conversation, remembering the 8-to-16-week OEM lead time. If you do not, sign a committed contract and revisit in two quarters. Either way you have a platform in production, which is more than most of these projects achieve in a year.

What I would skip

Fine-tuning, at first. Almost every "we need a fine-tune" turns out to be a retrieval problem or a prompt problem. Exhaust both before you build a training pipeline, because a fine-tune is a permanent maintenance obligation attached to a model version you will want to change.
Building your own gateway. LiteLLM is MIT-licensed and does more than your first three sprints would.
A vector database evaluation project. Pick one, index, ship, measure. Retrieval quality comes overwhelmingly from chunking, hybrid search and reranking, not from the store.
Rack-scale NVLink systems on day one. Covered above, but it bears repeating because somebody in the room will want one.
Multi-model routing before you have evals. Routing between models you cannot compare is a way to make quality unmeasurable.
Agent frameworks, for a while. A gateway, a well-scoped MCP server and a loop you wrote yourself will outperform a framework you do not understand, and the security properties will be ones you can actually reason about.

Where this leaves you

The stack is stable enough to build on: vLLM behind a gateway, open weights under a license your counsel has read, identity that survives the whole call path, MCP tools with audience-bound tokens, and evals that gate promotion. None of it is exotic in 2026, and all of it is achievable in a quarter with two good engineers.

The parts that will actually determine whether you succeed are not technical. Duty cycle decides your economics. Power and cooling decide your timeline. And the discipline of building evals before you need them decides whether you can ever upgrade the model, which decides whether the platform is an asset or a monument.

Two caveats on the numbers. NVIDIA and AMD publish no list prices, so treat every capex figure here as an indication and get a quote. And verify a model's license on its card, at the commit you pin, not in any table including mine.

References

Primary sources for the claims that decide something, so you can check them as they change.

Companion code
swirlai/private-llm-reference

Serving and Kubernetes
vLLM v0.29.0 ·
NVIDIA GPU Operator release notes ·
Kubernetes 1.34 DRA GA ·
llm-d

Gateways and identity
LiteLLM ·
Agent Router, formerly Envoy AI Gateway ·
Kong AI Gateway 3.14 ·
SPIRE

MCP
Spec 2026-07-28 changelog ·
Authorization spec ·
Security best practices ·
Enterprise-Managed Authorization, SEP-990 ·
Python SDK

Security and governance
OWASP GenAI LLM Top 10 2026 ·
OWASP Top 10 for Agentic Applications ·
CaMeL ·
EU AI Act timeline ·
Sigstore model-transparency

Model licenses change without notice. Read the LICENSE file at the exact commit you pin, including for anything named above.

Why Backstage search returns `team` when you search for `tech` and what we did about it

Sid Probstein — Thu, 10 Sep 2026 16:23:44 +0000

Type "tech" into Backstage search. You get "team".

We are the team behind SWIRL, and we spent this summer building a search engine module for Backstage. This is what we found, what we measured, and what we shipped. Every comparison below cites a Backstage issue number or a number we measured ourselves.

1. The complaint, in the maintainers' own issues

Four issues describe the same class of problem. One line each, with the number, so you can read them yourself.

27339: searching for "tech" returns entities named "team". The default engine adds an edit distance term to every query.
6177: partial entity names do not match unless the user types a wildcard.
8835: the Postgres engine cannot match part of a word at all.
32795: Elasticsearch support is still pinned to version 7, which reached end of life in January 2026.

None of these are bugs in the ordinary sense. Each engine is doing exactly what it was built to do. The problem is that a service catalog is not a document corpus, and the three default engines were all chosen for other reasons.

2. Why a shared index cannot fix it

The instinct is to point every portal at one big index and move on. That does not address any of the four.

Lunr lives in the backend process memory and is rebuilt on restart. It is a browser search library doing service catalog work. Its LunrSearchEngine appends an edit distance 2 term and a trailing wildcard term to every query, which is the single mechanism behind 27339. We reproduced it: "tech" returns 38 results on a stock instance, with team-a, team-d, team-c and team-b at ranks 8, 9, 10 and 11.

Postgres uses ts_rank over tsvector. That is a good full text engine for prose. It has no infix matching, which is issue 8835, and a catalog is almost entirely identifiers: payment-gateway-api, foo-bar.com, wayback-search. Identifiers are the one thing ts_rank was not designed for.

Elasticsearch would solve the relevance problem. It also asks you to run a JVM cluster next to your portal, and Backstage support is stuck on version 7 per 32795, with version 7 out of support since January 2026. Most teams we talk to chose Postgres search specifically to avoid this.

So the fix is not a bigger index. The fix is an index whose schema knows it is holding entity names.

3. The gauntlet

We wrote nine cases before we wrote the engine. Five come straight out of the issues above. We ran them against a live Backstage on the default Lunr engine, against SWIRL in process, and finally against the published container through the real ingest and search path.

The corpus is the Backstage example catalog, 71 documents, plus 5,000 synthetic entities. The synthetic set plants the entities the issues describe and 20 entities containing "team" with no "tech".

Query	Default engine	SWIRL
`tech`	38 results, team-a to team-d at ranks 8 to 11 (27339)	tech-prefixed title at rank 1, no team-only entity in the top 5
`abac`	0 results, no partial match (6177, 8835)	abacus at rank 1, sole hit
`foo-bar.com`	59 results, the first twelve all user entities	foo-bar.com at rank 1, sole hit
`store`	petstore absent from all 7 results (8835)	petstore in the top 3
`mes`	14 results, none containing the string, led by a person	zero results
`wayback`	wayback-search at rank 2	wayback-search at rank 1
`petsotre` (typo)	petstore at rank 1	petstore at rank 1, with fuzzy matching on
`wayback search` (phrase)	wayback-search at rank 1	wayback-search at rank 1
`service` filtered to kind=component, lifecycle=production	no attribute filter in the query API	all 25 hits match both attributes

Nine pass, zero fail, on the released image after our second fix pass. Two things worth saying plainly. The typo case is a genuine Lunr win: its built in edit distance finds petstore from petsotre with no tuning at all, and SWIRL only matches that with fuzzy matching enabled. And on the tech case the shipped path puts a tech-*-service entity at rank 1 rather than tech-radar; our relevancy pass prefers the longer title. The assertion the case exists to protect, no team-only entity in the top 5, holds. We are recording that rather than claiming it away.

The size and speed numbers, measured on 50,000 entities on one laptop:

34 MB index on disk
2.7 s to index all 50,000 documents, about 18,600 documents per second
0.26 ms mean query time, over 900 queries against a warm index

Relevance was the hard part. Speed was never close to a constraint.

4. Two lanes

The indexed lane. Your collators run unchanged. The engine module streams their documents to the SWIRL ingest endpoint, which writes them into a Tantivy index, one index per Backstage document type. The schema is the whole point: a stemmed and ASCII folded title_exact field boosted 3x, an n-gram title_ngram field for infix matching, and a stopworded text field. That n-gram field is the answer to 6177 and 8835. Ingest is upsert and delete by id, so there is no full rebuild and no transaction timeout.

The federated lane. SWIRL queries live sources per query, GitHub and Confluence today, and their results arrive in the same ranked list as swirl-federated documents. Nothing is copied into an index. This is the thing a shared index genuinely cannot do, because the source of truth stays where it is and the query goes out fresh.

What Backstage keeps. Its collators, its search page, and its permission filtering. SWIRL replaces the engine and nothing else. Permission fields pass through untouched, so filtering on indexed results works on day one. Your search page does not change when you swap the engine.

5. Install: one container, three steps

The container. One docker compose up -d with swirlai/swirl-backstage:0.1.1. Redis runs inside the image, started by the entrypoint. SQLite holds the app database. The index lives on the /data volume. No Postgres for SWIRL, no JVM, no cluster. On our smoke test it went from cold to healthy in 14 seconds.

Step 1. Edit .yarnrc.yml in your app root, then install:

nodeLinker: node-modules
npmMinimalAgeGate: 3d
npmPreapprovedPackages:
  - '@backstage/*'
  - '@swirl-search/*'

yarn --cwd packages/backend add @swirl-search/backstage-plugin-search-backend-module-swirl@0.1.1

That .yarnrc.yml line is not optional and it is worth explaining, because we got it wrong first. @backstage/create-app ships npmMinimalAgeGate: 3d, a Yarn 4 supply chain control that refuses any package published in the last 72 hours unless its scope is preapproved. Only @backstage/* is preapproved by default. So for three days after every release, including this one, yarn add fails with YN0016 for every user on a stock app. Our own smoke test hit it 25 minutes after we published. Preapproving the scope is the Backstage native escape hatch. Do not set npmMinimalAgeGate: 0, which turns the control off for every package in your repo.

Step 2. Wire it into packages/backend/src/index.ts, and remove the Postgres module that create-app ships:

backend.add(import('@backstage/plugin-search-backend'));
backend.add(
  import('@swirl-search/backstage-plugin-search-backend-module-swirl'),
);

Step 3. Point it at the container in app-config.yaml:

search:
  swirl:
    baseUrl: http://localhost:8000

Restart the backend. The first index went live 3.2 seconds after backend start on our test app.

6. What it costs to run, honestly

One container. About 2.5 GiB resident today, which is over our own 1 GB target and we are not going to pretend otherwise. Tantivy is about 150 MB of that. The rest is torch, presidio and litellm imported across three SWIRL processes, and trimming it is on the build plan rather than done.

One replica, deployment strategy Recreate, with the index on a PersistentVolumeClaim. All your Backstage pods point at the one SWIRL service. Restarts cost nothing: the index is on disk, so SWIRL reopens it and serves again in seconds, and it never reindexes. We verified that by restarting the container mid test, with the Backstage backend left running and no collator run, and the same query returned the same 43 results.

The failure mode, stated plainly: if the SWIRL pod is down, Backstage search returns an error rather than stale results. Catalog browsing is unaffected. Active passive HA is a roadmap item, not a launch promise.

The image is 7 GB on disk. That is large, and it is the same reason as the memory number.

7. Community and Enterprise

SWIRL for Backstage Community is free and Apache 2.0, and it is everything above: the indexed lane, the federated lane, GitHub and Confluence. The line between the editions is OAuth2. Community federated sources authenticate with one scoped service account per source, so every portal user sees whatever that account can read, and we require a scope restriction before a source will activate. Enterprise adds per-user OAuth2 identity passthrough, OAuth2-only sources, semantic cache and cross source dedup; if you need per-user permissions on federated results, contact us.

8. Go type "tech" into your portal

That is the whole ask. If you run Backstage, search for "tech" and tell me what comes back. If it is a team, you have reproduced 27339, and there is now an engine that does not do that.

Docs: https://docs.swirlaiconnect.com/backstage-overview
Install: https://docs.swirlaiconnect.com/backstage-install
GitHub: https://github.com/swirlai/swirl-backstage
Plugin directory: listed on backstage.io under Search
Demo video: https://youtu.be/VRH0r8iaJlo

npm is @swirl-search/backstage-plugin-search-backend-module-swirl at 0.1.1, and the image is swirlai/swirl-backstage:0.1.1. Issues and pull requests welcome, especially from anyone running Postgres search.

Get Every Email to Your Domain in One Gmail Inbox, Free, with Cloudflare

Sid Probstein — Fri, 14 Aug 2026 18:29:26 +0000

You own a domain. Your website lives on it. With Cloudflare's free plan you can make anything@yourdomain.com - any address, invented on the spot - land in your regular Gmail inbox. No Google Workspace subscription, no mail server, no changing registrars. About 30 minutes, $0.

What you get:

A catch-all: every address at your domain forwards to one inbox.
Free disposable addresses: give netflix@yourdomain.com to Netflix and bank@yourdomain.com to your bank. When spam arrives addressed to one of them, you know who leaked your address.
Your website, untouched.

One limitation: this is receive-only. You can reply as you@yourdomain.com from free Gmail via "Send mail as", but the mail goes out through Google's servers without your domain's blessing, so some recipients see "via gmail.com" and strict spam filters may object. If sending from the domain matters, that's what Google Workspace is for. For receiving, read on.

You need: a domain (registered anywhere; it stays there), a free Cloudflare account, and an inbox to receive the mail.

Step 1: Add your domain to Cloudflare

Cloudflare scans your existing DNS and imports what it finds. Review this list carefully - the scan is good but not guaranteed complete, and whatever it misses stops resolving after the switch. Check what your DNS really says from any terminal:

dig +short A yourdomain.com
dig +short MX yourdomain.com
dig +short NS yourdomain.com

If the MX query returns nothing, you have no existing email service and this migration is pure upside. If it returns something, understand what that mail service is before proceeding.

Step 2: The gotcha - where does your DNS actually live?

That NS query tells you who currently answers DNS for your domain, and it is not always your registrar.

In my case the nameservers pointed at NS1 (*.nsone.net) - Netlify DNS. My registrar just held the registration; the DNS zone lived in Netlify, from clicking "use Netlify DNS" when connecting my site years earlier. The A records Cloudflare imported were Netlify load balancer IPs, which Netlify rotates. Had I switched nameservers and kept those records, my site would have broken on the next rotation.

The fix: point at names, not IPs. For a Netlify site (same idea for Vercel, GitHub Pages, and friends):

Find your site's default subdomain in the Netlify dashboard (your-site.netlify.app).
In Cloudflare's DNS page, delete the imported A records.
Add two CNAMEs, both DNS only (click the orange cloud so it turns grey):
- www → your-site.netlify.app
- @ (apex) → your-site.netlify.app (Cloudflare flattens apex CNAMEs automatically)

DNS-only matters. Your host already runs a CDN and manages your HTTPS certificate; stacking Cloudflare's proxy in front invites redirect loops and cert renewal failures for zero benefit. Grey cloud means Cloudflare only answers DNS. Email Routing doesn't use the proxy, so it works either way.

Step 3: Switch nameservers at your registrar

Cloudflare assigns you two nameservers, like ada.ns.cloudflare.com and bob.ns.cloudflare.com. At your registrar, replace the current nameservers with those two. The registrar keeps the registration and the renewal bill; it just stops answering DNS questions.

Propagation is faster than its reputation - the change hits the .com registry within minutes:

dig +noall +authority NS yourdomain.com @a.gtld-servers.net

When that shows the Cloudflare nameservers, click Check nameservers now on your domain's Cloudflare overview page instead of waiting for its periodic poll. Then load your website. If you did Step 2 right, nothing changed.

Step 4: Turn on Email Routing

In the Cloudflare dashboard for your domain: Email → Email Routing → Get started.

Add a destination address - your Gmail. Click the link in the verification email Cloudflare sends.
On Routing rules, enable the Catch-all with action "Send to an email" pointed at your verified destination.
Accept the DNS records Cloudflare offers to add: three MX records (route1/2/3.mx.cloudflare.net) that direct your domain's mail to Cloudflare, plus SPF and DKIM records that keep forwarded mail from being spam-flagged. The odd-looking MX priorities are normal.

That single catch-all rule is the whole feature.

Step 5: Test it

dig +short MX yourdomain.com
# 36 route3.mx.cloudflare.net.
# 95 route2.mx.cloudflare.net.
# 98 route1.mx.cloudflare.net.

Send a message from an account other than the destination inbox (Gmail hides mail you send to yourself) to an address you just made up: whatever@yourdomain.com. It should arrive within seconds. Check spam the first time and hit "Not spam" if needed.

Caveats

Receive-only, as covered above. Email Routing forwards; it doesn't send or store.
Catch-alls attract spam eventually - dictionary-attack spam to random@yourdomain.com all lands on you. Gmail's filtering handles it, and the per-service trick turns the bug into a feature: add a Cloudflare rule to drop any address that gets burned.
Your old DNS host's features die at the switch - web redirects, parked pages, registrar email forwarding. That's the point, but inventory what you use before you flip.

Half an hour, zero dollars, and every address at your domain lands in one inbox.

Enterprise source integration as configuration: inside SWIRL's SearchProviders

Sid Probstein — Fri, 07 Aug 2026 03:51:36 +0000

The expensive part of federated search was never the query fan-out.
It's the N integrations behind it, each with its own API, auth, query
syntax, and response format. SWIRL's answer is to make each integration a configuration object called a SearchProvider, and the 66-second video above walks through one in the admin tool.

The anatomy, per provider:

Connection: which connector to use (HTTP GET/POST returning JSON, M365 Graph, SQL, Elastic DSL, ...), the endpoint, and the authenticator that supplies each user's credentials at query time.
Query: a template that maps SWIRL's query into the source's own syntax, plus query processors that adapt it automatically (synonyms, rewrites, source-specific quirks).
Results: JSONPath-style mappings that normalize each source's response into a common result shape: title, body, author, date, URL. This is what makes cross-source ranking honest; every result competes in the same schema, re-ranked by the same three-pass pipeline.

Because providers are rows, not code, the workflow is: activate a
preloaded provider (M365, Box, ServiceNow, Elastic, PostgreSQL,
BigQuery, and dozens more ship in the box), add credentials, search.
Change a mapping and the next query uses it; there is no deploy step.

Example provider JSON: https://github.com/swirlai/swirl-search/tree/main/SearchProviders

Guide: https://docs.swirlaiconnect.com/SP-Guide.html

Replaced Salesforce and HubSpot with Postgres, a Spreadsheet, and an AI Agent

Sid Probstein — Tue, 04 Aug 2026 16:00:51 +0000

We have used two real CRMs.

First Salesforce, because our founding sales person said it was essential. Then HubSpot, because our founding marketing person said it was essential. Both people were right that tracking the pipeline is essential... but really, neither system earned its keep at our scale.

What the data actually looked like, once it escaped, was a spreadsheet: three columns of company names labeled Engaged Prospects, Legal, and Partners/Channels. >100 organizations, no statuses, no activity history, no next steps.

A full CRM had twice proven to be overkill; the spreadsheet was underkill.

So I replaced both with a mini-CRM built in an afternoon with Claude Cowork, using tools we already run: local PostgreSQL, the spreadsheet itself, and SWIRL for search. Here is the full build, including the bug we shipped and caught.

Step 1: From three columns to a schema

The agent read the spreadsheet with openpyxl and proposed a schema built for the actual job, which is tracking activity, not just listing names:

CREATE TABLE organizations (
    id          SERIAL PRIMARY KEY,
    name        TEXT NOT NULL,
    category    TEXT NOT NULL CHECK (category IN ('engaged_prospect', 'legal', 'partner')),
    status      TEXT NOT NULL DEFAULT 'active',   -- free-form: active, stalled, won, dead...
    notes       TEXT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE (name, category)
);

CREATE TABLE contacts (
    id      SERIAL PRIMARY KEY,
    org_id  INT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    name    TEXT NOT NULL,
    title   TEXT,
    email   TEXT,
    phone   TEXT
);

CREATE TABLE activities (
    id            SERIAL PRIMARY KEY,
    org_id        INT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    activity_date DATE NOT NULL DEFAULT CURRENT_DATE,
    activity_type TEXT NOT NULL DEFAULT 'note',   -- call, email, meeting, demo...
    summary       TEXT NOT NULL,
    next_step     TEXT,
    next_step_due DATE
);

Two design decisions that paid off immediately:

Activities are append-only history, not a mutable "last touched" field. You never lose the story of an account.
Status is free-form text. We started with a suggested vocabulary, but the first real update was "dead", which was not on the list. A CHECK constraint on status would have turned a one-word instruction into a schema migration. Constrain the things that break joins (category); leave vocabulary to the humans.

Two views do most of the daily work:

-- Latest activity per organization
CREATE VIEW org_latest_activity AS
SELECT o.id, o.name, o.category, o.status,
       a.activity_date AS last_activity_date,
       a.activity_type AS last_activity_type,
       a.summary AS last_activity_summary,
       a.next_step, a.next_step_due
FROM organizations o
LEFT JOIN LATERAL (
    SELECT * FROM activities a
    WHERE a.org_id = o.id
    ORDER BY a.activity_date DESC, a.id DESC
    LIMIT 1
) a ON true;

-- The daily to-do list
CREATE VIEW followups_due AS
SELECT o.name, o.category, a.next_step, a.next_step_due
FROM activities a
JOIN organizations o ON o.id = a.org_id
WHERE a.next_step IS NOT NULL
  AND (a.next_step_due IS NULL OR a.next_step_due <= CURRENT_DATE)
ORDER BY a.next_step_due NULLS LAST;

An idempotent loader script pulled every name out of the three spreadsheet columns and inserted it with the right category. ON CONFLICT (name, category) DO NOTHING means re-running it after the spreadsheet grows is always safe.

Step 2: Make it searchable with SWIRL

We run SWIRL for federated search, so the obvious next move was a SearchProvider that folds the CRM into the same search box as everything else.

First, a denormalized view so a single ILIKE sweep can hit everything worth matching:

CREATE VIEW crm_search AS
SELECT o.id, o.name, o.category, o.status,
       COALESCE(o.notes, '') AS notes,
       la.last_activity_date, la.last_activity_type,
       COALESCE(la.last_activity_summary, '') AS last_activity_summary,
       COALESCE(la.next_step, '') AS next_step,
       la.next_step_due,
       COALESCE((SELECT string_agg(
                     c.name || COALESCE(' (' || c.title || ')', '')
                            || COALESCE(' <' || c.email || '>', ''), '; ')
                 FROM contacts c WHERE c.org_id = o.id), '') AS contacts,
       o.updated_at::date AS updated
FROM organizations o
LEFT JOIN org_latest_activity la ON la.id = o.id;

Then the provider. SWIRL's PostgreSQL connector takes a query template with mapped fields:

{
    "name": "Mini-CRM - PostgreSQL (SQL)",
    "connector": "PostgreSQL",
    "url": "localhost:5432:crm:<db-user>:<db-password>",
    "query_template": "select {fields} from {table} where {field1} ilike '%{query_string}%' or {field2} ilike '%{query_string}%' or {field3} ilike '%{query_string}%' or {field4} ilike '%{query_string}%' or {field5} ilike '%{query_string}%' or {field6} ilike '%{query_string}%'",
    "query_mappings": "fields=*,sort_by_date=updated,table=crm_search,field1=name,field2=notes,field3=last_activity_summary,field4=contacts,field5=category,field6=next_step",
    "result_processors": [
        "MappingResultProcessor",
        "CosineRelevancyResultProcessor"
    ],
    "result_mappings": "title='{name} ({category} / {status})',body='Last activity: {last_activity_summary} ({last_activity_type}, {last_activity_date}). Next step: {next_step} (due: {next_step_due}). Contacts: {contacts}. Notes: {notes}',date_published=updated"
}

Searching a company name, a status ("won"), a category ("partner"), or a phrase from a call summary all just work, ranked by SWIRL's relevancy pipeline alongside every other source.

The result_mappings line went through one iteration worth mentioning. The stock SQL-provider pattern is result_mappings: "DATASET", which collapses all rows into a single result carrying a table payload. Fine for analytics, wrong for a CRM: you want each account as its own result with its status and next step visible. Switching to explicit template mappings gives every org its own card:

Acme Manufacturing (engaged_prospect / active)
Last activity: intro call re search POC (call, 2026-07-15).
Next step: send scoping doc (due: 2026-07-25). ...

Mapping date_published=updated also makes date-sorting real instead of "unknown".

One more detail for the LLM era: SWIRL providers can carry query_instructions in their config, which is handed to an LLM when the source is queried through SWIRL's assistant or MCP server. Ours documents the full schema, the enumerated category/status values, and five SQL templates (activity history, due follow-ups, pipeline counts). That turns "what needs attention this week?" into real SQL against followups_due instead of a keyword guess.

Step 3: The update loop nobody hates

The first plan for data entry was conversational: the agent walks through each account and asks what happened. That died on contact with reality after one answer. Dictating 100+ updates one at a time is miserable.

The fix: the spreadsheet stays the editing surface. The agent added a second sheet, "Tracking", one row per organization, pre-filled from the database:

Name | Category | Status | Activity Date | Activity Type | Activity Summary | Next Step | Next Step Due | Notes

Edit any cells, save, run the sync:

Status or Notes differ from the DB: the org is updated.
Activity Summary is filled in and differs from the org's latest activity: a new activity row is inserted. History accumulates; the spreadsheet only ever shows the latest.
New name with a category: org created.
The script never deletes anything, and running it twice in a row is a no-op.

After any bulk change made directly in SQL, the sheet is regenerated from the database so the two never fight.

For bulk updates, plain English turned out to beat both surfaces. "Mark everything dead except these accounts; add this new deal as won via partner X" became one reviewed transaction. The agent applies it, prints the resulting pipeline counts, and refreshes the sheet.

The bug: names are not keys

The sync script's first version keyed organizations by name. One company in our data legitimately exists in two categories (it is both a service firm and a channel partner), which is exactly why the table's unique constraint is (name, category).

The script's in-memory state dict silently kept only one of the two rows, and its UPDATE ... WHERE name = X hit both. Net effect: syncing the sheet resurrected a row that a bulk update had just marked dead.

It surfaced immediately for one reason: the sync prints a change report, and we expected zero. A refresh-then-sync cycle should be a perfect no-op, and it reported "1 org updated" instead. That single unexpected line of output was the whole detection mechanism. The fix was mechanical (key by name plus category, scope every UPDATE and INSERT the same way), verified by re-running until the no-op was real.

If your loader is idempotent, "re-run it and demand zero changes" is the cheapest integration test you will ever write.

What we ended up with

A Postgres database with full activity history, one psql away.
A spreadsheet that is now a UI, not a database.
CRM accounts as first-class results in our federated search, next step and status on the card.
An LLM-queryable source: schema-aware SQL through SWIRL's MCP server.
A natural-language admin loop for bulk operations, with SQL you can read before it runs.

Total schema: three tables, three views, zero ORM, and two canceled CRM subscriptions. Claude Cowork wrote the schema, the loaders, the provider, and the bug; the change report caught the bug; the humans just answered questions and edited cells.

Would this scale to a 50-seat sales team? No, and it is not trying to. For a founder-led pipeline of a hundred accounts, the boring stack is hard to beat: every piece is inspectable, every update is a SQL statement you can read, and the search box already knew where to look.

Taught Claude Code to Install, Configure, and Debug SWIRL Enterprise Search Engine

Sid Probstein — Mon, 03 Aug 2026 14:49:47 +0000

Evaluating enterprise search software has a fixed cost that has nothing to do with the software: stand up a server, read the docs, connect a source, debug the connector, configure an LLM. By the time you see a real answer from your own data, a week is gone.

We built a Claude Code plugin that compresses that to about an hour, and along the way learned something more interesting than the time savings: a plugin skill is an executable runbook, and the best way to write one is to watch where the agent researches and move that knowledge upstream.

What it is

swirl-claude-plugin is free and Apache-2.0. It gives Claude Code eight guided workflows for SWIRL, a federated search + RAG engine:

claude plugin marketplace add swirlai/swirl-claude-plugin
claude plugin install swirl@swirl --scope user

Start a new Claude Code session and you have:

Command	What it does
`/swirl:start`	Assesses where you are and routes you
`/swirl:install`	Docker or local install, verified with a live search
`/swirl:provider`	Connect a source using 25+ built-in connectors
`/swirl:connector`	Develop a custom connector
`/swirl:rag`	Connect an LLM, configure ranking models, stand up cited answers
`/swirl:migrate`	Community-to-Enterprise moves and version upgrades
`/swirl:mcp`	Wire SWIRL's MCP server into Claude
`/swirl:troubleshoot`	Diagnose from logs; file a support ticket when diagnosis needs help

Why "an hour" is honest

SWIRL is federated. It queries SharePoint, OneDrive, Box, databases, and web APIs where they live, re-ranks results with local models, and feeds the best passages to the LLM you configure. Nothing is copied, ingested, or indexed; there is no vector database to build.

That is why the evaluation is fast: the usual multi-day step (moving your data into the search engine) does not exist. Claude installs the stack, connects two or three sources, and you are looking at ranked, cited answers from your own documents.

Skills are runbooks, not code

There is no code in the plugin. Each command is a skill: a markdown file whose frontmatter description routes the request and whose body loads focused operational knowledge into Claude's context. The install skill knows the real Docker quick start for each edition and insists on verifying with a live search before declaring success. The troubleshoot skill embodies a support rule we enforce on ourselves: the user's report is data; find the log line before proposing a fix. The skills also encode what NOT to do: never print API keys, never paste secrets into chat, never restart a stack someone is demoing on.

The development loop that actually improves it

Here is the pattern we now use for every release, with a real example.

During a walkthrough, a user asked Claude to "enable the cross-encoder and use larger embeddings instead of spaCy." Claude did what a capable agent does with missing knowledge: it researched. A dozen commands into settings files and compiled modules, the user steered it: ranking models in SWIRL are configured as AI Providers, in the database, like every other model choice. Once pointed there, Claude flipped the reader role to mxbai-embed-large served by the stack's own Ollama sidecar, verified the cross-encoder was already active in each result's explain output, and confirmed the change took effect with no restart.

That transcript became the next release. The rag skill now states it outright: ranking models are AI Providers; do not hunt through settings or source. The cross-encoder ships on by default, so confirm it rather than configure it. The embedding model is whatever provider holds the reader role. Changes apply at query time.

The general form: watch where your agent researches, then move that knowledge into the skill. Research is the tax an agent pays for what the skill does not say. Every walkthrough transcript is a free curriculum; the diff between "what the agent had to discover" and "what it should have known" is exactly the next version of the skill.

If you are building your own plugin

Three practices that will save you real time:

Give the plugin an explicit identity. Ship a .claude-plugin/plugin.json with the plugin's name and version, and keep component definitions (skills, commands) in exactly one place. Ambiguity here surfaces on the user's machine, not yours.
Test the exact path a stranger will run, from a clean environment. HOME=$(mktemp -d) claude plugin marketplace add you/your-repo && claude plugin install ..., then verify the user-visible surface: the actual command names in a session, not just "installed." Manifest validators check syntax, not experience.
Write the description for routing. The frontmatter description is what decides whether a user's phrasing reaches your skill. When a real user's words missed, we added their words to the description.

The loop closes with MCP

The part we like most: /swirl:mcp has Claude wire SWIRL's MCP server into itself. The same assistant that installed your knowledge layer then queries it, with permissions enforced server-side per user. Your agents get governed access to enterprise data; the data never moves.

Try it

Plugin: claude plugin marketplace add swirlai/swirl-claude-plugin
Docs: docs.swirlaiconnect.com/claude-plugin
One-page PDF: datasheet, no registration
SWIRL Community is open source; want an Enterprise license to evaluate? Work email in, license out; no human will contact you.

Issues and PRs welcome on the repo.

Turning Slackbot into an Enterprise Research Agent with MCP

Sid Probstein — Tue, 28 Jul 2026 15:04:03 +0000

We connected a Slack agent to our federated search engine over the Model Context Protocol, then asked it a question only our OneDrive could answer. It searched, read the document, and replied in Slack with the coverage limits in a table and a citation that opens the exact sentence in SharePoint.

This post covers the architecture, the wiring, and two gotchas that cost us real debugging time.

The architecture

Three pieces, all standard:

A Slack agent that speaks MCP as a client. Slack's agent platform lets a bot declare MCP servers in its manifest and connect to them at runtime.
SWIRL's MCP server. SWIRL is a federated search + RAG engine; its MCP server exposes search, search_rag (grounded answers with citations), read_document, score_document, list_providers, and chat. It's a standalone process that calls a running SWIRL deployment over HTTP, so licensing, throttling, and per-user permissions are enforced by SWIRL exactly as for any API client.
The sources. SharePoint, OneDrive, Box, databases, web APIs; SWIRL queries them where they live. Nothing is copied into the bot, the model, or a vector database.

The wiring

On the Slack side, the bot's manifest declares the MCP server and the bot connects with a slash command. On the SWIRL side, the server runs in HTTP transport mode:

SWIRL_MCP_TRANSPORT=http SWIRL_MCP_PORT=8675 \
SWIRL_MCP_TOKEN=<api-key> \
python -m swirl_mcp

For the demo we exposed it through a tunnel with auth disabled; do not do that in production. The production path is the server's OAuth 2.1 resource-server mode (SWIRL_MCP_AUTH=oidc): the MCP host runs PKCE against your IdP, the server validates each bearer JWT (issuer, audience, JWKS signature), and SWIRL maps the token to the real calling user, so every search is permission-trimmed per caller.

What a query looks like

Asked "Search OneDrive for SWIRL's insurance policy", the bot:

calls list_providers to find the OneDrive source id;
calls search scoped to that provider;
calls search_rag for a grounded answer over the results.

The reply carries the policy limits as a table, flags that professional services are excluded from that policy, and includes a text_fragment_url citation: a deep link that opens the source document scrolled to the exact quoted passage.

Two gotchas worth knowing

Timeout mismatch. Slack gives an MCP tool call about 60 seconds; SWIRL's search_rag polls up to 90 seconds for the RAG answer, sized for a cold local model. On a slow model the Slack call dies first and the bot reports a failure while the answer lands seconds later. Fix: fast model for Slack-facing RAG, or lower SWIRL_MCP_RAG_POLL_TIMEOUT so failures are honest.

Tool descriptions are UX. The bot's first act in the demo is describing SWIRL's tools, in its own words, accurately. That's not the bot being clever; it's the MCP server's tool descriptions being written for an LLM audience. If your agent misuses a tool, fix the description before touching the agent.

The takeaway

Agents don't need your data ingested into them; they need a governed search layer. The bot never held our documents. It held a connection to a server that could search them, as us, with permissions enforced and citations attached.

The same MCP server works from Claude Desktop and Claude Code. There's a Claude Code plugin that stands up the whole stack: swirl-claude-plugin, and the 2-minute demo video is here: [LINK-TO-VIDEO].

Docs: docs.swirlaiconnect.com

SWIRL 5 is GA: knowledge authority for enterprise AI

Sid Probstein — Wed, 22 Jul 2026 15:04:12 +0000

SWIRL 5 is generally available! I want to use this post to explain what it is at an engineering level, because the one-line pitch ("the knowledge authority layer for enterprise AI") does not tell you how it works or where the hard parts are. I would rather show you the machine.

Quick disclosure: I wrote the original version of SWIRL and I run the company. So read this as the person who built it explaining the design, not as a neutral survey. I have tried to be honest about the limits, including where a hand-tuned stack matches us.

The thing that changed

A year ago, "federated search across your systems, then re-rank the results" was a product you could sell. Today it is table stakes. MCP turned retrieval into a commodity: any model can reach any system through a connector, and every serious stack ships a re-ranker. If your pitch is "we retrieve and we re-rank," a technical evaluator will point at four other tools that do the same thing by lunch.

So the interesting question moved. It is no longer "can you find the document." Everything finds the document. It is "which of the versions you found is the one my organization actually stands behind." In a real enterprise corpus the answer to any given query does not exist once. It exists as a draft, three redlines, a copy someone saved to their desktop, and the executed final, all sitting in different systems, all semantically near-identical. Retrieval returns all of them. The model picks one, confidently, and it has no idea which one carries authority.

That is a governance problem wearing a search costume, and it is what SWIRL 5 is built to solve.

Canonical version election

The core primitive is version election. When SWIRL federates a query and gets back a cluster of near-identical documents, it does not just hand the pile to the model. It:

Clusters the versions it found across every source.
Scores each on signals that actually correlate with authority: source authority (an executed contract in iManage outranks a draft on someone's OneDrive), naming ("Executed", "Final", version numbers), and recency.
Elects one canonical version, and exposes its reasoning so a human can see why.

On top of that, teams can pin a canonical result for a query directly. Once pinned, every later search and every agent calling SWIRL gets the endorsed answer, full stop. Election is the automatic path; pinning is the human override. Both produce the same thing: a single answer the organization has stood behind, not the model's best guess.

This is the piece the frontier models cannot do for themselves. Claude and Copilot are extremely good at drafting and summarizing. Neither has any way to know which of your nine versions is ratified, because that fact does not live in the documents. It lives in your organization.

Ranking: three passes, run locally, no vector database

Under the election sits the ranking pipeline. It runs in three passes, and both models run locally in your tenant. Nothing goes over the wire to a third-party ranking service, and there is no vector database to build, secure, or keep in sync.

Keyword + BM25. Lexical first. Quoted phrases and exact terms are honored as written. In enterprise and legal content this matters: a defined term or a specific clause has to match exactly, not approximately.
Embedding re-rank. E5-Large-V2 embeddings with title-aware chunking, fused with the lexical scores using reciprocal rank fusion. Semantic recall, without letting it steamroll the exact matches from pass one.
Cross-encoder. MS-MARCO cross-encoder reads the query and each candidate document together, as a pair, and scores real relevance rather than vector similarity. It is the expensive pass, which is exactly why it runs last, only on the candidates that survived the first two.

The "no vector database" part is not a slogan, it is a deployment property. There is no second copy of your content sitting in an index that your information-governance policy never contemplated. The documents stay in iManage, Box, SharePoint, wherever they already live. Only the ranking happens in SWIRL. For a regulated buyer, removing that second copy removes an entire category of risk.

If you want the independent version of this argument, Meta's XetHub benchmarked keyword-only, vector-only, and hybrid re-rank, and hybrid won. Their post was literally titled "you don't need a vector database."

Assembly: a bounded prompt, not a context dump

Retrieval and ranking decide what is relevant. Assembly decides what the model actually sees, and that is where the token bill and a lot of the accuracy live.

SWIRL treats the prompt as a hard budget, around 3,000 tokens, and fills it deliberately: at most 10 sources, only those scoring above a relevance threshold, a topic matcher that scores the spans inside each source so only the passages that answer the query go in, per-source truncation to fit, and version de-duplication that drops the superseded copies before assembly. One bounded call, not the refine or tree-summarize modes that call the model once per chunk and multiply both tokens and latency.

The counterintuitive result, which we measured against LangChain and LlamaIndex defaults and checked against their source: in a versioned corpus, sending less produces a better answer. More context means more near-identical duplicates, and the model gets less certain, not more. We wrote that study up separately if you want the numbers and the honest caveats.

Grounding checks

Every generated answer is checked against the sources it cited. Claims that are not supported by the retrieved passages are flagged rather than shipped silently. This is not a hallucination cure, nothing is, but it changes the failure mode from "confident and wrong and unmarked" to "flagged for a human." In an enterprise setting that distinction is the whole game.

How it fits your stack

SWIRL 5 is headless and API-first, with a first-class MCP server. Any AI work surface calls it over MCP or REST and gets back ranked, permissioned, canonical answers with citations attached. It runs in your tenant, honors each source system's existing permissions on every query, and works with the model you choose, hosted Claude or GPT, Copilot, or a local model on your own hardware. The agent angle is the one I would watch: a human reading a superseded policy usually catches it, an agent does not pause, so serving agents the approved answer instead of a raw retrieval is a safety property, not a nicety.

Where it is hard, honestly

Version election is heuristic. Source authority, naming, and recency get you a long way, but naming conventions are messy and "most recent" is not always "operative." Pinning exists precisely because the automatic signal is not always enough. We show the reasoning so humans can correct it.
A carefully hand-tuned vanilla RAG can match our per-call token count. Low k, a good reranker, small chunks, plus your own de-dup layer. Our value is delivering that discipline by default and adding cross-version de-duplication that off-the-shelf stacks do not, not magic per-token compression.
Grounding checks reduce unsupported claims, they do not eliminate them. Treat the flag as a prompt for review, not a guarantee.

Try it

If you are putting AI on your own data, that is exactly the case this is built for. It is generally available now at swirlaiconnect.com, and I am happy to run it against a slice of your own systems so you can see the ranking, the citations, and the permission boundary on your data rather than a demo corpus. I built it, so you would be talking to the person who wrote the code.

How many tokens does your RAG stack actually send to the LLM?

Sid Probstein — Tue, 07 Jul 2026 13:13:00 +0000

The token bill for a RAG system is not set by your vector database. It's set one step later, by how you assemble the context you hand the LLM. Retrieval finds candidates; assembly decides how many of them, how much of each, and across how many LLM calls. That's where the money is spent.

So "vector DB vs. framework X vs. SWIRL" is the wrong axis. The real comparison is between context-assembly strategies. Here's an honest, source-checked look at how many tokens each common approach sends, and where SWIRL 5 actually costs less.

The honest headline first

SWIRL doesn't win by using smaller chunks. Anyone can lower top_k. On a single lean query, a minimal config like LlamaIndex's default (top_k=2) sends fewer raw tokens than SWIRL. If someone tells you SWIRL "always uses the fewest tokens," a technical evaluator will disprove it in five minutes.

SWIRL's advantage is structural, and it shows up exactly where enterprise content lives, in corpora full of document versions:

It never spends tokens on duplicate or superseded versions. Stock top-k returns whatever is nearest in embedding space, which in a versioned corpus means several near-identical copies. SWIRL collapses them to one canonical document before the LLM sees them.
It always answers in one bounded call. The multi-document synthesis modes people reach for when they want quality (LangChain refine/map_reduce, LlamaIndex refine/tree_summarize) multiply LLM calls, and refine multiplies tokens super-linearly. SWIRL never pays that tax.

What the common defaults actually do

Naive vector-DB RAG (Pinecone / Weaviate / Qdrant tutorials)

The pattern is: embed the query → retrieve top-k chunks → stuff them all into one prompt. Typical documented defaults:

Chunk size ~512 tokens (the common "start here"; the band is 256-1024)
Chunk overlap 10-20%
top_k 3-5
No de-duplication. None of the vendor quickstarts add a dedup or diversity step. Overlap alone guarantees adjacent chunks share text, and multiple versions of a document sit in the same embedding neighborhood, so top-k routinely returns redundant context and the pipeline sends all of it.

Input tokens ≈ k × chunk_tokens + overhead. At k=5 and ~1,000-token chunks, that's ~5,000 tokens, a large fraction of it redundant.

Sources: Pinecone, Weaviate, Qdrant.

LangChain

Verified from source:

Retriever default k=4 (similarity_search(k=4) in vectorstores/base.py)
RecursiveCharacterTextSplitter default 4,000 chars / 200 overlap (≈1,000 tokens) (text_splitters/base.py)

The combine-documents chains differ enormously in cost:

Chain	LLM calls (N docs)	Token behavior
`stuff`	1	all docs in one prompt
`map_reduce`	N + ≥1	one call per doc, then reduce
`refine`	N, sequential	re-sends the growing answer each step → super-linear tokens, no parallelism
`map_rerank`	N	one doc per call

refine is the trap: each step re-transmits the accumulating answer plus the next document, so the running answer is re-sent N-1 times and grows as it goes.

LlamaIndex

Verified from source (llama-index-core):

chunk_size 1,024 tokens, similarity_top_k 2, default response mode compact (constants.py, factory.py)
No default de-duplication. Node postprocessors are opt-in and the default similarity cutoff is off, so two versions of a doc in top-k both go to the LLM.

Response modes (docs):

Mode	LLM calls	Token behavior
`compact` (default)	~1	packs nodes into as few prompts as fit
`refine`	N	one call per node, re-sends evolving answer
`tree_summarize`	>1, recursive	summarize groups, then summaries-of-summaries
`accumulate`	N	query each node separately

compact with top_k=2 is genuinely lean, but that's a recall trade-off, and it still sends duplicate versions.

What SWIRL 5 does (verified in source)

SWIRL treats the LLM prompt as a hard budget and fills it deliberately:

Mechanism	Behavior
Hard prompt budget	RAG prompt capped at ~3,000 tokens (`SWIRL_RAG_TOK_DEFAULT`); assembly stops when full
Source cap + relevance gate	≤ 10 sources, only those scoring ≥ 0.8 are eligible
Scored semantic chunks	a BM25 topic-matcher scores spans within each source; on overflow SWIRL narrows to those scored spans, not the whole chunk
Truncation	per-source token-by-token truncation to fit the budget
Markup	relevant spans wrapped in `<SW-IMPORTANT>…</SW-IMPORTANT>` plus a compact per-source metadata header
De-dup before the LLM	version-cluster alternates dropped; if a doc is pinned, only the canonical is kept, so N versions collapse to 1
Single call	one stuff-style synthesis call (worst case +1 JSON repair)

Net: ~3,000 input tokens, one call, zero redundant-version tokens, a predictable ceiling that doesn't grow with document size or corpus size.

Putting numbers on it

Scenario: a query over a corpus where the relevant policy exists in 5 versions, plus 3 other relevant documents, a realistic enterprise shape. Total LLM input tokens per query (summed across calls):

Reading it honestly:

vs. a deliberately minimal stuff config (LlamaIndex compact, top_k=2): SWIRL is comparable per call, but that config buys its low count with poor recall and still ships duplicate versions.
vs. typical stuff RAG (k=5, ~1k chunks): SWIRL is ~40-45% fewer input tokens, and it removes the redundant-version half entirely.
vs. the quality-oriented multi-call modes (refine, tree_summarize, map_reduce): SWIRL is a 2-3× reduction in input tokens, and a larger reduction in output tokens, since those modes generate once per call.

Where the advantage is real, and where it isn't

Real and hard to get off-the-shelf: cross-version de-duplication. None of these frameworks do it by default. In a versioned corpus it's the difference-maker, and it compounds: the more versions, the more SWIRL saves.
Real: one bounded call vs. N-call synthesis; a predictable cost ceiling.
Honest caveat: a hand-tuned vanilla RAG (low k, a good reranker, small chunks, stuff, plus your own dedup layer) can match SWIRL's per-call token count. SWIRL's value is delivering that discipline by default, and adding version de-dup the others lack. It isn't magic per-token compression.

Takeaway

If you're answering questions over a corpus with real document versioning, the tokens you waste aren't in chunk size - they're in sending the LLM the same document five times and in synthesis modes that call the model once per chunk. SWIRL 5's design removes both by default. Measure your own stack the same way: total input and output tokens per query, summed across every LLM call. That number, not top_k, is your bill.

Links + Snippets Not Enough for RAG

Sid Probstein — Mon, 06 Jul 2026 12:00:00 +0000

Several people posting lately about how RAG + search is not enough:

"All the model gets is a list of links and snippets. It's not enough to make sense of most real business questions."

Agreed. If your pipeline stuffs ten search snippets into a prompt and hopes, you get confident mush. A snippet is a pointer to evidence, not the evidence. The fix is a step most RAG setups skip: page fetch plus a reader. Three parts.

1. Fetch the page, not the snippet

The search result is a pointer. So before generating, SWIRL fetches the actual page or document behind each top hit. Now the pipeline is working from the full source, not the 200 characters a search API happened to return.

2. Read it with a reader LLM

This is the part that matters. Some model reads each fetched page against the question and pulls out the passages that actually answer it, marking and scoring them. The chaff never reaches the expensive answering model.

Do extraction before generation, not instead of it.

3. Budget the context

Real pages do not fit neatly into a context window, so the reader either truncates to the highest-signal passages or marks them in place, best first, until the budget is spent. The answering model gets curated, ranked evidence instead of a pile of chunks competing for attention.

The result is the difference between "here are some links" and an answer you can act on. Same retrieval, far better grounding, because something actually read the sources before the model spoke.

That take is exactly why the reader step exists. Naive RAG earns its bad reputation. This is how you avoid it.

(And don't get me started about not having to put it all in a vector database first!)

Making RAG admit when it's guessing: source-grounded hallucination checks

Sid Probstein — Wed, 01 Jul 2026 15:12:48 +0000

The failure mode that scares me most in RAG isn't a wrong answer. It's a confident wrong answer with three citations that don't actually say what the answer claims.

So in SWIRL 5 I stopped trusting the model to police itself and added a check that runs after generation.

The flow:

Generate the answer with its citations, as usual.
Split the answer into atomic claims — roughly one assertion per sentence.
For each claim, pull the specific spans from the retrieved passages the model cited.
Run an entailment check: does the cited text actually support this claim, contradict it, or neither?
Any claim that isn't supported gets flagged in the UI, inline, before the user reads a word of it.

The interesting part wasn't the entailment model; it was everything around it.

Claim segmentation is harder than it sounds. Naive sentence splitting produces claims that are unverifiable on their own because the subject lives two sentences up.

Citations lie by omission. A model will cite a document that's topically relevant but doesn't contain the specific number it just quoted. The whole point of the check is to catch exactly that gap.

Latency budget. An honesty layer nobody waits for is an honesty layer nobody ships. SWIRL 5 batches and optionally caches passage embeddings and more.

The result isn't "SWIRL never hallucinates." Nothing can promise that. The result is: when it's on thin ice, it tells you, and it points at the exact sentence.

That's the version of trustworthy I can actually build.

SWIRL Community 4.5 Update

Sid Probstein — Sun, 28 Jun 2026 16:30:28 +0000

I wrote about SWIRL here last summer. Time for an update.

SWIRL Community 4.5 is out. It is the open-source, Apache-2.0 build of SWIRL: federated search and RAG across your apps, running on your own machine, no vector database required. Three things in this release are worth your attention.

1. Point RAG at any LLM

The big one. Community now lets you configure AI Providers and send your RAG queries to any model you want: OpenAI, Anthropic, Azure OpenAI, or a fully local model behind Ollama or vLLM. Pick the provider, set the model, done.

ai_provider:
  name: anthropic      # openai | azure | ollama | vllm
  model: your-model
  api_key: ${LLM_KEY}

This matters for two reasons. RAG quality is now your choice, not ours, so you can run a stronger model when accuracy counts. And you can keep the whole pipeline local, with retrieval and generation both inside your network. This was an Enterprise feature; it is in Community as of 4.5.

2. Ask RAG a question directly

RAG now accepts a natural-language question as its input. Before, you ran a search and generated over the results; now you can hand SWIRL a plain question and it does retrieval and generation in one step. Question in, cited answer out, which means a lot less glue code to wire SWIRL into a chat box or an app.

3. Updated Galaxy

Community now includes the new Galaxy 5 UI, including a search history widget + dashboard, cleaned up admin interface, built-in activity analytics and more.

That is the release. Free, self-hosted, and now model-agnostic.