Zero-downtime deploys for our self-hosted LLM gateway

#sre #llm #devops #infrastructure

TL;DR: Getting zero-downtime deploys on a self-hosted LLM gateway comes down to readiness probes that gate traffic, connection draining on shutdown, and running more than one instance behind a load balancer. We did it with Bifrost on EKS.

A rolling upgrade of our self-hosted LLM gateway dropped roughly 0.4% of in-flight requests one afternoon because old pods stopped accepting connections before the replacements passed their readiness checks. We run Bifrost, the open-source AI gateway from Maxim AI, in front of every internal service at Buildkite that calls an LLM, so a botched upgrade hits more than one team at once. This post covers how we got to actual zero-downtime deploys on a self-hosted LLM gateway, the health-check wiring that made it work, and what still bites us.

What zero-downtime means for a self-hosted LLM gateway

Zero-downtime deploys for a self-hosted LLM gateway mean no in-flight request fails while you swap versions. You need readiness probes that gate traffic before a pod joins the pool, connection draining so terminating pods finish their work, and at least two instances behind a load balancer so the pool keeps serving while one node restarts. Miss any of those and you drop requests.

We had the load balancer part sorted from day one. The draining and probe wiring is where we got it wrong, and the failure mode was quiet: a handful of 502s during each deploy that nobody noticed until a downstream team's retry budget started complaining.

Why our first rolling upgrade dropped requests

The bug was the classic Kubernetes shutdown race. When a pod is deleted, kubelet sends SIGTERM and removes the pod from Service endpoints at the same time, but those two events are not ordered against in-flight load balancer routing. The Kubernetes pod termination docs spell out that endpoint removal and SIGTERM happen concurrently, so a request already in flight can still land on a pod that has begun shutting down.

Because Bifrost is a Go binary, it exits fast on SIGTERM, which made the race worse. The process was gone before the load balancer stopped sending it traffic. The fix was a preStop hook to hold the pod open long enough for endpoint propagation, plus a readiness probe so new pods only take traffic once they can actually serve. Running it as a drop-in replacement for the OpenAI base URL meant none of our services needed code changes while we sorted the deploy mechanics.

Rolling upgrades with clustering and load balancing

Here is the deployment shape that stopped the drops. The preStop sleep covers endpoint propagation, and terminationGracePeriodSeconds gives in-flight LLM calls time to finish, since a streaming completion can run for tens of seconds.

spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: bifrost
          image: maximhq/bifrost:latest
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            periodSeconds: 5
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "15"]

With maxUnavailable: 0 and maxSurge: 1, the new pod has to pass its readiness probe before an old one leaves. Across three replicas, the request pool never drops below full capacity. Bifrost's automatic fallbacks and load balancing handle provider-side weighting, but the deploy-side availability is plain Kubernetes mechanics that you have to get right yourself.

For multi-node coordination we use Bifrost clustering so config and rate-limit state stay consistent across instances during a rolling swap. On the throughput side, a single instance sustains 5,000 RPS with about 11µs of added latency per request per the published benchmarks, so three replicas gave us plenty of headroom to lose one mid-deploy without backing up the queue.

Observability so we trust the deploy

A deploy you cannot watch is a deploy you cannot trust. We wired Bifrost observability into our existing Prometheus stack and tag every request with a deploy-version label using the x-bf-lh-* metadata headers. The metrics path uses async writes with under 0.1ms overhead, so the logging itself does not skew the latency we are trying to protect during a rollout.

Now each rolling upgrade shows up as a clean version cutover on the dashboard, with the 5xx rate flat through the transition. When something does regress, the per-version labels tell us within one scrape interval whether the new image is the cause. The governance and metrics resources cover the label model in more detail than I will here.

Trade-offs and limitations

Clustering is an enterprise feature, not part of the open-source core, so a single-instance hobby setup will not give you the cross-node state coordination described above. Be honest about that before you plan a multi-node topology.

Self-hosting also means you own the deploy. Bifrost gives you the gateway, but the readiness probes, preStop timing, and grace periods are yours to tune, and the right terminationGracePeriodSeconds depends on your longest streaming completion. The ecosystem is younger than LiteLLM, so you will find fewer random blog posts when you hit an edge case, though the docs covered what we needed. If you want a fully managed control plane instead of running pods, this self-hosted path is more work than it is worth.

Wrapping up

Zero-downtime deploys for a self-hosted LLM gateway are mostly about ordering: readiness before traffic, drain before exit, surplus capacity before you remove a node. Bifrost made the gateway layer fast and boring enough that the only thing left to get right was our Kubernetes wiring. If you want to see the clustering and governance pieces in action, you can book a demo.