DEV Community: Elise Moreau

Best Cloudflare AI Gateway Alternative in 2026

Elise Moreau — Tue, 02 Jun 2026 11:57:17 +0000

Cloudflare AI Gateway has earned its place as an accessible on-ramp for teams that want to proxy and observe LLM traffic without much setup friction. Free-tier analytics, basic caching, and rate limiting make it a reasonable starting point for lightweight apps running inside Cloudflare's edge network. But when AI workloads mature into production systems, processing thousands of requests per second across multiple providers, enforcing governance policies, and demanding gateway overhead in the microseconds range, the architectural ceilings become hard to work around.

Bifrost is the best Cloudflare AI Gateway alternative for teams that need production-grade performance, enterprise governance, and full deployment flexibility.

The Gaps That Appear at Scale

Cloudflare AI Gateway functions as a centralized proxy, but several constraints become significant as workloads grow:

Logging caps create operational blind spots. The free tier stores a maximum of 100,000 log events per month. The Workers Paid plan raises that ceiling to one million. Once either limit is reached, incoming requests stop being logged entirely, which means production teams lose request-level visibility exactly when traffic is heaviest.

Infrastructure costs are not as flat as they appear. The gateway itself carries no per-request charge, but it runs on Cloudflare Workers. At high volume, that means Workers billing kicks in: $0.30 per additional million requests and $0.02 per million CPU-milliseconds beyond the base allocation.

The ecosystem coupling creates migration risk. Cloudflare AI Gateway is built to run on Cloudflare's stack. Organizations not already invested in that ecosystem take on extra complexity and cost to adopt it, and any future migration requires rearchitecting the integration layer from scratch.

Enterprise governance coverage is limited. Recent Cloudflare updates added Unified Billing and basic content moderation. What's still missing: granular per-team budget controls, virtual key-level spend limits, role-based access control, and hierarchical cost management. These are table-stakes requirements for enterprise AI deployments.

No path to self-hosted deployment. Cloudflare AI Gateway runs exclusively on Cloudflare-managed infrastructure. Teams with data residency requirements, air-gapped environments, or regulatory constraints have no option to deploy it inside their own VPC or private cloud.

For prototype environments or low-volume projects already on the Cloudflare stack, these constraints may be workable. For teams running production AI at scale, they introduce real risk.

Why Bifrost Is the Right Alternative

Bifrost is an open-source AI gateway written in Go, built from the ground up for production AI infrastructure. It exposes a unified, OpenAI-compatible API across 1,000+ models from 12+ providers, including OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, Azure OpenAI, Mistral, Groq, Cohere, and Ollama.

Gateway Overhead Measured in Microseconds

Bifrost adds just 11 microseconds of overhead per request at 5,000 RPS on standard t3.xlarge instances, the lowest latency overhead measured across AI gateways in that class. Go's concurrency model eliminates the bottlenecks that affect Python-based gateways, and Bifrost avoids the infrastructure indirection of managed proxy layers entirely.

For real-time agents, customer-facing chatbots, and high-frequency tool-calling pipelines, that performance difference accumulates meaningfully at scale.

Governance at Every Level

Bifrost ships with a governance model designed around how enterprise teams actually work:

Budget controls scoped to virtual keys, teams, and individual customers, not just the top-level account
Rate limiting and access policies configurable per API key, per team, or per application
SSO via Google and GitHub for enterprise authentication
HashiCorp Vault integration for secure key management and rotation
Audit logging with no monthly caps, suitable for compliance and regulatory requirements

Automatic Failover Without Manual Intervention

Bifrost manages failure paths at the gateway level. When a provider is rate-limited, slow, or unavailable, automatic fallbacks route traffic to the next available provider or model with no downtime. Adaptive load balancing distributes requests across API keys and providers based on real-time availability and performance, and none of it requires manual intervention from the engineering team.

Semantic Caching That Reduces API Spend

Bifrost's semantic caching layer stores responses and retrieves them for future prompts that are semantically similar, not just exact string matches. For applications where users regularly ask overlapping questions, support bots, knowledge assistants, internal search tools, this approach captures a broader range of cache-eligible requests than basic prompt caching and meaningfully reduces spend on redundant provider calls.

A Native MCP Gateway for Agentic Applications

Bifrost includes built-in support for the Model Context Protocol, giving AI models a standardized interface to interact with external tools: filesystems, web search, databases, and custom services. LLM routing and MCP tool access run through the same gateway, removing the need for separate infrastructure components. Centralized tool governance controls which tools each team and application can access.

Real-Time Guardrails

Content filtering and safety guardrails run at the gateway layer, blocking unsafe outputs and enforcing compliance policies before responses reach end users. They operate in real time with no meaningful impact on request latency.

Observability Without Extra Infrastructure

Native Prometheus metrics, distributed tracing, and structured logging are built directly into Bifrost. No sidecars, no wrappers, no third-party integrations needed. Token counts, latency, error rates, and costs are all tracked at the per-request level, across models, teams, and environments. Paired with the Maxim AI observability platform, teams get a unified view across cost, latency, model behavior, and output quality.

Self-Hosted, Open Source, No Lock-In

Bifrost deploys via Docker, Kubernetes, or NPX and runs inside any infrastructure environment. A production-ready gateway is up in under 60 seconds:

npx -y @maximhq/bifrost

It is a drop-in replacement for OpenAI, Anthropic, and Google GenAI SDKs. One line change routes all traffic through Bifrost. Licensed under Apache 2.0 with no managed infrastructure dependency and no vendor billing surprises.

Feature Comparison: Bifrost vs. Cloudflare AI Gateway

Capability	Cloudflare AI Gateway	Bifrost
Gateway Overhead	Workers-dependent (variable)	11µs at 5,000 RPS
Self-Hosted Deployment	No (Cloudflare-managed only)	Yes (Docker, K8s, NPX)
Log Storage Limits	100K free / 1M paid per month	Unlimited (Prometheus + structured logs)
Budget Controls	Account-level only	Per-key, per-team, per-customer
MCP Support	No	Native MCP gateway
Caching	Basic prompt caching	Semantic similarity-based caching
Open Source	No	Yes (Apache 2.0)
Guardrails	Basic content moderation	Real-time safety and compliance guardrails
Failover	Retry + model fallback	Automatic multi-provider failover with adaptive load balancing

Who Should Move to Bifrost

Bifrost is built for teams that have hit the ceiling on what a managed, ecosystem-coupled gateway can offer:

High-throughput engineering teams that need microsecond-level gateway overhead without unpredictable Workers billing on top.

Enterprise organizations that require per-team budgets, RBAC, audit logging, SSO, and governance controls beyond what a basic proxy layer provides.

Agentic application builders that need unified LLM routing and MCP tool access through a single gateway rather than stitching together separate components.

Teams with data residency or compliance requirements that must self-host within their own VPC or air-gapped environment.

Cost-focused teams looking to reduce LLM API spend through semantic caching and intelligent fallback routing across providers.

Start Using Bifrost

Bifrost is open source and free to deploy. It takes under 60 seconds to get a production-ready gateway running locally.

Try Bifrost →
Book a Demo →
GitHub →
Documentation →

Our LiDAR detector spent 40% of its time in voxelization, not convs

Elise Moreau — Tue, 02 Jun 2026 05:40:34 +0000

TL;DR: We profiled a LiDAR object detector expecting the 3D backbone to dominate. It didn't. Voxelization plus the scatter-to-pillars step ate roughly 40% of per-frame latency on an A100, and pulling them out of the Python hot path took our p50 from 31ms down to 19ms.

The assumption that cost us two weeks

When I was at Valeo.ai we ran a PointPillars-style detector on nuScenes-scale point clouds, around 250k points per sweep. The mental model everyone carried was simple. The sparse conv backbone is heavy, so the backbone is where the milliseconds go. We spent a sprint trying to prune channels and fuse BatchNorm into the conv weights before anyone actually looked at a trace.

When we finally ran torch.profiler with CUDA activities enabled, the picture was not what we expected. The 2D CNN head and the pillar feature net were fast. The expensive part lived upstream, in the code nobody thought of as "the model."## What the trace actually showed

To be precise, two things dominated. First, the voxelization that buckets raw points into a fixed grid of pillars. Second, the scatter operation that writes encoded pillar features back into a dense BEV canvas before the 2D backbone runs.

Here is the kind of profiling output that made us stop:

import torch
from torch.profiler import profile, ProfilerActivity

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
) as prof:
    for _ in range(50):
        detections = model(point_cloud)
    torch.cuda.synchronize()

print(prof.key_averages().table(
    sort_by="cuda_time_total", row_limit=12))
# voxelize_cpu          12.4 ms  (self CPU)
# scatter_nd            3.1 ms   (CUDA)
# sparse_conv_backbone  9.8 ms   (CUDA)
# rpn_head              4.2 ms

The voxelization ran on CPU in a Python loop. Every frame paid a host-to-device copy after the points were bucketed, and the CPU work serialized against the GPU instead of overlapping with it. The backbone was not the problem. The plumbing around it was.## Moving the work to where the data already lives

The fix was not exotic. We replaced the CPU voxelizer with a GPU implementation from spconv 2.x, kept the point cloud resident on the device from the sensor decode onward, and let the scatter run as a single fused kernel instead of an indexed assignment in Python.

The nuance here is that the win came from removing a synchronization point, not from making any single kernel faster. Once voxelization happened on-device, the CPU could prepare frame N+1 while the GPU finished frame N. That overlap is invisible in a microbenchmark of one op and very visible in end-to-end p50.

Stage	Before (CPU voxelize)	After (GPU voxelize)
Voxelization	12.4 ms (CPU)	1.9 ms (CUDA)
Scatter to BEV	3.1 ms	1.4 ms
Sparse conv backbone	9.8 ms	9.6 ms
RPN head	4.2 ms	4.1 ms
p50 end-to-end	31 ms	19 ms

The backbone barely moved, which is the whole point. We had been optimizing the one part of the pipeline that was already efficient.

The labeling side had the same shape of bug

While we were in there, we hit a related issue in our auto-labeling loop. We used a VLM to spot-check a sample of frames where the detector confidence was low, around 3% of 1.2M frames. The captioning calls were a different kind of bottleneck, one that came from rate limits and the occasional provider timeout rather than a kernel.

We put those calls behind a gateway so a failed request to one provider would fail over to another without us babysitting it. Bifrost (https://github.com/maximhq/bifrost) was the one we landed on, mostly because it spoke an OpenAI-compatible API and we didn't want to rewrite the client. There are other options in that space. The lesson was the same as the LiDAR one. The slow part was rarely the model itself.

Trade-offs and limitations

Moving voxelization to the GPU is not free. The spconv GPU path holds more device memory for the hash tables that map points to voxels, so on a memory-constrained Jetson Orin we had to drop the max points-per-pillar from 32 to 20 to fit, which cost us about 0.4 mAP on the moderate split. On an A100 that tradeoff never came up.

There is also a portability cost. A CPU voxelizer runs anywhere. The GPU version pins you to a CUDA toolkit version that matches your spconv build, and we burned an afternoon on a mismatch between CUDA 11.8 and a wheel built for 12.1.

And profiling itself can mislead. CUDA kernels launch asynchronously, so without an explicit torch.cuda.synchronize() before reading timings, you measure launch overhead instead of real work. Half our early numbers were wrong for exactly this reason.

What I'd tell my past self

Profile before you optimize, and profile the whole pipeline, not the layer you find intellectually interesting. The detector network is the part you publish papers about. The preprocessing is the part that ships in production and quietly dominates latency.

For LiDAR specifically, watch the boundary between CPU and GPU. Every host-to-device copy per frame is a stall, and stalls hide from per-op benchmarks. The model is usually the easy part.

Centralising tool access for our prompt-assembly agent with Bifrost MCP gateway

Elise Moreau — Mon, 01 Jun 2026 14:55:36 +0000

TL;DR: Before our SDXL stack renders a single product photo, a small LLM agent assembles the request from product metadata, a template lookup, and a brand-colour database. Wiring those tools separately for each provider kept drifting. Bifrost's MCP gateway let us register the tools once and keep them when we fail over from GPT-4o-mini to Claude. Below is what it cost, and where LiteLLM and Portkey would honestly have served us better.

The step nobody benchmarks in a diffusion pipeline

At Photoroom I work on the diffusion stack that generates product photography. The denoising loop gets all the attention. The model is the easy part.

Before any of that runs, a small agent assembles the generation request. It reads a product metadata JSON off our object store, looks up a background template by category in Postgres, and sometimes runs a web search to pin down a brand's palette. Three tool calls, maybe 40 tokens of reasoning, then it emits a structured prompt and a set of conditioning parameters for SDXL.

To be precise, that agent is gpt-4o-mini about 90% of the time. It's cheap and the latency budget is roughly 600ms. The nuance here is that when OpenAI rate-limits us during a traffic spike, we fail over to claude-haiku, and the agent has to keep working with the exact same tools.

Why per-provider tool wiring broke

We had every tool defined twice. OpenAI function-calling schemas and Anthropic's tool blocks are close but not identical, and our 9 tools lived in both formats inside the agent service.

That duplication drifted. Someone would add a region argument to the template-lookup tool on the OpenAI path and forget the Anthropic one. The failover path then silently produced worse prompts, which we only caught because a reviewer noticed bland backgrounds in a sample set. Hard to test. Easy to miss.

The deeper issue: tool execution was glued into the agent process. Filesystem reads, the Postgres query, the web search, all ran inline, so each provider integration re-implemented the same plumbing.

What the MCP gateway changed

Bifrost exposes a Model Context Protocol gateway that lets models call external tools (filesystem, web search, databases) registered once at the gateway rather than per provider. We moved our tool definitions there. The agent now talks to one OpenAI-compatible endpoint, and the same tool set is presented whether the request lands on GPT-4o-mini or Claude.

Config looks like this:

providers:
  openai:
    keys:
      - value: env.OPENAI_KEY
  anthropic:
    keys:
      - value: env.ANTHROPIC_KEY

fallbacks:
  - openai/gpt-4o-mini
  - anthropic/claude-haiku

mcp:
  servers:
    - name: photoroom-tools
      tools: [template_lookup, brand_palette, metadata_read]

Failover and the tool surface are now described in the same file. When automatic fallback kicks in, the tools come along. No second schema to keep in sync.

One side benefit we didn't plan for: the Prometheus metrics Bifrost emits gave us per-tool call counts. We learned the web-search tool fired on only 4% of requests but accounted for a third of agent latency. We've since gated it behind a confidence check.

How it compares

I evaluated Bifrost against LiteLLM and Portkey, which we'd both used before. None of these is strictly best.

Concern	Bifrost	LiteLLM	Portkey
MCP tool gateway	Built in, registered once	Not a first-class feature	Limited
Provider breadth	23+	Largest provider list	Broad
Runtime	Go, low overhead	Python proxy	Hosted-first
Observability	Native Prometheus	Callbacks, more wiring	Strong hosted dashboard
Self-host simplicity	`npx` or Docker	pip + config	Possible, cloud-leaning

LiteLLM is the honest pick if your differentiator is provider coverage or you live entirely in Python and want callbacks in-process. Portkey's hosted analytics are more polished than anything I'd build on top of raw Prometheus, and their guardrails UI is genuinely nicer for non-engineers. We chose Bifrost because the MCP gateway matched our specific shape, an agent with shared tools across a failover path, and because the Go proxy added under a millisecond at p50 in our load tests.

Trade-offs and limitations

The MCP gateway centralises tool definitions, but it also centralises a failure mode. If the gateway is down, both the model call and the tools go with it. We run two replicas behind a load balancer and treat the gateway as a tier-1 dependency now, which is more operational weight than the old inline tools carried.

Debugging a tool call across the MCP boundary is harder than a local function. A bad Postgres query used to throw inside our process with a clean stack trace. Now I read gateway logs and correlate by request ID. Workable, not free.

And the comparison above is not eternal. LiteLLM was adding MCP support when I last checked, so the gap here may close. Evaluate against current versions, not this table.

Semantic caching, which Bifrost also offers, did nothing for us on this step. Each request is keyed on a unique product, so cache hit rate was near zero. We left it off. Worth saying plainly, since the feature is often pitched as a default win.

torch.compile recompiled our SDXL UNet 38 times in production

Elise Moreau — Fri, 29 May 2026 05:38:13 +0000

TL;DR: torch.compile gave us a 2.3x speedup on our SDXL pipeline in benchmarks, then quietly recompiled 38 times across the first 100 production requests because every customer uploads a product photo at a different resolution. The fix wasn't turning compile off. It was understanding what counts as a guard, bucketing inputs to fixed shapes, and reading the recompilation logs PyTorch 2.3 gives you for free.

The benchmark that lied to me

At Photoroom we run diffusion models for product photography. Someone uploads a sneaker on a kitchen table, and the model gives it a clean studio background. The UNet is the heavy part, so when PyTorch 2.3 promised free speedups through torch.compile, I spent a week wiring it in.

The benchmark looked great. Fixed 1024x1024 input, batch size 4, an A10G. 2.3x faster than eager mode after warmup. I shipped it to a 5% canary.

p99 latency went up. Not by a little. Some requests took 70 seconds longer than before the change.

What a guard actually is

torch.compile traces a graph for a specific set of input properties. Tensor shapes, dtypes, certain scalar values, device. Dynamo wraps that graph in what it calls guards, which are cheap runtime checks that say "this compiled kernel is valid only if the next input matches". Miss a guard, and it recompiles.

Compiling the SDXL UNet takes 40 to 90 seconds on an A10G. That cost is fine once, at startup. The nuance here is that it happens lazily, inside the first request that violates a guard. So the recompile lands in the middle of a customer waiting for their image.

And product photos do not have a fixed shape. Phones shoot 3024x4032, someone crops to 800x600, a Shopify export is 1200x1200. Every new resolution is a new shape, a new guard miss, another recompile mid-request.

Watching the recompiles happen

The thing that saved me was not a profiler. It was one environment variable.

TORCH_LOGS="recompiles" python serve.py

That prints a line every time Dynamo throws away a compiled graph, with the reason:

Recompiling function forward in unet.py:412
    triggered by the following guard failure(s):
    - tensor 'L['''sample''']' size mismatch at index 2.
      expected 128, actual 96

Index 2 is the latent height. A 1024px image becomes a 128-wide latent, a 768px image becomes 96. Different shape, recompile. I counted 38 distinct recompiles before the cache stabilized, and it never fully stabilized because new resolutions kept arriving.

Three ways to stop it

I tested three approaches over a week against real traffic from our logs. Here's what held up.

Approach	Recompiles after warmup	Speedup kept	Main cost
`torch.compile(model, dynamic=True)`	0	~1.6x	More general kernel, slower per step
Resolution bucketing	3 (warmed at boot)	~2.1x	Padding pixels wasted through VAE
Fixed canonical resolution	0	2.3x	Quality loss on extreme aspect ratios

torch.compile(model, dynamic=True) tells Dynamo to assume shapes vary from the start. No per-shape recompiles, but you pay with a more general kernel. We measured 1.6x instead of 2.3x. Honest, predictable, leaves speed on the table.

Bucketing won. We resize and pad every input so the long edge lands on one of {768, 1024, 1280}, then compile each bucket once at boot before the readiness probe goes green.

BUCKETS = [768, 1024, 1280]

def to_bucket(img):
    long_edge = max(img.height, img.width)
    target = min(BUCKETS, key=lambda b: abs(b - long_edge))
    return resize_and_pad(img, target)

# Warm all three at startup, before serving traffic
for b in BUCKETS:
    dummy = torch.zeros(1, 4, b // 8, b // 8, device="cuda")
    compiled_unet(dummy, t=torch.tensor([1.0], device="cuda"))

Three compiles total, all before the pod takes traffic. Cache hits forever after.

The part that isn't the UNet

There's a small LLM step before the diffusion model even runs. It rewrites the user's text prompt into something the model handles better, turning "make it look nice" into a structured scene description. Those calls go out to an external provider, and when that provider has a bad minute the whole render stalls behind it.

We route that step through an AI gateway so a provider hiccup fails over to a backup instead of blocking. We use Bifrost for this, though several gateways do automatic fallback. It's one box in the pipeline diagram, not the interesting one, but it kept the prompt-rewrite step from becoming a single point of failure once compile fixed the UNet side.

Trade-offs and Limitations

Bucketing wastes compute on padding. An 800x600 image padded to 1024 pushes roughly 60% extra pixels through the VAE decode. For us that's worth a stable p99. For a catalog that's all square crops, it'd be pure overhead, and dynamic=True would be the saner call.

Warmup adds about 4 minutes to pod startup. Our Kubernetes readinessProbe has initialDelaySeconds set to wait it out, which slows autoscaling response during traffic spikes. You feel this when a flash sale hits.

The Inductor compile cache is per-process by default. Every new pod recompiles from scratch. You can point TORCHINDUCTOR_CACHE_DIR at a shared volume to persist it, but the cache is keyed loosely on environment, and a mismatched CUDA driver version across nodes gave us a silent fallback to eager once. Test that path before you trust it.

And none of this helps if your bottleneck is the VAE or the scheduler loop. Profile first. I burned two days compiling a UNet that was already 40% of wall-clock.

Tracing our 4-stage product photo pipeline through Bifrost

Elise Moreau — Thu, 28 May 2026 14:55:01 +0000

TL;DR: We added OpenTelemetry tracing across the four LLM and VLM hops in our product-photo pipeline by routing them through Bifrost. Pipeline-level p95 went from 11.2s to 6.8s in two weeks, mostly because we could finally see which step was the bottleneck. The tracing was free once the gateway was in place; we weren't going to instrument four SDKs by hand.

Where the time went was a mystery

At Photoroom we run a four-stage pipeline for catalog photo cleanup. Background removal with an in-house model. Inpainting via SDXL with our internal LoRA. Upscaling through Real-ESRGAN. A caption step that hits an external VLM provider. Four hops, two of them outside our VPC.

A bug report sat in our queue for a week: "the pipeline is slow on Tuesdays." Designers were timing out at 12 seconds. Our internal Grafana showed the GPU jobs were fine. The external VLM call latency? Nobody had a number.

To be precise, we had per-service latency in Datadog, but the spans didn't stitch together across the external API hops. We could see step 1 took 320ms and step 4 took "8 to 10 seconds" but the "to" in there was the entire problem.

What we changed

I'd been evaluating Bifrost (github.com/maximhq/bifrost) for an unrelated project: semantic caching on the caption step. While reading the observability docs I noticed Prometheus metrics and OTel export were native, not plugins.

So I rewired both external calls (caption VLM and a prompt-rewrite call we make before SDXL) to go through Bifrost instead of directly to the provider. The Python change was about 9 lines. Swap the base URL.

The relevant slice of our config.json:

{
  "providers": {
    "openai": {
      "keys": [{"value": "env.OPENAI_KEY", "models": ["gpt-4o-mini"]}],
      "concurrency_and_buffer_size": {"concurrency": 32}
    },
    "anthropic": {
      "keys": [{"value": "env.ANTHROPIC_KEY", "models": ["claude-haiku-4-5"]}]
    }
  },
  "observability": {
    "prometheus_labels": ["x-team", "x-pipeline-stage"],
    "otel_endpoint": "http://otel-collector:4318"
  }
}

The two custom headers (x-team, x-pipeline-stage) attach to every Prometheus sample and every OTel span. In Tempo I can filter by pipeline stage and see exactly which hop is slow.

What we found

The Tuesday slowness was a regional capacity blip with our caption provider. p99 for the caption call was 14 seconds with no error, only slow token output. Once Bifrost sat in front of it we configured a fallback to a second provider and the 12-second pipeline timeouts disappeared.

Full p95 numbers, two weeks before and two weeks after:

Stage	p95 before	p95 after	Why
Background removal	340ms	340ms	Untouched, on our GPUs
Prompt rewrite	1.1s	480ms	Semantic cache hit ~62%
SDXL + LoRA	4.2s	4.2s	Untouched
Caption VLM	6.8s	1.4s	Failover plus cache
End-to-end	11.2s	6.8s

The semantic cache hit rate on the prompt-rewrite step is high because designers re-run very similar product descriptions. That was latency we didn't know we were leaving on the table.

How it compares to what we considered

We looked at LiteLLM and Portkey first. We already had a half-deployed LiteLLM somewhere in the cluster.

	LiteLLM	Portkey	Bifrost
Self-host	Yes	Yes (some features gated)	Yes
Prometheus native	Plugin	Yes	Yes
OTel spans	Plugin	Yes	Yes
Semantic cache	Yes	Yes	Yes
MCP support	No	Limited	Yes
Measured overhead	~9ms	cloud RTT dependent	~1.2ms

The nuance here is that Portkey is excellent if you are happy on their cloud. We couldn't be, because some product images carry customer-identifiable data and the entire pipeline runs inside our VPC. LiteLLM is mature and its community is larger than Bifrost's, but the observability story required wiring up the Prometheus plugin myself. Bifrost ships that out of the box per the docs.

Trade-offs and limitations

Bifrost is younger than LiteLLM and its plugin ecosystem is correspondingly smaller. We wrote a 40-line Go plugin to attach Photoroom-internal request IDs to spans. It works but it's one more thing we own.

The MCP feature is genuinely useful for tool-using agents. We aren't using it yet. Our pipeline isn't agentic; if you only call chat completions you can ignore MCP entirely.

Latency overhead is low (~1.2ms in our measurements) but it isn't zero. For an image pipeline where each hop is hundreds of milliseconds it's invisible. For a high-QPS embedding service it might matter. Benchmark in your own environment before assuming.

One operational note: the OTel exporter sends spans synchronously by default. We had to bump the batch interval to 5 seconds; otherwise the gateway pod's CPU climbed under load.

Semantic caching the VLM step in our product-photo pipeline

Elise Moreau — Wed, 27 May 2026 14:52:40 +0000

TL;DR: We put Bifrost in front of the VLM step that captions and rewrites prompts for our product-photo diffusion pipeline. Semantic caching cut that bill by ~62% in three weeks. The diffusion side, where the GPUs live, was never the cost we should have been worrying about.

The bill that surprised us

Our pipeline at Photoroom (paraphrased, not exact internal numbers) does three things per product image. A vision-language model reads the input and produces structured captions. A second LLM call rewrites the user's prompt into something the diffusion model behaves well with. Then SDXL with our internal LoRAs does the actual generation on our own A100s.

The diffusion step is what we obsess over. To be precise, it is what we benchmark and profile every sprint. So when we looked at the Q1 numbers, the surprise was that Claude and Gemini Vision together cost more than the GPU lease for the same workload. The VLM and prompt-rewrite layer was 58% of total inference spend.

The nuance here is that we had been calling the providers directly from a Python service with no caching. Same product image, same user request. The response paid for again.

Why we chose Bifrost over the alternatives

I looked at LiteLLM and Portkey first. Both are good. LiteLLM is the path of least resistance if you want a Python library inside an existing FastAPI service, and its provider coverage is excellent. Portkey has a polished hosted UX and very clean dashboarding.

We landed on Bifrost for three reasons specific to our setup. It runs as a Go binary, which means the gateway isn't competing for the same GIL-bound CPU as our inference service. Semantic caching is built in rather than an add-on. The OpenAI-compatible endpoint meant we didn't need to change any of our SDK calls, as documented here.

Honest comparison. LiteLLM has a larger Python ecosystem footprint and its routing config will feel more native if your stack is Python-first. Portkey's analytics UI is, frankly, prettier than what we get out of the box.

The setup

Bifrost runs as a sidecar next to the prompt-rewrite service. Both the captioning and rewrite calls now go through http://bifrost:8080/v1/chat/completions. Our config is small.

providers:
  anthropic:
    keys:
      - value: env.ANTHROPIC_KEY_PRIMARY
      - value: env.ANTHROPIC_KEY_BACKUP
  google_vertex:
    keys:
      - value: env.VERTEX_KEY

semantic_cache:
  enabled: true
  similarity_threshold: 0.94
  ttl_seconds: 86400

fallbacks:
  - primary: anthropic/claude-3-5-sonnet
    fallback:
      - google_vertex/gemini-1.5-pro

governance:
  virtual_keys:
    - id: vk_caption_team
      budget_usd_monthly: 800
    - id: vk_rewrite_team
      budget_usd_monthly: 400

Three things matter here. The cache threshold of 0.94 was tuned against a held-out set of 5,000 captioning calls. At 0.97 we missed too many obvious duplicates. At 0.90 we started returning captions that were close but wrong about colour, which is unforgivable for an e-commerce use case. The fallback list isn't theatre. We measured Anthropic 5xx rates of 0.4% over March, which on our volume is real customer-visible failures.

Numbers from week three

Metric	Before	After
Cache hit rate (caption)	0%	71%
Cache hit rate (rewrite)	0%	49%
p95 latency, caption step	1.8s	0.31s
Monthly VLM+LLM spend	baseline	-62%
Provider failover events handled	0 (we returned an error)	14

The rewrite step caches less well because user prompts vary more. Captioning is the big win, because product photos from the same merchant cluster heavily in embedding space. Roughly 70% of merchants in our top tier upload 80% of their catalogue images within a 90-day window.

What we actually changed in code

The migration was unromantic. Two lines.

client = openai.OpenAI(
    base_url="http://bifrost:8080/v1",
    api_key=os.environ["BIFROST_VIRTUAL_KEY"],
)

Everything else stayed. The VLM team didn't touch their code. The rewrite team flipped a config flag.

Trade-offs and Limitations

Semantic caching has a real failure mode. If your downstream model output is meant to vary across calls (creative generation, sampling-heavy use cases) you don't want this on. We disable it for the diffusion-prompt-suggestion endpoint that gives editors three variants. The cache would happily return the same triplet twice.

The Go binary is one more service to operate. For a small team this is non-trivial. LiteLLM-as-a-library has fewer moving parts if you don't need the cache.

Cost attribution through virtual keys is per-key, not per-end-customer-of-our-customers. If you need full multi-tenant chargeback down to the merchant level, you will write some glue.

The semantic cache uses an embedding model itself. Read the docs on what backs it before you assume your prompts stay inside your VPC.

The bf16 grad accumulator that killed our SDXL LoRA training

Elise Moreau — Wed, 27 May 2026 05:37:20 +0000

TL;DR: Our SDXL LoRA fine-tune for a Photoroom product photography model trained for six days while silently corrupting its adapter weights. The cause was bf16 gradient accumulation interacting badly with a custom adapter init we'd ported from a paper. Eval scores stayed in the same range the whole time, which is why nobody noticed.

The setup

We train SDXL LoRAs for product photography categories at Photoroom. Bottles, packaged food, soft goods. Each LoRA is 192MB. Training stack: PyTorch 2.3, bf16 mixed precision, gradient accumulation across 8 steps, A100 80GBs.

The LoRA init follows a small modification of the OFT paper for better stability on small datasets. To be precise, we orthogonalize the down-projection before training begins, then let the up-projection drift freely. This had been working for nine months.

What broke

Six days into a 7-day run, our automated CLIPScore check started showing variance that was technically inside our acceptance band but trending the wrong way. The nuance here is that our eval pipeline grades generations using a fan-out across three VLM providers (Claude vision, GPT-4o, Gemini 1.5) routed through an LLM gateway. We use Bifrost for that fan-out, which gives us provider-level failover when one of them rate-limits us mid-grade. Useful, and uneventful. The grade scores looked fine.

The real signal was a per-step gradient norm log we'd turned off a quarter earlier when it was spamming the dashboard. When I turned it back on for a sanity check, the grad norms had been collapsing to ~1e-5 in the down-projection layer since step 12,000.

Tracing it back

I added a hook to dump the raw bf16 gradient tensors before they hit the accumulator:

import torch

def grad_dump_hook(name):
    def hook(grad):
        finite = torch.isfinite(grad).all().item()
        absmax = grad.abs().max().item() if finite else float("nan")
        if absmax < 1e-4 or not finite:
            print(f"[{name}] finite={finite} absmax={absmax:.2e}")
        return grad
    return hook

for n, p in lora.named_parameters():
    if p.requires_grad and "lora_A" in n:
        p.register_hook(grad_dump_hook(n))

Output across 200 steps:

[blocks.7.attn1.lora_A] finite=True absmax=4.32e-06
[blocks.8.attn1.lora_A] finite=True absmax=2.11e-06
[blocks.9.attn1.lora_A] finite=True absmax=8.54e-07

Gradient magnitudes in bf16 land are bounded below by roughly 6e-8 before they round to zero. We were producing real gradients, but most of them were being silently quantized to zero during the accumulation step. The accumulator in our custom training loop accumulated in bf16, not fp32.

This is documented behavior. Standard PyTorch grad accumulation in Accelerate uses fp32 accumulators by default. Our custom loop, forked from an internal repo two years ago, did not.

The fix

# before
optim.param_groups[0]["grad_accumulator_dtype"] = torch.bfloat16

# after
optim.param_groups[0]["grad_accumulator_dtype"] = torch.float32

Single line. Six days. We re-ran training with fp32 accumulation. Grad norms stabilized in the expected 1e-3 to 1e-2 range. Eval scores moved up by ~6% in our internal background-consistency metric.

Why nothing else caught it

A few obvious questions:

Why didn't loss curves show it? They did, mildly. The loss was still going down, only slower. Within noise of a normal run.

Why didn't the VLM eval catch it? Because the generations were still "good." Product on a clean background, lighting roughly correct. The drift was in finer details (brand text legibility, soft-good fabric texture) that our three-VLM grading averages out. We're now adding a per-category CLIPScore-vs-reference check that runs without averaging.

Why did we trust the init? We had nine months of green runs. The OFT-style init only became a problem when we tightened the LR schedule three weeks ago, which made the gradient magnitudes smaller across the board.

Trade-offs and limitations

Approach	Memory cost	Bookkeeping
bf16 accumulator	baseline	low
fp32 accumulator (all params)	+4% peak	low
fp32 only for LoRA params	+0.6% peak	painful

The fp32 accumulator costs us ~4% more GPU memory per step. Not free. On the A100 80GBs it's invisible, but if you're tight on the H100 80GBs or sharing with another job, you'll feel it.

You can accumulate in fp32 only for the LoRA params and keep the base model gradients in bf16, but the bookkeeping is annoying. We took the 4% hit.

The deeper lesson: a custom training loop that worked for nine months is not a training loop you understand. It's a training loop that hasn't been stressed in the right place yet. I should have re-read it when we changed the LR schedule.

Per-customer budget caps on our caption pipeline: 3 weeks with virtual keys

Elise Moreau — Tue, 26 May 2026 14:53:00 +0000

TL;DR: We were burning around €4,200/month in vision-LLM costs across roughly 80 customers, with zero way to tell who was responsible. Bifrost virtual keys plus per-customer budgets gave us hard caps and clean attribution in a couple of days. Semantic caching saved another 34%, though it needed more babysitting than the README implies.

At Photoroom we ship diffusion-based product photography to about half a million users. The diffusion side is our own infra, GPU pool, custom UNet, the lot. The part nobody writes blog posts about is the captioning and safety-filtering step that runs before each generation, plus a prompt rewriter. Those calls go out to OpenAI's gpt-4o-mini and Anthropic's claude-haiku-4-5-20251001 depending on the route.

For months we treated those calls as overhead. Two API keys, one invoice per provider, no real attribution. Then in April our bill went from €1,800 to €4,200 in three weeks. To be precise: nothing had launched. A single enterprise customer was retrying caption generation in a loop because their pipeline interpreted our 429s as transient.

That was the trigger to put something in front of the providers.

What we tried first

The obvious first pass was a small Python proxy. Maybe 150 lines, sitting between our worker and OpenAI. It worked. For about a week. Then we needed per-customer rate limits, then budget caps, then a second provider, then someone in finance asked for usage reports by customer ID.

This is the point where most teams adopt a real gateway. We compared three.

Feature	Bifrost	LiteLLM	Portkey
Self-hostable	Yes	Yes	Yes (paid tier)
OpenAI-compatible endpoint	Yes	Yes	Yes
Per-customer virtual keys	Yes	Yes	Yes
Hierarchical budgets (key → team → customer)	First-class	Limited	Yes
Semantic caching built-in	Yes	Plugin	Yes
Prometheus metrics out of the box	Yes	Add-on	Hosted dashboard
Web UI for config	Functional	Minimal	The nicest

Portkey's dashboard is honestly the nicest of the three. LiteLLM has the longest tail of niche providers. We picked Bifrost because we wanted a self-hosted box where the budget hierarchy was first-class, and we wanted customer IDs to stay out of any third-party SaaS for GDPR reasons.

Setting up virtual keys

Step one was running Bifrost as a sidecar to our Python workers.

docker run -d \
  -p 8080:8080 \
  -v $(pwd)/config.json:/app/config.json \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  maximhq/bifrost

Then a virtual key per customer, with a monthly cap and an allow-list of models:

virtual_keys:
  - id: vk_acme_corp
    customer_id: acme_corp
    allowed_models:
      - openai/gpt-4o-mini
      - anthropic/claude-haiku-4-5-20251001
    budget:
      monthly_usd: 200
      hard_cap: true
    rate_limit:
      requests_per_minute: 60

Our worker now sends Authorization: Bearer vk_acme_corp instead of the raw provider key. When acme_corp hits €200, the gateway returns a typed 429 and we surface "monthly quota reached" in the customer's UI. The customer that triggered the April spike is capped at €150 now, and we sleep at night.

The whole rollout took a long Tuesday. Most of that was rewriting our worker to thread the customer_id through every call, not the gateway itself.

Where semantic caching helped (and where it bit us)

Bifrost's semantic cache hashes the embedding of the prompt and returns a cached completion when cosine similarity exceeds a threshold. For caption generation on near-duplicate product photos, this is a big deal. We saw a 34% hit rate over the first 10 days.

The catch: our prompt includes brand and SKU for context, but the brand string was occasionally elided when the customer's metadata was incomplete. Two prompts differing only in brand: null vs brand: Nike hashed close enough to collide, and a Nike trainer got captioned with a placeholder description. Not great.

We fixed it by raising the similarity threshold from 0.92 to 0.97 and adding the SKU as a cache namespace prefix. Worth a proper ablation paper-style. We have not had time.

Trade-offs and Limitations

A few things to know before you adopt this.

The Bifrost UI is functional but newer than Portkey's. If your finance team wants a self-service dashboard they can poke at without engineer help, Portkey is ahead today. We plumbed Bifrost's Prometheus output into our existing Grafana instance instead, which took an afternoon.

LiteLLM has a longer tail of obscure providers. If you route to OpenRouter sub-models or self-hosted vLLM endpoints with non-standard URL shapes, check the docs first.

Hard budget caps are hard. A customer who hits the cap mid-batch will see a 429 even if they're €0.10 over. We added a soft-cap warning at 80% and an autoscale-up flow for enterprise contracts so the experience does not feel punitive.

The semantic cache will collide on prompts that differ in metadata you forgot to include in the embedding. Tune the threshold, namespace aggressively, and assume you will discover one edge case per week for the first month.

Cost accounting for diffusion image generation at $0.0008 per render

Elise Moreau — Mon, 25 May 2026 14:53:09 +0000

TL;DR: Per-image cost on our SDXL-based product photography pipeline at Photoroom dropped from $0.0031 to $0.0008 over six months. Most of the win came from boring infrastructure work, not model tricks. An AI gateway in front of our text-conditioning calls saved more than I expected.

I spent most of Q1 staring at a Grafana panel labelled cost_per_render_eur. Our diffusion pipeline generates background-replaced product images at volume. When marketing asks for a million renders, the per-image number matters.

To be precise: the cost I track is GPU-seconds on A100/H100 SXM nodes plus any external API calls plus storage IO. Not amortised salaries, not the office espresso machine. Just the marginal cost of one more render.

Where the money actually goes

Before I started measuring properly, I assumed the UNet denoising loop was 80%+ of the cost. It wasn't.

Stage	% of wall time	% of cost	Notes
Text encoder (CLIP + T5)	4%	11%	T5-XXL is expensive on H100
LLM caption rewriting	8%	22%	External API, GPT-4o-mini initially
UNet denoising (25 steps)	71%	48%	DPM++ 2M Karras
VAE decode	9%	7%	fp16, no tricks
Storage IO + image post	8%	12%	S3 multipart, sharpen, resize

The caption-rewriting step shocked me. We use an LLM to take a customer prompt like "white sneaker on beach" and expand it into a diffusion-friendly description with lighting, framing, camera details. That single API call was 22% of cost.

Killing the bill in three places

Step 1 — UNet quantisation to int8. Used torchao + a small calibration set of 512 product images. Quality drop measured by CLIP-similarity on a held-out set: 0.847 to 0.841. Negligible. Throughput went from 14 renders/sec to 23 renders/sec on an H100. That's a 39% cost drop on the dominant stage.

Step 2 — Caching the text-encoder outputs. For our product taxonomy, only about 4,000 unique caption stems exist (variations on "minimalist white background", "studio lighting from upper-left", etc.). T5-XXL embeddings for these are 14KB each. I cached them in Redis with a 30-day TTL. Hit rate after two weeks: 91%. Text-encoder cost dropped from 11% to 1.2%.

Step 3 — The gateway problem. This is where it got interesting.

The LLM caption step was the messy one

The caption-rewriting calls were originally direct OpenAI API hits from our Python ranking service. When OpenAI had a partial outage in late January (the one that affected gpt-4o-mini specifically for ~40 minutes), we lost 280k renders. The cost of those failed renders, billed but not delivered, was around €890.

I put Bifrost in front. The choice was between LiteLLM, Portkey, and Bifrost. I'll be honest about the comparison.

LiteLLM has wider provider coverage in the Python ecosystem and a more mature semantic-cache integration with langchain-style apps. If your stack is pure Python and you live inside LangChain, it's a more natural fit.

Portkey's UI for prompt management is genuinely nicer than what Bifrost ships, and their guardrail catalog has more pre-built rules.

I picked Bifrost because (a) it's a Go binary with a single HTTP endpoint and our caption service is Go, (b) the automatic fallbacks between providers work without me writing routing logic, and (c) the semantic caching layer sits at the gateway so my Python preprocessing service and Go caption service share the cache.

Config that replaced about 140 lines of fallback logic in our caption service:

providers:
  openai:
    keys:
      - value: env.OPENAI_KEY_PRIMARY
        weight: 0.7
      - value: env.OPENAI_KEY_BACKUP
        weight: 0.3
  anthropic:
    keys:
      - value: env.ANTHROPIC_KEY
        weight: 1.0

fallbacks:
  - primary: openai/gpt-4o-mini
    secondary: anthropic/claude-haiku-4-5
    tertiary: openai/gpt-4o-mini

semantic_cache:
  enabled: true
  similarity_threshold: 0.94
  ttl_seconds: 604800

The 0.94 similarity threshold matters. We tested 0.90, 0.92, 0.94, 0.96 on 10,000 caption pairs and measured downstream image quality. Below 0.94, the cached caption sometimes mismatched the product category enough to confuse the UNet. Above 0.96, hit rate dropped under 30% and the cost win disappeared.

Current numbers after one month with the gateway in place:

Caption API spend: down 61% (semantic cache hit rate of 47%)
Caption-step latency p95: 340ms to 110ms on cache hits
Failed render rate from upstream LLM issues: 0.31% to 0.04%
New cost share for captions: 22% to 8.2%

Trade-offs and Limitations

Quantisation to int8 cost me about three weekends of calibration tuning. For very high-end fashion shoots where we render at 2048x2048, the quality drop becomes visible in fine fabric weave. We keep an fp16 path for those.

The semantic cache occasionally returns a "close enough" caption that doesn't match a niche product category. For our long-tail (about 4% of requests), I disable the cache via a header per-call. The gateway supports this through request metadata.

Bifrost's clustering features are gated to enterprise, which fine for our scale, but if I were running this across three regions I'd want to evaluate that cost honestly. Portkey's pricing for similar features came in lower for the team-collaboration tier.

I haven't migrated the image-generation outputs themselves through the gateway. The UNet runs on our own GPUs, not behind an LLM API, so the gateway adds no value there. Don't put infrastructure in places it doesn't earn its keep.

Why your diffusion model is slow at batch size 1 (and what actually helps)

Elise Moreau — Fri, 22 May 2026 05:37:23 +0000

TL;DR: Single-image diffusion inference is bottlenecked by kernel launch overhead and attention memory traffic, not raw FLOPs. torch.compile with mode="reduce-overhead", a fused attention backend, and CFG batching get you most of the way before you reach for distillation.

I spend a lot of time looking at flame graphs from production diffusion pipelines. The pattern is almost always the same. The team profiles their model, sees 50 steps of a UNet or DiT, and assumes the path to lower latency is fewer steps. So they try LCM, then TCD, then some flavor of consistency distillation, and the quality drops in ways the product team notices.

The nuance here is that at batch size 1, your GPU is mostly idle. You are not compute-bound. You are launch-bound and memory-bound. Distillation helps eventually, but only after you have fixed the boring things.

What the profiler actually shows

Run a vanilla SDXL or a 1B-parameter DiT at 1024x1024, batch 1, on an H100. Capture a trace with torch.profiler and zoom into a single denoising step.

You will see something like this, roughly:

~30-40% of wall time inside attention kernels
~20-25% inside conv and linear layers
~15-20% in layernorm, GELU, residual adds
The rest: kernel launch gaps, host-to-device syncs, Python overhead

That last bucket is the embarrassing one. On an H100 a kernel launch costs ~5 microseconds. A UNet step fires hundreds of kernels. A 50-step sample fires tens of thousands. You are paying for the privilege of dispatching work, not for the work itself.

To be precise: at batch 1, the same model at batch 8 often runs in less than 2x the wall time. That gap is your overhead bill.

Step one: torch.compile, but the right mode

The default torch.compile(model) call uses mode="default", which optimizes for compile time and flexibility. For inference you want:

import torch

unet = torch.compile(
    unet,
    mode="reduce-overhead",
    fullgraph=True,
    dynamic=False,
)

reduce-overhead enables CUDA graphs, which replay a captured sequence of kernels in one launch. This is the single largest win for batch 1 diffusion on modern GPUs. In my measurements on PyTorch 2.3, this alone takes a 1024x1024 SDXL UNet step from ~42ms to ~28ms on H100. No quality change, no architecture change.

The catch: fullgraph=True will yell at you about any graph break. CFG implementations that branch on guidance_scale need rewriting. Custom samplers that touch .item() between steps will break CUDA graph capture. Plan for a day of fighting this.

Step two: pick an attention backend on purpose

PyTorch's scaled_dot_product_attention dispatches to one of several backends. The defaults are not always right for high-resolution diffusion.

Backend	Best for	Notes
FlashAttention-2	Long sequences, H100/A100	Default on most setups, good general choice
FlashAttention-3	H100 only	~1.5x faster than FA2 on Hopper, requires manual install
xFormers memory-efficient	Older GPUs (V100, T4)	Lower memory, slower than Flash on modern hardware
Math (fallback)	Debugging only	Never ship this

For DiT-style models at 2K resolution the sequence length per attention block hits 16K+ tokens. FA3 on H100 is a real difference there. I have seen 18% end-to-end latency drop on a 2B DiT just from switching FA2 to FA3 via torch.nn.attention.sdpa_kernel.

Step three: batch your CFG

Classifier-free guidance runs the model twice per step, once conditional and once unconditional. Most reference implementations call the UNet twice sequentially. Do not do this.

Concatenate the two prompts into one batch of 2, run one forward pass, split the output. On batch 1 this nearly halves your per-step latency because you were leaving the GPU idle anyway. The memory cost is negligible at typical inference resolutions.

This is a 3-line change and somehow lives in maybe 60% of the codebases I review.

Step four, only now: think about steps

After the above, a 50-step SDXL sample on H100 is in the 1.2-1.5 second range. If your product needs sub-second, then yes, look at LCM, Hyper-SD, or DMD2. But evaluate quality on your own data, not on the curated examples in the paper. Distilled models lose the most quality on the long tail of prompts your users actually send, particularly text rendering and fine compositional structure.

Trade-offs and limitations

CUDA graphs hate dynamic shapes. If your service accepts arbitrary aspect ratios you will recompile constantly. Either bucket aspect ratios into a small set of fixed shapes, or accept the warmup cost on cold paths.

reduce-overhead mode increases memory usage because it pins workspace buffers. On a 24GB consumer card this can push you over the edge with larger models. Profile before deploying.

FlashAttention-3 requires building from source against a specific CUDA version. If your deployment runs across mixed GPU generations, the version matrix becomes painful. Pick one backend per deployment target.

And the obvious one: none of this fixes a slow VAE decode. If you are generating at 2K, the VAE can dominate. Tiled VAE decoding or a distilled decoder like TAESD is a separate fight.

Routing diffusion inference traffic across three providers

Elise Moreau — Thu, 21 May 2026 14:52:31 +0000

TL;DR: We route a mix of diffusion and LLM traffic across three providers from a single Go-based gateway called Bifrost. The 11 microsecond overhead is real, the failover works, and the part I care about most (weighted routing for cost vs latency tradeoffs) finally stopped being a custom Python service nobody wanted to maintain.

I work on diffusion models for product photography. Most of what I write about is training, but the boring truth is that inference traffic management eats more of my week than I would like to admit.

We have three categories of model calls in production. Hosted diffusion endpoints for fallback when our own GPU pool is saturated. LLM calls for prompt rewriting and caption generation. And a small embedding service for similarity search on reference images. Three providers, three SDKs, three retry policies. It was becoming a mess.

What we had before

A Python FastAPI service in front of everything. It worked. It was also slow, and the team had stopped trusting the metrics because the gateway itself was adding 40-80ms of overhead depending on the day.

The nuance here is that for a diffusion call taking 3 seconds, 60ms of gateway overhead is noise. For a small LLM rewrite that should take 200ms, it is a third of your budget. We were optimizing the wrong axis.

I spent a weekend evaluating replacements. Kong felt heavy. LiteLLM was the obvious choice for the LLM side but does not really speak the dialect of provider-specific diffusion APIs we need. Then a colleague pointed me at Bifrost.

Why a Go gateway actually matters here

To be precise: the language is not the point. The point is the runtime model. Bifrost runs as a single Go binary, uses goroutines for concurrency, and the published overhead is around 11 microseconds per request. I measured it on our own staging hardware and got numbers in the same ballpark, which is rare enough that I noticed.

For our embedding service this matters. For diffusion it does not. But having one gateway that does not become the bottleneck for the fast calls is what made the consolidation possible.

providers:
  openai:
    keys:
      - value: env.OPENAI_KEY_PRIMARY
        weight: 0.7
      - value: env.OPENAI_KEY_SECONDARY
        weight: 0.3
    network:
      retry:
        max_retries: 2
        backoff_initial_ms: 100
  anthropic:
    keys:
      - value: env.ANTHROPIC_KEY
  stability:
    keys:
      - value: env.STABILITY_KEY

mcp:
  - id: prompt-rewrite
    primary: openai/gpt-4o-mini
    fallbacks:
      - anthropic/claude-haiku-4-5

That config replaced about 400 lines of Python.

The weighted routing thing

This is the feature I did not know I wanted. We have two OpenAI accounts because of rate limits and billing isolation between research and production workloads. Previously we ran two separate clients with manual round-robin logic that always had off-by-one bugs.

Weighted routing in the gateway just handles it. 70/30 split, configured declaratively, and when one key hits a 429 the failover kicks in without us writing the retry code ourselves. Virtual keys on top of that let us issue per-team credentials that map to the underlying provider keys, so the research team and the production team see different rate limits and different cost dashboards.

Comparison with what we considered

Capability	LiteLLM	Kong	Bifrost
Per-request overhead	~50ms (Python)	~5ms but heavy footprint	~11μs
Failover across providers	Yes	Plugin required	Yes, built-in
Weighted key routing	Limited	Custom plugin	Native
Semantic caching	Via plugin	No	Native
Diffusion provider support	Weak	Generic HTTP only	Provider-aware
Operational footprint	Python service	Lua plugins, DB	Single Go binary

LiteLLM remains excellent for pure LLM-only stacks. Kong is the right answer if you already run Kong. For us, the combination of low overhead and provider-aware routing was the deciding factor.

Semantic caching on prompt rewrites

About 40% of our prompt-rewrite calls are near-duplicates. Same product, slightly different angle, same desired caption style. We were paying for every one of them.

Bifrost has semantic caching built in, using embeddings to match similar requests within a configurable threshold. I was skeptical because cache invalidation on semantic similarity is famously a footgun. We set the threshold conservatively (cosine similarity above 0.94) and audit the cache hits weekly. Hit rate is around 22%, cost savings are real, and we have not had a quality complaint yet. The audit is the part nobody talks about, but you need it.

Trade-offs and Limitations

It is a young project. The documentation has gaps, particularly around custom provider plugins. I had to read the source to understand how the streaming response handling works for SSE-heavy diffusion APIs.

Observability is functional but basic. We forward to our existing Prometheus setup and it works, but if you expect a polished UI for traffic analysis you will be disappointed. We built our own Grafana dashboards.

Semantic caching is only as good as your embedding model and threshold tuning. If your prompts have high lexical variation but identical intent, you will get false negatives. If your prompts are templated and only the parameters change, you will get false positives. Test on your own traffic before trusting it.

And one honest note: an 11 microsecond gateway does not make a 3-second diffusion call faster. It just stops being the reason your fast calls are slow. Know which problem you are solving.

Why your diffusion model is slow at batch size 1 (and what actually helps)

Elise Moreau — Tue, 19 May 2026 05:37:02 +0000

What the profiler actually shows

Run a vanilla SDXL or a 1B-parameter DiT at 1024x1024, batch 1, on an H100. Capture a trace with torch.profiler and zoom into a single denoising step.

You will see something like this, roughly:

~30-40% of wall time inside attention kernels
~20-25% inside conv and linear layers
~15-20% in layernorm, GELU, residual adds
The rest: kernel launch gaps, host-to-device syncs, Python overhead

To be precise: at batch 1, the same model at batch 8 often runs in less than 2x the wall time. That gap is your overhead bill.

Step one: torch.compile, but the right mode

The default torch.compile(model) call uses mode="default", which optimizes for compile time and flexibility. For inference you want:

import torch

unet = torch.compile(
    unet,
    mode="reduce-overhead",
    fullgraph=True,
    dynamic=False,
)

Step two: pick an attention backend on purpose

PyTorch's scaled_dot_product_attention dispatches to one of several backends. The defaults are not always right for high-resolution diffusion.

Backend	Best for	Notes
FlashAttention-2	Long sequences, H100/A100	Default on most setups, good general choice
FlashAttention-3	H100 only	~1.5x faster than FA2 on Hopper, requires manual install
xFormers memory-efficient	Older GPUs (V100, T4)	Lower memory, slower than Flash on modern hardware
Math (fallback)	Debugging only	Never ship this

Step three: batch your CFG

Classifier-free guidance runs the model twice per step, once conditional and once unconditional. Most reference implementations call the UNet twice sequentially. Do not do this.

This is a 3-line change and somehow lives in maybe 60% of the codebases I review.

Step four, only now: think about steps

Trade-offs and limitations

reduce-overhead mode increases memory usage because it pins workspace buffers. On a 24GB consumer card this can push you over the edge with larger models. Profile before deploying.

And the obvious one: none of this fixes a slow VAE decode. If you are generating at 2K, the VAE can dominate. Tiled VAE decoding or a distilled decoder like TAESD is a separate fight.

DEV Community: Elise Moreau

Best Cloudflare AI Gateway Alternative in 2026

The Gaps That Appear at Scale

Why Bifrost Is the Right Alternative

Gateway Overhead Measured in Microseconds

Governance at Every Level

Automatic Failover Without Manual Intervention

Semantic Caching That Reduces API Spend

A Native MCP Gateway for Agentic Applications

Real-Time Guardrails

Observability Without Extra Infrastructure

Self-Hosted, Open Source, No Lock-In

Feature Comparison: Bifrost vs. Cloudflare AI Gateway

Who Should Move to Bifrost

Start Using Bifrost

Our LiDAR detector spent 40% of its time in voxelization, not convs

The assumption that cost us two weeks

The labeling side had the same shape of bug

Trade-offs and limitations

What I'd tell my past self

Further Reading

Centralising tool access for our prompt-assembly agent with Bifrost MCP gateway

The step nobody benchmarks in a diffusion pipeline

Why per-provider tool wiring broke

What the MCP gateway changed

How it compares

Trade-offs and limitations

Further reading

torch.compile recompiled our SDXL UNet 38 times in production

The benchmark that lied to me

What a guard actually is

Watching the recompiles happen

Three ways to stop it

The part that isn't the UNet

Trade-offs and Limitations

Further Reading

Tracing our 4-stage product photo pipeline through Bifrost

Where the time went was a mystery

What we changed

What we found

How it compares to what we considered

Trade-offs and limitations

Further reading

Semantic caching the VLM step in our product-photo pipeline

The bill that surprised us

Why we chose Bifrost over the alternatives

The setup

Numbers from week three

What we actually changed in code

Trade-offs and Limitations

Further Reading

The bf16 grad accumulator that killed our SDXL LoRA training

The setup

What broke

Tracing it back

The fix

Why nothing else caught it

Trade-offs and limitations

Further reading

Per-customer budget caps on our caption pipeline: 3 weeks with virtual keys

What we tried first

Setting up virtual keys

Where semantic caching helped (and where it bit us)

Trade-offs and Limitations

Further Reading

Cost accounting for diffusion image generation at $0.0008 per render

Where the money actually goes

Killing the bill in three places

The LLM caption step was the messy one

Trade-offs and Limitations

Further Reading

Why your diffusion model is slow at batch size 1 (and what actually helps)

What the profiler actually shows

Step one: torch.compile, but the right mode

Step two: pick an attention backend on purpose

Step three: batch your CFG

Step four, only now: think about steps

Trade-offs and limitations

Further Reading

Routing diffusion inference traffic across three providers