DEV Community: Sebastian Buzdugan

Governing Every LLM and MCP Call Across the Enterprise: Virtual Keys, Budgets, and Guardrails with the Bifrost AI Gateway

Sebastian Buzdugan — Thu, 09 Jul 2026 14:25:38 +0000

You have API keys in a vault, a budget dashboard, and a policy doc that lists which models are approved. Procurement signed off. On paper, enterprise AI is under control.

So answer one question: who called which model last night, with whose budget, through which MCP tool?

Nobody can say.

The calls nobody is counting

Enterprise AI usage did not arrive as one system. It arrived as a hundred small ones.

A data team wires GPT-5.5 into a pipeline with a key from the shared vault. A product squad ships a feature on Claude through its own SDK. An agent framework spins up MCP servers that call tools, read files, and hit internal APIs. Each of these is a real AI call leaving your infrastructure, and almost none of them pass through a single point where you could see them, price them, or stop them.

The result is not a security hole in the dramatic sense. It is worse: it is an accounting gap. You cannot answer basic questions. Which teams are spending what. Which models are actually in use versus approved. Which MCP tools an agent is allowed to invoke, and which it quietly invokes anyway.

Most teams try to close this with process. A spreadsheet of keys. A wiki page of rules. A quarterly review. Process does not enforce anything at request time, and request time is the only time that matters.

The fix is not another policy. It is a control plane that every LLM and MCP call has to pass through.

One base URL in front of every model

Bifrost is an open-source AI gateway built by Maxim AI, written in Go and licensed Apache 2.0. It sits in front of your model providers and gives you one OpenAI-compatible API for all of them: over 1000 models across every major provider behind a single endpoint.

The adoption cost is deliberately close to zero. You do not rewrite call sites. You change a base URL.

# Before: every app talks straight to the provider
client = OpenAI(base_url="https://api.openai.com/v1")

# After: the same call, now governed, logged, and routable
client = OpenAI(base_url="http://localhost:8080/openai")

That one line is the whole point. The SDK, the model names, the request bodies all stay the same. What changes is that the call now lands on something you control before it reaches OpenAI, Anthropic, Bedrock, Vertex, or any of the others.

The gateway is not a toll booth that slows everything down. You are buying governance, not latency: in Bifrost's own t3.xlarge benchmark, that control layer cost 11 microseconds of added overhead per request at 5,000 requests per second, and held a 100 percent success rate.

Standing it up locally is a single command:

npx -y @maximhq/bifrost

That pulls the binary, runs migrations, and serves a web UI plus the API on port 8080. I ran exactly this while writing, and everything below is what the running gateway actually did, not a paraphrase of the docs.

Governance that follows a hierarchy, not a spreadsheet

Once traffic flows through one place, governance stops being a document and becomes a data model. In Bifrost that model has four levels: Customer, Team, Virtual Key, and Provider Config.

The Virtual Key is the unit you hand out. It is not a raw provider key. It is a governed credential with a sk-bf- prefix that carries its own budget, rate limits, and allow-lists. I created one through the API and it came back live:

curl -X POST localhost:8080/api/governance/virtual-keys \
  -d '{"name":"marketing-team-vk","is_active":true,
       "rate_limit":{"token_max_limit":1000000,"token_reset_duration":"1h",
                     "request_max_limit":1000,"request_reset_duration":"1h"}}'

The response handed back sk-bf-82a7fb23... with a rate-limit object already ticking: token_current_usage: 0, an hourly reset window, request counters primed. That key is the thing a team codes against. Everything you attach to it travels with every call it makes.

Budgets cascade. A Provider Config can cap the OpenAI spend inside a key, the key has its own ceiling, the team above it has another, and every applicable budget has to pass for the request to proceed. Cost deducts from each level at once. Budgets reset on windows from one minute to one year, and calendar_aligned snaps them to real month and year boundaries instead of rolling clocks. Rate limits come in two flavors, request-based and token-based, and unlike budgets they apply at the Virtual Key level only.

Access control is enforcement, not advice. Put a model or provider outside a key's allow-list and the call returns a 403 with a typed reason, model_blocked or provider_blocked. There is no soft warning to ignore.

One gotcha worth knowing before you wire this into config: a budget attaches as its own referenced object, not as an inline block on the key. I tried to embed the budget directly in the create call and it came back unattached. Create the budget, reference it by id. The docs are right; the shortcut is not.

Governing every MCP call, not just every model call

Chat completions were the easy half. The harder half, and the reason this topic exists, is MCP.

Model Context Protocol is what turns a chat model into something that acts: it calls tools, runs commands, reads and writes real systems. An ungoverned MCP layer is an agent with hands and no supervisor. Bifrost treats MCP as a first-class governed surface, and it does so with tool filtering that is deny-by-default at three levels.

Client config sets the baseline of which tools exist at all.
Request headers (x-bf-mcp-include-clients, x-bf-mcp-include-tools) narrow that set per call.
Virtual Key filtering overrides both, so a key with no MCP config gets no MCP tools, full stop.

Deny-by-default is the whole posture. A tool a key was never granted is not available to it, even if the underlying MCP server exposes it.

How the calls actually run is configurable, and the default is the safe one:

Tool Execution mode is human-in-the-loop. Bifrost does not auto-run tool calls; your app reviews them and explicitly hits /v1/mcp/tool/execute. That review point is where approval and audit live.
Agent Mode auto-executes only the tools you mark in tools_to_auto_execute, bounded by max_agent_depth (default 10).
Code Mode lets the model write Starlark to orchestrate many tools in one shot through four meta-tools, which Bifrost measures at up to 92.8 percent fewer input tokens than exposing every tool directly.
Bifrost as an MCP gateway exposes a /mcp endpoint that external hosts like Claude Desktop, Cursor, and Cline connect to. On the running instance that endpoint answered as a live streaming connection, held open the way an MCP host expects.

For teams that want reusable policy, MCP Tool Groups (an Enterprise feature) bundle curated tools and attach them to keys, teams, or users, enforced at request time.

The one-sentence version: the gateway governs what an agent is allowed to do, not just which model it is allowed to call.

What you can actually see

Governance you cannot observe is governance you cannot prove. Bifrost logs every request asynchronously, so the visibility does not cost latency.

Each entry carries the model, token counts, cost, and the identity behind the call. When a request fails over or rotates keys, an attempt_trail records every attempt, the selected_key_id, and the retry count, so a weird bill or a slow p99 has a paper trail instead of a shrug.

The exports are the standard enterprise set:

Prometheus metrics are built in on /metrics, live out of the box (on a fresh instance you will see the runtime metrics first; the request metrics fill in as traffic flows).
OpenTelemetry export speaks the genai_extension semantic conventions to Grafana, Datadog, New Relic, or Honeycomb.
A native Datadog connector and log exports to S3, GCS, and BigQuery are available on Enterprise.

If content is sensitive, disable_content_logging keeps the metadata, latency, cost, tokens, and identity, while dropping the request and response bodies. You get the accounting without hoarding the payloads.

The controls that make security sign off

Everything so far runs in the open-source gateway. The pieces a security team asks about before rollout live in Bifrost Enterprise, which is a strict superset: same config schema, same providers, more control.

Identity comes through OIDC and SCIM, with SSO against Okta, Microsoft Entra, Keycloak, Zitadel, and Google Workspace, plus group-to-role mapping and auto-deprovisioning. Access is role-based across Admin, Developer, and Viewer, the permission set widening at each tier instead of collapsing into a single all-or-nothing admin. Data Access Control adds row-level scoping, so an operator sees only their own, their team's, or all data.

Audit logs are immutable and HMAC-signed, exportable as JSON, JSON Lines, or Syslog straight into a SIEM. That is what SOC 2 Type 2, GDPR, HIPAA, and ISO 27001 evidence actually looks like.

Guardrails split into Rules, written in CEL, and Profiles that decide how to act: native Gitleaks-backed secret detection and PII regex, plus external providers including AWS Bedrock, Azure Content Safety, Google Model Armor, CrowdStrike AIDR, GraySwan Cygnal, and Patronus AI. For scale, clustering keeps state in sync across nodes with no single leader.

This is the layer that turns "we route through a gateway" into "we can show an auditor who did what, signed and exportable straight into the SIEM."

Extending the gateway to the laptop

The gateway governs every call that routes through it. The last mile is reach. Think of the app a developer installed this morning that talks straight to a provider from a laptop, never pointed at your gateway. Getting the same governance onto that machine is the other half of the job.

That is where Bifrost Edge and the gateway work together. Edge is a lightweight endpoint agent for macOS, Windows, and Linux that runs in the menu bar or system tray and intercepts AI traffic at the machine level. There is no base URL to change per app and nothing for the user to configure. Once a machine signs in through your existing identity provider, the same virtual keys, budgets, guardrails, and audit logs you already run on the gateway apply on the laptop too.

Edge adds two enforcement surfaces the endpoint needs:

App allow and deny, fleet-wide. An approved app runs fully governed; a denied one is blocked before any data leaves the machine.
MCP server governance on the device. Edge discovers the MCP servers configured inside Claude Code, Claude Desktop, Gemini CLI, OpenCode, Codex, and Cursor, dedupes them across the fleet, and lets you allow or deny each one centrally. Deny a server and it is stopped on the machine, even inside an app that had it configured before your policy existed.

Rollout is hands-off, through the MDM you already run: Jamf, Intune, and Kandji, with Omnissa Workspace ONE and JumpCloud in the mix as well. Admins work from an Approvals dashboard (Pending, Approved, Denied) and a Devices fleet view. The gateway centralizes the policy; Edge carries it to every machine.

What isn't ready yet

Edge is not something you can turn on this afternoon.

It is in limited alpha, or early access, and you register to be onboarded rather than downloading a shipping release. It is an Enterprise feature, not part of the open-source repo, so you will not find it in the Bifrost you clone from GitHub. That status is current as of July 2026; check the docs before you quote a timeline, because alpha windows move.

It is also not a replacement for your endpoint or network security. Edge governs AI traffic specifically. It runs alongside tools like CrowdStrike, Zscaler, and Tailscale, not instead of them.

None of that changes the part you can use today. The open-source gateway is real, it installs in one command, and every governance, MCP, and observability capability above ran on the instance I stood up while writing this. The endpoint layer is where the story is still being finished.

Where to start

Start with the gateway, because it is the part you can run today. Install it with npx -y @maximhq/bifrost, point one base URL at it, and route your team's traffic through it. Then spend an afternoon on the governance: issue a virtual key per team, attach budgets and rate limits, turn on audit logging. At that point every LLM and MCP call your infrastructure makes is attributed, bounded, and logged, and none of it required rewriting a call site.

The endpoint is the next problem, not the first one. When you are ready to bring the laptops under the same policy, Bifrost Edge is the extension that gets you there, on the timeline the alpha allows.

Every technical detail here is drawn from the Bifrost gateway and Edge documentation and confirmed against a running instance. If something in this piece does not match what you see in the docs, trust the docs.

Resources & References

Bifrost gateway documentation
Bifrost Edge documentation
Bifrost on GitHub (open source)
Register for Bifrost Edge alpha

Originally published on Medium.

How to Fine-Tune a 7B Model for Three Dollars on One GPU

Sebastian Buzdugan — Sun, 05 Jul 2026 11:43:18 +0000

You think fine-tuning a 7B model needs a rack of A100s. It needs one 16GB card and about three dollars of rented compute. The gap between those two beliefs is costing teams entire projects they never start.

The cluster you think you need does not exist

Ask most engineers what it takes to fine-tune an open model and you get some version of "we don't have the hardware." They picture full fine-tuning: every weight in fp16, gradients and optimizer states for every parameter, a 7B model ballooning past 100GB of VRAM before the first batch even lands.

That math is real, and it is why the instinct is to give up and go back to prompt engineering a frontier API.

But you are solving a problem that was fixed in 2023. Full fine-tuning is not the only option, and for the overwhelming majority of practical tasks it is the wrong one. QLoRA, introduced by Tim Dettmers and collaborators, dropped the memory floor so far that a 65B model fits on a single 48GB GPU while matching full 16-bit fine-tuning quality.

Scale that down to the sizes you actually ship. A 7B fine-tune runs comfortably on a 16GB card. A 13B fits on a 24GB desktop 4090. You are not renting a cluster. You are renting one GPU for an afternoon.

Why four bits is enough, and where the memory actually goes

Full fine-tuning spends VRAM on four things: the model weights, the gradients, the optimizer states, and the activations. QLoRA attacks the first three at once.

The trick is a clean split. The base model is frozen and stored in 4-bit, using a data type called NF4 (4-bit NormalFloat) that is information-theoretically optimal for the normally-distributed weights neural networks actually have. That 4-bit form is storage only. For the forward and backward pass, each block is de-quantized back to bf16 on the fly, so the math stays high-precision while the resting footprint drops roughly 4x.

You never compute gradients for those frozen weights. Instead you train small low-rank adapter matrices bolted onto each linear layer, typically well under one percent of the model's parameters. Optimizer state, the silent VRAM killer in full fine-tuning, now covers only those tiny adapters.

Two smaller innovations close the gap. Double quantization quantizes the quantization constants themselves, saving about 0.37 bits per parameter, roughly 3GB on a 65B model. Paged optimizers use NVIDIA unified memory to absorb the gradient-checkpointing spikes that would otherwise OOM you on a long sequence.

The result is not a compromise you tolerate. On the paper's own MMLU evaluation, 4-bit NF4 with double quantization replicated 16-bit LoRA performance across LLaMA 7B through 65B. Independent 2026 runs put QLoRA within one to two percent of full-precision LoRA on standard benchmarks. You give up almost nothing measurable.

What it actually costs to run

Here is the part that turns this from theory into a Tuesday afternoon. These are the realistic 4-bit VRAM footprints for a QLoRA fine-tune, base model plus adapters and activations:

7B: comfortable on a 16GB card
13B: fits on a 24GB card (a desktop RTX 4090)
30B to 32B: needs roughly a 40GB card
70B: about 46GB, which fits on a single 48GB card

Sequence length and batch size move these numbers, but not as much as you would fear. An 8B model at a 2048-token sequence length peaks around 6.6GB of reserved VRAM on a 4090 with Unsloth. You have headroom on hardware you may already own.

Hold that against full fine-tuning of the same 7B model. Weights, gradients, and Adam's two optimizer states in fp16 push you well past 100GB before activations, which means multiple 80GB A100s and a multi-GPU setup to coordinate. QLoRA takes that same job from a cluster you have to requisition down to a single card you can rent by the second. The 4x drop from 4-bit storage is only part of it; freezing the base and training sub-one-percent adapters is what deletes the optimizer-state mountain entirely.

Now put a clock and a price on it. A 7B QLoRA run of two to three epochs takes roughly two to four hours on an A100 and six to eight hours on an RTX 4090. On RunPod, a Community Cloud 4090 rents from about 0.34 USD per hour and an A100 PCIe from about 1.39 USD per hour, billed by the second.

Do the arithmetic. A full 7B fine-tune on a 4090 lands around two to three dollars. On an A100 you trade money for wall-clock and finish before lunch for roughly the same total. This is the number that should reframe your roadmap: a custom model is a coffee-run expense, not a capital request.

The recipe that works on the first try

The tooling collapsed into something you can hold in your head. bitsandbytes handles the 4-bit quantization, PEFT handles the adapters, and Unsloth wraps both with kernels that cut memory and roughly double throughput. Start by loading the base model in 4-bit.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meta-llama/Llama-3.1-8B",
    max_seq_length=2048,
    load_in_4bit=True,
    dtype=None,
)

If you would rather stay in plain Transformers, the equivalent is a BitsAndBytesConfig that encodes the same three ideas: NF4 storage, bf16 compute, double quantization on.

from transformers import BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

Next, attach the adapters. The single most important choice is targeting all the linear layers, not just the attention projections. Skipping the MLP layers is the most common reason a fine-tune underperforms for no obvious reason.

model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth",
    lora_dropout=0, random_state=3407,
)

Two hyperparameters carry the load. Rank r controls adapter capacity; 16 is a strong default, and 32 if the task is genuinely complex. For alpha, the old alpha = 2 * r convention still works, but Unsloth's 2026 ablations found alpha = r is the cleaner default. Gradient checkpointing set to "unsloth" buys you another 30 percent memory reduction, which is often the difference between fitting and not.

Then the training arguments. Prefer a small batch size with gradient accumulation rather than a large batch, because that is how you avoid out-of-memory errors while keeping an effective batch size in the healthy 4 to 16 range.

from trl import SFTConfig

args = SFTConfig(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    num_train_epochs=2,
    warmup_ratio=0.05,
    weight_decay=0.01,
    seed=3407,
)

That configuration is not a starting guess to tune for a week. It is close to the endpoint for most supervised fine-tuning jobs. Learning rate 2e-4, one to three epochs, effective batch 16, warmup around five percent. Change the data, not the knobs.

From a trained adapter to a model you can serve

What you get out of a QLoRA run is not a fresh multi-gigabyte model. It is an adapter: a small set of low-rank matrices, often around 100MB, that sits on top of the frozen base. That size is a feature. You can train a dozen task-specific adapters against one base model and store them all for less than the footprint of a single full checkpoint.

You have two ways to serve it. Keep the adapter separate and load it over the base at runtime, which lets you hot-swap behaviors, or merge it into the weights for a single self-contained model. For most deployments, merge and export.

model.save_pretrained_merged("llama-8b-support", tokenizer,
                             save_method="merged_16bit")
model.save_pretrained_gguf("llama-8b-support", tokenizer,
                           quantization_method="q4_k_m")

The merged 16-bit version drops straight into vLLM or TGI for a production endpoint. The GGUF export, quantized to something like q4_k_m, runs on Ollama or llama.cpp on a laptop with no GPU at all. The same afternoon that produced your adapter can end with the model answering requests on your own machine.

One caution on inference: if you serve the base in 4-bit to save memory, evaluate in that exact configuration. Quality measured on a 16-bit merge does not always transfer cleanly to a 4-bit serving path, and the gap is easiest to catch before you ship, not after.

When a fine-tune is the wrong answer

The fastest way to waste your three dollars is to fine-tune a problem that RAG should own. In 2026 the question is no longer "RAG or fine-tuning," it is where each piece of intelligence should live: in the weights, in retrieval, or in both.

Reach for retrieval when the knowledge changes often, is large, needs citations, or differs per user. Facts, documents, prices, anything you would be embarrassed to see go stale inside a model, belong in a vector store you can update without retraining. Fine-tuning bakes knowledge in at a moment in time; that is a liability for anything dynamic.

Reach for a fine-tune when you need consistent behavior that prompting cannot reliably enforce: a fixed output format, a specific tone or policy, domain reasoning, or structured output that has to be right every time. It also wins on economics when volume is high enough that a small fine-tuned model beats per-call frontier pricing, or when your latency budget cannot afford a retrieval hop.

The mature pattern is both. Fine-tune for style, format, and decision behavior; retrieve for the facts. Teams that pick a side out of principle usually ship a worse product than teams that route each concern to the tool built for it.

The failure modes nobody warns you about

QLoRA is forgiving, but three things quietly ruin runs.

Overfitting is the big one, and it hides in a metric that looks like success. If your training loss drops below 0.2, the model has likely memorized your data and will generalize worse, not better. When you see that, cut epochs, raise weight decay, or scale alpha down by half. More than three epochs almost always trades generalization for a prettier loss curve.

The second is trusting the loss curve at all. A falling loss is not a working model. You need a held-out eval that reflects the actual task, checked before and after, or you are flying blind. Domain-specific evaluation matters far more than any general benchmark score your base model advertises.

The third is a subtle capacity mismatch. If a fine-tune underperforms and your data is clean, the culprit is usually rank set too low or target modules set too narrow, not the method failing. Bump r to 32 and confirm you are targeting every linear layer before you conclude QLoRA cannot learn your task.

One optional upgrade is worth knowing: DoRA, weight-decomposed low-rank adaptation, often squeezes out a bit more accuracy at low ranks for a small speed cost. Try it once the plain recipe works, not before.

Final Thoughts

The reason to internalize this is not that fine-tuning is trendy. It is that the cost of trying just fell through the floor, and cost of trying is what governs how much you experiment.

When a custom 7B model costs three dollars and one afternoon, fine-tuning stops being a quarterly initiative that needs sign-off and becomes something you do on a hunch, twice, before standup. That changes which ideas are worth testing. The team that ships ten cheap fine-tunes and keeps the two that work will out-iterate the team still waiting on a GPU budget.

Pick one narrow, format-heavy task your frontier API keeps getting subtly wrong. Rent a 4090 for an evening. The worst case is you spent the price of a coffee to learn something concrete about your own data.

Resources & References

QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., arXiv)
LoRA Fine-Tuning Hyperparameters Guide (Unsloth Documentation)
Making LLMs Even More Accessible with 4-bit Quantization and bitsandbytes (Hugging Face)
GPU VRAM Requirements to Fine-Tune LLMs in 2026: Full, LoRA, and QLoRA Sizing (Spheron)
RAG vs Fine-Tuning for LLMs in 2026: What Actually Works in Production

Originally published on Medium.

Snapshot Once, Rollout a Thousand Times: A Practical RL Setup for Coding Agents

Sebastian Buzdugan — Wed, 01 Jul 2026 09:09:10 +0000

Your RL run has been going for six hours. The GPUs are warm, the reward curve is creeping up, the policy is learning. Good.

Now look at what those six hours actually bought you. For a large slice of them, your expensive accelerators sat idle, not waiting on the optimizer, not waiting on the model, but waiting on an environment to boot. Clone the repo. Install the dependencies. Restore the dataset. Then, finally, let the policy take its first action.

This is the part nobody puts in the RL paper. The reward function gets a section. The KL penalty gets a section. The thing that actually ate your wall-clock, replicating the environment for every single rollout, gets nothing, because it feels like plumbing.

It isn't plumbing. For agent RL, it's the bottleneck. And it's a bottleneck with a clean fix that almost nobody is using yet.

I spent the last few days testing that fix on real infrastructure. This piece is what I found: the primitive, a working rollout harness you can run yourself, the real timings, and the places it doesn't help so you know where the edges are.

The rollout is a state-replication problem wearing a compute costume

Strip an RL step down to its skeleton:

s0  ->  policy samples an action  ->  env transitions  ->  ...  ->  terminal  ->  reward

For a math problem, s0 is a string. For a coding agent, s0 is a world: this repo at this commit, this Python version, these installed packages, this dataset on disk, this half-built state from a setup script. Reconstructing that world is the price of admission for every rollout you run.

And in GRPO, GSPO, or any group-based method, you don't run one rollout per step. You run G of them, eight, sixteen, sixty-four completions, each needing its own clean copy of s0. Multiply by thousands of steps. The environment gets rebuilt tens of thousands of times across a single run.

There are three ways teams handle this today, and all three are workarounds for the same underlying problem.

Containers per rollout. Spin up a fresh container, pull the image, install, run, tear down. Clean isolation, but you pay the cold-start-plus-setup tax on every single rollout. Tens of seconds, every time, before the policy moves.

Warm pools. Keep a fleet of pre-built containers ready. Faster, but now you own pool drift, eviction logic, health checks, and the ops surface of a small distributed system. You traded a latency problem for an infrastructure problem.

One VM, sequential rollouts. Cheap and simple, until you realize rollout three poisoned the filesystem that rollout four needed, and you've serialized the one thing you most wanted to parallelize.

Every one of these is dancing around a single fact: replicating environment state is expensive, so we keep paying for it or building scaffolding to avoid paying for it. What if replicating state was just cheap?

The primitive: snapshot once, restore N

Here's the move. Instead of rebuilding s0 for every rollout, you build it once, freeze it, and hand out identical copies.

The substrate is a microVM sandbox. It boots in well under a second, and you can snapshot its entire state and restore that snapshot into fresh, independent VMs. I tested this on Tensorlake sandboxes, which is what the code below uses, but the pattern is the point. The setup is unremarkable:

from tensorlake.sandbox import Sandbox, CheckpointType

# Build the world ONCE.
sb = Sandbox.create(name="rl-canonical")
sb.run("bash", ["-lc", "mkdir -p /home/tl-user/proj"])
sb.write_file("/home/tl-user/proj/impl.py", buggy_module)
sb.write_file("/home/tl-user/proj/test_impl.py", hidden_tests)
sb.run("bash", ["-lc",
    "cd /home/tl-user/proj && "
    "pip install --break-system-packages -q numpy pandas requests pytest"])

# Freeze it.
snap = sb.checkpoint(checkpoint_type=CheckpointType.FILESYSTEM)

That snap is s0. Restoring it gives you a fresh, byte-identical world to roll out in:

fork = Sandbox.create(snapshot_id=snap.snapshot_id)   # a clean copy of s0

Before we go further, the obvious objection, because it's the right one to ask:

"Why not just bake a Docker image?"

Because an image and a snapshot are not the same kind of thing, and the difference is exactly what makes this work.

An image distributes an environment. A snapshot captures runtime state.

An image is a recipe for building an environment: the repo, the language, the dependencies. Great for distribution. But it's frozen at build time. It knows nothing about what happened after the container started: the files your setup script generated, the dataset you downloaded, the checkpoint your last run wrote, the config a previous step mutated.

A snapshot is taken at runtime. It captures the world as it actually is, mid-flight, including everything an image structurally cannot.

I tested this directly. I built a sandbox the way a real session would: installed deps, downloaded a file, generated an embeddings.npy from running code, wrote a checkpoint directory, and mutated a config. Then I snapshotted it and restored into a fresh sandbox. What survived:

deps:        requests + numpy importable
download:    downloaded_readme.md present
generated:   embeddings.npy  (1000, 128)
checkpoint:  step=4200 loss=0.13
run_state:   resume_from=4200

A Dockerfile gives you the first line. The other four, the runtime state that's often the entire point of s0, exist only because the snapshot captured them. That's the distinction in one sentence: an image distributes an environment; a snapshot captures runtime state.

A rollout harness you can actually run

Let's make it concrete. The toy task: an agent has to fix a buggy Python module so a hidden pytest suite passes. Small enough to fit in an article, structurally identical to anything real you'd train on. The reward is the fraction of tests that pass: clean, dense enough to learn from, impossible to game.

The starting state s0 is the module-with-bugs plus the installed environment, snapshotted once (the code above). Now the rollout step: for each of G policy samples, restore s0, apply the candidate, score it.

import concurrent.futures
from tensorlake.sandbox import Sandbox

def rollout(i, candidate_patch):
    # Each rollout gets its own clean copy of s0.
    fork = Sandbox.create(snapshot_id=snap.snapshot_id)
    fork.run("bash", ["-lc", f"cd /home/tl-user/proj && {candidate_patch}"])
    res = fork.run("bash", ["-lc",
        "cd /home/tl-user/proj && python3 -m pytest -q 2>&1 | tail -1"])
    reward = pass_fraction(res.stdout)      # 0.0 .. 1.0
    fork.terminate()
    return i, reward

# Fan out G rollouts from the one snapshot, in parallel.
G = 8
with concurrent.futures.ThreadPoolExecutor(max_workers=G) as ex:
    group = list(ex.map(lambda a: rollout(*a), enumerate(candidates)))

In a real loop, candidates comes from your policy. Here I scripted eight candidate patches of varying quality so the whole thing is deterministic and you can reproduce it without a model or an API key. The point isn't the policy, it's what the sandbox does underneath it.

Here's the actual output from my run:

rollout[0] reward=0.00   (3 failed)
rollout[1] reward=0.33   (2 failed, 1 passed)
rollout[2] reward=0.33   (2 failed, 1 passed)
rollout[3] reward=0.67   (1 failed, 2 passed)
rollout[4] reward=0.33   (2 failed, 1 passed)
rollout[5] reward=0.33   (2 failed, 1 passed)
rollout[6] reward=0.33   (2 failed, 1 passed)
rollout[7] reward=0.67   (1 failed, 2 passed)
group mean reward = 0.375     best = 0.67

That vector of rewards is the whole game. And here's the thing worth internalizing: getting G independent rollouts from one identical starting state is the expensive part, and it's identical across methods. What you do with the rewards afterward is where the techniques split. Rejection-sampling fine-tuning keeps the best completion and trains on it; GRPO and GSPO use the entire group, computing each rollout's advantage as its reward minus the group mean, and nudge the policy toward the above-average ones. Same expensive primitive underneath. The snapshot is what makes that primitive cheap.

One snapshot, G rollouts, one reward vector. Rejection sampling keeps the best; GRPO uses the whole group. The fan-out is the same either way.

The economics, which is the actual argument

Now the part that matters. Let's price the two approaches on the same workload, and let's be fair about it: both of them spin up a fresh sandbox per rollout, so both pay that cost. The only thing that differs is the build.

Two numbers from my runs. Building the world (installing numpy, pandas, requests, and pytest, plus generating a dataset) took 7.2 seconds. Restoring a sandbox from a snapshot took about 2 seconds (sub-second on capable infrastructure; free-tier contention pushed mine higher). Now price a full run, G = 8 rollouts per step, 1000 steps, 8,000 rollouts:

per rollout:    build 7.2s   +   restore ~2s
G = 8, 1000 steps  ->  8,000 rollouts
naive loop:     8,000 x (7.2s build + 2s restore)        ~=  20 hours
snapshot loop:  7.2s build once  +  8,000 x 2s restore   ~=  4.4 hours
                -----------------------------------------------------
saved:          ~16 hours, all of it rebuilding identical state

Same 1000-step run at G=8. Both loops pay the per-rollout sandbox spin-up (~4 hours). The snapshot loop deletes the ~16 hours of rebuilding identical state.

The ~4 hours of sandbox spin-up is real, and the snapshot loop still pays it. So does the naive loop. That part is a wash. What the snapshot deletes is the 16 hours spent rebuilding the same world 8,000 times. You're not making restores free; you're removing the redundant builds.

And that's with a trivial seven-second environment. Swap in a real coding-agent setup (a heavy requirements.txt, a model download, a dataset) and the build cost climbs from seconds to minutes while the restore stays flat. The gap widens until "rebuild every rollout" stops being slow and starts being the difference between a run you can afford and one you can't. Structurally, you've turned an O(G x steps) build cost into O(1).

One honest note on that restore number. I ran on the free tier (one vCPU, ten concurrent sandboxes), so eight simultaneous restores contended and per-rollout boots stretched to two to seven seconds rather than the sub-second you'd see on real infrastructure. I used ~2s as a fair middle estimate. Tensorlake quotes 10,000+ parallel environments on their full stack, where both the restores and the parallelism get much cheaper. None of it changes the shape: build once, not 8,000 times.

Where this pattern shows up

This isn't a Tensorlake trick. "Snapshot a starting state, fork it into many independent rollouts" is a shape that recurs the moment you need many runs from one point. Three places it's already load-bearing:

RL training. The case we just walked through. Tensorlake's own GSPO cookbook does exactly this at proper scale, dispatching G completions per step to parallel sandboxes, scoring each against a hidden test suite, using the group to update the policy. It's a working reference implementation if you want more than an architecture diagram.

Agent evaluation. Same primitive, different goal. Instead of training, you're scoring. Snapshot the benchmark's starting state, fork it across every task in the suite, run your agent in each isolated copy, collect pass/fail. Tensorlake plugs into Harbor (a framework for defining and verifying terminal tasks) as the execution runtime, running fleets of sandboxes for terminal-bench-style evaluation with real filesystem verification instead of trusting an agent's self-report.

Parallel search at inference. Drop the training entirely. Best-of-N sampling, tree search over tool calls, speculative execution of multiple plans, all of them are "fork the current state, explore N branches, keep the good one." The same snapshot-and-fan-out, just at inference time instead of training time.

The through-line: state is what makes parallelism expensive, and a cheap snapshot removes the cost. Once that's true, a pile of architectures that were too painful to build become reasonable.

Where it doesn't help (the honest part)

I don't trust a piece that only tells you the good news, so here's the map of the edges.

External state isn't in the snapshot. Anything your sandbox reaches over the network (a production database, a third-party API, a shared queue) is outside the VM and outside the freeze. Fork a sandbox mid-API-call and all N forks will independently try to finish that call. If your reward function touches an external service, idempotency and determinism stop being nice-to-haves.

Pick the right checkpoint type. Sandboxes give you a filesystem snapshot and a memory snapshot. For rollouts you almost always want the filesystem one: you want a clean disk reset from s0, not a frozen process tree. The memory snapshot, which additionally captures live RAM and running processes, is for pausing and resuming an actual session, and it costs more. In my tests the memory variant added a few hundred megabytes of captured RAM on top of the disk image. Use it deliberately, not by default. One genuinely counter-intuitive result: the heavier memory snapshot restored faster than the filesystem one in my runs (about 0.9s versus 1.9s) because the page cache comes back warm with it.

There's a per-restore floor. Restoring a snapshot isn't free. On capable infrastructure it's sub-second; under a throttled tier it's a few seconds. If your policy step is itself sub-second, that floor is a real fraction of your time. If your rollouts are LLM calls measured in seconds, it vanishes into the noise. Know which regime you're in.

Snapshot storage is real. A snapshot tracks the provisioned disk, not a tiny diff. Keep thousands of them around and you're paying for thousands of disk images. Snapshot deliberately and prune.

None of these sink the approach. They tell you where it fits: heavy environment, many rollouts, isolation that matters. Which is precisely the shape of agent RL.

The takeaway

If your training loop spends a meaningful chunk of its wall-clock waiting for environments to come up, you are paying a tax you don't have to pay. Rejection-sampling fine-tuning, GRPO and GSPO rollouts, agent eval harnesses, paired DPO sampling, every one of them is expensive for the same reason, and it's not the reason you think. It's not the GPUs. It's that you're rebuilding the same world, over and over, for no reason.

Snapshot it once. Fork it as many times as you have rollouts. Throw the forks away.

The cheapest way to feel the difference is to run it. Spin up a sandbox, build something stateful inside it, snapshot it, and restore that snapshot into three fresh sandboxes in parallel. Watch the world appear three times from one freeze. That shift, from rebuilding state to restoring it, is the kind of thing you have to see once before it changes how you architect the whole loop.

I tested everything in this piece against Tensorlake sandboxes on the free tier (June 2026). The rollout harness and benchmarks are reproducible: scripted candidates, no model or API key required.

Resources & References

Stay in Touch

Short takes and discussions on X -> https://x.com/sebuzdugan

Practical AI / ML videos on YouTube -> https://www.youtube.com/@sebuzdugan/

Partnerships & collabs -> sebuzdugan@gmail.com

Read the full piece on Medium

https://medium.com/data-science-collective/snapshot-once-rollout-a-thousand-times-a-practical-rl-setup-for-coding-agents-0f880a450610

Handling Multi-Model API Outages Without Melting Production

Sebastian Buzdugan — Wed, 24 Jun 2026 12:47:36 +0000

Your dashboards go red. Not one model. All of them. Retries spike. Latency climbs. Nothing recovers.

When every Claude call starts failing at once

If you ship anything serious on top of LLM APIs, you have lived this moment.

Requests that worked minutes ago now return errors. You flip the model flag. Same thing. You increase retries. Error rate goes up, not down. Queues back up. Downstream services feel it.

This is not a prompt issue. This is not a bad deploy. This is what a platform-level incident looks like from the client side.

The recent Anthropic incident labeled "Elevated error rate across multiple models" is a clean example of the failure mode that matters most in production. Multi-model impact. No numbers. No knobs to turn. Just errors.

The mistake teams make is assuming model diversity equals resilience. It does not.

Why model fallback quietly fails during platform incidents

Most LLM client stacks are built around a simple idea: if model A fails, try model B.

That works when failures are model-scoped. It fails when the blast radius is shared infrastructure.

Under the hood, models often share front-door API gateways, auth and quota systems, routing layers, and shared inference clusters or schedulers.

When error rate rises across multiple models, it usually means something above the model is degraded. Switching from Opus to Sonnet does not bypass the problem because the problem is not Opus or Sonnet.

The docs do not emphasize this. SDK examples actively encourage retry and fallback loops that assume independence. In production, this assumption is wrong often enough to matter.

What this failure looks like in real systems

Here is the common sequence I see during incidents like this. Error rate jumps from sub-1 percent to double digits. Clients retry aggressively. Retry traffic amplifies load on the already degraded system. Latency increases, triggering timeouts. Queues back up. Background jobs fall behind. Engineers start manually throttling.

The most dangerous part is that nothing is technically "down." Health checks pass. Some requests succeed. This keeps retry storms alive.

If you only built retry logic, you built a traffic amplifier.

The naive retry loop that makes things worse

This is the pattern I still see in production code:

from anthropic import Anthropic
import time

client = Anthropic()

def call_claude(prompt):
    for attempt in range(5):
        try:
            return client.messages.create(
                model="claude-3-sonnet",
                max_tokens=512,
                messages=[{"role": "user", "content": prompt}]
            )
        except Exception:
            time.sleep(2 ** attempt)
    raise RuntimeError("Claude failed after retries")

This looks reasonable. It is not.

During a platform incident, all retries hit the same failing surface, backoff still creates synchronized retry waves, and you contribute to global load at the worst possible moment. This pattern is survivable at low volume. At scale, it cascades.

The real fix starts with circuit breakers, not retries

The first control you need is not more retries. It is a circuit breaker that can say "stop calling this API."

A circuit breaker turns repeated failure into a fast local decision. Here is a minimal breaker using Redis as shared state:

import redis
import time

r = redis.Redis()

def claude_available():
    return r.get("claude_circuit") != b"open"

def trip_claude():
    r.setex("claude_circuit", 60, "open")

def call_claude_with_breaker(fn):
    if not claude_available():
        raise RuntimeError("Claude circuit open")
    try:
        return fn()
    except Exception:
        trip_claude()
        raise

The breaker is shared across workers. The open state expires automatically. Fail fast beats slow failure every time. This alone will save your system from self-inflicted damage.

Why retries still matter, but only inside boundaries

Retries are not evil. Unbounded retries are.

The pattern that works is a small retry count, a tight timeout, wrapped inside a breaker:

def safe_claude_call(prompt):
    def attempt():
        return client.messages.create(
            model="claude-3-sonnet",
            max_tokens=512,
            timeout=5,
            messages=[{"role": "user", "content": prompt}]
        )
    for _ in range(2):
        try:
            return call_claude_with_breaker(attempt)
        except Exception:
            time.sleep(0.5)
    raise RuntimeError("Claude unavailable")

Notice what is missing: no exponential backoff to 30 seconds, no retry count in the teens, no blind faith that persistence equals success. At incident scale, shorter and fewer retries win.

Model fallback is not resilience, provider fallback is

If multiple models fail together, the only real isolation is at the provider boundary. That means your fallback graph needs to cross vendors, not model families.

def generate(prompt):
    try:
        return safe_claude_call(prompt)
    except RuntimeError:
        return call_openai(prompt)

This is not about preference. It is about failure domains. If your business cannot tolerate degraded LLM output, you must pay the complexity tax of multi-provider support. There is no shortcut here.

Bulkheads stop one failing feature from drowning everything

Another failure pattern during incidents is resource starvation. Your summarization jobs eat all workers. User-facing requests time out. Everyone is unhappy.

Bulkheads prevent this by isolating workloads:

queues:
  llm_user_requests:
    concurrency: 50
  llm_background_jobs:
    concurrency: 10

During elevated error rates, user traffic stays responsive and background jobs slow down, not everything. If you only have one queue, you do not have isolation.

Idempotency saves you from retry side effects

Retries are dangerous when requests have side effects. If your LLM call triggers writes, notifications, or billing events, retries can duplicate work.

Use idempotency keys consistently:

idempotency_key = f"summary:{document_id}"

client.messages.create(
    model="claude-3-sonnet",
    idempotency_key=idempotency_key,
    messages=[{"role": "user", "content": prompt}]
)

When incidents happen, duplicate requests are common. Idempotency turns chaos into correctness.

What monitoring actually matters during elevated error rates

Most teams monitor request count and average latency. That is not enough. During incidents, you need error rate by provider, circuit breaker state, retry volume, and queue depth growth.

A simple metric that pays for itself is retry amplification:

select
  sum(retry_attempts) / sum(original_requests) as retry_factor
from llm_request_metrics
where timestamp > now() - interval '5 minutes';

If that number goes above 1.5, you are probably making things worse.

Why status pages do not help you in the moment

Status pages tell you two things: something is wrong, and someone else is fixing it. They do not tell you how long retries will fail, which endpoints are safe, or whether partial recovery is real.

Your system must assume uncertainty. That is why local control mechanisms matter more than external signals. If your only reaction is waiting for green, you are already behind.

Graceful degradation beats hard failure

When LLMs are unavailable, many products default to errors. This is often unnecessary. You can return cached responses, use smaller local models for basic tasks, skip non-critical enrichment, or defer generation to async jobs.

A simple cache fallback:

cached = cache.get(prompt_hash)
if cached:
    return cached

response = generate(prompt)
cache.set(prompt_hash, response, ttl=3600)
return response

During incidents, cache hit rates go up. That buys you time.

Where this advice breaks

There are cases where none of this helps: if your workload is real-time and uncachable, if output correctness is legally critical, or if you are contractually bound to one provider.

In those cases, your only option is admission control. Say no early. Protect the rest of the system. Rejecting requests cleanly is better than timing out everything.

Admission control and load shedding when breakers are not enough

Circuit breakers stop calls after failure. Admission control stops calls before failure. During platform incidents, the worst thing you can do is accept unlimited work you cannot complete. You need a hard cap.

A simple token bucket per provider works well:

import time
from collections import deque

class RateGate:
    def __init__(self, max_inflight):
        self.max_inflight = max_inflight
        self.inflight = 0

    def acquire(self):
        if self.inflight >= self.max_inflight:
            return False
        self.inflight += 1
        return True

    def release(self):
        self.inflight -= 1

claude_gate = RateGate(max_inflight=20)

Usage:

if not claude_gate.acquire():
    raise RuntimeError("Claude overloaded locally")
try:
    return safe_claude_call(prompt)
finally:
    claude_gate.release()

During partial outages, latency often doubles before error rate spikes. Inflight requests pile up silently. Your worker pool gets saturated even if the provider eventually errors.

Concrete numbers I have seen: normal p95 latency 1.2s, incident p95 latency 8 to 12s, worker exhaustion within 90 seconds. Load shedding keeps tail latency bounded and preserves capacity for critical paths. It is blunt, but it works.

Hedged requests sound smart and usually backfire with LLM APIs

Some teams try hedged requests: fire the same prompt at two models or providers and take the first response. This can help with single-request latency variance. During incidents, it is gasoline.

Example of what not to do:

def hedged_generate(prompt):
    return first_completed([
        lambda: call_claude(prompt),
        lambda: call_claude(prompt)
    ])

You double traffic exactly when capacity is constrained, trigger rate limits faster, and amplify retry storms upstream. Even cross-provider hedging is risky if you do not gate it:

def cautious_hedge(prompt):
    if system_healthy():
        return call_claude(prompt)
    return first_completed([
        lambda: call_claude(prompt),
        lambda: call_openai(prompt)
    ])

Rules that keep this safe: hedge only after a latency threshold, never hedge inside retries, and disable hedging automatically when error rate rises. Hedging is an optimization. Incidents are not the time to optimize.

How to actually test this before production does it for you

Most LLM outage handling code is untested. That is why it fails under stress. You can simulate platform incidents locally with three levers: forced error responses, artificial latency, and partial success rates.

import random
import time

def flaky_claude(prompt):
    r = random.random()
    if r < 0.3:
        raise RuntimeError("500")
    if r < 0.6:
        time.sleep(10)
    return "ok"

Then run load tests and watch circuit breaker open rate, queue depth over time, admission rejections, and the retry amplification metric. You want to see breakers opening quickly, inflight count staying flat, and non-critical queues slowing first. If your test shows recovery only after traffic stops, your controls are too weak.

This is not chaos for chaos' sake. It is validating that your failure handling actually reduces load instead of redistributing it.

Final Thoughts

Elevated error rates across multiple models are not edge cases anymore. They are normal failure modes for platform-scale AI services.

If your resilience strategy is model switching and hope, you will relive the same outage every time.

The fix is not clever prompts or better SDK defaults. It is the same discipline that keeps payment systems, search backends, and message brokers alive under stress.

Treat LLMs like the critical infrastructure they are. Your future incidents will still hurt, but they will not take everything down with them.

Resources & References

Stay in Touch

Short takes and discussions on X → https://x.com/sebuzdugan

Practical AI / ML videos on YouTube → https://www.youtube.com/@sebuzdugan/

Partnerships & collabs → sebuzdugan@gmail.com

Originally published on Medium.

Governing and Monitoring Enterprise AI Usage: Allow and Deny Controls for AI Apps on Managed Devices

Sebastian Buzdugan — Fri, 19 Jun 2026 12:57:05 +0000

Your gateway logs look clean. Every API call is accounted for, rate-limited, and piped to your SIEM. You have virtual keys per team, budgets per model, guardrails on PII. The compliance box is checked.

Then someone asks: which AI apps are actually running on our laptops right now?

Nobody knows.

The gateway only sees traffic that goes through it. Claude Desktop, ChatGPT, Cursor, Codex, and a browser full of AI sidebars talk directly to their own endpoints, using API keys that live in the user's home directory. None of that traffic has ever touched your proxy.

This isn't a configuration problem. It's a structural one. And it has a clean fix that most enterprise teams haven't wired up yet.

The gap nobody talks about

A gateway governs the traffic that flows through it. That is exactly what it is for, and Bifrost's gateway does it well: virtual keys, budgets, routing, guardrails, and audit logs, enforced on every request your infrastructure sends.

But your developers are not only calling AI through your infrastructure. They are running the Claude app on their laptop. They are using Cursor with a personal API key. They installed an MCP server that a GitHub README told them would make their coding agent smarter. That traffic leaves the machine straight for the provider. It never had a reason to pass through your gateway, so it doesn't.

This is not a gateway shortcoming. It is a question of where the traffic starts. A centralized proxy can only govern what reaches it, and a desktop app talking directly to api.anthropic.com was never going to reach it on its own. The endpoint is simply a second surface, and until something sits on the machine, that surface is dark.

Here is what lives there:

Desktop chat apps: Claude Desktop, ChatGPT, calling home directly
Browser AI: ChatGPT and Claude on the web, running over HTTPS
Coding agents: Claude Code, Cursor, Codex, OpenCode, configured per-user, per-project
MCP servers: local processes that give those agents filesystem access, shell execution, database connections, and external API calls, spun up from a few lines of JSON

That last one deserves a pause. An MCP server is not an app in the traditional sense. It's a process your developer launched that tells their AI agent: here are the tools you can use. File reads, shell commands, web fetches. All of it governed by whatever the developer put in a config file, with no visibility for your security team and no logs anywhere.

A coding agent with shell access, connected to an ungoverned MCP server, can exfiltrate credentials before your SIEM sees a single packet.

That is the surface Edge is built to cover, and it covers it by extending the gateway you already run, not by replacing it.

What your gateway sees vs. what's actually running on managed devices. The gap is everything on the right.

Two layers, one control plane

Bifrost, the open-source AI gateway built by Maxim and available on GitHub, splits the governance problem into the two layers it actually has.

The gateway is the centralized control plane. It sits between your infrastructure and 1000+ models behind one unified API, and enforces policy on every request: virtual keys, team budgets, rate limits, model routing, PII detection, secrets scanning, content guardrails, and audit logs that feed SOC 2 Type 2, GDPR, and HIPAA stacks. It is built to stay out of the way. In Bifrost's own t3.xlarge benchmark at 5,000 requests per second, the gateway added 11 microseconds of overhead per request at a 100 percent success rate.

The OSS version is self-hosted and drops in as a base URL change. You point your existing SDK at it and leave the rest of your code alone:

# before
client = anthropic.Anthropic(api_key="your-anthropic-key")

# after: same code, traffic now governed
client = anthropic.Anthropic(
    base_url="http://localhost:8080/anthropic",
    api_key="dummy-key",  # real keys live in Bifrost
)

No agent on the machine, no per-app config. This is the layer most teams already have, or can stand up in an afternoon.

Bifrost Edge is the endpoint enforcement layer. It is an Enterprise feature, not part of the open-source gateway. Edge runs locally on every managed machine and intercepts AI traffic at the OS level, before it reaches any application's networking stack. That traffic gets routed through your Bifrost gateway for policy evaluation, and the gateway's verdict comes back to Edge, which enforces it on the device.

The frame that makes this click: the gateway centralizes governance, Edge enforces it on every machine.

Every desktop app, every browser extension, every MCP server suddenly inherits the same virtual keys, budgets, guardrails, and audit logging that your API infrastructure already has. Zero per-app configuration. You install Edge once and governance follows the user.

Bifrost Edge on every managed machine, all routing to the same central gateway. Policy changes propagate to the fleet without touching individual devices.

How Edge sits on the machine

Installation is a single MDM push. Edge ships managed config for Jamf, Intune, Kandji, Omnissa Workspace ONE, and JumpCloud, so it rolls out across macOS, Windows, and Linux with no end-user action. The managed config carries only non-sensitive connection settings. The machine arrives pre-pointed at your Bifrost; identity and keys come from the user's sign-in, not from the device profile.

After install, the user signs in once through a browser SSO flow. That sign-in links the machine to the user and pulls down every policy assigned to them. From then on, Edge runs as a menu-bar agent on macOS or a system-tray process on Windows and Linux.

Interception happens at the machine level, not per application. Edge does not ask you to change base URLs or touch anything inside the AI apps. It uses an organization certificate, managed centrally from the Bifrost console, to route each app's AI calls through your gateway before they leave the machine. When Claude Desktop calls api.anthropic.com, Edge sees the request, sends it through Bifrost for policy evaluation, and hands the response back. The app never knows the difference.

That is what makes this work at scale. You have a hundred AI tools across the fleet; you configure one thing, and all hundred are governed.

Allow, deny, and the state in between

Edge discovers AI apps and MCP servers across the fleet and rolls them into a deduplicated catalog: the same tool on 200 machines shows up once. Every entry sits in one of three states.

Pending: discovered and awaiting review. It keeps working in the meantime, so turning Edge on does not break anyone's workflow on day one.
Approved: explicitly allowed. Traffic runs normally and stays under the gateway's full policy stack.
Denied: blocked. Edge stops it on the device, and the user sees a notice that the app is not permitted on company machines.

That default-pending posture is the part worth sitting with. You are not forced to choose between locking everything down up front or flying blind. New tools surface as Pending while you decide, so you get visibility immediately and enforcement when you are ready.

The workflow in practice: a developer installs a new coding agent. It appears as Pending in the console with its name, the devices running it, who last changed its status, and any notes. You approve it, deny it, or select every Pending item and act in bulk. Decisions take effect on each device at its next check-in, which is set by a sync interval you control centrally, down to a few seconds when you need policy to move fast.

Supported surfaces today: Claude Desktop, ChatGPT, Cursor, and Codex on the desktop; Claude Code, Codex CLI, and OpenCode as coding agents; ChatGPT and Claude on the web. If something you run is not on the list yet, the console has a "request support for an application" path.

MCP governance is its own layer

MCP servers get their own governance surface in Edge, separate from application-level allow/deny. That separation matters.

You might want Cursor allowed but a specific MCP server denied. Maybe it has filesystem write access your security team hasn't signed off on. Maybe it was pulled from an untrusted source. Edge lets you approve or deny at the server level, not just the app level.

The workflow:

Edge discovers MCP servers configured in supported AI apps across the fleet
The full inventory surfaces in the Bifrost admin console
Administrators approve or deny each server individually
Edge enforces those decisions at the device level

Default posture applies here too. Set Edge to deny all unknown MCP servers by default and new servers require explicit approval before anyone in the fleet can use them.

The fleet-wide MCP inventory alone is something most security teams have never had. Ask a room of security leads which MCP servers run on their developer machines and the honest answer is nobody knows. Edge surfaces each one with its name, whether it connects through a local command or a remote URL, and the exact tools it exposes, then gives you per-server approve and deny. MCP discovery currently covers Claude Code, Claude Desktop, Cursor, Gemini CLI, OpenCode, and Codex.

MCP governance in three steps: Edge discovers servers fleet-wide, admin sets per-server policy, Edge enforces on every device.

What you can see

With Edge deployed, the console becomes a live inventory of AI on the fleet.

The Devices view lists every machine running the agent: hostname, owner (name and email), platform with OS version and architecture, agent version, how many AI apps and MCP servers it has (versions on hover), when it was first seen, and its last check-in. You can filter by platform, by a specific app or MCP server, or by approval status, so "which machines are still running the app we denied" is a filter, not an investigation. Open a single device and you get its installed apps, its configured MCP servers and the tools they expose, and one-click approve, deny, or remove.

Underneath the inventory, the guardrails you already configured in the gateway now apply to all of this traffic, in both directions, before a prompt reaches a model and before a response comes back. That includes Gitleaks-backed secrets detection for leaked API keys, tokens, and private keys, PII redaction, and any third-party content-safety integration you run (AWS Bedrock, Azure, Google Model Armor, CrowdStrike, Patronus, and others). Edge does not re-implement any of it. It brings ChatGPT, Claude, and your coding agents under the same protection your API traffic already has.

Every routed request also inherits the gateway's structured logging: timestamp, user, model, token count, guardrail outcome. Pair that with the gateway's Prometheus, OpenTelemetry, and Datadog exports and you have one view over all AI usage, infrastructure and endpoint alike.

The containment argument

Here is the scenario that makes the security case concrete.

Your team discovers a new AI desktop app is exfiltrating conversation history to an undisclosed third party. You need it off every managed laptop, now.

Without Edge: you're filing an MDM change request, writing browser extension policies, opening a firewall ticket, coordinating across three teams, and probably missing the macOS desktop app entirely. Timeline: days to weeks.

With Edge: you open the console, set the app to Denied, and apply it to the fleet. Every machine picks it up at its next check-in, governed by the sync interval you set. The block lands before any data leaves the device. Audited, reversible, fleet-wide in seconds to minutes.

Same logic for MCP servers. A new server with dangerous tool permissions shows up in a GitHub repo. You deny it fleet-wide before any developer on your team installs it.

The governance speed matters as much as the governance capability. The threat landscape for AI tooling is moving fast. Waiting weeks to roll out a block is not a viable security posture.

The honest part

Edge is in limited alpha. You cannot self-serve into production today. You register, you get access when the alpha opens up. That is the reality of where it sits.

The capability set described here is what the alpha delivers. All of it is live and usable for teams in the program. But if you need this in production next week with an SLA behind it, that conversation starts with the Bifrost team, not a self-serve dashboard.

Also worth naming: the MCP app support list is current as of this writing. New apps and MCP servers get added, but if you have a specific tool you need covered, check the supported applications page before assuming it's there.

Getting started

The Bifrost gateway (OSS) is available now. It is a drop-in base URL replacement for any OpenAI-compatible SDK call. If you are not running a centralized gateway yet, that is the right starting point. Get visibility on your API traffic first, then add Edge for the endpoint layer.

For Edge, register for alpha access at docs.getbifrost.ai/edge/overview.

The path that makes sense:

Deploy the Bifrost gateway, route your team's API traffic through it
Configure virtual keys, budgets, and audit logging
Roll out Bifrost Edge via MDM to bring endpoint AI traffic under the same governance

Steps 1 and 2 give you coverage on everything your infrastructure controls. Step 3 closes the gap.

Every technical detail here is drawn from the Bifrost Edge documentation and the gateway overview. If something in this piece doesn't match what you see in the docs, trust the docs.

Originally published on Medium.

More writing on Medium → https://medium.com/@sebuzdugan
Short takes on X → https://x.com/sebuzdugan
Practical AI / ML videos on YouTube → https://www.youtube.com/@sebuzdugan/