DEV Community: Sebastian Buzdugan

Compliance-Ready AI Agents: Logging and Tracing Every MCP Tool Call with Bifrost

Sebastian Buzdugan — Wed, 29 Jul 2026 14:09:29 +0000

An AI agent on your team just deleted a customer record, moved a payment, or hit an internal API. It did it through an MCP tool, in a loop, with no human watching. The next morning, someone asks the only question that matters: which tool ran, whose access it used, and was it allowed. If the answer is a shrug, you do not have compliant agents. You have fast ones.

The morning after your agent moved money

Model Context Protocol turned chat models into actors. An agent no longer just talks. It calls tools, runs commands, reads files, and writes to real systems. That is the point of MCP, and it is also the problem.

Every one of those tool calls is an action with consequences, and most teams cannot reconstruct a single one after the fact. You can see that the agent used a tool. You cannot see which tool, on whose behalf, with what input, and whether a policy should have stopped it.

Auditors do not accept "the agent decided to." A SOC 2 review, an incident postmortem, a customer security questionnaire: they all ask the same thing. Show me the record.

The gap is not that agents are dangerous. It is that they are unaccountable by default.

Your logs have the conversation, not the action

Here is the mechanism that trips people up. Standard LLM logging records the request and response between your app and the model. You get the prompt, the completion, the tokens, the latency. What you do not get is the tool call the model set in motion.

When a model returns a tool call, that is a decision, not an execution. Something downstream still has to run delete_record or send_payment. If that execution happens inside your app code or a raw MCP client, it never touches the logging layer watching your model traffic. The action is invisible to the exact system you built to watch AI.

So you end up with two half-pictures. The model logs know the agent wanted to do something. Your database knows something happened. Nothing connects them.

Picture the incident. Your model logs show a completion that requested send_payment at 2am. Your ledger shows money left the account. Between those two facts sits the part you need and do not have: the call itself, its amount, the key it used, and whether anything checked it first. That missing middle is what an auditor circles.

Closing that gap means putting the tool execution itself behind something that records it. That is what an MCP gateway is for.

What a full MCP tool-call record looks like

I ran Bifrost locally to see exactly what it captures. It is an open-source AI gateway built by Maxim AI that also acts as an MCP gateway, so tool calls route through one place that logs them the same way it logs model calls.

You do not have to let the model auto-run anything. In the default flow, the model returns a tool call and your app decides whether to execute it by posting to one endpoint.

curl -X POST http://localhost:8080/v1/mcp/tool/execute \
  -H "Content-Type: application/json" \
  -d '{"id":"call_9f2","type":"function","function":{"name":"filesystem_list_directory","arguments":"{\"path\":\".\"}"}}'

The tool runs, returns its result, and the call lands in the log with everything an audit needs. Read it straight back from the logs API, filtered like any other request.

curl "http://localhost:8080/api/logs?status=success&content_search=filesystem_list_directory"

Each row carries the tool name, prefixed by its MCP client so filesystem_list_directory is unambiguous across servers, the input arguments, the result, the latency, the cost, the token counts, the timestamp in RFC3339, and the identity behind the call through selected_key_id. If the call failed over or rotated keys, an attempt_trail records every attempt in order. Logging runs on background workers and adds under 0.1ms per request, so you are not trading latency for the paper trail.

Line those fields up against the three questions from the incident and they answer cleanly. Which tool ran is the prefixed tool name. Whose access it used is selected_key_id. Whether it was allowed was decided before the call executed, at the key. One row, three answers, which is exactly what a reviewer wants and rarely gets.

This holds even when you let the agent run on its own. In agent mode Bifrost auto-executes tools in a loop up to a max_agent_depth of 10, and every step in that loop is logged the same way. Autonomy does not mean going dark.

For tracing across a whole run, Bifrost emits OpenTelemetry spans in the genai convention, and each tool call carries attributes like mcp.method.name and gen_ai.tool.name. A Prometheus metric, bifrost_mcp_client_operation_duration_seconds, breaks latency down by client, tool, and error type. Point it at the Grafana or Datadog you already run and MCP activity shows up next to everything else.

If the tool inputs are sensitive, set disable_content_logging. You keep the metadata that proves what ran, who ran it, and how long it took, while the request and response bodies are dropped.

One clarification saves a lot of confusion. Bifrost Enterprise has an Audit Logs tab, and it is easy to assume that is your tool-call record. It is not. That trail tracks configuration changes: who created a key, who changed a policy, who signed in. The record of what the agents actually did lives in the observability logs, which even get their own store. Look in the right place.

Tying every call to a real person

A log line is only useful if it points at a human. This is where identity matters, and Bifrost keeps it mechanical rather than manual.

A user signs in through your company identity provider over OIDC, and Bifrost provisions them from that directory or takes a SCIM push in real time. Their group or claim maps to a role. That role carries an access profile, which is a reusable policy template, and the moment the role applies, Bifrost clones the template into a per-user copy and issues a managed virtual key. The key carries exactly what the person is allowed to do, including which MCP tools, and it is write-protected so the holder cannot edit around it.

That last detail matters more than it sounds. A user who can edit keys still cannot widen their own tool access, because the profile owns the key, not the person. You change access in one place, on the role, and it flows down to everyone who holds it. No per-user key sprawl, no forgotten grants.

The offboarding half is the part security actually cares about. When the identity provider marks a user inactive, or a SCIM delete arrives, or the periodic reconciliation no longer finds them, Bifrost decommissions the user locally. Their key stops working at both model and tool-execution time. A departed employee's agent cannot keep acting on Monday, because the wiring that granted access is the same wiring that removes it.

So every tool-call record traces back to a provisioned person, and access has a real lifecycle instead of a spreadsheet of keys nobody revokes.

Three catches most people miss

The honest gotchas, because a compliance story with no caveats is a sales page.

First, the two ways to scope MCP tools behave in opposite directions. At the virtual key, filtering is deny-by-default. A key with no MCP config gets no tools at all.

curl -X POST http://localhost:8080/api/governance/virtual-keys \
  -H "Content-Type: application/json" \
  -d '{"name":"billing-agent","mcp_configs":[{"mcp_client_name":"billing","tools_to_execute":["check_status"]}]}'

MCP Tool Groups, the enterprise layer for reusing tool bundles across teams, do the reverse. If no group matches a request, the request falls through with all tools available. Mix the two mental models and you will believe an agent is locked down when it is wide open. Decide which layer is your source of truth and test the empty case.

Second, Bifrost records that a tool ran, but it does not record your human approve-or-deny decision. When you gate a tool behind a person clicking yes, that click is yours to capture. Send it to the same audit sink as the execution, or your trail shows the action without the authorization behind it.

Third, know the line between free and paid. The tool-call logging, the virtual-key filtering, the tracing, and the MCP gateway itself are open source and run today. Access profiles, tool groups, role-based access, directory provisioning, and signed exportable audit logs are Enterprise. Plan the rollout around that line, not around a screenshot.

Final Thoughts

Compliance-ready for agents is not a certification you buy. It is a plain capability: you can hand someone every tool call an agent made, tied to a real person, against a policy that says whether it was allowed.

Most of that is reachable now. Route your MCP tool calls through a gateway that logs them, scope them at the virtual key, and wire access to your identity provider so it lives and dies with employment. Add the enterprise layer when you need signed audit and provisioning at scale.

The teams that get burned this year will not be the ones whose agents did something wrong. They will be the ones who could not say what their agents did at all.

Resources & References

Stay in Touch

Short takes and discussions on X
→ https://x.com/sebuzdugan

Practical AI / ML videos on YouTube
→ https://www.youtube.com/@sebuzdugan/

Partnerships & collabs
→ sebuzdugan@gmail.com

Originally published on Medium.

Hivemind Wrote 19 Skills From My Coding History. The Best Part Was What It Refused to Write.

Sebastian Buzdugan — Mon, 20 Jul 2026 15:28:17 +0000

I ran one command, pointed Hivemind at my real Claude Code history, and about two minutes later it had written nineteen reusable skills describing how I actually work. Then it refused to write one more, because my own repository already documented that pattern.

That refusal is the moment the tool earned my attention. Most "AI memory" products hoard everything and call it a feature. This one made a judgment call. I spent a day wiring Hivemind into my production Claude Code setup, breaking it, tracing a real bug through its source, and measuring what it actually does. What follows is the honest version: the wins, and the parts that cost you.

Hivemind is Activeloop's continual-learning layer for coding agents. It captures your agent sessions as traces, mines the repeated patterns into SKILL.md files, and feeds those skills, plus recalled memory, back into every agent on your team. Claude Code, Cursor, Codex, and a few others. The pitch is simple: the agent your junior used this morning is sharper because of what your senior's agent figured out last week.

Hivemind positions itself as one shared brain across Claude, OpenClaw, and Codex

The One-Command Install, and the Two Things the README Skips

Setup really is one command, followed by a browser sign-in.

npm i -g @deeplake/hivemind
hivemind install --only claude
hivemind login

Two things the README does not advertise. First, it wants Node 22, and I was on 20.19.5. It installed and ran anyway with an engine warning, but that is an unsupported combination you should know about before you build on it. Second, on the very first write the memory table did not exist yet, and the create-table call hit a duplicate-key race on pg_type_typname_nsp_index. It retried and healed itself within seconds, then logged capture ok to cloud. No data was lost, but a cold start that self-heals is worth watching.

Nineteen Skills, Mined From My Own Sessions in Two Minutes

This is the part that sells the product. I ran the miner against my own Claude Code history, and it produced nineteen named skills, each with real provenance in the frontmatter and cross-references to the others.

Nineteen skills mined from my own sessions, each carrying source_sessions provenance.

name: detached-remote-ml-jobs
source_sessions:
- e2f3cd85-54b8-45bf-87e4-9952f50eb6ac
created_by_agent: claude_code
created_at: 2026-07-14T06:28:40.506Z

These were not stubs. staged-ablation-cost-optimization captured how I pilot cheap runs before committing GPU budget. debug-silent-container-failures captured my exact habit of distrusting a green job status and reading the real container logs. detached-remote-ml-jobs even linked to idempotent-ml-pipeline-stages by name. The tool read how I work and wrote it down more clearly than I would have.

The Part That Impressed Me: It Refused to Learn What My Docs Already Covered

I asked it to mine the sessions where I wired a bot into my Telegram notification fabric. It returned zero candidates, and it told me why.

[96ba8816] 0 skill candidate(s) ... bot-tg-fabric wiring workflow
is already documented in CLAUDE.md and largely repo-specific. No
repeatable-mistake pattern detected.

That restraint is the whole game. A memory layer that codifies everything becomes noise you have to wade through. This gate only keeps non-obvious, repeatable gotchas that are not already written down. On a well-documented repository it declines, correctly, and your skill library stays dense with signal.

The Same Bug, Two Agents: One Had the Skill, One Did Not

Claims are cheap, so I ran a controlled test. Same prompt, a silent Azure container job that reports success but does nothing. One fresh agent had the codified debug-silent-container-failures skill in scope, one did not.

Same prompt, same model. The codified skill changed the diagnosis.

With the skill, the agent named it and led with the exact root cause: a try/except block that swallows the exception and exits zero, so the orchestrator reports success. Without the skill, it gave a generic debugging checklist and reached for Log Analytics wiring first, never surfacing that root cause. A skill mined from my own past session made a fresh agent measurably smarter on a matching problem. That is the propagation promise, working.

Recall Was Broken on My Machine, So I Traced It and Fixed It

Out of the box, memory recall did nothing for me. Every backfill of session summaries failed, so there was nothing to recall, and every recall event logged hit=none. This is the kind of thing a puff piece skips. I went into the bundle source instead.

The wiki-summary worker shells out to claude -p with Haiku to summarize each session, on a fixed 120-second per-session timeout with a concurrency of four. My machine has session transcripts ranging from 5 to 18 megabytes. A Haiku summary of even a small session took about 91 seconds on its own, so running four in parallel with an 18 MB transcript in the batch pushes everything past 120 seconds, and it gets killed. The pipeline was not broken. It was starved.

I fixed it by staging summaries for the sessions I cared about outside the batch path, then flushing them to the cloud table. After that, recall fired.

{"event":"injected","mode":"semantic","score":0.68}

Semantic recall, injected into context, with a real relevance score. The embeddings daemon spawned on demand, and it worked.

Where Recall Pays Off, and Where It Quietly Costs You

Here is the honest split, because it matters. I ran a cache-controlled A/B: the same tasks, memory on versus off, with turns as the metric that caching cannot fake.

The result I am not hiding: on short, documented tasks, recall cost more than it saved.

On short, well-documented coding tasks, proactive recall made things worse: roughly 75 percent more turns and 23 percent more cost on average. I ran three wiring tasks. With memory off, each took 4 turns. With memory on, one actually improved to 3, one rose to 7, and one ballooned to 11 as the agent tried to force an injected pattern that did not fit. That is an average of 4 turns going to 7. It pulled in tangentially related memory that sent the agent exploring. This is the wrong workload for it, and the fix is to raise the relevance threshold or turn proactive recall off for short asks.

On the workload it is built for, it shines. I asked a fresh agent, with no local files in front of it, how I had fixed a login-detection bug in an earlier session. It pulled the exact verify_login logic, the session ID, and the commit straight from memory. That is real cross-session recall, and you cannot get it any other way. Use it for memory-heavy work, not for tasks your documentation already answers.

The Sleeper Feature Nobody Markets: A Live Code Graph

Hivemind also builds an AST graph of your repository from the same traces. On my monorepo that came to 2449 nodes and 4333 edges across 237 files, queryable as a filesystem. It correctly surfaced my deliberate vendored-copy pattern, nine identical copies of the same notification client, and mapped call and import edges with line numbers. As a fast index to find the files that matter before you open them, it is genuinely useful, and it honestly declares what it cannot resolve.

The graph caught all nine vendored copies of my notification client.

What to Watch Before You Turn This On

The biggest caveat is privacy. Capture is org-wide across every project, not just the one you are working in. My shared memory accumulated summaries of unrelated, sensitive work, readable by anyone I add to the workspace and surfaceable by recall into unrelated sessions. The summaries themselves are high fidelity, with exact function logic and commit SHAs, and Hivemind does redact some secrets such as image tags, but it left cloud subscription and workspace IDs in the clear. Use a per-project workspace, and review before you add a teammate.

Final Thoughts

Hivemind is two products wearing one name. The codification engine, which turns your real sessions into reusable skills that provably sharpen a fresh agent, is excellent and under-marketed. The proactive recall is powerful on memory-heavy work and a tax on short, documented tasks, so treat it as a setting you tune, not a default you trust everywhere. If you run coding agents across a team, the skill layer alone is worth the install this week. Just scope your workspaces first, and remember that a green status never meant your job did any work.

Resources & References

Originally published on Medium.

Your Agent Benchmark Is Lying to You

Sebastian Buzdugan — Mon, 13 Jul 2026 18:12:14 +0000

Your Agent Benchmark Is Lying to You

The hard part of agent evals is the sandbox.

Here is a number that should bother you. On a suite of coding tasks I ran last week, the eval harness reported an 87% pass rate. The real pass rate was 33%.

Same agents, same tasks, same machine. The only difference was whether the harness trusted what the agent told it or checked what the agent actually did. Fifty-three percent of the "passes" were tasks the agent never solved. The harness scored them green anyway.

This is not a story about a bad model. It is a story about a bad harness, and almost every agent benchmark you have seen a leaderboard for is running one.

The benchmark conversation right now is stuck on scores. Which model got what percent, whether the number is saturated, whether it is contaminated. That argument misses the thing underneath. Before you can trust a score, you have to trust the machine that produced it, and that machine, the eval harness, is the least audited piece of the whole stack. So I audited one. I built a small harness, seeded it with the exact cheats documented in the literature, and ran it live on real microVM sandboxes. The results are reproducible and the code is public. This is what I found, and why the fix is not a better scoring rule, it is isolation.

The headline result: a trusting harness reported 87% passing; ground-truth verification put the real number at 33%.

The scores are downstream of a harness nobody checks

February 2026, OpenAI stopped reporting SWE-bench Verified, the benchmark it had built and human-validated itself back in 2024. The reason was not that models got too good. It was that the benchmark was broken in ways that took two years to surface: an audit of the hard subset found that "at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions," and frontier models could reproduce the ground-truth gold patch verbatim from contamination. The recommendation was blunt: "we have stopped reporting SWE-bench Verified scores, and we recommend that other model developers do so too."

Around the same time, a team at UC Berkeley published an audit of ten popular agent benchmarks. Their exploit agent, which was engineered to do everything except solve the task, achieved "near-perfect scores on most of the benchmarks without solving a single task." On SWE-bench, the trick was a ten-line conftest.py that intercepted pytest and forced every test to report passed. On WebArena, the agent read the answer straight off the local filesystem.

Their diagnosis is the sentence this entire article hangs on: "Any evaluation that reads state from a shared environment without careful validation can be defeated by an agent that writes state to that environment." The system under test could touch the thing grading it.

That is not a scoring bug. It is a containment bug. And it means the scores are only as trustworthy as the sandbox they were produced in.

What an eval rollout actually is

Strip an agent evaluation down and it is the same shape as a reinforcement learning rollout, just pointed at a different goal.

s0  ->  agent acts (edits files, runs commands)  ->  terminal state  ->  verdict

For training, you use the verdict as a reward. For evaluation, you use it as a score. Either way, the expensive, load-bearing part is the same: reconstruct a clean starting state s0, let the agent loose in it, and then decide whether the final state is correct. You do this thousands of times, once per task, often many times per task.

Every failure mode of a benchmark lives in one of those three steps. Get the starting state wrong and tasks flake. Let the agent influence the verdict and it cheats. Trust the agent's own report of the verdict and it lies. The leaderboard sees none of this. It just sees green and red.

Three ways a harness quietly breaks

These are structural failures, not bugs you can patch with a stricter assertion. Each one is a direct consequence of where the agent runs relative to where the grading happens.

It trusts the agent. The most common harness runs the repository's own test suite inside the agent's environment and trusts the exit code. But the agent is inside that environment too. It can drop a conftest.py that rewrites every outcome to passed, overwrite the test file with assert True, or just print "all tests passed" and exit zero. The harness cannot tell the difference between a solve and a forged report, because it is reading the report from the same place the agent was writing.

It has no isolation. Run many tasks in one long-lived environment, resetting the workspace between them, and state leaks. A task that writes outside the reset path, a stray file in the Python site directory, a mutated global config, a poisoned cache, silently corrupts every task that comes after it. The failures do not look like contamination. They look like the later tasks were bad.

Its verification is gameable. Suppose you get smart and stop trusting the agent's test run. You inspect the artifact directly. Better, but not enough, because if the agent can see the checks, it can satisfy them without solving anything. An agent that hardcodes the exact visible test cases passes your check and fails the moment you try an input it has not seen. This is reward hacking, and the research is clear that it gets worse as models improve. Cursor found that "63% of successful Opus 4.8 Max resolutions retrieved the fix rather [than] derived it," and METR documented frontier models that "monkey-patch the competition evaluation function so that it judges every submission as successful" while insisting they would never cheat.

Notice that all three failures share a root cause. The agent is close enough to the evaluator to influence it: its report, its neighbors, or its checks.

The fix is a causal chain, and it ends at the sandbox

Once you see that root cause, the fix writes itself as a chain, and every link forces the next:

The verifier's checks must be hidden from the agent, or it games them. Hiding them requires the agent to run in a context it cannot read or write, which is per-task isolation. Isolation you can afford at ten thousand rollouts requires forks that are near-free to create, which means snapshotting a starting state once and restoring it into fresh microVMs in milliseconds. And that primitive, a snapshot you fork into thousands of sealed, disposable environments, is exactly what Tensorlake provides, and what the Harbor evaluation framework runs on top of it.

Each link forces the next. Trustworthy verification ends at the sandbox.

So the trustworthy harness looks like this. Snapshot the task's starting state once. For each task, fork a clean microVM. Let the agent work only inside its fork. Then verify against the real final state using held-out checks the agent never saw, from a context it could never reach. The agent cannot forge the report, cannot poison its neighbors, and cannot study the answer key, not because you asked it nicely, but because the walls are real.

Everything below is that harness, run for real.

The experiments

I will start with the experiment that actually proves the point, because the others only show that harnesses can be fooled. This one shows that isolation is the thing that fixes it.

The actual run on Tensorlake microVMs. Every number below comes from this output, and it is reproducible from the repo.

Held-out verification, and why it needs isolation

Give the agent a module with a buggy add and mul, and a visible test that checks add(2,3)==5 and mul(3,4)==12. Now compare two "agents". One genuinely fixes the functions. The other hardcodes the visible cases:

def add(a, b):
    return {(2, 3): 5, (-1, 1): 0}.get((a, b), 0)
def mul(a, b):
    return 12 if (a, b) == (3, 4) else 0

Both pass the visible check. Only one solved anything. The way you tell them apart is a held-out check the agent never saw, add(10,20)==30, run at verify time:

honest_fix        visible-check=PASS   held-out=PASS
hardcode_visible  visible-check=PASS   held-out=fail   <-- FALSE PASS on visible-only

The held-out check catches the fraud. But here is the part that matters: the held-out check only works because the agent could not see it or reach the process running it. In a shared environment, the agent can read the harness files, watch what gets executed, and tune its hardcoding to match. The secrecy of the checks is not a property of the checks. It is a property of the isolation. Fork-per-task is what keeps the answer key in a room the agent is not in.

That is the whole argument in one experiment. Everything else is showing how badly the naive alternatives fail.

The suite, the ground truth, and how to reproduce it

The setup is deliberately small so it is fully reproducible with no model in the loop. The "agents" are scripted, deterministic bash behaviors: an honest fix, a no-op, a partial fix, the hardcode above, and the three cheats from the literature. Because there is no model call, the only variance in the system is the environment, which lets me isolate the harness's own reliability.

There are three verifiers. The trusting harness believes the agent's pytest exit code or its stdout claim. The ground-truth verifier imports the artifact directly and checks behavior, deliberately not through pytest, so a hijacked conftest cannot forge it:

# ground truth: import the built artifact and check it, bypassing the agent's test machinery
import importlib.util as u
spec = u.spec_from_file_location("impl", "/task/impl.py")
m = u.module_from_spec(spec); spec.loader.exec_module(m)
assert m.add(2, 3) == 5 and m.add(-1, 1) == 0 and m.mul(3, 4) == 12

And the held-out verifier is that same check with inputs the agent never saw, injected only at verify time into a fork the agent could not touch.

Each task runs the same way: fork a clean microVM from the snapshot, run the agent behavior, verify, terminate. The full harness is public at github.com/sebuzdugan/agent-eval-harness, so "reproducible across runs" is something you can check, not something you have to take on faith. On the Tensorlake free tier it runs sequentially, capped at one concurrent sandbox.

First, is the harness itself trustworthy? I ran the same honest task five times from the clean snapshot. Ground-truth PASS, five out of five, identical every time. A bit-identical starting state removes environment-induced flakiness, which is not a given: a recent study of agentic evals found single-run pass rates "vary by 2.2 to 6.0 percentage points depending on which run is observed... even at temperature 0.0." A clean snapshot puts that source of variance to zero, so the only variance left is the model's, which you handle by re-running on purpose rather than suffering by accident.

Trusting the agent is worthless

Now the cheats. The trusting harness against the three documented exploits, plus honest and no-op baselines:

honest_fix        trusting-harness=PASS   ground-truth=PASS
noop              trusting-harness=fail   ground-truth=fail
cheat_conftest    trusting-harness=PASS   ground-truth=fail   <-- FALSE PASS
cheat_edit_test   trusting-harness=PASS   ground-truth=fail   <-- FALSE PASS
cheat_selfreport  trusting-harness=PASS   ground-truth=fail   <-- FALSE PASS

Every cheat passes the trusting harness. The conftest one is the exact Berkeley exploit: ten lines that make pytest report success no matter what the code does.

import pytest
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    outcome.get_result().outcome = "passed"

Trusting harness: cheating agents pass three out of three. Ground-truth on visible checks: zero out of three. Verifying what the agent did instead of what it said closes the gap instantly, at least for these three.

The lie rate

Put it on a suite. Fifteen tasks, seeded with those reward-hacking behaviors at rates in line with what the literature reports:

trusting-harness pass rate: 13/15 = 87%
ground-truth pass rate:      5/15 = 33%
LIE RATE (trusting pass, real fail): 8/15 = 53%

The trusting harness overstates real capability by 53 points. If you shipped that number on a leaderboard, more than half of your "successes" would be fiction. This is not a hypothetical about some future deceptive model. It is what a normal harness reports today when normal, documented cheats are present.

Isolation is not optional

The last one is about neighbors, not verdicts. I ran six honest tasks in a shared sandbox, resetting the workspace files between each, with one poisoned task in the middle that writes a single file into the Python user-site directory, outside the reset path.

shared sandbox (reset workspace between tasks):
  task 0: honest_fix  PASS
  task 1: honest_fix  PASS
  task 2: poison
  task 3: honest_fix  FAIL  <-- SPURIOUS (contaminated)
  task 4: honest_fix  FAIL  <-- SPURIOUS (contaminated)
  task 5: honest_fix  FAIL  <-- SPURIOUS (contaminated)

isolated harness (fresh fork per task):
  every honest task PASS, poison contained to its own fork
  spurious failures:  shared=3   isolated=0

One poisoned task, three innocent tasks failed. Fork-per-task isolation takes it to zero.

Three perfectly good tasks marked failed, because a prior task left residue the reset did not clean. In the isolated harness, forking a fresh microVM from the snapshot for each task, the poison is trapped in its own disposable environment and the count is zero. A shared harness cannot promise this. Fork-per-task gets it for free.

And free is close to literal. On Tensorlake the forks boot in well under a second, and the fork-run-verify-terminate loop clocked about 16 tasks per minute even on the free tier's single-slot cap. Lift the concurrency, which the paid tiers take to over a thousand parallel environments, and that throughput scales roughly linearly. You are not trading trust for speed. You get both.

Where this shows up

The fork-and-verify primitive is not benchmark-specific. It is the shape of any place you need many trustworthy runs from one starting point. RL for coding agents uses it as the reward signal, the same rollout, scored instead of leaderboarded. Continuous integration for agents uses it to catch regressions before a release ships. A/B-ing two models against the same task suite uses it to get a comparison that is not swamped by harness noise. In every case the requirement is identical: a clean start the agent cannot corrupt, and a verdict the agent cannot forge.

Where it does not save you

I do not trust a piece that only sells the upside, so here are the edges.

Verification is only as good as the checks you write. A held-out check is stronger than a visible one, but a finite oracle can still be memorized with enough attempts, which is why real suites rotate and expand their held-out sets. Isolation solves containment, not check quality. External services your sandbox reaches over the network, a live database, a third-party API, are outside the snapshot and outside the guarantee; if your verdict depends on them, determinism gets harder. There is a per-restore floor, small but nonzero. And snapshots consume storage, so you prune.

None of these undo the argument. They tell you where it applies: heavy environments, many tasks, verdicts that must be trustworthy. Which is precisely agent evaluation.

Trust the harness before the score

The next time you read an agent benchmark number, ask a different question than everyone else is asking. Not "is the score high enough," but "could the agent have touched the thing that produced it." If the agent and the evaluator shared an environment, the honest answer is that you do not know, and the Berkeley result says the number is probably wrong.

The fix is not a cleverer metric. It is a wall. Snapshot the world once, fork a sealed copy for every task, and verify against reality with checks the agent never saw. That is isolation, it is what makes a score mean something, and with a snapshot-and-fork substrate like Tensorlake and Harbor it is something you get by default instead of something you hope for.

Clone the harness, point it at your own tasks, and watch your pass rate move when you stop trusting the agent and start checking the artifact. The gap you find is the amount your current numbers are lying to you.

The full harness and every result in this piece are reproducible at github.com/sebuzdugan/agent-eval-harness, run live on Tensorlake sandboxes (July 2026), no model or model API key required.

Resources & References

Stay in Touch

Short takes and discussions on X -> https://x.com/sebuzdugan

Practical AI / ML videos on YouTube -> https://www.youtube.com/@sebuzdugan/

Partnerships & collabs -> sebuzdugan@gmail.com

Originally published on Medium.

Governing Every LLM and MCP Call Across the Enterprise: Virtual Keys, Budgets, and Guardrails with the Bifrost AI Gateway

Sebastian Buzdugan — Thu, 09 Jul 2026 14:25:38 +0000

You have API keys in a vault, a budget dashboard, and a policy doc that lists which models are approved. Procurement signed off. On paper, enterprise AI is under control.

So answer one question: who called which model last night, with whose budget, through which MCP tool?

Nobody can say.

The calls nobody is counting

Enterprise AI usage did not arrive as one system. It arrived as a hundred small ones.

A data team wires GPT-5.5 into a pipeline with a key from the shared vault. A product squad ships a feature on Claude through its own SDK. An agent framework spins up MCP servers that call tools, read files, and hit internal APIs. Each of these is a real AI call leaving your infrastructure, and almost none of them pass through a single point where you could see them, price them, or stop them.

The result is not a security hole in the dramatic sense. It is worse: it is an accounting gap. You cannot answer basic questions. Which teams are spending what. Which models are actually in use versus approved. Which MCP tools an agent is allowed to invoke, and which it quietly invokes anyway.

Most teams try to close this with process. A spreadsheet of keys. A wiki page of rules. A quarterly review. Process does not enforce anything at request time, and request time is the only time that matters.

The fix is not another policy. It is a control plane that every LLM and MCP call has to pass through.

One base URL in front of every model

Bifrost is an open-source AI gateway built by Maxim AI, written in Go and licensed Apache 2.0. It sits in front of your model providers and gives you one OpenAI-compatible API for all of them: over 1000 models across every major provider behind a single endpoint.

The adoption cost is deliberately close to zero. You do not rewrite call sites. You change a base URL.

# Before: every app talks straight to the provider
client = OpenAI(base_url="https://api.openai.com/v1")

# After: the same call, now governed, logged, and routable
client = OpenAI(base_url="http://localhost:8080/openai")

That one line is the whole point. The SDK, the model names, the request bodies all stay the same. What changes is that the call now lands on something you control before it reaches OpenAI, Anthropic, Bedrock, Vertex, or any of the others.

The gateway is not a toll booth that slows everything down. You are buying governance, not latency: in Bifrost's own t3.xlarge benchmark, that control layer cost 11 microseconds of added overhead per request at 5,000 requests per second, and held a 100 percent success rate.

Standing it up locally is a single command:

npx -y @maximhq/bifrost

That pulls the binary, runs migrations, and serves a web UI plus the API on port 8080. I ran exactly this while writing, and everything below is what the running gateway actually did, not a paraphrase of the docs.

Governance that follows a hierarchy, not a spreadsheet

Once traffic flows through one place, governance stops being a document and becomes a data model. In Bifrost that model has four levels: Customer, Team, Virtual Key, and Provider Config.

The Virtual Key is the unit you hand out. It is not a raw provider key. It is a governed credential with a sk-bf- prefix that carries its own budget, rate limits, and allow-lists. I created one through the API and it came back live:

curl -X POST localhost:8080/api/governance/virtual-keys \
  -d '{"name":"marketing-team-vk","is_active":true,
       "rate_limit":{"token_max_limit":1000000,"token_reset_duration":"1h",
                     "request_max_limit":1000,"request_reset_duration":"1h"}}'

The response handed back sk-bf-82a7fb23... with a rate-limit object already ticking: token_current_usage: 0, an hourly reset window, request counters primed. That key is the thing a team codes against. Everything you attach to it travels with every call it makes.

Budgets cascade. A Provider Config can cap the OpenAI spend inside a key, the key has its own ceiling, the team above it has another, and every applicable budget has to pass for the request to proceed. Cost deducts from each level at once. Budgets reset on windows from one minute to one year, and calendar_aligned snaps them to real month and year boundaries instead of rolling clocks. Rate limits come in two flavors, request-based and token-based, and unlike budgets they apply at the Virtual Key level only.

Access control is enforcement, not advice. Put a model or provider outside a key's allow-list and the call returns a 403 with a typed reason, model_blocked or provider_blocked. There is no soft warning to ignore.

One gotcha worth knowing before you wire this into config: a budget attaches as its own referenced object, not as an inline block on the key. I tried to embed the budget directly in the create call and it came back unattached. Create the budget, reference it by id. The docs are right; the shortcut is not.

Governing every MCP call, not just every model call

Chat completions were the easy half. The harder half, and the reason this topic exists, is MCP.

Model Context Protocol is what turns a chat model into something that acts: it calls tools, runs commands, reads and writes real systems. An ungoverned MCP layer is an agent with hands and no supervisor. Bifrost treats MCP as a first-class governed surface, and it does so with tool filtering that is deny-by-default at three levels.

Client config sets the baseline of which tools exist at all.
Request headers (x-bf-mcp-include-clients, x-bf-mcp-include-tools) narrow that set per call.
Virtual Key filtering overrides both, so a key with no MCP config gets no MCP tools, full stop.

Deny-by-default is the whole posture. A tool a key was never granted is not available to it, even if the underlying MCP server exposes it.

How the calls actually run is configurable, and the default is the safe one:

Tool Execution mode is human-in-the-loop. Bifrost does not auto-run tool calls; your app reviews them and explicitly hits /v1/mcp/tool/execute. That review point is where approval and audit live.
Agent Mode auto-executes only the tools you mark in tools_to_auto_execute, bounded by max_agent_depth (default 10).
Code Mode lets the model write Starlark to orchestrate many tools in one shot through four meta-tools, which Bifrost measures at up to 92.8 percent fewer input tokens than exposing every tool directly.
Bifrost as an MCP gateway exposes a /mcp endpoint that external hosts like Claude Desktop, Cursor, and Cline connect to. On the running instance that endpoint answered as a live streaming connection, held open the way an MCP host expects.

For teams that want reusable policy, MCP Tool Groups (an Enterprise feature) bundle curated tools and attach them to keys, teams, or users, enforced at request time.

The one-sentence version: the gateway governs what an agent is allowed to do, not just which model it is allowed to call.

What you can actually see

Governance you cannot observe is governance you cannot prove. Bifrost logs every request asynchronously, so the visibility does not cost latency.

Each entry carries the model, token counts, cost, and the identity behind the call. When a request fails over or rotates keys, an attempt_trail records every attempt, the selected_key_id, and the retry count, so a weird bill or a slow p99 has a paper trail instead of a shrug.

The exports are the standard enterprise set:

Prometheus metrics are built in on /metrics, live out of the box (on a fresh instance you will see the runtime metrics first; the request metrics fill in as traffic flows).
OpenTelemetry export speaks the genai_extension semantic conventions to Grafana, Datadog, New Relic, or Honeycomb.
A native Datadog connector and log exports to S3, GCS, and BigQuery are available on Enterprise.

If content is sensitive, disable_content_logging keeps the metadata, latency, cost, tokens, and identity, while dropping the request and response bodies. You get the accounting without hoarding the payloads.

The controls that make security sign off

Everything so far runs in the open-source gateway. The pieces a security team asks about before rollout live in Bifrost Enterprise, which is a strict superset: same config schema, same providers, more control.

Identity comes through OIDC and SCIM, with SSO against Okta, Microsoft Entra, Keycloak, Zitadel, and Google Workspace, plus group-to-role mapping and auto-deprovisioning. Access is role-based across Admin, Developer, and Viewer, the permission set widening at each tier instead of collapsing into a single all-or-nothing admin. Data Access Control adds row-level scoping, so an operator sees only their own, their team's, or all data.

Audit logs are immutable and HMAC-signed, exportable as JSON, JSON Lines, or Syslog straight into a SIEM. That is what SOC 2 Type 2, GDPR, HIPAA, and ISO 27001 evidence actually looks like.

Guardrails split into Rules, written in CEL, and Profiles that decide how to act: native Gitleaks-backed secret detection and PII regex, plus external providers including AWS Bedrock, Azure Content Safety, Google Model Armor, CrowdStrike AIDR, GraySwan Cygnal, and Patronus AI. For scale, clustering keeps state in sync across nodes with no single leader.

This is the layer that turns "we route through a gateway" into "we can show an auditor who did what, signed and exportable straight into the SIEM."

Extending the gateway to the laptop

The gateway governs every call that routes through it. The last mile is reach. Think of the app a developer installed this morning that talks straight to a provider from a laptop, never pointed at your gateway. Getting the same governance onto that machine is the other half of the job.

That is where Bifrost Edge and the gateway work together. Edge is a lightweight endpoint agent for macOS, Windows, and Linux that runs in the menu bar or system tray and intercepts AI traffic at the machine level. There is no base URL to change per app and nothing for the user to configure. Once a machine signs in through your existing identity provider, the same virtual keys, budgets, guardrails, and audit logs you already run on the gateway apply on the laptop too.

Edge adds two enforcement surfaces the endpoint needs:

App allow and deny, fleet-wide. An approved app runs fully governed; a denied one is blocked before any data leaves the machine.
MCP server governance on the device. Edge discovers the MCP servers configured inside Claude Code, Claude Desktop, Gemini CLI, OpenCode, Codex, and Cursor, dedupes them across the fleet, and lets you allow or deny each one centrally. Deny a server and it is stopped on the machine, even inside an app that had it configured before your policy existed.

Rollout is hands-off, through the MDM you already run: Jamf, Intune, and Kandji, with Omnissa Workspace ONE and JumpCloud in the mix as well. Admins work from an Approvals dashboard (Pending, Approved, Denied) and a Devices fleet view. The gateway centralizes the policy; Edge carries it to every machine.

What isn't ready yet

Edge is not something you can turn on this afternoon.

It is in limited alpha, or early access, and you register to be onboarded rather than downloading a shipping release. It is an Enterprise feature, not part of the open-source repo, so you will not find it in the Bifrost you clone from GitHub. That status is current as of July 2026; check the docs before you quote a timeline, because alpha windows move.

It is also not a replacement for your endpoint or network security. Edge governs AI traffic specifically. It runs alongside tools like CrowdStrike, Zscaler, and Tailscale, not instead of them.

None of that changes the part you can use today. The open-source gateway is real, it installs in one command, and every governance, MCP, and observability capability above ran on the instance I stood up while writing this. The endpoint layer is where the story is still being finished.

Where to start

Start with the gateway, because it is the part you can run today. Install it with npx -y @maximhq/bifrost, point one base URL at it, and route your team's traffic through it. Then spend an afternoon on the governance: issue a virtual key per team, attach budgets and rate limits, turn on audit logging. At that point every LLM and MCP call your infrastructure makes is attributed, bounded, and logged, and none of it required rewriting a call site.

The endpoint is the next problem, not the first one. When you are ready to bring the laptops under the same policy, Bifrost Edge is the extension that gets you there, on the timeline the alpha allows.

Every technical detail here is drawn from the Bifrost gateway and Edge documentation and confirmed against a running instance. If something in this piece does not match what you see in the docs, trust the docs.

Resources & References

Originally published on Medium.

How to Fine-Tune a 7B Model for Three Dollars on One GPU

Sebastian Buzdugan — Sun, 05 Jul 2026 11:43:18 +0000

You think fine-tuning a 7B model needs a rack of A100s. It needs one 16GB card and about three dollars of rented compute. The gap between those two beliefs is costing teams entire projects they never start.

The cluster you think you need does not exist

Ask most engineers what it takes to fine-tune an open model and you get some version of "we don't have the hardware." They picture full fine-tuning: every weight in fp16, gradients and optimizer states for every parameter, a 7B model ballooning past 100GB of VRAM before the first batch even lands.

That math is real, and it is why the instinct is to give up and go back to prompt engineering a frontier API.

But you are solving a problem that was fixed in 2023. Full fine-tuning is not the only option, and for the overwhelming majority of practical tasks it is the wrong one. QLoRA, introduced by Tim Dettmers and collaborators, dropped the memory floor so far that a 65B model fits on a single 48GB GPU while matching full 16-bit fine-tuning quality.

Scale that down to the sizes you actually ship. A 7B fine-tune runs comfortably on a 16GB card. A 13B fits on a 24GB desktop 4090. You are not renting a cluster. You are renting one GPU for an afternoon.

Why four bits is enough, and where the memory actually goes

Full fine-tuning spends VRAM on four things: the model weights, the gradients, the optimizer states, and the activations. QLoRA attacks the first three at once.

The trick is a clean split. The base model is frozen and stored in 4-bit, using a data type called NF4 (4-bit NormalFloat) that is information-theoretically optimal for the normally-distributed weights neural networks actually have. That 4-bit form is storage only. For the forward and backward pass, each block is de-quantized back to bf16 on the fly, so the math stays high-precision while the resting footprint drops roughly 4x.

You never compute gradients for those frozen weights. Instead you train small low-rank adapter matrices bolted onto each linear layer, typically well under one percent of the model's parameters. Optimizer state, the silent VRAM killer in full fine-tuning, now covers only those tiny adapters.

Two smaller innovations close the gap. Double quantization quantizes the quantization constants themselves, saving about 0.37 bits per parameter, roughly 3GB on a 65B model. Paged optimizers use NVIDIA unified memory to absorb the gradient-checkpointing spikes that would otherwise OOM you on a long sequence.

The result is not a compromise you tolerate. On the paper's own MMLU evaluation, 4-bit NF4 with double quantization replicated 16-bit LoRA performance across LLaMA 7B through 65B. Independent 2026 runs put QLoRA within one to two percent of full-precision LoRA on standard benchmarks. You give up almost nothing measurable.

What it actually costs to run

Here is the part that turns this from theory into a Tuesday afternoon. These are the realistic 4-bit VRAM footprints for a QLoRA fine-tune, base model plus adapters and activations:

7B: comfortable on a 16GB card
13B: fits on a 24GB card (a desktop RTX 4090)
30B to 32B: needs roughly a 40GB card
70B: about 46GB, which fits on a single 48GB card

Sequence length and batch size move these numbers, but not as much as you would fear. An 8B model at a 2048-token sequence length peaks around 6.6GB of reserved VRAM on a 4090 with Unsloth. You have headroom on hardware you may already own.

Hold that against full fine-tuning of the same 7B model. Weights, gradients, and Adam's two optimizer states in fp16 push you well past 100GB before activations, which means multiple 80GB A100s and a multi-GPU setup to coordinate. QLoRA takes that same job from a cluster you have to requisition down to a single card you can rent by the second. The 4x drop from 4-bit storage is only part of it; freezing the base and training sub-one-percent adapters is what deletes the optimizer-state mountain entirely.

Now put a clock and a price on it. A 7B QLoRA run of two to three epochs takes roughly two to four hours on an A100 and six to eight hours on an RTX 4090. On RunPod, a Community Cloud 4090 rents from about 0.34 USD per hour and an A100 PCIe from about 1.39 USD per hour, billed by the second.

Do the arithmetic. A full 7B fine-tune on a 4090 lands around two to three dollars. On an A100 you trade money for wall-clock and finish before lunch for roughly the same total. This is the number that should reframe your roadmap: a custom model is a coffee-run expense, not a capital request.

The recipe that works on the first try

The tooling collapsed into something you can hold in your head. bitsandbytes handles the 4-bit quantization, PEFT handles the adapters, and Unsloth wraps both with kernels that cut memory and roughly double throughput. Start by loading the base model in 4-bit.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meta-llama/Llama-3.1-8B",
    max_seq_length=2048,
    load_in_4bit=True,
    dtype=None,
)

If you would rather stay in plain Transformers, the equivalent is a BitsAndBytesConfig that encodes the same three ideas: NF4 storage, bf16 compute, double quantization on.

from transformers import BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

Next, attach the adapters. The single most important choice is targeting all the linear layers, not just the attention projections. Skipping the MLP layers is the most common reason a fine-tune underperforms for no obvious reason.

model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth",
    lora_dropout=0, random_state=3407,
)

Two hyperparameters carry the load. Rank r controls adapter capacity; 16 is a strong default, and 32 if the task is genuinely complex. For alpha, the old alpha = 2 * r convention still works, but Unsloth's 2026 ablations found alpha = r is the cleaner default. Gradient checkpointing set to "unsloth" buys you another 30 percent memory reduction, which is often the difference between fitting and not.

Then the training arguments. Prefer a small batch size with gradient accumulation rather than a large batch, because that is how you avoid out-of-memory errors while keeping an effective batch size in the healthy 4 to 16 range.

from trl import SFTConfig

args = SFTConfig(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    num_train_epochs=2,
    warmup_ratio=0.05,
    weight_decay=0.01,
    seed=3407,
)

That configuration is not a starting guess to tune for a week. It is close to the endpoint for most supervised fine-tuning jobs. Learning rate 2e-4, one to three epochs, effective batch 16, warmup around five percent. Change the data, not the knobs.

From a trained adapter to a model you can serve

What you get out of a QLoRA run is not a fresh multi-gigabyte model. It is an adapter: a small set of low-rank matrices, often around 100MB, that sits on top of the frozen base. That size is a feature. You can train a dozen task-specific adapters against one base model and store them all for less than the footprint of a single full checkpoint.

You have two ways to serve it. Keep the adapter separate and load it over the base at runtime, which lets you hot-swap behaviors, or merge it into the weights for a single self-contained model. For most deployments, merge and export.

model.save_pretrained_merged("llama-8b-support", tokenizer,
                             save_method="merged_16bit")
model.save_pretrained_gguf("llama-8b-support", tokenizer,
                           quantization_method="q4_k_m")

The merged 16-bit version drops straight into vLLM or TGI for a production endpoint. The GGUF export, quantized to something like q4_k_m, runs on Ollama or llama.cpp on a laptop with no GPU at all. The same afternoon that produced your adapter can end with the model answering requests on your own machine.

One caution on inference: if you serve the base in 4-bit to save memory, evaluate in that exact configuration. Quality measured on a 16-bit merge does not always transfer cleanly to a 4-bit serving path, and the gap is easiest to catch before you ship, not after.

When a fine-tune is the wrong answer

The fastest way to waste your three dollars is to fine-tune a problem that RAG should own. In 2026 the question is no longer "RAG or fine-tuning," it is where each piece of intelligence should live: in the weights, in retrieval, or in both.

Reach for retrieval when the knowledge changes often, is large, needs citations, or differs per user. Facts, documents, prices, anything you would be embarrassed to see go stale inside a model, belong in a vector store you can update without retraining. Fine-tuning bakes knowledge in at a moment in time; that is a liability for anything dynamic.

Reach for a fine-tune when you need consistent behavior that prompting cannot reliably enforce: a fixed output format, a specific tone or policy, domain reasoning, or structured output that has to be right every time. It also wins on economics when volume is high enough that a small fine-tuned model beats per-call frontier pricing, or when your latency budget cannot afford a retrieval hop.

The mature pattern is both. Fine-tune for style, format, and decision behavior; retrieve for the facts. Teams that pick a side out of principle usually ship a worse product than teams that route each concern to the tool built for it.

The failure modes nobody warns you about

QLoRA is forgiving, but three things quietly ruin runs.

Overfitting is the big one, and it hides in a metric that looks like success. If your training loss drops below 0.2, the model has likely memorized your data and will generalize worse, not better. When you see that, cut epochs, raise weight decay, or scale alpha down by half. More than three epochs almost always trades generalization for a prettier loss curve.

The second is trusting the loss curve at all. A falling loss is not a working model. You need a held-out eval that reflects the actual task, checked before and after, or you are flying blind. Domain-specific evaluation matters far more than any general benchmark score your base model advertises.

The third is a subtle capacity mismatch. If a fine-tune underperforms and your data is clean, the culprit is usually rank set too low or target modules set too narrow, not the method failing. Bump r to 32 and confirm you are targeting every linear layer before you conclude QLoRA cannot learn your task.

One optional upgrade is worth knowing: DoRA, weight-decomposed low-rank adaptation, often squeezes out a bit more accuracy at low ranks for a small speed cost. Try it once the plain recipe works, not before.

Final Thoughts

The reason to internalize this is not that fine-tuning is trendy. It is that the cost of trying just fell through the floor, and cost of trying is what governs how much you experiment.

When a custom 7B model costs three dollars and one afternoon, fine-tuning stops being a quarterly initiative that needs sign-off and becomes something you do on a hunch, twice, before standup. That changes which ideas are worth testing. The team that ships ten cheap fine-tunes and keeps the two that work will out-iterate the team still waiting on a GPU budget.

Pick one narrow, format-heavy task your frontier API keeps getting subtly wrong. Rent a 4090 for an evening. The worst case is you spent the price of a coffee to learn something concrete about your own data.

Resources & References

QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., arXiv)
LoRA Fine-Tuning Hyperparameters Guide (Unsloth Documentation)
Making LLMs Even More Accessible with 4-bit Quantization and bitsandbytes (Hugging Face)
GPU VRAM Requirements to Fine-Tune LLMs in 2026: Full, LoRA, and QLoRA Sizing (Spheron)
RAG vs Fine-Tuning for LLMs in 2026: What Actually Works in Production

Originally published on Medium.

Snapshot Once, Rollout a Thousand Times: A Practical RL Setup for Coding Agents

Sebastian Buzdugan — Wed, 01 Jul 2026 09:09:10 +0000

Your RL run has been going for six hours. The GPUs are warm, the reward curve is creeping up, the policy is learning. Good.

Now look at what those six hours actually bought you. For a large slice of them, your expensive accelerators sat idle, not waiting on the optimizer, not waiting on the model, but waiting on an environment to boot. Clone the repo. Install the dependencies. Restore the dataset. Then, finally, let the policy take its first action.

This is the part nobody puts in the RL paper. The reward function gets a section. The KL penalty gets a section. The thing that actually ate your wall-clock, replicating the environment for every single rollout, gets nothing, because it feels like plumbing.

It isn't plumbing. For agent RL, it's the bottleneck. And it's a bottleneck with a clean fix that almost nobody is using yet.

I spent the last few days testing that fix on real infrastructure. This piece is what I found: the primitive, a working rollout harness you can run yourself, the real timings, and the places it doesn't help so you know where the edges are.

The rollout is a state-replication problem wearing a compute costume

Strip an RL step down to its skeleton:

s0  ->  policy samples an action  ->  env transitions  ->  ...  ->  terminal  ->  reward

For a math problem, s0 is a string. For a coding agent, s0 is a world: this repo at this commit, this Python version, these installed packages, this dataset on disk, this half-built state from a setup script. Reconstructing that world is the price of admission for every rollout you run.

And in GRPO, GSPO, or any group-based method, you don't run one rollout per step. You run G of them, eight, sixteen, sixty-four completions, each needing its own clean copy of s0. Multiply by thousands of steps. The environment gets rebuilt tens of thousands of times across a single run.

There are three ways teams handle this today, and all three are workarounds for the same underlying problem.

Containers per rollout. Spin up a fresh container, pull the image, install, run, tear down. Clean isolation, but you pay the cold-start-plus-setup tax on every single rollout. Tens of seconds, every time, before the policy moves.

Warm pools. Keep a fleet of pre-built containers ready. Faster, but now you own pool drift, eviction logic, health checks, and the ops surface of a small distributed system. You traded a latency problem for an infrastructure problem.

One VM, sequential rollouts. Cheap and simple, until you realize rollout three poisoned the filesystem that rollout four needed, and you've serialized the one thing you most wanted to parallelize.

Every one of these is dancing around a single fact: replicating environment state is expensive, so we keep paying for it or building scaffolding to avoid paying for it. What if replicating state was just cheap?

The primitive: snapshot once, restore N

Here's the move. Instead of rebuilding s0 for every rollout, you build it once, freeze it, and hand out identical copies.

The substrate is a microVM sandbox. It boots in well under a second, and you can snapshot its entire state and restore that snapshot into fresh, independent VMs. I tested this on Tensorlake sandboxes, which is what the code below uses, but the pattern is the point. The setup is unremarkable:

from tensorlake.sandbox import Sandbox, CheckpointType

# Build the world ONCE.
sb = Sandbox.create(name="rl-canonical")
sb.run("bash", ["-lc", "mkdir -p /home/tl-user/proj"])
sb.write_file("/home/tl-user/proj/impl.py", buggy_module)
sb.write_file("/home/tl-user/proj/test_impl.py", hidden_tests)
sb.run("bash", ["-lc",
    "cd /home/tl-user/proj && "
    "pip install --break-system-packages -q numpy pandas requests pytest"])

# Freeze it.
snap = sb.checkpoint(checkpoint_type=CheckpointType.FILESYSTEM)

That snap is s0. Restoring it gives you a fresh, byte-identical world to roll out in:

fork = Sandbox.create(snapshot_id=snap.snapshot_id)   # a clean copy of s0

Before we go further, the obvious objection, because it's the right one to ask:

"Why not just bake a Docker image?"

Because an image and a snapshot are not the same kind of thing, and the difference is exactly what makes this work.

An image distributes an environment. A snapshot captures runtime state.

An image is a recipe for building an environment: the repo, the language, the dependencies. Great for distribution. But it's frozen at build time. It knows nothing about what happened after the container started: the files your setup script generated, the dataset you downloaded, the checkpoint your last run wrote, the config a previous step mutated.

A snapshot is taken at runtime. It captures the world as it actually is, mid-flight, including everything an image structurally cannot.

I tested this directly. I built a sandbox the way a real session would: installed deps, downloaded a file, generated an embeddings.npy from running code, wrote a checkpoint directory, and mutated a config. Then I snapshotted it and restored into a fresh sandbox. What survived:

deps:        requests + numpy importable
download:    downloaded_readme.md present
generated:   embeddings.npy  (1000, 128)
checkpoint:  step=4200 loss=0.13
run_state:   resume_from=4200

A Dockerfile gives you the first line. The other four, the runtime state that's often the entire point of s0, exist only because the snapshot captured them. That's the distinction in one sentence: an image distributes an environment; a snapshot captures runtime state.

A rollout harness you can actually run

Let's make it concrete. The toy task: an agent has to fix a buggy Python module so a hidden pytest suite passes. Small enough to fit in an article, structurally identical to anything real you'd train on. The reward is the fraction of tests that pass: clean, dense enough to learn from, impossible to game.

The starting state s0 is the module-with-bugs plus the installed environment, snapshotted once (the code above). Now the rollout step: for each of G policy samples, restore s0, apply the candidate, score it.

import concurrent.futures
from tensorlake.sandbox import Sandbox

def rollout(i, candidate_patch):
    # Each rollout gets its own clean copy of s0.
    fork = Sandbox.create(snapshot_id=snap.snapshot_id)
    fork.run("bash", ["-lc", f"cd /home/tl-user/proj && {candidate_patch}"])
    res = fork.run("bash", ["-lc",
        "cd /home/tl-user/proj && python3 -m pytest -q 2>&1 | tail -1"])
    reward = pass_fraction(res.stdout)      # 0.0 .. 1.0
    fork.terminate()
    return i, reward

# Fan out G rollouts from the one snapshot, in parallel.
G = 8
with concurrent.futures.ThreadPoolExecutor(max_workers=G) as ex:
    group = list(ex.map(lambda a: rollout(*a), enumerate(candidates)))

In a real loop, candidates comes from your policy. Here I scripted eight candidate patches of varying quality so the whole thing is deterministic and you can reproduce it without a model or an API key. The point isn't the policy, it's what the sandbox does underneath it.

Here's the actual output from my run:

rollout[0] reward=0.00   (3 failed)
rollout[1] reward=0.33   (2 failed, 1 passed)
rollout[2] reward=0.33   (2 failed, 1 passed)
rollout[3] reward=0.67   (1 failed, 2 passed)
rollout[4] reward=0.33   (2 failed, 1 passed)
rollout[5] reward=0.33   (2 failed, 1 passed)
rollout[6] reward=0.33   (2 failed, 1 passed)
rollout[7] reward=0.67   (1 failed, 2 passed)
group mean reward = 0.375     best = 0.67

That vector of rewards is the whole game. And here's the thing worth internalizing: getting G independent rollouts from one identical starting state is the expensive part, and it's identical across methods. What you do with the rewards afterward is where the techniques split. Rejection-sampling fine-tuning keeps the best completion and trains on it; GRPO and GSPO use the entire group, computing each rollout's advantage as its reward minus the group mean, and nudge the policy toward the above-average ones. Same expensive primitive underneath. The snapshot is what makes that primitive cheap.

One snapshot, G rollouts, one reward vector. Rejection sampling keeps the best; GRPO uses the whole group. The fan-out is the same either way.

The economics, which is the actual argument

Now the part that matters. Let's price the two approaches on the same workload, and let's be fair about it: both of them spin up a fresh sandbox per rollout, so both pay that cost. The only thing that differs is the build.

Two numbers from my runs. Building the world (installing numpy, pandas, requests, and pytest, plus generating a dataset) took 7.2 seconds. Restoring a sandbox from a snapshot took about 2 seconds (sub-second on capable infrastructure; free-tier contention pushed mine higher). Now price a full run, G = 8 rollouts per step, 1000 steps, 8,000 rollouts:

per rollout:    build 7.2s   +   restore ~2s
G = 8, 1000 steps  ->  8,000 rollouts
naive loop:     8,000 x (7.2s build + 2s restore)        ~=  20 hours
snapshot loop:  7.2s build once  +  8,000 x 2s restore   ~=  4.4 hours
                -----------------------------------------------------
saved:          ~16 hours, all of it rebuilding identical state

Same 1000-step run at G=8. Both loops pay the per-rollout sandbox spin-up (~4 hours). The snapshot loop deletes the ~16 hours of rebuilding identical state.

The ~4 hours of sandbox spin-up is real, and the snapshot loop still pays it. So does the naive loop. That part is a wash. What the snapshot deletes is the 16 hours spent rebuilding the same world 8,000 times. You're not making restores free; you're removing the redundant builds.

And that's with a trivial seven-second environment. Swap in a real coding-agent setup (a heavy requirements.txt, a model download, a dataset) and the build cost climbs from seconds to minutes while the restore stays flat. The gap widens until "rebuild every rollout" stops being slow and starts being the difference between a run you can afford and one you can't. Structurally, you've turned an O(G x steps) build cost into O(1).

One honest note on that restore number. I ran on the free tier (one vCPU, ten concurrent sandboxes), so eight simultaneous restores contended and per-rollout boots stretched to two to seven seconds rather than the sub-second you'd see on real infrastructure. I used ~2s as a fair middle estimate. Tensorlake quotes 10,000+ parallel environments on their full stack, where both the restores and the parallelism get much cheaper. None of it changes the shape: build once, not 8,000 times.

Where this pattern shows up

This isn't a Tensorlake trick. "Snapshot a starting state, fork it into many independent rollouts" is a shape that recurs the moment you need many runs from one point. Three places it's already load-bearing:

RL training. The case we just walked through. Tensorlake's own GSPO cookbook does exactly this at proper scale, dispatching G completions per step to parallel sandboxes, scoring each against a hidden test suite, using the group to update the policy. It's a working reference implementation if you want more than an architecture diagram.

Agent evaluation. Same primitive, different goal. Instead of training, you're scoring. Snapshot the benchmark's starting state, fork it across every task in the suite, run your agent in each isolated copy, collect pass/fail. Tensorlake plugs into Harbor (a framework for defining and verifying terminal tasks) as the execution runtime, running fleets of sandboxes for terminal-bench-style evaluation with real filesystem verification instead of trusting an agent's self-report.

Parallel search at inference. Drop the training entirely. Best-of-N sampling, tree search over tool calls, speculative execution of multiple plans, all of them are "fork the current state, explore N branches, keep the good one." The same snapshot-and-fan-out, just at inference time instead of training time.

The through-line: state is what makes parallelism expensive, and a cheap snapshot removes the cost. Once that's true, a pile of architectures that were too painful to build become reasonable.

Where it doesn't help (the honest part)

I don't trust a piece that only tells you the good news, so here's the map of the edges.

External state isn't in the snapshot. Anything your sandbox reaches over the network (a production database, a third-party API, a shared queue) is outside the VM and outside the freeze. Fork a sandbox mid-API-call and all N forks will independently try to finish that call. If your reward function touches an external service, idempotency and determinism stop being nice-to-haves.

Pick the right checkpoint type. Sandboxes give you a filesystem snapshot and a memory snapshot. For rollouts you almost always want the filesystem one: you want a clean disk reset from s0, not a frozen process tree. The memory snapshot, which additionally captures live RAM and running processes, is for pausing and resuming an actual session, and it costs more. In my tests the memory variant added a few hundred megabytes of captured RAM on top of the disk image. Use it deliberately, not by default. One genuinely counter-intuitive result: the heavier memory snapshot restored faster than the filesystem one in my runs (about 0.9s versus 1.9s) because the page cache comes back warm with it.

There's a per-restore floor. Restoring a snapshot isn't free. On capable infrastructure it's sub-second; under a throttled tier it's a few seconds. If your policy step is itself sub-second, that floor is a real fraction of your time. If your rollouts are LLM calls measured in seconds, it vanishes into the noise. Know which regime you're in.

Snapshot storage is real. A snapshot tracks the provisioned disk, not a tiny diff. Keep thousands of them around and you're paying for thousands of disk images. Snapshot deliberately and prune.

None of these sink the approach. They tell you where it fits: heavy environment, many rollouts, isolation that matters. Which is precisely the shape of agent RL.

The takeaway

If your training loop spends a meaningful chunk of its wall-clock waiting for environments to come up, you are paying a tax you don't have to pay. Rejection-sampling fine-tuning, GRPO and GSPO rollouts, agent eval harnesses, paired DPO sampling, every one of them is expensive for the same reason, and it's not the reason you think. It's not the GPUs. It's that you're rebuilding the same world, over and over, for no reason.

Snapshot it once. Fork it as many times as you have rollouts. Throw the forks away.

The cheapest way to feel the difference is to run it. Spin up a sandbox, build something stateful inside it, snapshot it, and restore that snapshot into three fresh sandboxes in parallel. Watch the world appear three times from one freeze. That shift, from rebuilding state to restoring it, is the kind of thing you have to see once before it changes how you architect the whole loop.

I tested everything in this piece against Tensorlake sandboxes on the free tier (June 2026). The rollout harness and benchmarks are reproducible: scripted candidates, no model or API key required.

Resources & References

Stay in Touch

Short takes and discussions on X -> https://x.com/sebuzdugan

Practical AI / ML videos on YouTube -> https://www.youtube.com/@sebuzdugan/

Partnerships & collabs -> sebuzdugan@gmail.com

Read the full piece on Medium

https://medium.com/data-science-collective/snapshot-once-rollout-a-thousand-times-a-practical-rl-setup-for-coding-agents-0f880a450610

Handling Multi-Model API Outages Without Melting Production

Sebastian Buzdugan — Wed, 24 Jun 2026 12:47:36 +0000

Your dashboards go red. Not one model. All of them. Retries spike. Latency climbs. Nothing recovers.

When every Claude call starts failing at once

If you ship anything serious on top of LLM APIs, you have lived this moment.

Requests that worked minutes ago now return errors. You flip the model flag. Same thing. You increase retries. Error rate goes up, not down. Queues back up. Downstream services feel it.

This is not a prompt issue. This is not a bad deploy. This is what a platform-level incident looks like from the client side.

The recent Anthropic incident labeled "Elevated error rate across multiple models" is a clean example of the failure mode that matters most in production. Multi-model impact. No numbers. No knobs to turn. Just errors.

The mistake teams make is assuming model diversity equals resilience. It does not.

Why model fallback quietly fails during platform incidents

Most LLM client stacks are built around a simple idea: if model A fails, try model B.

That works when failures are model-scoped. It fails when the blast radius is shared infrastructure.

Under the hood, models often share front-door API gateways, auth and quota systems, routing layers, and shared inference clusters or schedulers.

When error rate rises across multiple models, it usually means something above the model is degraded. Switching from Opus to Sonnet does not bypass the problem because the problem is not Opus or Sonnet.

The docs do not emphasize this. SDK examples actively encourage retry and fallback loops that assume independence. In production, this assumption is wrong often enough to matter.

What this failure looks like in real systems

Here is the common sequence I see during incidents like this. Error rate jumps from sub-1 percent to double digits. Clients retry aggressively. Retry traffic amplifies load on the already degraded system. Latency increases, triggering timeouts. Queues back up. Background jobs fall behind. Engineers start manually throttling.

The most dangerous part is that nothing is technically "down." Health checks pass. Some requests succeed. This keeps retry storms alive.

If you only built retry logic, you built a traffic amplifier.

The naive retry loop that makes things worse

This is the pattern I still see in production code:

from anthropic import Anthropic
import time

client = Anthropic()

def call_claude(prompt):
    for attempt in range(5):
        try:
            return client.messages.create(
                model="claude-3-sonnet",
                max_tokens=512,
                messages=[{"role": "user", "content": prompt}]
            )
        except Exception:
            time.sleep(2 ** attempt)
    raise RuntimeError("Claude failed after retries")

This looks reasonable. It is not.

During a platform incident, all retries hit the same failing surface, backoff still creates synchronized retry waves, and you contribute to global load at the worst possible moment. This pattern is survivable at low volume. At scale, it cascades.

The real fix starts with circuit breakers, not retries

The first control you need is not more retries. It is a circuit breaker that can say "stop calling this API."

A circuit breaker turns repeated failure into a fast local decision. Here is a minimal breaker using Redis as shared state:

import redis
import time

r = redis.Redis()

def claude_available():
    return r.get("claude_circuit") != b"open"

def trip_claude():
    r.setex("claude_circuit", 60, "open")

def call_claude_with_breaker(fn):
    if not claude_available():
        raise RuntimeError("Claude circuit open")
    try:
        return fn()
    except Exception:
        trip_claude()
        raise

The breaker is shared across workers. The open state expires automatically. Fail fast beats slow failure every time. This alone will save your system from self-inflicted damage.

Why retries still matter, but only inside boundaries

Retries are not evil. Unbounded retries are.

The pattern that works is a small retry count, a tight timeout, wrapped inside a breaker:

def safe_claude_call(prompt):
    def attempt():
        return client.messages.create(
            model="claude-3-sonnet",
            max_tokens=512,
            timeout=5,
            messages=[{"role": "user", "content": prompt}]
        )
    for _ in range(2):
        try:
            return call_claude_with_breaker(attempt)
        except Exception:
            time.sleep(0.5)
    raise RuntimeError("Claude unavailable")

Notice what is missing: no exponential backoff to 30 seconds, no retry count in the teens, no blind faith that persistence equals success. At incident scale, shorter and fewer retries win.

Model fallback is not resilience, provider fallback is

If multiple models fail together, the only real isolation is at the provider boundary. That means your fallback graph needs to cross vendors, not model families.

def generate(prompt):
    try:
        return safe_claude_call(prompt)
    except RuntimeError:
        return call_openai(prompt)

This is not about preference. It is about failure domains. If your business cannot tolerate degraded LLM output, you must pay the complexity tax of multi-provider support. There is no shortcut here.

Bulkheads stop one failing feature from drowning everything

Another failure pattern during incidents is resource starvation. Your summarization jobs eat all workers. User-facing requests time out. Everyone is unhappy.

Bulkheads prevent this by isolating workloads:

queues:
  llm_user_requests:
    concurrency: 50
  llm_background_jobs:
    concurrency: 10

During elevated error rates, user traffic stays responsive and background jobs slow down, not everything. If you only have one queue, you do not have isolation.

Idempotency saves you from retry side effects

Retries are dangerous when requests have side effects. If your LLM call triggers writes, notifications, or billing events, retries can duplicate work.

Use idempotency keys consistently:

idempotency_key = f"summary:{document_id}"

client.messages.create(
    model="claude-3-sonnet",
    idempotency_key=idempotency_key,
    messages=[{"role": "user", "content": prompt}]
)

When incidents happen, duplicate requests are common. Idempotency turns chaos into correctness.

What monitoring actually matters during elevated error rates

Most teams monitor request count and average latency. That is not enough. During incidents, you need error rate by provider, circuit breaker state, retry volume, and queue depth growth.

A simple metric that pays for itself is retry amplification:

select
  sum(retry_attempts) / sum(original_requests) as retry_factor
from llm_request_metrics
where timestamp > now() - interval '5 minutes';

If that number goes above 1.5, you are probably making things worse.

Why status pages do not help you in the moment

Status pages tell you two things: something is wrong, and someone else is fixing it. They do not tell you how long retries will fail, which endpoints are safe, or whether partial recovery is real.

Your system must assume uncertainty. That is why local control mechanisms matter more than external signals. If your only reaction is waiting for green, you are already behind.

Graceful degradation beats hard failure

When LLMs are unavailable, many products default to errors. This is often unnecessary. You can return cached responses, use smaller local models for basic tasks, skip non-critical enrichment, or defer generation to async jobs.

A simple cache fallback:

cached = cache.get(prompt_hash)
if cached:
    return cached

response = generate(prompt)
cache.set(prompt_hash, response, ttl=3600)
return response

During incidents, cache hit rates go up. That buys you time.

Where this advice breaks

There are cases where none of this helps: if your workload is real-time and uncachable, if output correctness is legally critical, or if you are contractually bound to one provider.

In those cases, your only option is admission control. Say no early. Protect the rest of the system. Rejecting requests cleanly is better than timing out everything.

Admission control and load shedding when breakers are not enough

Circuit breakers stop calls after failure. Admission control stops calls before failure. During platform incidents, the worst thing you can do is accept unlimited work you cannot complete. You need a hard cap.

A simple token bucket per provider works well:

import time
from collections import deque

class RateGate:
    def __init__(self, max_inflight):
        self.max_inflight = max_inflight
        self.inflight = 0

    def acquire(self):
        if self.inflight >= self.max_inflight:
            return False
        self.inflight += 1
        return True

    def release(self):
        self.inflight -= 1

claude_gate = RateGate(max_inflight=20)

Usage:

if not claude_gate.acquire():
    raise RuntimeError("Claude overloaded locally")
try:
    return safe_claude_call(prompt)
finally:
    claude_gate.release()

During partial outages, latency often doubles before error rate spikes. Inflight requests pile up silently. Your worker pool gets saturated even if the provider eventually errors.

Concrete numbers I have seen: normal p95 latency 1.2s, incident p95 latency 8 to 12s, worker exhaustion within 90 seconds. Load shedding keeps tail latency bounded and preserves capacity for critical paths. It is blunt, but it works.

Hedged requests sound smart and usually backfire with LLM APIs

Some teams try hedged requests: fire the same prompt at two models or providers and take the first response. This can help with single-request latency variance. During incidents, it is gasoline.

Example of what not to do:

def hedged_generate(prompt):
    return first_completed([
        lambda: call_claude(prompt),
        lambda: call_claude(prompt)
    ])

You double traffic exactly when capacity is constrained, trigger rate limits faster, and amplify retry storms upstream. Even cross-provider hedging is risky if you do not gate it:

def cautious_hedge(prompt):
    if system_healthy():
        return call_claude(prompt)
    return first_completed([
        lambda: call_claude(prompt),
        lambda: call_openai(prompt)
    ])

Rules that keep this safe: hedge only after a latency threshold, never hedge inside retries, and disable hedging automatically when error rate rises. Hedging is an optimization. Incidents are not the time to optimize.

How to actually test this before production does it for you

Most LLM outage handling code is untested. That is why it fails under stress. You can simulate platform incidents locally with three levers: forced error responses, artificial latency, and partial success rates.

import random
import time

def flaky_claude(prompt):
    r = random.random()
    if r < 0.3:
        raise RuntimeError("500")
    if r < 0.6:
        time.sleep(10)
    return "ok"

Then run load tests and watch circuit breaker open rate, queue depth over time, admission rejections, and the retry amplification metric. You want to see breakers opening quickly, inflight count staying flat, and non-critical queues slowing first. If your test shows recovery only after traffic stops, your controls are too weak.

This is not chaos for chaos' sake. It is validating that your failure handling actually reduces load instead of redistributing it.

Final Thoughts

Elevated error rates across multiple models are not edge cases anymore. They are normal failure modes for platform-scale AI services.

If your resilience strategy is model switching and hope, you will relive the same outage every time.

The fix is not clever prompts or better SDK defaults. It is the same discipline that keeps payment systems, search backends, and message brokers alive under stress.

Treat LLMs like the critical infrastructure they are. Your future incidents will still hurt, but they will not take everything down with them.

Resources & References

Stay in Touch

Short takes and discussions on X → https://x.com/sebuzdugan

Practical AI / ML videos on YouTube → https://www.youtube.com/@sebuzdugan/

Partnerships & collabs → sebuzdugan@gmail.com

Originally published on Medium.

Governing and Monitoring Enterprise AI Usage: Allow and Deny Controls for AI Apps on Managed Devices

Sebastian Buzdugan — Fri, 19 Jun 2026 12:57:05 +0000

Your gateway logs look clean. Every API call is accounted for, rate-limited, and piped to your SIEM. You have virtual keys per team, budgets per model, guardrails on PII. The compliance box is checked.

Then someone asks: which AI apps are actually running on our laptops right now?

Nobody knows.

The gateway only sees traffic that goes through it. Claude Desktop, ChatGPT, Cursor, Codex, and a browser full of AI sidebars talk directly to their own endpoints, using API keys that live in the user's home directory. None of that traffic has ever touched your proxy.

This isn't a configuration problem. It's a structural one. And it has a clean fix that most enterprise teams haven't wired up yet.

The gap nobody talks about

A gateway governs the traffic that flows through it. That is exactly what it is for, and Bifrost's gateway does it well: virtual keys, budgets, routing, guardrails, and audit logs, enforced on every request your infrastructure sends.

But your developers are not only calling AI through your infrastructure. They are running the Claude app on their laptop. They are using Cursor with a personal API key. They installed an MCP server that a GitHub README told them would make their coding agent smarter. That traffic leaves the machine straight for the provider. It never had a reason to pass through your gateway, so it doesn't.

This is not a gateway shortcoming. It is a question of where the traffic starts. A centralized proxy can only govern what reaches it, and a desktop app talking directly to api.anthropic.com was never going to reach it on its own. The endpoint is simply a second surface, and until something sits on the machine, that surface is dark.

Here is what lives there:

Desktop chat apps: Claude Desktop, ChatGPT, calling home directly
Browser AI: ChatGPT and Claude on the web, running over HTTPS
Coding agents: Claude Code, Cursor, Codex, OpenCode, configured per-user, per-project
MCP servers: local processes that give those agents filesystem access, shell execution, database connections, and external API calls, spun up from a few lines of JSON

That last one deserves a pause. An MCP server is not an app in the traditional sense. It's a process your developer launched that tells their AI agent: here are the tools you can use. File reads, shell commands, web fetches. All of it governed by whatever the developer put in a config file, with no visibility for your security team and no logs anywhere.

A coding agent with shell access, connected to an ungoverned MCP server, can exfiltrate credentials before your SIEM sees a single packet.

That is the surface Edge is built to cover, and it covers it by extending the gateway you already run, not by replacing it.

What your gateway sees vs. what's actually running on managed devices. The gap is everything on the right.

Two layers, one control plane

Bifrost, the open-source AI gateway built by Maxim and available on GitHub, splits the governance problem into the two layers it actually has.

The gateway is the centralized control plane. It sits between your infrastructure and 1000+ models behind one unified API, and enforces policy on every request: virtual keys, team budgets, rate limits, model routing, PII detection, secrets scanning, content guardrails, and audit logs that feed SOC 2 Type 2, GDPR, and HIPAA stacks. It is built to stay out of the way. In Bifrost's own t3.xlarge benchmark at 5,000 requests per second, the gateway added 11 microseconds of overhead per request at a 100 percent success rate.

The OSS version is self-hosted and drops in as a base URL change. You point your existing SDK at it and leave the rest of your code alone:

# before
client = anthropic.Anthropic(api_key="your-anthropic-key")

# after: same code, traffic now governed
client = anthropic.Anthropic(
    base_url="http://localhost:8080/anthropic",
    api_key="dummy-key",  # real keys live in Bifrost
)

No agent on the machine, no per-app config. This is the layer most teams already have, or can stand up in an afternoon.

Bifrost Edge is the endpoint enforcement layer. It is an Enterprise feature, not part of the open-source gateway. Edge runs locally on every managed machine and intercepts AI traffic at the OS level, before it reaches any application's networking stack. That traffic gets routed through your Bifrost gateway for policy evaluation, and the gateway's verdict comes back to Edge, which enforces it on the device.

The frame that makes this click: the gateway centralizes governance, Edge enforces it on every machine.

Every desktop app, every browser extension, every MCP server suddenly inherits the same virtual keys, budgets, guardrails, and audit logging that your API infrastructure already has. Zero per-app configuration. You install Edge once and governance follows the user.

Bifrost Edge on every managed machine, all routing to the same central gateway. Policy changes propagate to the fleet without touching individual devices.

How Edge sits on the machine

Installation is a single MDM push. Edge ships managed config for Jamf, Intune, Kandji, Omnissa Workspace ONE, and JumpCloud, so it rolls out across macOS, Windows, and Linux with no end-user action. The managed config carries only non-sensitive connection settings. The machine arrives pre-pointed at your Bifrost; identity and keys come from the user's sign-in, not from the device profile.

After install, the user signs in once through a browser SSO flow. That sign-in links the machine to the user and pulls down every policy assigned to them. From then on, Edge runs as a menu-bar agent on macOS or a system-tray process on Windows and Linux.

Interception happens at the machine level, not per application. Edge does not ask you to change base URLs or touch anything inside the AI apps. It uses an organization certificate, managed centrally from the Bifrost console, to route each app's AI calls through your gateway before they leave the machine. When Claude Desktop calls api.anthropic.com, Edge sees the request, sends it through Bifrost for policy evaluation, and hands the response back. The app never knows the difference.

That is what makes this work at scale. You have a hundred AI tools across the fleet; you configure one thing, and all hundred are governed.

Allow, deny, and the state in between

Edge discovers AI apps and MCP servers across the fleet and rolls them into a deduplicated catalog: the same tool on 200 machines shows up once. Every entry sits in one of three states.

Pending: discovered and awaiting review. It keeps working in the meantime, so turning Edge on does not break anyone's workflow on day one.
Approved: explicitly allowed. Traffic runs normally and stays under the gateway's full policy stack.
Denied: blocked. Edge stops it on the device, and the user sees a notice that the app is not permitted on company machines.

That default-pending posture is the part worth sitting with. You are not forced to choose between locking everything down up front or flying blind. New tools surface as Pending while you decide, so you get visibility immediately and enforcement when you are ready.

The workflow in practice: a developer installs a new coding agent. It appears as Pending in the console with its name, the devices running it, who last changed its status, and any notes. You approve it, deny it, or select every Pending item and act in bulk. Decisions take effect on each device at its next check-in, which is set by a sync interval you control centrally, down to a few seconds when you need policy to move fast.

Supported surfaces today: Claude Desktop, ChatGPT, Cursor, and Codex on the desktop; Claude Code, Codex CLI, and OpenCode as coding agents; ChatGPT and Claude on the web. If something you run is not on the list yet, the console has a "request support for an application" path.

MCP governance is its own layer

MCP servers get their own governance surface in Edge, separate from application-level allow/deny. That separation matters.

You might want Cursor allowed but a specific MCP server denied. Maybe it has filesystem write access your security team hasn't signed off on. Maybe it was pulled from an untrusted source. Edge lets you approve or deny at the server level, not just the app level.

The workflow:

Edge discovers MCP servers configured in supported AI apps across the fleet
The full inventory surfaces in the Bifrost admin console
Administrators approve or deny each server individually
Edge enforces those decisions at the device level

Default posture applies here too. Set Edge to deny all unknown MCP servers by default and new servers require explicit approval before anyone in the fleet can use them.

The fleet-wide MCP inventory alone is something most security teams have never had. Ask a room of security leads which MCP servers run on their developer machines and the honest answer is nobody knows. Edge surfaces each one with its name, whether it connects through a local command or a remote URL, and the exact tools it exposes, then gives you per-server approve and deny. MCP discovery currently covers Claude Code, Claude Desktop, Cursor, Gemini CLI, OpenCode, and Codex.

MCP governance in three steps: Edge discovers servers fleet-wide, admin sets per-server policy, Edge enforces on every device.

What you can see

With Edge deployed, the console becomes a live inventory of AI on the fleet.

The Devices view lists every machine running the agent: hostname, owner (name and email), platform with OS version and architecture, agent version, how many AI apps and MCP servers it has (versions on hover), when it was first seen, and its last check-in. You can filter by platform, by a specific app or MCP server, or by approval status, so "which machines are still running the app we denied" is a filter, not an investigation. Open a single device and you get its installed apps, its configured MCP servers and the tools they expose, and one-click approve, deny, or remove.

Underneath the inventory, the guardrails you already configured in the gateway now apply to all of this traffic, in both directions, before a prompt reaches a model and before a response comes back. That includes Gitleaks-backed secrets detection for leaked API keys, tokens, and private keys, PII redaction, and any third-party content-safety integration you run (AWS Bedrock, Azure, Google Model Armor, CrowdStrike, Patronus, and others). Edge does not re-implement any of it. It brings ChatGPT, Claude, and your coding agents under the same protection your API traffic already has.

Every routed request also inherits the gateway's structured logging: timestamp, user, model, token count, guardrail outcome. Pair that with the gateway's Prometheus, OpenTelemetry, and Datadog exports and you have one view over all AI usage, infrastructure and endpoint alike.

The containment argument

Here is the scenario that makes the security case concrete.

Your team discovers a new AI desktop app is exfiltrating conversation history to an undisclosed third party. You need it off every managed laptop, now.

Without Edge: you're filing an MDM change request, writing browser extension policies, opening a firewall ticket, coordinating across three teams, and probably missing the macOS desktop app entirely. Timeline: days to weeks.

With Edge: you open the console, set the app to Denied, and apply it to the fleet. Every machine picks it up at its next check-in, governed by the sync interval you set. The block lands before any data leaves the device. Audited, reversible, fleet-wide in seconds to minutes.

Same logic for MCP servers. A new server with dangerous tool permissions shows up in a GitHub repo. You deny it fleet-wide before any developer on your team installs it.

The governance speed matters as much as the governance capability. The threat landscape for AI tooling is moving fast. Waiting weeks to roll out a block is not a viable security posture.

The honest part

Edge is in limited alpha. You cannot self-serve into production today. You register, you get access when the alpha opens up. That is the reality of where it sits.

The capability set described here is what the alpha delivers. All of it is live and usable for teams in the program. But if you need this in production next week with an SLA behind it, that conversation starts with the Bifrost team, not a self-serve dashboard.

Also worth naming: the MCP app support list is current as of this writing. New apps and MCP servers get added, but if you have a specific tool you need covered, check the supported applications page before assuming it's there.

Getting started

The Bifrost gateway (OSS) is available now. It is a drop-in base URL replacement for any OpenAI-compatible SDK call. If you are not running a centralized gateway yet, that is the right starting point. Get visibility on your API traffic first, then add Edge for the endpoint layer.

For Edge, register for alpha access at docs.getbifrost.ai/edge/overview.

The path that makes sense:

Deploy the Bifrost gateway, route your team's API traffic through it
Configure virtual keys, budgets, and audit logging
Roll out Bifrost Edge via MDM to bring endpoint AI traffic under the same governance

Steps 1 and 2 give you coverage on everything your infrastructure controls. Step 3 closes the gap.

Every technical detail here is drawn from the Bifrost Edge documentation and the gateway overview. If something in this piece doesn't match what you see in the docs, trust the docs.

Originally published on Medium.

More writing on Medium → https://medium.com/@sebuzdugan
Short takes on X → https://x.com/sebuzdugan
Practical AI / ML videos on YouTube → https://www.youtube.com/@sebuzdugan/