DEV Community: claire nguyen

What is shadow AI and how to govern it

claire nguyen — Mon, 22 Jun 2026 06:14:58 +0000

Shadow AI is the use of AI tools inside an organization without IT's knowledge or approval. This guide explains what it is, why it creates real security and compliance risk, and how the Bifrost AI gateway together with Bifrost Edge brings that usage under governance on every machine.

Most of the AI tools employees rely on at work run on their own machines and reach a model provider directly, without passing through any corporate network checkpoint. A developer can install a desktop assistant, paste in proprietary source code, and send it to a third-party model before anyone in security knows the tool exists. Industry analysts call this pattern shadow AI: the use of AI tools or applications by employees without the approval or oversight of the IT department. The scale is no longer marginal, as a 2026 Gartner survey of cybersecurity leaders found that 69 percent have evidence or suspect that employees are using public generative AI at work.

What shadow AI is

Shadow AI is the use of AI tools, models, and services by employees without the knowledge, approval, or governance of an organization's IT or security teams. It is a subset of shadow IT, the broader category of hardware and software that IT has not approved, but it carries risks that older shadow IT controls were never designed to handle. The distinction matters for how an organization should respond.

Where shadow IT generally involves an unapproved application or storage service, shadow AI centers on systems that process, generate, and retain data in ways that are difficult to reverse. Common forms include:

Public chatbots and assistants used on work data, such as the ChatGPT app or Claude Desktop signed in with a personal account.
AI features inside browser tabs and SaaS products that an employee turns on without review.
Coding agents in the terminal and IDE, including Claude Code, Codex, and Cursor, that read source code and call external services.
MCP servers wired into those tools, which can read files, call APIs, and act on a user's behalf.

Salesforce's 2026 Workforce AI Survey found that 67 percent of employees use AI at work, while only 18 percent of organizations have a formal AI security policy. Adoption at that pace, with no governing layer underneath it, is what turns ordinary productivity into a security exposure.

Why shadow AI is a security risk, not just a policy gap

Shadow AI raises measurable security and compliance risk because sensitive data reaches systems that the organization cannot see, control, or audit. Gartner predicts that by 2030, more than 40 percent of organizations will experience security or compliance incidents tied to the use of unauthorized AI, and the reasons follow directly from how these tools are used.

The exposure goes well beyond a single pasted prompt. Several distinct failure modes make shadow AI harder to contain than earlier forms of unapproved software:

Data leaves the organization in a form that cannot be recalled. Once proprietary text enters a third-party model, it may be retained or used in ways the organization has no ability to reverse, unlike a file that can be deleted from a server.
Compliance and audit trails break down. Legal and security teams cannot demonstrate where regulated data went, whether retention rules were followed, or whether residency obligations were met when the traffic never passed through a governed path.
AI agents inherit standing access. An assistant connected through an MCP server can read email, repositories, and internal systems on a continuing basis, so the question shifts from what a user pasted once to what a connected tool can reach at any time.
Governance trails adoption. Most organizations still have no reliable way to see which AI tools and connections are in use, which leaves the bulk of this activity outside any policy or review.

These concerns are not hypothetical; they describe what happens when fast, employee-driven adoption runs ahead of any mechanism for seeing or controlling it.

Why existing controls miss shadow AI

Traditional network controls miss most shadow AI because the activity does not behave like the traffic those controls were built to inspect. Network proxies and data loss prevention systems observe what crosses the corporate network, yet a large share of AI usage runs on the endpoint and connects straight to a provider over an encrypted channel that resembles ordinary web traffic.

Three gaps recur across the older approaches:

Network filtering and data loss prevention operate at the perimeter, so AI requests that originate and resolve on the device fall outside their view.
Blocklists depend on a known list of destinations, and new AI tools, browser features, and MCP servers appear faster than any list can track.
Written policies state what employees should do, but a document does not enforce anything at the moment a prompt is sent.

The common thread points toward the fix: the AI runs on the endpoint, where the person and the tool actually meet, so the endpoint is the one place where every request can be seen and governed before data leaves the machine.

How the Bifrost AI gateway with Bifrost Edge govern shadow AI

Governing shadow AI well takes two things that fit together: one place to define policy, and a way to apply that policy to the AI running on every machine. Bifrost, the open-source AI gateway built by Maxim AI, is that one place. The gateway already holds the virtual keys, budgets, and rate limits that tie AI usage to a person or project, the guardrail profiles that inspect prompts and responses, and the audit logs that record every exchange. The limitation, until now, has been reach: those controls governed only the traffic that someone had configured to point at the gateway.

Bifrost Edge closes that gap by running on each machine and routing all supported AI traffic through Bifrost, so the same virtual keys, budgets, guardrails, and audit logs that already protect gateway traffic now apply to the desktop apps, browser AI, and coding agents people use day to day. The gateway stays the single control plane, and Edge becomes its reach to the endpoint, so there is no second policy model to build or maintain.

A request from any supported AI tool follows the same governed path on every machine:

A user works in a desktop app, a browser AI surface, or a coding agent exactly as before, with no base URL change and no SDK swap.
Bifrost Edge routes that request through the organization's Bifrost rather than letting it go straight to the provider.
Bifrost ties the request to the user's virtual key and its budget, runs the configured guardrails, and writes the exchange to the audit log.
The governed response returns to the original app, with sensitive content already caught or redacted.

Guardrails apply before data leaves the machine

The guardrail profiles configured in Bifrost apply to endpoint traffic with no extra setup on the device. A guardrail runs before a prompt reaches a model and again before a response returns, so secrets and personal data are caught or redacted before they leave the machine. Built-in coverage includes Gitleaks-backed secrets detection for leaked API keys, tokens, and credentials, a PII detection template built on custom regex, and content safety, alongside integrations with AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, CrowdStrike AIDR, GraySwan Cygnal, and Patronus AI.

Visibility into MCP servers across the fleet

Most organizations cannot say which MCP servers their employees have connected to AI tools. Bifrost Edge inventories the MCP servers configured inside each supported app and builds a live picture across the fleet of which servers are in use, on which apps, and on how many devices. Administrators then allow or deny each server individually, and Edge enforces that decision on the device, even for an app that had the server configured before the policy existed. MCP discovery covers the major AI apps that support it, including Claude Code, Claude Desktop, Gemini CLI, OpenCode, Codex, and Cursor.

App policy enforced on every device

App governance lets administrators decide which AI applications are permitted across the organization. Approved apps run normally, with their traffic governed through Bifrost, while disallowed apps are blocked before any data leaves the machine. When Edge encounters an app or MCP server it has not seen, it requests approval from the admin console, and administrators choose whether pending items are allowed or blocked while a decision is pending. Policy changes reach the whole organization at once, without anyone revisiting individual machines.

Rollout through your existing device management

Bifrost Edge deploys through the device management platforms an organization already runs, including Jamf, Microsoft Intune, Kandji, Omnissa Workspace ONE, and JumpCloud, across macOS, Windows, and Linux. The managed configuration carries only the connection settings that point each machine at the organization's Bifrost, and identity and keys come from the user's single sign-on, so no secrets sit on the device. After the first sign-in, governance stays in sync with the gateway, and central changes to app policy, MCP allow and deny lists, and routing reach the fleet on their own.

Common questions about shadow AI

Is shadow AI the same as shadow IT?

Shadow AI is a subset of shadow IT, but the risk profile differs. Shadow IT covers any hardware or software that IT has not approved, whereas shadow AI specifically involves tools that process and retain data in a model, which makes the exposure harder to reverse and more likely to spread across teams.

Can shadow AI be detected?

Shadow AI can be detected when AI requests are observed at the point where they originate. Because much of the usage runs on the endpoint, a layer that operates on the device, such as Bifrost Edge, can inventory the apps and MCP servers in use and route their traffic through a gateway where it becomes visible and auditable.

How do you govern AI coding agents?

Coding agents such as Claude Code, Codex, and Cursor run locally and often connect directly to model providers and MCP servers. Routing their traffic through Bifrost applies the same guardrails, budgets, and audit logging used for the rest of an organization's AI, while app and MCP policies determine which agents and tools are allowed on each machine.

Where this leaves enterprise teams

Shadow AI persists because the activity happens on the endpoint and moves faster than perimeter controls can follow, so better intentions and longer policy documents do not resolve it on their own. The organizations that handle it well treat it first as a visibility problem and then as an enforcement problem, governing AI where people actually use it rather than where the network happens to see it.

Pairing the Bifrost AI gateway with Bifrost Edge gives security and platform teams one control plane for that work, with the gateway defining the virtual keys, budgets, guardrails, and audit logs, and Edge, currently in alpha, extending them to every machine in the organization. Teams sizing up shadow AI can review how the combined approach works on the Bifrost Edge overview and register there for alpha access.

Fault-injecting our LLM provider to trust Bifrost fallbacks

claire nguyen — Fri, 19 Jun 2026 13:26:05 +0000

TL;DR: We run an LLM-backed build-failure summariser at Buildkite. To stop a provider wobble from breaking it mid-deploy, I ran a game day that fault-injected OpenAI with 429s and 500s and watched whether Bifrost's fallback config actually rerouted. It did, but only after I fixed two things I'd set up wrong.

We've got a small service that reads failed CI jobs and writes a one-paragraph summary into the build annotation, so engineers don't have to scroll 4,000 lines of test log to find the one assertion that broke. It calls an LLM. Handy when it works. Embarrassing when it doesn't, because a broken annotation makes people distrust every annotation.

The problem is the thing it depends on isn't ours. OpenAI rate-limits, has the occasional 5xx spell, and we don't get a heads-up. "Never had an outage" usually means you never tested the failure path. So I tested it.

Why a gateway at all

I didn't want fallback logic smeared across our service code. Retry-with-jitter, secondary provider, key rotation, all of that wants to live in one place with metrics attached. We put Bifrost in front, an OpenAI-compatible gateway, so our service keeps talking the same /v1/chat/completions it always did and the routing decisions move to config.

The pitch is plain. One endpoint, 23+ providers behind it, automatic fallbacks between them. Our code points at localhost:8080 instead of api.openai.com and stops caring which model actually answers.

Here's the fallback config I started the game day with:

{
  "providers": {
    "openai": { "keys": ["env.OPENAI_KEY_A", "env.OPENAI_KEY_B"] },
    "anthropic": { "keys": ["env.ANTHROPIC_KEY"] }
  },
  "fallbacks": [
    "openai/gpt-4o-mini",
    "anthropic/claude-haiku-4-5"
  ]
}

Two OpenAI keys for load balancing, then Anthropic as the lifeboat if OpenAI as a whole goes sideways. That was the theory.

The game day

A game day is just a planned outage you cause on purpose, with people watching. I scheduled 45 minutes, told the team, and put a toxiproxy in front of OpenAI so I could inject faults without waiting for the real thing to break.

Three scenarios:

429 storm. Every OpenAI response becomes a rate-limit for 5 minutes.
Hard 500s. OpenAI returns 503 on half of requests.
Latency tar pit. 30-second delays, no errors, the nastiest one.

Scenario one went fine. Bifrost saw the 429s, rotated between key A and key B, then gave up on OpenAI and the requests landed on Haiku. Annotations kept writing. Reckoned I was done.

Scenario two found my first mistake. I'd not set a sane retry ceiling, so on a 503 the gateway retried hard against the same struggling provider before failing over, and our p95 on annotation writes jumped to about 18 seconds. Fixed it by capping retries and letting the fallback fire sooner. The README's retries and fallbacks page covers the knobs; I'd skimmed it the first time.

Scenario three is the one everyone gets wrong. Slow isn't down. A 30-second response isn't an error, so naive fallback never triggers, the request just sits there. We added a request timeout so a tar-pitted provider counts as a failure and trips the lifeboat. That single change is the actual reason this exercise was worth running.

What the metrics showed

Bifrost ships native Prometheus metrics, so I didn't have to bolt on my own. I watched fallback rate and per-provider latency the whole time on a Grafana board.

Scenario	Without fallback	With Bifrost (tuned)
429 storm	annotations stall	reroute to Haiku, ~2.1s p95
Hard 503s	50% writes fail	0 user-visible failures
30s latency	every write hangs	timeout trips fallback in 4s

The numbers that mattered: zero broken annotations across all three once tuned, and the fallback decisions were visible in metrics instead of buried in logs nobody reads.

How it stacks up against LiteLLM and Portkey

I'd used LiteLLM before. Worth being honest here.

	Bifrost	LiteLLM	Portkey
OpenAI-compatible endpoint	yes	yes	yes
Automatic fallbacks	yes	yes	yes
Native Prometheus metrics	yes	yes	yes
Self-host story	single Go binary	Python proxy	gateway is OSS, control plane hosted
Maturity / ecosystem	newer	large, lots of integrations	polished dashboards

LiteLLM has been around longer and has a bigger pile of community integrations, which counts for something when you hit an edge case at 2am. Portkey's hosted dashboards are nicer than anything I'd build myself, and if you don't want to run infra that's a fair trade. We picked Bifrost mostly because a single Go binary is easy for an infra team to operate and the Prometheus output dropped straight into our existing board with no glue. Not a knock on the others. Different priorities.

Trade-offs and limitations

A gateway is one more hop you have to keep alive. If Bifrost falls over, every LLM call falls with it, so we run two replicas behind a load balancer and the game day included killing one of them too.

Fallback to a different model means a different model. Haiku doesn't write the exact same summary as gpt-4o-mini, and for a build annotation that's fine, but if you depend on a strict output schema you need to test the lifeboat actually produces it. We caught one prompt that assumed OpenAI-specific formatting.

And fault injection in front of a proxy isn't the real provider misbehaving. Toxiproxy gives you 429s and delays, not the weird partial-stream failures you see in the wild. It's a model of the failure, not the failure. Better than nothing, not the whole story.

Semantic caching is on the roadmap for us, not load-bearing yet, so I'm not going to claim numbers I haven't measured.

An 8-minute outage from a dead NLB and a JVM that cached DNS forever

claire nguyen — Fri, 19 Jun 2026 06:43:43 +0000

TL;DR: We drained a Network Load Balancer during a planned migration, and one internal service kept hammering the dead IPs for 8 minutes. The cause wasn't the failover. It was a JVM caching the DNS record forever. The fix was a 30-second TTL and a health-check tweak, not a smarter system.

Last month I ran a routine migration on one of our internal control-plane services at Buildkite. Move it behind a new NLB, drain the old one, done before lunch. We've got a small platform team, four of us, and this was meant to be a non-event.

It was not a non-event.

The bit where everything looked fine

I shifted the Route 53 record to the new NLB, watched the new targets go healthy, and started draining the old load balancer. Traffic on the new path climbed. Traffic on the old path... did not drop. One service, a Java scheduler that fans out build metadata, kept slamming the old NLB's IP addresses like nothing had changed.

The old targets were already deregistering. So we had requests landing on backends mid-shutdown, timing out at 10 seconds each, retrying, timing out again. Error rate on that service went from basically zero to 60%. For 8 minutes.

The annoying part? Every other service flipped over within about 60 seconds. Our Go services, our Node workers, all fine. Only the JVM one was stuck.

Why one runtime behaved differently

Here's the thing nobody on the team had clocked. The default DNS caching behaviour is wildly different depending on what's making the request.

| Runtime | Default DNS cache TTL | What actually happened |
|---|---|
| JVM (security manager off) | 30s | Usually fine |
| JVM (security manager on) | Forever (networkaddress.cache.ttl=-1) | Cached the dead NLB IPs until restart |
| Go net resolver | Honours record TTL | Re-resolved in ~60s |
| Node 18 | Honours record TTL | Re-resolved in ~60s |
| Python requests | No caching at the lib layer | Re-resolved per connection pool refresh |

NLBs hand you IP addresses behind a DNS name, and those IPs change when targets move. The contract is: you re-resolve and follow the record. The JVM, with a security manager active, sets networkaddress.cache.ttl to -1, which means cache the first answer for the life of the process. So our scheduler resolved the name once at boot, three weeks ago, and never looked again.

The DNS record had a 60-second TTL. Didn't matter. The JVM never asked DNS a second time.

The fix

Two lines in the JVM security config, pushed through our base image:

# $JAVA_HOME/conf/security/java.security
networkaddress.cache.ttl=30
networkaddress.cache.negative.ttl=5

Thirty seconds of caching, five seconds for negative lookups so a transient NXDOMAIN doesn't get pinned either. We bake this into the base image now so every JVM service inherits it. No app code change.

Then the second half of the problem: even with re-resolution, the old NLB was draining too slowly. Default deregistration delay is 300 seconds. During a planned cutover that's an eternity of half-dead targets accepting connections. We dropped it for this service:

aws elbv2 modify-target-group-attributes \
  --target-group-arn "$TG_ARN" \
  --attributes Key=deregistration_delay.timeout_seconds,Value=30

Short TTL plus short drain meant the next test cutover finished in under 90 seconds across every runtime. No error spike.

Where the LLM bit sneaks in

One reason this stung is that the same scheduler kicks off build steps that call an LLM for flaky-test classification. Those calls go out through an AI gateway, Bifrost in our case, which already does its own provider failover and re-resolution on the upstream side. So that path stayed healthy the whole time. Which made the outage extra confusing at first, because the part everyone assumed was fragile (the model calls) was solid, and the boring internal HTTP call was the thing on fire. Good reminder that the exotic dependency isn't always the one that bites you.

Trade-offs and limitations

A short DNS TTL isn't free, and I won't pretend it is.

More DNS queries. Dropping JVM cache to 30s means roughly 120x more lookups per hour per host than the old "resolve once" behaviour. For us that's noise against a Route 53 resolver, but if you're doing thousands of resolutions a second you'll want to check your resolver isn't the new bottleneck.

30s is still 30s. This shortens the failover window, it doesn't remove it. For a true hot failover you need connection-level health checks and active draining, not just DNS. We treat DNS as the coarse knob and deregistration delay as the fine one.

Negative caching tradeoff. A 5-second negative TTL means a brief DNS hiccup gets retried fast, but it also means a genuinely down name gets queried more aggressively. Fine for internal services, worth a second look if you're resolving something rate-limited.

It's per-runtime. There's no single switch. The JVM fix does nothing for a Go binary, and the Go default did nothing wrong here. You have to know each runtime's behaviour, which is exactly the gap that caused this.

What I'd tell past me

Test the failover, not the failover system. We had a lovely runbook for swapping NLBs and zero tests for what the clients actually did when we swapped. "Never had a DNS issue" just meant we'd never drained a load balancer with a JVM watching.

Next game day, we're killing an NLB on purpose and watching every runtime re-resolve. If something caches forever, I'd rather find out at 2pm on a Tuesday than during a real migration.

A provider latency spike stalled our whole build queue

claire nguyen — Thu, 18 Jun 2026 13:27:47 +0000

TL;DR: A provider slowdown turned a 2-second LLM call into a 70-second hang. Because our build agents block on that call, the queue backed up to roughly 400 jobs in twelve minutes. We put Bifrost in front with hard timeouts and a fallback model, and the queue stopped caring whether any single provider was healthy.

The bit nobody designs for

I work on compute and build orchestration at Buildkite. One of our internal services calls an LLM to triage flaky test output, group it, and suggest a likely owner. Small thing. Saves engineers a fair bit of digging.

The catch is that a build agent waits on that call before it releases its slot. So the latency of one HTTP request to a model provider quietly became part of our queue's throughput math. Nobody wrote it down that way, but that's what it was.

On a Tuesday in May the provider's p99 went sideways. Not an outage. Just slow. Our default client timeout was 60 seconds, our retry was three attempts with backoff, and suddenly a call that normally took 2 seconds was eating 70 before giving up. Agents held their slots the whole time. Within twelve minutes the run queue sat at about 400 jobs that had nothing to do with the LLM at all.

Classic head-of-line blocking. One slow dependency, a whole fleet stuck behind it.

What we changed

We'd been calling the provider SDK directly from the service. The reliability logic lived in our own code, which meant our timeout values, our retry counts, and our fallback policy were all bespoke and slightly wrong.

We moved the call behind Bifrost, a self-hosted Go gateway that speaks an OpenAI-compatible API. The point wasn't to add a hop. It was to move the failure handling out of our app and into config we could reason about during an incident.

Three things mattered for us.

First, fallbacks. If the primary model is slow or erroring, route to a different model or provider instead of retrying the sad one into the ground. Second, semantic caching, because flaky test output repeats far more than you'd reckon, and a cache hit is a call that can't hang. Third, native Prometheus metrics, so the LLM path showed up on the same SLO dashboards as everything else we run.

Here's the gist of the config:

{
  "providers": {
    "anthropic": {
      "keys": [{ "value": "env.ANTHROPIC_KEY", "weight": 1.0 }],
      "network_config": { "default_request_timeout_in_seconds": 8 }
    },
    "openai": {
      "keys": [{ "value": "env.OPENAI_KEY", "weight": 1.0 }],
      "network_config": { "default_request_timeout_in_seconds": 8 }
    }
  },
  "fallbacks": [
    { "model": "anthropic/claude-haiku-4-5", "weight": 1.0 },
    { "model": "openai/gpt-4o-mini", "weight": 1.0 }
  ]
}

The 8-second timeout is the real fix. Our triage call has no business taking longer than that, and if it does, we'd rather get a degraded answer from the fallback than hold a build slot hostage. The gateway fails over instead of stacking retries against a provider that's already struggling.

Did it work

Next time the same provider got slow (it happened again in June, naturally), the gateway tripped to the fallback model after 8 seconds. Triage quality dropped a touch for those minutes. The queue never noticed. Peak backlog during that window was 11 jobs, not 400.

The caching surprised me more than the failover. On a normal day we're seeing roughly 30% cache hits on triage prompts, because the same flaky test produces near-identical output across re-runs. Thirty percent fewer calls is thirty percent fewer chances to hang.

How it stacks up

We looked at LiteLLM and Portkey before landing here. All three do the core gateway job. The differences are real, so here's the honest version.

Thing I cared about	Bifrost	LiteLLM	Portkey
Self-hosted, no SaaS dependency	Yes, single Go binary	Yes, Python proxy	Yes, but SaaS is the main path
Fallback + load balancing config	Built in	Built in	Built in
Semantic caching	Built in	Built in	Built in
Prometheus metrics native	Yes	Via add-ons	Via their dashboard
Provider breadth	23+	Widest of the three	Broad

LiteLLM has the widest provider list and the biggest community, so if you're calling something obscure it's a safe bet. If your stack is already Python, its proxy drops in with less friction. Portkey's managed offering and guardrail tooling are more polished out of the box, and for a team that doesn't want to run another service that matters.

We picked Bifrost because it's one static binary written in Go, the config above is the whole story, and I didn't want a Python runtime sitting in a latency-sensitive path on our build hosts. That's a preference, not a verdict.

Trade-offs and limitations

You're adding a network hop. For us it's a sidecar on the same host, so it's sub-millisecond, but it's not zero and you should measure it rather than trust me.

The fallback model gives worse triage answers. We decided a rough answer that arrives beats a good one that never does, but that's a call you make per use case, not a universal truth.

Semantic caching can serve a stale-ish answer when two failures look similar but aren't. We tuned the similarity threshold conservatively and accept the occasional miss.

And a gateway is now a dependency too. We run two replicas behind a local load balancer and treat it like any other tier-1 service. If you deploy a single instance, you've moved your single point of failure, not removed it.

The broader lesson has nothing to do with LLMs. Any blocking call to a dependency you don't control belongs behind a timeout you do control. We just hadn't noticed an LLM had become one.

Scaling Claude Code Across Enterprise Engineering Teams

claire nguyen — Wed, 17 Jun 2026 08:50:38 +0000

Scaling Claude Code deployments across engineering teams requires governance, cost control, and observability that individual API keys cannot provide. Bifrost is an open-source AI gateway that centralizes Claude Code access without changing how developers work.

Claude Code runs against api.anthropic.com by default, which means each developer who installs it carries an individual provider credential and every request bypasses any central point of control. That arrangement works for a pilot, but it breaks down once hundreds of engineers run the agent daily and the organization needs to answer questions about spend, access, and reliability. Scaling Claude Code deployments to that size calls for an infrastructure layer between the agent and the model providers. Bifrost, the open-source AI gateway built by Maxim AI, fills that role by routing all Claude Code traffic through a single governed endpoint.

Why scaling Claude Code across teams is hard

Scaling Claude Code from a few volunteers to an entire engineering organization introduces four operational problems: untracked cost, ungoverned access, single-provider lock-in, and scattered telemetry. Each one is manageable for one developer and severe across a fleet. The list below maps the failure modes that surface as adoption grows.

Cost management

Per-developer API keys make it difficult to attribute spend to a team, project, or individual.
There is no shared view of where token budget is actually going.
Setting per-team quotas or hard spending caps is not possible with raw provider keys.

Access control

Distributing and rotating provider credentials across many developers is operationally fragile.
Revoking a single compromised key without disrupting everyone else is awkward.
Tracking which teams are entitled to which models has no central enforcement point.

Model flexibility

Claude Code ships locked to Anthropic's endpoint, so teams cannot route specific tasks to alternative providers.
There is no failover path when a provider returns errors or hits a rate limit.
Lighter models cannot be used for simple tasks to reduce per-request cost.

Observability

Logs live on individual developer machines rather than in a shared system.
Token usage, latency, and error rates are not monitored centrally.
Debugging a bad session or auditing usage after the fact is impractical at scale.

These issues move from inconvenient to blocking the moment Claude Code crosses from an experiment into production engineering workflows.

What an enterprise AI gateway does for Claude Code

An AI gateway is a unified entry point that routes, authenticates, governs, and observes traffic to multiple LLM providers from a single API. Placed in front of Claude Code, Bifrost intercepts every request the agent makes, applies policy, forwards it to the chosen provider, and returns the response in Anthropic's expected format. The client binary stays unmodified, so developers keep working exactly as before.

Integration is a base URL change. Point Claude Code at the Anthropic-compatible endpoint Bifrost exposes, supply a virtual key, and launch the agent:

# Point Claude Code at Bifrost's Anthropic-compatible endpoint
export ANTHROPIC_BASE_URL="http://localhost:8080/anthropic"

# Use a Bifrost virtual key instead of a raw provider key
export ANTHROPIC_API_KEY="vk_your_key"

# Launch Claude Code as usual
claude

Because Bifrost functions as a drop-in replacement for Anthropic's API surface, Claude Code never knows a gateway is in the path. Behind that endpoint, Bifrost routes to any of 20+ supported providers and over 1,000 models, giving a single Claude Code session access to Anthropic, AWS Bedrock, Google Vertex, Azure OpenAI, and others without client-side changes. The Claude Code integration guide documents the full setup, and the Claude Code resource hub collects the governance patterns teams use in production.

Key capabilities for scaling Claude Code deployments

Routing Claude Code through Bifrost turns four scaling problems into configuration. The capabilities below correspond directly to the cost, access, flexibility, and observability gaps described earlier.

Multi-team access control with virtual keys

Virtual keys are the primary governance entity in Bifrost. Instead of handing real provider credentials to developers, administrators issue virtual keys that carry their own permissions, budgets, and rate limits. This abstraction supports a hierarchy that mirrors how organizations are structured:

Organization level: set an overall spending ceiling across all teams.
Team level: allocate independent budgets to individual engineering teams or projects.
Developer level: track and cap usage for a single engineer.

Keys can be created, rotated, or revoked instantly without touching the underlying provider credentials. The result is centralized credential management, real-time spend visibility per team, automatic enforcement of budgets and rate limits, and clean onboarding and offboarding. Full policy options are covered on the governance resource page.

Cost optimization through model routing

Not every Claude Code task needs a frontier model. With routing rules, teams can direct simple operations to lighter, cheaper models while reserving the most capable ones for hard problems. Claude Code already organizes its work into tiers, and a gateway lets those tiers map to the most cost-effective model for each job:

Task type	Suggested model tier	Rationale
Lightweight edits, formatting	Claude Haiku or a comparable small model	Lowest per-token cost for routine work
Default coding and refactoring	Claude Sonnet	Strong balance of capability and cost
Complex reasoning, deep review	Claude Opus or another frontier model	Reserve highest cost for highest-value tasks

Because Bifrost normalizes requests across providers, the same routing logic can send a task to AWS Bedrock, Google Vertex, or another backend when that provider offers a better price or availability for a given model. Developers experience one consistent interface while the gateway optimizes spend underneath it.

Production-grade reliability

A single-provider setup is only as reliable as that provider's uptime. Bifrost adds automatic failover so that when a primary provider returns errors or rate limits, requests move to a configured backup with no interruption to the Claude Code session.

Automatic fallbacks: define an ordered chain of providers and models so requests complete even during an outage.
Load balancing: distribute traffic across multiple API keys and providers using weighted key management.
Semantic caching: Bifrost's semantic caching returns stored responses for semantically similar requests, cutting both latency and cost.

Unified observability

Bifrost consolidates telemetry for all Claude Code activity into one place, replacing logs scattered across developer laptops. The built-in observability layer tracks token usage by team, project, and developer, alongside request volume, latency, and error rates.

Real-time dashboards: monitor spend and performance across every Claude Code consumer.
Request inspection: review full request and response payloads and trace multi-turn conversations for debugging.
Enterprise integrations: export native Prometheus metrics and OpenTelemetry traces into existing monitoring stacks, with custom alerting on usage thresholds.

This visibility is what lets a platform team find optimization opportunities and catch reliability regressions before they affect a wider rollout.

Implementation architecture

Bifrost supports several deployment models, so teams can match the gateway to their security and operations requirements.

Self-hosted: run the open-source gateway inside your own infrastructure for maximum control.
Enterprise managed: use Bifrost Enterprise for managed deployment, clustering, and advanced governance.
Hybrid / in-VPC: keep the gateway in your own VPC with in-VPC deployment for workloads that cannot egress to the public internet.

Request flow through the gateway follows a consistent path from the developer workstation to the provider and back into centralized monitoring:

Developer workstation (Claude Code)
        |
   Bifrost gateway (localhost:8080/anthropic)
        |
[Virtual key -> team budget -> model selection -> routing rules]
        |
   Provider API (Anthropic, Bedrock, Vertex, Azure, ...)
        |
   Response + logging + metrics
        |
   Centralized dashboard

On the security side, enterprise deployments add role-based access control, SSO and OIDC integration with providers like Okta and Microsoft Entra, and immutable audit logs for SOC 2, GDPR, HIPAA, and ISO 27001 compliance. Sensitive workloads can be isolated at the network layer so that Claude Code traffic never leaves controlled infrastructure.

Best practices for production Claude Code deployments

Roll out a Claude Code gateway in stages rather than enforcing every policy at once. The following sequence keeps the migration low-risk:

Start with monitoring. Deploy Bifrost in observability mode first. Establish baseline usage, identify high-volume teams, and understand cost drivers before enforcing any limits.
Layer in tiered access. Build a virtual key hierarchy that matches your org chart, starting with conservative budgets and adjusting against real usage.
Configure failover chains. Set an ordered fallback path, for example a primary Anthropic route backed by an equivalent model on AWS Bedrock, so sessions survive a provider incident.
Enable MCP tools deliberately. Bifrost as an MCP gateway extends Claude Code with external tools through the Model Context Protocol. Begin with low-risk tools such as filesystem and search, then add database or internal-API tools as governance matures.
Track quality, not just cost. Pair usage data with quality signals from automated testing so AI-assisted output stays reliable as the deployment grows.

Common questions about scaling Claude Code

Does routing Claude Code through a gateway change the developer experience?

No. The integration is a base URL change, and Bifrost returns responses in Anthropic's native format. Developers continue to use the claude command and Claude Code's native features without modification.

Can the same Claude Code session use non-Anthropic models?

Yes. Because Bifrost exposes an Anthropic-compatible endpoint while routing to 20+ providers, a single session can target Anthropic, AWS Bedrock, Google Vertex, Azure OpenAI, and others, selected by routing rules or on the fly.

How does an AI gateway control Claude Code costs?

Spend is controlled through virtual keys that carry per-team and per-developer budgets, combined with model routing that sends routine tasks to cheaper models. Both are enforced centrally at the gateway, so cost limits apply automatically to every request.

Getting started

Scaling a Claude Code deployment with Bifrost takes a short setup. The steps below assume a local gateway; production deployments follow the same pattern against a hosted instance.

1. Install Claude Code

npm install -g @anthropic-ai/claude-code

2. Run Bifrost and create a virtual key

Start the gateway, add your provider credentials, and create a virtual key scoped to a team with its own budget through the Bifrost dashboard or configuration.

3. Point Claude Code at the gateway

export ANTHROPIC_BASE_URL="http://localhost:8080/anthropic"
export ANTHROPIC_API_KEY="vk_your_team_key"

4. Launch Claude Code

claude

Teams that prefer not to manage environment variables manually can use the Bifrost CLI, which configures the connection and launches Claude Code with the correct settings automatically. The same flow works for other CLI agents such as Codex CLI, Gemini CLI, and Cursor. Claude Code itself is documented in Anthropic's official docs.

Scale Claude Code with confidence

Scaling Claude Code deployments across an enterprise requires more than distributing API keys. It requires centralized governance, multi-provider flexibility, and unified observability, delivered without changing how developers work. Bifrost provides that control plane as an open-source AI gateway, giving platform teams cost visibility, automatic failover, and audit-ready logging while preserving the native Claude Code experience.

To see how Bifrost can govern and scale your Claude Code deployment, book a demo with the Bifrost team or explore Bifrost Enterprise for managed deployment and advanced governance.

Top 5 AI Agent Evaluation Tools in 2026

claire nguyen — Tue, 02 Jun 2026 11:22:41 +0000

Evaluating AI agents requires more than static benchmarks. This guide compares the five leading AI agent evaluation platforms in 2026: Maxim AI, Langfuse, Arize, LangSmith, and Galileo. Maxim AI is the best choice for teams that need end-to-end simulation, evaluation, and observability in a single platform built for cross-functional collaboration.

Production AI agents now handle customer support escalations, financial data analysis, and multi-step autonomous workflows. As these systems become mission-critical, systematic evaluation is no longer optional. Evaluation spans three concrete dimensions: output quality across diverse scenarios, cost control in multi-step workflows, and audit trail generation for regulatory requirements. Modern evaluation platforms address all three through tracing, automated testing, and production monitoring. This guide covers the five platforms that lead the field in 2026.

What Is AI Agent Evaluation?

AI agent evaluation is the process of measuring agent output quality, task completion, and behavior across real-world scenarios before and after deployment. Unlike static ML model scoring, agent evaluation must account for multi-step trajectories where a single failure can cascade downstream. Effective evaluation frameworks cover pre-production simulation, automated scoring at session and trace levels, and continuous monitoring once agents are live.

1. Maxim AI: End-to-End Simulation, Evaluation, and Observability

Maxim AI is an end-to-end platform for AI simulation, evaluation, and observability, purpose-built for teams shipping agentic applications. The platform brings pre-release experimentation, scenario-based simulation, and production monitoring into a single interface designed for both engineering and product teams.

Simulation and Testing

Maxim's simulation engine tests agents across hundreds of scenarios and user personas before any code reaches production. Evaluation operates at the conversational level: complete agent trajectories are analyzed for task completion, and simulations can be re-run from any step to isolate root causes and reproduce failures.

Evaluation Framework

The platform supports a unified framework for machine and human evaluations. Teams access off-the-shelf evaluators from the evaluator store or create custom evaluators tuned to their quality criteria. Evaluations are configurable at the session, trace, or span level, giving engineering teams full granularity across multi-agent systems.

Observability Suite

Maxim's observability layer provides real-time production monitoring with distributed tracing. Custom dashboards expose agent behavior across any dimension, and automated quality checks trigger alerts when production metrics fall outside defined thresholds.

Data Management

Teams curate multi-modal datasets directly from production logs. Human-in-the-loop workflows support continuous dataset enrichment, and synthetic data generation covers evaluation scenarios that production traffic has not yet reached.

Cross-Functional Collaboration

A no-code UI enables product managers to configure evaluations and build dashboards without engineering dependencies. Playground++ supports rapid prompt engineering and model comparison across quality, cost, and latency dimensions. This is a key differentiator from tools that restrict workflow ownership to engineering teams alone.

Best for: Teams requiring full lifecycle coverage from experimentation through production, organizations where product and engineering collaborate on agent quality, enterprises with human-plus-LLM evaluation workflows, and teams building multi-agent systems that require granular observability with audit trails.

See evaluation workflows for AI agents for a deeper look at how teams structure their eval pipelines.

2. Langfuse: Open-Source Tracing with Self-Hosting

Langfuse is an open-source LLM observability platform that offers self-hosted deployment for teams with strict data residency requirements. The platform covers tracing, evaluation, and monitoring with full infrastructure control.

Core capabilities include prompt management with version tracking and usage pattern analysis, LLM-as-a-judge evaluations with custom or pre-built evaluators, session-based analysis for user-facing applications, and dataset creation from production traces for offline evaluation.

Best for: Teams with data privacy requirements that prohibit third-party cloud processing, developers building custom evaluation workflows on an open-source foundation, and organizations that need self-hosted infrastructure at low cost. See the Maxim vs Langfuse comparison for a full capability breakdown.

3. Arize: Unified Monitoring for ML and LLM Systems

Arize (Phoenix platform) applies ML observability principles to LLM monitoring, providing a single monitoring layer across classical ML models and agent applications. This makes it relevant for organizations that run both traditional models and generative AI in the same production stack.

Key capabilities include drift detection and performance degradation monitoring, tool selection and invocation evaluators for agent workflows, OpenTelemetry-compatible tracing via OpenInference instrumentation, and integration with AWS Bedrock Agents and major orchestration frameworks.

Best for: Enterprises running hybrid ML and LLM systems that need a unified monitoring view, data science teams already familiar with traditional MLOps tooling, and regulated industries that require explainability across both model types. Compare platform depth in the Maxim vs Arize breakdown.

4. LangSmith: LangChain-Native Debugging and Tracing

LangSmith is the observability and debugging tool from LangChain, built specifically for applications developed on the LangChain framework. It offers detailed tracing and tight integration with LangChain abstractions, which reduces setup time for teams already in that ecosystem.

Capabilities include multi-turn evaluation for complete agent conversations, an Insights Agent that automatically categorizes usage patterns, offline and online evaluation workflows, and annotation queues for subject-matter expert feedback collection.

Best for: Teams with a significant investment in the LangChain ecosystem, developers who need rapid prototyping and iterative debugging, and organizations looking for a developer-first tracing experience within LangChain-based architectures. For teams evaluating their options, the Maxim vs LangSmith comparison shows how the platforms differ on collaboration and evaluation depth.

5. Galileo: Hallucination Detection and Production Guardrails

Galileo focuses on AI reliability for high-stakes use cases, offering research-backed hallucination detection, an eval-to-guardrail lifecycle, and Luna-2 small language models for cost-effective production monitoring.

Core capabilities include research-grounded metrics for factual accuracy and hallucination detection, automatic conversion of pre-production evaluations into production guardrails, agent-specific metrics covering tool selection accuracy and session success rates, and a reported 97% cost reduction in monitoring via Luna-2 model inference. Agent evaluation metrics and context coverage are narrower in scope compared to full-lifecycle platforms.

Best for: High-stakes domains (healthcare, finance, legal) where factual accuracy validation is the primary requirement, teams that need real-time guardrails controlling live agent behavior, and organizations with safety compliance obligations.

Platform Comparison

Platform	Primary Strength	Deployment	Pricing	Open Source
Maxim AI	End-to-end simulation, evaluation, and observability with cross-functional collaboration	Cloud, On-premise	Free tier; Pro from $29/seat/month	No
Langfuse	Open-source tracing with self-hosting	Cloud, Self-hosted	Free tier (50k observations/month); Pro from $59/month	Yes
Arize	Unified ML and LLM monitoring	Cloud, On-premise	Contact sales	No
LangSmith	LangChain-native debugging	Cloud, Self-hosted (Enterprise)	Free tier (5k traces/month); Contact sales	No
Galileo	Hallucination detection and guardrails	Cloud	Free tier; Contact sales	No

How to Choose an AI Agent Evaluation Platform

The right platform depends on your evaluation scope, team structure, and deployment requirements:

Comprehensive lifecycle coverage across engineering and product teams: Maxim AI is built for this. The no-code UI, simulation engine, and observability layer give both engineering and product full workflow ownership.
Open-source control and data residency: Langfuse is the primary option, with self-hosted deployment and an active open-source community.
Hybrid ML and LLM monitoring in a unified interface: Arize addresses this with its Phoenix platform and OpenTelemetry-based tracing.
LangChain-native development: LangSmith reduces integration overhead for teams already building on LangChain.
Real-time guardrails and hallucination detection in regulated industries: Galileo is designed specifically for this use case.

Platforms that only address one phase of the agent lifecycle (tracing only, or guardrails only) create gaps between pre-production testing and production monitoring. For most teams building at scale, a platform that spans the full development cycle reduces the overhead of maintaining separate tools and correlating data across systems.

Maxim AI's agent quality evaluation approach covers this in more depth, including how simulation connects to production monitoring in a unified workflow.

Summary

AI agent evaluation in 2026 requires platforms that cover the complete development lifecycle, not just one phase. Maxim AI leads for teams that need simulation, evaluation, and observability in one place with cross-functional collaboration built in. Langfuse is the right choice when data control and open-source infrastructure are the priority. Arize fits organizations running hybrid ML and LLM workloads. LangSmith is the natural pick for LangChain-focused teams. Galileo addresses the specific need for hallucination prevention in safety-critical domains.

Selecting the wrong tool adds cost and complexity, particularly when teams must maintain separate systems for pre-production testing and production monitoring. Matching the platform to your team's structure, deployment requirements, and evaluation scope is the decision that matters most.

To see how Maxim AI covers the full AI agent evaluation lifecycle, from simulation through production monitoring, book a demo or sign up for free.

The SIGTERM our build workers ignored, and the 90s that fixed it

claire nguyen — Tue, 02 Jun 2026 04:21:50 +0000

TL;DR: Our ECS build workers were quietly killing in-flight jobs every time we scaled in or deployed. The fix wasn't a bigger timeout, it was actually handling SIGTERM and bumping stopTimeout to 120s. Cut our "agent lost" failures from ~2% of runs to under 0.1%.

So, the thing that bugged me for weeks. We run a chunk of Buildkite's build compute on ECS, and every deploy or scale-in event would spike a small batch of failed builds. Not heaps. Maybe 2% of running jobs at that moment. Enough that someone in the team Slack would go "oi, my build died again" once a day.

The error was always the same flavour: agent disconnected mid-step, job marked as lost, customer retries, moves on. Annoying but not loud enough to page anyone. Which is exactly why it survived for a month.

What was actually happening

ECS sends SIGTERM to your container when it wants the task gone. Scale-in, deployment, spot reclaim, all of it. You get a grace window, then SIGKILL. The default stopTimeout is 30 seconds.

Our build agent process caught nothing. The PID 1 in the container was a shell wrapper, and the agent ran as a child. SIGTERM hit the shell, the shell shrugged, the agent kept running until SIGKILL ripped the whole thing out. A 6-minute integration test step that was 80% done? Gone.

The agent already supports graceful stop. It'll finish the current job and refuse new ones if you signal it properly. We weren't passing the signal through. Classic.

Here's the before. Spot the problem:

# before — shell eats the signal, agent never hears it
ENTRYPOINT ["/bin/sh", "-c", "buildkite-agent start"]

sh -c doesn't forward signals to the child by default. So PID 1 catches SIGTERM and does nothing useful with it.

The fix

Two parts. Run the agent as PID 1 so it gets the signal directly, and give it enough time to drain.

# after — exec replaces the shell, agent becomes PID 1
ENTRYPOINT ["buildkite-agent", "start"]

Then the ECS task definition:

{
  "containerDefinitions": [
    {
      "name": "build-agent",
      "stopTimeout": 120,
      "essential": true
    }
  ]
}

We set stopTimeout to 120 because most steps finish inside two minutes, and the agent's own --cancel-grace-period lines up so it doesn't get cut off mid-drain. The agent hears SIGTERM, stops accepting new work, lets the current job run to completion, then exits clean. ECS sees a tidy exit and moves on.

The agent does the right thing on its own once it actually receives the signal:

buildkite-agent start \
  --cancel-grace-period 120 \
  --disconnect-after-idle-timeout 10

That --disconnect-after-idle-timeout matters for the scale-in path. An idle agent disconnects fast so the autoscaler isn't waiting 120s to retire a worker doing nothing.

The numbers

Metric	Before	After
`agent lost` failures (% of runs)	~2%	<0.1%
Avg drain time on scale-in	n/a (SIGKILL)	11s idle / 70s busy
Builds killed per deploy	8–15	0
`stopTimeout`	30s	120s

The drain time on busy workers went up, sure. We trade a slower scale-in for builds that don't die. Easy call. Idle workers still retire in about 11 seconds because they've got nothing to finish.

One thing I'll flag for the LLM-curious crowd here on r/LocalLLaMA: a few of our build steps call a model for flaky-test classification, and those run through a small gateway sidecar (we use Bifrost) in the same task. That process needs the same treatment. It has to flush in-flight requests on SIGTERM or you get half-finished calls counted as failures, same bug in a different shirt. Once the agent drains properly, the sidecar's automatic failover stops papering over the real problem.

How we caught it for good

The fix is one thing. Trusting it is another. We added a game day check: kill a task running an active build, on purpose, and assert the build completes instead of going lost.

# during a game day — force-drain a worker mid-build
aws ecs stop-task --cluster build-prod --task "$TASK_ARN"
# then assert the build state, not just that the task stopped
bk-cli build get "$BUILD_ID" --field state # expect: passed

If you never test the drain, you don't have a drain. You've got hope. "Never had a lost build" usually means nobody's pulled the plug while watching.

Trade-offs and limitations

Longer stopTimeout means slower deploys when workers are busy. If you're running a fleet of 500, that's real wall-clock time during a rolling deploy. We accepted it because dead builds cost more than slow deploys.

stopTimeout has a hard ceiling of 120s on ECS. If a single build step legitimately runs longer than two minutes and can't be checkpointed, this won't save it. You'll still lose those. For us that's a rare long-running step, and we've moved most of them to job-level retries instead.

This also assumes your work is interruptible-after-current-task. If one agent holds a single 40-minute job, graceful stop just means waiting 120s then killing it anyway. Drain logic helps fleets of short jobs far more than it helps long monolithic ones.

And spot reclaim only gives you a 2-minute warning, which is right at the edge of our 120s window. Tight. We lean on retries as the backstop there rather than pretending the drain always wins.

Error budgets for an LLM dependency you don't control

claire nguyen — Mon, 01 Jun 2026 13:22:28 +0000

TL;DR: We shipped a natural-language build-query feature at Buildkite, then tried to put a 99.9% SLO on it. Turns out you can't promise uptime for a model provider you don't run. We put Bifrost in front, failed over across three providers, and now the error budget tracks our gateway's behaviour instead of OpenAI's status page.

Here's the moment it clicked for me. We were drafting an SLO doc for a feature that lets people ask "why did this build fail" in plain English. Someone wrote "99.9% availability". Cool. That's 43 minutes of allowed downtime a month. Then OpenAI had a wobble for about 50 minutes one Tuesday and we blew the whole budget before lunch.

The problem wasn't our code. Our service was up the entire time. The dependency wasn't.

You can't SLO something you don't operate

A normal SLO assumes you control the thing you're measuring. Postgres, your own API, an internal queue. You can add replicas, you can tune it, you can page someone who can fix it.

A hosted LLM is none of that. When Anthropic returns a 529 or OpenAI starts handing out 429s under load, there is no lever on your side. You wait. Our p99 for the feature was around 2.1 seconds on a good day, and during provider degradation it'd climb past 9 seconds or just fail outright.

So the question stopped being "how do I make the provider more reliable" and became "how do I make my dependency on any single provider less load-bearing." That's a routing problem, not a model problem.

Putting a gateway in the path

We run Bifrost as the single egress point for every LLM call now. It's an OpenAI-compatible gateway, so our service code didn't change much. The interesting part is the fallback config: if the primary provider errors or times out, the request gets retried against the next one without our app knowing.

{
  "providers": {
    "openai": { "keys": [{ "value": "env.OPENAI_KEY" }] },
    "anthropic": { "keys": [{ "value": "env.ANTHROPIC_KEY" }] },
    "bedrock": { "keys": [{ "value": "env.BEDROCK_KEY" }] }
  },
  "fallbacks": [
    "openai/gpt-4o-mini",
    "anthropic/claude-3-5-haiku",
    "bedrock/anthropic.claude-3-haiku"
  ]
}

Three providers, ranked. When OpenAI throttles, the call lands on Anthropic. When both are sad, Bedrock catches it. The feature degrades in quality maybe, but it stays up. That's the whole point of an error budget. Stay inside the line.

It also does load balancing across multiple keys, which mattered more than I expected. Half our "outages" early on were just one API key hitting its rate limit while another sat idle.

The metrics that actually feed the SLO

The bit that sold me was native Prometheus output. Bifrost exposes metrics straight out of the box, so I'm not scraping a vendor status page or parsing logs to know if we're burning budget.

Our availability SLI is now "requests Bifrost successfully resolved, including via fallback" over total requests. A request that failed on OpenAI but succeeded on Anthropic counts as a win, because the user got an answer. That's the number that should drive the SLO, not per-provider success.

# fast burn-rate over 1h: are we eating budget faster than allowed?
sum(rate(bifrost_requests_total{status="error"}[1h]))
/
sum(rate(bifrost_requests_total[1h]))
> (14.4 * 0.001)

We went from one provider doing about 99.4% effective availability to the fallback chain sitting around 99.93% over the last 60 days. Same models, same budget, just not betting the feature on one company's afternoon.

How it stacks up

We looked at LiteLLM and Portkey before landing here. None of these is strictly best. Depends what you're optimising for.

Thing I cared about	Bifrost	LiteLLM	Portkey
Self-host, no vendor in path	Yes, single Go binary	Yes	Possible, but hosted is the main path
Native Prometheus metrics	Built in	Via callbacks/config	Dashboard-first, export is extra
Provider failover config	Declarative fallback list	Yes, router config	Yes, configs/strategies
Hosted analytics UI	Basic built-in UI	Minimal	Strongest of the three
Python ecosystem depth	Smaller	Largest, huge community	Good

Honestly, if you live in Python and want the biggest provider list and community, LiteLLM is hard to beat. If you want a polished hosted dashboard and guardrails without running anything, Portkey is the comfortable pick. We're an infra team that wants metrics in our own Prometheus and a binary we can run on our own boxes, so Bifrost fit our shape. No worries either way.

Trade-offs and Limitations

Fallback hides failure, and that cuts both ways. If your alerting only watches the final success rate, you can be quietly running 80% of traffic on your third-choice provider for days and not notice the bill. We added a separate alert on per-provider fallback rate so degradation is visible, not just survivable.

Quality drift is real too. gpt-4o-mini and claude-3-5-haiku don't answer identically, so a build-failure summary can read differently mid-incident. For us that's acceptable. For anything doing structured extraction, you'd want to validate output shape per provider.

And a gateway is one more thing to run. It's a low-risk component, but it's in the hot path, so we run it with the same care as any other tier-1 service. If Bifrost is down, everything's down. We game-day it like the rest of our stack.

Self-hosting also means semantic caching, governance, and the rest are your config problem, not a managed feature. Fine for us. Worth knowing.

The Prometheus label that blew our monitoring bill out 6x

claire nguyen — Fri, 29 May 2026 04:21:15 +0000

TL;DR: Our metrics bill went 6x in a single month. Traffic was flat. One Prometheus label carrying per-build IDs spawned millions of time series, and the backend charges by active series. Here's how we caught it and the label rules we run now so it doesn't happen again.

The bill, not the traffic

I'm on the infra team at Buildkite. We run a fairly chunky Prometheus setup feeding a managed backend, and one Monday the monthly estimate had quietly gone from about $1,800 to a touch over $11k. Nobody shipped more traffic. Build volume was the same 40k-ish builds a day it'd been for weeks.

So it wasn't load. It was series count. Active series had climbed from roughly 1.2 million to nearly 9 million, and the backend prices on active series, not on request volume. That's the trap most people miss the first time.

What cardinality actually is

Think of every unique combination of metric name plus label values as its own drawer in a filing cabinet. http_requests_total{status="200"} is one drawer. Add region="ap-southeast-2" and now you've got a drawer per region. Add a label whose values are unbounded and you've got a cabinet the size of a warehouse.

Cardinality is the count of those drawers. Each one is a separate time series that has to be stored and indexed. Low-cardinality labels (status, region) are fine. High-cardinality ones are where the money leaks.

The one label that did it

A teammate had added build_id to a counter so they could debug a flaky deploy. Fair enough in the moment. Problem is every build has a unique ID, we do ~40k a day, and those IDs hang around for the full retention window.

40k unique values a day, multiplied across a handful of other labels, multiplied across retention. That's your several-million-series jump right there. One label.

Catching it

The fastest way to find the offender is to ask Prometheus which metric has the most series:

topk(10, count by (__name__)({__name__=~".+"}))

Then drill into the worst metric and see which label is doing the damage:

count(count by (build_id)(deploy_attempts_total))

When that second query came back with a number in the tens of thousands, we had our culprit.

The fix

You drop the label before it ever hits storage. metric_relabel_configs runs at scrape time, so you can strip a label without touching the app code:

scrape_configs:
  - job_name: "build-agents"
    metric_relabel_configs:
      - regex: "build_id"
        action: labeldrop

Per-build detail didn't vanish, we moved it to where unbounded identifiers belong: traces and logs. If you genuinely need a metric sliced per build, use exemplars so the high-cardinality bit lives in the trace store, not the series index.

Here's how we now reason about labels before adding one:

Label	Unique values	Safe to add?
status	~5	Yes
region	~6	Yes
instance_type	~15	Yes
agent_queue	~200	Usually fine
build_id	~40k/day	No, use a trace
user_email	unbounded	No, never

Rule of thumb we reckon on: if you can't name the upper bound of a label's values on a whiteboard, it doesn't go on a metric.

Same trap, different service

This isn't only a Prometheus-the-app thing. Any service that emits Prometheus metrics can sink you the same way. We run a small internal feature that summarises failed build logs through an LLM, and those calls go through Bifrost, an open-source AI gateway that ships native Prometheus metrics out of the box. Handy. But the instinct to tag those metrics with a per-request ID or per-virtual-key label is exactly the same footgun.

We keep its labels down to provider and model. That gives us cost-per-provider and latency-per-model without minting a new series for every call. The discipline travels with the metric, not the tool.

Trade-offs and Limitations

Dropping build_id means you can't slice a single build inside Prometheus anymore. For ad-hoc "what did build 84213 do" questions, you're now in the trace or log tooling, which is a context switch some folks grumbled about for a week.

Recording rules, the other common fix, aren't free either. They add evaluation load on the Prometheus side, and if you write a sloppy one you can quietly recreate the cardinality you were trying to kill. Test the output series count before you ship the rule.

Exemplars need backend support and a tracing system wired up. If you haven't got distributed tracing yet, that path's a bigger project than a one-line labeldrop. Be honest about where you are.

And labeldrop is a blunt instrument. Once it's gone at scrape, it's gone. If you later decide you wanted that dimension bounded rather than dropped, you're re-instrumenting.

Our PR-review bot kept hitting 429s. Bifrost key pooling fixed it.

claire nguyen — Thu, 28 May 2026 13:22:11 +0000

TL;DR: Our internal PR-review bot was getting 429'd by Anthropic between 9am and 11am Sydney time. We dropped Bifrost in front, pooled four keys, and the 429 rate fell from 8.2% to 0.07% in a fortnight. The migration was one env var swap. The interesting bits were the bits we got wrong.

The problem

We've got a PR-review bot that pings Claude on every pull request opened against our internal monorepo. It pulls the diff and ships a structured prompt to Claude. Gets back a summary plus a couple of "have you considered..." nudges. Saves our reviewers maybe 10 minutes per PR, on a team of 80 engineers, all sharing one Anthropic workspace that someone provisioned back in early 2024 and nobody bothered to split.

You can guess what happened.

Mornings in Sydney are brutal. Everyone arrives, opens their PRs from the night before, and our bot fires off 30-40 concurrent requests. Anthropic's per-org rate limit got chewed through by 9:15 most days. Bot started failing. Slack filled up with "did the review bot die again?" messages. Not a great look for the platform team.

What we tried first

The naive fix was a job queue with backoff. Wrote it in an arvo. Buildkite job, Redis-backed, exponential retry with jitter. It worked, sort of. Reviews now took 4-7 minutes to come back instead of 8 seconds, and engineers started ignoring the bot entirely because by the time the review landed they'd already merged, which kind of defeats the whole point of having a review bot.

Queueing wasn't the answer. We needed more headroom, which meant more keys, which meant somebody had to manage them.

Why Bifrost

I'd been kicking the tyres on a few gateways for an unrelated project. Bifrost (https://github.com/maximhq/bifrost) won on two specific points: load balancing across multiple API keys for the same provider is a documented first-class feature, and the OpenAI-compatible endpoint meant we didn't have to touch the bot's SDK code. It already spoke openai.ChatCompletion against an internal proxy URL.

Setup took about 40 minutes including the time to argue with our SSO admin about a new GitHub OAuth app.

{
  "providers": {
    "anthropic": {
      "keys": [
        { "value": "env.ANTHROPIC_KEY_1", "weight": 1.0 },
        { "value": "env.ANTHROPIC_KEY_2", "weight": 1.0 },
        { "value": "env.ANTHROPIC_KEY_3", "weight": 1.0 },
        { "value": "env.ANTHROPIC_KEY_4", "weight": 1.0 }
      ],
      "network_config": {
        "default_request_timeout_in_seconds": 30
      }
    }
  }
}

Bot config was a one-liner. Pointed OPENAI_API_BASE at our Bifrost ECS service on port 8080 and the bot didn't know it'd been moved.

Results after two weeks

| Metric | Before (queue + 1 key) | After (Bifrost + 4 keys) |
|---|---|
| Median review latency | 4m 30s | 11s |
| p95 review latency | 7m 12s | 28s |
| 429 rate | 8.2% | 0.07% |
| Reviews abandoned (timed out) | 14% | 0.4% |
| "Is the bot dead" Slack pings | ~6/day | 0 |

Costs went up about 22% because more reviews actually completed. Worth it.

Bifrost vs LiteLLM vs Portkey

I evaluated all three properly. None is strictly better; they hit different sweet spots.

| Concern | Bifrost | LiteLLM | Portkey |
|---|---|
| Multi-key load balancing | Native | Via Router | Native |
| OpenAI-compatible endpoint | Yes | Yes | Yes |
| Self-host complexity | Single Go binary | Python + deps | SaaS-first |
| Built-in web UI for config | Yes | Limited | Cloud-side |
| Semantic caching | Yes | Yes | Yes |
| MCP gateway | Yes | No | No |
| Community size | Growing | Larger | Larger |

LiteLLM's community is bigger and the integrations list is wider. If you want Python ergonomics, it's the easier ride. Portkey's hosted UX is slicker out of the box, but we needed self-host for compliance reasons. Bifrost being a single Go binary suited our ECS deploy model and our preference for fewer Python services in the critical path.

Trade-offs and limitations

It's not all roses.

Failover is per-request, not per-key cooldown. If one of our four keys gets stuck in a rate-limit hole, Bifrost retries the call elsewhere but doesn't proactively quarantine the bad key for a window. We're managing that with manual weight tweaks for now.
The web UI is handy but state lives in config files. Make changes via the UI in dev and forget to commit the config, and you've got drift. We learned that one the hard way.
Single point of failure. Anything you put in front of every LLM call becomes load-bearing. We run two Bifrost replicas behind an ALB. Tiny team running one node and a restart policy might be fine, but think about it before you ship.
Observability glue. Prometheus metrics are emitted natively, which is great. You'll still need to wire them into your existing dashboards. Took us an afternoon.

Surviving an AZ Failover for Our Build Runner Fleet at 3am

claire nguyen — Wed, 27 May 2026 13:24:15 +0000

TL;DR: We lost an AWS AZ for 47 minutes back in March. Our build runner fleet on EKS mostly survived, but the AI-assisted code review bots wedged because their LLM calls all routed to one region. Sticking Bifrost in front of those calls fixed the second problem. Here's what we changed.

It was 3:12am Sydney time when PagerDuty went off. ap-southeast-2a was having a wobble. Not a full outage — just enough packet loss that EKS nodes started flapping in and out of the cluster.

Our build runner fleet handled it fine. We've drilled this. Pod disruption budgets, multi-AZ node groups, the usual stuff. Builds rescheduled to 2b and 2c within about 90 seconds. No worries.

The bit that didn't handle it fine was the AI review bot we'd shipped six weeks earlier. That thing called Anthropic's API directly from inside the build container. When the AZ flapped, the egress NAT in 2a started dropping outbound TLS. The bot retried, hit our 30-second build timeout, and 4,200 builds went red over half an hour.

I want to talk about what we did the morning after, because the fix wasn't "make the bot more resilient." It was "stop pretending the LLM call is special."

The actual failure mode

Here's the rough shape of what was happening. Our review bot was a Go service running as a sidecar in the build pod. Pseudo-config looked like this:

review_bot:
  provider: anthropic
  api_key: ${ANTHROPIC_KEY}
  model: claude-sonnet-4-6
  timeout_ms: 25000
  max_retries: 2

Two retries, 25 second timeout each. Sounds reasonable. Except when the underlying network is dropping packets, you don't fail fast — you sit there waiting for TCP to give up. Two retries became 75 seconds of nothing. Build timeout kicked in. Build failed.

Worse, every single review bot in every single build was hitting the same NAT gateway in the same degraded AZ. We'd accidentally built a single point of failure into something we'd designed as a sidecar.

What we changed

I'd been kicking the tyres on Bifrost (https://github.com/maximhq/bifrost) for a few weeks already because I wanted central observability on LLM spend across our internal tools. The AZ incident pushed it to the top of the queue.

The plan was simple: stop letting build pods talk to providers directly. Run Bifrost as a deployment in our shared platform namespace, spread across all three AZs, and point the review bot at it. The bot's config went from anthropic.com to an internal service URL.

Bifrost's drop-in replacement (https://docs.getbifrost.ai/features/drop-in-replacement) meant we didn't touch the bot's code. Just the env var.

Then we configured fallbacks (https://docs.getbifrost.ai/features/retries-and-fallbacks) so a failed Anthropic call rolls over to AWS Bedrock's Claude. Same model family, different network path, different auth, different everything.

{
  "model": "anthropic/claude-sonnet-4-6",
  "fallbacks": [
    "bedrock/anthropic.claude-sonnet-4-6",
    "openai/gpt-4o-mini"
  ]
}

The GPT-4o-mini at the bottom is a deliberate downgrade. If both Anthropic paths are stuffed, we'd rather give the dev a worse review than no review and a red build.

What it looks like vs the alternatives

I evaluated three things properly. Here's the honest comparison from my notes:

Concern	LiteLLM	Portkey	Bifrost
Self-hosted Go binary	No (Python)	Partial	Yes
Provider failover config	Yes	Yes	Yes
Built-in web UI for config	Limited	Yes (cloud)	Yes (local)
Semantic caching	Plugin	Yes	Yes
Memory footprint on our nodes	~400MB	N/A (SaaS-first)	~180MB
MCP gateway	No	No	Yes (enterprise)

LiteLLM is genuinely good and we run it for one of our data science notebooks because the Python ergonomics are nice. Portkey has the slickest dashboard if you're happy with their cloud. Bifrost won here because we wanted a Go binary we could run on our own infra, and the resource overhead per pod mattered when we're scheduling hundreds of build pods.

The boring infra bit

We deployed Bifrost as three replicas, one per AZ, behind a ClusterIP service. Topology spread constraints to keep them honest. Each pod has its own provider key set via Kubernetes secrets, referenced through Bifrost's env var support (https://docs.getbifrost.ai/deployment-guides/config-json#environment-variable-references).

Prometheus scrape config picks up the native metrics endpoint. We graph p99 latency per provider and alert on fallback rate above 5% for more than 10 minutes. That alert would have fired during the March incident and given us a much better signal than "builds are timing out."

Trade-offs and limitations

This isn't a free win. A few things to flag.

The gateway is now a new hop in the request path. We measured about 8-12ms added per call. For our use case that's noise. For real-time inference it might not be.

Bifrost's clustering features are an enterprise thing. We're running it as independent replicas behind a service, which works because our config is mostly static. If you need shared state across replicas (live config sync, shared rate limit counters), you'll either pay for enterprise or accept some eventual consistency.

Semantic caching sounds great but we haven't turned it on for the review bot because code reviews are too context-specific. Cache hit rate would be near zero. Worth knowing before you assume it'll save you money.

And the obvious one: a gateway pod failing is now a thing that can break LLM calls. Spread your replicas, set sensible PDBs, don't be silly.

The Cost Math Behind Our CI Cache Hit Rate Going From 40% to 91%

claire nguyen — Wed, 27 May 2026 04:23:02 +0000

TL;DR: We were burning roughly AUD $14k/month on redundant CI compute because our cache hit rate sat at 40%. Three changes (content-addressed keys, a warmer tier, and killing one bad pre-commit hook) pushed it to 91% and shaved the bill to about $3.2k. Most of the savings came from a single weekend audit, not new tooling.

I run infra at Buildkite. We eat our own dog food, which means our internal monorepo runs on the same agents we sell to customers. About six weeks ago our finance team flagged that our CI compute line on AWS had crept up 38% quarter-on-quarter while team headcount only grew 11%. Something was off.

Turns out the culprit wasn't traffic. It was caches.

The starting point

Our setup, roughly:

~280 engineers across Sydney, Melbourne, San Francisco
Around 4,200 builds/day on the monorepo
Mix of Go, TypeScript, and a chunky Python ML eval service
Agents running on m6i.4xlarge spot instances in ap-southeast-2
Remote cache backed by S3 with a CloudFront distribution

When I first pulled the numbers, our cache hit rate (measured per build step, weighted by step duration) was sitting at 40.3%. For a healthy CI setup of this size I'd reckon you want 80%+. Anything under 60% means you're paying twice for the same compute.

Here's what the spend breakdown looked like before we touched anything:

Component	Monthly cost (AUD)	% of total
Spot EC2 (build agents)	$11,200	67%
S3 cache storage	$890	5%
CloudFront egress	$1,140	7%
LLM eval API calls (OpenAI + Anthropic)	$3,420	21%
Total	$16,650	100%

The LLM line is the one nobody expected. We run automated PR review on a subset of changes, plus regression evals on our search ranking service.

The three things that actually mattered

1. Content-addressed cache keys

We had cache keys like node_modules_v3_${branch_name}_${os}. That's already wrong but the worse bit was the v3 suffix that someone bumped six months ago and forgot why.

Switched to hashing the actual inputs: package-lock.json content hash + Node version + OS. Standard stuff but we'd just never done it properly.

steps:
  - label: ":node: install"
    plugins:
      - cache#v2.4.0:
          manifest: package-lock.json
          path: node_modules
          restore: file
          save: file
          key: "v1-{{ runner.os }}-node-{{ checksum 'package-lock.json' }}"

The restore: file bit matters. It means we only invalidate when package-lock.json actually changes, not when the branch name changes. Cache hit rate on the install step went from 31% to 96% overnight.

2. A warmer tier between memory and S3

S3 is cheap but the round-trip from ap-southeast-2 agents to S3 is about 18ms for small objects, and we were pulling thousands of them per build. We added an r6gd.large instance with NVMe local storage as an in-region warm cache. Agents check there first, fall through to S3.

Cost: about $180/month for the warm cache instance. Saves us roughly $1,400/month in CloudFront egress because most cache reads never leave the VPC now.

3. The bad pre-commit hook

This one is embarrassing. Someone added a pre-commit hook two years ago that ran find . -name "*.pyc" -delete before every test invocation. On a clean checkout this does nothing useful. On a cached checkout it deletes all the compiled Python bytecode, forcing Python to recompile on every test run. Average test step went from 4m20s to 2m45s after deleting eight lines of bash.

I genuinely could not believe it. We'd been paying for that for two years.

The LLM bit

The $3,420 LLM line was harder to chip away at because the calls themselves are useful. What we did:

Routed the PR review traffic through an AI gateway (we use Bifrost, which gives us semantic caching and a single endpoint) so identical or near-identical review prompts hit cache instead of provider
Moved the search ranking evals to a nightly batch rather than per-PR
Switched the bulk of the review traffic to a cheaper model and reserved the expensive one for changes touching /security/*

Semantic cache hit rate on PR review prompts settled around 34%, which doesn't sound massive but the prompts that hit cache tend to be the bigger ones (boilerplate "review this dependency bump" type stuff), so the dollar impact was bigger than the hit rate suggests.

Final LLM line came down to $1,180/month.

Where we landed

Component	Before	After	Change
Spot EC2	$11,200	$1,820	-84%
S3 + warm cache	$890	$1,070	+20%
CloudFront egress	$1,140	$140	-88%
LLM API	$3,420	$1,180	-65%
Total	$16,650	$4,210	-75%

Cache hit rate: 91.2% weighted.

Trade-offs and Limitations

The warm cache tier is a single point of failure. If that r6gd.large dies, we fall through to S3 cleanly but builds slow down by ~40 seconds each until we replace it. For us that's fine because spot interruption is more common than instance failure anyway. For a smaller team I'd skip it.

Content-addressed keys made cache busting harder for the rare case where you legitimately want to invalidate everything. We added a manual BUILDKITE_CACHE_EPOCH env var so a human can force-invalidate when needed. Used it twice in three months.

The pre-commit hook thing wasn't a tooling problem. It was institutional knowledge rot. There's no caching strategy that protects you from someone deleting your bytecode every commit. You need humans to actually read what runs.

DEV Community: claire nguyen

What is shadow AI and how to govern it

What shadow AI is

Why shadow AI is a security risk, not just a policy gap

Why existing controls miss shadow AI

How the Bifrost AI gateway with Bifrost Edge govern shadow AI

Guardrails apply before data leaves the machine

Visibility into MCP servers across the fleet

App policy enforced on every device

Rollout through your existing device management

Common questions about shadow AI

Is shadow AI the same as shadow IT?

Can shadow AI be detected?

How do you govern AI coding agents?

Where this leaves enterprise teams

Fault-injecting our LLM provider to trust Bifrost fallbacks

Why a gateway at all

The game day

What the metrics showed

How it stacks up against LiteLLM and Portkey

Trade-offs and limitations

Further Reading

An 8-minute outage from a dead NLB and a JVM that cached DNS forever

The bit where everything looked fine

Why one runtime behaved differently

The fix

Where the LLM bit sneaks in

Trade-offs and limitations

What I'd tell past me

Further Reading

A provider latency spike stalled our whole build queue

The bit nobody designs for

What we changed

Did it work

How it stacks up

Trade-offs and limitations

Further reading

Scaling Claude Code Across Enterprise Engineering Teams

Why scaling Claude Code across teams is hard

What an enterprise AI gateway does for Claude Code

Key capabilities for scaling Claude Code deployments

Multi-team access control with virtual keys

Cost optimization through model routing

Production-grade reliability

Unified observability

Implementation architecture

Best practices for production Claude Code deployments

Common questions about scaling Claude Code

Does routing Claude Code through a gateway change the developer experience?

Can the same Claude Code session use non-Anthropic models?

How does an AI gateway control Claude Code costs?

Getting started

Scale Claude Code with confidence

Top 5 AI Agent Evaluation Tools in 2026

What Is AI Agent Evaluation?

1. Maxim AI: End-to-End Simulation, Evaluation, and Observability

Simulation and Testing

Evaluation Framework

Observability Suite

Data Management

Cross-Functional Collaboration

2. Langfuse: Open-Source Tracing with Self-Hosting

3. Arize: Unified Monitoring for ML and LLM Systems

4. LangSmith: LangChain-Native Debugging and Tracing

5. Galileo: Hallucination Detection and Production Guardrails

Platform Comparison

How to Choose an AI Agent Evaluation Platform

Summary

The SIGTERM our build workers ignored, and the 90s that fixed it

What was actually happening

The fix

The numbers

How we caught it for good

Trade-offs and limitations

Further Reading

Error budgets for an LLM dependency you don't control

You can't SLO something you don't operate

Putting a gateway in the path

The metrics that actually feed the SLO

How it stacks up