DEV Community: Swapnoneel Saha

What Is Granular Access Control? RBAC, ABAC, and AI

Swapnoneel Saha — Wed, 09 Sep 2026 17:25:41 +0000

"Can this person use the AI API?" is one question.

"Can this service call gpt-5, at most 200 times an hour, with a 50 dollar cap this month, with no file system tools attached, and only until Friday?" is a completely different question.

Granular access control is what turns the first question into the second one. It means defining permissions at the smallest useful unit, so an identity can take exactly one action, on exactly one resource, within exactly one budget, under exactly one set of conditions, and nothing else.

That idea has a name, the principle of least privilege, and it just means giving each caller the minimum access it needs. Granular access control is how you actually write that down.

And the reason it stopped being a nice-to-have this year is that most of the things asking for permission are no longer people.

Access control is a dial, not a switch

Strip a permission down and it is three things: a subject (who is asking), an action (what they want to do), and a resource (what they want to do it to). Every access control system you have ever used is just a way of writing those three things down and checking them.

Coarse-grained access control leaves all three parts wide open. "Engineers can use the database" is one permission covering forty people, six verbs, and every table you own. It is one line of config, and it is fast to write, which is exactly why it survives so long.

Granular access control narrows the same three parts instead. "The reporting service can run SELECT on the orders table, and nothing else." One caller, one verb, one table.

Think of a house key against a hotel keycard. The house key opens everything you own, forever, and the only way to take it back is to change the lock. The keycard opens room 402, on floor 4, until Friday at 11 in the morning, and the front desk can kill it from the lobby without touching a single door.

Both are access control. The keycard is granular.

So granularity is not a product you buy or a model you adopt. It is how far down you are willing to turn the dial, and how much extra work you are willing to do in exchange. Turn it too far and you get a permissions table nobody can read. Leave it too coarse and one leaked credential owns the whole system.

The five dimensions you can narrow a permission on

Most explanations of granular access control stop at "give people less access", which is true and completely useless as advice. The useful version is that a permission can be narrowed along five separate axes, and you can turn each one independently.

Here they are, from the one everybody already does to the one almost nobody does.

1. Who is asking

The identity. This is the axis every system starts with, and the one that changed the most recently.

For a long time a subject meant a person, or a service account somebody created once and forgot about. Now a subject is just as likely to be a coding agent, a CI job, a background worker, or a chain of three agents where the last one has no idea which human started the request.

These are called non-human identities, and they are the ones quietly holding most of the over-permissioned access. Sonrai's cloud access research, published in May 2026, found that 92% of identities with sensitive permissions did not use them once in 90 days, and 87% of that group were machine identities. That is a vendor scanning its own customers and not an independent audit, so take the exact number lightly.

The fix here is easy to say and annoying to do: one identity per caller, never one shared credential per team.

2. What action they want to take

Read, write, update, delete. List versus fetch. Call a model versus look at what models exist.

Coarse systems have two levels here, usually named something like read and admin. Granular systems split up the verbs that can do different amounts of damage. Being able to view a log line and being able to export the whole log table are not the same permission, even though both are "reading".

3. Which resource, exactly

This is where granularity gets real, because a resource is rarely one flat thing.

A database can be scoped at the server, the database, the table, the column, or the row. An AI setup can be scoped at the provider, the model, or a single tool that a model is allowed to invoke. Each step down that ladder means less damage if the key leaks, and one more thing to maintain.

Row-level scoping is the one worth knowing by name. It answers "which of these records can you see", which is a different question from "can you see this table", and it is the difference between a support agent seeing their own tickets and a support agent seeing everybody's.

4. How much they can consume

Here is the axis classic access control mostly ignores, and it is the one that bites hardest with AI.

Permission has traditionally been a boolean. You can call this endpoint, or you cannot. But when a single call can cost real money and a loop can make ten thousand of them, "yes" without a ceiling is not a permission, it is an open tab.

So the granular version attaches quantity to the grant. This key may spend 200 dollars a month. This key may burn 10,000 tokens an hour. This key may make 100 requests a minute. Developers have been asking providers for exactly this for years, and OpenAI's own developer forum carries a feature request for per-key spending limits with people piling into it.

5. Under what conditions

The context around the request. Time of day, source IP, environment, device posture, whether the key has expired yet.

Expiry is the underrated one. A permission with no end date is a permission you will forget you granted, and every "how did this old key still work" incident starts there.

Is granular access control the same as RBAC?

No, and this trips up a lot of people, so it is worth separating properly.

RBAC (role-based access control) groups permissions into roles and hands roles to identities. It is the most common model in the world because it maps onto how companies actually think. You are a developer, developers get the developer role, done.

RBAC can be granular or coarse depending entirely on how you write the roles. That is the part people miss. A role called admin with every permission attached is RBAC and it is not granular at all.

The trouble shows up when you try to push RBAC down the dial. Every new condition needs its own role, so you get developer-staging, developer-staging-eu, developer-staging-eu-readonly, and pretty soon nobody can tell you what any of them do. The industry name for this is role explosion, and it is the standard failure mode of a team that discovered granularity and only had roles to express it with.

ABAC (attribute-based access control) fixes that by evaluating attributes at request time instead of pre-baking roles. Department, clearance, resource classification, time, location. It gets you much finer control without the role count exploding, and the price you pay is that when a request gets denied, you have to trace which attribute failed instead of just reading a role name.

ReBAC (relationship-based access control) asks a third question: how is this subject related to this resource? You can edit the doc because you created it. You can see the profile because you manage that person. Google Drive works this way, and so does every app where ownership is the real rule.

Well, which one should you use? Honestly, most real systems end up with RBAC for the broad strokes and one of the other two layered on for the cases roles cannot express. Granularity comes from the rules you write, and not from the model you picked. You can write a terrible coarse ABAC policy. Plenty of people have.

Why AI made this urgent

I run Claude Code and GPT Codex daily, and both of them do things a normal API consumer never did. They read files and call tools. And they chain several model calls together off one instruction I typed half-awake.

That breaks the two assumptions most permission systems were built on: that whoever is asking is a person, and that every request is something a person actually asked for.

The numbers back this up. 1Password surveyed 1,000 security and engineering staff at large US firms in late May and early June 2026, and found that agents in production reached roughly twice as much data as had actually been approved. In the same survey, 33% of developers running agents reported a breach or security incident tied to an over-privileged non-human identity, and 40% said they leave agents holding persistent access to systems and secrets after the task is finished.

None of that is an AI-specific vulnerability. It is the same over-permissioning problem as always, running at a speed and volume that humans never generated.

Where granular access control breaks in practice

Everybody agrees with least privilege. Almost nobody has it. So it is worth being specific about where the plan falls apart, because the reason is usually the setup, and not laziness.

A provider key cannot be narrowed. Your OpenAI or Anthropic key is a single credential with your whole account behind it. There is no version of it that means "gpt-4o only, 50 dollars, no tool calls". So the moment more than one service needs model access, you either share one key and lose all attribution, or you mint several and lose all central control.

Enforcement lives in the wrong place. If the permission check is inside your application code, then every new service, notebook, cron job, and intern's side script has to re-implement it correctly. One of them will not. And the ones that skip it will not show up in any policy list, because they never registered with the policy system in the first place.

You cannot enforce what you cannot attribute. At Zonko Labs I built an internal tool that captured our AI product's data logs and generated reports on latency and probable slowdowns, and the thing that made it useful was that every log line could be traced back to a specific caller. Without that, a spend spike is just a number going up. You cannot tighten a permission when you do not know which caller needs loosening.

So the pattern behind all three is the same. Granular access control needs a place in the request path that sees every call, knows who made it, and can say no before the call leaves your network.

Where the gateway comes in

Let me be honest about where this actually helps first. If you are one person with one API key on one machine, you do not need any of this, and a gateway will not make your side project safer. It also cannot fix a permission model you have not thought about. A gateway enforces the rules you write, so if you write "allow everything", you get everything, faster.

But the moment there are several services, several models, and a few agents in the mix, the check has to sit in the traffic path. That is what an AI gateway is: a proxy that every model call goes through. Which makes it the one place you can check a permission once, instead of in every service.

Bifrost is the open-source one I keep pointing people at, partly because the whole thing sits in a public GitHub repo, so you can read exactly what it enforces rather than trusting a feature list.

The thing you hand out is a virtual key. Your real provider credentials sit inside the gateway, and each caller gets its own scoped key instead. And the things you can scope that key on are almost exactly the five dimensions above.

Take the rule in plain words first:

This key belongs to the support team. It can only use OpenAI, only the gpt-4o model, and it stops working on the first of next year.

Written out in Bifrost's governance config, that is:

{
  "id": "vk-support-bot",
  "is_active": true,
  "allow_all_providers": false,
  "expires_at": "2027-01-01T00:00:00Z",
  "provider_configs": [
    {
      "provider": "openai",
      "allowed_models": ["gpt-4o"],
      "blacklisted_models": []
    }
  ],
  "team_id": "team-support"
}

Reading that back: allow_all_providers set to false means any provider not listed here is denied, including ones added to the gateway later. allowed_models narrows dimension three to a single model. expires_at is dimension five, so the key dies on its own without anybody remembering to revoke it. And team_id attaches it to a team, which is where the budget lives.

A few more things worth knowing:

Spending and rate caps sit on the key itself. A budget is max_limit plus a reset_duration of 1m, 1h, 1d, 1w, 1M, 1Q or 1Y, and rate limits are separate token_max_limit and request_max_limit counters. Blowing through them returns real status codes rather than a generic failure: 402 for budget_exceeded, 429 for token_limited, 403 for model_blocked. That distinction sounds small until you are the one reading the error at 2 in the morning.

Tool access is deny-by-default. If you have wired up MCP servers, a virtual key with no MCP configuration gets no tools at all. Where tools are granted, they are listed explicitly in tools_to_execute, and the key's list acts as a ceiling that a request cannot widen. That is dimension three pushed all the way down to individual tools, which matters a lot given how much damage one file system tool can do inside an agent loop! (If MCP is new to you, I wrote a beginner's guide to MCP servers first.)

Seeing the dashboard is a separate grant from calling a model. Bifrost ships three system roles, Admin, Developer and Viewer, carrying 42, 27 and 14 permissions, and you can build custom ones by toggling resource and operation pairs. On top of that sits data access control, which decides which rows a user sees at all: own-data, team-data, or all-data. Being allowed to open the logs page and being allowed to see everybody's logs are two different grants.

Write the policy once, not once per person. Access profiles let you define one policy and auto-issue a per-user virtual key from it, each with its own budget counter. Edit the template, propagate, and every key follows. Keys can rotate on a schedule from 1h up to 365d. This is the part that decides whether any of it survives the team getting bigger. If every new engineer needs a hand-written policy, somebody will quietly hand out a shared key by month three.

Audit logs are signed. Administrative events can be HMAC-signed and exported as JSON, JSON Lines, or syslog in the RFC 5424 format that SIEM tools ingest. Knowing who did something and being able to prove it later are the same job, and homegrown setups usually only manage the first half.

Bifrost publishes its own benchmarks (roughly 20 microseconds of added latency at 5,000 requests per second), and those are vendor benchmarks run on a vendor harness, so measure your own. The point holds whichever gateway you use: the check belongs on the path, and not copy-pasted into fifteen codebases.

If governance at this layer is the actual problem you are solving, I also went through the tools in this space in more detail.

Where should you set the dial?

The floor is not RBAC, and it is not ABAC either. The floor is being able to answer one question about your own system right now:

If a credential leaked this minute, what exactly could someone do with it, and how much could they spend before anyone noticed?

If the honest answer is "everything, and I have no idea", the fix is not a bigger access control model. It is a first cut at all five dimensions, in this order: one identity per caller, then a spending cap on each one, then an expiry date, then a model or resource allowlist, then conditions. That order is deliberate, because knowing who called and capping what they spend buy you the most safety for the least work, and conditions buy you the least.

You will not get to per-row, per-tool, per-hour granularity, and you should not try. Nobody is running the theoretically correct permission model. The teams doing well are the ones who pushed the dial two notches past a shared key and then actually maintained it.

Anyway, I am curious where other people drew this line. If you have a permission setup that survived a team getting bigger, or one that collapsed into a shared key by month three, tell me in the comments, I want to hear the failure stories more than the success ones.

You can find me on X, or read the rest of my writing on my site.

Three Insights for Building Great Claude Skills

Swapnoneel Saha — Tue, 08 Sep 2026 18:32:06 +0000

Most Claude skills that never fire are not broken. Their description is.

That one fact is behind almost every "why won't my skill trigger" post you'll find. Three things separate a working skill from one that sits there ignored: a description written like a routing rule, a SKILL.md kept short on purpose, and pushing fragile steps into scripts instead of long instructions. Get those three right, and everything else gets much easier.

First, what a Claude skill actually is

A skill is a folder with a SKILL.md file in it. At the top of that file, you have a small YAML header with a name and a description, followed by instructions written in markdown. That's the whole concept. Anthropic launched skills in October 2025 and made the format an open standard in December, so Claude Code, the API, and claude.ai all use the exact same folder structure.

The interesting part is how skills load. Claude doesn't read your entire skill upfront. When it starts up, it reads only the name and description of every installed skill, costing around 100 tokens each. Claude loads the main instructions only when it decides your skill fits what you asked for. And it reads extra helper files only when those instructions specifically point to them. Anthropic calls this progressive disclosure. Two of the three insights below come down to working with this design instead of fighting it.

This means skills are cheap to keep around, costing you tokens only when they actually run. That single design choice shapes what makes a skill good or bad.

Insight 1: Your description is a routing rule, not a summary

Here is where people get confused. Your description isn't a friendly summary for humans browsing your repository. It is a routing rule Claude uses to pick your skill out of the pile.

Remember what Claude actually sees at startup: only the name and description of each skill. Nothing else. When you send a prompt, Claude decides whether to open your skill's full text based purely on that description. If the description is vague, Claude never reads the instructions. The best instructions in the world won't help if your skill never triggers.

That's why a skill that won't trigger is almost never a code bug. It just means your description doesn't explain when to trigger using the words you actually type.

A good description handles two jobs: what the skill does, and when Claude should run it. Write it in the third person (like "Reviews pull requests" instead of "I review" or "Review this"). Because Claude reads this inside its system prompt, mixing perspectives can confuse the model. You have up to 1,024 characters, so use that space for the exact phrases you would normally type.

---
name: reviewing-pull-requests
description: "Reviews pull request diffs against the team's checklist and flags risky changes. Use when the user asks to review a PR, review a diff, or check staged changes before merging."
---

Notice that the description names the triggers out loud: "review a PR", "review a diff", "check staged changes". Compare that to "Helps with code review", which competes with everything Claude already knows about code review and loses.

The name matters less than the description, but it has strict rules that catch people off guard. It must use only lowercase letters, numbers, and hyphens, capped at 64 characters. It cannot include the words "anthropic" or "claude". Anthropic recommends names ending in -ing (reviewing-pull-requests, analyzing-spreadsheets) to clearly show what action the skill takes. A name like claude-pr-helper breaks two rules at once, while a generic name like helper gives no useful information.

There is real data behind this. One developer ran 650 activation trials on the same skills. They found that passive descriptions triggered far less reliably than direct instructions starting with "Use this skill when." While this isn't an official benchmark from Anthropic, the takeaway matches both the official documentation and real-world use: name the specific situation where Claude should trigger, not just what it can do.

Also, keep your scope realistic. If it's too broad ("helps with frontend"), Claude will pick its own built-in knowledge instead. If it's too narrow ("fixes React 18 hydration errors in Next.js 14 app router"), it won't run if you phrase your prompt slightly differently. You are describing a general task, not a specific bug ticket.

Insight 2: Progressive disclosure is a token budget you spend

Once a skill triggers, its SKILL.md body loads into the same context window as your conversation, your files, and every other skill's metadata. Anthropic makes a great point in their authoring guide: the context window is shared space. Every token you waste in your instruction file is memory the model can't use for your actual work.

This means your main skill file should be smaller than you might expect. The official recommendation is to keep SKILL.md under 500 lines. Move anything longer into separate files that load only when Claude needs them. Loading happens in stages, and each stage has a different cost:

Level	When it loads	Rough cost
Name and description	Always, at startup	~100 tokens per skill
SKILL.md body	When the skill triggers	Under 5k tokens
Bundled files and scripts	Only when referenced	Nothing until read

That third row is the real secret. You can include a 2,000-line API guide, datasets, or code examples, and they cost zero tokens until Claude actually needs them. So the best approach is a lean SKILL.md that acts like a table of contents, pointing Claude to larger files when necessary:

reviewing-pull-requests/
  SKILL.md          # short: the workflow and when to read what
  checklist.md      # loaded only when reviewing
  examples.md       # loaded only when Claude wants a sample
  scripts/
    diff_stats.py   # executed, never loaded into context

The documentation highlights two practical rules here. First, keep linked files only one level deep and link them directly inside SKILL.md. If you bury files inside subfolders, Claude often previews just the first 100 lines and misses the rest. Second, don't explain concepts Claude already understands. Anthropic uses a straightforward example: a good instruction to extract PDF text is about 50 tokens of code. A bad one uses 150 tokens because it starts by explaining what a PDF is. Claude already knows what a PDF is!

Notice what I'm doing here, too. I'm not explaining what a token is or how context windows work. If you're building skills, you already understand those basics. That's the exact same mindset your skills need: only write what the model doesn't already know.

I care about this point a lot because of my work building ANRL. It's a compact data format created to cut token waste and context fragmentation, and it reduced formatting overhead by more than 40%. When you spend weeks fighting to save 40% on tokens, watching someone waste 3,000 tokens in SKILL.md re-explaining JSON is painful. Keeping your skills lean isn't just about clean formatting—it protects the model's working memory.

Insight 3: Let code do the deterministic work

The last one separates skills that feel reliable from skills that mostly work.

If a step is deterministic (parse this file, validate this schema, sort these rows), do not write instructions asking Claude to be careful. Write a script and tell Claude to run it. When Claude runs a script through bash, only the output comes back into context, not the code. So a bundled validate.py is cheaper than asking Claude to generate the same validation every time, and it does the exact same thing on every run instead of a slightly different thing.

Anthropic uses a great phrase for writing these helper scripts: "solve, don't defer." If a file might be missing, handle that check right inside the script. Don't let the script crash and force Claude to guess a fix, because improvised fixes break reliability. The same rule applies to unexplained settings: setting a 30-second timeout with an explanatory comment is much better than leaving a random number like 47 that no one can explain later.

The bigger idea is giving Claude freedom only when the task allows it. Anthropic's docs share a helpful comparison:
Some tasks are like a narrow bridge with steep drops on both sides, like running database migrations in a strict order. For those, give Claude a single script and tell it not to alter the command. Other tasks are like an open field with no dangers, like reviewing code where the best feedback depends on what Claude finds. For those, give general guidelines and trust the model to choose the right path. Making mistakes here hurts in both directions: adding too many rigid rules in an open field makes the skill inflexible, while giving too little direction on a narrow bridge leads to errors.

For bulk edits or risky actions, the documentation suggests a three-step pattern: plan, validate, and execute. Have Claude write its planned changes to a file first. Next, run a script that checks that plan for mistakes. Finally, apply the changes only if the check passes. It might sound like extra work, but it's the difference between catching an error in a safe file and catching it after it breaks your database.

When one skill becomes twenty

Those three insights help you build a solid individual skill. But developers rarely stop at just one. Once your team starts collecting dozens of skills, two quiet problems start showing up.

First, skills can connect to external tools through MCP (if you haven't used MCP yet, check out my beginner's guide to MCP servers). The easiest way to think about the difference: MCP provides the tool, while the skill teaches Claude how to use it for your specific workflow. But pay close attention to Anthropic's security warning: a skill can tell Claude to run tools in ways the author never mentioned. That's why they recommend checking every file inside any skill you didn't write yourself.

Second, it's best practice to test skills across different models like Haiku, Sonnet, and Opus. A skill is only as reliable as the model running it. In a team environment, that means your skills end up calling different models across different API providers.

Combine that with several teammates installing community skills from GitHub, and you aren't just managing simple markdown files anymore. You have unmonitored tool calls and model requests running across your team. That's an entirely different problem.

Do you need a gateway for your skills?

If you're running just two skills on your laptop, you don't. Seriously, skip this section and go write your skills instead of setting up infrastructure. A gateway won't make an individual skill better, and it won't fix a bad description.

A gateway starts to matter when you can no longer answer simple questions: Who is allowed to run which tool? Which skill triggered this request? Whose API budget is paying for it? When you hit that point, routing all that traffic through one place makes life much easier. That place is an AI gateway, and the one I keep returning to is Bifrost (by Maxim AI), an open-source AI gateway and control plane for LLMs, MCP tools, and agents (repo here).

Two of its core features line up directly with the scaling problems above.

To secure tool calls, Bifrost acts as an MCP gateway with deny-by-default tool filtering. Without explicit permissions, zero tools are accessible. You define what to unlock across client settings, request headers, and virtual key configurations. This means a skill could ask for a filesystem MCP server, but its key can restrict execution strictly to read_file, completely blocking destructive actions like delete_file. You don't have to depend on plain text instructions in a prompt to keep your systems safe. (It even has an MCP Code Mode that runs tool workflows inside a sandbox instead of dumping every tool schema into your context, cutting token usage by over 92% across large tool sets.)

To manage model access, Bifrost provides virtual keys and governance rules. Each key gets its own list of approved models (across 25+ providers and 10,000+ models), an automatic budget cap with reset periods, and custom rate limits. So when you want your team to test a skill across Haiku, Sonnet, and Opus, you give them one key with an allow-list and a spending cap instead of juggling five API keys in a spreadsheet. Provider fallback and retries are built in, so a skill doesn't fall over if one provider has an outage.

Performance is where this actually holds up. If a gateway is slow, developers route around it. Bifrost is written in Go with memory pooling and pre-warmed connections, keeping added gateway overhead to about 11 microseconds at 5,000 requests per second on a single t3.xlarge instance. In Maxim's published benchmarks against LiteLLM (tested with 500 concurrent users on AWS), Bifrost delivered 9.5x higher throughput (424 req/s vs 44.84 req/s) and roughly 50x lower P50 latency (804 ms vs 38.65 s). Those are vendor benchmarks, so treat them as such, but the main point holds: the gateway manages your traffic without becoming the slow part.

None of this is about writing an individual skill. It's just what helps once your skills work well and you start scaling up.

So what actually makes a Claude skill great?

It's not about making a giant SKILL.md file. It's much simpler than that.

Write your description like a clear routing rule in the third person, using the exact words you'd naturally type in a prompt. Keep the main instructions short and let extra reference files handle the heavy lifting, because context memory isn't free. And move fragile, repetitive tasks into code scripts so they run consistently every single time instead of guessing. Those three habits make the difference between a skill that runs reliably and one that sits forgotten in a folder.

Using an AI gateway matters as well, but only later on. Don't worry about it until you have more skills and tools than you can easily keep track of in your head. You'll know when you cross that line. Until then, setting up a gateway is just extra work you don't need.

If you've built skills that trigger reliably, I'd love to hear how you wrote your descriptions to make them work. That's still the area where most people spend the least effort. Come share your thoughts on X, or check out my other posts at my site.

How to Cut Time to First Token (TTFT) in LLM Apps

Swapnoneel Saha — Fri, 04 Sep 2026 15:49:58 +0000

Whenever you type something into an AI app like ChatGPT or Gemini, and hit enter, you have to stare at nothing for half a second, one second, and sometimes even more. Then the words start pouring out fast. The wait time that you always experience in between these two events is called time to first token, or TTFT, and it is the single thing your users actually feel.

Time to first token (TTFT) is the gap between sending a request and the first token coming back. To cut it, shorten and cache your prompt so the model reads less, stream the reply, keep connections warm, and route around slow providers. The biggest lever is prompt length, since most of that wait is the model reading your input.

Now let me show you where the time actually goes, and how we can fix this. So, let's begin!

What Time to First Token Actually Is

When you send a prompt, the model does two very different jobs.

First it reads everything you sent. Every token of your system prompt, your chat history, your retrieved documents, your question. It runs all of that through the network in one big forward pass and builds an internal memory of it. This step is called prefill, and nothing comes back to you while it runs.

Then it starts writing, one token at a time. That second phase is where you see text stream in.

TTFT covers the first phase plus everything around it: the network trip to the provider, any time your request spends waiting in a queue, and the prefill itself. Here is the part that matters. On any prompt that is not tiny, prefill dominates, and the more tokens you send, the more prefill work the model does before it can start. A short prompt starts answering quickly. A prompt stuffed with 8,000 tokens of context makes the reader sit and watch the cursor.

What Is a Good Time to First Token?

It depends on what the app is for.

The Modular LLM Inference Handbook puts it simply: a chatbot usually needs TTFT under 500 milliseconds to feel responsive, and a code-completion tool needs it below 100 milliseconds or it feels broken. A batch job that emails a summary once a day can take five seconds and nobody cares.

So before you optimize anything, decide which of these you are. A chat UI and a background report have completely different budgets, and chasing 100ms on a nightly job is wasted effort.

The Fixes, Biggest Wins First

Here is the order I actually work through. The early steps cost you almost nothing and pay back the most. The later ones matter, but only after the cheap wins are done. Do not skip to the bottom.

1. Measure TTFT on its own, before you touch anything

You cannot cut a number you are not looking at. And most teams are not looking at this one, because their logging records total response time and stops there. Total time hides TTFT completely, since a slow first token and a slow-typing model both show up as one big number.

So split them. Log the time from request-sent to first-token-received as its own field, separate from the time to finish. Do it at the p50 and the p99, because the average will lie to you and the tail is where users rage-quit.

This is not theory for me. At Zonko Labs I built the internal tool that captured the data logs for Luffy, an AI co-worker that lived in Slack, and turned them into reports on latency and probable slowdowns. The first thing that tool taught everyone was that our "the bot is slow" complaints were almost always a first-token problem, not a typing-speed problem. You only learn that if you measure the two apart.

If your traffic runs through an AI gateway, you can track this without building custom timers in your app. For example, Bifrost exports a metric called bifrost_stream_first_token_latency_seconds. It measures TTFT automatically across OpenAI, Anthropic, or any other provider, so you get clean p50 and p99 charts right out of the box.

2. Shorten the prompt

This is the biggest lever you fully control, and it is the one no tool can pull for you.

Prefill scales with input length, so every token you delete is prefill work the model never has to do. Look hard at what you are actually sending. Most apps ship a system prompt that grew over months and never got trimmed, a full chat history when the last four turns would do, and RAG retrieval that dumps ten documents when three answer the question.

Cut the system prompt to what the model needs, not what makes you feel safe. Truncate or summarize old history instead of resending all of it. Retrieve fewer, better chunks. None of this is glamorous, and all of it moves TTFT more than the fancy stuff.

Be honest about the tradeoff, though. Cutting context can cost you answer quality, so trim, then check your evals, then trim again. This is the one place where going too far actually hurts, so it earns the care.

3. Cache the stable part of your prompt

Most of what you send is the same every single call. The system prompt does not change. The tool definitions do not change. Only the user's question at the end changes. Prefill does not know that, so it re-reads the whole thing every time, unless you turn on caching.

There are two flavors, and they are not the same thing.

Prompt caching, also called prefix caching, is offered by the model providers themselves. It stores the processed form of a stable prefix so the next call reuses that work instead of redoing it, which is what prefix caching buys you. Put the parts that never change at the very front of your prompt and the changing question at the end, so the cache has the longest possible prefix to hit.

Semantic caching is the bigger swing. Instead of caching the prompt's prefill, it caches the whole answer, and serves it when a new question means the same thing as an old one. On a hit, you skip the model entirely, so TTFT collapses from "prefill a big prompt" to "look up a vector," which is milliseconds. I wrote a full explainer on semantic caching if you want the mechanics.

Let me make the caching point concrete with a small test you can predict.

Call A sends a 3,000-token system prompt that never changes, then a 40-token question at the end. Call B folds the same information into a fresh 3,000-token prompt that gets reworded on every request, then the same 40-token question. Both send roughly the same number of tokens. On the hundredth request, which one has the lower TTFT?

Call A wins, and it is not close. Call A's 3,000-token prefix is identical each time, so after the first call it lives in the prompt cache and its prefill is skipped. Call B looks different to the cache every time, so it pays the full prefill on all one hundred calls. Same token count, wildly different first-token time, decided entirely by whether the front of your prompt holds still. That is the whole game with caching: keep the front stable.

4. Stream the response, and know its one limit

Streaming sends each token to the user the instant it is generated, instead of waiting for the full answer. Turn it on. Every serious chat UI streams, and it is a one-line change in most SDKs.

Now the honest part, because it trips people up. Does streaming reduce TTFT? No. Streaming does not make the first token arrive any sooner, since prefill still has to finish first. What it changes is everything after that first token, and it changes how fast the whole thing feels. People read a streaming reply as faster than a non-streaming one even when the total time is identical, which is why faster first tokens matter more than total speed.

So streaming is a perception win, not a TTFT win, and the two stack. Streaming only pays off if TTFT is already short, because a three-second blank screen followed by a burst still reads as three seconds of nothing. Fix TTFT first, then stream to make the rest feel quick.

5. Keep the connection warm and close

Some of your TTFT is pure plumbing, and it hides where nobody looks.

Every fresh HTTPS connection pays for a TLS handshake before a single byte of your prompt moves. If your app opens a new connection per request, you are paying that tax every time. Reuse connections with keep-alive and connection pooling so the handshake happens once, not on every call.

Distance costs you too. A request from a server in Mumbai to a model endpoint in Virginia spends real milliseconds just crossing the planet and back, twice, before prefill even starts. Pick a provider region near your users where you can, and cut the number of hops between your app and the model. If you are on a serverless setup, watch for cold starts, since a function spinning up from zero can add more delay than the model does.

6. Route around slow and rate-limited providers

Here is the failure mode that ruins TTFT and has nothing to do with your prompt. The provider is having a bad minute.

When a provider is overloaded, your request sits in its queue, and queue time is part of TTFT. When you hit a rate limit, you get a 429 and your client backs off and retries, which stacks seconds onto the first token before it ever appears. Neither of these is fixed by shortening your prompt, because the model never even started.

The fix is to not be stuck with one provider. Send the request to the fastest healthy option, and when one starts failing or crawling, fall over to another automatically instead of retrying into the same wall. This is adaptive load balancing applied to model traffic, and it is the difference between one bad provider-minute taking down your p99 and your users never noticing.

Where a Gateway Does This For You

Let me be honest about what a gateway cannot do first.

A gateway cannot shorten your prompt. It cannot make a provider's GPU run faster. If your app only calls one model and your prompt is already lean, you do not need a gateway for TTFT. You will get much bigger gains just by trimming your system prompt.

Where an AI gateway helps is when you want to handle caching, retries, and monitoring across multiple services without rebuilding them in every backend repo. The gateway sits between your application and your model providers as a reverse proxy, so you wire those fixes once in your infrastructure instead of in your app code.

I have used Bifrost, an open-source AI gateway built in Go, and it handles these levers in one place:

First, it gives you TTFT metrics out of the box. Measuring first-token latency manually usually means writing custom streaming wrappers in Python or Node.js to timestamp the first chunk. Bifrost measures bifrost_stream_first_token_latency_seconds at the proxy layer for every request, giving you clean p50 and p99 charts across all your providers without extra client telemetry.

Second, it provides semantic caching. If a user asks a question that matches a previous one, Bifrost serves the answer straight from cache in about 10 to 15 milliseconds. That turns an 800-millisecond prefill wait into almost nothing. Plus, cache writes happen asynchronously in the background, so cache misses do not add any latency to your stream.

Third, it runs automatic fallbacks. When a provider is overloaded or returns a 429 rate limit, your request sits in a queue and your TTFT spikes. With Bifrost, you can set fallback rules: if your primary model hangs or errors, it instantly shifts the request to a healthy backup provider instead of leaving your user staring at an empty screen.

Finally, there is the question of proxy overhead. Adding a gateway puts an extra hop in your network path. If that hop adds 30 or 50 milliseconds, it defeats the entire purpose of optimizing latency. Bifrost is written in Go and adds around 20 microseconds of overhead under load, meaning the hop itself is practically invisible.

You still have to fix your prompts first. But once your prompts are clean, putting a gateway in front of your models lets you handle caching, fallbacks, and latency tracking through simple configuration instead of messy glue code.

Start Here

If you do nothing else from this whole post, do three things, in this order.

Measure TTFT as its own metric, so you stop flying blind. Shorten and cache your prompt, because that is where the biggest, cheapest wins live. Then stream, so the rest of the answer feels fast.

The truth most infrastructure posts bury is that for an app that calls an API, time to first token is mostly about prompt length and caching, not about exotic GPU tricks you will never touch. The GPU tricks are real, but they belong to the people running the models. Your levers are the prompt you send, the cache in front of it, and the route it takes. Those you own completely.

Start by logging your p50 and p99 first-token time this week. I would bet the number surprises you, and I would bet a fat system prompt is the reason.

If you have a TTFT horror story, or a fix that worked that I left out, tell me. I am @swapnoneel123 on X, and I write more of these on my site.

What Is an MCP Registry? Discovering & Governing MCP Servers

Swapnoneel Saha — Thu, 03 Sep 2026 17:01:44 +0000

Most people consider an MCP registry as "npm for MCP servers." It is a nice line to explain the concept, but it is partially true and extremely half-baked.

An MCP registry is a searchable catalog of MCP servers. It stores metadata, like a server's name, where to install it, and how a client should connect, so an AI agent or a developer can find servers to plug in. But here is the part the npm comparison hides: a registry is a discovery layer, not a trust layer. It tells you a server exists. It does not tell you the server is safe, or that your agents should be allowed to run it.

That gap between "I found it" and "I trust it" is where teams get burned. Discovering MCP servers and governing MCP servers are two different jobs, and a registry only does the first one. So let's get both straight.

If you want to know more about what an MCP server is and how it works, I wrote a beginner's guide to MCP servers first. This post picks up where that one left off.

What an MCP Registry Actually Is

An MCP registry is a metadata catalog. It does not hold the server's code. It holds a small record that describes the server and points to wherever the code actually lives.

This trips people up, so let me separate the two clearly. Your code lives on a package registry: npm, PyPI, Docker Hub. That is where the actual bits get downloaded from. The MCP registry sits one level up and just stores a pointer. For example, a weather-mcp package can live on npm, and the MCP registry entry maps "weather v1.2.0" to npm:weather-mcp. The registry is the index card. npm is the shelf.

Each entry follows a standard shape called server.json. It records the server's unique name, where to find it (an npm package name, or a remote server URL), how to run it (command-line args, environment variables), and some discovery data like a description and what the server can do. That is it. A registry is a stack of these server.json cards, plus an API to search them.

And that is genuinely useful. Before any of this, finding an MCP server meant scrolling GitHub READMEs and random Discord links. A registry turns that into one queryable list. But notice what a stack of index cards can and cannot do for you, because that is where the trouble starts.

The Official MCP Registry

The one most people mean by "the MCP registry" is the official MCP Registry, which launched in preview on 8 September 2025. It is open source, and it is backed by a serious group: Anthropic, GitHub, PulseMCP, Microsoft, and others.

Its job is deliberately narrow. In the maintainers' own framing, it is a centralized metadata repository, and the metadata it stores is intentionally unopinionated. It does not rank servers, and it does not review or rate them. It just holds the server.json records and serves them over a REST API.

Here is the design choice that matters most, and almost nobody mentions it. The official registry is not really built for your app to read directly. It expects to be consumed by "downstream aggregators", which pull its data on a schedule (say, once an hour) and then layer their own curation, ratings, or search on top. The registry is the wholesale source of truth. The nice front-ends you actually browse are meant to be built by other people.

And in practice, as of 2026, most MCP clients still do not read the registry directly as a built-in source. So for now you usually meet it through one of those aggregators, not through your agent going and querying it live.

Two more things worth knowing early. First, the official registry does not host private servers, so an internal mcp.acme-corp.internal server has no place there. Second, it is in preview, which means data can reset and the shape can still change. So treat it as a fast-moving foundation, not a finished product.

MCP Registry vs Marketplace vs Gateway

This is the question I see confused most, so let me draw the lines hard. These are three different things doing three different jobs.

A registry is discovery in its rawest form. It answers one question: what servers exist, and how do I connect to them? It is a passive list.

A marketplace is discovery made pleasant. Sites like Smithery, Glama, and mcp.so give you search, categories, install buttons, and community ratings. Increasingly they read from the official registry underneath and add their own polish on top. A marketplace is a registry with a nice storefront and an opinion.

A gateway is something else entirely. It is not about finding servers, it is about controlling them at runtime. A gateway sits in the traffic path between your agents and your MCP servers, and it enforces the rules every time a call is made: who is allowed to use this tool, how often, at what cost, with what data. A registry is a directory. A gateway is a checkpoint.

The one line to remember: a registry without a gateway gives you discovery but no runtime governance. Your agents can find tools, but nothing stops an unauthorized call once they connect. Hold onto that, because it is the seam the rest of this post walks through.

How Discovery Actually Works

So how does a client go from "I need a GitHub server" to a running connection? Through the registry API and the server.json record, and the interesting part is how it proves a server is who it claims to be.

Server names use a reverse-DNS format: io.github.username/server-name, or com.example/server. That name is not decorative. It ties the server to a namespace that the publisher had to prove they own, through GitHub or DNS verification. If you publish under io.github.swpn0neel/*, the registry made you show you actually control that GitHub account first. So a name is a small ownership claim, verified at publish time.

Here is the piece that makes the whole thing extensible. The registry ships an open OpenAPI spec, which means anyone can stand up their own registry that speaks the exact same interface. A public marketplace can implement it. A company can implement a private one for internal servers. Any MCP client that already knows how to talk to the official registry can talk to those too, with no extra work. One shape, many registries.

That design is genuinely clever, and it is also exactly why governance cannot live at this layer. A standard that lets anyone publish, and lets anyone run a compatible registry, is a standard optimized for reach, not for safety. Openness and vetting pull in opposite directions, and the registry picked openness on purpose.

Discovery Is the Easy Half

So that is discovery, more or less solved. You have a standard catalog, verified namespaces, an API, and a way to run your own copy.

But discovery was always the easy half of the title. The hard half is governing what those servers actually do once your agents start calling them, and that is a problem a catalog structurally cannot touch. Let me show you why with one concrete example.

What a Registry Does Not Protect You From

Say you tell your coding agent, "add the Slack MCP server." It searches, finds a package named slack-mcp-server, sees an install command that looks completely official, and wires it in. Feels routine. Now walk that one action through each layer and watch where it does, and does not, get stopped.

Did the registry catch it? No, and this is by design. The official registry delegates security scanning to the underlying package registries and to downstream aggregators. Its own job is namespace authentication and metadata hosting. So namespace verification proves the publisher owns the name they used. It does not prove the code behind that name is safe. A registry entry is a verified business card, not a background check.

So typosquatting still works. This is not hypothetical. Security researchers have already documented look-alike PyPI packages like slack-mcp-server and slack-mcp-server-v2, whose install commands look official but are not. A name that reads as legitimate is not the same as a source you vetted, and a catalog optimized for discovery will happily list both.

And the tool itself can lie. The nastier class is tool poisoning: a malicious server hides instructions inside a tool's description, so when the model reads "what can you do?", it also reads "and quietly send recent messages to this URL." The model sees a normal tool. The registry saw a normal server.json. Neither one inspected behavior, because inspecting behavior is not what either one does.

Then there is the rug pull. A server can be perfectly clean the day it is approved and turn malicious three versions later. A registry records the new version. It does not re-judge it. This exact risk is why security researchers keep arguing that tool updates should trigger re-review instead of being trusted automatically.

None of this means the registry is broken. It means the registry is doing its one job, discovery, and quietly handing you a job it was never built to do: deciding what to trust, and enforcing that decision on every call. That job has to live somewhere else.

Do You Need a Private MCP Registry?

For a lot of teams, the first real answer to the trust problem is a private registry. And it genuinely helps, so let me be fair to it before I tell you where it stops.

A private, internal registry is an allowlist. Instead of letting agents pull from the open catalog, your platform team curates a list of vetted servers, mixes in your own internal ones, and points every agent at that. Because it implements the same OpenAPI shape, your existing clients keep working. This is the model enterprise registry products (Kong's, Portkey's, and others) are built around, and for regulated or multi-team setups it is close to mandatory, because a community catalog was never meant to carry your compliance requirements.

So a private registry moves the trust decision to a place you control. That is a real upgrade over pulling from the open internet.

But look closely at when it acts. A private registry governs discovery time: what a developer or agent is allowed to install and see. It does not govern runtime: what an approved server actually does on each individual call. Once an agent is connected to a vetted server, the registry is out of the loop. It cannot cap how much that agent spends, cannot say "this key may read files but never delete them," and cannot stop a poisoned tool description that slipped through your review. It curated the shelf. It is not standing at the counter.

You still need something in the traffic path.

Where the Gateway Comes In

Here is the honest part first, because it decides whether you should even read this section. If you are running one or two MCP servers you trust, you do not need a gateway. A registry, or even a hardcoded config, is plenty. A gateway earns its place when you have many servers, many agents, and real consequences if one of them misbehaves. And a gateway cannot vet server code for you either, so nothing here removes the need to review what you run.

For a few months I was the first layer of testing for an AI co-worker that lived inside Slack, catching its mistakes before the real users did. The lesson that stuck with me is simple: you cannot govern what you only hear about after the fact. Governing has to happen at runtime, in the path, while the call is being made.

What a gateway adds is the one thing a registry structurally cannot: enforcement on every call, at runtime. That is where Bifrost, an open-source AI gateway built by Maxim AI, fits the picture. It can act as an MCP gateway: you connect all your MCP servers to Bifrost, and your agents connect to Bifrost instead of to each server directly. One endpoint in front of everything, sitting exactly where the checks need to happen.

Once you are in that position, three things change.

First, access stops being all-or-nothing. Bifrost filters tools per virtual key, so a given key only sees the tools you explicitly listed for it. You configure it plainly: a client named filesystem, with tools_to_execute set to ["read_file", "list_directory"], and that key can do exactly those two things and nothing else. It is deny-by-default. The read-only agent literally cannot see a delete tool, so a poisoned or over-eager call has nothing to grab.

Second, the token problem gets solved instead of tolerated. Remember how a registry lets you connect dozens of servers? Once you do, every request stuffs every tool's description into the model's context, and with 150-plus tools the model burns most of its budget just reading the menu. Bifrost's Code Mode fixes this by exposing four generic meta-tools and letting the model write code to discover and run what it needs on demand, instead of pre-loading every definition. In their own benchmark of 508 tools across 16 servers, that cut input tokens by up to 92.8%, from 75.1M tokens down to 5.4M, while keeping a perfect pass rate. That is a vendor benchmark on a vendor harness, so weigh it as one, but the mechanism is sound and the direction is not subtle.

Third, control lives in one place. All your keys and connections sit behind the gateway instead of scattered across config files, and you can attach budgets and rate limits to each virtual key, so an agent that goes haywire hits a spending cap instead of your invoice. This is the runtime governance the registry handed off. The registry told your agents what exists. The gateway decides what is allowed, every single time.

So the two are not competitors. A registry and a gateway are the two halves the title promised, and they sit at different points in the flow: the registry at discovery, the gateway at runtime.

What Do You Actually Need?

Let me put it as plainly as I can, because the whole post comes down to one distinction.

A registry answers "what exists?" A gateway answers "what is allowed?" You will eventually want both, and the mistake I see most is assuming that a shiny catalog, public or private, has solved the second question. It has not. It was never trying to.

So here is the path I would actually take. If you are just exploring, use the official MCP registry or a marketplace to find good servers, and keep the count small. If you are a team standing this up for real, add a private registry so you control what gets discovered in the first place. And the moment agents are making real calls with real credentials and real budgets, put a gateway in the traffic path, because that is the only layer that can enforce a decision at the moment the decision matters.

Discovery is a catalog problem. Governance is a runtime problem. Solve them in the right places and MCP stops feeling scary. Try to solve governance with a directory, and you will find out the hard way that an index card never stopped anybody from doing anything.

I run coding agents like Claude Code every day, and my own MCP config has quietly grown from one server to a small pile without me really deciding to. That drift is exactly how this bites you, one harmless-looking addition at a time. So if you have hit the same sprawl, I would genuinely like to hear how you are handling it. Drop a comment, or come find me on X and let's compare setups.

What Is an MCP Server? A Beginner's Guide

Swapnoneel Saha — Wed, 02 Sep 2026 18:48:39 +0000

An MCP server is just a small program that gives an AI model access to tools and data it cannot reach on its own.

In plain words, it is a bridge. Through one shared standard called the Model Context Protocol, it lets an AI read your files, query a database, or call an API without you having to write custom glue code every single time. It can be a tiny script running on your laptop, or a service running in the cloud.

None of this is as complicated as the acronym makes it sound. And it is worth getting right, because MCP is quickly becoming the default way AI agents interact with the real world. So, let's dive in!

What Is an MCP Server

An MCP server is a program that exposes three things to an AI model: tools it can run, data it can read, and prompt templates it can reuse. Simply put, the model connects to the server, asks "what have you got?", and then uses whatever comes back. Strip away the jargon and an MCP server is a standard adapter between one AI and one outside system.

The "server" word trips people up, so let me kill that confusion first. It does not mean a rack in a data center. A filesystem MCP server can be a tiny script on your own laptop. It is called a server only because it serves requests, the same way your local dev server does when you run npm run dev. Small program, answers questions, done.

And that is really the core of what people mean by MCP server meaning: it is the thing on the other end of the connection that knows how to do actual work, like reading a file or querying a database, and knows how to describe that work in a format the model understands.

Where MCP Came From and Why It Caught On

The Model Context Protocol is an open standard for connecting AI models to outside tools and data. Anthropic released it in November 2024, and within a year it went from a niche idea to something Claude, ChatGPT, Cursor, and VS Code all speak.

Here is the problem it fixed. Before MCP, if you wanted an AI to use some tool, you wrote the glue yourself, every single time. A couple of years back at Keploy I built a RAG chatbot over their docs, wiring the vector embeddings and the retrieval together by hand. That was one data source and one integration, fully custom. Now picture doing that again for GitHub, then Slack, then your database, then for a different model that expects a different shape. It does not scale, and everyone was rebuilding the same connectors.

The official docs describe MCP as a USB-C port for AI. One connector shape, and anything can plug into anything. You build an MCP server once, and every client that speaks the protocol can use it. That is the entire reason it caught on so fast: it turned a pile of one-off integrations into one standard everyone could share.

How Does an MCP Server Actually Work?

An MCP server works through three roles: a host, a client, and a server. The host is the app you are actually using, like Claude Desktop or Cursor. For each server it wants to talk to, the host spins up one client, and that client keeps a dedicated line open to one MCP server. Messages go back and forth as JSON-RPC 2.0, which is a plain, boring, well-understood format, and that is a good thing.

The server can offer three kinds of thing, and it is worth knowing them by name:

Tools: actions the model can run, like "read this file", "run this query", or "create this GitHub issue".
Resources: data the model can read for context, like a file's contents or a database schema. Read-only, no side effects.
Prompts: ready-made templates for common tasks, so the model does not start from scratch every time.

The flow itself is simple. The client connects, then asks the server to list what it has (a tools/list call), then runs one of them (a tools/call). In JSON it looks about like this:

// "What can you do?"
{ "jsonrpc": "2.0", "id": 1, "method": "tools/list" }

// "Okay, do this one."
{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "get_weather",
    "arguments": { "city": "Kolkata" }
  }
}

One more thing worth knowing early: a server runs in one of two places. A local server talks over stdio (standard input and output), which is just two programs on the same machine passing text back and forth, no network involved. A remote server talks over HTTP, so it can live anywhere and serve lots of clients at once. Same protocol either way, which is the point.

MCP Server vs API: What's the Difference

This is the question I get most, so let me be direct. An MCP server usually sits on top of a normal API. It is not a replacement for one. The difference is who the thing is built for.

A regular API is built for a developer who already read the docs and hardcoded the exact calls. It assumes someone knows the endpoints ahead of time. An MCP server is built for a model that shows up knowing nothing, asks what is available, and figures out which tool to use on its own. That runtime discovery is the whole trick, and a plain API does not do it.

So when people ask about MCP vs API, the honest answer is that most MCP servers are a thin wrapper around an API you could have called yourself. What MCP adds is a standard way to describe the tools, so any model can find them and use them without a human wiring each call. The API does the work. The MCP server makes that work legible to an AI.

What Can You Actually Do With One

Plenty, and you probably already have the pieces. The most common MCP servers connect a model to everyday systems: a filesystem server for reading and writing files, a GitHub server for issues and pull requests, a Postgres server for querying a database, a Slack server for messages. Anthropic keeps a set of reference servers on GitHub, and most companies now ship an official one for their own product.

I run Claude Code every day, and that is where most people meet their first MCP server. You bolt one onto a coding agent or a desktop chat app, and suddenly it can touch your files, search the web, or open a pull request without you copy-pasting anything. The MCP tools it offers just show up as things the model can now do.

The pattern is always the same. Find a server for the system you care about, point your client at it, and the model gains a new skill. One server is genuinely a five-minute setup, and it feels a little bit like magic the first time an agent edits a real file on your disk!

Do You Need an MCP Gateway

Here is the honest part first: if you are running one or two MCP servers, you do not need a gateway. It solves a problem you do not have yet, so skip this section and go play with a server instead.

But things change once the count goes up. Say you have got ten servers wired into your agents. Three problems show up fast. First, every request dumps the full tool list of every server into the model's context, so with 150-plus tools the model burns most of its budget just reading the menu. Second, each server has its own keys and auth, scattered across config files with no single place to rotate them. And third, there is no shared way to say who is allowed to call what, or to cap spending. That is tool sprawl, and it turns a neat setup into a liability.

An MCP gateway fixes this by sitting right in the middle. Think of it like a power strip. Instead of plugging ten different cables into ten wall sockets, your AI agent connects to the gateway once. The gateway handles the rest.

This is where Bifrost, an open-source AI gateway built by Maxim AI, comes in. It can act as an MCP gateway, tying all your servers together so your AI talks to one neat endpoint instead of juggling ten.

It also fixes that token problem in a really clever way. Remember how 100 tools eat up your context? Normally, the model has to re-read every single tool's description on every prompt just to know what's on the menu. That burns tokens fast.

Bifrost fixes this using Code Mode. Instead of dumping every tool definition into the prompt, it gives the model four generic tools and lets it write code to find and run what it needs on demand. In their benchmarks, that cut input tokens by up to 92.8%. That is not a small difference, that is basically your whole bill! Very similar to the cut-costs-at-the-gateway idea I talked about with semantic caching.

On top of that, you get one central place for safety and control. All your API keys live in one spot instead of being scattered across random config files. You can set per-key tool filtering and budgets, meaning you decide exactly which agent can use which tools and how much it can spend. And by default, tools do not run automatically unless you explicitly turn that on. That last part matters a lot, because an agent running tools silently without your permission is a bad day waiting to happen.

Where to Start

You do not need to read a massive spec or write complex code to get started with MCP. The whole idea is small once the acronym stops scaring you: it is just a standard plug that lets your AI reach outside tools.

If you want to try it out today, do not overthink it. Pick an app you already use, like Claude Desktop, Cursor, or Claude Code. Then grab just one pre-built server, like the filesystem or GitHub server. Wiring it into your config file takes about five minutes.

Once it is running, give the model a real prompt: ask it to inspect a local folder or check a repo. The moment you see an agent reach outside its chat box and touch a real file on your machine, the whole protocol clicks.

And my advice would be to keep it tiny. You do not need a gateway or fancy routing when you are only running one or two servers. Play with one first, see how it feels, and only put a gateway like Bifrost in front when you have a dozen servers and your token bill starts climbing.

If you set up your first MCP server after reading this, I would genuinely like to hear how it went. Drop it in the comments, or come say hi on X and we can chat about it.

Frequently Asked Questions (FAQs)

What is an MCP server in simple terms? It is a small program that gives an AI model a set of tools and data through a shared standard, so the model can read a file, query a database, or call an API without custom glue code. Think of it as a standard adapter between one AI and one outside system.

What is the difference between an MCP client and an MCP server? They are the two ends of one connection. The server offers tools, data, and prompts. The client, created by an app like Claude Desktop, connects to the server and uses what it offers. One app can run many clients, one per server it talks to.

Are MCP servers safe to use? They are as safe as how you run them. A single local server you trust is low risk. The danger shows up when you run many unvetted servers with their own credentials and let an agent run tools automatically, which is exactly why keeping auto-execution off and routing servers through a gateway with tool filtering is worth doing once you scale.

How do you build your own MCP server? You use one of the official SDKs, define a few tools with their inputs, and expose them over stdio or HTTP. Start tiny, with one or two tools, get it working with a client you already use, and grow from there instead of trying to wrap an entire API on day one.

Fixing Claude Rate Limit Errors with an AI Gateway

Swapnoneel Saha — Wed, 26 Aug 2026 18:04:49 +0000

Ever had a 429 from Claude that just would not clear, no matter how patiently you backed off?

Yeah. That one was never a rate limit.

A Claude rate limit error is an HTTP 429 carrying "type": "rate_limit_error", and it means your organization crossed a per-minute request ceiling, a per-minute token ceiling, or its monthly spend cap. Which one decides the fix, and the response body tells you which before you write any retry code.

And the difference matters far more than it sounds, because two of those three conditions get strictly worse the moment you retry them.

The five failures behind one Claude rate limit error

rate_limit_error is not one condition. Anthropic returns the same string for several different situations, and the correct response may be to wait a few seconds or to change a setting that retrying cannot fix.

The Claude API errors reference documents them separately, so you can tell them apart from the response alone.

What you see	What it actually is	Does retrying help?
`429` + `rate_limit_error` + a `retry-after` header	A real per-minute limit. You went too fast.	Yes, after `retry-after` seconds
`429` + `rate_limit_error` + `details.error_code: enforced_spend_limit_reached`, and no `retry-after`	Your usage tier's monthly spend cap	No. Access returns at 00:00 UTC on the 1st
`400` + `invalid_request_error`, message starting `You have reached your specified API usage limits`	A spend limit you set yourself in the Console	No. Raise or remove your own limit
`429` right after a sharp traffic ramp	An acceleration limit, not your steady-state limit	Partly. Ramp up gradually instead
`529` + `overloaded_error`	Anthropic is busy. Nothing to do with you.	Yes, but the queue is global

That last row is the one that trips people up most. The Claude overloaded error gets searched more than any specific 429 phrasing, and almost every guide lumps it in with rate limits. It isn't yours. A 529 means the API is temporarily overloaded across all users, so no amount of tier upgrading or key rotation on your side moves it.

The second row is easy to misdiagnose. It looks identical to a normal rate limit at the status-code level, the official SDKs will happily auto-retry it, and every retry fails. The tell is the missing retry-after header plus that error_code.

So before anything else, log the full error body. Not err.status. The body.

import anthropic

client = anthropic.Anthropic()

try:
    msg = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": "hi"}],
    )
except anthropic.APIStatusError as e:
    print(e.status_code)                          # 429
    print(e.response.headers.get("retry-after"))  # None means spend cap
    print(e.body)                                 # the error_code lives here
    print(e.request_id)                           # req_018Ee... for support

The retry-after check on that second line is doing almost all of the diagnostic work. Present means wait. Absent on a 429 means stop retrying and go look at billing.

What actually counts against your Claude rate limits

Now, the per-minute limits. There are three of them running at once, per model, and you can trip any one.

RPM, requests per minute.
ITPM, input tokens per minute.
OTPM, output tokens per minute.

They are enforced per model class, so Sonnet traffic and Haiku traffic draw from separate buckets and can run at full speed at the same time. Worth knowing: on the current lineup, Sonnet 5 has its own bucket while Sonnet 4.6 and 4.5 share a combined one, and the same split applies to Opus 5 versus the Opus 4.x family. Pinning half your traffic to an older Sonnet does not double your Sonnet 5 headroom.

The rate limits documentation also says something that should change how you think about bursts. The limiter is a token bucket, which means capacity refills continuously instead of resetting on a clean minute boundary. A 60 RPM limit behaves much more like 1 request per second than like 60 free requests at the top of every minute. So if you fan out 60 concurrent calls at 12:00:00 you will get rate limited, even though your average for that minute was exactly at the ceiling.

I hit this exact shape of problem building Scholarian, which fetched and ranked over 10,000 academic papers. The moment you write a loop that fans out one model call per document, your average throughput looks perfectly fine on a dashboard and your p99 latency (the slowest 1% of calls) is a wall of 429s, because nothing in that loop is pacing itself.

The one number most people get wrong

For most Claude models, only uncached input tokens count against ITPM.

Specifically: input_tokens and cache_creation_input_tokens count, and cache_read_input_tokens does not. So a cache hit is free as far as your rate limit is concerned, and it is billed at a reduced rate on top of that. (Claude Haiku 3.5 is the exception and does count cache reads.)

Do the arithmetic on that, because it is not a small effect. Say you are on the Start tier with 2,000,000 ITPM on Sonnet 5, and you are running a coding agent that ships roughly 180,000 tokens of context per turn.

Without caching, 2,000,000 divided by 180,000 is about 11 turns per minute for your entire organization.

With prompt caching where 170,000 of those tokens are cache reads, only 10,000 counts against ITPM. That is 200 turns per minute off the exact same limit! Anthropic's own worked example frames it the same way, and it is why they publish your cache hit rate on the Usage page right next to the rate limit charts.

One nuance people miss: cache_creation_input_tokens does count. Writing the cache costs you full ITPM once, and reading it is free. So a workload that keeps rebuilding the cache because the prefix keeps shifting gets the worst of both.

Output is simpler. OTPM is measured on tokens actually produced, in real time, and max_tokens never factors into it. There is genuinely no rate limit penalty for setting a generous max_tokens, so stop shaving it down for that reason.

The headers you should be reading instead of guessing

Every response, including the successful ones, carries your current standing. And most people never look at them.

curl -sS -D - -o /dev/null https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","max_tokens":16,
       "messages":[{"role":"user","content":"hi"}]}'

Run that and you get back a block of anthropic-ratelimit-* headers. The ones worth wiring into a metric:

anthropic-ratelimit-requests-remaining
anthropic-ratelimit-input-tokens-remaining
anthropic-ratelimit-output-tokens-remaining
the matching -reset fields, which are RFC 3339 timestamps rather than durations

Two gotchas here. The remaining values are rounded to the nearest thousand, so treat them as a gauge and not as an accountant. And the generic anthropic-ratelimit-tokens-* triplet reports whichever limit is currently most restrictive, which means the number can jump between input and output accounting between requests without anything being wrong.

Identify the failure and read the remaining-capacity headers before changing retry logic. Backoff cannot fix a spend cap, and tuning it first wastes time.

At one of my previous roles I built an internal tool that pulled the data logs off an AI product and turned them into latency and slowdown reports, and the useful part was never the clever part. It was just having the numbers somewhere you could look at them without reproducing the bug first. Same idea here. Export those three headers as gauges and most rate limit debugging stops being detective work.

Retrying correctly, and where retrying stops helping

The rule in plain English: if the response gave you a retry-after, wait exactly that long. If it did not, double your wait each attempt and add randomness so your workers do not all wake up at the same instant.

import random, time, anthropic

client = anthropic.Anthropic(max_retries=0)  # we handle it ourselves

def call_with_retry(**kwargs):
    for attempt in range(5):
        try:
            return client.messages.create(**kwargs)
        except anthropic.APIStatusError as e:
            if e.status_code not in (429, 500, 502, 503, 504, 529):
                raise
            hinted = e.response.headers.get("retry-after")
            if e.status_code == 429 and hinted is None:
                raise  # spend cap, so retrying is pointless
            wait = float(hinted) if hinted else min(2 ** attempt, 30)
            time.sleep(wait * random.uniform(0.8, 1.2))
    raise RuntimeError("exhausted retries")

The jitter multiplier on the last line is the bit people skip, and it is the bit that matters at concurrency. Without it, 50 workers that all got rate limited at the same moment will all retry at the same moment, and you get a thundering herd that reproduces the original 429 on a fixed schedule forever.

Also notice max_retries=0 at the top. The official SDKs retry twice by default with their own backoff, honoring retry-after. That default is fine for a script. It is not fine when you are also running your own retry loop, because the two layers multiply and you quietly end up with 3x the attempts you thought you configured.

But here is the honest limit of all of this, and it is why the post does not end here.

Retries are a queueing strategy. They smooth out bursts against a ceiling you are near. They do nothing at all when you are structurally above the ceiling, because every retry is just another request competing for the same bucket. If your steady-state demand is 3,000,000 input tokens per minute and your limit is 2,000,000, no backoff curve in the world fixes that. You need more ceiling.

Setup is done, now the part that actually raises the ceiling

Everything above happens inside one API key against one endpoint. Reading the error properly and then caching aggressively will genuinely resolve most Claude rate limit errors, and if you are a solo developer on one key, honestly stop reading here and go turn on prompt caching. You will get more out of that one afternoon than out of any infrastructure.

But if you are past that point, the interesting fact is that "Claude capacity" is not one pool. It is four, and they are metered by different organizations running different quota systems.

Four separate places Claude capacity lives

1. The direct Claude API, by usage tier. Start, Build, Scale, then Custom. Each tier carries both per-minute limits and a monthly spend cap, and as of mid-2026 those caps sit at 500USD on Start, 1,000USD on Build, and 200,000USD on Scale. Tiers move up automatically as you build usage history, and you can also request an increase from the Console.

2. Amazon Bedrock. Claude on Bedrock runs against AWS service quotas, which are per-account and per-region and adjustable through the Service Quotas console. These have no relationship to your Anthropic tier at all. Bedrock even splits its own quotas across two inference endpoints, tracked separately for the same underlying model.

3. Google's platform. Claude models run there through the model garden against Google Cloud quotas, again on a completely separate meter.

4. Priority Tier. Anthropic sells committed capacity with its own anthropic-priority-* headers and its own bucket, sitting alongside your standard limits rather than replacing them.

The consequence is the interesting part. An organization pinned at 100% of its direct-API ITPM might have an entirely idle Bedrock quota sitting in two regions. The capacity exists, and it is already paid for. The problem is that your application has one base URL and one key, so it cannot reach any of it.

One clarification before moving on, because this is the most common mix-up of all. If your error reads API Error: rate limit reached inside Claude Code on a Pro or Max plan, none of this section applies to you, because that is a subscription meter and not the API. Two different systems, two different clocks, and I wrote about Claude Pro usage limits separately.

Where a gateway changes the math

This is the job an AI gateway does, which you will also see called an LLM gateway or an LLM proxy. It is a proxy that speaks every provider's API shape, sits between your code and the model, and holds the routing decisions your application shouldn't have to.

Let me lead with what it cannot do, because that part gets oversold constantly. A gateway does not raise Anthropic's ceiling. Your Start tier is still your Start tier. If you run one Anthropic key through a gateway and change nothing else, you will hit exactly the same 429 at exactly the same token count, plus a few microseconds of hop.

What it changes is how many buckets a single request can reach, and how fast it gives up on a bad one. Bifrost is Maxim AI's high-performance, open-source AI gateway that unifies access to 20+ providers through a single OpenAI-compatible API. Its retry and fallback layer distinguishes upstream failures from credential failures: it retries 5xx and network errors against the same key, rotates keys for 429, 401, 402, and 403 responses, and moves to the next provider only after the current provider's retry budget is exhausted. That behavior maps directly to the different Claude failure modes in this post.

Pooling keys so one bucket is not the whole story

Rate limits are set at the organization level, so a second organization with its own key is a second bucket. A gateway can treat several keys as one logical pool.

{
  "providers": {
    "anthropic": {
      "keys": [
        { "name": "primary",  "value": "env.ANTHROPIC_API_KEY",   "models": ["*"], "weight": 0.7 },
        { "name": "overflow", "value": "env.ANTHROPIC_API_KEY_2", "models": ["*"], "weight": 0.3 }
      ]
    }
  }
}

Traffic splits 70/30 by weight. But the part that matters for 429s is what happens on failure. Bifrost rotates keys when the failure is bound to the credential rather than to the request, which it defines as 429, 401, 403 and 402. A rate limited key gets marked used for that cycle and the request carries on against another key. Auth and billing failures mark the key dead for the remainder of that request. Once every key has been tried, the set resets and a fresh weighted round begins.

Key selection is scored rather than round robin, using recent error rates, latency, and observed rate limit hits, with weights recomputed on a short interval and penalties decaying once a key recovers. That is adaptive load balancing applied to credentials instead of servers, and it matters here because a key that just returned a 429 is a bad destination for the next 30 seconds specifically, and not permanently.

Falling back across channels, not just keys

Key pooling only helps if you have more keys. Falling back across the four capacity pools helps even when you don't.

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{ "role": "user", "content": "..." }],
  "fallbacks": [
    "bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0",
    "vertex/claude-sonnet-4-5@20250929"
  ]
}

Same request, same Anthropic model family, three completely independent quota systems. If the direct API is capped, the call lands on Bedrock's per-region quota, which knows nothing about your Anthropic tier.

The execution detail worth knowing is that each provider in the chain gets its own full retry budget. A primary at max_retries: 3 with two fallbacks also at 3 means up to 12 attempts on one call, so set your client timeout with that in mind. Retries and fallbacks documents the backoff as min(initial × 2^attempt, max) × jitter(0.8-1.2), with defaults of 500ms initial and a 5000ms cap.

And a real gotcha, stated plainly because it will bite you: max_retries defaults to 0. Retries are off until you turn them on.

{ "network_config": { "max_retries": 3, "retry_backoff_initial": 500, "retry_backoff_max": 5000 } }

Two more things that buy headroom

Caching at the gateway instead of per service. If four services ask the same question, an application-level cache inside each one misses four times. A gateway cache hits three of them, and every hit is a request that never touches your ITPM at all. Bifrost's version only engages when a request carries an x-bf-cache-key header, defaults to a 5 minute time-to-live and a 0.8 cosine similarity threshold, and can run in exact-match mode with no embedding provider at all. That threshold is the whole product, and I went into why in a post on semantic caching.

Stopping one tenant from eating the org bucket. Your organization limit is shared, so one runaway batch job starves the interactive traffic sitting behind it. Governance limits let you cap consumption per virtual key before the request ever reaches Anthropic, in tokens or in requests, with calendar-aligned reset periods running from 1m up to 1Y. When a cap trips, the gateway reports the breach with its own code instead of a vague 429: token_limited or request_limited on a 429, budget_exceeded on a 402, and 403 for a blocked model or provider. The governance docs carry the full mapping.

That is a real difference in debuggability. "Your batch job hit its token cap" is a fix you can act on in a minute. "Anthropic said 429" is an afternoon.

Pointing Claude Code at it

Since a lot of people arriving at this error are inside a coding agent rather than a Python service, the integration is two environment variables. Bifrost exposes an Anthropic-compatible endpoint, so the client never knows anything changed.

export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
export ANTHROPIC_API_KEY=dummy-key

The path is /anthropic, and not /v1/anthropic. And the key can be a placeholder because the real credentials live inside the gateway.

Gateway traffic is billed per token against API keys. It uses a separate meter from Pro or Max subscriptions, not a larger subscription allowance.

What a gateway will not fix

Four things, because a post that only lists the wins is a brochure.

It does not help with a 529. An overloaded error is Anthropic-wide capacity, so a fallback to Bedrock might dodge it, but no amount of key pooling on the direct API will.

It does not make a cross-provider fallback behave identically. Sonnet 5 on the direct API and Sonnet 4.5 on Bedrock are not the same model, and if your prompts are tuned tightly or you lean on a specific tool-calling shape, your fallback path needs its own evals. A fallback that silently produces worse answers is arguably worse than a clean 429.

It does not fix a workload that is simply too large. If you need 5x your current ceiling permanently, the answer is a tier increase or committed capacity, and routing is the bridge that gets you there without downtime in the meantime.

And the vendor benchmarks are vendor benchmarks. Bifrost's page claims 20 microseconds of added latency at 5,000 requests per second, and a 50x advantage on that same p99 number over LiteLLM. Those are Maxim's own numbers on Maxim's own harness, and you should treat them the way you treat every published benchmark, which is as a reason to go run your own.

What you should actually do, in order

Ordered by what each one costs you, cheapest first. Stop at the first one that fixes it.

One. Log the full error body and check for retry-after. If it is missing on a 429, you have a spend cap and no retry strategy will ever help you.

Two. Export the three anthropic-ratelimit-*-remaining headers as gauges. You cannot tune what you cannot see.

Three. Turn on prompt caching and check your hit rate on the Usage page. This is the single biggest win available and it costs you an afternoon, because cached reads do not count against ITPM on most models.

Four. Add retry-after-aware backoff with jitter, and make sure you are not stacking it on top of the SDK's own retries.

Five. Move anything non-interactive to the Message Batches API, which has its own separate rate limits.

Six. Request a tier increase. It is free and it is slow, so start it before you need it.

Seven. Only now, put a gateway in front and pool keys or channels. This is real infrastructure with real operational cost, and it is worth it once you have genuinely run out of ceiling, and mostly not before.

The reason I would put it dead last rather than first is that most 429s I have run into were not capacity problems at all. They were a cache that never got turned on, or a retry loop with no jitter, or a spend cap that everybody kept retrying into. Fix those and the ceiling stops being the constraint.

If you have hit a 429 shape that doesn't fit any of the five rows in that table, drop it in the comments, I'd genuinely like to see it. And if you want more of this sort of thing, I'm on X and I write everything up first on my site.

Claude Pro Usage Limits Explained and How to Work Around Them

Swapnoneel Saha — Tue, 25 Aug 2026 17:05:36 +0000

In March 2026, Claude Max subscribers started watching a 5-hour window drain in 90 minutes. One person reported going from 21% used to 100% used on a single prompt, and everybody assumed it was a bug.

It wasn't. Anthropic's Thariq Shihipar posted on X and said that they're adjusting the 5 hour session limits during peak hours to manage growing demand, affecting roughly 7% of users.

Claude Pro meters you on two clocks at once: a rolling 5-hour session window and a weekly cap, shared across claude.ai, Claude Code and the desktop app. Neither one is a message counter. Both are token meters, and what drains them fastest is not how much you ask, it is how much you make Claude re-read.

Once you read the meter properly, most of the "limits" problem becomes context management, and context is something you control.

What are Claude Pro's usage limits, exactly?

Two windows, running at the same time, and you can hit either one.

The 5-hour session window starts with your first message and rolls. It is not a clock that ticks at the top of the hour, and it is not tied to a calendar day. Send a message at 2pm and that window closes at 7pm, whether you sent one message in it or four hundred.

The weekly window sits on top of that and resets at a fixed time assigned to your account. This one is the real ceiling for heavy users, because you can stay inside every 5-hour window all week and still run out on a Thursday.

Both windows are shared. Anthropic's docs are explicit that usage across claude.ai, Claude Code and Claude Desktop counts toward the same pool. So the hour you spent arguing with Claude about your resume in the browser is the same hour you don't get in your terminal later.

And here is the part that annoys everyone, me included: Anthropic publishes no token number for any plan. Not for Pro, not for Max. Pro is described as more usage than free, and Max is sold as 5 times or 20 times more usage than Pro. Any blog quoting you an exact message count for Pro is guessing.

The plans themselves, as of August 2026: Pro is 17USD per month billed annually or 20USD per month billed monthly, and Max starts at 100USD per month. Same model access on both. The only thing you buy with Max is a bigger number on the same two clocks.

There is one more distinction worth holding onto, because it changes what you do when you get blocked. A message saying you hit your session limit or weekly limit is plan-wide, and switching models with /model will not save you. A message saying you hit your Opus limit or Sonnet limit is model-specific, and switching to a different model family genuinely does keep you working.

Why your limit drains faster than your typing

Two developers, same Pro plan, same afternoon.

Developer A opens a fresh Claude Code session, asks 40 short questions about a small file, and closes it.

Developer B opens one session at 9am, works on and off all day across three unrelated tasks without ever clearing, and sends 12 messages total.

Developer B burns more of the plan. Anthropic's cost documentation explains why:

Claude Code sends your full conversation with every request, and each time Claude uses tools it sends another request carrying that batch of tool results.

So a one-line question at 4pm, in a session that has been open since 9am, draws usage for the entire conversation sitting behind it. Every file that got read, every tool result, every previous answer. Twelve messages carrying a whole day of accumulated context beats 40 messages carrying almost nothing, every single time.

Three more things burn the meter while you are not looking, and none of them are obvious.

Cache misses. Prompt caching is what keeps a long session affordable, because re-read history gets billed at the cached rate instead of the full one. But the cache has a lifetime. On a subscription it is one hour, and your first message after a longer break misses it and reprocesses your entire context at full price. That is the real answer to "why did one question cost me so much".

Extended thinking. It's on by default, thinking tokens bill as output tokens, and the default budget can run to tens of thousands of tokens per request depending on the model. You are paying for reasoning on tasks that may not need any.

Agent teams. If you have them switched on, they use roughly 7 times more tokens than a normal session when teammates run in plan mode, because every teammate carries its own full context window. Five agents is five conversations, and not one.

I have some personal skin in this part. I have been building ANRL, an AI-native representation language with a Rust parser-compiler, specifically because delimiter overhead and context fragmentation waste an absurd share of a context window. It cuts delimiter token overhead by over 40% in my benchmarks. Spending months on that problem is what made me stop reading the limit as a quota and start reading it as a bill for context I chose to carry.

Why did Claude cut me off after one question?

Because that one question was not one question.

If you resumed a large session after a long break, you paid a cache miss on the whole history. If a scheduled task or a background job fired while the session sat idle, it sent your full context along with it. If you ran /compact, that operation itself reads the entire conversation it is summarizing, so compacting a huge context is itself a huge request.

The tell is always the same: the size of the request has almost nothing to do with the size of what you typed.

When do Claude usage limits reset?

The 5-hour window resets 5 hours after your first message in that window, rolling. There is no fixed reset hour, so the practical move is just to notice when you started.

The weekly window resets at a fixed time tied to your account rather than a global Monday. When you actually get blocked, the error message tells you the reset time, and that is the only authoritative source you have.

Worth knowing: Claude Code v2.1.234 and later can wait out a limit and resume the interrupted task automatically once the window resets. It lives in /rate-limit-options, and it beats sitting there refreshing.

How do you see where your usage actually went?

This is the step almost everybody skips, and it is the one that changes behaviour.

Run /usage inside Claude Code. On a paid plan it shows plan usage bars plus a breakdown that is genuinely useful:

Attribution, meaning how much of your recent usage went to skills, subagents, plugins and each individual MCP server, as a percentage of the total.
Behavior flags, which name the behaviour costing you the most whenever one crosses 10% of recent usage, so "long context" or "cache misses" gets called out by name instead of you having to guess.
Loops, the heaviest scheduled tasks that ran recently, with per-run token counts.

Press d or w to flip between the last 24 hours and the last 7 days.

Then run /context to see what is sitting in your context window right now, and /insights for a report on how you actually work, written out to ~/.claude/usage-data/report.html.

One honest caveat. These numbers are computed from local session history on that machine, so usage from your other devices and from claude.ai is not in there. Don't treat the bars as your true remaining balance.

I built something adjacent to this on a contract a while back, an internal tool that captured an AI product's logs and turned them into reports on latency and probable slowdowns (keeping it vague here, can't say much more than that). The lesson transferred completely: teams argue about model quality for weeks and never once look at the token accounting, which usually takes an afternoon and explains most of the pain.

How do you actually work around Claude Pro's usage limits?

Let me be blunt about one thing before the list. Nothing here raises your subscription cap. There is no flag and no proxy that makes Anthropic hand you more of a plan you didn't buy, and anybody selling you a "bypass" is selling you something that doesn't exist.

What you can do is make each token buy more work, and give yourself somewhere to go when you get blocked. Both are real, and together they are worth more than a plan upgrade.

1. Clear between unrelated tasks

/clear does more work than anything else on this list, because it attacks the long-context problem directly. Switching from a bug fix to writing docs? Clear. Use /rename first so you can /resume that session later if you need it.

2. Match the model to the job

Sonnet handles most coding work and costs meaningfully less than Opus. Keep Opus for architecture calls and multi-step reasoning, and switch with /model mid-session rather than committing at the start. For simple subagents, set model: haiku in the subagent config.

Leaving Opus as your default all day is the most common way people burn a weekly cap without noticing.

3. Turn the thinking budget down when you don't need it

Lower the effort level with /effort, disable thinking in /config for simple work, or cap it with an environment variable:

MAX_THINKING_TOKENS=8000

That line says: give me at most 8,000 tokens of reasoning per request. On adaptive-reasoning models the budget is ignored, so use effort levels there instead.

4. Shrink what loads at session start

Your CLAUDE.md is loaded into context at the start of every single session, so a 600-line file is a tax on unrelated work all week. Anthropic's own guidance is to keep it under 200 lines and move specialised instructions into skills, which load only when invoked.

Same idea for MCP servers. Run /mcp and switch off the ones you aren't using, and prefer CLI tools like gh or aws where they exist, since those add nothing to context until Claude actually runs them.

5. Push verbose work out of the main conversation

Two mechanisms, same principle: keep the noisy stuff out of the conversation that gets re-sent every turn.

Subagents for running tests and chewing through logs. The verbose output stays in the subagent's context and only a summary comes back.

Hooks for preprocessing. A PreToolUse hook that greps a 10,000-line log for ERROR before Claude ever sees it takes that context from tens of thousands of tokens down to hundreds.

6. Stop paying for cache misses

The cache lifetime is one hour on a subscription. So the pattern that hurts is: work for an hour, go to a meeting for 90 minutes, come back and drop a one-liner into the same session.

Either come back inside the hour, or start fresh instead of resuming. On Pro and Max, Claude Code will offer to resume a large session from a summary rather than the full history, and you should take it.

7. Understand what usage credits really cost

You can switch on usage credits and keep working past your cap, billed at standard API rates. Fine. But there's a trap in the small print that nobody mentions: once you are drawing on usage credits, the prompt cache lifetime drops from one hour to five minutes!

So the moment you go over your cap, your long sessions get dramatically more expensive per message, and not just marginally. You can choose the TTL yourself to keep the one-hour lifetime, and if you use credits at all, you should.

8. Give yourself a second lane

The seven above make one plan go further. This one is different, because it gives you somewhere to fail over to when the plan is genuinely spent.

Claude is available from Anthropic directly, from Amazon Bedrock and from Google Vertex AI, and each of those has its own independent rate limits. Your subscription cap has no bearing whatsoever on your Bedrock quota. So "I'm blocked until Tuesday" is really a routing problem wearing a costume, and routing problems have known solutions.

What an AI gateway does about this, and what it can't

An AI gateway is a proxy that sits between your tools and the model providers you use. Bifrost is Maxim AI's high-performance, open-source AI gateway that unifies access to 20+ providers through a single OpenAI-compatible API. For usage limits, the relevant features are ordered provider fallbacks and virtual keys with their own budgets and rate limits. Together, they create a separately metered API route for overflow traffic, but they do not increase the Claude Pro subscription allowance.

A gateway cannot raise your Claude Pro cap. Your subscription is an OAuth relationship between Claude Code and Anthropic, and no proxy in the world changes that arithmetic. Worse, the moment you point Claude Code at a gateway you are authenticating with API keys, which means that traffic bills per token and doesn't touch your subscription at all. Different cost model, and you should walk into it knowingly.

Which is exactly why it works as an overflow lane. It's a separate meter, and not a bigger one.

Pointing Claude Code at it is two lines in ~/.claude/settings.json, or %USERPROFILE%\.claude\settings.json on Windows, which is where mine lives:

"env": {
  "ANTHROPIC_BASE_URL": "http://localhost:8080/anthropic",
  "ANTHROPIC_AUTH_TOKEN": "your-virtual-key"
}

The path is /anthropic and not /v1/anthropic, which is the mistake everyone makes exactly once. From there, /model openai/gpt-5.5 or /model vertex/claude-haiku-4-5 switches providers mid-session, and the harness never knows anything changed.

The gateway adds four controls that help manage the separate API meter.

Failover that fires on the error you care about. Fallback chains are declared as an ordered list, and every provider in the chain gets its own full retry budget:

"fallbacks": [
  "anthropic/claude-sonnet-4-6",
  "bedrock/anthropic.claude-sonnet-4-6-v1:0"
]

The detail I like here is how a 429 is handled. Hitting a rate limit does not immediately jump you to the next provider. It rotates to another key within the same provider first, and only falls through once that provider's retries are exhausted. Backoff doubles on each retry until it hits a ceiling, with a bit of randomness thrown in so a thousand clients don't all retry on the same tick. In practice that is min(initial × 2^attempt, max) × jitter, defaulting to 500ms initial and 5,000ms maximum, so a provider having a bad minute doesn't get hammered on the way out.

Budgets that refuse instead of warn. Virtual keys are scoped credentials carrying their own budgets and rate limits. You give the coding agent one key and the side project another, and when a budget is exhausted the request gets refused with a real status code rather than quietly costing you money. This is the piece Anthropic's own plan structure genuinely does not give an individual.

Caching, so repeated work stops being paid work. A cache hit is a completion you never pay for, in tokens or in seconds. I went through the mechanics, the threshold trap and the realistic hit rates in what semantic caching actually is, including why the 95% figure quoted everywhere is not a hit rate at all.

Numbers you can act on. Every request through the gateway is logged with tokens, cost, latency and provider. Same accounting /usage gives you, except it survives across machines and covers the tools that aren't Claude Code.

If you want the routing side in more depth, I wrote up adaptive load balancing separately. And the overhead question has a real answer: Bifrost adds tens of microseconds per request at 5,000 requests per second, which is nothing next to a model call.

Is Claude Max worth it, or should you just use the API?

Depends on one number, and you can go get it today.

Open /usage, press w, and look at what share of your weekly cap you actually consumed. If you regularly finish the week under 70%, Max is not your problem and better context habits will get you the rest. Upgrading to fix a habit is an expensive way to avoid typing /clear.

If you are hitting the weekly wall by Wednesday every week, then it's a real capacity problem and you have two honest options. Max at 100USD per month buys you 5 times the same plan, predictably, and it's the right answer if your usage is steady. Pay-per-token through a gateway has no ceiling at all, but also no floor, so it's the right answer if your usage is spiky.

For reference on the pay-per-token side, Anthropic's enterprise deployments average around 13USD per developer per active day and stay under 30USD per active day for 90% of users. On steady daily use, that math does not favour the API. On three heavy days a month, it does.

My own answer, and yours might be different: subscription for the daily driver, gateway for the overflow and for everything that isn't Claude Code.

One more thing about the limits themselves

They move, and this year they have mostly moved in your favour.

On 6 May 2026, Anthropic doubled Claude Code's 5-hour rate limits across Pro, Max, Team and Enterprise, and removed the peak-hours reduction that caused the March mess. That came attached to a compute deal for more than 300 megawatts of new capacity, over 220,000 GPUs.

Then weekly Claude Code limits went 50% higher as a promotion from 13 May, originally through 19 August 2026, and later extended to 31 August. Anthropic's own wording about making it permanent was careful: they hope to, but capacity may be tight.

So treat any specific limit you read about, including the ones in this post, as a snapshot. The mechanism is stable. The numbers are not.

What I actually do

Nothing exotic, and it took about a week to settle into.

Claude Code stays on the Pro subscription, because that is the best price per unit of work I can get anywhere. I /clear between unrelated tasks, which I resisted for a long time and was wrong about. Sonnet is my default and Opus is a deliberate choice, and not a setting I forgot to change. And when something feels slow or expensive, I open /usage and read the behavior flags instead of guessing, which is the habit that actually stuck.

Underneath the harnesses that aren't Claude Code, my traffic goes through the gateway with a fallback rule, so one provider having a bad day doesn't end my afternoon. I also keep a second subscription with a different vendor, which is less a strategy than an admission that a single meter is a single point of failure.

Start with /usage this week. Find your biggest behavior flag and fix that before you spend a rupee on a bigger plan. If it says long context, you have a /clear habit to build. If it says cache misses, you have a scheduling problem. /usage tells you which habit to change first.

And if you have found a genuinely clever way to stretch a Pro plan that isn't on this list, I want to hear it in the comments. I write more about LLM infrastructure and building with AI over at swapnoneel.site, and I'm on X (swapnoneel123) if you want to argue about any of this.

Top 5 AI Agent Governance Platforms in 2026

Swapnoneel Saha — Sat, 22 Aug 2026 12:58:20 +0000

Most teams shopping for AI agent governance in 2026 end up buying a dashboard, and then find out six months later that their actual problem was an agent holding a database credential with no spending cap on it.

AI agent governance platforms decide what your autonomous agents are allowed to do and how much they can spend doing it. The category splits into five layers, and no single product covers all five well. The five that matter in 2026 are Bifrost, Microsoft Agent 365, Zenity, Arthur AI, and Credo AI.

I have ranked them by the thing most lists skip: how much each one can actually stop, in real time, before a bad action lands. Not how good the report looks afterwards.

What AI agent governance actually means

Let's start from first principles, because "governance" is a word that has been stretched until it means almost nothing.

Traditional software governance assumes a human is somewhere in the loop. A person logs in, a person clicks the button, and the audit log records which person did it. Permissions attach to that person.

An AI agent breaks that assumption in a very specific way. The agent holds the credential, decides on its own which tool to call, and calls it a few hundred times while you are asleep. There is no click to attribute.

So governing an agent means controlling five separate things, and they really are separate:

Identity. Which agent is this, who owns it, and what is it allowed to be?
Authorization. Which models, tools, data and systems can it reach?
Spend. How many dollars and how many tokens can it consume before it gets cut off?
Behavior. Was the input a prompt injection, was the output leaking a card number, was the answer just confidently wrong?
Evidence. Can you reconstruct, six weeks later, exactly what it did and why?

Miss any one of those and you don't have governance, you have a partial view with a nice chart on top of it.

Gartner polled more than 3,400 organizations investing in agentic AI and predicted that over 40% of agentic AI projects will be canceled by the end of 2027, naming escalating costs and inadequate risk controls as two of the three reasons. Notice that both of those are governance failures and not model failures. The model was fine. Nobody had put a limit on it.

And the readiness gap is not subtle. In a Deloitte survey of 3,235 IT and business leaders across 24 countries, only 21% said their organization has a mature governance model in place for agentic AI. So four out of five teams are running agents on trust.

The five control points, walked through one incident

This is the part I wish the other lists on this topic did, so let me do it here.

Take one concrete setup. A customer support agent running on Claude Sonnet, wired to two MCP servers: one for Linear so it can file tickets, and one Postgres connection so it can look up an order. A support engineer built it in an afternoon. It works fine.

Now walk through what goes wrong, as four separate events.

Event one. The Postgres MCP server got registered with a connection string that has write access, because that was the string already sitting in the engineer's .env file. Nobody noticed, because the agent never tried to write anything.

Event two. A customer pastes a support message that contains an instruction addressed to the agent, telling it to look up recent orders for a different email address. This is item one on the OWASP Top 10 for Agentic Applications, published in December 2025, and this is the boring, non-exotic version of it.

Event three. The agent complies, and because it is trying to be helpful, it iterates. It runs the same lookup pattern 4,000 times over a two-hour window, each call carrying a large system prompt plus the full retrieved context.

Event four. Somebody finds out the next morning, from a bill.

So which layer catches which event?

An identity layer catches event one, and only event one. If the agent has an identity with declared scopes, then a read-only agent holding a write-capable credential is a visible contradiction, and you can alert on it. But identity has nothing at all to say about a malicious support message.

A behavior layer, meaning guardrails, catches event two. A prompt injection classifier sitting in front of the model can flag that instruction and refuse the turn. But guardrails run per call, and they are stateless. Four thousand individually-innocent calls are four thousand clean passes.

An evidence layer, meaning tracing and observability, sees all four events, perfectly, in full detail, after they have already happened. That is not a criticism of it, that is the definition of the layer. I wrote a whole post on what you should capture on every agent call and I still think tracing is the first thing you should turn on. It is just not a control.

The only layer that can stop event three while it is happening is the layer that every single call physically passes through, counting tokens as they go, with the authority to return a 402 on call number 900.

That layer is the gateway. Which is exactly why it is number one on this list.

Now, the five AI agent governance platforms

That is the map. Five control points, and one hard rule underneath all of it: a tool can only enforce what it sits in front of.

So here are the five, ordered by how much of the request path each one owns, with what it actually does and where it runs out of road.

1. Bifrost, governance on the request path

Bifrost is Maxim AI's open-source AI gateway, written in Go, and it is at number one here because it is the only entry on this list that sits inside the call. Every model request and every MCP tool call passes through it, which makes it the only one that can refuse.

The governance model is a three-tier hierarchy: customer, then team, then virtual key. A virtual key is the thing your application actually authenticates with, and it is the primary governance entity. One looks roughly like this:

{
  "id": "vk-001",
  "name": "support-agent-prod",
  "value": "sk-bf-*",
  "is_active": true,
  "provider_configs": [
    {
      "provider": "anthropic",
      "weight": 0.5,
      "allowed_models": ["claude-sonnet-4-5"],
      "key_ids": ["anthropic-primary"]
    }
  ],
  "team_id": "team-support-001",
  "expires_at": "2026-12-01T00:00:00Z"
}

Read that config out loud as a sentence and you get the whole point of the layer! This agent may talk to Anthropic, may use exactly one model, may use exactly one of your provider API keys, belongs to the support team, and stops working in December whether anyone remembers it or not.

Three parts of this are worth pulling out properly.

Budgets that cascade

Budgets attach independently at each of the three tiers, and they get checked cumulatively. The virtual key's budget is checked, then the team's, then the customer's, and the request only proceeds if all of them still have room.

{
  "id": "budget-support-vk",
  "virtual_key_id": "vk-001",
  "max_limit": 100.00,
  "reset_duration": "1M",
  "current_usage": 0.0
}

Reset durations run from one minute up to one year (1m, 1h, 1d, 1w, 1M, 1Q, 1Y), and there is a calendar-aligned mode that resets on real UTC month and quarter boundaries instead of a rolling window. That sounds like a tiny detail until you try to reconcile a rolling 30-day agent spend against a finance team's calendar month.

There is also a budget override, with an effective limit that equals the base budget plus the override amount, granted either for a fixed number of cycles or forever. That primitive exists because the alternative, in every organization I have ever seen, is somebody quietly raising the real limit at 2am and never lowering it back.

Rate limits that count tokens, and not just requests

Rate limits live only at the virtual key tier, and they are two fully independent counters:

{
  "token_max_limit": 10000,
  "token_reset_duration": "1h",
  "request_max_limit": 100,
  "request_reset_duration": "1m"
}

That separation is the thing that would have caught event three in the walkthrough. A requests-per-minute cap does nothing against an agent making a modest number of enormous calls. A token cap with its own independent reset window is the control that actually maps to how agents burn money.

And the failures come back as honest HTTP semantics, which matters more than it sounds: 402 for a budget exhausted, 429 for a rate limit hit, 403 for an inactive key, an expired key, or a blocked model. Your client code already knows how to handle those. You are not parsing an error string to find out what happened.

MCP tools as a governed surface

This is the part that almost nothing else in the category does yet.

Bifrost's MCP gateway treats tools the way it treats models, so tool filtering is deny-by-default at three levels. The client config sets which tools exist at all, per-request headers (x-bf-mcp-include-clients, x-bf-mcp-include-tools) narrow that set for a single call, and virtual key filtering overrides both. A key with no MCP configuration gets no MCP tools at all, period! You can also bundle a curated set of tools from several different servers into one MCP Tool Group and attach that group to a key.

Now go back to event one in the walkthrough. A write-capable Postgres tool that the support agent's key was never granted is a tool the agent cannot even see, regardless of what is sitting in anybody's .env file.

For the behavior layer, Bifrost does not try to build its own classifiers. It plugs into AWS Bedrock Guardrails, Azure AI Content Safety, Google Model Armor and Patronus AI, and it emits OpenTelemetry out of the box. It also does adaptive load balancing with provider failover, semantic caching, and multi-node clustering, which is the reliability half of the same control plane.

The numbers. Maxim's own benchmark puts the added latency in the tens of microseconds (the repository claims under 15 microseconds per request at 5,000 requests per second, the product page says 20), with 3.3 GB peak memory, and against LiteLLM it claims 9.5 times the throughput and 54 times faster latency at the 99th percentile, meaning the slowest one request in a hundred. Treat vendor benchmarks as vendor benchmarks, but the architectural reason behind them is real: this is a Go binary in your request path, and not a Python process. The core is Apache 2.0 on GitHub at around 7.4k stars, and you can have it running with one command:

npx -y @maximhq/bifrost

Where it runs out of road. Bifrost governs the call, and it does not discover agents. If a team stands up an agent that talks to OpenAI directly and skips the gateway, Bifrost will never know that agent exists. So it is simultaneously the strongest enforcement point available and the easiest one to route around, which means it only works if you make it the only path out. Also, enterprise RBAC, SSO and audit logs sit in the commercial tier and not in the open-source core.

Pick it if you want a control that can actually say no, and you are willing to make gateway egress mandatory.

2. Microsoft Agent 365 with Entra Agent ID

Microsoft Agent 365 is the most serious attempt yet at solving layer one properly.

The idea underneath it is Entra Agent ID, and it is a genuinely good idea: every agent gets a first-class identity in the directory, the same way every employee has one. Once an agent is a directory object, everything Microsoft already built for humans starts applying to it. Conditional Access policies. Least-privilege scoping. Purview for data classification, Defender for threat detection, Intune for endpoint controls. Agent 365 then adds the registry and the admin hub on top, so you get a real inventory of the agent fleet, including the agents somebody spun up in Copilot Studio and never told you about.

That registry is the answer to the exact problem a gateway cannot solve. Agent sprawl is a discovery problem before it is an enforcement problem, and you cannot govern an agent you have not found yet.

Where it runs out of road. The gravity is real. Coverage is deepest for Copilot Studio and Azure AI Foundry agents and for agents on managed endpoints, and it thins out fast for a Python agent your ML team runs in a container on another cloud. It also does not do token budgets in any meaningful sense. It will tell you an agent exists and what it may touch, and it will not stop that agent at dollar 4,000.

Pick it if you are already an Entra shop, which honestly means pick it anyway, because identity is the one layer you cannot fake.

3. Zenity, posture and runtime action validation

Zenity comes at this from the security side rather than the platform side, and its strength is the part of your estate that nobody has an inventory of.

It discovers agents living inside Copilot Studio, Power Platform, AWS Bedrock and Google Vertex AI, then keeps watching what those agents touch and share. The framing is AI Security Posture Management, and the genuinely useful mechanic is buildtime policy: it flags an over-permissioned connector before the agent ships, which is the cheapest possible moment to catch event one from the walkthrough. Gartner named Zenity a 2025 Cool Vendor in Agentic AI TRiSM (their acronym for trust, risk and security management), and that low-code coverage is why.

So why does the low-code angle matter this much? Well, because that is where the agents you don't know about actually live. An engineer's Python agent is at least in a repository somewhere. A business analyst's Copilot Studio agent with access to a SharePoint site full of contracts is nowhere you are currently looking.

Where it runs out of road. It is a security product, so it governs risk and not cost. There is no budget primitive here and no token accounting at all. And its discovery is strongest in exactly the places Microsoft's is strongest, so if you already have Agent 365, the overlap deserves a hard look before you pay for both.

Pick it if a lot of your agents are being built by people who are not engineers.

4. Arthur AI, guardrails and evaluations without a framework tax

Arthur AI launched its Agent Discovery and Governance platform in December 2025, and the thing I find genuinely interesting about it is that it is framework-neutral in a category that mostly is not.

Discovery works across four vectors: OpenTelemetry streams, MCP server monitoring, network-layer analysis, and the platform APIs of Vertex AI, Bedrock and Azure AI Foundry. And because the tracing is built on OpenTelemetry and OpenInference rather than a proprietary SDK, it works with LangChain, LlamaIndex, raw OpenAI and Anthropic calls, and whatever your team decides to pick next quarter.

On top of that it runs two things that are worth keeping separate in your head. Runtime guardrails, which intercept before and after the model call for PII, prompt injection and toxicity. And continuous evaluations against live production traffic, which is the part that catches hallucination and quality drift, the failures that never throw an error and never show up in a latency graph.

That second distinction is the one I care most about, and it comes from doing the work by hand and getting it half right. On a contract with a startup building an AI co-worker that lives in Slack (keeping it a bit vague, can't reveal much more than this lol), I was the first layer of internal testing, and I built an internal tool that captured the product's data logs and generated reports on latency and probable slowdowns. It was useful, and the core users got a better product out of it. But it told me when the thing was slow, and it never once told me when the thing was confidently wrong. Those are two different systems, and I had only built one of them.

Where it runs out of road. Same ceiling as any observability-first product: it sees everything and blocks a narrow slice of it. The guardrails are per-call, so the accumulation problem from event three stays unsolved, and there is no spend primitive here either.

Pick it if your stack is heterogeneous and output quality is your real risk, rather than permissions.

5. Credo AI, the compliance program of record

Credo AI is on this list for a reason that has nothing to do with runtime, and I would rather be blunt about that than pretend it belongs in the same bucket as the other four.

Its Policy Packs translate regulation into something an engineering team can actually act on, mapping the EU AI Act, ISO 42001 and the NIST AI Risk Management Framework down into concrete requirements. It keeps an AI Agent Registry of what is in production, plus GAIA, an assistant for the governance workflow itself. When a regulator or an enterprise customer's security review asks what your agents do and which controls apply to them, this is the layer that produces the answer.

And the timing here is worth getting right, because a lot of content on this topic is now stale. The Digital Omnibus on AI entered into force on 27 July 2026, and it deferred the high-risk obligations for standalone Annex III systems all the way out to 2 December 2027, and to 2 August 2028 for AI embedded in regulated products. So the deadline a lot of teams panic-bought a platform for has moved.

What did not move is Article 50. The transparency duties landed on 2 August 2026, and those catch every chatbot and every piece of synthetic content, which in practice means they catch most agent deployments regardless of risk tier. The high-risk regime got 16 more months. The disclosure regime is live right now.

Where it runs out of road. Credo AI cannot block a request. It produces policy and evidence, and if you buy it expecting enforcement, you have made the exact purchasing mistake this whole post is about.

Pick it if you sell into regulated industries, or your legal team has started asking questions in writing.

What none of these five actually do

Time for the honest part, because a list that ends on five recommendations and no caveats is just an ad.

Agent-to-agent calls are mostly ungoverned. When agent A hands a task off to agent B, whose budget does it hit, whose identity does it carry, and does B's guardrail see the original user's prompt or A's rewritten version of it? Inter-agent communication is item seven on the OWASP agentic list, and I have not seen anybody solve it convincingly yet. Gateways get closest, because that hop is still an HTTP call, and "closest" is doing a lot of work in that sentence.

Memory is a governance hole. An agent that writes a poisoned fact into its long-term store carries that fact past every per-call guardrail forever, because on every future call it is not an injection anymore, it is just context. Nothing on this list governs writes to agent memory the way it governs writes to a database.

Agent identity is not portable. Entra Agent ID is real and it is good, and it is also Microsoft's. There is no cross-vendor standard yet, so an agent's identity in your directory means nothing to a partner's system. We are roughly where federated human identity was before SAML showed up.

And the buying mistake, which is the whole reason I ordered this list the way I did. A team worried about a runaway bill buys a compliance platform, and a team worried about a regulator buys a tracing tool. Both walk away with a real product doing a real job, and neither one has touched the thing that wakes them up at 3am.

Frequently asked questions

Is an AI gateway enough on its own for agent governance? For spend, authorization and tool access, yes, and it is the strongest control you can get for those three. For discovery and identity, no. A gateway cannot see an agent that does not route through it, so pair it with an identity or discovery layer, and make gateway egress mandatory at the network level rather than by policy document.

Do I need an AI agent governance platform for a small team? You need two of the five control points from day one, and neither of them is a purchase. Put every model call behind one gateway with a per-key spending cap, and turn tracing on. My own side project Scholarian has fetched and ranked over 10,000 papers across 250-plus search sessions, and the reason I care about per-key caps is not enterprise policy, it is that a retry loop in a research pipeline is perfectly capable of spending real money overnight.

What is the difference between AI agent governance and AI observability? Observability tells you what happened, governance decides what is allowed to happen in the first place. They get sold together because tracing is how you prove a policy was enforced, but a trace has no authority. If your platform can produce a beautiful timeline of an incident and could not have prevented that incident, you bought observability.

Where do guardrails fit into all this? Guardrails are the behavior layer, and they run per call, screening inputs for injection and outputs for leakage or toxicity. They are necessary, and they are stateless, which is why they miss failures made out of many individually-fine calls. Run them at the gateway rather than inside each application, so that one policy update covers every agent instead of six repositories.

Does the EU AI Act require an agent governance platform? No regulation names a product. But the Article 50 transparency duties are in force as of 2 August 2026, and the deferred high-risk regime still expects risk management, logging and human oversight when it lands in December 2027. Those obligations are much cheaper to meet if the logging was there from the start.

What would I actually pick?

If I could only run one thing, it is the gateway, and that is Bifrost. Put every model call and every MCP tool call behind it, then set a token budget with its own reset window on every virtual key and deny tools by default. It is the only layer on this list that can turn a policy into a refusal, it is Apache 2.0 so the argument costs you nothing but an afternoon, and at microsecond-scale overhead you are not trading latency for control.

Then add identity second, and not fifth. Agent 365 if you are on Entra, which most enterprises already are. Discovery is the one problem a gateway genuinely cannot solve for you, and most of the rest of this list is a refinement of a control you would already have.

And here's my bet for the next 18 months: this category does not stay five layers wide. Identity and posture will collapse into whoever already owns your directory, evaluations will fold into observability, and the thing that survives as a separate purchase is the request path, because that is the only place enforcement is physically possible. If I turn out to be wrong about that, I'll happily write the follow-up.

That's all from my side, folks. If you are running agents in production and you have found a control that actually stopped something, I want to hear about it, so drop it in the comments. And if you want more of this, I write about AI infrastructure regularly on X and on my site.

What Is Semantic Caching, and Where It Quietly Breaks

Swapnoneel Saha — Wed, 19 Aug 2026 17:39:37 +0000

Two people open your support chatbot within the same minute. One types How do I reset my password? and the other types i forgot my password, how do i get a new one.

Same question, same answer, two full model calls, and your Redis cache stores both as separate keys without ever hitting on either.

Semantic caching stores past LLM responses and serves them to new queries that mean the same thing, even when the words are completely different. It works by turning every query into a vector, searching for the nearest stored vector, and returning that cached answer if the similarity clears a threshold you set.

Why your existing cache does nothing for LLM traffic

Every cache you have ever written works on exact equality. You take the request, hash it, look up the hash, and either the bytes match or they don't. Redis and Memcached both work this way, and so does the HTTP layer sitting in front of them. It works brilliantly.

It works because the traffic it was designed for is machine-generated. GET /api/users/42 is always spelled the same way by the same client, every time.

Human language is not like that. People can ask for a refund policy in many ways, and a hash function treats each wording as a different key. One extra space, one lowercase letter, one "please" at the end, and you get a completely different key.

So your hit rate on natural language collapses to nearly zero, and you go on paying for the same answer over and over.

Semantic caching fixes the matching function instead of the cache. The storage stays boring. What changes is that you stop asking "are these two strings identical" and start asking "are these two strings close enough in meaning".

The difficult part is deciding what counts as close.

How semantic caching works

The rule in plain English:

Turn the question into a point in space. Look for the nearest point we have already answered. If it is near enough, reuse that answer.

In practice, it takes five steps:

A query comes in.
You send it to an embedding model, which returns a vector of floats.
You search your vector store for the nearest stored vector, using cosine similarity.
If the best match scores above your threshold, you return the stored response and never call the model at all. That's a cache hit.
If nothing clears the threshold, you call the model, return the real answer, and write the query vector plus the response into the store with an expiry time.

In code it is almost insultingly short:

def ask(query, threshold=0.92):
    vec = embed(query)                         # step 2
    match, score = store.nearest(vec)          # step 3

    if score >= threshold:                     # step 4
        return match.response                  # cache hit, zero model tokens

    answer = llm.complete(query)               # step 5, cache miss
    store.put(vec, answer, ttl=3600)
    return answer

Notice what step 4 is really doing. It is taking a floating point number and using it to decide whether a human being gets a fresh answer or a recycled one. There is no other logic in this system, no parsing and no intent classification, nothing else that ever looks at what was actually asked.

If you want the mechanics of what store.nearest is doing underneath, I built one of these from scratch, cosine similarity and the HNSW graph and all, in my post on building a vector database from scratch. The short version is that it is an approximate nearest neighbour search, so it is fast, and it is also allowed to be a little bit wrong.

The similarity threshold is the whole product

Consider three pairs of queries and how an embedding model scores them.

Pair one. What is your refund policy? and Can I get my money back? Different words, same intent. This one should score high, and it does.

Pair two. Show me the sales numbers for Q1 2025 and Show me the sales numbers for Q3 2024. Almost identical strings, completely different answers. You would want this to score low. It does not. It scores extremely high, because most of the tokens are shared and the embedding barely notices which quarter you asked about.

Pair three. Is this drug safe for pregnant patients? and Is this drug not safe for pregnant patients? One word apart, opposite meaning.

Embedding models are often poor at handling negation. A validity audit published in August 2026 tested 9 encoder configurations and found that negation and antonym pairs score above genuinely similar pairs on every model tested, at average cosines of 0.93 to 0.999. In the production system that paper audits, flipping an instruction from "withhold the study drug" to "administer the study drug" scored 0.9608, and the safety gate that existed specifically to catch that never fired.

The reversed instruction scored higher than most legitimate paraphrases would.

A threshold like 0.92 does not cleanly separate "same question" from "different question". What it often separates is surface form, and surface form is not meaning. Two sentences that share a grammatical frame and differ in one date, one entity, or one negation will sit above almost any threshold you are willing to set.

That's the trap. Raise the threshold to 0.98 and you kill your hit rate, because honest rephrasings stop matching. Lower it to 0.85 and you start serving Q3 2024's numbers to someone asking about Q1 2025. There is no single number that fixes both, because the failure is in the measurement and not in the cutoff.

What helps is refusing to let similarity be the only gate. Partition the cache by anything the embedding is bad at holding: user, tenant, model, and any structured parameter your queries carry. If dates and IDs are pulled out into the cache key instead of being left sitting inside the prose, the embedding never gets a chance to blur them.

Semantic caching vs prompt caching vs KV caching

These three names get used interchangeably online, but they refer to different layers.

KV caching lives inside the GPU. During inference the model computes key and value tensors for every token in your context, and the KV cache keeps them around so the next token does not need to recompute attention over everything before it. This is always on, you do not configure it, and it is the reason generation gets faster after the first token.

Prompt caching, sometimes called prefix caching, is what OpenAI and Anthropic sell you at the API level. It reuses those KV tensors across requests when two requests share a common prefix. So if you send a 4,000 token system prompt on every call, the provider can skip recomputing it and charges you less for those tokens. Important detail: it matches on exact prefix bytes, so two prompts that mean the same thing but start differently will miss it entirely.

Semantic caching sits in front of the model, in infrastructure you control. It stores whole request and response pairs and matches on meaning. When it hits, you save 100% of the call, because the model is never invoked.

You can use all three together. A request can try the semantic cache first, fall through to the provider's prompt cache on a miss, and only then pay for full inference. Semantic caching is the outer layer, so it is also the one you control.

What is a realistic cache hit rate?

The 95% figure quoted in semantic-caching marketing is easy to misread.

You will see 95% quoted constantly. Trace that number back and it almost never refers to hit rate. It refers to match accuracy, meaning the cached response was correct 95% of the time it was served. Those are entirely different claims, and the second one tells you nothing about how much money you saved.

Actual production numbers are much lower. A breakdown of real deployment data puts typical hit rates at 20 to 45%, with Portkey seeing around 20% on retrieval-augmented workloads and an EdTech platform hitting about 45% on student question-and-answer traffic. Open-ended chat sits at 10 to 20% because its queries repeat less often.

Academic results land in a similar band. The GPT Semantic Cache paper reports cutting API calls by up to 68.8%, but that is on query categories picked for repetition, which is exactly the workload where this technique looks its best.

Even a 20% hit rate can matter. On a $5,000 monthly bill, that is $1,000 saved. A cache hit returns in under 5 milliseconds instead of the 2 to 5 seconds a real completion takes, so the speed difference may matter even more than the cost.

Set expectations using your own traffic.

What semantic caching costs you to run

You pay for an embedding call and a vector search on every request, including the 60 to 80% that miss. Embeddings are cheap compared to a chat completion, so the money side is fine. The latency is the thing to watch, because you have just added a network round trip to the front of every request in your system, including all the ones the cache cannot help with.

At Keploy I built a retrieval-augmented chatbot over their documentation using vector embeddings, and docs traffic is close to the best case for this technique. People ask the same twenty questions in fifty phrasings, forever. Even there, you trade a small cost on every request for a larger saving on the requests that hit. Measure that ratio before assuming the cache pays off.

Then there is staleness. Your cache does not know your prices changed on Tuesday. The stored answer is a frozen snapshot of what the model said, plus whatever context it was given at the time, and it keeps being served until its expiry time runs out. Short expiry times are safer and hit less. Long ones are the opposite. Pick deliberately.

Multi-turn conversations are worse. A follow-up like "and what about the second one?" embeds to almost nothing useful, because the meaning lives in the previous four messages and not in that sentence. The safer approach is to stop caching after a few turns of history.

And you now operate a vector store. That is one more thing to size, monitor, and pay for.

One cache, many tenants, and the leak nobody plans for

This is the part that turns semantic caching from a performance feature into a security decision, and it is why I would not hand-roll one at the application layer in an enterprise setting.

A semantic cache with one global namespace returns the nearest previous response across every user in it. Not the nearest response belonging to that user. The nearest one, period.

Picture two customers of the same SaaS product asking structurally similar questions about their own account data. Their prompts embed within 0.93 cosine of each other because they are the same question about different companies, and the cache hands one customer the other's cached answer. Nothing errors or logs a violation. It looks like a successful cache hit, making the leak hard to detect.

There is a subtler version too. Even with no wrong response served, cache hits are dramatically faster than misses, and that timing difference is observable from outside. Somebody probing your API can learn which questions have already been asked by other tenants just by watching time to first token.

The fix is not clever, it is structural. The cache key has to be partitioned by a tenant identifier resolved from something you trust, meaning the API key, a virtual key, or a signed token claim. Never from the request body, because the request body belongs to the attacker. And the lookup has to be scoped to that namespace so a cross-tenant match is not merely unlikely, it is unreachable.

This is why I prefer to handle semantic caching once at the gateway instead of rebuilding tenant isolation inside every service.

When should you not use semantic caching?

Anything that must be current. Live inventory, account balances, order status. A stale answer here is worse than a slow one.

Anything where the parameters matter more than the phrasing. Analytical queries over dates, IDs, and entities are precisely where embeddings blur the thing you needed preserved. If you cannot pull those values out into the cache key, skip it.

Anything high-stakes and low-volume. Medical, legal, financial advice. The negation problem described above can cause serious harm in those domains. And if your volume is low, you were not saving much anyway.

Long open-ended conversations. A 10 to 20% hit rate while adding latency to 100% of requests is a bad trade.

Where it does earn its place: support bots, docs assistants, FAQ layers, onboarding flows, internal knowledge search, and any product where a large user base asks a small set of questions in a lot of different ways.

How Bifrost does semantic caching at the gateway

Bifrost is Maxim AI's high-performance, open-source AI gateway that unifies access to 20+ providers through a single OpenAI-compatible API. Because semantic caching runs at the gateway, one cache policy is applied before requests reach any configured model provider.

Its semantic cache combines exact hash matching with vector similarity search, supports per-request TTL and threshold overrides, and keeps cache entries separate by model and provider by default.

Bifrost uses two cache layers.

Every request first goes through a direct hash lookup. If the prompt is byte-identical to something already cached, it returns immediately with zero embedding overhead, which matters because you just skipped the round trip that would otherwise tax every request in the system. Only on a direct miss does it embed the query and run the similarity search. So the cheap path stays cheap, and the expensive path only runs when it might actually pay off.

Caching only activates when a request carries a cache key, passed as an x-bf-cache-key header. There is no global-namespace mode for you to accidentally ship. If you want per-tenant isolation, the tenant identifier goes in that header, and cross-tenant matches then cannot happen because those entries are not in the same partition. Requiring a cache key makes isolation mandatory rather than optional.

Its other defaults are:

threshold defaults to 0.8 for semantic hits, overridable per request with x-bf-cache-threshold.
ttl, the time to live on an entry, defaults to 5 minutes. This reduces stale answers, and you can override it per request.
conversation_history_threshold defaults to 3, which means it stops caching once a conversation runs past 3 messages.
cache_by_model and cache_by_provider are both on by default, so a cached GPT answer never gets served to a Claude request.
The vector store is pluggable across Redis or Valkey, Weaviate, Qdrant, and Pinecone, so you are not forced into adopting a new database.

Every response carries a cache_debug block with cache_hit, hit_type (direct or semantic), the actual similarity score, and a cache_id. The similarity score helps diagnose a bad match, while cache_id lets you remove the affected entry or clear its partition through the API.

Every cache hit avoids the completion cost. At the gateway, the cache sits next to budgets, virtual keys, and routing. The same layer that chooses which provider gets a request can first decide whether the request needs a provider at all. I cover the routing side separately in what adaptive load balancing actually is.

Bifrost adds under 100 microseconds of overhead at 5,000 requests per second, which is negligible next to the embedding call.

What should you do first?

Do not start by building the cache.

Start by measuring how repetitive your traffic is. Take a week of your logs, embed the queries, and count how many land within 0.92 of an earlier one. That gives you an upper bound on the possible hit rate. A week of your own traffic will tell you more than a benchmark based on someone else's workload.

If that number comes back at 30% or better, turn semantic caching on at your gateway. Partition the cache key by tenant from day one and keep the expiry short. For the first few weeks, watch the similarity scores on cache hits instead of trusting the threshold.

If it comes back at 8%, the workload is too varied for semantic caching to save much.

If you have run a semantic cache in production and watched it serve something it should not have, tell me about it in the comments. Those failures are the useful stories.

AI Observability Explained: What It Is and How It Works

Swapnoneel Saha — Tue, 18 Aug 2026 06:25:25 +0000

Traditional monitoring rests on one quiet assumption that nobody ever writes down: the same input gives you the same output. Something breaks, you replay the request, you watch it break again, you fix it.

Now send the same request to a model twice. You get two different answers, and neither one of them threw an error.

AI observability is the practice of recording what happened inside an AI system on every request: the prompt, the model version, tokens, cost, latency, tool calls, and a judgement of whether the output was any good. Monitoring tells you the service is up. Observability tells you why it answered that way.

That gap is the whole story here.

Why your current monitoring stack misses all of this

Your existing setup is watching for crashes. Status codes, error rates, p99 latency, memory. All of it is designed around the idea that a broken thing looks broken.

An AI feature failing looks nothing like that. It returns HTTP 200 in 900ms, with grammatically perfect prose that happens to be wrong, or that quietly ignored the document you retrieved for it, or that called the refund tool when the user only asked a question.

Your dashboard sees a healthy service, because by every measure it has, the service is healthy.

And there are whole categories of failure your stack has no field for. It has nowhere to put "this response cost 14 cents", or "the model version changed under us last Tuesday", or "the retrieved context was garbage". Those are not infrastructure facts, and standard telemetry was never built to carry them.

Something has to hold those fields instead, which is the entire reason this tooling exists. Bifrost is Maxim AI's high-performance, open-source AI gateway that unifies access to 20+ providers through a single OpenAI-compatible API. Its position between the application and the model providers makes it useful for observability: every model request and response passes through the same layer, regardless of provider. Bifrost's built-in observability records the input, output, provider, model, token usage, cost, latency, errors, and status for each call, then makes those fields searchable in a live dashboard. Those are the details standard HTTP telemetry misses even when the request returns 200.

What one AI request actually looks like when you trace it

This is the part that made it click for me, so let me walk through a real shape.

At Keploy I built a retrieval-augmented chatbot over their documentation, using vector embeddings, so developers could ask a question instead of hunting through pages. A single question to something like that is not one operation. It is a chain, and a trace is just that chain written down.

One request breaks into spans, where a span is one step with its own start time, end time, inputs and outputs:

The user's question comes in and opens the root span.
The question gets embedded into a vector. That is a span, with its own model and its own cost.
The vector search runs and returns, say, five chunks of documentation. That is a span, and the important bit is that it records which five chunks came back.
Those chunks get stuffed into a prompt template along with the chat history.
The model call goes out. This span carries the model name and version, the temperature, the prompt tokens, the completion tokens, the cost in dollars, the total latency, and the time to first token.
If the model calls a tool, every one of those is its own child span too.

Now here is why anybody bothers with all that plumbing.

When the bot gives a bad answer, you do not have to guess. You open the trace and look at step 3. If the vector search pulled back five irrelevant chunks, your problem is chunking or embeddings, and the model did nothing wrong. If the search pulled back exactly the right documentation and the model still answered from thin air, your problem is the prompt.

Two completely different fixes, and without the trace you cannot tell them apart. All you have is "the bot said something dumb", which is the single most useless bug report in the world.

The things worth capturing on every call

You will notice I have not called this section "the three pillars of observability". Everyone else writing about this does, and I dropped it on purpose, because logs, metrics and traces is a framing built for deterministic systems and it has no slot at all for "was the answer any good".

So here is the actual list:

The full prompt and the response, as they really went over the wire, after every template and system message got assembled. Not the template, the final text.
Model, version and parameters. Providers ship silent updates. If you cannot say which exact version answered a request, you cannot explain last month's regression.
Tokens in, tokens out, and cost in dollars per request, attributed to a user or a feature.
Latency, split into total time and time to first token. Those two numbers feel completely different to a user, and one can get worse while the other improves.
Tool calls, retries and fallbacks. Which key was tried, what failed, what it fell back to.
A trace ID that ties the whole chain together, and ideally a session or user identifier so you can reconstruct a full conversation.
A quality score, attached after the fact. More on that next.

I built one of these myself earlier this year, an internal tool at a contract role that captured an AI product's logs and turned them into reports on latency and probable slowdowns (keeping it vague on purpose, cannot say much more than that). The honest takeaway was not that the tool was clever. It was that a team can ship for months on vibes, and the moment somebody puts the per-request numbers on a screen, problems nobody previously had words for suddenly have words.

How do you measure quality when there is no right answer?

Well, you do not measure it the way you measure a unit test, because there is no expected string to compare against.

The industry has mostly settled on three overlapping things. LLM-as-judge, where you send the input and output to a second model with a rubric and it scores relevance or faithfulness or tone. It is imperfect, and it is far better than nothing. Human annotation on a sample, which is slow, expensive, and still the ground truth everything else gets calibrated against. And implicit user signals, like thumbs, edits and retries, which are noisy but free.

Run those continuously and you get drift detection, which is just the same score measured over time. When your faithfulness score drops 8 points over two weeks and nobody deployed anything, something moved underneath you, and that is usually the model provider.

This is the half most teams skip, and there are numbers on it. In LangChain's 2026 State of Agent Engineering report, which surveyed 1,300+ practitioners, 89% said they had observability running on their agents while only 52% were running evaluations. So most people are recording what happened and still have no systematic opinion on whether it was good.

Which is also why I keep saying you cannot test an AI feature the way you test code. I went into that failure mode properly in my post on testing AI coding agents.

Where AI observability actually lives in your stack

Two choices here, and you can do both.

You can instrument your application directly, wrapping every model call in your own code. That gives you the most context, because your code knows what the user was doing. It also means every service, every language and every framework has to be instrumented separately, and someone has to keep it consistent.

Or you put it in the gateway. If all your model traffic already goes through one proxy, that proxy sees every request and every response by definition, and you get telemetry for services you never touched.

And the reason you can do both without doubling the work is that there is finally a shared standard. The OpenTelemetry GenAI semantic conventions define agreed attribute names for exactly this, like gen_ai.request.model, gen_ai.usage.prompt_tokens and gen_ai.usage.cost. Emit those and your AI spans slot into the same traces as the rest of your system, in whatever backend you already pay for.

Bifrost is a reasonable thing to look at here, since it does both halves. It records inputs, outputs, tokens, cost and status for every call into SQLite or Postgres with a dashboard on top, and it exports OpenTelemetry spans using those GenAI conventions plus native Prometheus counters like bifrost_input_tokens_total and bifrost_cost_total. The logging runs in background goroutines, which is why its documentation puts the added overhead under 0.1ms per request.

That last detail is the pattern to steal, whichever tool you end up picking. Telemetry gets emitted off the hot path, after the response is already on its way back to the user. Observability that slows down the thing it observes gets switched off within a week.

The routing side of that same gateway is worth knowing about too, and I covered it in my post on adaptive load balancing.

What it costs you to run

Now the uncomfortable part, because none of this is free.

Storage adds up fast. You are storing full prompts and full responses, and prompts got long. A retrieval app can easily push 8,000 tokens of context per call. At real traffic that is a serious volume of text, and this is where sampling comes in: keep 100% of errors and slow requests, keep a small percentage of the healthy ones.

Your prompts contain user data. Every support chat, every uploaded document, every email a user pasted in. The moment you log all of it, your observability store is now a system holding personal data, with all the retention and access rules that implies. Redact at the point of capture, not later.

And someone has to actually look at it. This is the one that quietly kills the whole effort. The traces get collected, the dashboard gets built, nobody opens it, and six months later it is a very expensive write-only database.

Frequently asked questions

Is AI observability the same as LLM monitoring? Close, and monitoring is the narrower one. Monitoring tracks known metrics like uptime, latency and error rate, and answers "is it working". Observability keeps enough per-request detail that you can answer questions you had not thought of yet, like "why did this one user get that answer". In practice most tools sell both under one name.

Do I need OpenTelemetry for this? No, but it is the sensible default in 2026. The GenAI semantic conventions mean your AI spans use the same attribute names everywhere, so you can change vendors without reinstrumenting, and your model calls appear inside the same traces as your database queries. Note that parts of the spec are still marked experimental, so pin your versions.

What is the difference between observability and evals? Evals are the measurement, observability is the pipe. Evals score whether an output was good; observability captures the request, the context, the cost and the trace so the score has something to attach to. You can run evals offline in CI against a fixed dataset, but you can only run them on real traffic if the traffic is being recorded.

So what should you actually do?

If you have an AI feature in production right now and you cannot pull up the exact prompt, the model version and the cost of a request from last Tuesday, that is the gap, and it is worth a day of your week.

Start with capture, and not with dashboards. Get every request logged with its prompt, response, model, tokens, cost and a trace ID, put it wherever you already look at data, and give it two weeks. You will find something. Everyone does. Quality scoring, drift alerts and per-feature cost budgets are worth adding later, but every one of them sits on top of the boring capture layer, so there is no point doing them first.

If your model calls already go through a gateway, turn on the telemetry it ships with before you write any of this yourself. That is the single highest-value hour available to you here, and it is mostly a config change.

That is my read on it, and your setup might look nothing like mine. If you have built this kind of tracing yourself, or you have had an AI observability bill genuinely surprise you, drop it in the comments, I would like to hear how it went.

You can find me on X where I post about most of what I am building, and the rest of my writing lives at swapnoneel.site.

What is Adaptive Load Balancing, and Why AI Needs It

Swapnoneel Saha — Mon, 17 Aug 2026 17:53:52 +0000

Five identical servers sitting behind one load balancer, each getting exactly one-fifth of the requests. So why is one of them pinned at 90% CPU while another one sits half idle?

Well, because an equal share of requests is not an equal share of work.

Adaptive load balancing is a routing strategy that picks a destination using live health signals like error rate, latency and utilization, instead of a fixed rotation. The balancer keeps scoring every backend while traffic flows, shifts weight toward the ones behaving well, and pulls weight away from the ones going bad.

What is a load balancer?

A load balancer is just a thing sitting in front of your servers, deciding which one gets the next request. That's it.

The simplest version is round robin. Request 1 goes to server A, request 2 to server B, request 3 to server C, then back to A again. It's a rotation, and it never once looks at what is actually happening inside those servers.

Static algorithms like this quietly assume two things: every request costs the same, and every server has the same capacity right now. Both assumptions survive about five minutes of real production traffic.

Think of a supermarket. Round robin is the sign saying "next customer to the next till, in order." Adaptive is a floor manager who watches which till is genuinely moving, spots the one stuck behind a price check, and sends people elsewhere.

Same queue, very different Saturday.

How does an adaptive load balancer decide where to send a request?

It collects signals, turns them into a score per route, and turns those scores into weights.

Three signals do most of the work:

Error rate. Is this backend returning failures? Usually the heaviest signal, and usually time-decayed, so a spike from ten minutes ago stops dominating the decision.
Latency. How slow is it right now, both against its peers and against its own recent baseline? A route that always takes 2 seconds is fine. A route that usually takes 200ms and is now taking 2 seconds is in trouble.
Utilization. How much of its capacity is already committed, so that no single fast route gets hammered into becoming a slow one.

Those collapse into one number per route, and a higher number means a bigger share of the traffic.

And the weights are not recalculated per request, because that would drop real work onto the hot path. They get recalculated on a background loop, and each incoming request simply reads the numbers that were computed a moment ago.

Bifrost is Maxim AI's high-performance, open-source AI gateway that unifies access to 20+ providers through a single OpenAI-compatible API. That makes it a useful case study here: the gateway sees provider, model, and API-key routes from one layer, which is the full decision surface its adaptive balancer needs. The documented implementation scores routes on error penalty, token-aware latency, utilization, and recovery momentum. It recalculates weights every 5 seconds, monitors both provider and key performance, and keeps route selection under 10 microseconds on the hot path.

That provider-and-key view matters because a provider can remain healthy while one API key is throttled or exhausted. A balancer that only watches provider health would miss that failure and keep sending traffic to a route that cannot serve it.

The other half of the mechanism is state. A good adaptive balancer doesn't just have a dial, it has an opinion about what each route currently is: healthy, degraded, failed, or recovering.

And the recovering state is the one people forget. A route that failed does not get cut off forever, it gets a thin trickle of live traffic so the balancer can find out when it is better. Without that, your balancer is just a fancy circuit breaker that never closes again.

Adaptive load balancing vs round robin and least connections

Each static strategy fails in a different way.

Round robin rotates blindly. Fine when every server and every request is genuinely identical, which is basically never.

Weighted round robin lets you say "server A is beefier, give it double." Better, but you set those weights by hand, based on what was true when you deployed. It has no idea what is true at 3am during a traffic spike.

Least connections picks whichever server has the fewest open connections. This one is genuinely dynamic, and it is a solid default. But an open connection is a rough proxy for load, since one connection doing heavy work counts exactly the same as one connection idling.

Least response time, usually implemented with an exponentially weighted moving average of latency, gets close to adaptive. It measures the thing you actually care about.

Adaptive load balancing is the version that stops relying on any single number. It combines errors, latency and capacity, keeps a health state per route, and has explicit behavior for pulling a route out and easing it back in.

Does the extra machinery pay off? Envoy's own benchmark for its Peak EWMA policy puts it at a 99.9% success rate under a 1-second timeout, against 99% for least-loaded and 95% for round robin (Envoy docs). That gap between adaptive and round robin is made entirely of user-visible failures.

Why an old routing idea matters again

Adaptive load balancing has been used in networking gear and reverse proxies for years. For a normal web app behind NGINX, least connections is often enough.

AI traffic changes that because it breaks the assumptions behind the older algorithms.

Why LLM traffic makes static load balancing fall apart?

Four things go wrong at once.

Requests are wildly different sizes. A "summarize this sentence" call and a "read these 40 pages and reason about them" call hit the same endpoint, and one of them costs a hundred times more. When I built Scholarian, a deep-research pipeline over academic papers, it ended up fetching and ranking over 10,000 papers across 250+ search sessions. Some sessions were one cheap query. Some were a long chain of very expensive ones. A rotation cannot tell those apart, so it cheerfully fires the expensive one at the route that is already drowning.

The limits are not counted in requests. Model providers rate-limit you on requests per minute and tokens per minute, and it is usually the token ceiling you hit first. So a balancer counting requests is watching the wrong meter, and you find out about it through a 429 error in production.

The backends are not yours. You cannot SSH into OpenAI. There is no CPU graph, no memory reading, nothing except the latency and error rate you observe from outside. Observed behavior is the only signal you have, and observed behavior is exactly what adaptive balancing runs on.

Every API key is its own bottleneck. Rate limits are per key, so teams end up holding several keys per provider. Now you are not balancing across servers anymore, you are balancing across a grid of providers and keys, each with separate limits and separate health.

What adaptive load balancing looks like inside an AI gateway?

The gateways handling this properly split the decision in two.

The first level picks the provider and model for a request, based on live capacity and error rates across all of them. The second level picks which API key inside that provider actually gets used, weighted by how each individual key is performing.

That two-level shape matters because the failures are different at each level. A provider goes down for everybody at once. A single key just quietly hits its own token ceiling while its siblings are perfectly fine. One balancer trying to handle both would be making the wrong call half the time.

I went through the practical side of this in my post on Bifrost's enterprise features, where the adaptive routing sits right next to audit logs and guardrails. Worth a read if you want the version with an actual dashboard in front of you.

When you should not reach for adaptive load balancing

You need real telemetry before you need adaptive routing. At a contract role earlier this year I built an internal tool that captured our AI product's logs and generated reports on latency and probable slowdowns, and the uncomfortable lesson was that most of the wins came from simply seeing the numbers. Half the routing problems people want an adaptive balancer to solve turn out to be one bad prompt template or one undersized instance, and a dashboard finds those faster than an algorithm hides them.

Low traffic means no signal. Scoring on error rate and latency needs enough requests per window to mean anything. At 5 requests a minute, an adaptive balancer is mostly reacting to noise, and reacting to noise is worse than not reacting at all.

It can paper over a capacity problem. If every route is degraded, adaptive balancing will smoothly and confidently send you to the least-bad option, forever, while the real answer was "add capacity" or "get off the free tier."

And it is one more moving part. More state, more tuning, one more thing to reason about at 3am. If round robin across two identical boxes is working for you, keep it.

Choosing the right routing strategy

If you run a plain web service on infrastructure you control, least connections is fine and adaptive load balancing is over-engineering.

If you are routing to model providers, it is not optional anymore. You are balancing across backends you cannot inspect, with limits measured in tokens, with per-key ceilings, and with failure modes that arrive as a slow degradation instead of a clean crash. A fixed rotation has no mechanism to even notice that. Use a gateway that scores routes on live behavior and moves the traffic for you, and go spend that attention on your product instead. And also, if what Bifrost is doing intrigued you, you can easily book a demo, and see how it fits in your organization.

That is my take, and your setup might look nothing like mine. If you have built this kind of routing yourself, or you have watched an adaptive balancer make a genuinely stupid decision, drop it in the comments, I want to hear it.

You can find me on X where I post about most of what I am building, and the rest of my writing lives at swapnoneel.site.

Bifrost for Enterprises: Audit Logs, MCP Tool Groups, Adaptive Routing, and Guardrails

Swapnoneel Saha — Sun, 09 Aug 2026 15:41:05 +0000

Bifrost is Maxim AI's high-performance, open-source AI gateway that unifies access to 20+ providers through a single OpenAI-compatible API. Bifrost Enterprise is a strict superset of that gateway, adding high-availability clustering, adaptive load balancing, identity and access controls, audit logs, guardrails, and private deployment options.

Those additions matter when several teams share the gateway. Routing rules, access policies, MCP tool permissions, and administrative changes can be managed and audited in one layer instead of being implemented separately in every AI application.

After writing about the open-source gateway in my previous two posts, I asked the Bifrost team for access to the Enterprise version. They gave me limited access to the paid features for free. This article covers what I tested and whether those features make sense for personal and enterprise use.

Transparency through Audit Logs

So, before creating any new Enterprise configuration, I wanted to see whether Bifrost could actually tell me what was happening behind the scenes. Audit Logs tell you exactly that. It’s different from the regular LLM logs because they show the requests going through the gateway, while Audit Logs focus on changes and administrative activity inside Bifrost. So if someone creates a virtual key, changes a routing rule, updates a guardrail, or modifies the cluster configuration, this is where we should be able to find it.

For a personal setup, audit logs may be more detail than you need. In a company, they give security and platform teams a record of who changed shared routing, access, and guardrail configuration.

What are MCP Tool Groups?

In my previous blog, I already mentioned how I connected the MCP Gateway to OpenCode using the Virtual Key, which enabled Bifrost to expose all of my configured MCP tools through that one endpoint.

But that also raised a question: do I really want every harness to have access to every tool? So, this time I decided to try MCP Tool Groups.

The idea is pretty simple. We can create a group of selected MCP tools and attach that group to a virtual key. OpenCode already uses my dedicated Enterprise virtual key, so I can control the tools available to it without changing the rest of my MCP setup.

In the previous blog, you might have seen that I’ve used the Context7 MCP server. It provided access to two tools:

resolve-library-id
query-docs

So, while creating the tool group, I decided to drop the query-docs and kept only the resolve-library-id activated.

Then, under Associations, I attached the group only to my opencode-enterprise virtual key. I didn't attach it to any teams, customers, providers, or other keys.

The OpenCode MCP configuration itself didn't need much change. It was already pointing to Bifrost's remote MCP endpoint:

{
  "mcp": {
    "bifrost": {
      "type": "remote",
      "url": "https://bifrost-enterprise.agitracker.io/mcp",
      "enabled": true,
      "oauth": false,
      "headers": {
        "Authorization": "Bearer {file:./bifrost-virtual-key}"
      }
    }
  }
}

I kept the virtual key inside a separate local file, so it never had to be pasted into the configuration or committed to Git.

After restarting OpenCode, I used a deliberately small prompt,

Use the Context7 MCP to resolve the React library. Reply with only the returned library ID.

The request executed successfully, as you can see in the above screenshot.

So, is this feature useful? The idea is excellent, especially when different coding harnesses should have access to different MCP tools. And, it gives you a central place to manage tool access instead of duplicating MCP configuration across every client.

What is Adaptive routing?

The next feature I wanted to try was Adaptive Routing. The idea behind it is quite useful, especially if you are running several models, providers, or API keys through the same Bifrost gateway.

Normally, requests are distributed using fixed weights. For example, if two Gemini keys have the same weight, Bifrost can send roughly half of the traffic to each one. The problem is that fixed weights do not know whether one key has become slower, started returning errors, or hit a rate limit.

Adaptive Routing tries to solve that automatically.

Bifrost monitors the latency, error rate, success rate, and utilization of each available route. It then recalculates their weights every few seconds. And, a healthy and faster route receives more traffic, while a failing or slow route receives less. Bifrost still sends a small amount of traffic to recovering routes so it can detect when they become healthy again.

The routing happens at two levels:

Bifrost can select which provider should handle a model request.
After selecting the provider, it can choose the best API key configured for that provider.

This makes the feature more useful for companies that maintain multiple provider accounts or keys. Instead of manually changing weights whenever a provider starts acting up, Bifrost can react to the recent performance data on its own.

How to set Guardrails?

Provider safety controls do not cover every organization's policies. Putting custom guardrails at the gateway lets the same input and output checks run across every model behind it.

This feature is meant to protect both the prompts sent to a model and the responses coming back from it. Bifrost separates the feature into two parts: rules decide when a check should run, while profiles define what kind of check should be performed. For this test, I wanted to avoid adding another external API key, so I chose Bifrost's built-in Custom Regex provider. According to the Guardrails documentation, Custom Regex runs locally and can be used for deterministic pattern checks.

So, I created two guardrail rules. One for the input, and the other one for the output.

So, now if I send a request like Reply with exactly: BIFROST_GUARDRAIL_TEST, I get a regex pattern matched error. That’s where the guardrail is actually doing its job.

The regex rule blocked the matching output without calling an external guardrail service.

Final thoughts

When I started exploring Bifrost Enterprise, I expected the paid version to feel like the open-source gateway with a few extra switches. That wasn’t true.

The core experience stayed familiar, and I could continue using OpenCode through one Bifrost endpoint while the gateway handled the provider connection underneath. That part was convenient. I did not need to change my workflow every time I switched between OpenAI and Gemini.

The enterprise features that made the most sense to me were Audit Logs and MCP Tool Groups. Audit Logs give teams a central record of what happened, while MCP Tool Groups make it easier to control which tools a client can access.

Would I use Bifrost Enterprise for my personal setup? Probably not if I only had one provider, one API key, and a handful of requests. The extra governance and operational features would be more machinery than I need.

For a team running several models, provider keys, MCP clients, and internal users, the situation is different. A shared gateway, centralized logs, access controls, guardrails, and tool restrictions can remove a lot of repeated setup from individual applications. And if you think you are the right candidate, you can always book a demo!