DEV Community: Swapnoneel Saha

Top 5 AI Agent Governance Platforms in 2026

Swapnoneel Saha — Sat, 22 Aug 2026 12:58:20 +0000

Most teams shopping for AI agent governance in 2026 end up buying a dashboard, and then find out six months later that their actual problem was an agent holding a database credential with no spending cap on it.

AI agent governance platforms decide what your autonomous agents are allowed to do and how much they can spend doing it. The category splits into five layers, and no single product covers all five well. The five that matter in 2026 are Bifrost, Microsoft Agent 365, Zenity, Arthur AI, and Credo AI.

I have ranked them by the thing most lists skip: how much each one can actually stop, in real time, before a bad action lands. Not how good the report looks afterwards.

What AI agent governance actually means

Let's start from first principles, because "governance" is a word that has been stretched until it means almost nothing.

Traditional software governance assumes a human is somewhere in the loop. A person logs in, a person clicks the button, and the audit log records which person did it. Permissions attach to that person.

An AI agent breaks that assumption in a very specific way. The agent holds the credential, decides on its own which tool to call, and calls it a few hundred times while you are asleep. There is no click to attribute.

So governing an agent means controlling five separate things, and they really are separate:

Identity. Which agent is this, who owns it, and what is it allowed to be?
Authorization. Which models, tools, data and systems can it reach?
Spend. How many dollars and how many tokens can it consume before it gets cut off?
Behavior. Was the input a prompt injection, was the output leaking a card number, was the answer just confidently wrong?
Evidence. Can you reconstruct, six weeks later, exactly what it did and why?

Miss any one of those and you don't have governance, you have a partial view with a nice chart on top of it.

Gartner polled more than 3,400 organizations investing in agentic AI and predicted that over 40% of agentic AI projects will be canceled by the end of 2027, naming escalating costs and inadequate risk controls as two of the three reasons. Notice that both of those are governance failures and not model failures. The model was fine. Nobody had put a limit on it.

And the readiness gap is not subtle. In a Deloitte survey of 3,235 IT and business leaders across 24 countries, only 21% said their organization has a mature governance model in place for agentic AI. So four out of five teams are running agents on trust.

The five control points, walked through one incident

This is the part I wish the other lists on this topic did, so let me do it here.

Take one concrete setup. A customer support agent running on Claude Sonnet, wired to two MCP servers: one for Linear so it can file tickets, and one Postgres connection so it can look up an order. A support engineer built it in an afternoon. It works fine.

Now walk through what goes wrong, as four separate events.

Event one. The Postgres MCP server got registered with a connection string that has write access, because that was the string already sitting in the engineer's .env file. Nobody noticed, because the agent never tried to write anything.

Event two. A customer pastes a support message that contains an instruction addressed to the agent, telling it to look up recent orders for a different email address. This is item one on the OWASP Top 10 for Agentic Applications, published in December 2025, and this is the boring, non-exotic version of it.

Event three. The agent complies, and because it is trying to be helpful, it iterates. It runs the same lookup pattern 4,000 times over a two-hour window, each call carrying a large system prompt plus the full retrieved context.

Event four. Somebody finds out the next morning, from a bill.

So which layer catches which event?

An identity layer catches event one, and only event one. If the agent has an identity with declared scopes, then a read-only agent holding a write-capable credential is a visible contradiction, and you can alert on it. But identity has nothing at all to say about a malicious support message.

A behavior layer, meaning guardrails, catches event two. A prompt injection classifier sitting in front of the model can flag that instruction and refuse the turn. But guardrails run per call, and they are stateless. Four thousand individually-innocent calls are four thousand clean passes.

An evidence layer, meaning tracing and observability, sees all four events, perfectly, in full detail, after they have already happened. That is not a criticism of it, that is the definition of the layer. I wrote a whole post on what you should capture on every agent call and I still think tracing is the first thing you should turn on. It is just not a control.

The only layer that can stop event three while it is happening is the layer that every single call physically passes through, counting tokens as they go, with the authority to return a 402 on call number 900.

That layer is the gateway. Which is exactly why it is number one on this list.

Now, the five AI agent governance platforms

That is the map. Five control points, and one hard rule underneath all of it: a tool can only enforce what it sits in front of.

So here are the five, ordered by how much of the request path each one owns, with what it actually does and where it runs out of road.

1. Bifrost, governance on the request path

Bifrost is Maxim AI's open-source AI gateway, written in Go, and it is at number one here because it is the only entry on this list that sits inside the call. Every model request and every MCP tool call passes through it, which makes it the only one that can refuse.

The governance model is a three-tier hierarchy: customer, then team, then virtual key. A virtual key is the thing your application actually authenticates with, and it is the primary governance entity. One looks roughly like this:

{
  "id": "vk-001",
  "name": "support-agent-prod",
  "value": "sk-bf-*",
  "is_active": true,
  "provider_configs": [
    {
      "provider": "anthropic",
      "weight": 0.5,
      "allowed_models": ["claude-sonnet-4-5"],
      "key_ids": ["anthropic-primary"]
    }
  ],
  "team_id": "team-support-001",
  "expires_at": "2026-12-01T00:00:00Z"
}

Read that config out loud as a sentence and you get the whole point of the layer! This agent may talk to Anthropic, may use exactly one model, may use exactly one of your provider API keys, belongs to the support team, and stops working in December whether anyone remembers it or not.

Three parts of this are worth pulling out properly.

Budgets that cascade

Budgets attach independently at each of the three tiers, and they get checked cumulatively. The virtual key's budget is checked, then the team's, then the customer's, and the request only proceeds if all of them still have room.

{
  "id": "budget-support-vk",
  "virtual_key_id": "vk-001",
  "max_limit": 100.00,
  "reset_duration": "1M",
  "current_usage": 0.0
}

Reset durations run from one minute up to one year (1m, 1h, 1d, 1w, 1M, 1Q, 1Y), and there is a calendar-aligned mode that resets on real UTC month and quarter boundaries instead of a rolling window. That sounds like a tiny detail until you try to reconcile a rolling 30-day agent spend against a finance team's calendar month.

There is also a budget override, with an effective limit that equals the base budget plus the override amount, granted either for a fixed number of cycles or forever. That primitive exists because the alternative, in every organization I have ever seen, is somebody quietly raising the real limit at 2am and never lowering it back.

Rate limits that count tokens, and not just requests

Rate limits live only at the virtual key tier, and they are two fully independent counters:

{
  "token_max_limit": 10000,
  "token_reset_duration": "1h",
  "request_max_limit": 100,
  "request_reset_duration": "1m"
}

That separation is the thing that would have caught event three in the walkthrough. A requests-per-minute cap does nothing against an agent making a modest number of enormous calls. A token cap with its own independent reset window is the control that actually maps to how agents burn money.

And the failures come back as honest HTTP semantics, which matters more than it sounds: 402 for a budget exhausted, 429 for a rate limit hit, 403 for an inactive key, an expired key, or a blocked model. Your client code already knows how to handle those. You are not parsing an error string to find out what happened.

MCP tools as a governed surface

This is the part that almost nothing else in the category does yet.

Bifrost's MCP gateway treats tools the way it treats models, so tool filtering is deny-by-default at three levels. The client config sets which tools exist at all, per-request headers (x-bf-mcp-include-clients, x-bf-mcp-include-tools) narrow that set for a single call, and virtual key filtering overrides both. A key with no MCP configuration gets no MCP tools at all, period! You can also bundle a curated set of tools from several different servers into one MCP Tool Group and attach that group to a key.

Now go back to event one in the walkthrough. A write-capable Postgres tool that the support agent's key was never granted is a tool the agent cannot even see, regardless of what is sitting in anybody's .env file.

For the behavior layer, Bifrost does not try to build its own classifiers. It plugs into AWS Bedrock Guardrails, Azure AI Content Safety, Google Model Armor and Patronus AI, and it emits OpenTelemetry out of the box. It also does adaptive load balancing with provider failover, semantic caching, and multi-node clustering, which is the reliability half of the same control plane.

The numbers. Maxim's own benchmark puts the added latency in the tens of microseconds (the repository claims under 15 microseconds per request at 5,000 requests per second, the product page says 20), with 3.3 GB peak memory, and against LiteLLM it claims 9.5 times the throughput and 54 times faster latency at the 99th percentile, meaning the slowest one request in a hundred. Treat vendor benchmarks as vendor benchmarks, but the architectural reason behind them is real: this is a Go binary in your request path, and not a Python process. The core is Apache 2.0 on GitHub at around 7.4k stars, and you can have it running with one command:

npx -y @maximhq/bifrost

Where it runs out of road. Bifrost governs the call, and it does not discover agents. If a team stands up an agent that talks to OpenAI directly and skips the gateway, Bifrost will never know that agent exists. So it is simultaneously the strongest enforcement point available and the easiest one to route around, which means it only works if you make it the only path out. Also, enterprise RBAC, SSO and audit logs sit in the commercial tier and not in the open-source core.

Pick it if you want a control that can actually say no, and you are willing to make gateway egress mandatory.

2. Microsoft Agent 365 with Entra Agent ID

Microsoft Agent 365 is the most serious attempt yet at solving layer one properly.

The idea underneath it is Entra Agent ID, and it is a genuinely good idea: every agent gets a first-class identity in the directory, the same way every employee has one. Once an agent is a directory object, everything Microsoft already built for humans starts applying to it. Conditional Access policies. Least-privilege scoping. Purview for data classification, Defender for threat detection, Intune for endpoint controls. Agent 365 then adds the registry and the admin hub on top, so you get a real inventory of the agent fleet, including the agents somebody spun up in Copilot Studio and never told you about.

That registry is the answer to the exact problem a gateway cannot solve. Agent sprawl is a discovery problem before it is an enforcement problem, and you cannot govern an agent you have not found yet.

Where it runs out of road. The gravity is real. Coverage is deepest for Copilot Studio and Azure AI Foundry agents and for agents on managed endpoints, and it thins out fast for a Python agent your ML team runs in a container on another cloud. It also does not do token budgets in any meaningful sense. It will tell you an agent exists and what it may touch, and it will not stop that agent at dollar 4,000.

Pick it if you are already an Entra shop, which honestly means pick it anyway, because identity is the one layer you cannot fake.

3. Zenity, posture and runtime action validation

Zenity comes at this from the security side rather than the platform side, and its strength is the part of your estate that nobody has an inventory of.

It discovers agents living inside Copilot Studio, Power Platform, AWS Bedrock and Google Vertex AI, then keeps watching what those agents touch and share. The framing is AI Security Posture Management, and the genuinely useful mechanic is buildtime policy: it flags an over-permissioned connector before the agent ships, which is the cheapest possible moment to catch event one from the walkthrough. Gartner named Zenity a 2025 Cool Vendor in Agentic AI TRiSM (their acronym for trust, risk and security management), and that low-code coverage is why.

So why does the low-code angle matter this much? Well, because that is where the agents you don't know about actually live. An engineer's Python agent is at least in a repository somewhere. A business analyst's Copilot Studio agent with access to a SharePoint site full of contracts is nowhere you are currently looking.

Where it runs out of road. It is a security product, so it governs risk and not cost. There is no budget primitive here and no token accounting at all. And its discovery is strongest in exactly the places Microsoft's is strongest, so if you already have Agent 365, the overlap deserves a hard look before you pay for both.

Pick it if a lot of your agents are being built by people who are not engineers.

4. Arthur AI, guardrails and evaluations without a framework tax

Arthur AI launched its Agent Discovery and Governance platform in December 2025, and the thing I find genuinely interesting about it is that it is framework-neutral in a category that mostly is not.

Discovery works across four vectors: OpenTelemetry streams, MCP server monitoring, network-layer analysis, and the platform APIs of Vertex AI, Bedrock and Azure AI Foundry. And because the tracing is built on OpenTelemetry and OpenInference rather than a proprietary SDK, it works with LangChain, LlamaIndex, raw OpenAI and Anthropic calls, and whatever your team decides to pick next quarter.

On top of that it runs two things that are worth keeping separate in your head. Runtime guardrails, which intercept before and after the model call for PII, prompt injection and toxicity. And continuous evaluations against live production traffic, which is the part that catches hallucination and quality drift, the failures that never throw an error and never show up in a latency graph.

That second distinction is the one I care most about, and it comes from doing the work by hand and getting it half right. On a contract with a startup building an AI co-worker that lives in Slack (keeping it a bit vague, can't reveal much more than this lol), I was the first layer of internal testing, and I built an internal tool that captured the product's data logs and generated reports on latency and probable slowdowns. It was useful, and the core users got a better product out of it. But it told me when the thing was slow, and it never once told me when the thing was confidently wrong. Those are two different systems, and I had only built one of them.

Where it runs out of road. Same ceiling as any observability-first product: it sees everything and blocks a narrow slice of it. The guardrails are per-call, so the accumulation problem from event three stays unsolved, and there is no spend primitive here either.

Pick it if your stack is heterogeneous and output quality is your real risk, rather than permissions.

5. Credo AI, the compliance program of record

Credo AI is on this list for a reason that has nothing to do with runtime, and I would rather be blunt about that than pretend it belongs in the same bucket as the other four.

Its Policy Packs translate regulation into something an engineering team can actually act on, mapping the EU AI Act, ISO 42001 and the NIST AI Risk Management Framework down into concrete requirements. It keeps an AI Agent Registry of what is in production, plus GAIA, an assistant for the governance workflow itself. When a regulator or an enterprise customer's security review asks what your agents do and which controls apply to them, this is the layer that produces the answer.

And the timing here is worth getting right, because a lot of content on this topic is now stale. The Digital Omnibus on AI entered into force on 27 July 2026, and it deferred the high-risk obligations for standalone Annex III systems all the way out to 2 December 2027, and to 2 August 2028 for AI embedded in regulated products. So the deadline a lot of teams panic-bought a platform for has moved.

What did not move is Article 50. The transparency duties landed on 2 August 2026, and those catch every chatbot and every piece of synthetic content, which in practice means they catch most agent deployments regardless of risk tier. The high-risk regime got 16 more months. The disclosure regime is live right now.

Where it runs out of road. Credo AI cannot block a request. It produces policy and evidence, and if you buy it expecting enforcement, you have made the exact purchasing mistake this whole post is about.

Pick it if you sell into regulated industries, or your legal team has started asking questions in writing.

What none of these five actually do

Time for the honest part, because a list that ends on five recommendations and no caveats is just an ad.

Agent-to-agent calls are mostly ungoverned. When agent A hands a task off to agent B, whose budget does it hit, whose identity does it carry, and does B's guardrail see the original user's prompt or A's rewritten version of it? Inter-agent communication is item seven on the OWASP agentic list, and I have not seen anybody solve it convincingly yet. Gateways get closest, because that hop is still an HTTP call, and "closest" is doing a lot of work in that sentence.

Memory is a governance hole. An agent that writes a poisoned fact into its long-term store carries that fact past every per-call guardrail forever, because on every future call it is not an injection anymore, it is just context. Nothing on this list governs writes to agent memory the way it governs writes to a database.

Agent identity is not portable. Entra Agent ID is real and it is good, and it is also Microsoft's. There is no cross-vendor standard yet, so an agent's identity in your directory means nothing to a partner's system. We are roughly where federated human identity was before SAML showed up.

And the buying mistake, which is the whole reason I ordered this list the way I did. A team worried about a runaway bill buys a compliance platform, and a team worried about a regulator buys a tracing tool. Both walk away with a real product doing a real job, and neither one has touched the thing that wakes them up at 3am.

Frequently asked questions

Is an AI gateway enough on its own for agent governance? For spend, authorization and tool access, yes, and it is the strongest control you can get for those three. For discovery and identity, no. A gateway cannot see an agent that does not route through it, so pair it with an identity or discovery layer, and make gateway egress mandatory at the network level rather than by policy document.

Do I need an AI agent governance platform for a small team? You need two of the five control points from day one, and neither of them is a purchase. Put every model call behind one gateway with a per-key spending cap, and turn tracing on. My own side project Scholarian has fetched and ranked over 10,000 papers across 250-plus search sessions, and the reason I care about per-key caps is not enterprise policy, it is that a retry loop in a research pipeline is perfectly capable of spending real money overnight.

What is the difference between AI agent governance and AI observability? Observability tells you what happened, governance decides what is allowed to happen in the first place. They get sold together because tracing is how you prove a policy was enforced, but a trace has no authority. If your platform can produce a beautiful timeline of an incident and could not have prevented that incident, you bought observability.

Where do guardrails fit into all this? Guardrails are the behavior layer, and they run per call, screening inputs for injection and outputs for leakage or toxicity. They are necessary, and they are stateless, which is why they miss failures made out of many individually-fine calls. Run them at the gateway rather than inside each application, so that one policy update covers every agent instead of six repositories.

Does the EU AI Act require an agent governance platform? No regulation names a product. But the Article 50 transparency duties are in force as of 2 August 2026, and the deferred high-risk regime still expects risk management, logging and human oversight when it lands in December 2027. Those obligations are much cheaper to meet if the logging was there from the start.

What would I actually pick?

If I could only run one thing, it is the gateway, and that is Bifrost. Put every model call and every MCP tool call behind it, then set a token budget with its own reset window on every virtual key and deny tools by default. It is the only layer on this list that can turn a policy into a refusal, it is Apache 2.0 so the argument costs you nothing but an afternoon, and at microsecond-scale overhead you are not trading latency for control.

Then add identity second, and not fifth. Agent 365 if you are on Entra, which most enterprises already are. Discovery is the one problem a gateway genuinely cannot solve for you, and most of the rest of this list is a refinement of a control you would already have.

And here's my bet for the next 18 months: this category does not stay five layers wide. Identity and posture will collapse into whoever already owns your directory, evaluations will fold into observability, and the thing that survives as a separate purchase is the request path, because that is the only place enforcement is physically possible. If I turn out to be wrong about that, I'll happily write the follow-up.

That's all from my side, folks. If you are running agents in production and you have found a control that actually stopped something, I want to hear about it, so drop it in the comments. And if you want more of this, I write about AI infrastructure regularly on X and on my site.

What Is Semantic Caching, and Where It Quietly Breaks

Swapnoneel Saha — Wed, 19 Aug 2026 17:39:37 +0000

Two people open your support chatbot within the same minute. One types How do I reset my password? and the other types i forgot my password, how do i get a new one.

Same question, same answer, two full model calls, and your Redis cache stores both as separate keys without ever hitting on either.

Semantic caching stores past LLM responses and serves them to new queries that mean the same thing, even when the words are completely different. It works by turning every query into a vector, searching for the nearest stored vector, and returning that cached answer if the similarity clears a threshold you set.

Now, let's understand how it actually works!

Why your existing cache does nothing for LLM traffic

Every cache you have ever written works on exact equality. You take the request, hash it, look up the hash, and either the bytes match or they don't. Redis and Memcached both work this way, and so does the HTTP layer sitting in front of them. It works brilliantly.

It works because the traffic it was designed for is machine-generated. GET /api/users/42 is always spelled the same way by the same client, every single time.

Human language is not like that. There are roughly infinite ways to ask for a refund policy, and a hash function treats every one of them as a different universe. One extra space, one lowercase letter, one "please" at the end, and you get a completely different key.

So your hit rate on natural language collapses to nearly zero, and you go on paying for the same answer over and over.

Semantic caching fixes the matching function instead of the cache. The storage stays boring. What changes is that you stop asking "are these two strings identical" and start asking "are these two strings close enough in meaning".

And that one word, close, is where all the difficulty in this topic lives.

How does semantic caching actually work?

The whole thing is five steps, and none of them are complicated on their own.

Before any code, here is the rule in plain English:

Turn the question into a point in space. Look for the nearest point we have already answered. If it is near enough, reuse that answer.

Now the actual shape of it:

A query comes in.
You send it to an embedding model, which returns a vector of floats. Usually 768 or 1536 dimensions, depending on the model.
You search your vector store for the nearest stored vector, using cosine similarity.
If the best match scores above your threshold, you return the stored response and never call the model at all. That's a cache hit.
If nothing clears the threshold, you call the model, return the real answer, and write the query vector plus the response into the store with an expiry time.

In code it is almost insultingly short:

def ask(query, threshold=0.92):
    vec = embed(query)                         # step 2
    match, score = store.nearest(vec)          # step 3

    if score >= threshold:                     # step 4
        return match.response                  # cache hit, zero model tokens

    answer = llm.complete(query)               # step 5, cache miss
    store.put(vec, answer, ttl=3600)
    return answer

Notice what step 4 is really doing. It is taking a floating point number and using it to decide whether a human being gets a fresh answer or a recycled one. There is no other logic in this system, no parsing and no intent classification, nothing else that ever looks at what was actually asked.

If you want the mechanics of what store.nearest is doing underneath, I built one of these from scratch, cosine similarity and the HNSW graph and all, in my post on building a vector database from scratch. The short version is that it is an approximate nearest neighbour search, so it is fast, and it is also allowed to be a little bit wrong.

The similarity threshold is the whole product

Let's do something most articles on this topic skip. I'll give you three pairs of queries, you predict what a cosine similarity score should look like for each, and then we will check against what embedding models actually return.

Pair one. What is your refund policy? and Can I get my money back? Different words entirely, same intent. This one should score high, and it does. This is the case semantic caching was built for, and it works.

Pair two. Show me the sales numbers for Q1 2025 and Show me the sales numbers for Q3 2024. Almost identical strings, completely different answers. You would want this to score low. It does not. It scores extremely high, because most of the tokens are shared and the embedding barely notices which quarter you asked about.

Pair three, and this is the one that should worry you. Is this drug safe for pregnant patients? and Is this drug not safe for pregnant patients? One word apart, opposite meaning.

Here's what actually happens. Embedding models are largely blind to negation, and this is not a rumour, it is measured. A validity audit published in August 2026 tested 9 encoder configurations and found that negation and antonym pairs score above genuinely similar pairs on every model tested, at average cosines of 0.93 to 0.999. In the production system that paper audits, flipping an instruction from "withhold the study drug" to "administer the study drug" scored 0.9608, and the safety gate that existed specifically to catch that never fired.

Read that once more. The reversed instruction scored higher than most legitimate paraphrases would.

So the prediction most people carry into this, that a threshold like 0.92 cleanly separates "same question" from "different question", is just wrong. What the threshold separates is surface form, and surface form is not meaning. Two sentences that share a grammatical frame and differ in one date, one entity, or one negation will sit above almost any threshold you are willing to set.

That's the trap. Raise the threshold to 0.98 and you kill your hit rate, because honest rephrasings stop matching. Lower it to 0.85 and you start serving Q3 2024's numbers to someone asking about Q1 2025. There is no single number that fixes both, because the failure is in the measurement and not in the cutoff.

What actually helps is refusing to let similarity be the only gate. Partition the cache by anything the embedding is bad at holding: user, tenant, model, and any structured parameter your queries carry. If dates and IDs are pulled out into the cache key instead of being left sitting inside the prose, the embedding never gets a chance to blur them.

Semantic caching vs prompt caching vs KV caching

These three get used interchangeably online and they are three completely different layers. Getting this straight is worth more than any amount of threshold tuning.

KV caching lives inside the GPU. During inference the model computes key and value tensors for every token in your context, and the KV cache keeps them around so the next token does not need to recompute attention over everything before it. This is always on, you do not configure it, and it is the reason generation gets faster after the first token.

Prompt caching, sometimes called prefix caching, is what OpenAI and Anthropic sell you at the API level. It reuses those KV tensors across requests when two requests share a common prefix. So if you send a 4,000 token system prompt on every call, the provider can skip recomputing it and charges you less for those tokens. Important detail: it matches on exact prefix bytes, so two prompts that mean the same thing but start differently will miss it entirely.

Semantic caching sits in front of the model, in infrastructure you control. It stores whole request and response pairs and matches on meaning. When it hits, you save 100% of the call, because the model is never invoked.

They stack, and they should. A request should try the semantic cache first, fall through to the provider's prompt cache on a miss, and only then pay for full inference. The savings are multiplicative rather than competing, and the layer you own is the outermost one.

What is a realistic cache hit rate?

This is where I want to be blunt, because the marketing on this topic is bad.

You will see 95% quoted constantly. Trace that number back and it almost never refers to hit rate. It refers to match accuracy, meaning the cached response was correct 95% of the time it was served. Those are entirely different claims, and the second one tells you nothing about how much money you saved.

Actual production numbers are much lower. A breakdown of real deployment data puts typical hit rates at 20 to 45%, with Portkey seeing around 20% on retrieval-augmented workloads and an EdTech platform hitting about 45% on student question-and-answer traffic. Open-ended chat sits at 10 to 20%, because open-ended chat is genuinely open-ended.

Academic results land in a similar band. The GPT Semantic Cache paper reports cutting API calls by up to 68.8%, but that is on query categories picked for repetition, which is exactly the workload where this technique looks its best.

And 20 to 45% is still a very good deal! On a 5,000 dollar monthly bill, a 20% hit rate is 1,000 dollars back, and the latency win is bigger than the money win. A cache hit returns in under 5 milliseconds against 2 to 5 seconds for a real completion, which changes what the feature feels like to use, not just what it costs.

Just size your expectations off your own traffic. Which brings me to the honest part.

What semantic caching costs you to run

Nobody puts this in the intro paragraph, so here it is.

You pay an embedding call and a vector search on every single request, including the 60 to 80% that miss. Embeddings are cheap compared to a chat completion, so the money side is fine. The latency is the thing to watch, because you have just added a network round trip to the front of every request in your system, including all the ones the cache cannot help with.

At Keploy I built a retrieval-augmented chatbot over their documentation using vector embeddings, and docs traffic is close to the best case for this technique. People ask the same twenty questions in fifty phrasings, forever. Even there, the honest framing is that you are trading a guaranteed small cost on 100% of requests against a large saving on a minority of them, and you need to actually measure that ratio before assuming it comes out ahead.

Then there is staleness. Your cache does not know your prices changed on Tuesday. The stored answer is a frozen snapshot of what the model said, plus whatever context it was given at the time, and it keeps being served until its expiry time runs out. Short expiry times are safer and hit less. Long ones are the opposite. Pick deliberately.

Multi-turn conversations are worse. A follow-up like "and what about the second one?" embeds to almost nothing useful, because the meaning lives in the previous four messages and not in that sentence. Most sane implementations just refuse to cache beyond a few turns of history, and that is the correct call.

And you now operate a vector store. That is one more thing to size, monitor, and pay for.

One cache, many tenants, and the leak nobody plans for

This is the part that turns semantic caching from a performance feature into a security decision, and it is why I would not hand-roll one at the application layer in an enterprise setting.

A semantic cache with one global namespace returns the nearest previous response across every user in it. Not the nearest response belonging to that user. The nearest one, period.

So picture two customers of the same SaaS product asking structurally similar questions about their own account data. Their prompts embed within 0.93 cosine of each other, because they are the same question about different companies, and the cache cheerfully hands one customer the other's cached answer. Nothing errored. Nothing logged a violation. It looks exactly like a successful cache hit, which is the worst property a data leak can possibly have.

There is a subtler version too. Even with no wrong response served, cache hits are dramatically faster than misses, and that timing difference is observable from outside. Somebody probing your API can learn which questions have already been asked by other tenants just by watching time to first token.

The fix is not clever, it is structural. The cache key has to be partitioned by a tenant identifier resolved from something you trust, meaning the API key, a virtual key, or a signed token claim. Never from the request body, because the request body belongs to the attacker. And the lookup has to be scoped to that namespace so a cross-tenant match is not merely unlikely, it is unreachable.

If that sounds like something you would rather not rebuild inside every service you own, I agree with you, and that is the real argument for doing this at the gateway.

When should you not use semantic caching?

Some genuine "don't bother" cases, because this is not free and it is not universal.

Anything that must be current. Live inventory, account balances, order status. A stale answer here is worse than a slow one.

Anything where the parameters matter more than the phrasing. Analytical queries over dates, IDs, and entities are precisely where embeddings blur the thing you needed preserved. If you cannot pull those values out into the cache key, skip it.

Anything high-stakes and low-volume. Medical, legal, financial advice. The negation research above is not a curiosity in those domains, it is a lawsuit. And if your volume is low, you were not saving much anyway.

Long open-ended conversations. A 10 to 20% hit rate while adding latency to 100% of requests is a bad trade.

Where it does earn its place: support bots, docs assistants, FAQ layers, onboarding flows, internal knowledge search, and any product where a large user base asks a small set of questions in a lot of different ways. That describes a very large share of enterprise AI traffic, which is why this matters at all.

How Bifrost does semantic caching at the gateway

I have been running Bifrost as my AI gateway for a while now, and its semantic cache is the cleanest implementation of everything above that I have read, mostly because I could actually read it. The whole thing is open source on GitHub under Apache 2.0, so none of what follows is a claim you have to take on trust.

The design choice I like most is that it is two layers, not one.

Every request first goes through a direct hash lookup. If the prompt is byte-identical to something already cached, it returns immediately with zero embedding overhead, which matters because you just skipped the round trip that would otherwise tax every request in the system. Only on a direct miss does it embed the query and run the similarity search. So the cheap path stays cheap, and the expensive path only runs when it might actually pay off.

Then there is the thing that answers the multi-tenancy section directly. Caching only activates when a request carries a cache key, passed as an x-bf-cache-key header. There is no global-namespace mode for you to accidentally ship. If you want per-tenant isolation, the tenant identifier goes in that header, and cross-tenant matches then cannot happen at all, because those entries are not in the same partition. Making the safe thing mandatory instead of optional is a real design decision, and most implementations get it wrong.

A few more of the knobs, since the defaults tell you what the authors actually believe:

threshold defaults to 0.8 for semantic hits, overridable per request with x-bf-cache-threshold.
ttl, the time to live on an entry, defaults to 5 minutes. That is a deliberately conservative staleness stance, and you can override it per request too.
conversation_history_threshold defaults to 3, which means it stops caching once a conversation runs past 3 messages. That is exactly the multi-turn failure I described earlier, handled by default.
cache_by_model and cache_by_provider are both on by default, so a cached GPT answer never gets served to a Claude request.
The vector store is pluggable across Redis or Valkey, Weaviate, Qdrant, and Pinecone, so you are not forced into adopting a new database.

And every response carries a cache_debug block with cache_hit, hit_type (direct or semantic), the actual similarity score, and a cache_id. That last one is what makes invalidation possible, since you can delete a single poisoned entry by its ID, or clear an entire partition by cache key, straight over the API. If you have ever had to explain to a customer why the bot gave them the wrong answer twice, you will understand why having that similarity score visible per request is worth a lot.

The cost story is the obvious one, and the numbers are the ones from earlier in this post rather than anything I can promise you. Every cache hit is a completion you did not pay for at all. What a gateway changes for enterprises is that the cache now sits next to budgets, virtual keys, and routing in one place, so the same layer deciding which provider gets a request is also deciding whether the request needs a provider at all. That routing side is a whole topic of its own, and I wrote it up separately in what adaptive load balancing actually is.

Bifrost adds under 100 microseconds of overhead at 5,000 requests per second, which is a rounding error next to the embedding call, and honestly next to anything else in an LLM request path.

What should you actually do first?

Do not start by building the cache.

Start by measuring how repetitive your traffic actually is. Take a week of your logs, embed the queries, and count how many of them land within 0.92 of an earlier one. That single number is your ceiling, and getting it takes an afternoon. I did a version of this on a contract a while back, building an internal tool that captured an AI product's logs and turned them into reports on latency and probable slowdowns (keeping it vague here, can't say much more than that lol). The thing I took away from it is that you learn more from one honest week of your own traffic than from every benchmark on the internet.

If that number comes back at 30% or better, turn semantic caching on at your gateway, partition the cache key by tenant from day one, keep the expiry short, and watch the similarity scores on your hits for the first few weeks instead of trusting the threshold.

If it comes back at 8%, you have found a much more interesting problem than caching, which is that your users are all asking different things.

And if you have run a semantic cache in production and watched it serve something it absolutely should not have, please tell me about it in the comments, those stories are the useful ones. I write more about LLM infrastructure and building with AI over at swapnoneel.site, and I'm on X (swapnoneel123) if you feel like arguing about thresholds.

AI Observability Explained: What It Is and How It Works

Swapnoneel Saha — Tue, 18 Aug 2026 06:25:25 +0000

Traditional monitoring rests on one quiet assumption that nobody ever writes down: the same input gives you the same output. Something breaks, you replay the request, you watch it break again, you fix it.

Now send the same request to a model twice. You get two different answers, and neither one of them threw an error.

AI observability is the practice of recording what happened inside an AI system on every request: the prompt, the model version, tokens, cost, latency, tool calls, and a judgement of whether the output was any good. Monitoring tells you the service is up. Observability tells you why it answered that way.

That gap is the whole story here.

Why your current monitoring stack misses all of this

Your existing setup is watching for crashes. Status codes, error rates, p99 latency, memory. All of it is designed around the idea that a broken thing looks broken.

An AI feature failing looks nothing like that. It returns HTTP 200 in 900ms, with grammatically perfect prose that happens to be wrong, or that quietly ignored the document you retrieved for it, or that called the refund tool when the user only asked a question.

Your dashboard sees a healthy service, because by every measure it has, the service is healthy.

And there are whole categories of failure your stack has no field for. It has nowhere to put "this response cost 14 cents", or "the model version changed under us last Tuesday", or "the retrieved context was garbage". Those are not infrastructure facts, and standard telemetry was never built to carry them.

Something has to hold those fields instead, which is the entire reason this tooling exists. My team uses Bifrost, so I will use it as the example throughout this post. It's an open-source AI gateway from Maxim, so anything I claim about what it records per request is something you can go check line by line. Most tools here put their telemetry story on a marketing page and stop there.

What one AI request actually looks like when you trace it

This is the part that made it click for me, so let me walk through a real shape.

At Keploy I built a retrieval-augmented chatbot over their documentation, using vector embeddings, so developers could ask a question instead of hunting through pages. A single question to something like that is not one operation. It is a chain, and a trace is just that chain written down.

One request breaks into spans, where a span is one step with its own start time, end time, inputs and outputs:

The user's question comes in and opens the root span.
The question gets embedded into a vector. That is a span, with its own model and its own cost.
The vector search runs and returns, say, five chunks of documentation. That is a span, and the important bit is that it records which five chunks came back.
Those chunks get stuffed into a prompt template along with the chat history.
The model call goes out. This span carries the model name and version, the temperature, the prompt tokens, the completion tokens, the cost in dollars, the total latency, and the time to first token.
If the model calls a tool, every one of those is its own child span too.

Now here is why anybody bothers with all that plumbing.

When the bot gives a bad answer, you do not have to guess. You open the trace and look at step 3. If the vector search pulled back five irrelevant chunks, your problem is chunking or embeddings, and the model did nothing wrong. If the search pulled back exactly the right documentation and the model still answered from thin air, your problem is the prompt.

Two completely different fixes, and without the trace you cannot tell them apart. All you have is "the bot said something dumb", which is the single most useless bug report in the world.

The things worth capturing on every call

You will notice I have not called this section "the three pillars of observability". Everyone else writing about this does, and I dropped it on purpose, because logs, metrics and traces is a framing built for deterministic systems and it has no slot at all for "was the answer any good".

So here is the actual list:

The full prompt and the response, as they really went over the wire, after every template and system message got assembled. Not the template, the final text.
Model, version and parameters. Providers ship silent updates. If you cannot say which exact version answered a request, you cannot explain last month's regression.
Tokens in, tokens out, and cost in dollars per request, attributed to a user or a feature.
Latency, split into total time and time to first token. Those two numbers feel completely different to a user, and one can get worse while the other improves.
Tool calls, retries and fallbacks. Which key was tried, what failed, what it fell back to.
A trace ID that ties the whole chain together, and ideally a session or user identifier so you can reconstruct a full conversation.
A quality score, attached after the fact. More on that next.

I built one of these myself earlier this year, an internal tool at a contract role that captured an AI product's logs and turned them into reports on latency and probable slowdowns (keeping it vague on purpose, cannot say much more than that). The honest takeaway was not that the tool was clever. It was that a team can ship for months on vibes, and the moment somebody puts the per-request numbers on a screen, problems nobody previously had words for suddenly have words.

How do you measure quality when there is no right answer?

Well, you do not measure it the way you measure a unit test, because there is no expected string to compare against.

The industry has mostly settled on three overlapping things. LLM-as-judge, where you send the input and output to a second model with a rubric and it scores relevance or faithfulness or tone. It is imperfect, and it is far better than nothing. Human annotation on a sample, which is slow, expensive, and still the ground truth everything else gets calibrated against. And implicit user signals, like thumbs, edits and retries, which are noisy but free.

Run those continuously and you get drift detection, which is just the same score measured over time. When your faithfulness score drops 8 points over two weeks and nobody deployed anything, something moved underneath you, and that is usually the model provider.

This is the half most teams skip, and there are numbers on it. In LangChain's 2026 State of Agent Engineering report, which surveyed 1,300+ practitioners, 89% said they had observability running on their agents while only 52% were running evaluations. So most people are recording what happened and still have no systematic opinion on whether it was good.

Which is also why I keep saying you cannot test an AI feature the way you test code. I went into that failure mode properly in my post on testing AI coding agents.

Where AI observability actually lives in your stack

Two choices here, and you can do both.

You can instrument your application directly, wrapping every model call in your own code. That gives you the most context, because your code knows what the user was doing. It also means every service, every language and every framework has to be instrumented separately, and someone has to keep it consistent.

Or you put it in the gateway. If all your model traffic already goes through one proxy, that proxy sees every request and every response by definition, and you get telemetry for services you never touched.

And the reason you can do both without doubling the work is that there is finally a shared standard. The OpenTelemetry GenAI semantic conventions define agreed attribute names for exactly this, like gen_ai.request.model, gen_ai.usage.prompt_tokens and gen_ai.usage.cost. Emit those and your AI spans slot into the same traces as the rest of your system, in whatever backend you already pay for.

Bifrost is a reasonable thing to look at here, since it does both halves. It records inputs, outputs, tokens, cost and status for every call into SQLite or Postgres with a dashboard on top, and it exports OpenTelemetry spans using those GenAI conventions plus native Prometheus counters like bifrost_input_tokens_total and bifrost_cost_total. The logging runs in background goroutines, which is why its documentation puts the added overhead under 0.1ms per request.

That last detail is the pattern to steal, whichever tool you end up picking. Telemetry gets emitted off the hot path, after the response is already on its way back to the user. Observability that slows down the thing it observes gets switched off within a week.

The routing side of that same gateway is worth knowing about too, and I covered it in my post on adaptive load balancing.

What it costs you to run

Now the uncomfortable part, because none of this is free.

Storage adds up fast. You are storing full prompts and full responses, and prompts got long. A retrieval app can easily push 8,000 tokens of context per call. At real traffic that is a serious volume of text, and this is where sampling comes in: keep 100% of errors and slow requests, keep a small percentage of the healthy ones.

Your prompts contain user data. Every support chat, every uploaded document, every email a user pasted in. The moment you log all of it, your observability store is now a system holding personal data, with all the retention and access rules that implies. Redact at the point of capture, not later.

And someone has to actually look at it. This is the one that quietly kills the whole effort. The traces get collected, the dashboard gets built, nobody opens it, and six months later it is a very expensive write-only database.

Frequently asked questions

Is AI observability the same as LLM monitoring? Close, and monitoring is the narrower one. Monitoring tracks known metrics like uptime, latency and error rate, and answers "is it working". Observability keeps enough per-request detail that you can answer questions you had not thought of yet, like "why did this one user get that answer". In practice most tools sell both under one name.

Do I need OpenTelemetry for this? No, but it is the sensible default in 2026. The GenAI semantic conventions mean your AI spans use the same attribute names everywhere, so you can change vendors without reinstrumenting, and your model calls appear inside the same traces as your database queries. Note that parts of the spec are still marked experimental, so pin your versions.

What is the difference between observability and evals? Evals are the measurement, observability is the pipe. Evals score whether an output was good; observability captures the request, the context, the cost and the trace so the score has something to attach to. You can run evals offline in CI against a fixed dataset, but you can only run them on real traffic if the traffic is being recorded.

So what should you actually do?

If you have an AI feature in production right now and you cannot pull up the exact prompt, the model version and the cost of a request from last Tuesday, that is the gap, and it is worth a day of your week.

Start with capture, and not with dashboards. Get every request logged with its prompt, response, model, tokens, cost and a trace ID, put it wherever you already look at data, and give it two weeks. You will find something. Everyone does. Quality scoring, drift alerts and per-feature cost budgets are worth adding later, but every one of them sits on top of the boring capture layer, so there is no point doing them first.

If your model calls already go through a gateway, turn on the telemetry it ships with before you write any of this yourself. That is the single highest-value hour available to you here, and it is mostly a config change.

That is my read on it, and your setup might look nothing like mine. If you have built this kind of tracing yourself, or you have had an AI observability bill genuinely surprise you, drop it in the comments, I would like to hear how it went.

You can find me on X where I post about most of what I am building, and the rest of my writing lives at swapnoneel.site.

What is Adaptive Load Balancing, and why AI needs it

Swapnoneel Saha — Mon, 17 Aug 2026 17:53:52 +0000

Five identical servers sitting behind one load balancer, each getting exactly one-fifth of the requests. So why is one of them pinned at 90% CPU while another one sits half idle?

Well, because an equal share of requests is not an equal share of work.

Adaptive load balancing is a routing strategy that picks a destination using live health signals like error rate, latency and utilization, instead of a fixed rotation. The balancer keeps scoring every backend while traffic flows, shifts weight toward the ones behaving well, and pulls weight away from the ones going bad.

That's the definition. But the definition is the boring part, so let's get into what the balancer is actually measuring, how fast it reacts, and why this suddenly matters a lot more in 2026 than it did five years ago.

What is a load balancer?

A load balancer is just a thing sitting in front of your servers, deciding which one gets the next request. That's it.

The simplest version is round robin. Request 1 goes to server A, request 2 to server B, request 3 to server C, then back to A again. It's a rotation, and it never once looks at what is actually happening inside those servers.

Static algorithms like this quietly assume two things: every request costs the same, and every server has the same capacity right now. Both assumptions survive about five minutes of real production traffic.

Think of a supermarket. Round robin is the sign saying "next customer to the next till, in order." Adaptive is a floor manager who watches which till is genuinely moving, spots the one stuck behind a price check, and sends people elsewhere.

Same queue, very different Saturday.

How does an adaptive load balancer decide where to send a request?

It collects signals, turns them into a score per route, and turns those scores into weights.

Three signals do most of the work:

Error rate. Is this backend returning failures? Usually the heaviest signal, and usually time-decayed, so a spike from ten minutes ago stops dominating the decision.
Latency. How slow is it right now, both against its peers and against its own recent baseline? A route that always takes 2 seconds is fine. A route that usually takes 200ms and is now taking 2 seconds is in trouble.
Utilization. How much of its capacity is already committed, so that no single fast route gets hammered into becoming a slow one.

Those collapse into one number per route, and a higher number means a bigger share of the traffic.

And the weights are not recalculated per request, because that would drop real work onto the hot path. They get recalculated on a background loop, and each incoming request simply reads the numbers that were computed a moment ago.

Bifrost, the open-source AI gateway from Maxim, is the clearest published example of this that I have come across. Its adaptive load balancer scores routes on error penalty (50% of the score), a token-aware latency score (20%), and utilization (5%), plus a momentum bias that speeds up recovery once a bad route starts behaving again. Weights recalculate every 5 seconds, and route selection adds under 10 microseconds to the hot path, per its documentation.

The entire source code is on GitHub, so you can go read how the scoring is implemented rather than taking a feature page's word for it. Most load balancers describe their algorithm as "intelligent" and then stop talking.

The other half of the mechanism is state. A good adaptive balancer doesn't just have a dial, it has an opinion about what each route currently is: healthy, degraded, failed, or recovering.

And the recovering state is the one people forget. A route that failed does not get cut off forever, it gets a thin trickle of live traffic so the balancer can find out when it is better. Without that, your balancer is just a fancy circuit breaker that never closes again.

Adaptive load balancing vs round robin and least connections

Everyone puts these in a table. I would rather just tell you where each one stops working.

Round robin rotates blindly. Fine when every server and every request is genuinely identical, which is basically never.

Weighted round robin lets you say "server A is beefier, give it double." Better, but you set those weights by hand, based on what was true when you deployed. It has no idea what is true at 3am during a traffic spike.

Least connections picks whichever server has the fewest open connections. This one is genuinely dynamic, and it is a solid default. But an open connection is a rough proxy for load, since one connection doing heavy work counts exactly the same as one connection idling.

Least response time, usually implemented with an exponentially weighted moving average of latency, gets close to adaptive. It measures the thing you actually care about.

Adaptive load balancing is the version that stops relying on any single number. It combines errors, latency and capacity, keeps a health state per route, and has explicit behavior for pulling a route out and easing it back in.

Does the extra machinery pay off? Envoy's own benchmark for its Peak EWMA policy puts it at a 99.9% success rate under a 1-second timeout, against 99% for least-loaded and 95% for round robin (Envoy docs). That gap between adaptive and round robin is made entirely of user-visible failures.

Alright, so far this is a decades-old idea

And it genuinely is. Adaptive load balancing has been in networking gear and reverse proxies forever, and if you run a normal web app behind NGINX, least connections is probably good enough and you can stop reading here.

So why is the term suddenly everywhere again?

Because AI traffic breaks nearly every assumption the older algorithms were built on. That is where this stops being system-design trivia and starts being your on-call pager, so let's get into it.

Why LLM traffic makes static load balancing fall apart?

Four things go wrong at once.

Requests are wildly different sizes. A "summarize this sentence" call and a "read these 40 pages and reason about them" call hit the same endpoint, and one of them costs a hundred times more. When I built Scholarian, a deep-research pipeline over academic papers, it ended up fetching and ranking over 10,000 papers across 250+ search sessions. Some sessions were one cheap query. Some were a long chain of very expensive ones. A rotation cannot tell those apart, so it cheerfully fires the expensive one at the route that is already drowning.

The limits are not counted in requests. Model providers rate-limit you on requests per minute and tokens per minute, and it is usually the token ceiling you hit first. So a balancer counting requests is watching the wrong meter, and you find out about it through a 429 error in production.

The backends are not yours. You cannot SSH into OpenAI. There is no CPU graph, no memory reading, nothing except the latency and error rate you observe from outside. Observed behavior is the only signal you have, and observed behavior is exactly what adaptive balancing runs on.

Every API key is its own bottleneck. Rate limits are per key, so teams end up holding several keys per provider. Now you are not balancing across servers anymore, you are balancing across a grid of providers and keys, each with separate limits and separate health.

What adaptive load balancing looks like inside an AI gateway?

The gateways handling this properly split the decision in two.

The first level picks the provider and model for a request, based on live capacity and error rates across all of them. The second level picks which API key inside that provider actually gets used, weighted by how each individual key is performing.

That two-level shape matters because the failures are different at each level. A provider goes down for everybody at once. A single key just quietly hits its own token ceiling while its siblings are perfectly fine. One balancer trying to handle both would be making the wrong call half the time.

I went through the practical side of this in my post on Bifrost's enterprise features, where the adaptive routing sits right next to audit logs and guardrails. Worth a read if you want the version with an actual dashboard in front of you.

When you should not reach for adaptive load balancing

Now the honest part, because none of this is free.

You need real telemetry before you need adaptive routing. At a contract role earlier this year I built an internal tool that captured our AI product's logs and generated reports on latency and probable slowdowns, and the uncomfortable lesson was that most of the wins came from simply seeing the numbers. Half the routing problems people want an adaptive balancer to solve turn out to be one bad prompt template or one undersized instance, and a dashboard finds those faster than an algorithm hides them.

Low traffic means no signal. Scoring on error rate and latency needs enough requests per window to mean anything. At 5 requests a minute, an adaptive balancer is mostly reacting to noise, and reacting to noise is worse than not reacting at all.

It can paper over a capacity problem. If every route is degraded, adaptive balancing will smoothly and confidently send you to the least-bad option, forever, while the real answer was "add capacity" or "get off the free tier."

And it is one more moving part. More state, more tuning, one more thing to reason about at 3am. If round robin across two identical boxes is working for you, keep it.

So what's the final message?

If you run a plain web service on infrastructure you control, least connections is fine and adaptive load balancing is over-engineering.

If you are routing to model providers, it is not optional anymore. You are balancing across backends you cannot inspect, with limits measured in tokens, with per-key ceilings, and with failure modes that arrive as a slow degradation instead of a clean crash. A fixed rotation has no mechanism to even notice that. Use a gateway that scores routes on live behavior and moves the traffic for you, and go spend that attention on your product instead. And also, if what Bifrost is doing intrigued you, you can easily book a demo, and see how it fits in your organization.

That is my take, and your setup might look nothing like mine. If you have built this kind of routing yourself, or you have watched an adaptive balancer make a genuinely stupid decision, drop it in the comments, I want to hear it.

You can find me on X where I post about most of what I am building, and the rest of my writing lives at swapnoneel.site.

Bifrost for Enterprises: Adaptive Routing, Guardrails and much more

Swapnoneel Saha — Sun, 09 Aug 2026 15:41:05 +0000

As you all might have seen, in the past two blogs, I wrote about how I explored the different features of Bifrost, and how each one of them improved my workflow, and how I interact with different harnesses through one common gateway. If you haven’t read them yet, go check them out from here. So after exploring all the free features, I was getting the urge to try the Enterprise version as well (for my personal use, though). So, I contacted the Bifrost team, and thanks to them, they gave me limited access to try out their paid features for free!! So, in this blog, I will be exploring the most prominent paid features, and would give an honest verdict on whether it’s great for personal use or not, or whether you should even give it a try for your enterprise use case.

Transparency through Audit Logs

So, before creating any new Enterprise configuration, I wanted to see whether Bifrost could actually tell me what was happening behind the scenes. Audit Logs tell you exactly that. It’s different from the regular LLM logs because they show the requests going through the gateway, while Audit Logs focus on changes and administrative activity inside Bifrost. So if someone creates a virtual key, changes a routing rule, updates a guardrail, or modifies the cluster configuration, this is where we should be able to find it.

It is not the most exciting feature on its own, and for personal use, this might not be that useful, but in an enterprise setting, it is probably one of the most important ones to have, when you are working with a large group of people, and you have to keep tabs on everything that’s going on.

What are MCP Tool Groups?

In my previous blog, I already mentioned how I connected the MCP Gateway to OpenCode using the Virtual Key, which enabled Bifrost to expose all of my configured MCP tools through that one endpoint.

But that also raised a question: do I really want every harness to have access to every tool? So, this time I decided to try MCP Tool Groups.

The idea is pretty simple. We can create a group of selected MCP tools and attach that group to a virtual key. OpenCode already uses my dedicated Enterprise virtual key, so I can control the tools available to it without changing the rest of my MCP setup.

In the previous blog, you might have seen that I’ve used the Context7 MCP server. It provided access to two tools:

resolve-library-id
query-docs

So, while creating the tool group, I decided to drop the query-docs and kept only the resolve-library-id activated.

Then, under Associations, I attached the group only to my opencode-enterprise virtual key. I didn't attach it to any teams, customers, providers, or other keys.

The OpenCode MCP configuration itself didn't need much change. It was already pointing to Bifrost's remote MCP endpoint:

json
{
  "mcp": {
    "bifrost": {
      "type": "remote",
      "url": "https://bifrost-enterprise.agitracker.io/mcp",
      "enabled": true,
      "oauth": false,
      "headers": {
        "Authorization": "Bearer {file:./bifrost-virtual-key}"
      }
    }
  }
}

I kept the virtual key inside a separate local file, so it never had to be pasted into the configuration or committed to Git.

After restarting OpenCode, I used a deliberately small prompt,

Use the Context7 MCP to resolve the React library. Reply with only the returned library ID.

The request executed successfully, as you can see in the above screenshot.

So, is this feature useful? The idea is excellent, especially when different coding harnesses should have access to different MCP tools. And, it gives you a central place to manage tool access instead of duplicating MCP configuration across every client.

What is Adaptive routing?

The next feature I wanted to try was Adaptive Routing. The idea behind it is quite useful, especially if you are running several models, providers, or API keys through the same Bifrost gateway.

Normally, requests are distributed using fixed weights. For example, if two Gemini keys have the same weight, Bifrost can send roughly half of the traffic to each one. The problem is that fixed weights do not know whether one key has become slower, started returning errors, or hit a rate limit.

Adaptive Routing tries to solve that automatically.

Bifrost monitors the latency, error rate, success rate, and utilization of each available route. It then recalculates their weights every few seconds. And, a healthy and faster route receives more traffic, while a failing or slow route receives less. Bifrost still sends a small amount of traffic to recovering routes so it can detect when they become healthy again.

The routing happens at two levels:

Bifrost can select which provider should handle a model request.
After selecting the provider, it can choose the best API key configured for that provider.

This makes the feature more useful for companies that maintain multiple provider accounts or keys. Instead of manually changing weights whenever a provider starts acting up, Bifrost can react to the recent performance data on its own.

How to set Guardrails?

Every model has a set of their own guardrails by default, but while working on them you might need to put your own custom guardrails as well. And the best place to do that is to integrate it directly into your AI gateway!

This feature is meant to protect both the prompts sent to a model and the responses coming back from it. Bifrost separates the feature into two parts: rules decide when a check should run, while profiles define what kind of check should be performed. For this test, I wanted to avoid adding another external API key, so I chose Bifrost's built-in Custom Regex provider. According to the Guardrails documentation, Custom Regex runs locally and can be used for deterministic pattern checks.

So, I created two guardrail rules. One for the input, and the other one for the output.

So, now if I send a request like Reply with exactly: BIFROST_GUARDRAIL_TEST, I get a regex pattern matched error. That’s where the guardrail is actually doing its job.

It is as simple as that. No fancy setup needed for a working guardrail.

Final thoughts

When I started exploring Bifrost Enterprise, I expected the paid version to feel like the open-source gateway with a few extra switches. That wasn’t true.

The core experience stayed familiar, and I could continue using OpenCode through one Bifrost endpoint while the gateway handled the provider connection underneath. That part was convenient. I did not need to change my workflow every time I switched between OpenAI and Gemini.

The enterprise features that made the most sense to me were Audit Logs and MCP Tool Groups. Audit Logs give teams a central record of what happened, while MCP Tool Groups make it easier to control which tools a client can access.

Would I use Bifrost Enterprise for my personal setup? Probably not if I only had one provider, one API key, and a handful of requests. The extra governance and operational features would be more machinery than I need.

For a team running several models, provider keys, MCP clients, and internal users, the situation is different. A shared gateway, centralized logs, access controls, guardrails, and tool restrictions can remove a lot of repeated setup from individual applications. And if you think you are the right candidate, you can always book a demo!

Deep Diving Into Bifrost: Virtual Keys, MCP and Skills

Swapnoneel Saha — Wed, 05 Aug 2026 12:46:50 +0000

In my previous blog, I mentioned how I got tired of switching providers every time I hit a rate limit, and how I finally found Bifrost, which actually solves that. If you haven't read it, please do check it out from here!

Since then, I have been daily driving Bifrost with multiple coding harnesses including OpenCode, jcode and Pi, to name a few. Initially, I was perfectly fine with just the fallback mechanism and complexity routing. But every time I skimmed through the dashboard, the other features kept intriguing me.

So, I sat down and decided to explore all of them one-by-one, and see if any of them solve problems that I'm not even aware of yet. So, let's begin!

Setting Limits With Virtual Keys

So, you can create a virtual API key and set custom limits on how many tokens and requests it can allow in a set time period.

Setting it up is pretty easy through the dashboard.

Just give it a name and an expiry duration, then add the providers and the models you want to allow through this key. And done! You can take this key and set it up in your desired harness, similar to the way we discussed in the earlier blog.

But this time, we should make a separate file for the key, and keep it in the same directory as our opencode config file. I've named mine bifrost-virtual-key, and it holds nothing but the key itself. OpenCode can read a value straight out of a file using the {file:...} syntax, and a relative path there resolves against the config file's own directory, not wherever you happen to launch the terminal from. So the key never has to sit inside the config, and never has to go into git.

After that, pasting this in our config:

{
  "model": "bifrost-local/gemini/gemini-2.5-flash",
  "small_model": "bifrost-local/gemini/gemini-2.5-flash",
  "provider": {
    "bifrost-local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Bifrost Local",
      "options": {
        "baseURL": "http://localhost:8080/v1",
        "apiKey": "{file:./bifrost-virtual-key}"
      },
      "models": {
        "gemini/gemini-2.5-flash": {
          "name": "Gemini 2.5 Flash via Bifrost"
        },
        "opencode-zen/big-pickle": {
          "name": "Big Pickle via Bifrost"
        }
      }
    }
  }
}

The small_model field there is worth a word, since it's easy to skip past. OpenCode uses it for cheap background work like generating session titles, and if you don't set it, it goes hunting for a cheaper model on its own. Pointing it at the same model keeps everything flowing through one virtual key, which is the whole point of the exercise.

This feature is pretty useful when you are experimenting with different models, and for long-horizon tasks, where you don't want the model to run indefinitely and get stuck in a loop, just burning expensive tokens. Or whenever you want fine control over the MCPs, tools or skills that your harness can get access to.

So let's set a deliberately tiny limit and watch it bite.

As you can see, I've set a maximum of 1000 tokens and a maximum of 10 requests, both resetting every hour. That token budget is small on purpose, because I want to hit the wall quickly rather than wait around for it. I've already connected this key to my OpenCode setup, so a couple of ordinary messages should be enough to get me blocked.

And that's exactly what happened, though the way it happened is the interesting part. My first message cost 627 tokens and went through fine. The second one went through too, and quietly cost 12.94K tokens by itself. Every single request after that got rejected, which is why the success rate sits at 16.67%, only 2 requests out of 12.

So the limit isn't checked against the size of the request you're about to make, it's checked against what you have already spent. A request gets waved through as long as you're under budget at that moment, and it can then blow straight past the ceiling on its own. Worth knowing before you set a budget you actually care about, because the cap decides when the blocking starts, and not how much you can spend in total.

One more thing to notice in that screenshot: each failed attempt shows up twice, once against gemini and once against big-pickle. That's the fallback rule from the last blog doing its job. Bifrost tried the fallback, and the fallback got refused by the same virtual key, which is exactly what you'd want.

Now, let's see what more we have in our box!

The MCP Gateway

This is the one that I regret not trying earlier.

Using the MCP gateway we can connect all our MCPs in one place, and then selectively allow access to our harnesses using the virtual key. So instead of every harness carrying its own copy of every MCP config, Bifrost holds them all, and each key gets to see only the slice you've allowed it.

There are a huge number of MCPs already present in the MCP Server library, 487 of them at the time I'm writing this, and it's just a one-click installation from there. As you can see in the screenshot below, I have already installed the Context7 MCP. And now, I will just add this to my virtual key from the MCP Client Configuration option, and we are good to go.

As we have already added our virtual key to our OpenCode harness, we can now easily use the Context7 MCP from there. Bifrost exposes every MCP you've installed at a single endpoint, /mcp, and the virtual key you send as a Bearer token decides which tools come back. So from OpenCode's side, this looks like one ordinary remote MCP server, no matter how many you have installed behind it.

Here's the block to add to the config file:

{
  "mcp": {
    "bifrost": {
      "type": "remote",
      "url": "http://localhost:8080/mcp",
      "enabled": true,
      "oauth": false,
      "headers": {
        "Authorization": "Bearer {file:./bifrost-virtual-key}"
      }
    }
  }
}

The oauth: false line matters more than it looks. OpenCode will try to start an OAuth flow on its own when a remote MCP server answers with a 401, so you have to tell it not to bother here, because we are authenticating with a fixed key instead.

Let's test it by asking OpenCode something that can only be answered by fetching live documentation, and then check whether the tool calls actually show up on the Bifrost side.

And there they are. Two tool calls, resolve-library-id followed by query-docs, both against the Context7 server, both successful. So the MCP is working perfectly as intended, and I never had to put a Context7 config into OpenCode at all.

Now, Bifrost has a similar thing going on for prompts and skills as well, so let's check them out too!

The Prompt and Skills Repositories

So, before coming to the skills, let's talk about the Prompt Repository. Here, we can test our prompts against various models, see the results and tweak them. This is really important, because I was able to thoroughly test my prompts here before turning them into a skill.

And as you can see, I've created a code review prompt that accepts the language, the review focus and the code snippet, and gives you a detailed review of it, along with the revised code if your code needs any fixing.

The variables are the neat bit here. Anything you write as {{ code }} or {{ language }} inside the prompt gets picked up automatically and turned into a field you can fill in, so you're changing the inputs and not rewriting the prompt every time you want to try something.

And prompts here are versioned, which I didn't expect. You keep editing in a session, and when something is actually good you commit it as a version, so the thing your application calls later is a version you deliberately shipped, and not whatever you happened to be typing five minutes ago.

Now once you are done testing your prompts, you can create a skill out of that as well, and add it in the skills repository.

I've created this skill, safe-bug-fix, that can be used to fix bugs in a codebase. It's a plain SKILL.md, with a "when to use" section, a numbered workflow and a set of safety rules, and it's targeted at OpenCode.

After creating the skill, we need to configure OpenCode as well, so that it can actually use the skills.

For that we can paste the following code snippet in our OpenCode config:

{
  "permission": {
    "skill": {
      "*": "allow"
    }
  }
}

The * there means every skill is allowed. You can be pickier if you want, since the same block takes deny and ask alongside allow, and the keys accept wildcards, so something like internal-* can be denied while everything else stays open.

And you will be able to use it inside OpenCode anytime you want!

Now, the next most useful feature that I found is setting up custom log headers. Let's talk about that then!

Custom Log Headers

So, while I was primarily using my key on OpenCode, I started to use it on jcode as well. But I was unable to identify from the logs which request was coming from which harness.

Hence I decided to find a solution, and found that we can create custom log headers in Bifrost from the Log settings. It was pretty simple to set up.

The idea is straightforward: you name a header, and from then on Bifrost copies that header off every incoming request and stores it in the log entry's metadata. I added the header name X-Request-Source.

After that, I configured OpenCode to actually send that header, by pasting this in the config file:

{
  "provider": {
    "bifrost-local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Bifrost Local",
      "options": {
        "baseURL": "http://localhost:8080/v1",
        "apiKey": "{file:./bifrost-virtual-key}",
        "headers": {
          "X-Request-Source": "opencode"
        }
      }
    }
  }
}

And, that's it!

As you can see in the screenshot above, there's now an extra column in the logs, and I can tell that these requests are coming from OpenCode. Repeat the same block in your other harness with a different value, and the whole picture separates out.

There's also a shortcut I found afterwards, which I'd have used if I had known about it. Any header you send with an x-bf-lh- prefix gets captured into the log metadata automatically, without configuring anything on the Bifrost side at all. The prefix gets stripped and whatever is left becomes the key. So x-bf-lh-source: opencode would have got me the same result with one less step.

It was as simple as that!

So, What's Actually Worth Using?

Honestly, more of it than I expected, and a bit annoyingly so. Every one of these had been sitting in that sidebar the whole time I was happily using Bifrost as a fallback router and nothing else.

The MCP gateway is the one I'd tell you to try first. Moving every MCP out of individual harness configs and into one place, then handing each harness a key that decides what it can see, fixed a mess I had stopped noticing because I'd been living in it for so long. Virtual keys are the ones I'd keep the tightest grip on, now that I know the limit gets checked against what you have already spent and not against what you're about to spend. And custom log headers took about two minutes and solved something I'd been squinting past for weeks.

The prompt and skills repositories I'm still making my mind up about. Testing a prompt properly before shipping it is genuinely useful, and the versioning is better than what I was doing before, which was nothing. But I've not used them long enough to tell you whether they replace the way you already keep your prompts, or just sit beside it.

So if you're running Bifrost as a gateway and never got past routing, go open that sidebar. There's more in there than I expected, and all of it stays local. And if you want to get a taste of the enterprise version, then you can easily book a demo.

And if you try any of these, or you've found something in Bifrost that I've still not touched, tell me about it, or come find me on X.

Trying Bifrost: An AI Gateway That Simplified My Setup

Swapnoneel Saha — Tue, 04 Aug 2026 09:24:41 +0000

I try a lot of models daily, and I kept ending up with a separate API key for every model provider I wanted to test with my desired harness. So I decided to try Bifrost on my local machine, to see if it would actually fix that.

Bifrost is a high-performance, open-source AI gateway, built in Go. It puts multiple AI providers behind a single OpenAI-compatible API, and it does that with ultra-low latency, automatic failover, load balancing, and enterprise governance features baked in.

If you've used LiteLLM before, it may sound familiar on paper, but the experience is pretty different in practice. LiteLLM is a Python library and proxy you configure and run yourself; Bifrost ships as a standalone Go binary with a full web dashboard baked in, so there's no separate observability stack to stand up just to see what's actually happening to your requests. That dashboard ended up being the thing I used the most, as you'll see below.

I wired it into OpenCode, an open-source coding harness similar to Claude Code and Codex. I'm using an OpenCode Zen key and a Gemini key, and together, these give me access to multiple SOTA models for free, without touching a separate dashboard for each provider.

Installation and Setup

The installation and setup was very simple, and quick.

First, I installed the Bifrost CLI using this command:

npx -y @maximhq/bifrost

Note: This requires Node and NPM to be installed on your machine, otherwise it won't work. If you want to know how to install and manage node versions, you can follow this blog that I've written earlier.

Now, it's time to run Bifrost! I'm doing it in a directory level, but you can also do it in a system level as well, if you want. For that, you can easily follow the Bifrost documentation.

npx -y @maximhq/bifrost -app-dir ./my-bifrost-data

I ran this command in my working directory, and this will create the configuration files and the logs db using SQLite.

And also, this will expose our dashboard in port 8080, and from there we can easily set our API keys and use them in our applications, which I will get to you later in this blog.

Connecting Your Providers

Now it's time to grab your API keys, and connect it to Bifrost.

As I've mentioned previously, I will be using Zen and Gemini. Both of these have generous free tiers, and getting the API keys doesn't require any credit card details.

As you can see from the above screenshot, adding your keys is pretty simple. You just need to select your provider, assign a name to your key and you are good to go!

Additionally, you can also add allowed models to your specific key, because most of these keys come with a lot of available models, and if you don't want to use all of them, or limit your pool, you can do it from here as well.

Integrating With OpenCode

Now that you have connected your providers to Bifrost and it is already running, we can now integrate Bifrost and these models directly to OpenCode.

Now, adding a connection like Bifrost means you have to manually edit the config file. Based on your operating system, the location of the config file might vary. More detailed info about that you can check on OpenCode's official documentation.

But the file that we need to edit is opencode.json. In the provider block, we have to add something similar to this:

"bifrost-local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Bifrost Local",
      "options": {
        "baseURL": "http://localhost:8080/v1"
      },
      "models": {
        "opencode-zen/big-pickle": {
          "name": "Big Pickle via Bifrost"
        },
        "gemini/gemini-2.5-flash": {
          "name": "Gemini 2.5 Flash via Bifrost"
        }
      }
    }

Note: I gave this its own bifrost-local provider block instead of just pointing OpenCode's built-in openai provider at Bifrost, the way Bifrost's own docs show it, because I wanted it visually obvious in the model selector which models are going through Bifrost versus hitting a provider directly, in case I ever wire up a real OpenAI key in the same config later. Either approach works functionally; this is just how I like to keep them apart.

Now this will vary based on your selected model and provider. It is better if you can check it out from the Bifrost docs itself.

Now that we are connected, we can launch OpenCode from our terminal (or app), and in the model selector we will see something like this:

Now we can select this model and use it to do our task on OpenCode.

But wait, till now I just described how Bifrost sits between your model provider and harness. So, let's get to the crux and find out what more things we can do with Bifrost!

Monitoring and LLM Logs

Bifrost gives us a clear understanding of how our models are performing, the amount of tokens they are consuming, the latency and the cost as well.

In the screenshot below, you will be able to see how Bifrost does that:

We can see and check the detailed logs as well.

All of your data is stored locally and nothing gets sent to the cloud, other than the messages we're sending to the model providers, which is obvious. Everything is kept air-gaped.

Model Routing using Automatic Fallback

Now the most interesting part: how do we automatically route the models?

During production, one model might fail to respond, and it's kinda common. So for those scenarios, we can set rules like:

If model A is not available, then use model B.

This is the simplest version though. Using CEL expressions, we can create custom routing rules for almost anything that we want.

If models are available and we have a specific logic in mind, we can implement that in Bifrost easily.

So, for now, let's create a simple rule, such that: if Gemini models aren't available, we will route the traffic through Zen models instead.

As you can see in the screenshot above, I have created a global rule with maximum priority, such that if the provider is gemini and the model is gemini-2.5-flash, and if the user is using that specifically, we will fall back to opencode-zen/big-pickle instead.

Now as the rule is set and applied, I will temporarily disable the gemini services to see what happens.

Let's come to OpenCode, and type a "hi", and let's see what happens.

As you can see, the user didn't get any interruption at all. Let's check the logs to find out what actually happened here.

As we can see, the gemini model gave an error, and it automatically fell back to big-pickle, and gave me the response.

That's the magic of Bifrost.

The Complexity Router

Now that we have understood Model Routing, let's see how we can determine and set what kind of requests should go to which kind of models.

For example, we can route trivial queries to cheap models, while preserving the expensive ones for difficult tasks.

The idea is simple: divide the incoming requests into four tiers: simple, medium, complex and reasoning.

So I've configured my complexity routing profile, and you can easily do it as well based on your own requirement. Here's my profile:

For my simple and medium queries, I want the gemini model to handle that, and for complex and reasoning-based queries, I want them to go to big-pickle.

Now, we need to create custom routing rules for that.

Now that the rules have been created, let's test them, and check the LLM logs. I'll send two queries from OpenCode:

Hello, briefly define REST API. (expected to go to gemini)
How to debug an async API authentication failure step by step, explain the root cause, and recommend an architecture fix. (expected to go to big-pickle)

And we can see in the screenshot above that it worked exactly how we intended.

Bifrost Can Do a Lot More

Model routing and the complexity router are the two features that got me the most excited, but Bifrost isn't limited to just these two. It also ships guardrails, virtual keys, and cluster mode for scaling across machines, and that's still on my list to explore.

Honestly, I loved this. Setting it up took maybe fifteen minutes end to end, and it quietly fixed a problem I'd been living with for a while: juggling separate API keys and dashboards for every provider I wanted to test. The fallback and complexity routing worked exactly the way the docs said they would, no surprises, and that alone made this whole exercise worth it.

If you're juggling more than one model provider and keep swapping keys by hand, give Bifrost a shot. And if you've already tried it, let me know what you built with it, and also if the enterprise version intrigues you, you can book a demo as well.

Nobody's Testing AI Coding Agents Enough

Swapnoneel Saha — Thu, 23 Jul 2026 22:52:30 +0000

Code review used to be the part everyone complained about. Slow, nitpicky, the thing standing between you and shipping.

And for a while, AI coding agents made it feel optional. The agent writes the code, the code compiles, the tests pass, ship it.

But have you ever wondered, what does that actually look like once you zoom out to the whole industry, and not just your own repo? Not great. Somewhere between 40 to 62% of AI-generated code got shipped with security or design flaws by March 2026, and roughly one in five breaches this year traces back to AI-written code, according to industry analysis on the verification gap. Code generation got really fast. Verification did not pick up the same pace. Testing AI coding agents properly is where that gap actually lives, whether your team has staffed for it or not.

What's actually breaking?

Let's get specific.

In late June 2026, security researchers at Adversa AI disclosed something called GuardFall, a shell-interpretation bypass that worked against 10 of 11 popular open-source AI coding and computer-use agents, including Aider, Cline, Goose, and OpenHands. The agents were checking the raw command text for danger before running it, but bash rewrites that text through quoting, substitution, and expansion before it actually executes. So a command that looks harmless to the safety check can still detonate once the shell gets its hands on it. And only one tool in the survey actually held up!

And it's not just an edge case for people running agents locally. A scan of 5,600 vibe-coded apps already in production found 2,000 highly critical vulnerabilities and 400 exposed secrets, some of them exposing medical records and payment information. Georgia Tech's Vibe Security Radar tracked the trend line getting worse, and not better: 6 confirmed AI-generated vulnerabilities in January 2026, 15 in February, 35 in March.

Why is this happening if the models got so much better?

Well, that's exactly the part people get backwards.

Better models didn't remove the need for verification, they just moved the bottleneck. Generating a solution stopped being the hard part a while ago. Deciding whether you can actually trust that solution is the hard part now, and 55.4% of enterprise decision-makers already name agent reliability and hallucination management as their top production challenge. The code compiles, the tests pass, and reviewers still have to reconstruct what the change was even trying to do before they can tell if it's safe.

I've actually had this job, and it's not glamorous

I'm not writing this from the outside. I recently worked with an early-stage startup that's figuring out their PMF before going full-throttle. I was the first layer of internal testing for the product, which was a self-evolving super agent (keeping it a bit vague, can't reveal more than this lol), and my entire job was catching bugs before the core users on it ever saw them. I also built an internal tool which was an agent chain based on strict rules that pulled the agent's data logs and evaluated them to ensure that the self-learning from feedback and real-life scenarios aren't being hallucinated, and if the agent is trying to manipulate the guidelines itself. Also, I generated reports on latency and probable slowdowns, because "it seems to be working" isn't a testing strategy, numbers are.

Before that, I spent time at Keploy building and improving sample apps specifically to demo API testing. So when I say testing agent output is a real, staffable job and not a checkbox, that's not a hot take pulled from a headline, it's what I got paid to do.

And to be fair to the agents themselves: they are genuinely fast, and genuinely useful. I use Claude Code, Antigravity and Codex daily, and I'm not about to pretend otherwise. The problem was never that the code they write is bad on average. The problem is that "on average" is exactly the wrong bar for security and correctness, because the failures cluster in the 5-10% you didn't specifically check.

(And yes, this post was drafted by an agent skill I built, and I'm going to go through and edit it before it goes anywhere near publish. That's not irony, that's the actual point: the draft can get the facts and the structure right, but deciding which of my receipts actually belong here, and how hard to steelman the agents, is still a job for a human. Mine, in this case.)

What should you actually do differently?

Stop reviewing agent output the way you review your own code, and start reviewing it the way you'd review a fast junior developer's very first PR: assume competence, verify everything, especially the parts that touch execution.

Concretely: never let an agent pipe raw, unreviewed strings into a shell without a real evaluator in between, GuardFall exists because teams assumed string-matching was enough. Budget actual human review time as a fixed cost of using these tools, not a nice-to-have. And track your own vulnerability trend line the way Georgia Tech tracked the industry's, because "it hasn't broken yet" is not the same thing as "it's fine."

And testing an agent isn't only about catching bugs before they ship, it's also about watching what the agent quietly costs you over time. That's exactly why I built that internal latency-tracking tool in the first place, numbers on slowdowns catch problems long before a user ever complains. If you don't want to build that yourself, tools are starting to do it for you: I recently found Bifrost from Maxim AI, a gateway that sits between you and your coding agents and gives you latency, cost, and token usage in one dashboard, plus fallback logic for when a model starts misbehaving. Worth a look if you're juggling as many agents and LLMs as most of us are these days.

Do use these agents. Really do, they're not going anywhere and they've earned their place in my own workflow. But treat testing them as the actual job, not the afterthought, because right now, for most teams, it still is one.

If you're building your own testing layer for an AI tool, or you've been burned by one that didn't have one, I'd genuinely like to hear about it, drop it in the comments. I write more about agent tooling and building with AI at swapnoneel.site, including how I built a self-improving writing agent, and you can find me on X (swapnoneel123).

Kimi K3 and the Rise of Open Weight Frontier Models

Swapnoneel Saha — Thu, 23 Jul 2026 12:09:46 +0000

Apparently everyone is talking about the launch of Kimi K3 right now. So I wanted to share my two cents on this, especially for those who consider this to be just a "cheap chinese model". Because honestly, that joke doesn't land well anymore.

Here is the actual reason why I'm saying so. Moonshot AI released Kimi K3 on July 16, 2026, a 2.8 trillion parameter open weight model, and it landed at #4 on the Artificial Analysis Intelligence Index, just behind Claude Fable 5 and GPT-5.6 Sol, and ahead of Claude Opus 4.8. Full weights are dropping on July 27. Pricing is 3USD and 15USD per million input and output tokens respectively, a fraction of what the closed labs charge. That's why open weight frontier models are suddenly a real conversation and not just a budget footnote.

The old belief was that closed labs own the frontier

For most of the last two years, the assumption was simple: if you want the smartest model, you pay OpenAI or Anthropic, period. Open weight models were the budget option, good enough for chatbots and side projects, but never good enough for the actual frontier.

But those walls started shaking when we got GLM 5.2 from Z.ai, just a few weeks ago. And then came Kimi K3, which is the moment that assumption stopped being obviously true. Not because it beats GPT-5.6 Sol and Fable 5 outright, it doesn't, but because the gap has gotten small enough that "just use the closed model" is no longer an automatic decision. According to Nathan Lambert's analysis on Interconnects, the gap between open and closed, and between US and Chinese labs, has shrunk from a debated 6 to 9 months down to something closer to 3 to 5 months (interconnects.ai).

What's actually inside Kimi K3

The specs are genuinely wild. It's a mixture-of-experts model with 896 experts, and it only activates 16 of them per token, so despite being 2.8 trillion parameters total, the compute cost per token stays manageable (kimi.com). It ships with a 1 million token context window, native vision, and a new attention mechanism called Kimi Delta Attention.

On raw benchmarks, K3 takes first place on Program Bench, SWE Marathon, BrowseComp, and Frontend Code Arena. Program Bench specifically jumped from 53.6 to 77.8 over its predecessor, a 45% jump (wan27.org). In blind developer testing on Arena, people preferred Kimi K3 over both Fable 5 and GPT-5.6 Sol for front-end coding specifically (codersera.com).

Does that mean the closed labs are done?

Well, not entirely. GPT-5.6 Sol and Fable 5 still sit ahead on the general Intelligence Index, around 59 and 60 versus K3's 57. And Moonshot's own success became a problem within days. Demand strained their compute capacity hard enough that they had to pause new subscriptions. That's not a small footnote. Running a 2.8T model at scale is expensive even when you're the one giving the weights away for free, and it shows that "open" doesn't automatically mean "infinitely available."

And here's the honest catch on price too. K3 spends way more tokens in reasoning. So, Sol and Fable both tend to get to an answer in noticeably fewer tokens than K3 needs for the same task, so once you look at cost per task instead of cost per million tokens, the gap almost closes. The sticker price makes K3 look like a steal, the actual bill at the end of the month is a lot closer.

Why open weight frontier models actually matter to you

I ship AI products for a living, and most of my projects live and die by which model I pick underneath them, and cost per task isn't some abstract line item for me, it directly decides whether a feature is worth shipping. When a model that's a few points behind on intelligence lands close on actual cost and you can self-host it once the weights are out, that's not a footnote, that's a real decision every team building on LLMs now has to make. I ran into a version of this same tradeoff when I wrote about testing AI coding agents, model choice was never just about the leaderboard, it was about what actually held up under my own usage.

And there's a bigger reason than cost. A model whose weights you hold cannot be shut off by someone else's pricing decision, rate limit, or policy change. That's the sovereignty argument people keep making about open weight models, and it stops being theoretical the moment your product depends on an API you don't control.

The verdict

Open weight models are no longer the consolation prize. Kimi K3 is proof that you can be a handful of benchmark points behind the absolute frontier and still be the more rational choice for a huge chunk of real work, especially coding. GPT-5.6 Sol and Fable 5 are still the smartest models on the planet right now, and if you need every last point of reasoning, use them. But if you want to not depend on someone else's uptime, and you're fine with the actual bill landing close either way, going and actually trying K3 instead of assuming the closed model wins by default is worth your afternoon.

That's just me though, and your workflow might be different depending on what you're actually building.

If you're experimenting with model choice for your own AI products, drop a comment with which model you've moved to since K3 landed.

Why Vercel is still my default for shipping frontend projects

Swapnoneel Saha — Mon, 13 Jul 2026 15:39:49 +0000

Last week, I was working on a client project with a fast approaching deadline. The work had already piled up, so I had to move really fast; I was constantly making changes, pushing them straight to GitHub, checking them through the preview link of the deployment, and going straight to the next task. And while doing so, I barely stopped and worried about hosting, because Vercel was already connected. And after successfully delivering the project within the stipulated time, it hit me that I probably could not have moved that quickly if the deployment itself had been another thing to manage.

That made me realise: Vercel has been my default choice for a long time, and it is not because I am completely locked into the platform. From time to time, I still reach for other services like Cloudflare, Netlify, and Railway as well, but for my personal projects and fast development cycles, I somehow always end up coming back to Vercel.

I mostly use Next.js, so I know the tech nerds out there will assume that, it is the entire reason why I choose Vercel, and that's a fair assumption to make, because it's partly true. Vercel develops and maintains Next.js, so of course it provides the best hosting for Next.js, but that's just one side of the coin.

How I use Vercel in my projects

If you check the projects section on my portfolio, you will find that most of the web projects I currently have are deployed through Vercel. And not all of them are Next.js applications; you will find projects with React, TanStack tooling, Node.js, and Bun as well. These are not just weekend experiments or hobby projects, either. Some of them have real users as well! Let me give you Scholarian as an example. It is a research platform built on Next.js, and according to my latest project analytics, it currently has more than 75 active users and over 700 chat sessions.

The funny thing is that I did not think twice about deploying most of these projects. I connected the repository, gave Vercel the required environment variables, and pushed the code. That absence of thought is the whole point. But then again, I also use Cloudflare Pages and Railway for actual work, so this is not a “Vercel is perfect and everything else is bad” argument. I have reasons for coming back, but I also know where the platform starts becoming the wrong tool. So, let's discuss!

Why do I keep coming back to Vercel?

First of all, Vercel's preview deployment workflow! It makes my development cycle much smoother. By default, every non-production branch can receive its own preview URL, and I can share that URL before merging the branch. That's extremely useful for catching visual problems before they reach production. A pull request may look completely fine during code review, but you can never know when the actual interface breaks at a particular viewport width. This has happened to me a lot. Just a few days ago, I shipped the near-final version of a project to one of my clients without noticing that, in the mobile version, a heading was overlapping one of the image assets. Preview deployments let people test the thing instead of trying to imagine it from a diff.

For a solo developer and freelancer like me, this saves a lot of time because, as you can see in the above screenshot, Vercel adds a toolbar to preview deployments where collaborators can leave comments directly on the page. This was especially useful during hackathons, when we were short on time. And our team always communicated in that way, and my teammates would drop in and leave comments like "the link to this button is redirecting to the pricing page instead of the features page" or "the color is way too contrasty." The small catch is that they need a Vercel account to comment, and external collaboration has some plan-specific limits, so it is not entirely frictionless, but still, it is much easier than sending Loom videos, annotated screenshots, or five messages explaining which button or font your client or peers want. And they have an optional third-party integration as well that can convert a preview comment into a GitHub issue. This makes conversations with my clients and non-technical collaborators much easier!

The Next.js experience is the other reason I keep using it. Vercel develops the framework, so features such as Incremental Static Regeneration, Server Actions, React Server Components, route handlers, and streaming work with very little platform-specific configuration, and I don't have to spend an afternoon figuring out how a new Next.js feature maps onto the hosting environment. Vercel covers that part for me by default.

Now, to be fair, other platforms have improved a lot, and Netlify currently supports the major Next.js features through its OpenNext adapter, including Server Components, Server Actions, streaming, ISR, and Partial Prerendering. Cloudflare can also run Next.js using its own OpenNext-based adapter. So the difference is no longer that Next.js features simply do not work elsewhere, because that would be an outdated argument. The difference is that Vercel remains the first-party deployment target, and that means there is one less compatibility layer to worry about. And this removes a pain point for me, especially when I am using a newer framework feature. And that's the edge I'm actually talking about. For a normal static React or Vite application, this advantage matters much less, but for a serious Next.js project, it becomes my go-to option.

And then there's the DX at the dashboard level. These are minute things, but together, they make a big difference for me. For example, the environment variables are scoped per environment (local, preview, and production; all of them are isolated). Rolling back to any previous deployment takes two clicks. And the deployment logs actually tell you what failed, not just that it did, and because of that, they become much easier to fix if you are taking the “pasting it into Claude Code” route.

What are the alternatives to Vercel?

Well, when we are talking about the alternatives, Cloudflare Pages is the one that comes the closest. And we know how much tech Twitter is divided on this one, and how frequently we see their representatives fight each other on open threads regarding this (I enjoy watching those heated arguments, lol). And yeah, Cloudflare is genuinely fast, and their edge network spans 300+ locations, and for static content, the performance gap over Vercel is actually quite measurable. And what I appreciate most is that the pricing is much more predictable; because, first of all, there are no egress fees, and they also provide unlimited bandwidth on the free tier. And as a bonus, I have also seen them helping start-ups from time to time as well, which is a great initiative, in my opinion.

I respect all of this, but the problem is that Cloudflare's Workers environment runs on V8 isolates, which is different from a standard Node.js runtime. And this is where I face the most problems. For purely static sites or projects that have lightweight edge functions, it's totally fine, but sometimes, with specific packages that exclusively require a Node.js runtime, you start to face error messages. And although nodejs_compat mode now supports a substantial portion of the Node API, the compatibility is still not perfect.

There is also a trade-off in how the two platforms approach infrastructure. If you want databases, KV stores, or smart routing in your project, you must understand Cloudflare's broader ecosystem, like D1, KV, and routing rules, which is great when you want that level of control. But Vercel abstracts all of that by default. It is basically a trade-off of infrastructure control for speed, and I prefer Vercel's simpler deployment workflow in this regard.

Netlify was my original platform before I switched. I have nothing against it, honestly. It is very similar to Vercel in a lot of ways. But Vercel's integration with Next.js, which I just discussed in detail in the previous section, makes Netlify feel like it's one step behind. Features like Server Actions and React Server Components work natively on Vercel, while on Netlify, they have to go through adapters that often lag behind new framework releases, which is a big compromise. And another thing: Netlify's core CDN infrastructure also has fewer edge locations than Vercel's 100+ node network, and that's visible in the global TTFB numbers as well. I'd still use Netlify for a simple static site with a form or two because their built-in form handling is actually clever. But for a Next.js project, Vercel is my primary choice.

Now, for Railway, it's a bit different, and I use it when I need a persistent backend, like maybe a WebSocket server, a background job, or something that can't be serverless. In Scholarian, I have a long-running task where the background worker has to produce a long report using Gemini, and that process generally takes three to four minutes, so I switched the backend of my app to Railway. For Vercel, that's where it genuinely breaks down. If you need a long-running process, you're either doing something hacky with edge functions or you're reaching for a different platform. And there are a couple of good options besides Railway, and for that, Encore would be my personal recommendation.

Where Vercel actually falls short

The pricing model! And that's the part I like the least. The free Hobby plan is capped, so it cannot generate an on-demand surprise bill, but if a hobby project exceeds its allowances, it may be paused or restricted instead. That's why I keep an eye out for my portfolio site, because that's the one that gets the most traffic. The bigger billing concern begins as soon as you switch to the Pro plan, where usage beyond the included credit can be charged across multiple resources. Being mindful enough is particularly important here, because I have seen a lot of posts on Reddit and X where developers have complained about the same issue.

Vercel Pro currently has a 20 USD monthly platform fee, which includes one deploying seat and 20 USD of usage credit. And for additional developer seats that can deploy or configure the projects, they will cost you another 20 USD per month, but the read-only viewer seats are free. And that combination can become difficult to predict when a project grows.

Vercel provides spending alerts and lets paid teams configure actions such as pausing projects after reaching a limit. Hence, it's better to enable those controls instead of assuming that traffic will always stay predictable. Sudden bot traffic, a poorly optimized function, image transformations, or a sudden spike in legitimate users can all consume usage faster than expected. So, it's always better to keep those factors in mind so that you don't get overcharged accidentally.

The other limitation is compute. Vercel Functions can handle APIs, server-rendered routes, streaming, and other request-driven tasks, and the current function limits are far more generous. But if your application requires a continuously running background process or custom Docker containers, Vercel isn't the right fit. There are platforms like Render or Northflank that are built for that kind of workload. Vercel is a frontend cloud, so the moment you need full-stack infrastructure, you're pairing Vercel with something else anyway. Hence, the title of my blog says why I prefer it for frontend projects, and not full-stack projects!

And then there is also vendor lock-in, although I do not think it is as simple as people make it sound. And it's not limited to Vercel either; almost every service provider has its own kind of vendor lock-in. A static React or Vite project is easy to move, but a Next.js application that depends heavily on Vercel’s caching behavior, image optimization, routing, integrations, and deployment settings will take more effort to migrate. The more platform-specific behavior you adopt, the less portable your application becomes, and that's true for Vercel, Cloudflare, AWS, and almost every other cloud platform. I have not experienced the issue myself, but I have seen people complaining about it, so I included it in the blog.

Is the Vercel free tier actually good enough?

For most side projects, yes! And their Hobby plan is free for personal use and is pretty generous. It comes with unlimited projects, automatic HTTPS, custom domains, preview deployments, and 100 GB of bandwidth per month, and that's mostly enough for personal use. And I never felt the need to purchase their paid Pro plans.

But the moment you add a team or need more bandwidth or function execution, you have to go to the Pro plan that starts at $20/month for each member.

What is the verdict?

Vercel is still my default for frontend deployment and a part of my development cycle. The preview URL workflow alone has saved me more debugging cycles than I can count. I've tried the alternatives in actual projects, and none of them gave me back the time I was spending on deployment issues.

That said, if you're cost-conscious and don't mind the learning curve, then Cloudflare is still a great choice. And if you need a backend, pick Railway and point your Vercel frontend at it. And if you've moved away from Vercel for something specific, or if you have a setup that works better for you, I'd genuinely love to hear about it in the comments! Made a reddit post for this discussion as well, I would really appreciate if you can discuss there..

And I have also recorded a short video, where I compared the deployment of one single Next.js project on three platforms, namely Vercel, Netlify and Cloudflare, so if you want to check it out, you can do that from here.

And if you want to see the kinds of projects I've actually shipped on Vercel, check them out at my site. Also, you can find me on X or GitHub if you want to talk or connect.

GEO for Developers: Get Cited by ChatGPT and Perplexity

Swapnoneel Saha — Tue, 07 Jul 2026 23:53:53 +0000

Have you ever asked ChatGPT or Perplexity a coding question and get a suspiciously specific, correct answer, with zero link back to whoever actually wrote it?

Yeah, that's been happening to my blog too, and I finally sat down to fix it.

So here's the real answer: GEO for developers doesn't need an agency or a 40-page audit. It just needs three simple things: a properly structured content so a model can lift one paragraph and have it make sense on its own, real evidence instead of vague claims, and let the right bots get into your robots.txt. That's genuinely most of it, and in this blog I'll walk you through exactly how!

What is GEO, actually?

GEO stands for Generative Engine Optimization, and it's the practice of writing content so AI answer engines (ChatGPT, Perplexity, Google's AI Overviews) quote it directly instead of just ranking it in ten blue links.

It's pretty self-explanatory, and it sits right next to SEO (Search Engine Optimization). We used to optimize purely for search engines like Google, but the times have changed, so now you're writing for generative engines too, Perplexity, Gemini, ChatGPT, all of them.

The term comes from an actual peer-reviewed study, presented at KDD 2024 by researchers from Princeton, Georgia Tech, and IIT Delhi, and it's become its own line item in 2026 marketing budgets. The paper's numbers are wild: adding statistics to a page boosted its visibility in AI answers by 41%, and content optimized for generative engines improved visibility by up to 40% overall.

Why doesn't most GEO advice fit a dev blogger?

Here's the thing though, and I say this as someone who's done SEO freelancing and used Semrush since my first year of college: almost every GEO guide I found while researching this is written for a marketing team running brand-mention trackers across a hundred pages. That's not you if you're publishing one post a week on your own domain, or worse, on Hashnode.

You don't need consensus-signal dashboards. You need to know which five things to do to your next post, and that's what I'm giving you.

What actually works in GEO?

Answer the question in your first 40-60 words, standalone. Roughly 44% of everything AI engines quote comes from the first third of a page, so don't bury your point under three paragraphs of throat-clearing. Say the thing, then explain it. If you notice carefully, I have already mentioned the three GEO optimisation steps in the introduction itself.

Phrase your headings as questions. Not "Benefits of X", but "Why does X matter?" or "What is X?". This is also just a more natural way to write, so it's a rare case where the AI-friendly move and the human-friendly move are the same move. Check how I have framed the headings of each section =]

Every section has to make sense if someone rips it out of the page. AI engines lift paragraphs, not entire posts. If your section starts with "This also means...", restate what "this" is. Small habit, big difference.

Put real numbers in, with sources. Not "many developers prefer X", but "X handles 50,000 requests per second, per their own benchmark, published in June 2026". Vague claims don't get quoted. Specific, sourced ones do. You will find multiple such statements in this blog itself, and also with linked citations.

Fix your robots.txt. This one's just a config file, and most bloggers never touch it (yes, that's a real file sitting on your domain right now, doing nothing). The bots you want to allow for citations are different from the bots that scrape for training data: OAI-SearchBot and PerplexityBot are the ones fetching pages to answer live questions, while GPTBot and ClaudeBot are the training crawlers. You can allow the first pair and still block the second, if that's the line you want to draw.

Should you bother with llms.txt?

Honestly? Probably not yet, and I want to be straight with you about this because most GEO posts won't be.

The idea is simple: drop a markdown file at /llms.txt summarizing your site so a model doesn't have to parse your HTML. Over 844,000 sites have added one already, including Anthropic's own docs. But no major AI company has confirmed they actually read it, and Google's John Mueller called it a "temporary crutch" that isn't done for search at all.

So add one if you want, it costs you ten minutes and can't hurt. Just don't mistake it for the thing that's going to get you cited. That's the structure and the sourcing, not the file.

What if you don't even own your blog?

If you cross-post to Hashnode or Dev.to like I do, you don't control the robots.txt on that domain, and you can't add schema markup either. That's the platform's call, not yours.

What you can still control everywhere: the content structure itself, and your canonical URL. Always point the canonical tag back to your personal site, always write the answer-first paragraphs regardless of platform, and let the schema/robots.txt tactics apply fully only where you actually own the domain.

Is GEO worth your time?

Yes, but not the version most people are selling you. I recently rebuilt my own blog-writing process around exactly this, direct-answer blocks first, real stats with sources, extractable sections, and it's honestly made the drafts read better for humans too, not just for whichever bot happens to crawl them. I wrote up the whole build here, if you want the longer version.

FAQ

Do I need to block GPTBot to protect my writing?
That's a separate decision from GEO. Blocking GPTBot stops your content from training future models, but blocking OAI-SearchBot too would also stop you from showing up in ChatGPT's live search results. Decide which trade-off you actually want.

Will GEO replace SEO for blogs?
No, they overlap more than they compete. Structuring for extraction and citing real sources helps you rank in Google too. Think of GEO as SEO with an extra, stricter bar for evidence and standalone clarity.

How long before I see actual citations?
I don't have a clean number for this yet, since I only rebuilt my own process around it in mid-2026. Perplexity re-crawls constantly, so that's the faster feedback loop; ChatGPT search is slower and more selective about which pages it trusts.

Do I need schema markup if I only publish on Hashnode or Dev.to?
Not directly, since the platform controls that layer. Focus your energy on content structure and canonical URLs instead, those travel with you no matter where you publish.

If you're building or rebuilding your own writing process around this, I'd genuinely love to hear what worked for you, drop it in the comments. You can also find me on X (swapnoneel123) or check out more of my work at swapnoneel.site.

How I made an AI Agent write in my voice

Swapnoneel Saha — Mon, 06 Jul 2026 09:54:48 +0000

Let's be honest, AI-written blogs have a certain... vibe. You know it, I know it, and your readers can smell it from the very first paragraph.

But here's my take: you can make AI write in your voice, just not with a "generic" prompt. What actually worked for me is an agent skill with three parts: a voice profile built from seven of my real writing samples, a kill list of AI phrases, and a feedback loop that turns my edits into permanent rules. And here comes the twist, the blog you are reading right now is the very first output of that system!

So, let me walk you through exactly how I built it, and you can judge for yourself whether it sounds like a human or not.

Why does AI writing sound so... AI?

Before fixing the problem, let's understand it from the ground up.

An LLM is trained on billions of documents, so by default, it writes like the average of all of them. That's where phrases like "in today's fast-paced world"s come from, and those perfectly balanced conclusions that never pick a side. It's not that the model is dumb. It's that the average of a million voices is no voice at all.

And your voice is the exact opposite of average. It's the specific way you break grammar rules, and the things you're willing to admit that others won't.

I've written multiple technical blogs for different startups including Keploy, Devbytes and many more, and have been blogging on Hashnode since 2023. So when I asked AI to draft posts "in my style" with a simple prompt, the result was always the same: grammatically perfect, structurally neat, and absolutely not me.

So, can you actually make AI write in your voice?

Well, yes. But you have to show it, not describe it.

"Write in a friendly, conversational tone" gives everyone on the internet the same friendly, conversational tone. What you need instead is a system that extracts the mechanics of your writing from real samples, and then enforces them like rules. Mine has three parts.

Part 1: The voice profile

I gave the agent seven samples of my writing: two journey blogs, one tutorial, one opinion piece, one comparison, three cold intros, and a small questionnaire about my tastes. And these are not just "any" samples, three of them are my past works that was cherry-picked by the system. And the other four were literally the topics given to me by Fable 5, so that it can understand my writing style better.

But here's the important part, the profile it built isn't a list of adjectives. It's mechanics:

My sentences constantly open with And, But, So, and Now (this exact paragraph included).
My posts move forward by asking the reader's next question, and then answering it.
Every big claim needs a personal receipt with a number, not a vague "many developers say".
At most two "!!" per post. Yes, it literally counts them.

And one more thing: newer samples always outrank older ones. My 2023 writing had habits I've dropped since, and the system knows my current voice wins every conflict.

Part 2: The kill list

The second file is a banned-patterns list. Every AI-ism I hate goes there: "delve", "seamless", "game-changer", rule-of-three sentences, em-dash chains, hedged conclusions that refuse to pick a winner, and emojis (all of them, I don't use emojis in my blogs, period).

The rule is zero tolerance. If a banned pattern shows up in a draft, the agent doesn't just delete it, it rewrites the sentence the way I would say it.

Part 3: The feedback loop (this is the part that actually matters)

Now, the first two parts get you maybe 80% of the way. The remaining 20% is where every "write like me" tool I've seen gives up.

Here's my loop: the agent writes a draft, I edit it like I normally would, and then a second skill diffs my final version against the draft. Every meaningful change gets generalized into a rule. If I cut a long intro once, that's a hypothesis. If I do it twice, it gets promoted to a confirmed rule that every future draft must follow.

And there's a hard cap of 30 active rules. Why? Because this whole system runs on a smaller, cheaper model, and a smaller model follows 30 rules well and drowns in 80. The intelligence lives in the files, not the model.

But does it learn from every single edit?

Well, no. And this was a deliberate design decision.

A one-off change (fixing a fact, rephrasing something topic-specific) teaches nothing about my voice, so it gets logged and forgotten. Only patterns become rules. Otherwise the agent would overfit to whatever mood I was in during one editing session.

There's also one rule I consider non-negotiable: the agent can never invent a story about me. All personal facts live in a single profile file, and if a post needs an anecdote that isn't in there, the agent has to stop and ask me. An AI confidently fabricating a personal memory in your published blog is so much worse than a boring paragraph.

Does it actually work?

Honest answer: I don't fully know yet, and I won't pretend otherwise.

This post is literally draft number one. The feedback loop has learned exactly zero rules from my edits so far, because there were no edits before this. You are looking at the "before" photo. If you can tell which sentences I touched after the agent wrote them, tell me in the comments, seriously!

And another honest admission: setting this up took me more effort than just writing 2-3 posts by hand (I literally wrote four new blogs as an assignment lol, so that the LLM can infer my writing style better). The payoff only makes sense because it compounds, every post I edit makes the next draft closer to me.

But the direction feels right, and I'm clearly not alone in thinking this way. The dev community has moved past one-shot prompting: Peter Steinberger's viral post ("you shouldn't be prompting coding agents anymore, you should be designing loops that prompt your agents") pulled 6.5 million views in June 2026 and set the timeline on fire for a week. And the Hacker News discourse in 2026 has shifted from shiny demos to making agents repeatable and trustworthy. A writing agent with a feedback loop is just that same idea, pointed at a blog.

What are agent skills, anyway?

If the term is new to you, let's zoom out for a second.

An agent skill is basically an onboarding document for an AI. It's a markdown file (usually called SKILL.md) with step-by-step instructions, plus supporting files it should read, that a coding agent like Claude Code loads before doing a task. Think of it like the difference between telling a new intern "write a blog" and handing them your company's full writing playbook.

And the beautiful part is that skills are portable and dumb-model-friendly. I have orchestrated multiple agentic workflows, and the lesson from there was the same: agents don't fail because the model is weak, they fail because the instructions are vague.

FAQ

How many writing samples do you need to clone your voice?
Seven worked for me, but coverage beats volume. One sample per content type (tutorial, opinion, comparison, narrative) teaches far more than ten samples of the same type, because your voice changes with the mode.

Can this work with a cheaper model?
That's the whole point. The voice profile, kill list, and rules carry the intelligence, so a smaller model just has to follow instructions. Save the expensive model for building the system, not running it.

How do you stop the AI from making up facts about you?
One canonical profile file, and a hard rule: if the fact isn't in the file, ask the human. Never generate a personal claim from thin air. Accepting that AI can't do the entire job for you, and you have to keep yourself in the loop, creates the difference.

Does this replace writing?
No, and I don't want it to. It replaces the first draft and the SEO chores. The opinions and the final edit are still mine, and honestly, that's the part I enjoy anyway. Even the current sentence that you are reading right now, was actually inserted by me during the edit.

So, should you build one?

If you publish regularly, yes. Build the voice profile. Really do. But don't skip the feedback loop, because without it you've just built a fancy prompt that will drift back into AI-slop within three posts.

And start smaller than I did: pick your five most representative pieces, extract the mechanics (not adjectives!), list ten phrases you'd never say, and make reviewing the diffs a habit. If you want a more detailed analysis about the system, just comment down below and I would be happy to help you all!

I'll be sharing more about this system as the feedback loop matures, including the numbers on how many edits it actually takes before drafts start needing none. If you want to follow that experiment, you can find me on X (swapnoneel123) or check out my other works at swapnoneel.site.

And that's a wrap! Have you tried making AI write like you? What worked, and what came out sounding like a LinkedIn bot? I would love to hear your experience. Thank you for reading, and have a nice day ahead!!