Your AI governance problem probably isn't a policy problem; it’s a routing one.
Let’s paint a common scenario: Your team uses OpenAI, another adds Anthropic, then the product team connects another model. Then someone builds an agent with MCP tools. A few months later, the company has several AI applications, multiple providers, and a growing bill.
The question is: who can use which model, how much can they spend, and what happens when a provider fails?
That is when AI governance stops being a policy document and becomes an infrastructure problem.
In this article, we'll look at why enterprise AI governance gets harder as LLM traffic grows, where an AI gateway fits into the architecture, and how Bifrost can govern access, routing, budgets, rate limits, observability, and AI agent tools.
Bifrost is an open-source AI gateway built in Go by Maxim AI. It sits between your applications and AI model providers, routing requests through a unified API instead of making each application manage every provider separately.
Let’s get started.
Why enterprise AI governance becomes an infrastructure problem
The first generation of enterprise AI was mostly about choosing models. The problem is different now.
LLM traffic is not like ordinary API traffic. Cost depends on tokens. Latency varies by model and provider. A single agent can make many model and tool calls. Providers have different limits and capabilities. Teams may want different models for different jobs.
So the traffic itself becomes something the platform team needs to manage.
At scale, the same request pattern repeats across applications, teams, providers, and agents. Governance gets difficult simply because the system grows.
The problem with governing AI in every application
Every AI request an enterprise makes needs an answer to questions like: who is allowed to make it, which model or provider it's allowed to use, and whether anyone can find a record of it afterward.
That's what "governing AI traffic" means in practice: not a policy document, but those questions getting answered correctly, every time, for every request.
The first move might be to answer them inside each application. Give the app its own provider keys, its own retry logic, its own spending limits, its own logging. For one team, on one service, this works fine.
It stops working once there are twenty teams doing it. Each one solves the same questions once, at different times, in a slightly different way, because nothing forces any two of them to do it the same way.
So when a provider changes its API, or a new model gets approved, or security wants a new rule, that's not one change anymore. It's twenty changes, made by twenty people, on twenty schedules, and every application has quietly become part of the governance system whether it was built for that job or not.
The limit is not the volume of traffic, but not having a single place where any of this policy lives. That's the gap an AI gateway is built to close.
The AI gateway as the control plane
A better way to go about handling different requests, as shown in the diagram above, is to route applications and agents through an AI gateway connected to the AI model providers.
The gateway is the control plane where identity, access, routing, budgets, rate limits, guardrails, and observability can be applied consistently.
It’s not simply having another proxy; rather, it’s having a place where policy meets the request while the request is happening.
A typical request can move through:
Identity: who is making the request, usually a Virtual Key tied to an app, team, or agent.
Access policy: which providers, models, and tools that identity is allowed to use.
Budget and rate limit: whether the request fits within the spending and volume limits set for it.
Routing: which provider and model should actually handle the request.
Guardrails: a check for policy violations or sensitive data before anything reaches the user.
Provider: the model provider, like Groq, OpenAI, or Anthropic, that generates the response.
**Observability: **a record of the request, response, cost, and outcome for later audit.
If a check fails, the request stops there, emphasizing the principle of one governed path, where the idea is to effectively manage traffic along its existing route.
What should an AI gateway govern?
When traffic passes through one layer, the governance questions become much easier to answer. You can start with something as simple as knowing who is making a request. That might be a user, an application, a team, or a Virtual Key. From there, you can decide what that caller is allowed to access and keep its usage separate from everyone else's.
The same applies to models and providers. Not every application needs access to everything. A support application might only need a few approved models, while an internal agent might need access to a different set.
There is also the question of how much traffic each application should generate. Budgets can keep spending under control, while request and token limits prevent a single application from consuming more than its share. The Budget and Limits controls let those limits be applied at different levels.
With several models and providers in the mix, the gateway also needs to decide where each request should go. Routing can direct traffic to particular providers or models, distribute requests, and provide a fallback when something goes wrong.
Some requests should never make it that far. Guardrails can inspect what is going into and coming out of a model and block requests that violate the rules.
And after the request has been handled, you still need to understand what happened. How much traffic did an application generate? Which model handled it? How long did it take? Did a provider fail? Good observability turns those questions from guesses into things you can actually answer.
That is what governance looks like at the traffic layer: knowing who is making the request, what they can do, how much they can use, where the request goes, and what happened along the way.
Agents make the governance problem bigger
As AI agents get more development and adoption, the governance challenge gets bigger. At first, the question was simple: "Can this application call GPT?" But for agents, that's just the start.
These agents can do a lot more than just call a model. They can read files, query databases, search internal systems, use Git, and call other tools through the Multi-Channel Protocol. Because of this, governance needs to include control over tool access as well.
For example, a research agent might only need to read files but shouldn’t be able to delete them. On the other hand, an engineering agent might need access to Git.
Bifrost’s MCP Tool Filtering lets you set up a Virtual Key. This key can explicitly allow certain clients and tools to be accessed. If you don’t set up any MCP configuration, the default is to deny access to everything; this keeps things secure.
So instead of asking, "Which model can this application use?" we need to ask, "What can this AI system actually do?" This shift means we have to think more about how to govern these agents and what permissions they really need to work safely and effectively.
Cost and reliability belong in the same layer
The gateway is also where governance meets business economics. If one team's agent suddenly makes ten times as many requests. With gateway-level budgets and usage data, you can see which key, team, provider, or workflow caused the increase.
The same layer can handle reliability. Bifrost's routing controls can distribute requests across configured provider or model paths, which also caters to any fallback behavior when a provider path is no longer available.
Governance is not only about saying no. It also keeps approved traffic reliable.
The governed AI traffic layer
This is where Bifrost comes in as an open-source, high-performance AI gateway. It gives applications a unified API and interface to connect across the AI model providers.
To learn more about the engineering story behind Bifrost, watch the Runtime episode on YouTube. It covers why the team built Bifrost in Go, the unified API, routing and failover, observability, MCP, and the infrastructure decisions behind a high-throughput AI gateway.
Let's put this into practice
We built a Customer Support Assistant using Next.js and Groq for AI inference. The application does not call a provider directly. Its baseURL points to Bifrost, and the Bifrost Virtual Key identifies the workload, with the API key credential entered in the Bifrost dashboard.
The Customer Support application only sends chat requests through the AI gateway.
Also, the governance policy includes a request and token limit, a monthly budget, and an approved model list configured in a config.json file and also managed in the Bifrost dashboard.
The Bifrost Gateway dashboard runs at http://localhost:8080
The Virtual Key Rankings attribute requests, tokens, and cost to the Customer Support Assistant workload.
In the image above, the enterprise-support-assistant Virtual Key accounts for all 35 attributed requests and $0.0019 spent. This observability enforces policy and gives finance and platform teams a shared view of LLM usage.
The Virtual Key Rankings tab also shows how Bifrost attributes traffic back to each workload. In this demo, every request from the Customer Support Assistant arrives with the Virtual Key, and Bifrost maps that key to the workload name, rolls up requests, tokens, and costs in one place.
From the LLM logs tab in the dashboard, we can view total requests, success rate, average latency, total tokens, and cost, along with a log and timestamp for every request the application sent.
An example of a policy block is the HTTP 429 code when the rate limit has been exceeded. This enforcement is done from the AI gateway before the request is sent to the AI provider.
When an AI model is allowed in the Bifrost configuration, it shows an HTTP 403 message, as shown below.
The LLM logs also show the 403 error in the dashboard, with the Groq logo displayed like so:
You can also view the AI model rankings in the Bifrost dashboard and export the data when needed.
For the Customer support application use case, this shows which AI models have been used for the responses.
From the image, we can see that there are two models in use:
- openai/gpt-oss-20b
- openai/gpt-4o-mini
It also shows the requests on each model and the success rates, tokens used, costs, average latency, and throughput. This gives more insights into the AI models, which help with better understanding and decision-making.
As shown in the Customer support application above, Enterprise governance is enforced centrally, in the request path. The LLM traffic flows through one gateway; budgets and limits control growth; model and tool access can be restricted; routing can keep approved traffic moving; and requests remain traceable.
The LLM provider (for instance, OpenAI, Anthropic) the application needs is defined through a provider configuration. Then the application gets its own identity with a Virtual Key, which gives a clear boundary for access and usage.
From there, you can put required limits around the traffic. The budgets and rate limits help keep an application or team from using more than it should.
As the system grows and you add more models or providers, and introduce routing so requests can go to the right place, with fallback when an LLM provider is unavailable.
And once the application becomes an agent, MCP Tool Filtering lets you decide which tools that agent can actually access.
You can get started with one application and one provider, then add the controls as the traffic and the system grow.
Next steps
Enterprise AI governance is not only about writing the rules. It is about enforcing them once AI traffic starts spreading across teams, applications, models, providers, and agents.
That is why an AI gateway matters, giving you a place to see the traffic, apply policies, control access and spend, route requests, govern agent tools, and keep a record of what happened.
Instead of asking every application to solve these problems on its own, you can handle them at the layer every request already passes through, and your organization can scale AI securely without slowing innovation.
To get started with Bifrost, check out the documentation.
Top comments (2)
Great share, Ekemini!!
Thank you Swapnoneel!