Enterprises spent the last decade building comprehensive API programs to manage internal services. A large organization now runs thousands of internal APIs, each featuring a specific owner and authentication scheme. These systems provide the essential context agents need for customer records, inventory, and support tickets as they enter production.
Resolving Scalability Issues in Agent Connectivity
The Model Context Protocol (MCP) established a standard way to connect agents to external tools. However, the common method of exposing APIs through MCP involves creating one tool for every API operation. This specific strategy fails quickly when applied to a large enterprise estate. Typical enterprise CRM or ERP systems often contain hundreds of operations. When an agent requires several of these APIs, it becomes overwhelmed by hundreds of different tool definitions.
Three major problems occur when using this one-to-one mapping approach. First, the financial cost increases significantly. Every tool definition must load into the model context before any work begins. Every intermediate result then flows back through the model, which means the organization pays for every hop in tokens. A simple workflow like checking a shipment status becomes expensive very quickly.
Accuracy also suffers as the list of tools grows. Once the number of tools exceeds what the model can reasonably handle, selection logic degrades. The agent often picks the wrong tool or makes unnecessary calls. Finally, latency becomes a major issue for the end user. Processing more context and managing constant back-and-forth communication slows the agent down significantly.
Limitations of Traditional Integration Curation
Teams frequently attempt to fix these issues by manually trimming tool lists. They split large APIs into smaller, specialized servers to keep things manageable. While this helps in the short term, it creates a new problem. Every specific agent use case then requires its own hand-maintained integration. This leads to the same type of integration sprawl that original API programs were designed to prevent.
The Shift Toward Code Mode Interfaces
The industry is currently moving toward a different pattern known as code mode. In this setup, a server exposes an API as a single interface that resembles a software development kit. The agent writes short code snippets against this interface, which then run in a secure sandbox. Only the final result of that execution returns to the model context.
This process involves three distinct steps: search, understand, and execute. The agent searches the API surface for a specific operation and pulls the schema on demand. This keeps the full API available while ensuring only the necessary parts reach the context window. It aligns perfectly with how large language models function, as they are trained extensively on code.
Integrating Agent Traffic with Existing Gateways
Efficiency gains from using code mode are quite dramatic. Because the agent works with code, a single snippet can chain multiple calls together using loops or conditionals. This reduces the number of round trips required for complex workflows. In internal testing, a workflow that previously used 120,000 tokens dropped to just 1,200 tokens. This represents a massive reduction in resource consumption.
However, running agent-written code requires a trustworthy sandbox. Platform teams must have full visibility into every call the code makes to ensure security. Adopting this pattern means treating observability as a primary requirement. Without these safeguards, the benefits of code mode could be offset by new security vulnerabilities within the internal network.
Utilizing Centralized Policy Enforcement
Governance remains a critical concern even after solving the context window problem. Any agent calling a production API needs the same protections as any other client. This includes robust authentication, authorization, and rate limiting. Many teams mistakenly recreate these controls for every new integration. This leads to credentials being hidden in server configurations and traffic bypassing the standard gateway.
A superior strategy involves treating the context layer as a direct extension of the existing API platform. The specifications and access policies already live where the API is managed. Organizations should generate the agent-facing interface from this existing source of truth. By routing agent calls through the same gateway used by other consumers, existing policies apply automatically.
Improving Discovery Through Centralized Catalogs
Discovery is another vital component of a successful agent strategy. Each agent-ready interface should be registered in a central catalog. This allows other teams to find and reuse existing integrations instead of building new ones for the same system. Centralization ensures that the agent becomes a governed client rather than a shadow IT project.
Composing Specialized Context Surfaces
APIs are only one part of the broader enterprise data picture. Most organizations already run various servers from SaaS vendors and internal teams. A truly useful agent needs to draw on all these sources simultaneously. A well-designed context layer allows platform teams to compose these sources into a single interface.
This composition should be scoped to the specific job of the agent. For example, a support agent might need access to ticketing and shipping data but should never see payroll information. This scoping serves as a vital security control by limiting the agent’s reach. It also serves as a performance control by keeping the model focused on relevant data.
Implementation at the Infrastructure Level
Curation is the ultimate goal of a context layer. Simply providing more context does not result in a better agent. The objective is to make enterprise systems available while keeping the data sent to the model small. Many companies are now treating this context as a form of infrastructure. They generate it from managed systems and govern it with existing policies.
The evolution of APIs into the backbone of digital business happened because they became secure and discoverable. AI agents require that same level of discipline to be effective in an enterprise setting. Most organizations have already built the necessary foundation. Now, the focus must shift to making those systems accessible to autonomous agents in a controlled and efficient manner.
Managing AI and API Traffic Together
New tools like the Kong AI Gateway are designed to bridge the gap between traditional APIs and AI needs. By generating code mode interfaces, these tools allow agent access to reside on the same platform used for standard API traffic. This unified approach ensures that as agents become more prevalent, they do not disrupt the existing security architecture of the organization.
As enterprises move forward, the focus will remain on selectivity. The fastest-moving companies will be those that can deliver context on demand without sacrificing performance. By using the principles of code mode and centralized governance, IT managers can ensure their API programs remain relevant in the age of artificial intelligence.
Top comments (0)