Uber’s MCP Gateway Keeps 5,000 Tools Out of the Context Window

Uber just detailed their MCP Gateway handling 800+ servers and 5,000+ tools. The hard part wasn’t the protocol. It was keeping the model’s context window from drowning in tool schemas while still letting agents discover and call internal APIs without rewriting every service.

The scale problem

Early MCP integrations at Uber were ad-hoc. Teams wired their own servers. Discovery was manual. Security was inconsistent. Once hundreds of teams started shipping agents, the fragmentation showed up as duplicated infrastructure and tools that agents couldn’t reliably find.

Their answer is a single gateway: MCP Registry as the control plane (catalog, ownership, enablement) and Proxy Gateway as the data plane (runtime translation and execution). Existing HTTP, gRPC, and TChannel services stay untouched. AutoCrawler (Cadence workflows watching the IDL registry) scans Protobuf/Thrift definitions, generates MCP schemas, runs an LLM over the comments to produce agent-friendly descriptions, and registers everything disabled-by-default. Owners review and flip the enable bit. Config changes propagate without restarts.

Native MCP servers heartbeat, get discovered the same way, and are proxied. Third-party ones (Jira, Google) go through token exchange plus the gateway’s auth and redaction.

Context bloat is the real tax

MCP has no built-in cross-server search. If you wire 800 servers into an agent prompt you burn the context budget before the first useful call. Uber’s Omni MCP is the escape hatch: one proxy that exposes four tools:

• discover_server — query intent → matching servers
• discover_tools — tools on a server
• get_tool_schema — full schema only when needed
• invoke_tool — call it

Agents do incremental discovery instead of loading the entire catalog. Response Projection adds a GraphQL-style field so the gateway trims the response to the paths the model actually asked for. Code Mode (aifx mcp search / aifx mcp call) lets coding agents write results to files instead of stuffing them into the context window.

That pattern is the same one that shows up in any real agent loop once the tool count leaves double digits.

What this looks like for a local tool

Most of us aren’t running Cadence workflows. We still hit the same constraints when the “tools” are local processes instead of internal microservices.

A live call assistant on Windows is a good example. You need:

• mic capture (WASAPI exclusive or shared)
• system-audio loopback (render endpoint opened as input)
• streaming STT
• optional local LLM for turn-taking or translation
• an overlay that doesn’t get captured by screen-share

Each of those is a “tool” with its own latency budget, format constraints, and failure modes. If you dump every schema and every intermediate buffer into the model context you blow the window and the latency budget at the same time.

The practical equivalent of Omni MCP is a thin dispatcher that keeps the full capability list out of the prompt until the model asks for it. Something like:

// pseudo — the real path is in the capture crate
enum Capability {
    MicCapture { sample_rate: u32, channels: u16 },
    Loopback { device_name: String },
    SttSession { language: String, keyterms: Vec<String> },
}

fn dispatch(intent: &str, state: &AppState) -> Result<Capability, Error> {
    // cheap keyword/embedding match first, full schema only on hit
    match intent {
        s if s.contains("system audio") => Ok(Capability::Loopback { 
            device_name: state.default_render_device() 
        }),
        s if s.contains("transcribe") => Ok(Capability::SttSession { 
            language: state.language(), 
            keyterms: state.keyterms() 
        }),
        _ => Err(Error::UnknownIntent),
    }
}

On the Windows side the loopback path is the one that usually surprises people. With cpal you open an input stream on a render device; WASAPI treats it as loopback. Shared mode goes through the audio engine mixer, exclusive mode skips it but you fight the rest of the machine for the endpoint. Either way you have to resample to 16 kHz mono before the STT model sees it, and you have to gate the mic track so the speakers don’t get re-transcribed.

The latency stack ends up looking like:

1. WASAPI buffer (10–30 ms depending on period)
2. resample + mono mix (~1–2 ms)
3. STT first-token (model-dependent, 50–200 ms for good local ones)
4. optional LLM turn decision

Total under a few hundred milliseconds is usable for a call overlay. Anything that forces a full tool catalog into the prompt adds another round-trip and usually kills the budget.

Tradeoffs that actually matter

• Disabled-by-default + owner approval prevents the “agent called the wrong service” class of bugs. The cost is that new tools sit dark until someone flips the switch.
• Protocol translation in the proxy means the model only ever speaks MCP/JSON-RPC. Downstream protobuf/Thrift never leaks into the prompt. The downside is an extra hop; Uber routes it through their service mesh so the hop is already paid for.
• Incremental discovery + response projection keeps token use sane. You trade a couple of extra tool calls for not having to truncate schemas or risk the model hallucinating arguments it never saw.
• For local apps the same idea applies: keep the heavy schemas (full WASAPI config, model tokenizers, overlay window handles) out of the prompt. Expose a small set of high-level actions and let the agent pull details only when it needs them.

The gateway pattern is just service-mesh thinking applied to agent tools. Once the number of callable things exceeds what fits comfortably in a context window, you either build something like Omni MCP or you watch every agent session start with a 30 k token preamble of tool definitions.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.