DEV Community

fcn06
fcn06

Posted on

Unifying Agent Orchestration and Model Gateways in Rust: Solving the Stateless Turn Problem

#rust #ai #agents #gateway

While developing multi-agent systems and LLM integrations in Rust, I found myself repeatedly confronting two related architectural challenges.

The first was an operational divide between two layers of infrastructure: a lightweight model gateway for routing and caching LLM requests, and an agent orchestration runtime responsible for tool execution, planning, and task resolution. Teams often end up deploying two separate systems for these workloads, duplicating provider configurations, authentication layers, and connection pools.

The second challenge was the "stateless turn" problem: many agent loops initialize each execution turn from scratch ([system_prompt, current_user_message]), immediately discarding previous tool outputs and conversational turns. While simple, this makes multi-turn reasoning brittle and deprives agents of contextual continuity.

This article shares how we approached these challenges around a unified Tokio-based architecture, and how we recently implemented an opt-in, four-tier context engine to give agents persistent conversational memory without sacrificing backward compatibility.

The Dual-Mode Architectural Pattern

Instead of maintaining a separate proxy daemon and an agent orchestrator, the system is organized around two complementary operational modes sharing the same core runtime:

+--------------------------------------------------------------------------------------------------+
|                                        RUNTIME MODES                                             |
+--------------------------------------------------------------------------------------------------+
|                                                                                                  |
|   MODE 1: MULTI-AGENT & MCP ORCHESTRATION               MODE 2: MODEL GATEWAY SERVER             |
|   (kickstart/multi_agent_orchestration_kickstart/)      (kickstart/gateway_kickstart/)           |
|                                                                                                  |
|   • Planner Agent (Dynamic DAG generation)              • POST /v1/chat/completions (OpenAI API) |
|   • Executor Agent (Workflow DAG execution)             • POST /v1/responses (Open Responses)    |
|   • Domain Specialists with native MCP Tools            • Stateful multi-turn chaining           |
|   • Pluggable Context & Memory Services                 • Multi-provider routing (Groq, Gemini,  |
|   • Evaluation & Self-Correction (Judge)                  OpenAI, Ollama, vLLM, llama.cpp)       |
|   • Agent-to-Agent (A2A) protocol contracts             • High-throughput lock-free cache         |
|                                                                                                  |
+--------------------------------------------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

Mode 1: Agent Orchestration via Model Context Protocol (MCP)

Mode 1 coordinates specialized agents through explicit message contracts:

  • Planner Agent: Analyzes high-level requests and constructs a directed acyclic graph (DAG) of tasks.
  • Executor Agent: Traverses the DAG, resolving dependencies and dispatching step execution.
  • Domain Specialists: Execute tools using the Model Context Protocol (MCP), handling tools over standard I/O, Server-Sent Events (SSE), or streamable HTTP transports.
  • Evaluation Service: An optional LLM-as-a-Judge validation step that assesses tool outputs and prompts the agent for correction if the result is malformed. 
User Request ──> Planner ──> Task DAG ──> Executor ──> Specialist Agent ──> MCP Tool ──> Evaluation ──> Response
Enter fullscreen mode Exit fullscreen mode

Mode 2: OpenAI-Compatible Model Gateway

Mode 2 acts as a standard inference gateway:

  • /v1/chat/completions: Drop-in compatibility with existing OpenAI SDKs, IDE extensions, and client pipelines.
  • /v1/responses: Stateful chaining that correlates turns using explicit response IDs.
  • Unified Provider Routing: TOML-configured routing across commercial APIs (Groq, Google Gemini, OpenAI) and local runtimes (Ollama, vLLM, llama.cpp).

Sharing a runtime means both modes share the same connection pooling, HTTP client configuration, secret management, and tracing telemetry.

Two Deployment Patterns: Start Unified, Let Usage Decide

To accommodate different operational requirements, we implemented two concrete deployment patterns:

  1. "Full Swarm" (Agents + MCP + Gateway): Runs the entire orchestration stack — planner, DAG executor, domain specialists with MCP tool servers, and the model gateway — in a single unified deployment. This provides an end-to-end autonomous environment where agents talk to LLMs directly through the local gateway with zero network hops, sharing connection pools and in-process memory.
  2. "Gateway-Only Swarm": Strips away agent orchestration and runs purely as a high-performance inference gateway. It handles /v1/chat/completions, provider failover, caching, and stateful response chaining (/v1/responses) for external client applications, developer tools, or existing third-party agent stacks.

The underlying philosophy is pragmatic: start unified, and let actual production usage patterns tell you if they should stay glued. If your inference proxying traffic grows 100x faster than agent orchestration, or if security requirements dictate isolating live tool execution behind private networks while keeping the gateway at the ingress, the gateway can be peeled off into an independent service without modifying client contracts or configuration formats.

The Problem with Stateless Agent Loops

During early iterations, our MCP agent loop followed a common pattern:

// A common pattern in basic agent runtimes
let messages = vec![
    Message {
        role: "system".to_string(),
        content: Some(system_prompt),
    },
    user_message,
];
Enter fullscreen mode Exit fullscreen mode

Every user interaction started with a fresh message array. While clean and deterministic for single-shot question answering, this architecture has distinct failure modes:

  1. Session Amnesia: Follow-up questions fail because prior exchanges and tool outputs vanish once the request terminates.
  2. Write-Only Logging: Memory traits are frequently defined with only a log() method, recording conversation logs for telemetry while providing no retrieval mechanism back to the agent.
  3. Prompt Coupling: Ad-hoc attempts to inject history often result in manual string concatenation in the HTTP handler, coupling transport logic to prompt formatting.

Designing a Pluggable 4-Tier Context Engine

To address this cleanly, we recently restructured the context architecture into four distinct operational tiers:

┌────────────────────────────────────────────────────────────────────────┐
│                        CONTEXT TAXONOMY                                │
├────────────────────────────────────────────────────────────────────────┤
│ 1. Conversational Working Context (Multi-turn session & thread state)  │
│ 2. Long-Term & Semantic Memory (User preferences, key facts, recall)   │
│ 3. Sovereign Identity & Security Context (Roles, clearance, metadata)  │
│ 4. Procedural & Skill Knowledge (Domain runbooks, safe tool sequencing)│
└────────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Rather than baking assumptions about prompt format or storage backends directly into the core execution loop, we introduced a ContextProvider abstraction into the commons layer:

#[async_trait]
pub trait ContextProvider: Send + Sync {
    /// Enrich the prompt or messages before the LLM thinking phase
    async fn enrich_context(
        &self,
        request: &ContextRequest,
        messages: &mut Vec<Message>,
    ) -> Result<()>;

    /// Post-turn hook to record dialogue or extract persistent facts
    async fn on_turn_complete(
        &self,
        request: &ContextRequest,
        user_message: &Message,
        assistant_message: &Message,
    ) -> Result<()>;
}
Enter fullscreen mode Exit fullscreen mode

How the Execution Pipeline Works

When a request arrives, McpAgent executes a two-phase context pipeline:

User Request (with session_id & metadata)
          │
          ▼
┌────────────────────────────────────────────────────────┐
│              CONTEXT ASSEMBLER PIPELINE                │
│                                                        │
│  1. Base System Prompt                                 │
│  2. Identity Provider   ──> Injects identity block     │
│  3. Skill Provider      ──> Injects tool runbooks      │
│  4. Fact Memory Provider──> Recalls relevant facts     │
│  5. History Provider    ──> Loads sliding window turns │
│  6. Current User Turn                                  │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
           ┌───────────────────────────────┐
           │      McpAgent Run Context     │
           │      (Enriched Message List)  │
           └───────────────┬───────────────┘
                           │
                           ▼
           ┌───────────────────────────────┐
           │    McpAgent Execution Loop    │
           │ Thinking ──> Executing Tools  │
           └───────────────┬───────────────┘
                           │ (Assistant Response)
                           ▼
┌────────────────────────────────────────────────────────┐
│                   POST-TURN FLUSH                      │
│  • Persist User & Assistant turns to MemoryService     │
│  • Invoke `on_turn_complete` on all context providers  │
└────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

  1. Sliding Window History (HistoryContextProvider):
    The provider queries MemoryService::get_conversation(session_id, limit), fetching the last $N$ turns (configurable via TOML) and injecting them between the system prompt and the latest user turn. Older turns can be safely compacted or truncated without blowing past LLM context limits.

  2. Persistent Fact Recall (SemanticMemoryContextProvider):
    A dedicated db_facts store exposes store_fact and recall_facts. Before LLM execution, facts matching the user's query or category are recalled and attached as structured context.

  3. Identity & Security Context (IdentityContextProvider):
    Injects caller metadata, tenant identification, active permissions, and clearance level into an <identity_context> block, ensuring the model stays aligned with authorization boundaries.

  4. Zero-Breaking-Change Guarantee:
    To ensure existing deployments remain stable, all methods added to the MemoryService trait provide default no-op implementations. If an agent does not configure memory services or context providers, it continues operating in the original stateless mode with zero performance regression.

Why Rust for Multi-Agent & Gateway Infrastructure?

Rust is well-suited for this specific intersection of workloads for several reasons:

  • Concurrency Without Global Locks: Shared state across concurrent sessions (such as DashMap-backed memory tables and cached tool schemas) operates cleanly across Tokio worker threads without coarse mutex contention.
  • Low Latency & Predictable Footprint: When proxying high-token streaming requests or managing complex tool calls, predictable memory ownership avoids garbage-collection pause spikes.
  • Compile-Time Protocol Validation: Using strongly typed structs for A2A and MCP message boundaries surfaces serialization and schema mismatches at compile time rather than midway through an automated workflow.

Configuration Example

Enabling context memory in an agent runtime is handled declaratively in the configuration file:

[agent_mcp]
agent_mcp_model_id = "openai/gpt-oss-20b"
agent_mcp_llm_url = "https://api.groq.com/openai/v1/chat/completions"
agent_mcp_system_prompt = "You are a specialized customer domain agent."

# Context Engine Settings
agent_mcp_history_length = 10
agent_mcp_enable_memory_recall = true
agent_mcp_enable_identity_context = true
Enter fullscreen mode Exit fullscreen mode

Trade-offs and Open Questions

In designing this system, we encountered several trade-offs that have no single "correct" answer:

  1. Sliding Window vs. Recursive Summarization: A fixed sliding window is fast, deterministic, and adds zero LLM cost. However, long-running threads inevitably drop older details. A recurring compactor that periodically condenses earlier turns into a narrative summary solves this, but incurs additional latency and inference cost.
  2. Key-Value Fact Storage vs. Vector Embeddings: For local and edge environments, key-value fact matching (by key, tag, and query substring) avoids the operational burden of running an embedding model or vector database. For large-scale unstructured corpora, however, vector embeddings remain necessary.
  3. Glued vs. Decoupled Deployments: Providing both "Full Swarm" and "Gateway-Only" patterns allows teams to start simple and observe traffic patterns empirically. Rather than guessing the ideal microservice boundary on day one, letting actual load, blast radius considerations, and operational ownership dictate whether the gateway and orchestrator remain glued has proved much less painful than premature decomposition.

We would be glad to hear from developers building agent systems or LLM gateways:

  • How do you manage conversation history compaction across extended agent workflows?
  • Do you prefer combining proxy routing and agent execution into a single binary, or keeping them decoupled across services?

Project & Code

If you are interested in exploring the implementation, testing the multi-agent kickstarts, or examining the MCP runtime contracts, the project is open-source under Apache-2.0:

👉 Swarm on GitHub

Contributions, critiques, and discussions are welcome.

Top comments (2)

Collapse
 
raknaos profile image
Baptiste Le Bouquin •

The "stateless turn" diagnosis hits a real nerve. I run a small fleet of agents on a VPS and the failure mode is rarely model quality — it's loops that rebuild context from scratch each turn and drop the intermediate state (tool outputs, half-resolved errors) where the useful information actually lived.

The question that would help me evaluate the four-tier design: what's the eviction policy between tiers? In practice persistent memory is not a storage problem, it's a dropping problem — when the budget is tight, a tiered model without explicit demotion criteria ends up being a cache with extra steps.

On the gateway+orchestrator unification: did you hit contention between the two modes on the shared pools — long agent turns holding connections while short inference requests queue? That's the reason we keep the routing layer separate, and I'm curious whether tokio's scheduling actually made it a non-issue.

Collapse
 
reidmarlow profile image
Reid Marlow •

The unified Tokio runtime makes sense for keeping connection pooling and provider failover in one place, especially when internal agents make dozens of short tool calls that would otherwise suffer HTTP hop latency against an external proxy.

The trap I run into with the four-tier context split is fact invalidation across turns. A sliding window preserves immediate recency, but if a persistent fact provider caches environment state from turn one, such as a file path or a tool output schema, and a specialist agent alters that state in turn four, the prompt ends up injecting contradictory records into the system context.

Pairing the key-value fact store with a turn-indexed invalidation key or dirty flag has saved me from debugging cases where the model hallucinates simply because it trusted an earlier recalled fact over its current working memory.