fcn06

Posted on Aug 18

Introducing Swarm: Multi-Agent Orchestration and an LLM Gateway in Pure Rust 🦀

#rust #ai #mcp #llm

While experimenting with multi-agent systems, I kept ending up with two separate pieces of infrastructure: an orchestration layer for agents and tools, and a gateway layer for routing LLM requests.

I wanted both to share the same runtime, provider abstractions, state management, and protocol contracts.

So I built Swarm, an open-source AI orchestration framework and model gateway written in Rust.

The Dual-Mode Architecture

Many AI stacks end up separating these concerns: a dedicated proxy for lightweight routing and a separate orchestrator for more complex reasoning. Swarm unifies both patterns around a single high-performance Tokio runtime.

+--------------------------------------------------------------------------------------------------+
|                                        SWARM MODES                                               |
+--------------------------------------------------------------------------------------------------+
|                                                                                                  |
|   MODE 1: MULTI-AGENT & MCP ORCHESTRATION               MODE 2: MODEL GATEWAY SERVER             |
|   (kickstart/multi_agent_orchestration_kickstart/)      (kickstart/gateway_kickstart/)           |
|                                                                                                  |
|   • Planner Agent (Dynamic plan generation)             • POST /v1/chat/completions (OpenAI)     |
|   • Executor Agent (Workflow DAG execution)             • POST /v1/responses (Open Responses)    |
|   • Domain Specialists with MCP Tool integration        • Stateful multi-turn chaining          |
|   • Discovery & Memory services                         • Multi-provider (Groq, Gemini, OpenAI,  |
|   • Evaluation & Judge Service                            Ollama / vLLM / local endpoints)       |
|   • Resilient OAuth2 / JWT authentication               • High-throughput lock-free cache        |
|                                                                                                  |
+--------------------------------------------------------------------------------------------------+

The key idea: Swarm can run as a full agent orchestration stack or as a standalone LLM gateway without requiring two unrelated frameworks.

Mode 1: Deterministic Multi-Agent Orchestration with MCP

Coordinating multiple agents becomes much easier when service boundaries and message contracts are explicit.

Mode 1 splits responsibilities across decoupled, specialized services:

Planner Agent: Analyzes incoming user requests and dynamically constructs execution DAGs.
Executor Agent: Resolves task dependencies and controls step execution.
Domain Specialists : Execute live tools via a native Model Context Protocol (MCP) runtime (supporting SSE and streaming tool calls).
Discovery & Memory : Maintain service registries and conversational state.
Evaluation Service : Built-in LLM-as-a-Judge validation loop for output verification and self-correction.

Inter-agent communication relies on type-safe agent-to-agent (A2A) message contracts, catching many contract and integration errors during development and compilation.

User Request
    ↓
Planner
    ↓
Execution DAG
    ↓
Executor
    ↓
Weather Agent
    ↓
MCP Weather Tool
    ↓
Evaluation
    ↓
Final Response

Mode 2: OpenAI-Compatible Model Gateway

Mode 2 exposes an OpenAI-compatible gateway for client applications, developer tools, and automated pipelines.

OpenAI Compatibility (POST /v1/chat/completions): Works with standard OpenAI SDKs, Cursor, and developer extensions.
Stateful Responses (POST /v1/responses): Supports multi-turn conversation chaining using explicit previous_response_id references.
Unified Multi-Provider Routing: Route requests across Groq, Google Gemini, OpenAI, or local backends such as Ollama, vLLM, and llama.cpp through TOML configuration.

[server]
bind_address = "0.0.0.0:8080"
log_level = "info"

[models]
default_model = "openai/gpt-oss-20b"

[providers.groq]
api_url = "https://api.groq.com/openai/v1/chat/completions"

[providers.google]
api_url = "https://generativelanguage.googleapis.com/v1beta/models"

[providers.custom]
# Local inference (Ollama / vLLM / llama.cpp / LocalAI)
api_url = "http://localhost:11434/v1/chat/completions"
recommended_models = ["llama3.2:latest", "mistral:latest", "deepseek-r1:8b"]

Why Rust?

Rust gives Swarm a few useful properties for orchestration and gateway workloads:

⚡ Low-Overhead Request Handling: Built on Tokio and Hyper for asynchronous, high-concurrency gateway workloads.
🔒 Concurrent Session Management: DashMap and Arc-based stores allow shared state across concurrent requests without a global application lock.
🪶 Small Runtime Footprint: No garbage collector and predictable memory ownership make the gateway suitable for lightweight deployments.
🛡️ Strongly Typed Protocols: Internal MCP and A2A message contracts reduce schema mismatches and integration errors.

Quickstart

You can test either mode locally in minutes:

git clone https://github.com/fcn06/swarm.git
cd swarm

# Configure your provider keys
cp .env.example .env

Option A: Launch the Model Gateway (Mode 2)

./kickstart/gateway_kickstart/01_launch_gateway.sh

# Test OpenAI-compatible completions
curl -X POST http://localhost:8080/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [{"role": "user", "content": "Explain Swarm architecture in 2 sentences."}]
  }'

Option B: Launch Multi-Agent MCP Suite (Mode 1)

# Launch agents, MCP server, and infrastructure services
./kickstart/multi_agent_orchestration_kickstart/01_launch_all.sh

# Run a live MCP tool test query
./kickstart/multi_agent_orchestration_kickstart/02_test_weather_query.sh "What is the current weather in Boston ?"

Open Source & Community

Swarm is fully open-source under the Apache-2.0 license. We rely on and contribute back to the emerging Rust AI ecosystem, including the official MCP Rust SDK and A2A Protocol.

I'm especially interested in feedback from people running agent systems or LLM gateways in production:

Would you rather deploy the orchestration and gateway as one runtime, or keep them completely separate?

If you try Swarm, I'd also love feedback on the MCP runtime, gateway compatibility, and APIs.

DEV Community