You Don't Need to Choose Between a Gateway and an Agent Framework

When I first published Swarm on GitHub, most questions weren't about Rust or MCP. They were about timing and categorization:

"We just need a lightweight gateway for multi-provider routing; agents feel like overkill."

"We already run an orchestration framework; why would we replace our proxy?"

This reaction highlights a false dichotomy currently plaguing the AI infrastructure ecosystem: the assumption that a gateway and an agent orchestrator must be two completely different products.

In practice, teams rarely wake up needing full-blown multi-agent autonomous swarms on Day 1. But when they start with a standalone proxy, they inevitably hit a wall — patching together Python microservices, external vector state stores, MCP bridges, and ad-hoc eval scripts. Every evolution requires a rewrite.

The core premise of Swarm is different: a single, pure-Rust runtime where you don't choose between a gateway and an orchestrator — you simply choose which capabilities to turn on.

---

The AI Adoption Ladder

Most engineering teams evolve their LLM stack along a predictable trajectory:

Rung 1: OpenAI-Compatible Gateway   (Drop-in replacement for hardcoded SDKs)
  └── Rung 2: Multi-Provider Fallbacks (Groq, Gemini, Ollama, vLLM via TOML)
        └── Rung 3: Stateful Sessions     (Previous response chaining & context)
              └── Rung 4: Native MCP Tools    (SSE + Streamable HTTP tool execution)
                    └── Rung 5: Multi-Agent DAGs  (Planner + Executor + Specialists)
                          └── Rung 6: Built-in Evals    (LLM-as-a-Judge & policy gates)

You can stop at any rung and have a lean, production-grade binary. When you're ready for the next level, you change a configuration flag — not your architectural foundation.

---

Rung 1 — Just a Low-Latency Gateway

If your immediate goal is simply eliminating hardcoded API keys and single-vendor SDK locks, Swarm acts as an OpenAI-compatible drop-in front door with sub-millisecond native routing overhead.

# Spin up the gateway in seconds
./kickstart/gateway_kickstart/01_launch_gateway.sh

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "Explain progressive disclosure in software."}]
  }'

You get instant OpenAI compatibility. No agent overhead, no background worker queues, no forced abstractions.

---

Rung 2 — Multi-Provider & Local Model Routing

When rate limits hit or you need cost-effective fallbacks across cloud and local runtimes (Groq, Anthropic, Gemini, Ollama, vLLM, llama.cpp), routing is declared cleanly in config.toml:

[providers.groq]
api_url = "https://api.groq.com/openai/v1/chat/completions"
weight = 80

[providers.local_vllm]
api_url = "http://localhost:8000/v1/chat/completions"
recommended_models = ["meta-llama/Llama-3.3-70B-Instruct"]

Your applications continue calling the same /v1/chat/completions endpoint. Failover, load distribution, and local-inference routing happen invisibly inside the runtime.

---

Rung 3 — Stateful Conversations via /v1/responses

Multi-turn chat state is where teams often bolt on an external Redis or PostgreSQL session manager. Swarm provides explicit turn-by-turn state management natively through /v1/responses using previous_response_id chaining:

curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "groq/llama-3.3-70b-versatile",
    "input": "Calculate the Q3 cloud infrastructure spend.",
    "previous_response_id": "resp_01JMW892KPA7XYZ"
  }'

State is managed by the runtime, eliminating client-side conversation bloat while keeping state inspection simple and deterministic.

---

Rung 4 — Native Model Context Protocol (MCP)

When your model needs real-world context — database schemas, filesystem access, API calls — you shouldn't have to migrate to a heavy agent framework just to call tools.

Swarm natively supports MCP (over both SSE and streamable HTTP) directly inside the gateway layer:

[mcp_servers.postgres_db]
transport = "sse"
url = "http://localhost:3001/sse"

[mcp_servers.git_tools]
transport = "http"
url = "http://localhost:3002/mcp"

Tool discovery, argument validation, and streaming tool execution run within the same engine that routes your completions.

---

Rung 5 — Coordinated Multi-Agent Workflows

When single-prompt loops cannot solve compound tasks, Swarm activates its autonomous orchestration engine:

User Intent
   │
   ▼
[ Planner ] ──► Builds Execution DAG (Dependencies & Concurrency)
   │
   ▼
[ Executor ] ──► Dispatches tasks across Domain Specialists
   │
   ├── Specialist A (Data Analyst + Postgres MCP)
   └── Specialist B (Report Writer + File MCP)
   │
   ▼
Unified Response

Why this matters: Rung 5 reuses the identical provider configurations, fallback pools, state engine, and MCP tool connectors established in Rungs 1–4. There is no secondary agent daemon or translation bridge.

---

Rung 6 — Built-in LLM-as-a-Judge Evaluation

The final rung is the one most gateways and agent frameworks omit entirely: closing the loop on quality.

Instead of exporting logs to an external SaaS pipeline, Swarm embeds an LLM-as-a-Judge loop. It scores intermediate DAG outputs, validates MCP tool results against deterministic schemas, and flags hallucinated responses before they reach client applications:

curl -X POST http://localhost:8080/v1/eval/judge \
  -H "Content-Type: application/json" \
  -d '{
    "response_id": "resp_01JMW892KPA7XYZ",
    "criteria": ["correctness", "grounding", "conciseness"],
    "judge_model": "openai/gpt-4o"
  }'

This foundational layer enables our upcoming roadmap items: policy-based dynamic routing, durable state checkpoints, and human-in-the-loop validation gates.

---

Architectural Coherence Beats Glue Code

The individual capabilities of Swarm — gateway proxying, MCP tool invocation, DAG planning, automated evaluation — exist across different open-source projects.

What is rare is finding them integrated into a single, zero-dependency, memory-safe binary where adopting multi-agent orchestration doesn't invalidate the proxy architecture you set up on Day 1.

The architectural bet of Swarm is simple: the tools you choose when you only need a gateway should never become technical debt the day you need agents.

---

Discussion

If you're currently scaling your LLM infrastructure:

• At which rung on this ladder has your team spent the most engineering time?
• Have you had to replace a gateway when moving to agents (or vice-versa)?

Check out the project and try the kickstart scripts on GitHub: github.com/fcn06/swarm