Every framework tutorial for building an AI agent ends the same way: a demo that works great on your laptop, talking to a hosted LLM API, with no story for what happens when you actually need to run it — reachable, monitored, and talking to other agents — in production. That's where self hosted AI agent infrastructure stops being optional and starts being the actual engineering problem.
"Self-hosted" here doesn't mean "reject every SaaS API." It means: you control the pieces that matter for reliability, cost, and data custody — compute, storage, secrets, and networking — instead of renting all of them from a vendor who can change pricing, rate limits, or terms under you. This post walks through the pieces of that stack, one at a time, and where each one commonly breaks.
1. Compute: where the agent loop actually runs
An agent isn't a single API call — it's a loop: plan, call a tool, observe, repeat. That loop needs somewhere to live that isn't a serverless function with a 10-second timeout.
Options that show up repeatedly in self-hosted setups:
- A long-running process on a VM or bare metal, supervised by systemd or a process manager.
- Containers behind a lightweight orchestrator (Docker Compose for single-node, k3s/Nomad for anything bigger).
- Disposable microVMs spun up per task and torn down after, useful when an agent needs to run untrusted or exploratory code without touching your main host.
The common failure mode: teams pick a serverless platform because it's the fastest way to ship a demo, then hit wall-clock limits the moment an agent needs to wait on a slow tool call or a human-in-the-loop step.
2. Memory and retrieval: vector + structured storage
Agents need state beyond the context window: conversation history, embeddings for retrieval, structured facts. Self-hosting this usually means running your own vector store (pgvector on Postgres is the least-surprising choice if you already run Postgres; Qdrant or Weaviate if you want a dedicated engine) alongside a regular relational or key-value store for anything that needs transactional guarantees.
The mistake to avoid: treating the vector store as your only database. Embeddings answer "what's similar," not "what's true right now" — you still need a system of record for anything an agent needs to act on reliably.
3. Orchestration and tool-calling
This is the layer frameworks like LangGraph, CrewAI, and the Model Context Protocol (MCP) live in — a well-defined way for an agent to call a tool, get a typed result back, and decide what to do next. MCP specifically standardizes "here's a tool, here's its schema" so any MCP-compatible agent can use it without custom glue code per tool. Self-hosting your orchestration layer means you own the tool registry and its permission boundaries, instead of trusting a third party's hosted tool marketplace with your credentials.
4. Secrets and permissions
An agent that can call tools is an agent that holds credentials. The self-hosted version of "least privilege" means scoping what each tool grant can touch, not handing an agent one long-lived API key with account-wide access. Anything with an install step should ask for scoped grants up front rather than ambient, everything-or-nothing access.
5. Networking: the piece nobody plans for until it breaks
This is the part that gets skipped in most write-ups, and it's the one that actually kills self-hosted agent deployments in practice. The moment you have more than one agent — or one agent behind a NAT'd home server, a laptop, or a container on a cloud provider that doesn't hand out public IPs — you need:
- A stable address for each agent that survives restarts and IP changes.
- Encrypted transport between agents, not plaintext HTTP over the open internet.
- NAT traversal, because most self-hosted compute (home labs, containers, cloud VMs without public IPs) sits behind one.
- An explicit trust model, so "on the network" and "trusted to talk to me" are two different things, not one.
Most people reach for a VPN (Tailscale, ZeroTier, Nebula) here, and that's a reasonable choice — they solve NAT traversal and encrypted transport well and are mature, widely-deployed tools. Where they differ from agent-native networking is the trust model: on a VPN, joining the network usually implies you're reachable by everyone else on it, which is designed for people connecting their own devices, not for arbitrary agents that should only talk to specific peers they've explicitly approved.
Pilot Protocol is an open-source overlay network built for this specific case: every agent gets a permanent virtual address, traffic runs over encrypted UDP tunnels (X25519 key exchange + AES-GCM) with NAT traversal built in (STUN, hole-punching, relay fallback), and — the part that matters for multi-agent setups — trust is a separate, explicit step from being on the network. Two agents can both be reachable overlay-wide and still refuse to talk to each other until they mutually approve a handshake. The daemon itself is Go with zero external dependencies (stdlib only), which matters if "self-hosted" is supposed to mean you can actually audit and run the thing yourself. It's AGPL-3.0 and the source is on GitHub.
Pilot also ships an app-store layer on top of the networking piece: installable capability apps (JSON in, JSON out) that run locally on your daemon, discoverable through a shared directory. If you're already self-hosting the networking layer, that's a way to add capabilities — search, data lookups, deploy targets — without standing up a new service and wiring auth for it yourself.
curl -fsSL https://pilotprotocol.network/install.sh | sh
pilotctl appstore catalogue
That's not a replacement for MCP — MCP defines how a single agent calls a tool; Pilot defines how agents (and their tools) find and reach each other across networks. The two compose: an MCP-speaking agent can sit on a Pilot address and be reachable across NAT the same way any other Pilot node is.
6. Observability
Once an agent is out of your terminal and running unattended, you need logs of every tool call and every decision it made, not just the final output. Self-hosted usually means your existing stack — structured logs into whatever you already run (Loki, ELK, plain files rotated to S3) — rather than a hosted agent-observability SaaS with its own retention limits and pricing tiers.
Putting the stack together
None of these six layers is optional at scale, but you don't need to build all of them from scratch. A reasonable self-hosted starting point looks like:
- A supervised long-running process or small container cluster for compute.
- Postgres + pgvector for memory, unless you already have real reasons to run a dedicated vector database.
- MCP (or an equivalent typed tool-calling layer) for orchestration.
- Scoped, per-tool credentials — not one master key.
- An overlay network with NAT traversal and explicit per-peer trust for anything that needs multiple agents or machines talking to each other.
- Your existing logging stack, pointed at the agent's tool-call trace.
FAQ
Is self-hosted AI agent infrastructure only for large teams?
No — a single self-hosted VM running a supervised agent process with Postgres and an overlay network for connectivity is a complete stack for a solo developer or small team. The complexity scales with the number of agents and machines involved, not with team size.
Do I need a VPN or an overlay network if I only run one agent?
Not immediately — a single agent talking to hosted APIs over normal HTTPS doesn't need it. It becomes necessary the moment that agent needs to be reached by another agent, run behind a NAT, or move between hosts without breaking its address.
Does self-hosting mean avoiding all hosted LLM APIs?
No. Self-hosting infrastructure and using a hosted model API aren't mutually exclusive — most self-hosted agent stacks still call out to a hosted LLM; what's self-hosted is the loop, the memory, the tool permissions, and the networking around that call.
How is an overlay network different from just using a VPN?
Both solve encrypted transport and NAT traversal. The difference that matters for agents is trust: a VPN typically treats "connected to the network" as "reachable by the network," while an agent-oriented overlay separates network membership from per-peer trust, since agents often shouldn't be reachable by every other node just because they're online.
Read more on the Pilot Protocol blog.
Top comments (0)