AI Agent Cannot Reach Peer on Another Cloud? Here's the Fix

You spin up an agent on AWS. It needs to call a tool-serving agent your teammate deployed on GCP. You get connection refused, or worse, a silent timeout. Both boxes are up. Both processes are listening. The agent still can't reach its peer on another cloud.

This is one of the most common failure modes in multi-agent systems once you move past a single-host demo, and it's worth understanding exactly why it happens before reaching for a fix.

Why "it works on localhost" stops working across clouds

When an AI agent needs to call a peer — another agent, a tool server, an MCP endpoint — that peer needs a stable address and an open path. Cross-cloud, you lose both by default:

1. Security groups and firewalls block inbound by default. AWS security groups, GCP firewall rules, and Azure NSGs all deny inbound traffic unless you explicitly allow it. That's correct behavior for security, but it means your agent on GCP has no way to accept a connection from your agent on AWS until someone opens a port — and opening ports to the public internet for agent-to-agent traffic is exactly the kind of exposure most teams don't want.

2. Ephemeral IPs and NAT hide the peer entirely. Most agents run behind NAT (a Kubernetes pod, a container on a laptop, a serverless function) with no public IP at all. Even if you did open a port, there's no stable address to open it on — the IP changes on every redeploy, every autoscale event, every restart.

3. Ingress rules assume human traffic, not agent traffic. Load balancers and ingress controllers are built around HTTP request/response from browsers or known API clients. Long-lived, bidirectional, agent-initiated connections don't map cleanly onto that model, which is part of why long-running agent tasks that rely on webhooks time out or drop mid-conversation.

Put together: two agents can be perfectly healthy and still have no valid network path between them the moment they're on different clouds, different VPCs, or different NAT boundaries.

The usual workarounds, and where they fall short

• Public IP + open port on the peer. Works, but now that agent is internet-facing. You're trusting your firewall rules and your process's own auth to hold the line, and every redeploy risks a new IP you have to re-propagate.
• VPN (site-to-site or mesh, e.g. Tailscale/Nebula/ZeroTier). These solve the reachability problem well — once a node has joined the mesh, it has a stable virtual IP and can reach other joined nodes. The tradeoff for agent-to-agent use is that in most VPN models, joining the network and being trusted are the same event: once a node is on the mesh, it's generally reachable by other mesh members by design. For agent swarms where you want fine-grained control over which agent can talk to which other agent — not just which agents are on the network — you often want that decision decoupled from membership.
• Central message broker (Redis, RabbitMQ, a hosted queue). Reliable, but now every agent needs to reach that broker's address instead, and you've added a hop and an operational dependency — you're just moving the reachability problem, not eliminating it.

Overlay addressing: give the agent an identity, not just an IP

The fix that generalizes across all three failure modes above is to stop routing to IP addresses and start routing to a permanent virtual address that belongs to the agent, not the host it happens to be running on. This is the same idea behind CGNAT-friendly overlay networks broadly, applied specifically to AI agents.

Pilot Protocol is an open-source overlay network built for exactly this: every agent gets a permanent virtual address that survives restarts, IP changes, and moving across clouds. Under the hood:

• Transport is encrypted UDP tunnels (X25519 key exchange + AES-GCM), with userspace reliability layered over UDP — no dependency on inbound firewall rules being open.
• NAT traversal uses STUN plus hole-punching, with relay fallback when a direct hole-punch isn't possible. An agent behind restrictive NAT is still reachable through the overlay.
• Discovery happens through a rendezvous registry and nameserver, so agents find each other by name or tag rather than by tracking IPs.
• Trust is explicit and decoupled from membership. Being on the network doesn't mean being reachable by everyone on it — each peer relationship requires a mutual handshake (both sides approve) before a tunnel opens. That's a deliberate design choice for exactly the multi-agent case above: you may want hundreds of agents discoverable on the same overlay without every one of them able to talk to every other one.

It's implemented in Go with zero external dependencies (stdlib only), open source under AGPL-3.0.

Getting two agents on different clouds talking

Install the daemon on both hosts — the AWS box and the GCP box, or wherever your agents live:

curl -fsSL https://pilotprotocol.network/install.sh | sh

Start the daemon on each:

pilotctl daemon start --hostname agent-a   # on the AWS host
pilotctl daemon start --hostname agent-b   # on the GCP host

From agent-a, initiate a handshake with agent-b — this is the one step that replaces "open a security group rule and hope the IP doesn't change":

pilotctl handshake agent-b "coordinating a task pipeline"

Agent-b approves it (or an auto-trust policy can do this automatically for known sources):

pilotctl approve <node_id_of_agent_a>

Once mutual trust is established, either side can message the other by hostname — no public IP, no open inbound port, no re-propagating addresses after a redeploy:

pilotctl send-message agent-b --data "status check" --wait

Check that the tunnel is actually up:

pilotctl ping agent-b
pilotctl peers   # shows transport state — direct or relayed

That's the whole loop: install, start, handshake, approve, send. The address for agent-b never changes even if GCP assigns it a new ephemeral IP tomorrow.

Beyond point-to-point messaging: capabilities as apps

Once agents can reach each other reliably, the next problem is usually "what can I actually call on the other end." Pilot also ships an app store — installable capabilities that run locally on a daemon as typed JSON-in/JSON-out services, auto-spawned on install:

pilotctl appstore catalogue
pilotctl appstore install io.pilot.cosift
pilotctl appstore call io.pilot.cosift cosift.search '{"q":"your query","k":"5"}'

The pattern is discover → install → call, and it composes with the peer addressing above: an agent on any cloud can reach a peer's installed capabilities the same way it reaches the peer itself.

FAQ

Is this the same problem as a webhook timing out? Related but distinct. Webhook timeouts happen because HTTP request/response wasn't designed for long-running agent tasks. Cross-cloud unreachability happens because there's no valid network path at all — the ingress/NAT/firewall layer never let the connection through in the first place. An overlay network with persistent tunnels addresses both: it replaces the fragile HTTP round-trip with a standing, encrypted connection.

Do I need to open any inbound ports? No — that's the point. NAT traversal (STUN + hole-punching, relay fallback) is designed so neither peer needs a public IP or an open inbound port.

What if the peer is on a completely different cloud provider, not just a different VPC? Doesn't matter — the overlay address is independent of the underlying network. That's the difference between routing to an IP (tied to a specific host, cloud, and subnet) and routing to a virtual address (tied to the agent's identity).

Does joining the network mean every agent can reach every other agent? No. Trust is a separate, explicit step from being on the network — each peer relationship requires mutual approval before traffic flows.