DEV Community

Philip Stayetski
Philip Stayetski

Posted on

Two Agents Can't Reach Each Other? Here's the Debugging Checklist I Use

You write Agent A, deploy it on a cloud VM. You write Agent B, run it on your laptop. Agent A sends a message. Nothing comes back. No error, no timeout — just silence.

This has happened to me more times than I'd like to admit. When two agents cannot reach each other, the root cause is almost always something mundane: a missing handshake, a NAT that quietly drops UDP, or — embarrassingly often — a daemon that never started. But because the failure mode is always the same ("no reply"), you end up chasing ghosts in the wrong layer.

Here's the ordered checklist I walk through every time I need to debug why two agents can't reach each other. It saves hours.

Step 1: Is the daemon actually running?

This is the most embarrassing check on this list, which is exactly why it goes first. I've spent twenty minutes debugging a connectivity failure only to realize I'd never started the agent's daemon on the target machine.

ps aux | grep daemon
Enter fullscreen mode Exit fullscreen mode

If nothing shows up, start it. If it starts and immediately exits, check the logs — a port conflict or a missing config file is the usual suspect. On a fresh deploy, confirm the daemon survived the provisioning script.

Step 2: Is trust mutual?

Most agent-to-agent networking systems use some form of trust handshake. If Agent A trusts Agent B but Agent B never approved Agent A, traffic flows one way only — which looks exactly like "no reply."

This is the single most common cause of asymmetric connectivity failures in peer-to-peer setups. The receiver gets the handshake request, ignores it, and never sends back.

If you're building your own trust layer, you need three things: (a) an out-of-band exchange of identities, (b) a way for each side to confirm the other's approval, and (c) a timeout for stale pending requests. Get any of these wrong and messages vanish silently.

Step 3: Can they actually see each other through NAT?

This is the hard one. If both agents are behind NAT — which is almost always true for agents on different networks, say one on a cloud VM and one on home WiFi — standard UDP doesn't work without help.

The NAT problem breaks down into three sub-checks:

Can either agent receive unsolicited traffic? If an agent is behind a symmetric NAT or a carrier-grade NAT, no incoming UDP reaches it without a relay. No middlebox punching technique works on every NAT type. You need server-mediated hole-punching plus a relay fallback for the cases that won't cooperate.

Do you have a relay set up? Without a relay, two agents behind symmetric NATs will never connect. With one, they trade a few extra milliseconds of latency for guaranteed delivery.

Did the NAT mapping expire? UDP NAT bindings have a timeout — often 30 to 120 seconds for TCP-assisted NATs, sometimes as low as 15 seconds for pure UDP flows. If your agent sends one message and goes silent for a minute, the mapping drops, and the reply has nowhere to go. Keepalive pings every 10 to 30 seconds prevent this.

Step 4: Are they speaking the same wire format?

This one hurts because it's a design bug, not an ops bug. Agent A serializes with protobuf using a schema it generated yesterday. Agent B expects the JSON wire format from the spec published last month. Both run. Both call themselves "agents." Neither can parse what the other sent.

The fix is a shared contract. If both agents use the same SDK, you get this for free. If they don't — one is Python and the other is Go — you need an explicit wire format that both sides independently implement, and a test harness that confirms messages round-trip across languages.

Step 5: What does the wire say?

When checks 1 through 4 pass and messages still don't arrive, look at the packets.

tcpdump -i any port <agent-port> -X
Enter fullscreen mode Exit fullscreen mode

You're looking for:

  • Packets arriving at all — zero packets means the problem is on the network path, not in the agent
  • Packets arriving but getting a RST or ICMP unreachable — the port isn't listening
  • Corrupted payload — truncated frames, wrong encoding

This check ends most debugging sessions, because the packet trace doesn't lie. If you see ICMP Port Unreachable, the process wasn't listening. Go back to Step 1.


When the checklist itself is the problem

Having a checklist is good. Needing it every time is a sign that the problem shouldn't exist in the first place.

What I wanted was a networking layer for agents that makes Step 1 (daemon lifecycle) a framework guarantee, Step 2 (trust) a single command on both sides, Step 3 (NAT traversal) transparent with STUN, hole-punching, and automatic relay fallback, and Step 4 (wire format) an SDK contract that doesn't drift between versions.

That's what Pilot Protocol does — an open-source overlay network built for agents. Each agent gets a permanent virtual address that survives restarts, IP changes, and NAT. Connectivity is authenticated via mutual handshakes. Encrypted UDP tunnels establish automatically with STUN, hole-punching, and relay fallback when direct communication isn't possible. The SDKs for Go, Python, Node, and Swift guarantee the wire format stays identical across languages.

If you're tired of running this checklist for every pair of agents you deploy:

curl -fsSL https://pilotprotocol.network/install.sh | sh
pilotctl daemon start
pilotctl handshake <peer> "let's talk"
Enter fullscreen mode Exit fullscreen mode

Otherwise, print this list and tape it to your monitor. You'll need it.

Top comments (0)