DEV Community: Philip Stayetski

Bring Your Own Identity Provider for AI Agents: Binding an OIDC Identity to an Overlay Address

Philip Stayetski — Mon, 10 Aug 2026 20:39:58 +0000

Your agents have keys. Your company has an identity provider. Right now, those two worlds don't talk, and it's your job to connect them.

Here's the situation I kept running into: the platform team stands up a fleet of autonomous agents, each one minting its own cryptographic identity. Then the security team asks the inevitable question — how do we bring your own identity provider for AI agents, so an agent's identity means something to the rest of the org? They want the agent to authenticate against the same OIDC/OAuth IdP as every employee and service account. Same governance, same lifecycle, same audit trail.

This post is a practical how-to for that exact problem: linking an agent's keypair-based identity to an existing IdP so the agent is addressable both inside the overlay network and inside your corporate identity system.

Why an agent's identity starts as a keypair, not a login

Agents on a peer-to-peer network don't get usernames. They get addresses derived from keys. On Pilot Protocol, for example, every agent holds a permanent virtual address that survives restarts, IP changes, and moves across clouds — the identity is the key, not the host.

That's the right foundation: a machine identity that can sign and encrypt, that can't be spoofed by claiming a hostname. X25519 key exchange and signed handshakes beat password auth for unattended processes, because there's no human to type a password and no session to steal.

But a keypair answers "who is this agent?" only in the cryptographic sense. It doesn't answer "is this agent allowed to touch our production data?" That's a question your IdP already answers for every other principal in the company. The trick is making the two answers agree.

The two-identity problem

You now have two identity systems describing the same agent:

World	Identity	Issued by	Verified by
IdP (OIDC/OAuth)	`client_id` + claims	Your identity provider	Token signature (JWKS)
Overlay network	Public key / virtual address	The agent itself	Peers, at handshake

The overlap problem is real. An agent with address N:1234.ABCD.5678 might map to service account agents/checkout-worker in your IdP — or it might be a rogue instance claiming to be that worker. Nothing in the overlay world knows about the IdP world unless you make it.

The good news: you don't need to replace either system. You need a binding between them. Here are the three patterns I've actually used, in increasing order of ceremony.

Pattern 1: Present IdP-verified claims at handshake time

The overlay's trust model is an explicit handshake — two peers mutually approve before any traffic flows. That approval step is the natural enforcement point for IdP policy.

The flow:

The agent authenticates to your IdP (OIDC client-credentials flow, or a device flow if a human bootstraps it) and receives an ID token.
During the overlay handshake, the agent presents that token alongside its public key.
The receiving peer (or a policy agent acting on its behalf) validates the token signature against your IdP's JWKS endpoint, checks the claims — aud, expiry, group membership — and only then approves the handshake.

No new infrastructure. Your IdP stays the source of truth for "is this principal valid", and the overlay stays the source of truth for "can this address reach me". The binding is the signed token.

Pattern 2: A broker agent vouches for its fleet

If your agents are short-lived or spawn dynamically, making every peer verify JWKS signatures is noisy. Instead, run one well-known "registration agent" that holds a long-lived trust relationship with both worlds.

Agents authenticate to the IdP once, through the broker.
The broker verifies the token, then handshakes the new agent and attests to the fleet: "I vouch for this address; it holds IdP claims for agents/checkout-worker."
Other peers trust the broker's attestation instead of doing their own token dance.

This is the same delegation pattern you already use for employee onboarding — a trusted authority performs the verification, everyone else trusts the result. It also gives you one place to revoke: kill the broker's attestation and the agent loses its standing in the fleet.

Pattern 3: Private networks with join rules

For the coarsest control, don't let strangers onto the shared network at all. Pilot Protocol supports private networks with group-level connectivity and join rules — you define who may join, and agents outside the allowlist simply can't reach the group.

This doesn't replace IdP binding, but it layers on top of it: the IdP decides who the agent is, the join rules decide whether it's in your network. Combined with pattern 1 or 2, you get identity verification at two independent layers, which is exactly what a zero-trust posture wants.

What the handshake actually looks like

Concretely, with Pilot's CLI, the enforcement point is the trust commands. On the receiving side:

pilotctl pending                # incoming handshake requests
pilotctl approve <node_id>      # approve after your policy check

And on the initiating side:

pilotctl handshake <peer> "agent authenticating as agents/checkout-worker (IdP token attached)"

The reason this pattern works is that Pilot's trust model is explicit — membership and trust are decoupled. Joining the network doesn't make you trusted; a peer has to approve you. That's the hook your IdP policy hangs on: approval can be gated on token verification instead of being automatic.

Where the two worlds meet, cleanly

What I like about this setup is that neither system contorts itself:

The agent keeps its cryptographic identity — a permanent address it can be reached at, regardless of NAT or cloud moves. That's the overlay's job.
The IdP keeps its job — issuing verifiable claims about principals, with expiry and revocation.
The binding is a signed token exchanged at an explicit trust boundary.

If you want the full picture of the addressing and trust model — what an address is, how handshakes work, how private networks and join rules are configured — the trust model documentation covers it in detail, including the exact commands for handshakes, approvals, and network join rules.

Bring your own identity provider for AI agents: the takeaway

Bringing your own identity provider for AI agents isn't about choosing between cryptographic identity and corporate identity. It's about binding them at the right enforcement point. Handshake-time token verification, a vouching broker, and network join rules are three ways to do it, and they compose.

The agents get addresses that survive infrastructure churn. The IdP gets a seat at the table for every agent that joins your fleet. And the security team gets an answer to their question that doesn't start with "well, actually...".

If you're setting this up yourself, the install is one command:

curl -fsSL https://pilotprotocol.network/install.sh | sh

Then handshake, verify, approve — and your agents are both addressable and auditable.

Consensus Between Three or More AI Agents Without a Leader: How Peer Agreement Works

Philip Stayetski — Sun, 09 Aug 2026 13:35:16 +0000

When you have three or more AI agents that need to reach consensus without a leader — deciding task ordering, settling a value, or converging on shared state — the default move is to appoint a coordinator. One agent owns the decision, everyone else asks it. That works until the coordinator disappears, falls behind, or becomes the bottleneck every message has to route through. Leaderless consensus is the alternative, and it is far more practical than the theory makes it sound. This post is an explainer of what "without a leader" really demands, which existing tools fit, and where most peer consensus attempts actually fail.

Why "No Leader" Is a Different Problem

Leader-based consensus (Raft, Paxos) is well understood. The catch: election itself is a consensus problem, and it assumes a topology that stays put. Autonomous agents violate that assumption constantly — they restart, move across clouds, change IPs, and drop off the network mid-conversation. If your consensus protocol depends on a stable leader, you have just re-imported the failure mode you were trying to avoid.

Leaderless agreement quietly assumes three substrate properties:

Stable identity. A participant you can name today is the same participant tomorrow, even after a restart.
Reachability. Messages actually arrive, including for agents behind NAT or firewalls.
Verified trust. You know who is sending you a vote, and they know you.

Most write-ups on agent consensus skip straight to the algorithm. In practice, the substrate is where these systems fall apart.

The Existing Toolkit: Quorums, CRDTs, BFT

You do not need to invent an algorithm. The pieces exist; the job is picking the right one for the kind of agreement you need.

Quorum voting — the default shape

Each agent broadcasts a proposal, peers respond with signed acks or counter-proposals, and the proposal applies once a threshold of the group has agreed. No agent is special; the quorum is a property of the group, not of any single member. This is the right shape for task ordering and value settlement between agents that already trust each other.

CRDTs — agreement by merging

For state convergence, you can skip consensus entirely. Convergent replicated data types let every agent apply updates locally and merge with peers; as long as everyone eventually exchanges deltas, the state converges without any vote. This is the cheapest form of "agreement" and the right default when the question is what is the current state, not whose proposal wins.

BFT — only when you expect adversaries

Byzantine fault-tolerant protocols (PBFT, HotStuff) handle malicious participants, and they cost accordingly — extra rounds, extra messages, and a minimum group size that grows with the number of tolerated faults. Worth it for open networks with untrusted actors. Overkill for a closed set of collaborating agents.

The honest summary: for the common cases — ordering work, settling values, converging state among agents you know — you need a quorum loop, a merge function, and the substrate properties above. You rarely need Byzantine tolerance, and you never need a coordinator.

The Layer That Kills Peer Consensus Before It Starts

Here is the part that rarely shows up in the algorithm papers. To run any of the above, every agent needs an address that survives restarts, a path to peers behind NAT, and a way to verify who it is agreeing with. If those three things are bolted on as an afterthought, your consensus loop will spend most of its time failing to deliver messages, not failing to agree.

This is the gap Pilot Protocol fills, and it is an honest fit rather than a stretch: it is a networking layer built specifically so agents can talk to each other as peers. Each agent gets a permanent virtual address that survives restarts and IP changes. Transport is encrypted UDP tunnels (X25519 key exchange, AES-GCM) with STUN, hole punching, and relay fallback, so agents behind NAT are reachable. Trust is an explicit per-peer handshake where both sides approve — membership and trust are decoupled, which is exactly the pairwise-trust property a quorum needs. The network carries 243k+ agents, and it is open source under AGPL-3.0 with zero external dependencies.

The commands are the substrate in practice:

pilotctl handshake <peer-address> "collaborating on shared task state"
pilotctl peers          # who you can actually reach, and how
pilotctl send-message <peer> --data '<message>'

The handshake detail matters for consensus specifically: joined does not imply trusted. Each agent approves who it will accept messages — and votes — from, so your quorum is made of relationships the participants chose, not the network's default. And because addresses are permanent, the agent that voted in round one is the same agent in round ten. The Pilot Protocol docs cover the addressing and trust model in detail.

A Minimal Leaderless Agreement Loop

Putting it together, a leaderless agreement loop for three or more agents looks like this:

Propose — an agent broadcasts {proposal, seq, signature} to every peer.
Collect — each peer replies with a signed ack or a counter-proposal.
Apply — the proposer applies once it holds acks from a simple majority of the group.
Converge — for shared state, merge CRDT deltas instead of voting on a single winner.

A sketch of the loop:

QUORUM = 2  # simple majority of a 3-agent group

def propose(msg, peers):
    acks = {me: msg}
    for p in peers:
        if p.ack(msg):          # signed, delivered over the peer tunnel
            acks[p] = msg
    if len(acks) >= QUORUM:
        return apply(msg)       # durable, ordered, attributed
    return backoff_and_retry(msg)

The transport here is whatever your peers speak; the loop is the point. What the transport must give you is addressing that survives restarts and verified sender identity. That is the substrate layer — and it is the part you should not have to build yourself.

Consensus Between Three or More AI Agents Without a Leader: What to Reach For First

If you are designing a multi-agent system that needs leaderless agreement, the order of operations is:

Give every agent a stable identity and address that survives restarts and NAT.
Establish trust pairwise and explicitly — no ambient "everyone on the network is trusted."
Then pick the mechanism: quorum for ordering and settlement, CRDTs for state convergence, BFT only when you genuinely expect malicious actors.

If the loop is stuck, check the substrate before the algorithm. Nine times out of ten the failure is a message that never arrived or a sender that could not be verified — not a flaw in the agreement logic.

Where to Start

Leaderless consensus between agents is mostly a substrate problem wearing an algorithm costume. Once addressing, reachability, and trust are real, the agreement layer reduces to a quorum loop and a merge function. If you want to see the substrate part working before you build on it, install Pilot, handshake a couple of peers, and look at who you can actually reach:

curl -fsSL https://pilotprotocol.network/install.sh | sh

Then run a quorum loop over direct peer messages and watch how much of the hard part was already handled for you. The docs at pilotprotocol.network/docs are the reference for the addressing and trust model behind it.

What an Agent Should Do When It Loses Connectivity to All Peers: A Graceful Degradation Playbook

Philip Stayetski — Sun, 09 Aug 2026 09:07:41 +0000

Your agent depends on a network. Then one day every peer is unreachable at once. What should an agent do when it loses connectivity to all peers — keep retrying in a tight loop until it burns its rate limits, or degrade gracefully and come back when the network returns?

I've watched autonomous agents handle total network loss in production, and most of them handle it badly. Not because the engineers who wrote them were careless, but because we design for partial failure — one peer down, one timeout, one retry — and total failure is a different beast. This post is my opinion on what the right behavior actually looks like, shaped by what I've seen work and what I've seen fail.

Losing all peers is a different failure from losing one peer

A single peer disappearing is a retry problem. You have a timeout, you have a backoff schedule, you have other peers to talk to while this one is away. The system keeps functioning; one leg of the mesh is just offline.

Losing all peers at once is a different failure class. It usually means one of three things:

your host's network path is down (the local uplink, a firewall change, a suspended VM),
the network itself is partitioned, or
the discovery/rendezvous layer you depend on is unreachable.

None of those get better by hammering the same failed request fifty times in ten seconds. Yet that's exactly what a lot of agent loops do, because the default loop is "try, fail, retry" with no branch for total failure.

What an agent should do when it loses connectivity to all peers

My short answer: detect the shape of the outage, back off with exponential delay, operate locally, and reconnect with a defined recovery path. Four phases, in order:

Detect — determine whether you lost the network or everyone did. Can you still reach the rendezvous/registry? Can you reach the internet at all? That single check tells you whether to keep trying or to stop trying.
Back off exponentially — the first reconnect attempt is immediate, then the delay doubles each failure, with a cap and jitter so a fleet of agents doesn't reconnect in lockstep.
Operate locally — the agent still has work it can do that doesn't require peers: local computation, queueing intents, preparing state. Total disconnect is not total idleness.
Reconnect deliberately — probe at the backoff cadence, and when connectivity returns, resync state and drain the queue.

The important mental shift: reconnection is not "try harder." It's "try less often, more deliberately, while doing useful local work in between."

The backoff loop, concretely

Exponential backoff with jitter is boring and correct:

import random, time

def reconnect_loop(is_connected, attempt_connect, max_delay=60):
    delay = 1.0
    while True:
        if is_connected():
            delay = 1.0
            time.sleep(10)          # steady-state health check
            continue
        try:
            attempt_connect()
        except ConnectionError:
            pass
        time.sleep(delay * (0.5 + random.random()))  # jitter
        delay = min(delay * 2, max_delay)

The cap matters. Without it, an agent that lost its uplink for three hours spends three hours sleeping for increasingly absurd stretches. With it, you get a steady, polite heartbeat — one probe every minute or so — and an agent that's ready to act the moment the network returns.

Operate locally while disconnected

The phase most agent designs skip entirely is local operation. An agent that only reacts to network events is dead weight during an outage. An agent that also does local work keeps earning its keep:

Queue intents. If the agent's job is to relay or transform messages, have it write outbound intents to a durable local queue instead of dropping them. When the network returns, drain the queue.
Do local-only work. Computation, data preparation, rendering, checks against local state — anything that doesn't need a peer.
Log the outage state. A clear "disconnected since X, reason Y, N queued items" is worth more than a wall of connection errors when you come back to debug.

The rule of thumb I use: if a task doesn't strictly require a peer, it should not block on the network.

Reconnect to the same address, not a new one

Here's where the network design underneath the agent starts to matter. If your agent's identity is tied to a machine, an IP, or a cloud region, then an outage that moves the machine means the agent comes back as a different endpoint — and every peer has to re-learn who it is. That turns a connectivity blip into a trust and discovery problem.

A design I've come to respect handles this by giving the agent a permanent virtual address that survives restarts, IP changes, and moving across clouds. On Pilot Protocol, agents get exactly that: a stable address, encrypted UDP tunnels, and NAT traversal with relay fallback, so "reachable" doesn't depend on which network your agent happens to be behind. The recovery story becomes well-defined — when connectivity returns, the agent reconnects to the same address, peers re-establish the same trust relationship, and the mesh resumes where it left off. No re-discovery dance, no new identity to propagate. (Their documentation walks through the addressing and trust model in detail.)

I'm not saying an overlay network makes outages impossible — networks fail regardless of how clever your addressing is. But it does make the recovery deterministic, which is the part that's actually in your control.

The opinion part: resilience is a behavior, not a network property

Here's my real takeaway, and it's an opinion: we spend a lot of effort making networks resilient and very little effort making agents resilient. The network either comes back or it doesn't — you can't control that. But the agent's behavior during the outage — whether it thrashes or waits, whether it drops work or queues it, whether it comes back clean or half-broken — that's entirely yours to design.

So the next time you deploy an agent that depends on peers, ask yourself what it does when every peer vanishes. If the answer is "retries in a tight loop," you haven't finished writing it. Add the exponential backoff, add the local work, add the deliberate reconnect. The outage will happen. Your agent's behavior during it is a choice.

If you're building agents that talk to each other, the one-command way to give them stable addresses and encrypted tunnels is: curl -fsSL https://pilotprotocol.network/install.sh | sh — then pilotctl gets you connected to peers in minutes.

Automate Agent Onboarding in a Multi-Agent Fleet: Address, Trust, Discovery — Done Once

Philip Stayetski — Sat, 08 Aug 2026 20:33:47 +0000

Onboarding a new agent into a multi-agent fleet is the least-automated part of most agent deployments. You provision the container, install the runtime, and then the manual part starts: give the agent an identity, tell it which peers to trust, point it at the capabilities it's allowed to call. Two agents and it's a chore. A fleet and it's a project. The way out isn't a thicker runbook — it's recognizing that onboarding is a short, fixed sequence. Every new agent needs three things: an address, trust with its peers, and discovery of what it can call. Automate that sequence once per deployment and every agent after the first is free.

What onboarding actually is: address, trust, discovery

Most onboarding checklists are really three different problems stapled together. Separate them, and each one has a clean answer — and each answer is automatable.

1. An address that outlives the machine

An agent in a fleet has to be reachable. Not just today, on this IP, in this cloud — but after restarts, IP changes, and moves between clouds. That's the property DNS doesn't give you: a DNS record is bound to a host or endpoint that changes underneath you. And if your agents sit behind NAT, there's no stable endpoint at all.

What a fleet needs is a permanent virtual address that belongs to the agent, not to the machine it's running on. The address survives restarts, IP changes, and cloud migrations; the transport underneath — encrypted UDP tunnels, with NAT traversal when the network demands it — is the runtime's problem, not yours.

2. Trust, established explicitly — not implied by membership

The second thing a new agent needs is a set of peers it can actually talk to. This is where a lot of fleets quietly get the model wrong. In a typical overlay VPN, joining the network means being trusted: membership and trust are the same thing. For agent fleets, that's the wrong conflation — you often want agents from different teams or organizations on the same network without them trusting each other wholesale.

The model that fits fleets is explicit, mutual trust: a handshake between two agents that both sides approve, per peer. "Can reach" and "is trusted" stay separate questions, and each new agent gets exactly the trust it needs — no more.

3. Discovery: what can I call, and who else is here?

The third piece is discovery, in both directions. A new agent needs to find the peers and capabilities it's supposed to use, and the fleet needs to know the agent exists. A rendezvous registry plus a nameserver that resolves names and tags to addresses covers the peer side; capabilities get their own layer, which we'll get to.

Automating agent onboarding in a multi-agent fleet

Once the sequence is defined, automating it is a provisioning problem, not a runtime problem. Because the address is permanent, onboarding happens once per deployment — not on every restart. The agent's identity lives in a local keypair file, and everything after first start is idempotent: run the same steps again and nothing breaks.

Here's the whole sequence as a script, with Pilot Protocol as the runtime — a Go daemon with zero external dependencies that gives each agent a permanent virtual address, encrypted UDP tunnels (X25519 key exchange + AES-GCM), and NAT traversal:

# 1. Address — first start gives the agent a permanent virtual address
curl -fsSL https://pilotprotocol.network/install.sh | sh
pilotctl daemon start

# 2. Trust — explicit and mutual, per peer
pilotctl handshake <peer-address> "onboarding fleet member"
pilotctl trust          # confirm the link is mutual before relying on it

# 3. Discovery — find peers and capabilities by name or tag
pilotctl send-message list-agents --data '/data {"search":"weather"}'
pilotctl appstore catalogue

Three commands cover the whole checklist. The handshake is the only interactive part, and even that is scriptable: your fleet's control plane approves incoming handshakes the way it approves any other request — pilotctl pending to see them, pilotctl approve <id> to accept. Write this once into your deployment pipeline, and "add an agent" stops being a human task.

Capability onboarding: discover → install → call

Address and trust get an agent onto the network. The third step — capabilities — is where the fleet's actual work happens, and it deserves its own loop: discover → install → call.

An agent that joins a fleet needs to know what it can use. Pilot's app store makes capability onboarding look like package management:

pilotctl appstore catalogue                    # what's available
pilotctl appstore install <id>                 # install a capability app
pilotctl appstore call <id> <app>.help         # what does it do, what params
pilotctl appstore call <id> <app>.<method> '<json>'   # use it

The apps run locally on the agent's daemon as typed IPC services — JSON in, JSON out — and the daemon re-checks each app's pinned signature on every spawn. Permissions are granted at install time, scoped per app, so a new agent picks up capabilities without ambient authority. Install is an explicit, auditable event, the same shape for every agent in the fleet — and each app you publish is discoverable by the 243k+ agents already on the network.

One fair comparison: MCP did the ecosystem a real service by standardizing how agents invoke tools. What an app store adds on top is packaging and trust — signature-verified adapters, scoped grants, a single install command. The two are complementary: the store is about how capabilities get installed and vetted, not how they're called.

Start from a pre-wired fleet and skip the sequence entirely

If your fleet has a shape you keep re-deploying — a content pipeline, a code-review loop, a monitoring stack — you don't have to write the onboarding sequence from scratch at all. Pilot ships pre-wired multi-agent fleets: orgs where the agents, skills, and trust links are already wired, ready to deploy. The address-trust-discovery sequence is already done; you add your agents to a fleet that already knows how to behave.

Onboarding is a script, not a ceremony

The difference between a fleet you can grow and a fleet you're stuck with is whether adding an agent is a script or a ceremony. Address, trust, discovery — three steps, done once per deployment. Automate them and the marginal cost of a new agent approaches zero; leave them manual and every new agent is a new chance for drift.

Try the sequence once with two agents on one host, then script it:

curl -fsSL https://pilotprotocol.network/install.sh | sh

After that, onboarding is the boring part of your fleet — which is exactly what it should be.
.

Encrypted Tunnel vs TLS for Agent Communication: Same Crypto, Different Trust Model

Philip Stayetski — Fri, 07 Aug 2026 16:34:12 +0000

Encrypted tunnels and TLS both give you encryption. If you're picking between them for agent communication, the cipher suite is the wrong place to look — the real difference is in the trust model and the connection model. TLS secures a stream and anchors identity in a certificate authority. An encrypted tunnel secures a channel and anchors identity in a mutual handshake between the two endpoints. Same crypto family, different answers to "who is this peer?" — and for AI agents, that answer drives everything from reconnects to zero-trust posture.

I've spent the last few months wiring up agent-to-agent traffic, and this distinction keeps coming up in the wrong framing: people ask "which has stronger encryption?" and miss that both are strong. The question that actually matters is whether you want a stream or a channel, and whether you want a CA or the peer itself to vouch for identity.

What TLS gives you: a secured stream

TLS is a stream protocol. It runs on top of a reliable, ordered transport — almost always TCP — and it protects a byte stream between two sockets. The handshake negotiates a session key, and certificates bind public keys to identities. In the common case, the server presents a certificate signed by a certificate authority, and the client validates that signature against its trust store.

It is excellent at what it was built for: browsers talking to web servers, APIs over the public internet, anything where a large population of clients needs to trust a small set of servers without prior coordination. The CA ecosystem does that job at planetary scale, and mTLS extends the model so both sides present certificates.

The trade-offs show up when your endpoints are agents:

Connection-scoped. A TLS session lives and dies with the connection. The agent restarts, the IP changes, the cloud migrates — the session is gone and the handshake has to happen again, certificates and all.
Reachability assumed. TCP needs a listener your peer can actually reach. Agents behind NAT or firewalls aren't reachable by default, so you end up adding port forwarding, a relay, or a public-facing endpoint.
CA-anchored identity. Even with mTLS, trust is anchored in a certificate authority — a root you may not control, plus certificate lifecycle to manage (issue, renew, revoke).

None of these are failures. They're properties of a design aimed at the public web. But agent-to-agent traffic has different properties: peers are long-lived, mobile, frequently behind NAT, and mutually untrusted until they've explicitly agreed to talk.

What an encrypted tunnel gives you: a secured channel

A tunnel is a different shape. Instead of securing a stream, it creates a persistent, encrypted channel between two endpoints, and whatever you send through it is protected. The channel has an identity of its own, independent of any single connection, so the endpoints can reconnect, restart, and move without re-establishing trust from scratch.

Pilot Protocol is a concrete example of this model. It's an open-source overlay network for AI agents: every agent gets a permanent virtual address that survives restarts, IP changes, and moving across clouds. Traffic moves over encrypted UDP tunnels using X25519 key exchange and AES-GCM, with reliability handled in userspace rather than by TCP. NAT traversal is built in — STUN, hole-punching, and a relay fallback — so agents behind NAT are reachable without opening ports.

The trust model is where it diverges most sharply from TLS. Instead of a certificate authority, identity is established by an explicit per-peer handshake: both sides mutually approve before any traffic flows. Membership and trust are decoupled — being on the network doesn't mean being trusted, which is a different posture from a VPN where "joined" effectively means "trusted." Discovery is handled by a rendezvous registry and nameserver, so agents find each other by name or tag rather than by hardcoded IP.

Channel vs stream: why the shape matters for agents

The stream/channel distinction isn't academic — it changes what breaks and what doesn't.

With TLS, a connection is the unit of trust and state. If your agent's connection drops, everything riding on it — the session keys, the negotiated parameters, the application state that assumed a live socket — needs to be rebuilt. Retry logic, backoff, and reconnection are your problem to own.

With a channel, the unit of trust is the endpoint pair. The encrypted tunnel persists as a concept even when individual packets or connections come and go. The agent reconnects to the same virtual address; the mutual trust already established doesn't evaporate with a socket close. That's a meaningful difference for autonomous agents that run for weeks, restart after updates, or get migrated between clouds mid-conversation.

There's also a payload angle. TLS is protocol-specific in practice — you negotiate ALPN, you pick a protocol per connection. A channel is transport: once it exists, you can run HTTP, gRPC, MCP, or anything else through it. You establish trust once and multiplex whatever your agents actually speak.

CA trust vs mutual trust: who answers "who is this peer?"

This is the question the "vs" in encrypted tunnel vs TLS really lives in.

TLS answers it with a signature chain: a CA vouches for the server (and in mTLS, both sides). That's powerful when you want many clients to trust few servers they've never met — the browser scenario. It's heavier when every peer is both client and server and you'd rather not operate a private CA or pay for certificates per agent.

Mutual handshakes answer it directly: each peer decides, for itself, whether to trust the other. No third party is in the loop, and trust is revocable per peer. For zero-trust agent infrastructure — where you assume no ambient trust and verify explicitly — that maps cleanly: an agent only talks to peers it has deliberately approved, and it can drop that approval at any time.

Encrypted tunnel vs TLS: side by side

	TLS	Encrypted tunnel
Connection model	Stream, per-connection, over TCP	Persistent channel, independent of any connection
Trust anchor	Certificate authority / PKI	Mutual per-peer handshake
Peer identity	Certificates (CN/SAN), CA-validated	Explicit mutual approval between endpoints
After a restart / IP change	Session gone, re-handshake needed	Same virtual address, trust persists
NAT traversal	Needs reachable listener or relay	Built in (STUN, hole-punching, relay fallback)
What you can run through it	One negotiated protocol per connection	Anything: HTTP, gRPC, MCP, custom

Which should you use for agent communication?

Honestly: both, and they're not really competitors. TLS is the right tool when your agent talks to public web services, browsers, or anything where the CA model already fits — it's a solved problem and you shouldn't re-invent it. An encrypted tunnel is the right tool for the connections between your own agents: long-lived, mutually untrusted until handshaken, behind NAT, moving between environments. Plenty of setups use both — a tunnel for reachability, TLS inside it when a specific protocol wants it.

If you want to see the channel model in practice, Pilot Protocol is a working implementation: Go, zero external dependencies, AGPL-3.0, with the trust handshake, NAT traversal, and permanent addressing described above. The Pilot Protocol docs walk through the model, and 243k+ agents are already on the network.

The takeaway: stop comparing cipher strength and start comparing trust models. TLS asks "can this certificate be traced to a root we trust?" A tunnel asks "have these two peers explicitly agreed to talk?" For agent communication, that second question is usually the one you actually need answered.

Reference: try the tunnel model yourself — curl -fsSL https://pilotprotocol.network/install.sh | sh

Timeout and Retry Between Two Autonomous Agents: When No Broker Owns the Retry

Philip Stayetski — Fri, 07 Aug 2026 11:36:02 +0000

Two autonomous agents are talking. Agent A sends a request to Agent B and waits. Nothing comes back. Is B slow, or is B gone? How long does A hold the connection, and what does it do next? Timeout and retry between two autonomous agents is one of the most quietly important reliability problems in agent systems — because unlike a classic API call, there is no broker, no queue, and no load balancer that owns the retry. Somebody has to own it, and in a peer-to-peer setup that somebody is you.

This post is the practical version: right-sized timeouts, exponential backoff with jitter, idempotency, and a retry budget — the logic, with code, that lets two agents survive each other's disappearances.

Why Timeout and Retry Between Two Autonomous Agents Is Different

In a traditional client–server architecture, retries are infrastructure's problem. A message queue redelivers unacked messages. A load balancer fails over to a healthy replica. A serverless platform retries the invocation for you. The broker owns the retry because the broker owns the request lifecycle.

Two autonomous agents talking directly have none of that. The failure domain is bigger and fuzzier:

The peer may be gone. Crashed, redeployed, moved to another cloud.
The peer may be alive but busy. Agent B could be mid-task on something that takes minutes. From A's side, a slow response and a dead peer look identical.
Nobody is watching the wire. If the reply is lost in transit, nothing redelivers it; the request simply evaporates.
Both sides may be retrying at once. A retries its request while B retries its own — the same message arrives twice, or the two agents hammer each other in a feedback loop.

Once you stop assuming a broker, every retry decision becomes an application decision. The good news: the toolbox is small and well understood.

Right-Sized Timeouts: Slow Is Not Dead

The first mistake is picking a timeout out of thin air. Too short turns every slow-but-healthy peer into a "failure"; too long makes the whole system hang on a dead peer. The right size comes from measurement: log real round-trips between your agents, find the slow tail, and set the per-attempt timeout a comfortable multiple above it. And separate the two things you're actually measuring:

Round-trip time — how long the wire takes. This is what you probe with a ping or heartbeat.
Operation time — how long the peer's actual work takes. This varies per request type and is what your per-call timeout should be based on.

Keep them as different values. A liveness probe with a short timeout tells you the peer is reachable; the operation timeout tells you the work finished. In a P2P or overlay setting, ping the transport before you assume the peer died — reachability and responsiveness are different facts.

On top of per-attempt timeouts, set a hard overall deadline for the whole request. Per-attempt timeout says "this try gave up"; the deadline says "this request is over, stop spending." A minimal loop looks like this:

import asyncio

async def call_with_budget(coro, attempt_timeout: float, total_deadline: float):
    """Run coro with a per-attempt timeout and a hard overall deadline."""
    deadline = asyncio.get_running_loop().time() + total_deadline
    while True:
        remaining = deadline - asyncio.get_running_loop().time()
        if remaining <= 0:
            raise TimeoutError("overall deadline exceeded")
        try:
            return await asyncio.wait_for(coro(), timeout=min(attempt_timeout, remaining))
        except asyncio.TimeoutError:
            # handled by the retry loop below
            raise

The retry loop that wraps this is where backoff lives.

Exponential Backoff With Jitter

When a retry fails, you back off — but not at a constant rate, and not in lockstep with the other side. Exponential backoff with jitter is the standard answer for exactly this situation:

import random

def backoff_delay(attempt: int, base: float = 0.2, cap: float = 8.0) -> float:
    """Full jitter: random delay in [0, base * 2**attempt], capped."""
    return random.uniform(0, min(cap, base * (2 ** attempt)))

Two details matter more than they look:

The exponent. Each failed attempt multiplies the window, so the delay grows 0.2s → 0.4s → 0.8s → 1.6s… up to the cap. A peer that's restarting gets time to come back; a peer that's gone stops being hammered.
The jitter. Without randomness, every agent retries on the same schedule — a thundering herd of synchronized retries, and when both agents retry at the same cadence they keep colliding forever. Jitter breaks the symmetry. It's the one part people skip, and the part that prevents retry storms.

Cap the backoff and stop at a bounded number of attempts. Which brings us to the safety condition that makes retries legal.

Idempotency Is What Makes Retries Safe

Here's the uncomfortable fact: a retry is a duplicate. If Agent B actually processed the request and only the reply was lost, the retry makes B do the work again. For reads that's fine. For anything with side effects — a payment, a job dispatch, a state change — it's a bug.

The fix is the same one distributed systems have used for decades: at-least-once delivery plus deduplication on the receiver. Give every request a unique ID; the receiver remembers which IDs it has already processed and answers duplicates without re-running the work:

# Receiver side: dedup by request id
processed: dict[str, object] = {}

async def handle(request: dict) -> object:
    rid = request["request_id"]
    if rid in processed:          # already ran this one — reply from cache
        return processed[rid]
    result = await do_side_effectful_work(request)
    processed[rid] = result
    return result

The sender generates the ID once and keeps it across all retry attempts; the receiver treats "same ID" as "same logical request." Retry becomes a transport concern instead of a correctness one.

Retry Budgets and Circuit Breakers

Left to itself, retry logic retries forever — which means the "reliability" mechanism becomes the outage. Give every call a budget: a maximum number of attempts and a maximum total time. The deadline in the first section is the total-time half; the attempt cap is the other half:

async def retry_loop(coro, attempt_timeout, total_deadline, max_attempts):
    delay = 0.2
    for attempt in range(max_attempts):
        try:
            return await call_with_budget(coro, attempt_timeout, total_deadline)
        except (TimeoutError, ConnectionError):
            if attempt == max_attempts - 1:
                raise
            await asyncio.sleep(backoff_delay(attempt))
    raise TimeoutError("max attempts exceeded")

Beyond the per-call budget, add a circuit breaker at the peer level: track consecutive failures to Agent B, and once they cross a threshold, stop sending for a cooldown window instead of burning each call's budget on a peer that is clearly down. In a brokered world the queue provides this backpressure. In a brokerless world, your circuit breaker is the backpressure — the only thing standing between "B is having a bad minute" and "A spends its afternoon retrying B."

In a P2P World, the Other Side Retries Too

Everything above assumed you control one side. In agent-to-agent communication, both sides run the same playbook — so both sides can be retrying the same logical exchange from opposite directions. That's how you get duplicate work on both ends and message storms that neither side's backoff fixes on its own.

The mitigations are protocol-level agreements, not library choices:

Request IDs end to end, generated by whoever originates, honored by whoever receives — on both sides.
At-least-once semantics on both sides, with dedup as the default assumption rather than the exception.
Shared timeout conventions, so A's "slow" isn't B's "normal."

This is also where agent-native networking infrastructure earns its keep. If you're building agent-to-agent links over raw sockets or HTTP, you own the whole transport stack on top of the retry logic: addressing, reachability, the distinction between "peer gone" and "peer moved." An overlay network for agents — like Pilot Protocol's docs describe — gives every agent a permanent virtual address that survives restarts and IP changes, so a retry to the same address still reaches the same agent, plus encrypted tunnels and NAT traversal so the peer is reachable at all. The transport handles the wire; your timeout and retry logic handles the operations. You can absolutely build this yourself — but it's worth knowing where the line between "application logic" and "transport plumbing" sits before you own both.

The Checklist

The working version of this post is short:

Measure real round-trips; set per-attempt timeouts above the slow tail.
Separate liveness (ping the peer) from responsiveness (time the operation).
Back off exponentially, with jitter, capped, with a max attempt count.
Deduplicate by request ID on the receiver — retries are only safe if they're idempotent.
Budget and break — a total deadline plus a circuit breaker so retrying never becomes the outage.
Agree on semantics with the peer, because they're running this same playbook at you.

No broker is coming to save you. That's fine — the tools above are older than agents, and they still work.

Reference: Pilot Protocol docs — addressing, transport, and the trust model for agent networks. Install: curl -fsSL https://pilotprotocol.network/install.sh | sh.

Log Agent-to-Agent Message History for Debugging: When Both Agents Insist They're Right

Philip Stayetski — Fri, 07 Aug 2026 06:44:37 +0000

Two agents, one conversation, and suddenly both of them insist they never said what the other one claims. Sound familiar? When you run multi-agent systems long enough, you will eventually need to log agent-to-agent message history for debugging — not because your agents lie, but because "what was actually sent" stops being obvious the moment a retry, a timeout, or a silent drop enters the picture.

This post is a practical how-to: what to capture when two agents disagree, why the obvious "just add logging" answer keeps failing, and how to pick between shipping messages to external storage or using what the transport already keeps.

Why "just add logging" keeps failing

The naive approach is to log at the application layer, inside the agent code. Each agent writes what it thinks it sent. That's exactly the problem: the log ends up being a record of intentions, not a record of the wire.

Three failure modes show up over and over:

The he-said/she-said. Agent A logs "sent", Agent B logs "never received". Both logs are truthful from their own perspective. The retry logic, the timeout, or the queue that ate the message lives somewhere neither agent can see.
The wrong clock. Timestamps generated at the application layer drift across hosts. When you line up two logs side by side, the ordering doesn't match anything.
The missing correlation. Two agents talking over a mix of transports — webhooks, a message bus, a direct socket — produce four different log formats with no shared identifier. You can't join them.

The fix is to log at a layer that sees the actual exchange, with a consistent shape, and to decide up front where the record lives.

Log agent-to-agent message history for debugging: the five fields that matter

Whatever mechanism you pick, every agent-message log entry needs these five fields to be useful later:

Timestamp — ideally captured by the logging layer, not the application, so clocks are consistent.
From / to — stable agent identifiers, not hostnames or IPs that change.
Payload — the message body (or a hash of it, if the payload is large or sensitive).
Delivery outcome — acked, delivered, failed, or unknown. "Unknown" is a legitimate value and often the most interesting one.
Correlation ID — one identifier threaded through retries so a single logical message can be traced across multiple attempts.

With those five fields, "what was said" becomes a query instead of an argument.

Option A: external storage — the kitchen-sink approach

The most common pattern is to push every agent message into your own store: a Redis stream, a JSONL file, a Postgres table. A small sidecar subscribes to your agent traffic and appends one line per message.

# sidecar: append every agent message to an append-only log
def on_message(msg):
    entry = {
        "ts": msg.received_at.isoformat(),
        "from": msg.sender,
        "to": msg.recipient,
        "payload": msg.data,
        "outcome": msg.ack,
        "correlation_id": msg.correlation_id,
    }
    with open("agent-messages.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

External storage wins on control: you set retention, you can search, you can replay conversations into a fresh agent. It costs you infrastructure, a second failure mode, and the discipline to keep the sidecar running. If your agents already talk over a message bus, this is often the natural move — the bus is the log.

Option B: use the transport's own record

Before you build the sidecar, check what the transport already keeps. Some messaging systems persist messages as part of the protocol — and then "debugging what was said" is just reading the mailbox.

This is where Pilot Protocol comes in. It's an open-source overlay network for AI agents, and its documentation is explicit about which of its four communication models are stored:

Stream (connect / send) — synchronous request-response, not stored.
Data exchange (send-message / send-file) — async, stored on arrival in ~/.pilot/inbox/ and ~/.pilot/received/.
Pub/Sub — real-time fan-out to active subscribers, not stored.
Datagram — fire-and-forget, not stored.

The data-exchange model is the one designed for "delivery matters more than realtime response," and it gives you the message log for free. Every message lands on the recipient as a JSON file:

{
  "type": "JSON",
  "from": "0:0000.0000.0005",
  "data": "{\"task\":\"analyze\"}",
  "bytes": 18,
  "received_at": "2026-01-15T10:30:00Z"
}

That file is the he-said/she-said record: sender, payload, size, arrival time, captured by the transport on the receiving side. When an agent claims a message never arrived, you read the inbox instead of arguing:

pilotctl send-message other-agent --data '{"task":"analyze","input":"data.csv"}' --type json
# ...later, on the recipient:
pilotctl inbox

The same model covers files: send-file stores what arrived in ~/.pilot/received/.

There's a diagnostics side too. When the disagreement is about connectivity rather than content — one agent swears the tunnel was up — the daemon reports connection state, bytes sent and received, retransmissions, and congestion stats per connection:

pilotctl connections   # per-connection transport stats
pilotctl info          # peers, encryption status, traffic

So the built-in path gives you two complementary records: what was delivered (the inbox) and what the tunnel was doing (the diagnostics). Neither requires extra infrastructure, because the transport was designed to keep them.

Option C: hybrid — transport record plus external archive

Honestly, most production systems end up with both. The transport's inbox is your ground truth for "did it arrive", and it's the fastest thing to check when an agent complains. External storage gives you long retention, full-text search, and replay across many agents and transports.

A cheap hybrid: let the transport keep the live record, and have a scheduled job append the inbox to your archive. You get the zero-infra debugging experience day-to-day and the queryable history when you need to reconstruct a whole conversation weeks later.

Which one should you pick?

Pick external storage when your agents already talk over a bus, when you need full-text search or long retention, or when messages cross many different transports.
Pick transport-native history when you want the record to exist without building anything, when "did it arrive" is the question you debug most, and when you trust the transport's capture point more than your own logging code.
Pick both when the disagreement is expensive enough that you want receipts from two independent sources.

The underlying rule is the same in every case: log at the layer that actually moves the message, capture the five fields, and make sure the record survives the argument. Your future self, debugging at 2am, will thank you.

Pilot Protocol is an open-source overlay network that gives AI agents a permanent virtual address, encrypted tunnels, and built-in trust — with messaging models that decide up front whether a message is stored. The messaging and diagnostics docs cover the full model.

Get started: curl -fsSL https://pilotprotocol.network/install.sh | sh

Agent to Human Notification Without Email or SMS: A Recipe That Skips SMTP and SMS Gateways

Philip Stayetski — Thu, 06 Aug 2026 23:13:04 +0000

Your agent just spent two hours on a batch job. At 3 a.m. it hits something it can't resolve: an anomaly, a failed retry, a decision that needs a human. It has to alert you — and it has neither an email account nor a phone.

Agent to human notification without email or sms sounds like a constraint you'd fight. In practice it's the better design. The alert channel becomes a capability your agent calls, not a credential your agent carries. This is a use-case recipe for exactly that: the options that actually exist, and a concrete discover → install → call flow you can run this afternoon.

Agent to Human Notification Without Email or SMS: Four Channels That Work

You don't need SMTP and you don't need an SMS gateway. You need a delivery path from the agent to a screen a human actually watches. Four shapes cover the realistic options:

Channel	What it takes	When it fits
Message to the operator's own node	Operator runs a daemon; one `send-message` call	Zero third parties; delivery that survives disconnects
Chat-platform webhook	Bot token or incoming webhook, outbound egress	Your team already lives in Slack, Discord, or Telegram
Notification API service	One account and API key per service	One-off pushes, retries handled for you
Typed IPC call to an app	One `install`, then one `call`	Agents that shouldn't own any channel credentials

The first row is the interesting one, so let's start there, then build the recipe around the last.

The Recipe: Make Notification a Typed IPC Call

The pattern that keeps showing up in agent work: don't give the agent credentials, give it a capability. Pilot Protocol's agent app store is built around that idea. Apps are installable capability services that run locally on your daemon as typed IPC endpoints — JSON in, JSON out — and they auto-spawn the moment you install them. The loop is discover → install → call:

pilotctl appstore catalogue        # find an app
pilotctl appstore view <id>        # read what it does
pilotctl appstore install <id>     # one command, auto-spawned
pilotctl appstore call <id> <app>.<method> '<json>'   # do the work

Every app is discoverable by 243k+ agents on the overlay, and installing is a single command. The security story matters here because your agent is unsupervised: installs are signature-verified (the manifest pins a sha256 and an ed25519 signature, re-checked on every spawn), permissions are grant-scoped and accepted at install time rather than ambient, and the daemon supervises the app's lifecycle. Runtime discovery is a convention: every app answers <app>.help with its methods and parameters. No browser, no REST plumbing.

Get started with the one-command install:

curl -fsSL https://pilotprotocol.network/install.sh | sh

A Concrete Run: Agent Alerts Operator

Say the agent is an overnight risk scan. It flags something it can't classify and needs a human before it does anything else. Two honest paths, depending on what "alert" means here.

Path A — the operator runs a node. This is the purest version of "without email or sms": no third-party service in the path at all. The operator's node has a permanent virtual address, reachable even behind NAT via encrypted UDP tunnels. The agent sends a typed async message:

pilotctl send-message operator-node --data '{"severity":"high","task":"nightly-risk-scan","detail":"spike in failed logins"}' --type json

Data-exchange messages persist to the operator's ~/.pilot/inbox/ and survive disconnections — the docs recommend them exactly when delivery matters more than real-time response. The operator reads them with pilotctl inbox. If you want fan-out to a whole team, the same protocol has a pub/sub model: agents publish events, subscribers receive them in real time.

Path B — SMS or email is genuinely the right channel, but the agent still doesn't own it. The app store's communications apps hold the channel for you. AgentPhone gives the agent a real phone number — voice calls, SMS and iMessage, conversations over REST. Primitive provisions a managed email inbox in one call — send, receive, reply, search real mail over one REST API. In both cases the agent's side is identical: a typed IPC call with a JSON payload. The phone number or inbox lives in the app, granted at install, re-verified on every spawn.

Why the Channel Shouldn't Live in the Agent

Email means SMTP credentials, an account to provision, and outbound port decisions. SMS means a phone number and a provider account. Both are state your agent carries, rotates, and can leak — and both are exactly the kind of ambient authority that makes security reviews of autonomous agents painful.

An app-store call inverts the dependency. The notification capability is installed once, scoped to what it needs, and invoked with JSON. If the app is updated or replaced, the agent's code doesn't change — the call contract stays the same. That's the same reason the publish side exists: bring an existing API, describe its methods in the guided submission, and Pilot builds, signs, and reviews the adapter. Once approved, install works everywhere on the overlay. Turn an API into an agent app, and every agent on the network gains that capability — notification included.

Which Channel When

Operator runs a node → send-message to the inbox. No accounts, no egress, delivery that survives disconnects.
Team lives in chat → webhook to the platform they already watch. You manage the token; it works well.
SMS or email is the actual requirement → install the app that owns the channel (AgentPhone for a number, Primitive for an inbox) and call it over IPC.
Multiple watchers, live updates → pub/sub events.

None of these need the agent to hold an email account or a phone. The channel is a capability, the call is JSON, and the credentials stay where they're scoped and verified.

If your agent has been silently failing at 3 a.m. because alerting meant provisioning mail infrastructure or buying SMS credits, this is the unblock:

curl -fsSL https://pilotprotocol.network/install.sh | sh

Then browse the catalogue with pilotctl appstore catalogue, install the app that fits your channel, and make the alert a one-line call.

Run an AI Agent That Calls Other Agents as Tools — What “Call” Actually Means on the Wire

Philip Stayetski — Thu, 06 Aug 2026 18:26:37 +0000

You want to run an AI agent that calls other agents as tools. An orchestrator that hands subtasks to a research agent, a coding agent, a human-approval agent. On paper, this is just tool calling — the same mechanism as a calculator or a web-search tool. In practice, "calling an agent" is two problems stacked together, and most tutorials only solve one of them.

The first problem is framework-level tool calling: the interface the model uses to discover and invoke a tool. The second is transport-level agent reachability: whether one agent's process can actually open a connection to another agent's process. They are different layers, and confusing them is why multi-agent systems that demo beautifully on a laptop fall apart the moment the agents live on different machines.

Tool calling is an interface, not a connection

Start with what the framework layer actually does. Function calling in an LLM API gives the model a set of typed functions with JSON schemas; the model returns a structured call, and your runtime executes it. MCP standardizes that further — tools become resources a client can discover and invoke, with the server running over stdio or HTTP.

Both are genuinely useful. MCP in particular is a real step forward: one client talks to many servers, and the ecosystem is growing fast. But notice what both of them assume: that the tool's process is reachable. A stdio MCP server lives in your process tree. An HTTP MCP server lives behind a URL that has to resolve, route, and accept your connection. Neither layer has any mechanism for making an agent on a laptop behind NAT reachable from an agent in a cloud VPC. That is not their job.

The part everyone skips: the other agent has to be reachable

Here's the scenario that breaks the demo. Your orchestrator runs on a VM in the cloud. The agent you want to call as a tool — the one holding the private data, or the one that can trigger an action only it can perform — runs on a laptop at home, or inside a customer's private network. The framework layer is ready: the orchestrator knows the tool exists, knows its schema, and can emit a valid call. Then the call goes out and dies. No route. NAT. A firewall that only allows outbound. An IP that changed this morning.

This is the transport problem, and it's the half of "calling an agent" that tool-calling frameworks explicitly don't solve. The usual workarounds each cost you something: expose a public endpoint (inbound port, static IP, DNS to babysit), poll a shared queue (latency and state), or run everything on one host (fine for a demo, wrong for production).

What transport-level agent reachability actually means

For agent A to call agent B as a tool, A's process needs a stable, working path to B's process. That's a networking problem, and it has a networking answer: an overlay network built for agents. Same family as a mesh VPN, with the agent use case in mind:

A permanent virtual address per agent, surviving IP changes and moves between clouds.
Encrypted UDP tunnels — X25519 key exchange, AES-GCM — so agent traffic is not plaintext on the public internet.
NAT traversal via STUN, hole punching, and relay fallback, so agents behind NAT are reachable without opening inbound ports.
Explicit trust: per-peer handshake, mutually approved, so "connected" never silently means "trusted."

The point of this layer: reachability stops being an assumption. When an agent is on the network, it has an address other agents can reach, wherever it runs.

Run an AI agent that calls other agents as tools: the two-layer stack

So the working shape of a multi-agent system has two layers, and you want both.

Layer one — the framework: the model discovers tools and emits structured calls. Function calling, or MCP with a client and servers.

Layer two — the transport: the processes can reach each other. Stable addresses, encrypted tunnels, a trust relationship.

Here's what that looks like in practice. Keep MCP as your framework layer — it's good at that. Put an agent-native network underneath it. Pilot Protocol, for example, ships an MCP server (pilot-mcp) that bridges the two: your MCP client discovers other agents as tools, and each tool call travels over an encrypted tunnel to a peer behind NAT. The model sees an ordinary tool; the wire sees a reachable peer.

The transport layer also gives you the plumbing that makes calls land:

# establish mutual trust with the agent you want to call
pilotctl handshake <peer> "orchestrator calling you as a tool"
pilotctl trust          # confirm mutual trust

# then call it — stable address, encrypted tunnel
pilotctl send-message <peer> --data '{"task": "summarize the incident", "return": "tool_result"}'

Because the tool layer and transport layer are separate, the tools themselves can be delivered the same way. Pilot's app store ships capability apps as typed IPC services — JSON in, JSON out, auto-spawned on install, discoverable by the 243k+ agents on the network. The loop is pilotctl appstore catalogue → install → call:

pilotctl appstore catalogue
pilotctl appstore install <id>
pilotctl appstore call <id> <method> '{"query": "..."}'

The model's tool call becomes a JSON message to a local, signature-verified app. Framework on top, transport underneath — both explicit instead of assumed.

Why you need both

Skip the framework layer and you have reachable agents nothing can invoke: the processes can talk, but no model is driving them in a loop. Skip the transport layer and you have a beautiful tool-calling setup where every call to another agent dies with a connection error at deploy time. The first is a design gap. The second is the one that costs you a weekend.

The honest takeaway: MCP and function calling solved a real problem — the interface. Don't bolt a custom tool-calling protocol onto an overlay; use the standard framework layer. And don't treat reachability as an implementation detail of tool calling; it's a network property. Solve the interface with the frameworks the ecosystem standardized, and solve reachability with an agent-native network underneath — permanent addresses, NAT traversal, and explicit trust.

Run an AI agent that calls other agents as tools, and it will fail on the wire before it fails on the schema. Build both layers, and "calling an agent" becomes what it always looked like in the demo: one line, and it just works.

Get started:

curl -fsSL https://pilotprotocol.network/install.sh | sh

Then read the Pilot Protocol docs for the MCP bridge and app store commands. Source: github.com/pilot-protocol.

Send Large Files Between AI Agents Without a Cloud Storage Bucket: Direct Transfers Over an Encrypted Tunnel

Philip Stayetski — Thu, 06 Aug 2026 09:36:10 +0000

Your agent finished training and the other agent needs the weights. Or a pipeline stage produced a dataset and the next stage lives on a different cloud. The reflex is to stand up an S3 bucket, hand credentials to both sides, and pay the egress bill while you're at it.

But if you need to send large files between AI agents without a cloud storage bucket, there's a more direct path: give the agents a real network channel between them and transfer the file point-to-point. No bucket policy, no shared NFS mount, no third party sitting in the middle of your data.

This tutorial walks through what the direct approach actually requires, then shows a working agent-to-agent file transfer with Pilot Protocol — an open-source overlay network that gives every agent a permanent address and encrypted tunnels through NAT.

Why buckets became the default

Let's be fair to object storage first. S3 (and GCS, and Azure Blob) became the standard way to move artifacts between machines for a good reason: reachability. Two servers behind NAT can't open connections to each other, but both can reach a bucket. Upload from one side, download from the other. Problem solved — at the cost of:

Credentials on both sides (plus rotation, plus a bigger leakage surface)
A third party holding your data, even briefly
Egress and storage costs for data that was just passing through
Bucket policies, lifecycle rules, and versioning you never wanted to configure

A shared filesystem (NFS and friends) only works inside one trust domain — the same VPC, the same cluster. The moment your agents live on different clouds, or one sits behind a corporate firewall, the shared mount is off the table.

The underlying problem was never "we need storage." It was "these two machines can't reach each other."

The direct approach: agent-to-agent file transfer

If two agents have a persistent, encrypted, NAT-traversing channel, moving a file becomes a point-to-point operation. You don't need a bucket to transfer data between two endpoints that can already talk to each other.

That's the shape of an overlay network like Pilot Protocol. Each agent gets a permanent virtual address that survives restarts and IP changes. Traffic flows over encrypted UDP tunnels (X25519 key exchange + AES-GCM, with reliability handled in userspace), and STUN plus hole-punching with a relay fallback means agents behind NAT are reachable without any port forwarding. Trust is explicit: a handshake between the two peers, approved on both sides — joining the network doesn't make you trusted by anyone.

File transfer is built into the daemon, not bolted on. No object-store client library, no SDK ceremony for the common case. One command sends, one command lists what arrived.

Tutorial: moving an artifact between two agents

Here's the whole loop. Two hosts, worker-a and worker-b, on different networks.

1. Install on both hosts

curl -fsSL https://pilotprotocol.network/install.sh | sh

Same one-line install for the daemon and the pilotctl CLI, on both machines.

2. Start each daemon

pilotctl daemon start

Each agent registers and gets its permanent virtual address. From here on, that address is how the other side reaches it — no public IP, no port forwarding, no DNS gymnastics.

3. Establish trust

Private nodes require mutual trust before any data flows. On worker-a:

pilotctl handshake worker-b "shipping model weights"

On worker-b, approve the incoming request:

pilotctl approve <node-id>

That's the entire trust model — membership and trust are decoupled. Being on the network doesn't grant anyone access to your agent; you explicitly approve who can talk to you.

4. Send the file

On worker-a, where the artifact lives:

pilotctl send-file worker-b ./model-weights.bin

The command returns delivery metadata: filename, bytes, destination, sha256, verified, transport. Those last two fields are the integrity story — the receiving daemon verifies the transfer, so you're not shipping silent corruption risk across the wire.

5. Receive on the other side

On worker-b:

pilotctl received

Files land in ~/.pilot/received/ on the target. Delivery is asynchronous — the daemon stores files on arrival, so the recipient doesn't need an interactive session running at the exact moment the file lands. When you're done with them, pilotctl received --clear wipes the directory.

What sending large files without a cloud storage bucket actually requires

For big artifacts, the default transfer is streamed in chunks rather than sent as a single frame, so artifact size isn't what gates the transfer. What does matter:

Both sides must be reachable on the overlay. That's what the NAT traversal is for — it's handled for you, relay fallback included.
Disk space on the receiving side. The file lands in ~/.pilot/received/; the daemon doesn't garbage-collect for you.
Mutual trust before you send. The handshake is a one-time setup and it persists.
If your agent isn't the CLI, there's an SDK. Go, Python (pilotprotocol on PyPI), Node, and Swift bindings cover the programmatic case.

The whole thing is open source (AGPL-3.0, Go, zero external dependencies), so the tunnel and the trust logic are inspectable rather than a black box.

When to keep the bucket anyway

Direct transfer is a delivery mechanism, not a storage service. Keep the object store when:

You need durable archival — the artifact must outlive both agents
You're sharing with parties outside the overlay — humans, other orgs, pre-signed URLs
You need CDN distribution — many readers, one source
You want an audit trail of who accessed what — bucket logging gives you that

For the common case — one agent handing a large artifact to another agent it trusts, across clouds, through NAT — the direct channel removes a whole class of credential management and takes the middleman out of the data path. The network already carries 243k+ agents and users, so the addressing and trust layer is a solved problem rather than something you assemble from parts.

Try it

Install the daemon on the two hosts that need to trade files:

curl -fsSL https://pilotprotocol.network/install.sh | sh

Then handshake, send-file, and check received. The full command reference for file transfer, the inbox, and the stream and pub/sub models for everything that isn't a file lives in the Pilot Protocol docs.

The next time an agent needs to hand a large artifact to another agent, ask whether you actually need a bucket — or just a direct connection.

AI Agent Credential Rotation Without Downtime: Keeping In-Flight Communication Alive

Philip Stayetski — Wed, 05 Aug 2026 18:11:19 +0000

Your agent has been talking to a peer for six hours. There are in-flight requests, a signing key, an API token, and a long-lived tunnel that peers discovered through NAT. Security says: rotate everything by Friday. You swap the secret — and every in-flight message dies, every reconnect fails, and the peer on the other end has a cached identity that no longer matches anything you present. AI agent credential rotation without downtime is the problem nobody hands you a runbook for, so here's the one I wish I'd had.

Why rotating credentials breaks a running agent

Web services rotate keys all the time. The difference is that a stateless HTTP request lives for milliseconds: by the time a key rotates, every in-flight request has already completed. Agents are stateful. They hold sessions open for hours or days, and the messages in flight at the moment of rotation were signed with the old key.

Three failure modes show up in practice:

In-flight requests get rejected. A message signed with the old key arrives after rotation. Strict verification drops it. The peer retries, the retry is signed with the new key, and now the two sides disagree about which key is valid.
Peers cache your identity. Your peer remembered your public key (or token) at handshake time. A hard rotation invalidates that cached identity and forces a full re-handshake — mid-conversation.
Reconnect storms. The moment the old credential stops working, every peer that was silently connected tries to re-establish at once. That's the worst time to be doing key exchange.

None of this is new — TLS and secrets infrastructure solved most of it decades ago. The trick is porting those playbooks to agents before you need them.

The classic playbooks: dual-key rotation and grace periods

The core idea behind zero-downtime rotation is simple: never have a moment where the old key is invalid and the new key isn't trusted yet. You keep both alive through an overlap window.

Versioned keys. Give every key a kid (key ID) plus an activation window. Sign with the newest active key; verify with any key whose window hasn't closed. This is exactly how JWKS rotation works and how TLS cross-signed certificates ease a CA transition.

class RotatingKeyring:
    """Old keys stay verifiable during the grace window; only signing moves forward."""

    def __init__(self, keys):  # [{"kid", "key", "active_from", "expires_at"}]
        self.keys = keys

    def sign(self, payload):
        current = max(self.keys, key=lambda k: k["active_from"])
        return current["kid"], sign(current["key"], payload)

    def verify(self, kid, sig, payload):
        entry = next(k for k in self.keys if k["kid"] == kid)
        if entry["expires_at"] < now():
            raise KeyExpired(f"{kid} outside grace window")
        return verify(entry["key"], sig, payload)

Grace period sizing. The overlap window must cover the worst case: the longest a message can be in flight, plus the longest a peer can reasonably take to notice the new key, plus reconnect backoff. A few multiples of your keepalive interval is a sane starting point.

Staggered rotation. Rotate one node at a time, watch error rates, then continue. Fleet-wide rotation turns a single mistake into an outage with a thousand witnesses.

Short-lived secrets. If a token expires every hour instead of every year, rotation stops being a fire drill and becomes a background event. Dynamic secrets from a secrets manager (Vault-style leases are the classic example) give you this for free: agents fetch, use, and refresh without any human in the loop.

Where agents make it worse

The standard playbooks get you most of the way. Agents add three complications:

In-flight messages are signed, not just authorized. A token check happens at request time; a signature check happens at verify time, which can be after the sender already rotated. Your verification path must tolerate the old key for a while.
The identity is the credential. If your agent's "key" is also its identity — the thing peers use to recognize it — rotating it is indistinguishable from a new node appearing. Peers will treat the post-rotation agent as a stranger.
The transport is stateful. Agents often sit behind NAT with a mapped endpoint. A daemon restart to "apply the new key" can drop that mapping, and the agent becomes unreachable until the next keepalive. Rotation that requires a restart is downtime, just delayed.

The through-line: decouple the identity from the credential. Who you are should be stable; what you present to authenticate should be replaceable.

Decouple identity from credential (the actual fix)

TLS did this with long-term certificates and ephemeral session keys: the certificate is the identity, the per-connection key exchange is disposable. Agent infrastructure needs the same split, and it's exactly the design choice Pilot Protocol makes.

Pilot Protocol is an open-source overlay network for agents — a permanent virtual address per agent, encrypted UDP tunnels, NAT traversal, and a per-peer trust model. Two properties make rotation boring there:

Addressing is decoupled from the key. Every agent gets a stable virtual address (like 0:0000.0000.0001) assigned at registration. It survives restarts, IP changes, and moving across clouds. Peers reach you by address, not by key — so rotating key material never changes how you're found.
Tunnel secrets are per-connection. Each tunnel derives its own shared secret via X25519 key exchange, with AES-256-GCM for the traffic. Long-term identity material (an Ed25519 keypair in ~/.pilot/identity.json) is used for trust handshake signing, not for the data path. Rotating long-term material doesn't tear down active tunnels, because the active tunnels aren't encrypted with it.

That split is what makes rotation without downtime structurally possible rather than a sequence of lucky timings. And the tooling treats it as a first-class operation: pilotctl rotate-key generates a fresh identity keypair, and there's a documented recovery flow (pilotctl recovery enroll / new-key / recover) so that even a lost key doesn't cost you the address — you rotate to a fresh key and reclaim the same address with recovery material.

Trust is handled the same way: explicit per-peer handshakes that persist across daemon restarts and can be revoked with untrust. Membership and trust are separate — rotating your key doesn't reset your trust graph, and revoking a peer doesn't require re-keying the network. The Pilot Protocol docs walk through the addressing, transport, and trust model in detail.

That's not magic — it's the same identity/credential separation that made TLS rotation boring, applied to the agent's whole networking layer instead of just one connection.

A rotation checklist for running agents

Whatever transport you use, the same checklist applies:

Version your keys. kid + activation window on everything you sign.
Rotate one node at a time. Watch error rates before continuing.
Size the grace window honestly. It must cover in-flight message lifetime + peer reconnect backoff.
Separate identity from credential. The thing peers use to find you should never be the thing you rotate.
Prefer session-key separation. If active sessions carry their own keys, long-term rotation is a background event.
Test on a staging peer. Rotate a throwaway node first, watch the handshake logs, then do production.

FAQ

What does "credential rotation without downtime" mean for agents?
Replacing an agent's API tokens, signing keys, or identity keypair while it keeps running and keeps talking to peers — no dropped in-flight messages, no forced reconnects, no window where the agent is unreachable.

How long should the overlap window be?
Long enough to cover the longest in-flight message plus the time a peer can take to notice the new key, plus reconnect backoff. A few multiples of your keepalive interval is a reasonable starting point.

Do I need to restart an agent to rotate a key?
Only if your architecture couples the credential to the process. If identity and credential are decoupled and tunnels carry per-connection session keys, rotation is a config-level operation.

What's the difference between rotating an API token and rotating an identity key?
A token authorizes requests and is cheap to rotate. An identity key is what peers use to recognize you; rotating it without an overlap window looks like a new node appearing. That's why the stable-address pattern matters.

Rotation is a normal maintenance event, not an outage in waiting. The agents that survive it are the ones whose identity doesn't change when their keys do.

Get started with Pilot Protocol:

curl -fsSL https://pilotprotocol.network/install.sh | sh

Then pilotctl rotate-key and pilotctl recovery enroll are a command away. Docs: pilotprotocol.network/docs

Test Agent-to-Agent Connectivity Locally Before Deploying: Two Daemons, One Host

Philip Stayetski — Wed, 05 Aug 2026 13:14:23 +0000

You wrote two agents. Unit tests pass, the tool-call mocks are beautiful, and the first integration test — both agents in one process — works too. Then you deploy, and agent A cannot reach agent B. Not a logic bug. A connectivity bug: the handshake times out, the address doesn't resolve, or the message lands on a channel the peer never approved.

The fix is boring and effective: test agent-to-agent connectivity locally before deploying. Run two daemon processes on one host, give each a real identity, and exercise the same failure modes you'll hit at distance — restarts, address changes, trust decisions, lossy links. If the pair survives that loop, deploying is a formality.

Here's the dev loop I use, and the exact commands.

What "agent-to-agent connectivity" actually means

Before you test anything, name the layers. Four things have to be true for agent A to talk to agent B:

Addressing — A can name B. A name that survives restarts and IP changes, not a socket that dies with the process.
Transport — A's bytes actually reach B. Through NAT, through firewalls, without a static IP on either side.
Trust — B accepts messages from A. Explicit, per-peer, mutual — not "everything on the VPN is trusted."
Discovery — A can find out that B exists at all.

Unit tests mock all four. Integration tests usually run both agents inside one process, which makes them impossible to break — localhost always answers. Neither tells you anything about production.

The two-process dev loop

The core move: stop testing agents and start testing the network between them. Two daemon processes on one host, each with its own identity and address, talking over the same transport they'll use in production.

Pilot Protocol is an open-source overlay network built for exactly this shape, and its daemon is the identity layer, so the harness is a two-minute setup. Start node one and node two with separate identity files:

# node one
pilotctl daemon start --identity ~/.pilot/identity-a.json --hostname agent-a

# node two
pilotctl daemon start --identity ~/.pilot/identity-b.json --hostname agent-b

Each daemon registers a permanent virtual address that survives restarts and IP changes. That's the property you're about to test.

Simulate what goes wrong at distance

Localhost hides everything. Two daemons with real identities don't. Walk the failure modes in order:

1. Restart one daemon. Kill node B, start it again. Does its address survive? If a daemon re-registers with a new address on every boot, peers can't find it, and your "distributed" system breaks on a deploy. With Pilot, the identity file persists, so the address should come back the same:

pilotctl info          # note the address
# restart the daemon...
pilotctl info          # same address? good.

2. Replace the identity. Start node B with a new identity file. The peer should no longer recognize it — trust is tied to identity, not to the IP it came from. If your test harness happily accepts a stranger, your trust model is fake.

3. Send before approval. Try to message node B from A before any handshake. It should fail. pilotctl pending should show the request waiting, and nothing should flow until both sides approve. Membership and trust are decoupled: "joined the network" must never mean "trusted by everyone."

pilotctl pending       # incoming requests waiting on you
pilotctl approve <node-id>

4. Round-trip both directions. Trust is mutual, so test it both ways:

pilotctl ping <peer-address>
pilotctl send-message <peer-address> --data 'hello from agent-a' --wait

5. Move one node behind a different network. A container with its own network namespace, a hotspot, a cloud VM — anything that puts real NAT in the path. This is the step localhost can't fake, and it's where most A2A setups die. Pilot handles it with STUN plus hole-punching and a relay fallback (the beacon), so the same commands keep working when the direct path doesn't.

A concrete test script

Here's the whole loop as a script you can run before every deploy:

#!/usr/bin/env bash
set -e

# 1. identity survives restart
ADDR_BEFORE=$(pilotctl info --json | jq -r .data.address)
pilotctl daemon stop && pilotctl daemon start
ADDR_AFTER=$(pilotctl info --json | jq -r .data.address)
[ "$ADDR_BEFORE" = "$ADDR_AFTER" ] || { echo "FAIL: address changed"; exit 1; }

# 2. unapproved peer cannot send
pilotctl pending | grep -q . && echo "note: pending requests exist"

# 3. message round-trips
pilotctl ping agent-b
pilotctl send-message agent-b --data 'ping' --wait

echo "PASS: agent-to-agent connectivity OK"

Run it, fix what breaks, run it again. That's the loop.

Test agent-to-agent connectivity locally before deploying — the checklist

If you take nothing else from this post, use this list:

[ ] The address survives a daemon restart.
[ ] A message round-trips over the tunnel in both directions.
[ ] An unapproved peer cannot deliver messages.
[ ] Approval flips delivery on, with no code changes.
[ ] A fresh identity is rejected by the peer.
[ ] Discovery works: A can find B by name, not by hardcoded IP.
[ ] The same commands pass with real NAT in the path.

Every item is a thing that breaks in production and costs an hour to debug. Every item is testable on one machine.

Why the daemon shape matters

The reason two daemons on one host is the right harness, rather than a mock, is that you're testing the actual artifacts you'll deploy: the process, its identity, its trust decisions, its transport. When the pair works locally — including a restart and a NAT hop — the remaining risk between you and production is configuration, not architecture.

Pilot Protocol is one honestly-fitting option for this loop: it's open source (AGPL-3.0, Go, no external dependencies), gives every agent a permanent virtual address, encrypts tunnels with X25519 key exchange and AES-GCM, and makes trust an explicit per-peer handshake. The Pilot Protocol docs cover the addressing, transport, and trust model in full. It also has an app store of agent-native tools you can install with a single command once your nodes are talking — but that's a follow-up post.

Get started and run the loop yourself:

curl -fsSL https://pilotprotocol.network/install.sh | sh

Two daemons, one host, a few minutes. Cheaper than the first production outage.