Timeout and Retry Between Two Autonomous Agents: When No Broker Owns the Retry

#ai #agents #networking #python

Two autonomous agents are talking. Agent A sends a request to Agent B and waits. Nothing comes back. Is B slow, or is B gone? How long does A hold the connection, and what does it do next? Timeout and retry between two autonomous agents is one of the most quietly important reliability problems in agent systems — because unlike a classic API call, there is no broker, no queue, and no load balancer that owns the retry. Somebody has to own it, and in a peer-to-peer setup that somebody is you.

This post is the practical version: right-sized timeouts, exponential backoff with jitter, idempotency, and a retry budget — the logic, with code, that lets two agents survive each other's disappearances.

Why Timeout and Retry Between Two Autonomous Agents Is Different

In a traditional client–server architecture, retries are infrastructure's problem. A message queue redelivers unacked messages. A load balancer fails over to a healthy replica. A serverless platform retries the invocation for you. The broker owns the retry because the broker owns the request lifecycle.

Two autonomous agents talking directly have none of that. The failure domain is bigger and fuzzier:

The peer may be gone. Crashed, redeployed, moved to another cloud.
The peer may be alive but busy. Agent B could be mid-task on something that takes minutes. From A's side, a slow response and a dead peer look identical.
Nobody is watching the wire. If the reply is lost in transit, nothing redelivers it; the request simply evaporates.
Both sides may be retrying at once. A retries its request while B retries its own — the same message arrives twice, or the two agents hammer each other in a feedback loop.

Once you stop assuming a broker, every retry decision becomes an application decision. The good news: the toolbox is small and well understood.

Right-Sized Timeouts: Slow Is Not Dead

The first mistake is picking a timeout out of thin air. Too short turns every slow-but-healthy peer into a "failure"; too long makes the whole system hang on a dead peer. The right size comes from measurement: log real round-trips between your agents, find the slow tail, and set the per-attempt timeout a comfortable multiple above it. And separate the two things you're actually measuring:

Round-trip time — how long the wire takes. This is what you probe with a ping or heartbeat.
Operation time — how long the peer's actual work takes. This varies per request type and is what your per-call timeout should be based on.

Keep them as different values. A liveness probe with a short timeout tells you the peer is reachable; the operation timeout tells you the work finished. In a P2P or overlay setting, ping the transport before you assume the peer died — reachability and responsiveness are different facts.

On top of per-attempt timeouts, set a hard overall deadline for the whole request. Per-attempt timeout says "this try gave up"; the deadline says "this request is over, stop spending." A minimal loop looks like this:

import asyncio

async def call_with_budget(coro, attempt_timeout: float, total_deadline: float):
    """Run coro with a per-attempt timeout and a hard overall deadline."""
    deadline = asyncio.get_running_loop().time() + total_deadline
    while True:
        remaining = deadline - asyncio.get_running_loop().time()
        if remaining <= 0:
            raise TimeoutError("overall deadline exceeded")
        try:
            return await asyncio.wait_for(coro(), timeout=min(attempt_timeout, remaining))
        except asyncio.TimeoutError:
            # handled by the retry loop below
            raise

The retry loop that wraps this is where backoff lives.

Exponential Backoff With Jitter

When a retry fails, you back off — but not at a constant rate, and not in lockstep with the other side. Exponential backoff with jitter is the standard answer for exactly this situation:

import random

def backoff_delay(attempt: int, base: float = 0.2, cap: float = 8.0) -> float:
    """Full jitter: random delay in [0, base * 2**attempt], capped."""
    return random.uniform(0, min(cap, base * (2 ** attempt)))

Two details matter more than they look:

The exponent. Each failed attempt multiplies the window, so the delay grows 0.2s → 0.4s → 0.8s → 1.6s… up to the cap. A peer that's restarting gets time to come back; a peer that's gone stops being hammered.
The jitter. Without randomness, every agent retries on the same schedule — a thundering herd of synchronized retries, and when both agents retry at the same cadence they keep colliding forever. Jitter breaks the symmetry. It's the one part people skip, and the part that prevents retry storms.

Cap the backoff and stop at a bounded number of attempts. Which brings us to the safety condition that makes retries legal.

Idempotency Is What Makes Retries Safe

Here's the uncomfortable fact: a retry is a duplicate. If Agent B actually processed the request and only the reply was lost, the retry makes B do the work again. For reads that's fine. For anything with side effects — a payment, a job dispatch, a state change — it's a bug.

The fix is the same one distributed systems have used for decades: at-least-once delivery plus deduplication on the receiver. Give every request a unique ID; the receiver remembers which IDs it has already processed and answers duplicates without re-running the work:

# Receiver side: dedup by request id
processed: dict[str, object] = {}

async def handle(request: dict) -> object:
    rid = request["request_id"]
    if rid in processed:          # already ran this one — reply from cache
        return processed[rid]
    result = await do_side_effectful_work(request)
    processed[rid] = result
    return result

The sender generates the ID once and keeps it across all retry attempts; the receiver treats "same ID" as "same logical request." Retry becomes a transport concern instead of a correctness one.

Retry Budgets and Circuit Breakers

Left to itself, retry logic retries forever — which means the "reliability" mechanism becomes the outage. Give every call a budget: a maximum number of attempts and a maximum total time. The deadline in the first section is the total-time half; the attempt cap is the other half:

async def retry_loop(coro, attempt_timeout, total_deadline, max_attempts):
    delay = 0.2
    for attempt in range(max_attempts):
        try:
            return await call_with_budget(coro, attempt_timeout, total_deadline)
        except (TimeoutError, ConnectionError):
            if attempt == max_attempts - 1:
                raise
            await asyncio.sleep(backoff_delay(attempt))
    raise TimeoutError("max attempts exceeded")

Beyond the per-call budget, add a circuit breaker at the peer level: track consecutive failures to Agent B, and once they cross a threshold, stop sending for a cooldown window instead of burning each call's budget on a peer that is clearly down. In a brokered world the queue provides this backpressure. In a brokerless world, your circuit breaker is the backpressure — the only thing standing between "B is having a bad minute" and "A spends its afternoon retrying B."

In a P2P World, the Other Side Retries Too

Everything above assumed you control one side. In agent-to-agent communication, both sides run the same playbook — so both sides can be retrying the same logical exchange from opposite directions. That's how you get duplicate work on both ends and message storms that neither side's backoff fixes on its own.

The mitigations are protocol-level agreements, not library choices:

Request IDs end to end, generated by whoever originates, honored by whoever receives — on both sides.
At-least-once semantics on both sides, with dedup as the default assumption rather than the exception.
Shared timeout conventions, so A's "slow" isn't B's "normal."

This is also where agent-native networking infrastructure earns its keep. If you're building agent-to-agent links over raw sockets or HTTP, you own the whole transport stack on top of the retry logic: addressing, reachability, the distinction between "peer gone" and "peer moved." An overlay network for agents — like Pilot Protocol's docs describe — gives every agent a permanent virtual address that survives restarts and IP changes, so a retry to the same address still reaches the same agent, plus encrypted tunnels and NAT traversal so the peer is reachable at all. The transport handles the wire; your timeout and retry logic handles the operations. You can absolutely build this yourself — but it's worth knowing where the line between "application logic" and "transport plumbing" sits before you own both.

The Checklist

The working version of this post is short:

Measure real round-trips; set per-attempt timeouts above the slow tail.
Separate liveness (ping the peer) from responsiveness (time the operation).
Back off exponentially, with jitter, capped, with a max attempt count.
Deduplicate by request ID on the receiver — retries are only safe if they're idempotent.
Budget and break — a total deadline plus a circuit breaker so retrying never becomes the outage.
Agree on semantics with the peer, because they're running this same playbook at you.

No broker is coming to save you. That's fine — the tools above are older than agents, and they still work.

Reference: Pilot Protocol docs — addressing, transport, and the trust model for agent networks. Install: curl -fsSL https://pilotprotocol.network/install.sh | sh.