Log Agent-to-Agent Message History for Debugging: When Both Agents Insist They're Right

Two agents, one conversation, and suddenly both of them insist they never said what the other one claims. Sound familiar? When you run multi-agent systems long enough, you will eventually need to log agent-to-agent message history for debugging — not because your agents lie, but because "what was actually sent" stops being obvious the moment a retry, a timeout, or a silent drop enters the picture.

This post is a practical how-to: what to capture when two agents disagree, why the obvious "just add logging" answer keeps failing, and how to pick between shipping messages to external storage or using what the transport already keeps.

Why "just add logging" keeps failing

The naive approach is to log at the application layer, inside the agent code. Each agent writes what it thinks it sent. That's exactly the problem: the log ends up being a record of intentions, not a record of the wire.

Three failure modes show up over and over:

1. The he-said/she-said. Agent A logs "sent", Agent B logs "never received". Both logs are truthful from their own perspective. The retry logic, the timeout, or the queue that ate the message lives somewhere neither agent can see.
2. The wrong clock. Timestamps generated at the application layer drift across hosts. When you line up two logs side by side, the ordering doesn't match anything.
3. The missing correlation. Two agents talking over a mix of transports — webhooks, a message bus, a direct socket — produce four different log formats with no shared identifier. You can't join them.

The fix is to log at a layer that sees the actual exchange, with a consistent shape, and to decide up front where the record lives.

Log agent-to-agent message history for debugging: the five fields that matter

Whatever mechanism you pick, every agent-message log entry needs these five fields to be useful later:

• Timestamp — ideally captured by the logging layer, not the application, so clocks are consistent.
• From / to — stable agent identifiers, not hostnames or IPs that change.
• Payload — the message body (or a hash of it, if the payload is large or sensitive).
• Delivery outcome — acked, delivered, failed, or unknown. "Unknown" is a legitimate value and often the most interesting one.
• Correlation ID — one identifier threaded through retries so a single logical message can be traced across multiple attempts.

With those five fields, "what was said" becomes a query instead of an argument.

Option A: external storage — the kitchen-sink approach

The most common pattern is to push every agent message into your own store: a Redis stream, a JSONL file, a Postgres table. A small sidecar subscribes to your agent traffic and appends one line per message.

# sidecar: append every agent message to an append-only log
def on_message(msg):
    entry = {
        "ts": msg.received_at.isoformat(),
        "from": msg.sender,
        "to": msg.recipient,
        "payload": msg.data,
        "outcome": msg.ack,
        "correlation_id": msg.correlation_id,
    }
    with open("agent-messages.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

External storage wins on control: you set retention, you can search, you can replay conversations into a fresh agent. It costs you infrastructure, a second failure mode, and the discipline to keep the sidecar running. If your agents already talk over a message bus, this is often the natural move — the bus is the log.

Option B: use the transport's own record

Before you build the sidecar, check what the transport already keeps. Some messaging systems persist messages as part of the protocol — and then "debugging what was said" is just reading the mailbox.

This is where Pilot Protocol comes in. It's an open-source overlay network for AI agents, and its documentation is explicit about which of its four communication models are stored:

• Stream (connect / send) — synchronous request-response, not stored.
• Data exchange (send-message / send-file) — async, stored on arrival in ~/.pilot/inbox/ and ~/.pilot/received/.
• Pub/Sub — real-time fan-out to active subscribers, not stored.
• Datagram — fire-and-forget, not stored.

The data-exchange model is the one designed for "delivery matters more than realtime response," and it gives you the message log for free. Every message lands on the recipient as a JSON file:

{
  "type": "JSON",
  "from": "0:0000.0000.0005",
  "data": "{\"task\":\"analyze\"}",
  "bytes": 18,
  "received_at": "2026-01-15T10:30:00Z"
}

That file is the he-said/she-said record: sender, payload, size, arrival time, captured by the transport on the receiving side. When an agent claims a message never arrived, you read the inbox instead of arguing:

pilotctl send-message other-agent --data '{"task":"analyze","input":"data.csv"}' --type json
# ...later, on the recipient:
pilotctl inbox

The same model covers files: send-file stores what arrived in ~/.pilot/received/.

There's a diagnostics side too. When the disagreement is about connectivity rather than content — one agent swears the tunnel was up — the daemon reports connection state, bytes sent and received, retransmissions, and congestion stats per connection:

pilotctl connections   # per-connection transport stats
pilotctl info          # peers, encryption status, traffic

So the built-in path gives you two complementary records: what was delivered (the inbox) and what the tunnel was doing (the diagnostics). Neither requires extra infrastructure, because the transport was designed to keep them.

Option C: hybrid — transport record plus external archive

Honestly, most production systems end up with both. The transport's inbox is your ground truth for "did it arrive", and it's the fastest thing to check when an agent complains. External storage gives you long retention, full-text search, and replay across many agents and transports.

A cheap hybrid: let the transport keep the live record, and have a scheduled job append the inbox to your archive. You get the zero-infra debugging experience day-to-day and the queryable history when you need to reconstruct a whole conversation weeks later.

Which one should you pick?

• Pick external storage when your agents already talk over a bus, when you need full-text search or long retention, or when messages cross many different transports.
• Pick transport-native history when you want the record to exist without building anything, when "did it arrive" is the question you debug most, and when you trust the transport's capture point more than your own logging code.
• Pick both when the disagreement is expensive enough that you want receipts from two independent sources.

The underlying rule is the same in every case: log at the layer that actually moves the message, capture the five fields, and make sure the record survives the argument. Your future self, debugging at 2am, will thank you.

---

Pilot Protocol is an open-source overlay network that gives AI agents a permanent virtual address, encrypted tunnels, and built-in trust — with messaging models that decide up front whether a message is stored. The messaging and diagnostics docs cover the full model.

Get started: curl -fsSL https://pilotprotocol.network/install.sh | sh