DEV Community: Stephen Wanjohi

A2A push notifications: what the spec guarantees, and what it leaves to you

Stephen Wanjohi — Tue, 04 Aug 2026 09:15:01 +0000

The Agent2Agent protocol solved a real problem. Long-running agent tasks do not fit inside a request and response, and holding an SSE stream open for forty minutes is not something you want to depend on. So A2A lets the client register a webhook and disconnect, and the remote agent posts back when something worth knowing about happens.

The specification is careful about the parts it covers. It is also explicit, by omission, about a part it does not cover, and that part is the one that decides whether your integration works at three in the morning.

What the spec actually gives you

A push notification is configured with a PushNotificationConfig. The fields are small and sensible:

url, the HTTPS endpoint that will receive the POST
token, an optional opaque value the receiver can check to confirm the notification belongs to a task it knows about
authentication, optional details describing how the sending agent will authenticate itself to your webhook You can supply it inline on the initial SendMessage or SendStreamingMessage call, or manage it separately for an existing task. The current specification exposes create, get, list, and delete operations for these configs, so a client can rotate an endpoint without restarting the task.

On the wire, the notification body uses the same StreamResponse shape as streaming, carrying exactly one of a task, a message, a statusUpdate, or an artifactUpdate. If you have already written a streaming consumer, you have most of a webhook consumer.

The spec also tells you when to expect one. The server decides, but the guidance is that notifications fire on significant state changes. A task is in one of eight states. Four are terminal: COMPLETED, FAILED, CANCELED, REJECTED. Four are interim: SUBMITTED, WORKING, INPUT_REQUIRED, AUTH_REQUIRED. In practice you will be notified on the terminal ones, and on INPUT_REQUIRED and AUTH_REQUIRED, because those are the two interim states where the task has stopped and is waiting on you.

On authentication it is genuinely helpful. Bearer tokens, API keys, HMAC signatures, and mTLS are all named, with a JWT and JWKS example for key distribution. The security guidance is good in both directions. The sending agent should treat a client-supplied URL as hostile and defend against SSRF with domain allowlisting, ownership verification, and egress controls. The receiving webhook should verify signatures against trusted keys, validate the token if one was set, and use timestamps and nonces or jti claims to reject replays.

That is a well-specified protocol. Read it and you know exactly what a notification looks like and how to prove it came from who it claims to.

The sentence that is not in the specification

Nowhere does A2A say what happens if your webhook is down.

There is no retry policy. No backoff guidance. No timeout threshold. No statement about how many attempts a sending agent should make, or whether it should make any. No definition of at-least-once or at-most-once. The spec says notifications are delivered by HTTP POST and leaves everything after that word to the implementation.

This is a defensible choice. Protocols that try to mandate delivery semantics tend to age badly. But it means something specific for anyone building on A2A: the reliability of your agent notifications is not a property of the protocol. It is a property of whichever implementation happens to be on the other end, and you probably have not read its source.

Two A2A-compliant agents can behave completely differently here. One retries ten times over six hours with exponential backoff. One tries once, catches the exception, logs it, and moves on. Both are conformant. Your integration passes its tests against either.

Why this bites harder for agents than for ordinary webhooks

A dropped order.paid webhook is bad. A dropped A2A terminal notification is worse, and the reason is structural.

Push notifications exist precisely for the long-running case. The client disconnected on purpose. That is the feature. So when the agent finishes a task that took forty minutes of real compute, there is often exactly one POST that says so, and nobody is watching the connection anymore.

If that POST fails and is not retried, three things are true at once. The work happened. The result exists on the remote agent. And your side of the system believes the task is still WORKING, forever, because the notification that would have moved it was the notification that got lost.

With streaming you would have noticed the disconnect. That is the trade you made when you chose push. It is a good trade, but it moves the burden of noticing onto the delivery layer.

There is a second-order version of this that is nastier. Because the sender chooses when to notify, an agent can legitimately send you the same state transition twice, and a retrying sender certainly will. If your webhook handler is not idempotent, a duplicate COMPLETED notification does not just log twice. It runs whatever your completion handler runs, twice.

What the receiving side owes

Your webhook is a public HTTPS endpoint that a third party posts to. Treat it that way.

Verify before you trust. Check the signature or bearer token against the key you expect, compare the token field against the value you registered, and do both with constant-time comparison so you are not leaking the credential a byte at a time through timing. Return 401 on a mismatch and do not explain which part failed.

Reject replays. A timestamp window plus a seen-nonce set is enough. The jti claim exists for this if you are on JWTs.

Be idempotent, and key it correctly. The natural key for an A2A notification is the task ID combined with the state being reported, not a random message ID, because the thing you want to happen only once is "this task became COMPLETED", regardless of how many POSTs carry that news.

Return quickly. Acknowledge with a 2xx as soon as the notification is durably written on your side, then do the actual work asynchronously. A handler that takes eleven seconds because it calls three internal services will eventually take longer than the sender's timeout, and then you are being retried for work you already completed.

What the sending side owes

If you are the one running the agent, the notification is your outbound delivery problem, and it looks exactly like every other outbound delivery problem.

Write it down before you send it. The state transition should be durable in your own database before the POST is attempted, because a process that crashes between "task completed" and "notification sent" otherwise loses the only record that a notification was owed.

Retry, but retry with judgement. A connection timeout or a 503 deserves another attempt on a backoff schedule with jitter, because the endpoint is probably coming back. A 422 or a 400 does not, because the payload will be equally wrong in ten minutes and all you are doing is delaying the moment somebody discovers the integration has been broken since it shipped. A 410 Gone is the receiver telling you to stop, and the right response is to disable the config rather than to keep trying.

Fail somewhere visible. When the retry budget is exhausted, the notification should land in a dead letter queue that a person actually reads, stored whole rather than as a log line, so that when the endpoint comes back somebody can replay it. Alert on the endpoint rather than on the individual notification, or you will teach everyone to ignore the alert.

Defend the URL. The client hands you an arbitrary HTTPS address and asks you to POST to it from inside your network. That is a textbook SSRF vector, which is why the spec calls it out. Allowlist, verify ownership, and keep egress controlled.

Build or buy, honestly

None of the above is exotic. A competent backend team can build a persisted queue, a retry worker with jitter, a dead letter table, and a replay endpoint. Teams do it every quarter.

The cost is not the build. It is that you have now taken ownership of a piece of infrastructure whose failure mode is silence, and it has to keep working correctly while you build the agent your users are actually paying for. Delivery infrastructure does not page you when it breaks. It just stops mentioning things.

If you are running one agent, notifying one endpoint your own team operates, write the retry loop. That is the correct amount of engineering for that problem. The line worth watching is not volume. It is who owns the other end. Once you are posting to endpoints operated by other people, on their uptime and their deploy schedule, the failure modes stop being yours and you find out about them from a customer.

The short version

A2A tells you what a push notification looks like and how to prove where it came from. It does not tell you what happens when the POST fails, and that gap is yours to fill on both sides of the connection.

Persist the state transition before you send it. Retry the retryable and stop fast on everything else. Make the receiver idempotent on task ID plus state, because duplicates are not a bug in the sender, they are the sender doing its job. Verify credentials in constant time and reject replays. And put the notifications that never made it somewhere a person will look, because a task that silently stays WORKING forever is the one failure mode nobody has a dashboard for.

Mittr is the reliable action layer for webhooks and AI agent actions, including A2A push notifications. It handles both auth presets from the spec, bearer token and OIDC or OAuth2 JWT, with constant-time credential checks and a 401 on mismatch. Every receipt is recorded in a request inspector, idempotency keys are derived from task plus state, and notifications are correlated by task ID. Outbound, every action is written to Postgres before delivery, retried on a front-loaded schedule with jitter behind a per-endpoint circuit breaker, and dead-lettered rather than dropped if it exhausts its budget. The free tier is 3,000 messages a month, no card. More at mittr.io.

Dead letter queues for webhooks: after the last retry

Stephen Wanjohi — Wed, 29 Jul 2026 13:18:26 +0000

Almost every webhook system has a retry policy. Far fewer have a good answer to the question that comes immediately after it: what happens when the retries run out?

That moment is where most delivery systems quietly lose data. Not in a dramatic outage, but in a catch block that logs an error nobody reads, on an endpoint that was down for eleven minutes.

Three endings, and most systems ship two

An outbound event has exactly three terminal states:

Delivered.
Still trying.
Gave up.

In-house delivery code usually models the first two with real care. There is a retry loop, exponential backoff, maybe some jitter. State three is where the design thins out. The loop exits, an error is logged, and the event stops existing in any form the team can act on.

A dead letter queue is what you build so that state three is a place rather than an ending.

A dead letter queue is not a queue of failures

It is a queue of decisions nobody has made yet.

That distinction sounds like semantics and it is not. Every entry in a well built dead letter queue is an event where the system has already exhausted everything it can do automatically, and a human now has to choose one of three things: replay it as is, fix the receiving endpoint and then replay it, or discard it deliberately.

Once you frame it that way, the design follows. A queue of decisions needs enough context for someone to decide, it needs to reach a person who is capable of deciding, and it needs an obvious action to take once they have.

Most dead letter implementations fail on at least one of those three.

Deciding when to stop

Before anything lands in a dead letter queue, you have to decide what "give up" means. Three rules cover most cases.

Separate retryable from non-retryable. A 500, a 502, a 503, a connection reset, a timeout: retry all of those, because the endpoint may well be fine in thirty seconds. A 400, 401, 403 or 422 will not fix itself on the next attempt. Something about the request is wrong, or the credentials are wrong, and hammering it twelve more times over six hours just adds noise. Fail those fast and dead letter them immediately, because the sooner a human sees a 401 the better.

A 429 sits in the middle. Retry it, but honour the Retry-After header rather than your own backoff curve.

Cap by attempts and by wall clock, not just attempts. Exponential backoff compounds quickly. If you retry twelve times with doubling delays, the last attempt lands days after the first. By then the event may be meaningless and the customer has long since noticed.

A front-loaded schedule handles this better than a pure doubling curve. Something like 1s, 5s, 30s, 2m, 10m, 30m, 1h, 6h, 24h catches the common case (an endpoint that blips for a few seconds) almost immediately, while still giving a genuinely down endpoint most of a day to come back. Add roughly ten percent random jitter to every delay, or a thousand retries queued during the same outage will stampede the endpoint the moment it recovers.

Use a circuit breaker so one dead endpoint does not fill your queue. If every delivery to a given endpoint has failed for the last ten minutes, stop trying and start parking. Otherwise a single customer whose server is down for an afternoon generates tens of thousands of doomed attempts, saturates your workers, and slows delivery for everyone else.

There is a fourth case worth handling separately. If an endpoint returns 410 Gone, or 404s consistently for days, the subscription itself is dead. Disable it and notify the account owner rather than dead lettering thousands of events nobody will ever replay.

What to store

The most common mistake in a homegrown dead letter queue is storing the error instead of the event.

A row that says POST failed: 503 is a log line, not a recovery mechanism. You cannot replay it. You have preserved the evidence of the problem and thrown away the thing you actually needed.

A dead letter entry needs the full request as it would be sent again:

the exact payload body
the headers, including the signature and its timestamp
the destination URL and the subscription or endpoint id
the event id and the idempotency key
the original event creation time, separate from the time it was dead lettered
the complete attempt history: timestamp, status code, latency, and response body for every single try
the reason delivery stopped: attempts exhausted, time budget exceeded, circuit open, non retryable status

The response body is the field teams most often skip, usually for storage reasons, and it is the field that saves the most time. A bare 500 tells you nothing about who owns the problem. A 500 with {"error":"unknown_currency: KES"} in the body tells you this is a broken integration rather than a flaky network, and it tells you in one glance instead of one afternoon.

Truncate the body if you need to. Two kilobytes is usually plenty. Do not drop it.

The dead letter queue nobody looks at

A dead letter queue without alerting is /dev/null with extra steps and a storage bill.

This is the failure mode I would watch for most closely, because it feels like success. The table exists. Rows are being written. Delivery has technically been made recoverable. But nobody is subscribed to it, so events accumulate for weeks and the first person to look is an engineer investigating a customer complaint that has already escalated.

Two rules make it real:

Alert on rate, not on individual entries. One dead letter is noise, and paging on every single one trains people to ignore the channel within a week. Forty from the same endpoint inside ten minutes is an incident. Alert on the second thing.

Alert on age. An entry that has been sitting untouched for seven days is a decision nobody made. That is a different alert with a different urgency, and it should go to whoever owns the integration rather than to whoever is on call.

Replay is harder than it looks

The replay button is the payoff for all of this, and it is also where you can do real damage. Three questions are worth asking before you press it.

Is the event still true? Some events are facts and stay valid forever. payment.succeeded was true when it happened and it is still true four days later. Other events are snapshots of state. order.status.updated from Tuesday may now describe a status that has been superseded twice, and replaying it can walk the receiver's state backwards. Facts are safe to replay indefinitely. Snapshots should either expire or be replaced by a fresh read of current state.

Will the receiver double process it? At least once delivery means duplicates are inevitable and replay makes them likely. The idempotency key you generated when the event was first created has to survive into the dead letter entry and go out with the replay, unchanged. If replaying charges a card twice, the dead letter queue has made things worse rather than better.

Does order matter? This one gets skipped and it is the one that corrupts data. If you replay event 40 after events 41 through 60 have already been delivered, the receiver processes an outdated change on top of newer ones. Where ordering matters for a given entity, replay needs to pause that entity's stream, replay in sequence, then resume. Where it does not matter, say so explicitly in your docs so consumers know not to rely on it.

There is a fourth option worth building, which is to edit before you replay. A surprising share of dead letters are not transport failures at all. The endpoint URL changed, or the payload carries a field the receiver rejects. If the only action available is replay as is, those events are stuck forever and someone ends up writing a one off script. Being able to correct the destination or the body and then resend turns a dead end into a two minute fix.

Poison messages

There is one specific pattern worth designing against.

An event that reliably crashes the receiver will fail every attempt, land in the dead letter queue, get replayed by a well meaning engineer, fail again, and come straight back. Left alone it can loop indefinitely, and if you replay in bulk it can take the receiving service down each time it comes around.

Count replays per event. After two or three, quarantine it and require an explicit override to try again. The loop is almost always a malformed payload or a schema mismatch, and no number of retries will fix either.

What building this actually costs

None of the individual pieces here are hard, and that is exactly why teams underestimate the whole.

The rough shape is an events table with a real state machine, a separate attempts table so history survives, a scheduler that wakes up delivery at the right time, a worker pool that does not starve under load, backoff with jitter, circuit breaker state tracked per endpoint, signature generation and zero downtime secret rotation, SSRF protection on every destination URL your customers can set, an admin interface where a human can actually see and act on dead letters, authentication and an audit trail on the replay action because replay is a write operation with real consequences, and a retention policy so the tables do not grow without bound.

Any competent backend team can build that. The honest cost is not the initial build, it is that you now own a piece of infrastructure that has to keep working correctly while you build the product your customers are actually paying for, and its failure modes are silent by nature. Delivery infrastructure does not page you when it breaks. It just stops mentioning things.

If your event volume is low and your endpoints are internal, building it is a perfectly reasonable call. If you are delivering to endpoints you do not control, the surface area grows faster than most teams expect.

The short version

Retries buy you time. The dead letter queue is what turns the events that time did not save into something recoverable.

Store the whole event, not the error. Alert on rate and on age. Keep idempotency keys intact through replay. Decide up front which of your events are facts and which are snapshots. And make sure a real person receives the queue, because a dead letter queue nobody reads is just a slower way to lose data.

Mittr is the reliable action layer for webhooks and AI agent actions. Every action is written to Postgres before delivery, retried on a front-loaded schedule with jitter behind a per-endpoint circuit breaker, and logged on every attempt with status code, latency, and response. Anything that exhausts its retry budget is dead-lettered rather than dropped, where it can be inspected, edited, and replayed, with alerting on Slack, PagerDuty, or email. More at mittr.io.

Why retries matter more than prompts

Stephen Wanjohi — Thu, 23 Jul 2026 07:00:00 +0000

Somewhere right now, an engineering team is holding a retro about their AI agent. It worked in every demo. In production, a customer says it "just didn't do anything." The team is debating a prompt rewrite.

The logs tell a different story. The agent reasoned correctly. It chose the right tool. It made the call. The call timed out. Nothing retried it. The action never happened, no one was notified, and the model took the blame for a networking problem.

This is the most common agent failure we see, and no prompt will fix it.

Agents are event-driven systems wearing a trench coat

Strip away the novelty and an agent action is three steps: decide, act, confirm. The deciding gets all the attention. The acting is usually a bare HTTP call: fetch(url) and hope.

Backend engineers already know how this movie ends, because webhooks taught us. Networks fail. Endpoints rate-limit. Processes restart mid-request. The industry's answer wasn't smarter senders. It was delivery infrastructure: persistence, retries, idempotency, observability. Payments companies have run on it for a decade.

Agents have quietly inherited the exact same problem. Every tool call, callback, and notification an agent produces is an event that needs to arrive. And most agent stacks today deliver those events with less care than a 2015 webhook integration.

What a lost action costs

When a prompt is mediocre, the output is mediocre: visibly, immediately, fixably.

When a delivery fails silently, the cost is deferred and compounding. The customer wasn't notified. The workflow half-completed. And because fire-and-forget leaves no trail, there's nothing to debug: no status code, no attempt history, no way to distinguish "the agent decided wrong" from "the agent decided right and the network ate it."

That last distinction is everything. Without it, every infrastructure failure becomes an indictment of your model, and you'll spend your time tuning prompts that were never the problem.

The reliability checklist that actually moves the needle

None of this is exotic. It's the same discipline reliable webhook systems use, applied to agent actions.

Persist before you attempt. Write the event down before trying to deliver it. If the process dies or the deploy restarts, the intent survives and delivery resumes. Anything held only in memory is a coin flip.

Retry with backoff, and know when to stop. Most delivery failures are transient: a timeout, a 502, a rate limit. Exponential backoff resolves them without hammering the endpoint. A circuit breaker stops you from making a struggling endpoint worse.

Make retries safe. At-least-once delivery means occasional duplicates, so every send needs an idempotency key. In Mittr's MCP tools, mittr_send_event takes an idempotencyKey parameter for exactly this reason, so the model can retry a tool call without double-charging anyone.

Log every attempt. Status code, latency, response body, timestamp. "The webhook never arrived" should be a 30-second lookup, not a day of grepping.

Keep a replay button. Some events fail past any reasonable retry budget, like when an endpoint is down for a full hour. That should be recoverable, not fatal: dead-letter it, alert on it, replay it when the endpoint is back.

Correlate by run. Agent debugging has a dimension webhooks never had: one reasoning loop produces many actions. Tagging every event with a run identifier (agentRunId, in Mittr's case) means you can pull up everything an agent did in a single run and see precisely which action failed, when, and why.

Where the leverage is

Here's the argument in one comparison.

Improving your prompt makes the agent decide a bit better, some of the time. Improving your delivery layer makes every action the agent takes (every tool call, every callback, every notification) either arrive or leave a recoverable, visible trail. One is a marginal gain on the intelligent part. The other removes an entire failure class from the system.

Prompts are a craft. Delivery is a guarantee. Production systems are built on guarantees.

The boring ending

The uncomfortable truth about production agents is that the differentiating work (your product, your reasoning, your UX) sits on top of a pile of undifferentiated plumbing: queues, retry engines, dead-letter queues, audit logs, replay systems.

You can build that pile. Teams do, and it takes weeks to build and years to babysit. Or you can treat delivery as infrastructure, point your agent at it, and spend those weeks on the part of the system your customers actually see.

That second option is why Mittr exists. Your agent sends the event. We do the rest.

Mittr is delivery infrastructure for events, webhooks, and agent actions: persisted first, retried with backoff, logged on every attempt, replayable any time. Agents connect over MCP in one config block: docs.mittr.io/guides/ai-agents. The free tier is 3,000 messages a month, no credit card.