Replaying dead-letter messages across Azure, AWS and GCP, and saying "verification required" when it can't be proven

It is 2am. A queue backs up and messages start landing in the dead-letter queue. Everyone's instinct is the same: replay them.

And then the questions start, the ones you usually can't answer quickly:

• Why did these fail? One error, or five different ones mixed together?
• Is the cause actually fixed? If not, you are about to send the same messages straight back into the same failure.
• Who replayed what, and when? Next week nobody will remember.
• Did the replay work? Or are the messages quietly sitting in the dead-letter queue again?

I hit these on every cloud I work with, so I built ServiceHub: a self-hosted, MIT-licensed tool for Azure Service Bus, AWS SQS/SNS and GCP Pub/Sub. This post is the pain first, then what ServiceHub does about each part of it.

Try it in your browser, nothing to install: https://debdevops.github.io/servicehub/ (made-up data, no server, connects to nothing).

The pain is different on each cloud

Replaying sounds simple until you meet the details:

• Azure Service Bus: every queue and subscription has its own dead-letter queue, so a bad hour is spread across many places.
• AWS SQS: there is no non-destructive peek. Every read is a receive, and it counts toward the queue's maxReceiveCount. Looking at a dead letter is not free.
• GCP Pub/Sub: reading a dead letter counts as a delivery attempt too, and Pub/Sub records no failure reason, so you see the message but not why it failed.

Three clouds, three behaviours, and the same 2am question: can I safely put these back?

Pain 1: "I can't see what failed, or why"

What ServiceHub does: one Dead letters page per cloud, with failures grouped by reason, so five different problems no longer look like one big pile.

Azure is watched automatically:

AWS and Google Cloud only know what you ask them to look at, because looking has a cost. ServiceHub never reads their dead letters on its own. You press Look now; it reads up to 100 per queue and keeps them, so you can open, read and replay them.

Where a cloud gives no reason, the page says "no reason recorded" instead of guessing.

Pain 2: "Replaying is a leap of faith"

What ServiceHub does: every replay, whether a person clicks it or the Agent does it, goes through one gate. Before anything is sent it shows a proposal: scope, a sample, the risk, the policy and the stop condition. If a check can't run, the replay is blocked, not waved through.

Replaying everything is previewed too, grouped by failure reason, with a pace and a Stop now:

Pain 3: "I can't tell whether the replay worked"

This is the part I care about most.

Azure can confirm whether a replayed message stayed out of the dead-letter queue. AWS and GCP can't, without extra infrastructure. So on those clouds a replay reads "verification required", never "verified". A tool that says "fixed" when it doesn't know is worse than no tool.

Messages that fail again are labelled for what they are. Here is Azure's Replayed page in my test setup, where the sample messages fail on purpose:

4.2.0 adds an opt-in, per-namespace setting, Confirm fixes on this cloud. If ServiceHub gets a complete view of the cloud's dead-letter queue, a replay can read recovered or came back. A view either saw everything or the replay stays "verification required". There is no confidence score.

Pain 4: "Automation that replays on its own can loop"

What ServiceHub does: the Agent replays on its own only for a failure signature it has earned trust on, per signature, not globally. Until then the messages wait for a person. It has brakes: a circuit breaker, an emergency stop (type STOP, give a reason, both recorded), and one place to approve or decline waiting approvals. And in 4.2.0, Auto Replay backs off by itself if 2 of a failure's last 10 verified replays didn't hold.

Pain 5: "Nobody can say afterwards what happened"

What ServiceHub does: every recovery is written to a hash-chained ledger you can read in Advanced.

Run it yourself

Pick whichever is easiest. All three are free and local.

1. In the browser (no install)

Open https://debdevops.github.io/servicehub/ and press Demo. Six namespaces across three clouds, about 750 dead letters. Replay, approve, switch on an Auto Replay rule, press Emergency stop. Each action says "Demo - nothing was sent".

2. Clone and run from source

For macOS, Linux, or a WSL 2 terminal on Windows. You only need bash, curl (or wget), tar, git and an internet connection. ./run.sh downloads the .NET 10 SDK and Node.js for you if they are missing (into ~/.servicehub/tools, no sudo, checksum-verified).

git clone https://github.com/debdevops/servicehub.git
cd servicehub
./run.sh

The first start takes a few minutes while it downloads and builds. When you see ServiceHub is ready, open http://localhost:3000 and press Demo. Press Ctrl-C to stop everything it started. If port 3000 or 5153 is busy it picks the next free one and tells you.

This mode is for trying and developing. If you already have .NET 10 and Node 22 and want nothing downloaded, use ./run.sh --no-install.

3. Docker

export SERVICEHUB_ENCRYPTION_KEY="$(openssl rand -hex 32)"   # keep it: it protects the cloud credentials you store
docker run -d --name servicehub --restart unless-stopped \
  -p 127.0.0.1:8080:8080 -v servicehub-data:/data \
  -e SECURITY__ENCRYPTIONKEY="$SERVICEHUB_ENCRYPTION_KEY" \
  ghcr.io/debdevops/servicehub:4.2.0

Open http://localhost:8080 and press Demo. The image is public and runs on linux/amd64 and linux/arm64.

Then connect a real cloud

Open Connections and add a cloud: an Azure Service Bus connection string, an AWS access key for one account and region, or a Google project and service account. Credentials are encrypted and stay on your machine. Azure needs a Manage-level policy to list queues and count messages (make a dedicated policy); replay uses Send.

What else is new in 4.2.0

• A 13-stop guided tour that drives the real screens.
• Interrupted replays are no longer sent twice. They become outcome unknown, block further attempts on that message, and an Approver closes them by recording what they found.
• Failures are told apart by error message, so two different failures on one queue no longer share trust.
• A read-only MCP server (ten fixed GET addresses; no tool that replays, purges or approves).

I also found and fixed a real bug against live SQS: the whole-queue check had never worked on a real queue.

Limits, stated plainly

• Replay on Production namespaces is blocked for everyone.
• No sign-in: keep it on localhost, a private network, or behind an authenticating proxy.
• It stores the first 500 characters of each message body in a local SQLite file in plain text. Protect the data folder.
• 4.1.0 and later is a rewrite with no upgrade path from 4.0.0.
• Azure, AWS and GCP were exercised live on 2026-10-14 for the screenshots above. Not observed live: a cured replay on GCP, and 1,000+ message queues. The infra/ Terraform modules have not been run against a real account.

Repo and release notes: https://github.com/debdevops/servicehub. If something confused you or broke on first run, open an issue. I want to hear it.