DEV Community: Deb

Replaying dead-letter messages across Azure, AWS and GCP, and saying "verification required" when it can't be proven

Deb — Wed, 14 Oct 2026 18:38:17 +0000

It is 2am. A queue backs up and messages start landing in the dead-letter queue. Everyone's instinct is the same: replay them.

And then the questions start, the ones you usually can't answer quickly:

Why did these fail? One error, or five different ones mixed together?
Is the cause actually fixed? If not, you are about to send the same messages straight back into the same failure.
Who replayed what, and when? Next week nobody will remember.
Did the replay work? Or are the messages quietly sitting in the dead-letter queue again?

I hit these on every cloud I work with, so I built ServiceHub: a self-hosted, MIT-licensed tool for Azure Service Bus, AWS SQS/SNS and GCP Pub/Sub. This post is the pain first, then what ServiceHub does about each part of it.

Try it in your browser, nothing to install: https://debdevops.github.io/servicehub/ (made-up data, no server, connects to nothing).

The pain is different on each cloud

Replaying sounds simple until you meet the details:

Azure Service Bus: every queue and subscription has its own dead-letter queue, so a bad hour is spread across many places.
AWS SQS: there is no non-destructive peek. Every read is a receive, and it counts toward the queue's maxReceiveCount. Looking at a dead letter is not free.
GCP Pub/Sub: reading a dead letter counts as a delivery attempt too, and Pub/Sub records no failure reason, so you see the message but not why it failed.

Three clouds, three behaviours, and the same 2am question: can I safely put these back?

Pain 1: "I can't see what failed, or why"

What ServiceHub does: one Dead letters page per cloud, with failures grouped by reason, so five different problems no longer look like one big pile.

Azure is watched automatically:

AWS and Google Cloud only know what you ask them to look at, because looking has a cost. ServiceHub never reads their dead letters on its own. You press Look now; it reads up to 100 per queue and keeps them, so you can open, read and replay them.

Where a cloud gives no reason, the page says "no reason recorded" instead of guessing.

Pain 2: "Replaying is a leap of faith"

What ServiceHub does: every replay, whether a person clicks it or the Agent does it, goes through one gate. Before anything is sent it shows a proposal: scope, a sample, the risk, the policy and the stop condition. If a check can't run, the replay is blocked, not waved through.

Replaying everything is previewed too, grouped by failure reason, with a pace and a Stop now:

Pain 3: "I can't tell whether the replay worked"

This is the part I care about most.

Azure can confirm whether a replayed message stayed out of the dead-letter queue. AWS and GCP can't, without extra infrastructure. So on those clouds a replay reads "verification required", never "verified". A tool that says "fixed" when it doesn't know is worse than no tool.

Messages that fail again are labelled for what they are. Here is Azure's Replayed page in my test setup, where the sample messages fail on purpose:

4.2.0 adds an opt-in, per-namespace setting, Confirm fixes on this cloud. If ServiceHub gets a complete view of the cloud's dead-letter queue, a replay can read recovered or came back. A view either saw everything or the replay stays "verification required". There is no confidence score.

Pain 4: "Automation that replays on its own can loop"

What ServiceHub does: the Agent replays on its own only for a failure signature it has earned trust on, per signature, not globally. Until then the messages wait for a person. It has brakes: a circuit breaker, an emergency stop (type STOP, give a reason, both recorded), and one place to approve or decline waiting approvals. And in 4.2.0, Auto Replay backs off by itself if 2 of a failure's last 10 verified replays didn't hold.

Pain 5: "Nobody can say afterwards what happened"

What ServiceHub does: every recovery is written to a hash-chained ledger you can read in Advanced.

Run it yourself

Pick whichever is easiest. All three are free and local.

1. In the browser (no install)

Open https://debdevops.github.io/servicehub/ and press Demo. Six namespaces across three clouds, about 750 dead letters. Replay, approve, switch on an Auto Replay rule, press Emergency stop. Each action says "Demo - nothing was sent".

2. Clone and run from source

For macOS, Linux, or a WSL 2 terminal on Windows. You only need bash, curl (or wget), tar, git and an internet connection. ./run.sh downloads the .NET 10 SDK and Node.js for you if they are missing (into ~/.servicehub/tools, no sudo, checksum-verified).

git clone https://github.com/debdevops/servicehub.git
cd servicehub
./run.sh

The first start takes a few minutes while it downloads and builds. When you see ServiceHub is ready, open http://localhost:3000 and press Demo. Press Ctrl-C to stop everything it started. If port 3000 or 5153 is busy it picks the next free one and tells you.

This mode is for trying and developing. If you already have .NET 10 and Node 22 and want nothing downloaded, use ./run.sh --no-install.

3. Docker

export SERVICEHUB_ENCRYPTION_KEY="$(openssl rand -hex 32)"   # keep it: it protects the cloud credentials you store
docker run -d --name servicehub --restart unless-stopped \
  -p 127.0.0.1:8080:8080 -v servicehub-data:/data \
  -e SECURITY__ENCRYPTIONKEY="$SERVICEHUB_ENCRYPTION_KEY" \
  ghcr.io/debdevops/servicehub:4.2.0

Open http://localhost:8080 and press Demo. The image is public and runs on linux/amd64 and linux/arm64.

Then connect a real cloud

Open Connections and add a cloud: an Azure Service Bus connection string, an AWS access key for one account and region, or a Google project and service account. Credentials are encrypted and stay on your machine. Azure needs a Manage-level policy to list queues and count messages (make a dedicated policy); replay uses Send.

What else is new in 4.2.0

A 13-stop guided tour that drives the real screens.
Interrupted replays are no longer sent twice. They become outcome unknown, block further attempts on that message, and an Approver closes them by recording what they found.
Failures are told apart by error message, so two different failures on one queue no longer share trust.
A read-only MCP server (ten fixed GET addresses; no tool that replays, purges or approves).

I also found and fixed a real bug against live SQS: the whole-queue check had never worked on a real queue.

Limits, stated plainly

Replay on Production namespaces is blocked for everyone.
No sign-in: keep it on localhost, a private network, or behind an authenticating proxy.
It stores the first 500 characters of each message body in a local SQLite file in plain text. Protect the data folder.
4.1.0 and later is a rewrite with no upgrade path from 4.0.0.
Azure, AWS and GCP were exercised live on 2026-10-14 for the screenshots above. Not observed live: a cured replay on GCP, and 1,000+ message queues. The infra/ Terraform modules have not been run against a real account.

Repo and release notes: https://github.com/debdevops/servicehub. If something confused you or broke on first run, open an issue. I want to hear it.

I built a forensic debugger for Azure Service Bus dead-letter queues — here's how it works

Deb — Fri, 01 May 2026 08:33:50 +0000

It's 2 AM. Your monitoring fires. 5,000 messages in the Dead-Letter Queue.
You need to know three things immediately:

What's actually inside those messages
Which failure pattern caused this
Which messages are safe to replay right now

That investigation used to take me 6 hours. It now takes 45 minutes. Not because I got faster — because I built the right tool.

ServiceHub is an open-source, self-hosted web application that treats dead-letter queue debugging as the forensic investigation it actually is.

Here's what it does that a standard management console cannot:
Full message body inspection — click any message, active or dead-letter, and see the complete JSON with syntax highlighting, every system property, and the exact broker error description.

Client-side AI pattern detection — clusters your DLQ messages into error groups with confidence scores. Runs entirely in your browser. Zero data exfiltration. No API key needed.

Auto-replay rules — define conditions, set rate limits, add exponential backoff. The engine runs autonomously and shows live Pending → Replayed → Success counters.

30-day DLQ Intelligence — persists every scan to local SQLite. Know whether tonight's spike is a new incident or the same pattern from last Tuesday.

Correlation Explorer — paste any Correlation ID and trace that message across every queue, topic, and namespace in one search.

The entire thing is read-only by design. Uses PeekMessagesAsync — your consumers are never affected. Works with Listen permission only.

bash
git clone https://github.com/debdevops/servicehub.git
cd servicehub && ./run.sh

One command. Auto-installs .NET 10 and Node.js.

Open http://localhost:3000.

Or try the hosted demo in 30 seconds — no install:
👉 https://app-servicehub-prod.azurewebsites.net/

The hosted version authenticate via Microsoft Entra ID (Azure AD). No user database. No personal data stored. Connection strings AES-GCM encrypted at rest in your session.

What does your current DLQ investigation workflow look like? Drop it in the comments — I'm genuinely curious what the most painful part still is.

⭐ GitHub: https://github.com/debdevops/servicehub — MIT license, PRs welcome.

I Built a Free Web Tool to Debug Azure Service Bus DLQs Without the 2 AM Headache

Deb — Fri, 03 Apr 2026 15:58:26 +0000

I Built a Free Web Tool to Debug Azure Service Bus DLQs Without the 2 AM Headache

It’s 2 AM. Your pager goes off.

Azure Service Bus is showing thousands of dead-letter messages, but the Azure Portal only tells you a number. No message content. No failure reasons. No easy way to understand what went wrong or how to safely fix it.

After experiencing this pain too many times, I decided to build ServiceHub — an open-source web-based forensic debugger for Azure Service Bus.

Try It Live — No Installation Needed

I’ve deployed a free hosted demo so you can test it instantly:

→ Try ServiceHub Live Demo

Just paste your Service Bus connection string and click Connect.

It works safely with just Listen permission (read-only by default).

What ServiceHub Gives You

Full Message Visibility — Browse active queues, topics, and subscriptions with complete JSON body, system properties, custom headers, and syntax highlighting.
Powerful DLQ Intelligence — Persistent history, auto-categorization of failures (Transient, MaxDelivery, Expired, etc.), forensic timeline, trend charts, and export to JSON/CSV.
Auto-Replay Rules — Define smart rules once and automatically replay matching messages with live success statistics and rate limiting.
Safe Batch Replay — Replay multiple messages safely with strong confirmation guards.
AI-Powered Insights — Client-side pattern detection and anomaly flagging (no data leaves your browser).
Testing Tools — Built-in Message Generator with realistic scenarios and anomaly injection.

All of this in a clean, modern web interface built with React 18 and .NET 10.

Run It Locally (Completely Free & Open Source)


bash
git clone https://github.com/debdevops/servicehub.git
cd servicehub
./run.sh