DEV Community: Robbe Verhelst

Resend Prometheus Monitoring: Alert on Email Bounces in Grafana

Robbe Verhelst — Wed, 22 Jul 2026 14:06:51 +0000

If you are looking for Resend Prometheus monitoring, Resend Grafana alerts, or a way to alert on email bounces, that is exactly what resend-exporter is for.

It is a small Prometheus exporter and webhook receiver for Resend email events. It receives Resend webhooks, verifies their Svix signatures, exposes clean Prometheus metrics, and gives Grafana something useful to alert on when transactional email delivery breaks.

The reason I built it was simple.

An appointment confirmation email bounced.

The appointment itself existed. The application did its job. Resend knew the email had failed. But unless someone checked the right dashboard or the customer asked about it, that failure could quietly sit there as a support problem instead of an operational signal.

That felt wrong.

Transactional email is part of the product. Password resets, appointment confirmations, invoices, login links, onboarding flows, receipts: when they fail, the user does not care that your API returned 200 OK.

It turns this:

Resend knows an email bounced somewhere in a vendor dashboard.

Into this:

Prometheus sees resend_email_events_total{event_type="email.bounced"} increase and Grafana can alert me.

The problem with silent Resend email failure

Most web apps treat email as a side effect.

The request succeeds, the database is updated, and some email provider gets asked to deliver a message. At that point the application often considers the work done.

But delivery is where the user value actually happens.

If an appointment confirmation bounces, the appointment is technically created but operationally degraded. If a password reset fails, authentication is broken for that user. If an invoice email never arrives, someone eventually has to untangle the mess.

Email providers like Resend already emit useful delivery events:

email.sent
email.delivered
email.delivery_delayed
email.bounced
email.failed
email.complained

The missing piece, for me, was getting those events into the monitoring stack I already use.

I did not want another dashboard to remember.

I wanted Prometheus metrics, Grafana dashboards, and alert rules.

What the Resend Prometheus exporter does

resend-exporter is deliberately small:

Resend
  -> Svix-signed webhook
  -> resend-exporter
       - verify
       - count
       - log
  -> /metrics for Prometheus
  -> Grafana dashboards and alerts
  -> JSON logs for Loki or stdout

It exposes four HTTP endpoints:

Method	Path	Purpose
`POST`	`/webhooks/resend`	Receive Resend webhook events
`GET`	`/metrics`	Prometheus metrics
`GET`	`/healthz`	Liveness probe
`GET`	`/readyz`	Readiness probe

The webhook handler verifies Resend's Svix signature, validates the payload, updates Prometheus counters and gauges, and emits one structured JSON log line per accepted event.

No database. No queue. No Resend API key required for the basic webhook mode.

Just receive, verify, count, log.

Metrics are not logs

The most important design choice was separating metrics from details.

Prometheus should answer questions like:

how many emails bounced?
did failures increase in the last five minutes?
which event types are happening?
are failures concentrated by sending domain or broad recipient domain?
has the webhook gone stale?

It should not store every recipient address, subject line, or Resend email ID as labels.

That way lies cardinality pain.

So the exporter exposes metrics like:

resend_webhook_events_total{event_type="email.bounced",domain="acme.dev"}
resend_email_events_total{event_type="email.failed",from_domain="acme.dev",to_domain="outlook.com"}
resend_webhook_signature_failures_total
resend_webhook_handler_errors_total{reason="invalid_json"}
resend_webhook_last_event_timestamp_seconds{event_type="email.delivered"}

Those labels are intentionally boring.

The more sensitive forensic detail goes into structured logs:

{
  "level": "warn",
  "event_type": "email.bounced",
  "resend_email_id": "3ebe19b6-1dcc-4534-8442-9dc689ee439b",
  "from_domain": "example.com",
  "to_domain": "outlook.com",
  "recipient_count": 1,
  "reason": "Recipient mail server not found",
  "event_created_at": "2026-07-21T00:24:00Z"
}

Metrics tell you that something is wrong.

Logs tell you what broke.

Keeping those jobs separate makes the exporter much safer to operate.

Keeping Prometheus labels under control

Email data is cardinality bait.

It is very tempting to add labels like:

to="customer@example.com"
subject="Your appointment confirmation"
email_id="..."

That would make the first dashboard look convenient and the second month of Prometheus look stupid.

resend-exporter avoids high-cardinality labels by design:

full recipient emails are never metric labels
subjects are never metric labels
Resend email IDs are never metric labels
recipient domains are bucketed
the long tail can collapse into other
extra recipient domains can be allowlisted when needed

For private deployments, the logs can include more detail. For shared or stricter setups, redaction can stay strict or hash sensitive fields.

The current redaction modes are:

Mode	Behavior
`strict`	Do not log recipient email or subject
`hash`	Log stable SHA-256 hashes for correlation
`none`	Log raw values for private deployments

The default is strict.

Because "just put it in a label" is how dashboards become landmines.

Resend alerting examples for Prometheus and Grafana

Once the events are in Prometheus, the alert rules become ordinary PromQL.

Any failed email:

increase(resend_email_events_total{event_type="email.failed"}[5m]) > 0

Any bounced transactional email:

increase(resend_email_events_total{event_type="email.bounced"}[5m]) > 0

Repeated delivery delays:

increase(resend_email_events_total{event_type="email.delivery_delayed"}[30m]) >= 3

Signature failures:

increase(resend_webhook_signature_failures_total[5m]) > 0

That last one is useful because webhook endpoints are public by necessity. Bad signatures should be rejected, counted, and visible.

Running resend-exporter

With Docker:

docker run -p 8080:8080 \
  -e RESEND_WEBHOOK_SECRET=whsec_... \
  ghcr.io/robbeverhelst/resend-exporter:latest

With Helm:

helm install resend-exporter oci://ghcr.io/robbeverhelst/charts/resend-exporter \
  --set resend.webhookSecret=whsec_... \
  --set serviceMonitor.enabled=true

For local testing, the repo includes a compose playground with the exporter, Prometheus alert rules, and a provisioned Grafana dashboard:

docker compose up --build

Then configure a Resend webhook to point at:

https://your-host/webhooks/resend

The webhook path must be internet-reachable.

The metrics endpoint should not be.

What ships today

The repository currently includes:

Bun/TypeScript implementation
Svix webhook signature verification
Prometheus metrics via prom-client
structured JSON logging
strict/hash/none redaction modes
recipient-domain bucketing
Docker image
Helm chart
docker-compose playground
Grafana dashboard
Prometheus alert examples
CI and automated releases

Repo: github.com/robbeverhelst/resend-exporter

The current release is v0.2.1.

What I still want to add

The next useful feature is optional Resend API reconciliation.

Webhook-only mode is intentionally simple, but an API key could enable:

backfilling recent events after exporter downtime
checking whether required Resend webhook events are configured
enriching sparse webhook events
exposing configuration health metrics

I also want to add delivery-delay histograms, bounce-type breakdowns, and first-class PrometheusRule support in the Helm chart.

The larger point

This project is small, but the pattern matters.

If a user-facing side effect can fail after your application says "success", it deserves observability.

Transactional email is not background noise. It is production behavior.

And if production behavior breaks, I want the same boring machinery that watches everything else to notice.

That is all resend-exporter really does.

It makes email delivery failures page-shaped.

AI-Powered Media Server Management with OpenClaw and Tsarr

Robbe Verhelst — Mon, 30 Mar 2026 08:39:30 +0000

Last weekend I wanted to know which TV series were eating the most storage on my media server. Simple question.

My AI assistant (running on OpenClaw) tried to answer it. It took multiple approaches, five attempts, and several minutes of fumbling before it found the data. Top Gear at 494 GB. Pokémon at 362 GB. South Park at 344 GB. The answer was there, but getting to it was painful.

The thing is, I'd already built a tool that does this in one command:

tsarr sonarr series list --json

Tsarr is a type-safe CLI I built for managing Radarr, Sonarr, and the rest of the *arr stack. My AI assistant just didn't know it existed.

The Problem

AI assistants are great generalists but terrible specialists. They'll try raw API calls, shell scripts, whatever they can think of. Anything except the purpose-built tool sitting right there. They don't read your README. They don't know your CLI flags. They improvise, and improvisation with infrastructure is how things break.

The Solution: AgentSkills

ClawHub is a skill registry for OpenClaw agents. A "skill" is a SKILL.md file that teaches an AI how to use a specific tool: what commands exist, when to use them, safety rules, and common workflows.

I wrote one for Tsarr. It's structured like this:

skill/
├── SKILL.md                          # Entry point: routing + safety rules
└── references/
    ├── setup.md                      # Installation, config, connectivity
    ├── common-workflows.md           # Health checks, search, add, queue, history
    └── service-cheatsheet.md         # Every command mapped per service

The SKILL.md tells the agent:

Start with tsarr doctor if anything seems off
Use --json when extracting data
Inspect before mutating: always fetch before delete
Ask which service if the user just says "library" without specifying Radarr or Sonarr

The references give it a cheatsheet of every command pattern so it doesn't have to guess.

The Result

Same question, "what series are the largest?", now gets answered instantly:

tsarr sonarr series list --json
# sorted, filtered, exact byte counts, all metadata

No fumbling. No five attempts. The AI knows the tool because I taught it.

Installing the skill on any OpenClaw instance:

npx clawhub install tsarr

One command. The agent immediately knows how to manage your entire *arr stack.

What It Can Do Now

With the skill installed, I just ask in plain English and the agent handles it:

"What's eating my storage?"

tsarr sonarr series list --json  # sorted by size, instant answer
tsarr radarr movie list --json   # same for movies

"Are there any download issues?"

tsarr sonarr queue list          # stuck downloads, warnings, errors
tsarr radarr queue list          # same for movies

"Check if everything is healthy"

tsarr doctor                     # tests all configured services at once

"Find me that new show everyone's talking about"

tsarr sonarr series search "Shogun"
tsarr sonarr series add <id> --quality-profile HD --root-folder /media/shows

"Are subtitles missing for anything?"

tsarr bazarr movie list --json   # check subtitle status across your library

"What got added recently?"

tsarr sonarr history list        # recent grabs, imports, upgrades
tsarr radarr history list

Tsarr supports six services: Radarr (movies), Sonarr (TV), Lidarr (music), Readarr (books), Prowlarr (indexers), and Bazarr (subtitles). Every service follows the same CLI pattern, so the agent learns one and knows them all.

The skill doesn't just list commands. It teaches the agent when to use each one, what flags matter, and when to ask clarifying questions before doing anything destructive.

Why This Matters

Every homelab runner, every DevOps engineer, every developer has tools their AI doesn't know about. Custom CLIs, internal scripts, niche utilities. The knowledge gap between "this tool exists" and "the AI can use it effectively" is just a SKILL.md file.

If you've built a CLI that you use regularly, consider writing a skill for it. Your future self (and your AI) will thank you.

Tsarr: github.com/robbeverhelst/Tsarr
ClawHub skill: clawhub.ai/robbeverhelst/tsarr
OpenClaw: openclaw.ai

Guard Wolves: a Minecraft plugin I shouldn't have been able to build

Robbe Verhelst — Fri, 13 Mar 2026 16:16:57 +0000

I run a small Minecraft server for friends. We had a problem: every time we'd go exploring, mobs would wreck our base. The obvious solution? Guard dogs. Minecraft has wolves, they can be tamed, but tamed wolves are useless as guards. They either sit there doing nothing or follow you around like lost puppies.

I wanted wolves that would stay at a location, patrol a radius, and attack hostile mobs. Basically guard dogs. Minecraft doesn't have that.

The language I'd never seen

The way to script custom behavior in Minecraft (without writing a full Java plugin) is DenizenScript. It's a YAML-like scripting language specific to Minecraft servers. It has its own syntax, its own commands, its own way of handling entities, events, and flags.

I'd never written a line of it. My day job is TypeScript. DenizenScript looks like this:

- if <[wolf].location.distance[<[guard_center]>]> > 15:
  - walk <[wolf]> <[safe_center]> speed:0.3
  - flag <[wolf]> return_attempts:++

That's not YAML. That's not any language I know. The angle brackets, the dot-chained methods, the flag system. It's its own thing entirely.

Learning DenizenScript properly for one project didn't make sense. I'd use it once and forget it. So I described what I wanted in plain English and let AI translate it into DenizenScript.

What I asked for

The concept was simple:

Right-click a tamed wolf with a stick to toggle guard mode
Guard wolf stays at that location and patrols a 15-block radius
It attacks hostile mobs but never players, never creepers (explosions near the base = bad), and never passive animals
If it gets stuck or wanders too far, teleport it back
When the owner comes back, right-click with a stick to get your wolf back

Simple on paper. 463 lines of DenizenScript in practice.

The easy part

AI handled the basic structure well. The event handlers, the combat scanning, the flag system for tracking guard state. Within an hour I had a script that could toggle guard mode and make wolves attack nearby zombies.

The owner removal trick was clever: when a wolf enters guard mode, you remove the owner (so it stops following you and won't teleport to you), but save the original owner in a flag. When you disable guard mode, restore the owner. AI figured that out without me asking.

# Enter guard mode
- flag <[wolf]> original_owner:<player>
- adjust <[wolf]> owner:!
- flag <[wolf]> guard_mode

# Exit guard mode
- adjust <[wolf]> owner:<[wolf].flag[original_owner]>
- flag <[wolf]> guard_mode:!

The hard part

Then reality hit. The script worked in theory. In practice, wolves are dumb.

Pathfinding failures. Wolves would try to walk back to their guard point and get stuck on a fence. Or a half-slab. Or literally nothing visible. The AI-generated "walk back" command wasn't enough. I needed stuck detection: if a wolf hasn't moved in 60 seconds despite being told to walk, teleport it home.

- if <[time_stuck]> > 60:
  - teleport <[wolf]> <[guard_center]>
  - flag <[wolf]> invulnerable_until:<util.time_now.add[3s]>

Teleport safety. Teleporting a wolf sounds simple. But if you teleport it into a block, it suffocates. If you teleport it mid-air, fall damage. If you teleport it while a mob is hitting it, it dies during the invulnerability gap. Every teleport needed a 3-second invulnerability window with protection from multiple damage types.

The creeper problem. First version: wolves attacked everything hostile. I realized pretty quickly that creepers would be a disaster. Wolf attacks creeper, creeper explodes, half your base is gone. Worse than no guard wolf at all. Had to explicitly exclude creepers from the target list.

Server restarts. Guard wolves need to survive server restarts. Their guard points, health, and state all stored in Denizen flags. But on restart, max health resets to vanilla values. Needed a startup script that re-discovers all guard wolves and restores their stats.

Health management. Tamed wolves in Minecraft can be healed by feeding them meat. But guard wolves have their owner removed (that's how they stay put). Without an owner, the healing mechanic doesn't work the same way. I had to add a warning message: "Disable guard mode first, then heal, then re-enable."

Each of these took multiple deploy-test-fix cycles. Change the script, reload on the server, spawn some zombies, watch the wolf, see what breaks, repeat.

What the result looks like

After all the iteration, the plugin is actually solid:

Wolves patrol a 15-block radius and return to their post
They attack zombies, skeletons, spiders, and other hostiles on sight
They ignore creepers, players, and passive mobs
They survive server restarts with all their stats intact
Stuck wolves teleport home with invulnerability
You get death notifications when a guard wolf dies
A /gw ls command shows all your wolves with health, location, and armor status

========== Your Wolves (5) ==========
• Fang (Pale) ❤ 32/40 GUARDING at -487,78,-18,world
• Rex (Ashen) ❤ 20/20 Following at 123,65,456,world
• Scout (Woods) ❤ 8/8 Sitting at -200,70,300,world

The unexpected payoff

Once the guard wolves script was solid, something clicked. I had 463 lines of working DenizenScript that covered events, flags, entity management, combat, persistence. It was basically a reference implementation.

When I wanted other scripts afterward, like keeping dolphins alive (they drown when you wander too far from them), I could point AI at the guard wolves script and say "make something like this but for dolphins." It already understood the patterns: how Denizen handles entities, how flags work, how to hook into server events.

The first script took a day. The dolphin script took maybe 20 minutes. Having a working example in the same language made AI dramatically better at generating new scripts. It's like giving it a style guide instead of asking it to figure everything out from scratch.

The honest breakdown

AI got me maybe 60-70% of the way there. The basic structure, the event system, the flag management. Things that are well-documented and pattern-based.

The remaining 30-40% was all edge cases that only surface when you actually run the thing. Pathfinding quirks, teleport safety, combat targeting rules, persistence across restarts. AI can't predict that wolves get stuck on half-slabs or that creepers near your base are a disaster. That knowledge comes from testing.

Total time: about a day. Without AI, I probably wouldn't have built it at all. DenizenScript is too niche to justify learning for one project. With AI, I could focus on what I wanted instead of how DenizenScript works.

The plugin is open source if you want to use it or build on it: guard-wolves on GitHub.

Anyone else building weird Minecraft plugins? I'd love to see what people are doing with DenizenScript.

Tsarr: a type-safe TypeScript SDK & CLI for the Servarr ecosystem

Robbe Verhelst — Fri, 13 Mar 2026 15:00:13 +0000

If you run Radarr, Sonarr, or any of the *arr apps, you've probably written a quick script to automate something. Bulk imports, library cleanup, monitoring. And you've probably hit the same wall: the APIs are big, undocumented in practice, and every app has slightly different conventions.

I got tired of writing raw fetch calls and guessing at response shapes, so I built Tsarr: a fully type-safe TypeScript client and CLI that covers the entire Servarr ecosystem.

What it does

Tsarr provides auto-generated TypeScript clients for:

Radarr (movies)
Sonarr (TV series)
Lidarr (music)
Readarr (books)
Prowlarr (indexers)
Bazarr (subtitles)

Every client is generated directly from the official Swagger/OpenAPI specs, so types are always accurate and up-to-date. When the *arr devs update their API, Tsarr follows automatically.

How it works under the hood

Each *arr app ships an OpenAPI/Swagger spec that describes their full API. Tsarr uses @hey-api/openapi-ts to generate typed TypeScript clients from those specs.

The flow looks like this:

Fetch the latest OpenAPI specs from each *arr app's GitHub repo
Run @hey-api/openapi-ts to generate fully typed clients (request types, response types, enums, everything)
Bundle it all into one package with modular imports per app
The CLI layer wraps those same clients with citty for the command-line interface

Because the generation is automated, keeping up with API changes is just a matter of re-running the generator. Renovate handles dependency bumps, and semantic-release cuts new versions automatically when PRs merge.

No hand-written API types. No manual maintenance. If the official spec changes, the next build catches it.

Use it as an SDK

import { RadarrClient, SonarrClient } from 'tsarr';

const radarr = new RadarrClient({
  baseUrl: 'http://localhost:7878',
  apiKey: 'your-api-key'
});

// Full type safety: autocomplete, type checking, the works
const movies = await radarr.getMovies();
const status = await radarr.getSystemStatus();

You get proper TypeScript types for every request and response. No more guessing what fields a movie object has.

Use it as a CLI

# Install globally
npm install -g tsarr

# Setup
tsarr config init

# Manage your library from the terminal
tsarr radarr movie list
tsarr radarr movie search --term "Interstellar"
tsarr sonarr series list
tsarr prowlarr indexer list

# Check all connections at once
tsarr doctor

# JSON output for scripting
tsarr radarr movie list --json | jq '.[] | .title'

The CLI supports table output for humans and JSON for scripts. Shell completions included for bash, zsh, and fish.

Why I built this

I self-host everything on a Kubernetes cluster at home. When you manage multiple *arr instances, automation isn't optional. I also run an AI assistant that manages my homelab, and it works way better with CLIs than raw APIs. Being able to say "add Interstellar to Radarr" and have it run tsarr radarr movie add behind the scenes is a game changer.

So I needed:

Type safety so I catch API changes at compile time, not at 3 AM when my automation breaks
One package instead of six different libraries for six different apps
CLI access because sometimes you just want to check something from the terminal without opening a browser

Nothing like this existed for TypeScript. Managarr is a great Rust TUI for interactive management, but if your stack is Node.js/TypeScript and you want to build automation, you were on your own.

Installation

Available everywhere:

# npm
npm install tsarr

# Homebrew
brew install robbeverhelst/tsarr/tsarr

# Docker
docker run --rm ghcr.io/robbeverhelst/tsarr doctor

# AUR (Arch Linux)
yay -S tsarr-bin

# Nix
nix profile install github:robbeverhelst/tsarr?dir=packaging/nix

# Or grab a standalone binary, no runtime needed
curl -L https://github.com/robbeverhelst/tsarr/releases/latest/download/tsarr-linux-x64 -o tsarr

What you can build with it

Some things I use it for:

Bulk movie imports: feed it a list, let it add everything with proper quality profiles
Library audits: find movies without subtitles, series with missing episodes
Monitoring scripts: check queue status, disk space, indexer health in cron jobs
Cross-app automation: when Sonarr grabs a series, trigger Bazarr to fetch subtitles

The CLI is also great for quick checks without leaving the terminal:

# What's in my download queue right now?
tsarr radarr queue list --table

# Any indexer issues?
tsarr prowlarr indexer list --table

It's open source

MIT licensed. Contributions welcome.

If you run any *arr apps and write TypeScript, give it a try. Issues and PRs welcome, especially if you find edge cases in specific API endpoints.

What's your *arr automation setup like? I'd love to hear what people are building.