DEV Community: Anand Rathnas

Grafana 11 Provisions Alerts Before Dashboards — and That Breaks folderUid

Anand Rathnas — Fri, 04 Sep 2026 05:49:41 +0000

We rolled a tidy little change into our Grafana provisioning: pin the dashboard folder's UID so deep links don't break across redeploys. Restarted the container. Watched it exit 1 with this:

failed to create folder for provisioned dashboards" folder=jo4 ... err="a folder with the same name already exists in the current location"

That error makes no sense. We hadn't created any folder called jo4 anywhere else. The provisioning files in /etc/grafana/provisioning/dashboards/ are the only place the name appears. So who created the first one?

The answer turned out to be: Grafana did. Before our dashboard provider ran. And it's not what the docs imply.

What We Expected

The Grafana provisioning docs are organized in roughly the order we'd expect a fresh install to come up:

Datasources (so dashboards have something to query)
Dashboards (the things humans look at)
Alerting (rules that fire against dashboards/queries)

Our providers.yaml set the folder name and the folder UID, on the reasonable theory that pinning the UID protects external deep links from rotating after a rebuild:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    folderUid: jo4-folder           # <-- our innocent addition
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards/jo4

If dashboards are provisioned before alerting, this is fine. The dashboard provider creates a folder named jo4 with UID jo4-folder. Alerting provisioning runs later, references folder: jo4 by name, matches the existing one. Everyone goes home.

That is not what Grafana 11.2.0 actually does.

What Actually Happens

We spun up a sandbox container, dropped in the exact same provisioning tree, and tailed the logs. Here's the relevant window, milliseconds and all:

06:26:57.106  msg="Initialising datasources"
06:26:57.204  msg="starting to provision alerting"
06:26:57.322  msg="finished to provision alerting"
06:26:57.344  msg="starting to provision dashboards"
06:26:57.346  level=error msg="failed to create folder for provisioned dashboards"
              folder=jo4 ... err="a folder with the same name already exists in the current location"
06:26:57.346  msg="Stopped background service" service=*provisioning.ProvisioningServiceImpl
              reason="failed to provision dashboards"
06:26:57.346  level=fatal msg="Server shutdown" reason="..."

Three things to read off this:

Alerting provisions before dashboards. It's not even close — alerting starts 140ms before dashboards and finishes 22ms before dashboards even start. This is the opposite of how the docs are organized.
The alerting provisioner created the folder. Two milliseconds into dashboard provisioning, the error is already firing.
The whole provisioning service stops. Not "skip this dashboard provider and continue." It stops everything. The Grafana process then exits 1 because provisioning is a required background service. The container restart-loops.

Why

Once you accept the ordering, the rest falls out cleanly.

The alerting provisioner takes a folder field — a name, not a UID. There is no folderUid field accepted in alerting rule provisioning files. When alerting runs first and sees folder: jo4, it needs that folder to exist. It doesn't. So Grafana creates it, with a UID it generates on the spot (something like ffn7ba7vmp3i8a — looks random, is deterministic for this install).

Now the dashboard provisioner runs. Our providers.yaml says: "make sure there's a folder named jo4 with UID jo4-folder." Grafana does the name lookup first, finds an existing folder named jo4 — but its UID is ffn7ba7vmp3i8a, not jo4-folder. The provisioner's reconciliation logic decides this is a different folder and tries to create one with the pinned UID. The unique-name constraint on folders fires. Provisioning service stops. Container exits 1.

So folderUid in the dashboard provider is fundamentally incompatible with also having alerting rules in the same folder — at least under Grafana 11.2.0's current ordering — because alerting will have created the folder first, with its own auto-generated UID, and your pinned UID can never win the race.

The Fix

One removed line. The dashboard provider keeps folder: jo4 (the human-readable name) and drops folderUid entirely:

Before:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    folderUid: jo4-folder
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/jo4
      foldersFromFilesStructure: false

After:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/jo4
      foldersFromFilesStructure: false

That's it. With no folderUid, the dashboard provider does a name-only lookup, finds the folder alerting already created, reuses it, and provisions all the dashboards into it. The container comes up clean.

We also left a load-bearing comment in providers.yaml so the next person who sees a dangling UID in deep links and thinks "I'll just pin this" gets the full story before they break the deploy:

# IMPORTANT: do NOT set `folderUid`. Grafana 11.2.0 provisioning order is
# datasources → alerting → dashboards (verified empirically). Alerting
# provisioning auto-creates the folder by name (only field accepted) with a
# Grafana-generated UID. If we then pin a different folderUid here, Grafana
# tries to create a SECOND folder with the same name, errors, stops the
# provisioning service, and the container exits 1.

Verifying It Worked

Re-running the sandbox after removing folderUid, here's the timeline:

msg="starting to provision alerting"
msg="finished to provision alerting"
msg="starting to provision dashboards"
msg="finished to provision dashboards"

No error. Process stays up. Then we hit the Grafana API to see who owns the folder:

curl -s -u admin:admin http://localhost:3000/api/folders | jq '.[] | {title, uid}'

Output:

{ "title": "jo4", "uid": "ffn7ba7vmp3i8a" }

And the dashboards inside that folder all report folderUid: ffn7ba7vmp3i8a. So do the alerting rule groups. One folder, one UID, both provisioners pointing at it. Exactly what we wanted — we just don't get to choose the UID.

If you genuinely need a stable folder UID for deep links (we don't, but you might), the only path we've found that works is: let alerting auto-create it, read the resulting UID back via the API once after the first install, and codify it as an external constant. Don't try to pin it through providers.yaml.

Lessons Learned

Documentation ordering is not runtime ordering. The Grafana docs describe provisioning subsystems in the order you'd intuitively initialize them, but the actual startup sequence in 11.2.0 is datasources → alerting → dashboards. If something cross-references between subsystems, find out who runs first the empirical way — read the logs.
A unique-name constraint plus two creators is always a race. Whenever two independent provisioners can both produce the "same" named resource and only one of them lets you specify the UID, the one without UID control wins by going first. Your pinned UID has no path to victory.
Provisioning service failure = container failure. Grafana doesn't gracefully degrade when one provisioning file is bad. The whole service stops and the process exits. Treat your provisioning YAML with the same rigor as a database migration: it can take down the system.
folderUid in a dashboard provider is a foot-gun the moment you also have alerting in the same folder. Drop it. Let the folder be created by whoever runs first and look it up by name everywhere else.
Leave the comment. Future-you, or the next operator, will look at the removed folderUid and wonder why. A five-line comment in the YAML is cheaper than another two-hour debugging session.

Got bit by Grafana provisioning ordering? What was your symptom? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Your sed -i Edit Isn't Reaching the Container: The Bind-Mount Inode Trap

Anand Rathnas — Wed, 02 Sep 2026 05:35:45 +0000

We tweaked a scrape config in our Prometheus YAML, ran sed -i on the host, confirmed the file on disk had the new content, and waited for Prometheus to pick it up. It didn't. An hour later we were staring at a stat output that explained everything we'd ever misunderstood about bind mounts. If you've ever edited a file Docker mounted into a container and watched the container act like nothing happened, this post is for you.

The Symptom

Our Prometheus container has its config bind-mounted from the host:

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - /opt/impress/prometheus.yml:/etc/prometheus/prometheus.yml

We needed to add a scrape target. Easy:

sed -i 's/scrape_interval: 15s/scrape_interval: 30s/' /opt/impress/prometheus.yml
docker exec prometheus kill -HUP 1   # signal Prometheus to reload

The host file looked correct. Prometheus's /-/reload endpoint returned 200. But the runtime config in the UI still showed 15s. We re-ran the SIGHUP. Same. We checked Prometheus logs — it said it had reloaded the config. From its own file. Which apparently still said 15s.

That's the moment we stopped trusting our assumptions and asked the container what it was actually seeing.

Diagnosis: `stat` Tells the Truth

We compared the two views of "the same file":

# On the host
$ md5sum /opt/impress/prometheus.yml
a3f9c1e8...  /opt/impress/prometheus.yml

$ stat /opt/impress/prometheus.yml
  File: /opt/impress/prometheus.yml
  Size: 2847       Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 1835421     Links: 1

# Inside the container
$ docker exec prometheus md5sum /etc/prometheus/prometheus.yml
7b2d4f06...  /etc/prometheus/prometheus.yml

$ docker exec prometheus stat /etc/prometheus/prometheus.yml
  File: /etc/prometheus/prometheus.yml
  Size: 2791       Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 1835098     Links: 0

Two things jumped out:

Different MD5s. The container is reading different bytes than the host file contains.
Links: 0 inside the container. Zero hard links means no directory entry points at this inode anywhere on the filesystem. The file has been unlinked. The container is the only thing keeping the inode alive.

Different inode numbers (1835421 on the host, 1835098 in the container) confirmed it: these aren't the same file anymore. They were, when the container started. They aren't now. The container is holding open a ghost — an inode that exists only because it still has an open file descriptor pointing at it.

Why sed -i Is the Culprit

This is the part of sed -i nobody reads the man page about. sed -i does not edit in place. Despite the flag name, it does this:

Open the target file for reading.
Open a temp file in the same directory (sedXYZ or similar).
Stream the transformed content to the temp file.
rename(2) the temp file over the target name.

That rename(2) is the killer. The original inode (the one with the old content) is no longer reachable by name, but it doesn't get freed because the container has it open. The directory entry /opt/impress/prometheus.yml now points at a brand new inode with the new content. On the host that's the same path with new bytes — looks fine. Inside the container, nothing changed: the bind mount was set up at container start against the original inode, and bind mounts of single files are bound to inodes, not directory entries.

So the container keeps reading the ghost. Forever. Or until you restart it.

This is also why Links: 0. The original directory entry is gone (it now points to the new inode), so the old inode has zero references in the directory tree — but it has one open file descriptor inside the container, so the kernel keeps it alive.

The Fix

There are two correct ways out.

Option A: edit in place for real

You want to write into the same inode, not replace it. cat > file does exactly that — it opens the existing inode with O_TRUNC and writes new content. The inode number doesn't change. The bind mount still points at it. The container sees the update immediately:

# Build new content somewhere, then truncate-and-write the SAME inode
sed 's/scrape_interval: 15s/scrape_interval: 30s/' /opt/impress/prometheus.yml > /tmp/prom.new
cat /tmp/prom.new > /opt/impress/prometheus.yml
rm /tmp/prom.new

Confirm with stat that the inode number didn't change. If it did, you used a tool that does atomic-rename. Back up and try again.

Option B: restart the container

If the file change is paired with anything else (image bump, env-var change, a sibling config edit), just rebuild the container. This is what we did:

docker compose up -d --force-recreate prometheus

--force-recreate gives us a fully fresh container — new bind mounts against current inodes, plus any env-var or compose-file changes get picked up. Less surgical than cat > file, but bulletproof when you're not sure what else drifted.

docker compose restart prometheus also works for the inode problem alone, but it won't pick up env-var or compose-file changes — it restarts the existing container in place.

Tools That Have the Same Trap

sed -i is the famous one, but the trap is everywhere atomic-rename is the safe-write idiom:

mv newfile oldfile — literally rename(2). Same trap.
vim and emacs by default — both write to a backup/swap file and rename over the target, for crash safety. Configurable, but the defaults bite.
perl -i -pe '...' — same temp-file-and-rename dance as sed -i.
awk '...' file > tmp && mv tmp file — the explicit version of the same thing.
Most "safe save" code in editors and language tooling — gofmt -w, prettier --write, black, you name it. They rename for atomicity, which is correct for preventing half-written files on crash, and wrong for bind-mounted single files.

Anything that promises atomic writes is doing this. Atomic writes and bind-mounted single files are fundamentally incompatible.

The Bulletproof Workflow

After this incident we made two rules for editing bind-mounted config:

Prefer directory mounts over single-file mounts. A bind mount of a directory tracks the directory entries inside it, not specific inodes. Atomic-rename inside that directory works the way you expect. The cost is exposing sibling files to the container.
When a single-file mount is unavoidable, edit with cat > file. Build the new content in a tempfile somewhere outside the mounted path, then truncate-and-write the target. Verify with stat that the inode number is unchanged.

For our Prometheus case we kept the single-file mount (we don't want other files in /opt/impress visible to Prometheus) and added a tiny wrapper script that does the cat-redirect dance. No more sed -i on bind-mounted files. Ever.

Lessons Learned

sed -i is a lie. It's not in-place. It's "write a temp file and rename." That rename(2) creates a new inode and orphans the old one.
Bind mounts of single files are inode-bound. Whatever inode the directory entry pointed to at container-start time is the inode the container will hold forever. Replace that inode on the host and the container keeps the old one.
Links: 0 on a file inside a container is the smoking gun. It means the container is the last reference to an inode that's been unlinked from the filesystem. You're looking at a ghost.
md5sum on both sides of the mount is the fastest diagnostic. If host and container disagree, you've hit some flavor of this trap.
Atomic-rename is correct for safety, wrong for bind mounts. Use cat > file for in-place truncate-and-write, or mount the parent directory instead of the single file.
When in doubt, --force-recreate the container. It's the bigger hammer but it's reliable and it picks up env-var changes too.

Ever spent an hour chasing this? What was your tell? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Why pgrep Can't Detect cloud-init's Apt Races on DigitalOcean

Anand Rathnas — Mon, 31 Aug 2026 06:51:34 +0000

If you've ever written a wait loop that polls pgrep apt-get before running your own apt-get, congratulations: you have a race condition. We had one too. On DigitalOcean's Ubuntu 24.04 droplets, our cold-boot setup script would intermittently die on the apt frontend lock — even though we'd "carefully" waited for apt to be idle. Here's why our defensive wait loop was lying to us, and the one line that actually fixed it.

The Setup

We run a small observability droplet — Prometheus + Grafana behind nginx, called jo4-impress. It's bootstrapped from a GitHub Actions workflow that SCPs impress/do-setup/ to the freshly-provisioned droplet and runs setup-impress.sh. The script installs Docker via the convenience script (curl -fsSL https://get.docker.com | sh), then brings up the compose stack.

Because we knew Ubuntu's unattended-upgrades and the apt-daily timers fire shortly after boot, we put a pgrep-based wait loop in front of the Docker install. The idea: poll until no apt-get is running, then run our own. Belt-and-braces. What could go wrong.

The Problem

On a fresh droplet, the script intermittently failed with the classic:

E: Could not get lock /var/lib/dpkg/lock-frontend.
   It is held by process 2746 (apt-get)

The frustrating part: our wait loop reported "no apt-get running" two seconds before this error. We weren't ignoring it. We were checking. And yet PID 2746 — an apt-get we never saw — grabbed the lock the instant our curl get.docker.com | sh invoked its own apt-get update.

Here's the timeline pulled from /var/log/apt/history.log on the droplet (workflow run 26424379616):

Time (UTC)	Event
23:48:13	Droplet boots
23:49:01–27	`apt-get install ca-certificates gnupg curl …` (spawned by `install-do-agent`)
23:49:35–38	`apt-get install do-agent` (still inside `install-do-agent`)
23:49:42	Our pgrep wait loop concludes: "no apt-get running, proceeding"
23:49:43	`curl get.docker.com \
23:49:45	Lock contention error. Holder: PID 2746
23:49:48–49	{% raw %}`apt-get install droplet-agent` completes (this was PID 2746)

PID 2746 was spawned by install-dotty-agent, a DigitalOcean cloud-init vendor script. It started in the four-second gap between install-do-agent finishing and our curl | sh racing into apt-get update. Our pgrep loop caught the quiet moment between two vendor scripts and concluded — correctly, in the instant it sampled — that apt was idle. By the time we acted on that conclusion, the next vendor script had already launched.

The Two Sources of Contention

This is where it gets interesting. On a DigitalOcean Ubuntu 24.04 droplet at first boot, there are two completely independent things competing for the apt lock, and most defenses only address one of them.

(1) The systemd timer units. These are the ones every "first-boot apt fix" tutorial talks about:

unattended-upgrades.service
apt-daily.service, apt-daily.timer
apt-daily-upgrade.service, apt-daily-upgrade.timer

They fire on a randomized schedule shortly after boot. systemctl stop + systemctl mask handles them. This is the well-known half of the problem.

(2) cloud-init's modules-final → config-scripts_vendor stage. This is the half nobody warns you about. DigitalOcean ships two vendor scripts that cloud-init runs during this stage:

install-do-agent — installs the DigitalOcean monitoring agent
install-dotty-agent — installs droplet-agent (the web-console SSH bridge)

Both shell out to apt-get install … directly. They are not systemd units. They don't appear in systemctl list-units. Masking the apt timers does precisely nothing to them, because they were never running through those timers in the first place.

On the run captured above, install-do-agent ran 23:48:29–23:49:41 (71.8 seconds total) and install-dotty-agent ran 23:49:41–23:49:53 (12.2 seconds). Between them: a short window where pgrep apt-get returns empty — but cloud-init is very much still going to launch the next one.

Why the pgrep Loop Failed

The pgrep approach has a structural flaw, not a tuning flaw. Let's be explicit:

# The old wait loop (approximately)
for i in $(seq 1 60); do
  if ! pgrep -x apt-get >/dev/null; then
    echo "apt is free, proceeding"
    break
  fi
  sleep 5
done
apt-get install -y docker-ce  # races whatever cloud-init launches next

The check answers one question: is there an apt-get process running **right now? That's a point-in-time observation. It tells you nothing about whether something about to be launched will run an apt-get in the next few seconds.

Cloud-init's vendor-scripts stage is a sequence. While it's executing the sequence, the gaps between individual apt-get invocations are normal — script setup, package downloads, post-install hooks — and routinely large enough to make pgrep look clean. The wait loop is sampling a process state when it should be checking a higher-level lifecycle state: "is cloud-init done yet?"

The systemd-timer half of the problem made this look like a tuning issue ("just sleep longer", "increase the poll count"). It wasn't. No amount of polling helps when the next contender hasn't been spawned yet.

The Fix

Cloud-init ships a command that answers the actual question:

cloud-init status --wait

It blocks until cloud-init reaches its done state — vendor scripts inclusive. Once it returns, every install-*-agent invocation has completed and won't be spawning more apt-gets. We can install Docker without surprise contenders.

The diff that fixed setup-impress.sh:

- # Wait until any boot-time apt is finished.
- for i in $(seq 1 60); do
-   if ! pgrep -x apt-get >/dev/null; then
-     break
-   fi
-   sleep 5
- done
+ # Defense-in-depth: silence the systemd timer units...
+ APT_UNITS=(
+   unattended-upgrades.service
+   apt-daily.service apt-daily.timer
+   apt-daily-upgrade.service apt-daily-upgrade.timer
+ )
+ systemctl stop "${APT_UNITS[@]}" 2>/dev/null || true
+ systemctl mask "${APT_UNITS[@]}" 2>/dev/null || true
+
+ # Load-bearing: block until cloud-init's vendor scripts finish.
+ echo "⏳ Waiting for cloud-init to finish (DO vendor scripts release apt)..."
+ cloud-init status --wait
+
  echo "📦 Installing Docker via convenience script..."
  curl -fsSL https://get.docker.com | sh
  systemctl enable --now docker
+
+ # Lift the masks once Docker is in.
+ systemctl unmask "${APT_UNITS[@]}" 2>/dev/null || true
+ systemctl enable --now apt-daily.timer apt-daily-upgrade.timer 2>/dev/null || true
+ systemctl enable --now unattended-upgrades.service 2>/dev/null || true

Two things to notice. First: cloud-init status --wait is doing the load-bearing work. It's the one line that closes the race. Second: we still mask the apt timers, but only as defense-in-depth against the scenario where cloud-init has finished but a timer fires inside the Docker-install window. We re-enable them at the end so the droplet's normal patching cadence resumes.

The full context lives in impress/do-setup/setup-impress.sh, commented inline so the next person who reads it doesn't have to re-derive any of this.

Lessons Learned

pgrep answers "is X running now?" — not "will X run in the next ten seconds?" Any defensive wait built around pgrep has this blind spot. Use it for "wait for this specific PID to exit", not "wait for a class of process to be permanently quiet."
On Ubuntu 24.04 cloud images, there are two boot-time apt actors, not one. Systemd timers and cloud-init vendor scripts run independently. Masking the timers and ignoring cloud-init buys you the silence right up until cloud-init speaks.
cloud-init status --wait is the right primitive. It encodes the lifecycle question ("is cloud-init done?") that your script actually wants the answer to. No polling, no false positives, no off-by-a-few-seconds.
Belt-and-braces is fine; one of them has to actually be load-bearing. We kept the systemd masks because they cost us nothing and close a real (if narrower) window. But the masks alone never fixed the bug — cloud-init status --wait did.
When an intermittent bug has a four-second window, the logs from one bad run are worth more than a week of staring at the code. /var/log/apt/history.log on the failing droplet handed us the answer in the form of PID 2746 and an exact second. The fix took ten minutes once we had it.

Hit a similar cloud-init-vs-apt race? Drop the timeline in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

How to Plug Claude Routines Into Your SaaS With MCP

Anand Rathnas — Fri, 28 Aug 2026 12:37:56 +0000

A while back a user asked, "Can I run jo4 from a Claude routine?"

I said "sure, eventually", which is what I always say when something sounds like a weekend and I know damn well it's a long weekend.

It was a long weekend.

But jo4.io now speaks MCP. Claude routines (and any other MCP-aware client) can shorten links, list URLs, pull click stats, update titles, and delete links — all through the same Spring Boot API that powers the dashboard.

Here's what shipped, why it's mostly an OAuth story, and how to wire your own client to it.

TL;DR

Jo4 exposes an MCP server at https://jo4-api.jo4.io/mcp. It's a remote, OAuth-protected MCP server — not a local stdio one. So:

Your client discovers the auth server at /.well-known/oauth-authorization-server.
Your client registers itself dynamically (no human in the loop).
Your client does an OAuth authorization-code + PKCE flow.
Your client gets a bearer token and starts calling MCP tools like jo4_shorten_url.

That's the whole song. The rest of this post is the harmonies.

What I Thought I Was Building vs. What MCP Actually Required

I had a mental model that went "add a few tool annotations to my service classes, point Claude at the URL, ship a tweet."

I had not read the MCP authorization spec. The MCP spec defers to a small constellation of OAuth RFCs:

RFC	What it gives you	Why MCP needs it
6749	OAuth 2.0 core (auth code, refresh)	Standard token issuance
7591	Dynamic Client Registration	MCP clients have no human to fill out a portal
7636	PKCE	Public clients can't keep secrets
8414	Authorization Server Metadata	Clients discover endpoints, not config them
9728	Protected Resource Metadata	Resources point at their own auth server

A "remote MCP server" is mostly "a resource server with these five RFCs implemented correctly." The MCP-specific bits (tool registration, transport, JSON-RPC framing) take maybe 20% of the effort. OAuth eats the rest.

If your backend already runs a hand-rolled OAuth server, congratulations — you're 80% of the way there. If it doesn't, you're about to learn a lot about RFC 7591.

Anatomy of the Shipped Code

Roughly the new files:

io/jo4/jo4/mcp/
├── Jo4McpTools.java                    -- @McpTool methods (read + write)
├── Jo4McpTransportConfig.java          -- transport bean override
└── OAuthMcpContextExtractor.java       -- carries auth across reactor threads

io/jo4/jo4/controller/noauth/
├── OAuthDcrController.java             -- POST /oauth/register (RFC 7591)
└── OAuthMetadataController.java        -- GET /.well-known/* (RFC 8414, 9728)

io/jo4/jo4/security/
├── DcrRateLimiter.java                 -- per-IP DCR rate limiting
├── OAuthScopeEnforcementFilter.java    -- read vs. write scope gating
├── RedirectUriValidator.java           -- https/localhost only
└── ResourceIndicatorValidator.java     -- RFC 8707

Plus a couple of Liquibase changesets adding audience to access tokens and DCR fields to clients. About 2,700 net lines, mostly tests. (You will write a lot of tests for OAuth code. Please write a lot of tests for OAuth code.)

The Four RFCs That Bit Me

1. RFC 7591 — Dynamic Client Registration

The thing I most wanted not to implement and absolutely had to.

Picture an MCP client like Claude. It doesn't have a developer to log into your portal and click "Create Client". It needs to register itself, get a client_id, and start an OAuth flow — all in a single uninterrupted dance.

The endpoint is POST /oauth/register. The body is the metadata of the client being created (redirect URIs, grant types, token auth method). The response is the same body plus a server-issued client_id (and client_secret if it's confidential).

Two things tripped me up:

token_endpoint_auth_method: "none" is required for public clients. Native and SPA-style MCP clients can't keep a secret, so they register as public clients with none. Your token endpoint must accept tokenless requests for these. PKCE is what protects them — without code_challenge, the flow must reject.

Open registration means rate limiting. Anyone on the internet can POST to /oauth/register. I added a sliding-window per-IP limiter (DcrRateLimiter) gated behind a config flag. Without it you have a delightful little DDoS surface on day one.

2. RFC 8414 — Authorization Server Metadata

A small controller serving JSON at /.well-known/oauth-authorization-server:

{
  "issuer": "https://jo4-api.jo4.io",
  "authorization_endpoint": "https://jo4-api.jo4.io/oauth/authorize",
  "token_endpoint": "https://jo4-api.jo4.io/oauth/token",
  "registration_endpoint": "https://jo4-api.jo4.io/oauth/register",
  "code_challenge_methods_supported": ["S256"],
  "scopes_supported": ["read", "write"],
  "grant_types_supported": ["authorization_code", "refresh_token"]
}

code_challenge_methods_supported: ["S256"] is non-negotiable for MCP — plain PKCE is forbidden by the MCP authorization spec. List S256. Only S256.

3. RFC 9728 — Protected Resource Metadata

This one I almost skipped because it felt redundant. It's not.

The MCP client has a resource URL (your /mcp endpoint). Before doing anything, it asks the resource: "who issues your tokens?" The answer comes from /.well-known/oauth-protected-resource:

{
  "resource": "https://jo4-api.jo4.io",
  "authorization_servers": ["https://jo4-api.jo4.io"],
  "scopes_supported": ["read", "write"],
  "bearer_methods_supported": ["header"]
}

In our case the resource and the AS are the same Spring app, so the URLs match. When mcp.jo4.io becomes its own subdomain in front of the same backend, only the resource field changes. That's the whole point of having a separate document.

4. RFC 8707 — Resource Indicators

Small but sharp. Tokens issued for the MCP resource carry an aud (audience) claim pinning them to that resource. A token your dashboard issued for browser API calls cannot be replayed at /mcp, and vice versa. A ResourceIndicatorValidator enforces it on both authorize and token endpoints.

If you skip this, a token issued for one purpose works for any purpose. That's fine until the day it isn't.

The MCP-Specific Bit (Finally)

Once OAuth is sorted, the MCP layer is small. We use Spring AI's MCP server starter. The tools look like this:

@McpTool(
    name = "jo4_shorten_url",
    description = "Create a new shortened jo4 link..."
)
public Map<String, Object> shortenUrl(
        McpSyncServerExchange exchange,
        @McpToolParam(description = "Destination URL", required = true) String longUrl,
        @McpToolParam(description = "Optional title", required = false) String title,
        @McpToolParam(description = "Optional custom slug", required = false) String customSlug,
        ...) {
    OAuthTokenAuthentication auth = requireWriteScope(exchange);
    UrlEntity entity = UrlEntity.builder()
        .longUrl(longUrl).title(title).shortUrl(customSlug).build();
    return urlSummary(urlService.createUrl(entity, auth.getUser().getId(),
                                            auth.getUser().getTenantId()));
}

Six tools shipped: jo4_get_url, jo4_list_my_urls, jo4_shorten_url, jo4_update_url, jo4_delete_url, jo4_get_stats. They delegate to the same UrlService the REST controllers use. Tenant isolation is automatic because the OAuth token already carries the user.

The thread-local trap

The non-obvious bit: the Spring AI MCP transport hands tool execution off to a Reactor boundedElastic thread. Spring Security's SecurityContextHolder is a thread-local. By the time your @McpTool method runs, the security context is empty.

The fix is a small McpTransportContextExtractor that snapshots the auth on the servlet thread and stuffs it into MCP's transport context (which does propagate):

@Override
public McpTransportContext extract(ServerRequest request) {
    Authentication auth = SecurityContextHolder.getContext().getAuthentication();
    if (auth instanceof OAuthTokenAuthentication oauthAuth) {
        return McpTransportContext.create(Map.of(KEY_OAUTH_AUTH, oauthAuth));
    }
    return McpTransportContext.EMPTY;
}

Then in tool methods: exchange.transportContext().get(KEY_OAUTH_AUTH). Took an evening to track down the first time a tool returned null for userId.

How To Connect Your Own Client (The Tutorial Bit)

Say you're writing your own MCP client (or you want to test the flow with curl).

Step 1 — Discover

curl https://jo4-api.jo4.io/.well-known/oauth-protected-resource
curl https://jo4-api.jo4.io/.well-known/oauth-authorization-server

The first tells you which AS issues tokens for the resource. The second tells you that AS's endpoints. A real client chains them.

Step 2 — Register dynamically

curl -X POST https://jo4-api.jo4.io/oauth/register \
  -H 'Content-Type: application/json' \
  -d '{
    "client_name": "my-mcp-client",
    "redirect_uris": ["http://localhost:8765/callback"],
    "grant_types": ["authorization_code", "refresh_token"],
    "response_types": ["code"],
    "token_endpoint_auth_method": "none",
    "application_type": "native",
    "scope": "read write"
  }'

You get back a client_id. Keep it.

localhost redirects are accepted (per the MCP spec's loopback exception). Anything else must be https://. No http:// to non-loopback hosts.

Step 3 — PKCE authorization

Build a verifier and challenge:

VERIFIER=$(openssl rand -base64 32 | tr -d '=+/' | cut -c -43)
CHALLENGE=$(printf '%s' "$VERIFIER" | openssl dgst -sha256 -binary | \
            base64 | tr -d '=+/' | tr '/+' '_-')

Open the user's browser at:

https://jo4-api.jo4.io/oauth/authorize
  ?response_type=code
  &client_id=YOUR_CLIENT_ID
  &redirect_uri=http://localhost:8765/callback
  &scope=read%20write
  &code_challenge=$CHALLENGE
  &code_challenge_method=S256
  &resource=https://jo4-api.jo4.io
  &state=xyz

The user logs in. Your callback receives ?code=.... Exchange it:

curl -X POST https://jo4-api.jo4.io/oauth/token \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "grant_type=authorization_code" \
  -d "code=THE_CODE" \
  -d "redirect_uri=http://localhost:8765/callback" \
  -d "client_id=YOUR_CLIENT_ID" \
  -d "code_verifier=$VERIFIER" \
  -d "resource=https://jo4-api.jo4.io"

You get an access_token (plus refresh_token if you asked for offline access).

Step 4 — Call the MCP server

The MCP endpoint is https://jo4-api.jo4.io/mcp. Streamable HTTP transport. Standard JSON-RPC 2.0. Use whichever MCP SDK matches your language; from a real MCP client this is one config line:

{
  "mcpServers": {
    "jo4": {
      "url": "https://jo4-api.jo4.io/mcp",
      "transport": "streamable-http"
    }
  }
}

Most clients with built-in OAuth support will run the discovery + DCR + PKCE dance for you the first time the server is added. You stop seeing OAuth and start seeing tools.

Step 5 — Try a tool

From a Claude routine, after the OAuth handshake, this just works:

Behind the scenes the client invokes jo4_shorten_url with longUrl and customSlug. Jo4 returns the new short URL. The routine continues on its merry way.

For analytics:

That's jo4_get_stats with slug=spring-launch and a 7-day window. Routine reads the breakdown, summarizes, done.

Things I'd Tell My Past Self

Read the MCP authorization spec end-to-end before writing a line of code. It cross-references five RFCs and you need all of them. Skim, then re-read.

Treat DCR as a public endpoint from minute one. Rate limiting and validation aren't day-two concerns. Anyone can POST /oauth/register against your prod server the moment it's live.

Audience-bind your tokens. Tokens minted for one resource leaking into another resource is exactly the class of bug RFC 8707 exists to prevent. Just enforce it.

Test the IDOR path. When tools accept slugs by string, every tool needs an explicit "does this user own this slug" check that returns NOT_FOUND (not FORBIDDEN — never reveal that the slug exists for someone else). I caught this in code review on jo4_get_stats. You don't want to catch it later.

Watch out for thread-locals on async transport. If you read auth from SecurityContextHolder inside a tool method, you'll get null and won't know why. Use the transport context.

What's Next

Right now we ship six tools — read and write for the URL primitive plus stats. Next on the list: bio-page tools (jo4_create_bio_page, jo4_add_bio_link) and webhook-management tools so a routine can subscribe itself to click events.

The OAuth scaffolding is done. New tools are now a one-method-and-a-test affair.

Building an MCP server for your own SaaS? Drop the question that's stuck — happy to compare notes on whichever RFC is currently ruining your evening.

Building jo4.io — a URL shortener with analytics, now reachable from your favorite MCP-aware AI.

From Polling to SSE in RTK Query Without Breaking Reconnect Logic

Anand Rathnas — Wed, 26 Aug 2026 02:27:22 +0000

We had a React notification bell that polled /notifications/unread-count every 30 seconds. It worked. It also sent thousands of pointless requests when the answer was "still zero, still zero, still zero." We replaced it with Server-Sent Events.

The pattern looks small from the outside — open a stream, listen for events, update state. The reality has more sharp edges than I expected. Here's the implementation that survived production.

The TL;DR

Native browser EventSource can't send custom headers, so it can't carry an Authorization Bearer token. Use @microsoft/fetch-event-source instead.
The bell still does a one-shot bootstrap fetch via RTK Query so first paint isn't blank. SSE drives subsequent updates.
SSE updates are pushed into the same RTK Query cache the bootstrap fetch wrote into, so every reader of useGetUnreadCountQuery updates without prop drilling.
Reconnect with exponential backoff. Force logout on 401. Survive tab backgrounding.

The Component

const SSE_ENABLED = import.meta.env.VITE_USE_SSE_NOTIFICATIONS === 'true';
const POLL_INTERVAL_FALLBACK_MS = 30_000;

export function NotificationBell() {
  const dispatch = useAppDispatch();

  // Bootstrap: one-shot for first paint. SSE drives subsequent updates when on,
  // RTK Query polls when off — easy kill switch via env var.
  const { data: unreadData } = useGetUnreadCountQuery(
    undefined,
    SSE_ENABLED ? undefined : { pollingInterval: POLL_INTERVAL_FALLBACK_MS },
  );

  const streamRef = useRef<{ close: () => void } | null>(null);

  useEffect(() => {
    if (!SSE_ENABLED) return;

    const apiBase = import.meta.env.VITE_API_BASE ?? '';
    streamRef.current = subscribeToNotifications(apiBase, (count) => {
      // Patch RTK Query's cache so every reader of useGetUnreadCountQuery
      // updates, not just this component.
      dispatch(
        notificationApi.util.updateQueryData(
          'getUnreadCount',
          undefined,
          (draft) => {
            if (draft?.response) {
              draft.response.count = count;
            }
          },
        ),
      );
    });

    return () => {
      streamRef.current?.close();
      streamRef.current = null;
    };
  }, [dispatch]);

  const unreadCount = unreadData?.response?.count ?? 0;
  // ... render bell ...
}

A few choices to call out:

Feature flag with a polling fallback. SSE_ENABLED is an env var. When it's off, the same component falls back to RTK Query's pollingInterval. This is the kill switch you'll thank yourself for the first time SSE behaves badly behind a customer's corporate proxy.

updateQueryData, not local state. The bootstrap fetch and the SSE updates write into the same RTK Query cache key. Any other component that calls useGetUnreadCountQuery — a sidebar, a modal — re-renders automatically when the SSE stream pushes a new count. Local component state would have meant prop drilling or a parallel state tree.

streamRef for the close handle. The cleanup callback closes the stream on unmount. The ref pattern is necessary because the stream object is created inside useEffect but referenced from the cleanup closure.

The Subscribe Function

This is the gnarly part. EventSource is the native browser API, but it's a non-starter:

That's a hard limit — you can't pass an Authorization: Bearer <jwt> header. There's a workaround (cookie auth + withCredentials) but we're not using cookies for our API. So: @microsoft/fetch-event-source, which is a fetch-based polyfill that supports headers and exposes lifecycle hooks.

import { fetchEventSource, type EventSourceMessage }
  from '@microsoft/fetch-event-source';
import { getAccessToken, forceLogout } from '../../utils/authTokenManager';

const RECONNECT_INITIAL_DELAY_MS = 1000;
const RECONNECT_MAX_DELAY_MS = 30_000;
const RECONNECT_BACKOFF_FACTOR = 2;

class FatalAuthError extends Error {
  constructor() {
    super('Auth failed for SSE stream');
    this.name = 'FatalAuthError';
  }
}

export function subscribeToNotifications(
  apiBase: string,
  onUnreadCount: (count: number) => void,
): { close: () => void } {
  const controller = new AbortController();
  let closed = false;
  let reconnectDelay = RECONNECT_INITIAL_DELAY_MS;
  const url = `${apiBase}/api/v1/protected/notifications/stream`;

  const handleMessage = (msg: EventSourceMessage) => {
    if (msg.event !== 'unread-count' || !msg.data) return;
    try {
      const payload = JSON.parse(msg.data);
      if (typeof payload.count === 'number') {
        onUnreadCount(payload.count);
        // Successful event = healthy connection; reset backoff.
        reconnectDelay = RECONNECT_INITIAL_DELAY_MS;
      }
    } catch (err) {
      console.warn('[notificationStream] malformed event payload', err);
    }
  };

  const connect = async () => {
    if (closed) return;
    const token = await getAccessToken();
    if (!token) {
      // Auth state hasn't settled yet; retry after delay.
      scheduleReconnect();
      return;
    }

    try {
      await fetchEventSource(url, {
        signal: controller.signal,
        headers: {
          Authorization: `Bearer ${token}`,
          Accept: 'text/event-stream',
        },
        // Critical: don't reload on tab visibility changes (default behavior).
        openWhenHidden: true,
        async onopen(response) {
          if (response.status === 401 || response.status === 403) {
            throw new FatalAuthError();
          }
          if (!response.ok) {
            throw new Error(`SSE handshake failed: ${response.status}`);
          }
        },
        onmessage: handleMessage,
        onclose() {
          // Server closed the stream (e.g., 30-min lifetime cap). Reconnect.
          if (!closed) scheduleReconnect();
        },
        onerror(err) {
          if (err instanceof FatalAuthError) {
            closed = true;
            forceLogout();
            throw err; // stop fetchEventSource's retry loop
          }
          // Network blip — bail out and let our catch handle reconnect.
          throw err;
        },
      });
    } catch (err) {
      if (err instanceof FatalAuthError || closed) return;
      console.warn('[notificationStream] disconnected, reconnecting', err);
      scheduleReconnect();
    }
  };

  const scheduleReconnect = () => {
    if (closed) return;
    const delay = reconnectDelay;
    reconnectDelay = Math.min(reconnectDelay * RECONNECT_BACKOFF_FACTOR,
      RECONNECT_MAX_DELAY_MS);
    setTimeout(() => { void connect(); }, delay);
  };

  void connect();

  return {
    close() {
      closed = true;
      controller.abort();
    },
  };
}

The Sharp Edges

openWhenHidden: true. The default behavior of fetch-event-source is to close the stream when the tab is backgrounded and reopen on focus. This sounds reasonable until you realize that's exactly when you most want the connection to stay alive — so the unread count is up to date when the user comes back. Set it to true.

401/403 = hard stop, not retry. A network blip is recoverable. An auth failure is not — the token is bad, retrying every second won't fix it. We throw a tagged FatalAuthError, force-logout, and break out of the retry loop. Without this, a stale token caused our SSE client to hammer the server with retries forever.

Reset backoff on successful event. Without this, a connection that hiccups, reconnects, runs fine for an hour, then hiccups again would start the second reconnect at the previous attempt's max delay. Resetting backoff on every successful event means each disconnect starts fresh.

Server onclose is normal. Our backend caps stream lifetime at 30 minutes (so JWT expiry is handled by reconnect). The client should treat onclose as routine: schedule reconnect, log nothing scary. Don't surface it as an error.

AbortController for cleanup. fetchEventSource doesn't expose a "stop" method; the only way to terminate it is via the signal. Wire up controller.abort() in the close handler.

Cache Patching, Not Local State

The single most useful pattern in this whole thing:

dispatch(
  notificationApi.util.updateQueryData('getUnreadCount', undefined, (draft) => {
    if (draft?.response) {
      draft.response.count = count;
    }
  }),
);

updateQueryData is RTK Query's escape hatch for updating cached data outside of normal query/mutation flow. It mutates the cache as if the query had refetched and gotten the new value. Every component subscribed to useGetUnreadCountQuery re-renders. No new endpoint, no new slice, no prop drilling.

This is the same primitive RTK Query exposes for optimistic updates. Repurposing it for "we got a server-pushed update" is uncontroversial and pays off the moment a second component (a sidebar count, a modal indicator) needs the same data.

What We Didn't Do

A few patterns we considered and skipped:

WebSockets. Overkill for one-way server-to-client updates. SSE is simpler, has built-in reconnect semantics in browsers, and works through every HTTP intermediary that supports Transfer-Encoding: chunked.
Long polling. Same byte cost as SSE in steady state, more complicated state machine, no native reconnect helper.
Mobile SSE. Mobile apps use FCM/APNs push for real-time, plus an on-resume REST call for the unread count when the app foregrounds. JS runtimes on mobile suspend when backgrounded, which makes long-lived SSE unworkable.

Lessons Learned

EventSource can't send headers. If you do bearer-token auth, you need a fetch-based SSE client (we use @microsoft/fetch-event-source).
openWhenHidden: true is almost always what you want. The default of "close on tab hide" is exactly backwards for an idle long-lived stream.
Treat 401/403 differently from network errors. Auth failures are unrecoverable from the client's perspective — force logout, stop retrying. Network blips are recoverable — exponential backoff.
Reset backoff on successful events, not just on successful reconnects. A long-running stream that hiccups should re-enter retry at the initial delay, not the previous attempt's max delay.
Server-side stream lifetime caps are a feature, not a bug. They give you a clean reconnect boundary that handles mid-stream JWT expiry without any explicit re-validation logic.
RTK Query's updateQueryData is the right place to put server-pushed updates. Cache patches propagate to every subscriber. Local component state forces prop drilling.
Always make new transports feature-flagged with a polling fallback. First time SSE behaves badly behind a customer's corporate proxy, you'll flip the flag and forget about it.

Have you migrated polling to SSE in production? What's the gotcha that bit you? Drop it in the comments.

Building jo4.io — a URL shortener whose real-time updates work the way the user expects, even when the network does not.

Server-Sent Events Across Multiple Pods: Redis Pub/Sub + Spring WebFlux

Anand Rathnas — Mon, 24 Aug 2026 02:25:05 +0000

We replaced our notification-bell polling with Server-Sent Events. On a single pod, this is easy — Spring WebFlux + a Flux of events, done in 30 lines. On multiple pods, it gets interesting fast: a write that happens on pod A needs to reach a subscriber on pod B, and naively, it doesn't.

Here's the full pattern: how the SSE endpoint is shaped, how Redis pub/sub fans events across pods, and the subtle ordering and lifecycle gotchas we hit along the way.

The Endpoint

A /notifications/stream SSE endpoint that emits the current unread count on subscribe, then live updates whenever the count changes:

@GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public ResponseEntity<Flux<ServerSentEvent<NotificationStreamEvent>>>
        streamNotifications() {
    authContext.requireScope("read");
    Long userId = authContext.getCurrentUser().getId();

    ServerSentEvent<NotificationStreamEvent> initial = ServerSentEvent
        .<NotificationStreamEvent>builder()
        .event("unread-count")
        .data(NotificationStreamEvent.unreadCount(
            notificationService.getUnreadCount(userId)))
        .build();

    Flux<ServerSentEvent<NotificationStreamEvent>> events =
        notificationEventBus.subscribe(userId)
            .map(e -> ServerSentEvent.<NotificationStreamEvent>builder()
                .event(e.type()).data(e).build());

    Flux<ServerSentEvent<NotificationStreamEvent>> heartbeat =
        Flux.interval(Duration.ofSeconds(30))
            .map(t -> ServerSentEvent.<NotificationStreamEvent>builder()
                .comment("ping").build());

    Flux<ServerSentEvent<NotificationStreamEvent>> body = Flux.concat(
            Flux.just(initial),
            Flux.merge(events, heartbeat))
        .take(Duration.ofMinutes(30));

    HttpHeaders headers = apiHeaders();
    headers.set(HttpHeaders.CACHE_CONTROL, "no-cache");
    headers.set("X-Accel-Buffering", "no");
    return ResponseEntity.ok().headers(headers).body(body);
}

A few things worth highlighting:

Initial emit on subscribe. The first ServerSentEvent carries the current count synchronously. Clients get the answer immediately and don't need a separate REST round-trip for first paint.

Heartbeat every 30 seconds. SSE looks idle to TCP intermediaries. Comment events (:ping) keep the connection alive through any proxy/CDN idle timeout.

X-Accel-Buffering: no. Reverse proxies (nginx and friends) buffer responses by default. SSE is point-the-other-way — buffering means events sit in the proxy until the buffer fills, defeating the entire stream. This header is a hint to skip buffering for this response.

take(Duration.ofMinutes(30)). Server-initiated close. The stream lives for 30 minutes, then the server hangs up. The client reconnects with a fresh JWT. This is how we handle mid-stream token expiry without inventing our own re-validation logic — close-and-reconnect is the mechanism.

The Per-User Sink

The notificationEventBus.subscribe(userId) call lazily creates a per-user Sinks.Many<NotificationStreamEvent> and returns its Flux. New subscribers latch onto the same sink; the sink is removed when the last subscriber unsubscribes:

public Flux<NotificationStreamEvent> subscribe(Long userId) {
    return Flux.defer(() -> {
        Sinks.Many<NotificationStreamEvent> sink = sinks.computeIfAbsent(userId,
            k -> Sinks.many().multicast().directBestEffort());
        return sink.asFlux();
    }).doFinally(signal -> cleanupIfEmpty(userId));
}

private void cleanupIfEmpty(Long userId) {
    sinks.computeIfPresent(userId, (k, sink) ->
        sink.currentSubscriberCount() == 0 ? null : sink);
}

Two non-obvious choices:

Flux.defer instead of Flux.create or eager construction. Without defer, this race fires:

subscribe(userId) returns a Flux referencing a sink.
Before the downstream attaches, the previous subscriber for that user unsubscribes.
cleanupIfEmpty removes the sink from the map.
A publish arrives, looks up the sink in the map, finds nothing, drops the event.
The downstream finally attaches — to the now-orphaned sink.

Flux.defer rebuilds the lookup at the moment downstream attaches, which makes the computeIfAbsent/attach pair effectively atomic from the publisher's perspective.

directBestEffort backpressure. A slow subscriber drops events rather than terminating the stream. For unread-count specifically, this is correct: the latest count replaces any dropped one, so dropping is harmless.

The Cross-Pod Problem

So far so good on one pod. The break comes the moment you scale horizontally.

Pod A holds the SSE connection for User 42. A write on pod B increments User 42's unread count and publishes to pod B's local sink map. Pod A's sink map has no idea anything happened. The user sees nothing change.

The fix is Redis pub/sub. Every pod subscribes to a Redis pattern; every write fans out to that pattern; every pod with a local subscriber for the affected user emits to its local sink.

@Component
public class RedisNotificationEventBus implements NotificationEventBus, MessageListener {

    public static final String CHANNEL_PREFIX = "jo4:notify:user:";
    public static final String CHANNEL_PATTERN = CHANNEL_PREFIX + "*";

    private final ConcurrentHashMap<Long, Sinks.Many<NotificationStreamEvent>> sinks
        = new ConcurrentHashMap<>();

    @Override
    public void publish(Long userId, NotificationStreamEvent event) {
        String channel = CHANNEL_PREFIX + userId;
        try {
            String payload = objectMapper.writeValueAsString(event);
            redisTemplate.convertAndSend(channel, payload);
        } catch (Exception e) {
            log.warn("Redis publish failed for userId={}; falling back to local emit",
                userId, e);
            emitLocally(userId, event);
        }
    }

    @Override
    public void onMessage(Message message, byte[] pattern) {
        try {
            String channel = new String(message.getChannel(), StandardCharsets.UTF_8);
            if (!channel.startsWith(CHANNEL_PREFIX)) return;
            Long userId = Long.parseLong(channel.substring(CHANNEL_PREFIX.length()));
            NotificationStreamEvent event = objectMapper.readValue(
                message.getBody(), NotificationStreamEvent.class);
            emitLocally(userId, event);
        } catch (Exception e) {
            // Never throw from MessageListener — would tear down the listener container.
            log.error("Failed to handle pub/sub on channel={}", message.getChannel(), e);
        }
    }
}

The trick:

Publishes go to Redis only. They don't touch the local sinks directly. Even pod A's own writes route through Redis, which means every replica's local sinks get updated through one consistent path (no "but we wrote it locally" branch to maintain).
Subscribes happen on every pod, via MessageListener. The pattern subscription is wired in a @Configuration class (MessageListenerContainer.addMessageListener(bus, new PatternTopic("jo4:notify:user:*"))).
Routing is by channel suffix. Channel jo4:notify:user:42 carries events for user 42; every pod that has a local sink for 42 emits, every pod that doesn't ignores the message. No global broadcast to every subscriber.

The Fallback to Local

Notice this branch:

} catch (Exception e) {
    log.warn("Redis publish failed for userId={}; falling back to local emit",
        userId, e);
    emitLocally(userId, event);
}

Redis pub/sub is fire-and-forget and the connection can die. If the publish fails, we still emit to the local sink so users on the current pod don't miss the event. Users on other pods miss it during the outage, but that's strictly better than everyone missing it.

The Transactional-Listener Bridge

How do events actually get into bus.publish(...)? Not from the controller — the controller is a passive subscriber. They come from the rest of the application, via Spring's transactional event listener:

@Component
@RequiredArgsConstructor
public class UnreadCountStreamPublisher {

    private final NotificationService notificationService;
    private final UnreadCountCacheService cacheService;
    private final NotificationEventBus eventBus;

    @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
    public void onUnreadCountChanged(UnreadCountChangedEvent event) {
        Long userId = event.userId();
        try {
            long count = notificationService.getUnreadCountFresh(userId);
            cacheService.evict(userId);
            eventBus.publish(userId, NotificationStreamEvent.unreadCount(count));
        } catch (Exception e) {
            // Listener must not throw — would mark the just-committed transaction
            // as failed in some configurations and disrupt unrelated handlers.
            log.error("Failed to publish unread-count change for userId={}",
                userId, e);
        }
    }
}

Three things this is doing right:

AFTER_COMMIT, not BEFORE_COMMIT. The SSE event must not fire until the DB write commits. Otherwise the client could see a count that's about to be rolled back.

Fresh DB read, not the cached value. After a write, the cache may briefly hold the pre-write count. Reading fresh from the DB is the only safe authority for what we're publishing. Then we evict the cache (rather than write-through it) so the next reader hits the DB and re-populates.

Catch-all that never throws. A @TransactionalEventListener that throws from AFTER_COMMIT can mark the just-committed transaction as failed in some configurations. Listeners must be defensive.

The application code that mutates unread counts publishes a UnreadCountChangedEvent and moves on. It doesn't know about Redis, doesn't know about SSE, doesn't know about cache eviction. All of that is downstream of the event.

Lessons Learned

A single-pod SSE stream is simple. A multi-pod SSE stream with shared subscribers needs a fan-out plane. Redis pub/sub is the cheapest one that works.
Always route publishes through the same path, even local ones. "Publish locally + publish to Redis" creates double-delivery possibilities that "publish to Redis only, subscribe from everywhere" avoids by construction.
Backpressure is a product decision. For unread counts, dropping is correct because newer values supersede older ones. For chat messages, dropping is incorrect. Pick directBestEffort, buffer, or failFast per use case.
Flux.defer for any sink that may be created/destroyed by lifecycle. Eager Flux construction races against unsubscribe-then-subscribe sequences in subtle ways.
Server-initiated close is the right way to handle JWT expiry on long-lived streams. Hang up after a fixed lifetime; the client reconnects with a fresh token. No mid-stream auth gymnastics.
MessageListener callbacks must never throw. A thrown exception tears down the listener container and your app stops receiving events with no obvious symptom.
AFTER_COMMIT is the only safe phase for "broadcast that the data changed" listeners. Anything earlier risks broadcasting a value that's about to roll back.

How did you scale your real-time stream beyond one process? Pub/sub, sticky sessions, websocket cluster? Drop your pattern in the comments.

Building jo4.io — a URL shortener with real-time updates that work the same on one pod or twenty.

We Were Hitting Auth0 /userinfo on Every Request. Here's the Fix.

Anand Rathnas — Fri, 21 Aug 2026 02:25:31 +0000

If you're using Auth0 as an OIDC provider for a Spring backend, here's a question worth asking right now: what fields does your backend assume are in the access token, but aren't actually there by default?

For us, the answer was email. And the default behavior, when our user-sync filter found no email claim, was to fall back to a synchronous HTTP call to Auth0's /userinfo endpoint. Per request. With caching, but not enough caching.

Once we noticed, the fix took an afternoon. Here's the whole story.

The Default Auth0 Access Token

Auth0 access tokens are minimal by design. Out of the box, an access token's claims look something like:

{
  "iss": "https://your-tenant.auth0.com/",
  "sub": "auth0|abc123",
  "aud": "https://your-api/",
  "iat": 1716950000,
  "exp": 1716953600,
  "scope": "openid profile email"
}

Notice what's not there: email, name, picture, email_verified. Those live in the ID token by default, or come back from a separate /userinfo call.

Most tutorials encourage you to use the access token for authorization (verify signature, check scopes) and the ID token for user identity. Fine for a SPA. Awkward for a Spring backend that only sees a single bearer token from Authorization: Bearer <jwt> and doesn't know which one it is.

We took a tempting shortcut: assume the access token has email (because some tenants are configured to inline it), and if it doesn't, fall back to /userinfo to grab the rest of the profile.

What That Looked Like

String email = jwt.getClaimAsString("email");
String firstName = jwt.getClaimAsString("given_name");
String lastName = jwt.getClaimAsString("family_name");
String picture = jwt.getClaimAsString("picture");

if (StringUtils.isBlank(email)) {
    log.warn("Email claim missing from JWT, falling back to /userinfo");
    Auth0UserInfo userInfo = fetchUserInfoFromAuth0(jwt.getTokenValue());
    email = userInfo.email;
    if (StringUtils.isBlank(firstName)) firstName = userInfo.givenName;
    if (StringUtils.isBlank(lastName)) lastName = userInfo.familyName;
    if (StringUtils.isBlank(picture)) picture = userInfo.picture;
}

The fallback was wrapped in a Redis cache plus a sync lock, so on a hot path most requests hit cache and skipped the HTTP call. But:

Cache misses on cold start, after deploys, after Redis restarts, after cache evictions.
New users always missed the cache by definition.
The mobile app's startup fan-out (3-5 parallel requests) thundered against the lock, and at least one of them did the actual fetch.

When we measured: a single user could trigger several /userinfo calls per session. Multiplied across the user base and the rate of token rotations, that's a non-trivial dependency on Auth0 staying fast and available — for every API request to our own service.

The Right Fix Is on the Auth0 Side

The clean answer is to never need /userinfo at request time. Put the claims you need into the access token itself, at issue time. Auth0 supports this via Post-Login Actions.

A Post-Login Action runs once per login, on Auth0's side, and can mutate the access token before issuance. The Action that fixed our problem looks like this (Auth0 Action JavaScript):

exports.onExecutePostLogin = async (event, api) => {
  const namespace = 'https://jo4.io/';
  api.accessToken.setCustomClaim(`${namespace}email`,
    event.user.email);
  api.accessToken.setCustomClaim(`${namespace}email_verified`,
    event.user.email_verified);
  api.accessToken.setCustomClaim(`${namespace}given_name`,
    event.user.given_name);
  api.accessToken.setCustomClaim(`${namespace}family_name`,
    event.user.family_name);
  api.accessToken.setCustomClaim(`${namespace}picture`,
    event.user.picture);
};

Two important details about that namespace prefix.

Auth0 strips non-namespaced custom claims. If you setCustomClaim('email', ...), Auth0 silently drops it because it conflicts with the OIDC reserved name. Custom claims must be namespaced URIs. We used https://jo4.io/, but it's a symbolic URI — there's no requirement Auth0 actually fetch from it. It just needs to look like a URI.

Match the namespace exactly on the backend. We made the namespace configurable so the Action and the filter can't drift:

@Value("${app.auth0.email-claim-key:https://jo4.io/email}")
private String emailClaimKey;

@Value("${app.auth0.given-name-claim-key:https://jo4.io/given_name}")
private String givenNameClaimKey;

@Value("${app.auth0.picture-claim-key:https://jo4.io/picture}")
private String pictureClaimKey;

And read from those keys directly:

String email = jwt.getClaimAsString(emailClaimKey);
String firstName = jwt.getClaimAsString(givenNameClaimKey);
String lastName = jwt.getClaimAsString(familyNameClaimKey);
String picture = jwt.getClaimAsString(pictureClaimKey);

No fallback. No /userinfo call. The claims either arrive in the access token or they don't, and if they don't, there's a real configuration bug to fix — not a symptom to paper over.

The Graceful Degradation

That said, "no fallback" is too brittle for a deploy where the Action ships before, after, or in the middle of the backend rollout. We added one bounded escape hatch: if the email claim is missing and the user already exists in our DB, log a loud warning and proceed with the existing record:

if (StringUtils.isBlank(email)) {
    Optional<UserEntity> existing = userService.findByExternalAuthId(auth0Id);
    if (existing.isPresent()) {
        log.warn("Email claim '{}' missing from JWT for auth0Id={}; "
            + "using existing DB record. Verify Auth0 Post-Login Action "
            + "sets this custom claim.", emailClaimKey, auth0Id);
        userSyncCacheService.markUserSynced(auth0Id, existing.get().getEmail());
        return existing.get();
    }
    log.error("Email claim '{}' missing from JWT and user not found "
        + "by auth0Id={}. Auth0 Post-Login Action must set this custom claim.",
        emailClaimKey, auth0Id);
    throw new IllegalStateException(
        "Email claim missing from JWT — Auth0 Post-Login Action misconfigured");
}

If the Action gets disabled accidentally, existing users keep working (with a noisy warning that you'll see the next time you check your logs). New signups fail fast and loud. The error message names the exact misconfiguration. Future-you will thank present-you.

The Migration Window

We also moved email_verified from the standard claim name into the namespaced one in a separate change. The migration has a wrinkle: tokens issued before the migration still have the old claim, and they don't expire instantly.

So during the window we accepted both:

Boolean emailVerified =
    jwt.getClaimAsBoolean("https://jo4.io/email_verified");
if (emailVerified == null || !emailVerified) {
    // Tmp fallback for previously issued tokens.
    // Remove this if-block after [token-rotation cutoff date].
    emailVerified = jwt.getClaimAsBoolean("email_verified");
}

We dated the cleanup in the comment. After the cutoff (when all pre-migration tokens had expired), we deleted the fallback in a separate commit. One commit to migrate, another to clean up — never the same commit, because the cleanup needs to happen on a clock not on a deploy.

Bonus: Lock Release

A subtler bug we found while in there. The original sync code acquired a Redis lock around the userinfo fetch, but on the success path the lock was never explicitly released — it just expired via TTL after 30 seconds. That meant concurrent requests for that user during the next 30 seconds would all skip the sync entirely.

The fix is the obvious one: explicit release in finally:

if (!userSyncCacheService.tryAcquireSyncLock(auth0Id)) {
    return userService.findByExternalAuthId(auth0Id).orElse(null);
}

try {
    return performSync(jwt, auth0Id);
} finally {
    userSyncCacheService.releaseSyncLock(auth0Id);
}

The release is best-effort — if Redis is briefly unavailable, the TTL still cleans up. But on the happy path, the lock evaporates the instant we're done, instead of holding everyone else off for half a minute.

Lessons Learned

Auth0 access tokens have minimal claims by default. Don't assume email or name are in there — read the actual JWT and find out.
Don't paper over missing claims with a synchronous external call. The cache-the-callout approach scales until it doesn't, and the failure mode is "everything is slow" not "auth is broken," which is harder to diagnose.
Push enrichment to the issuer. Auth0 Post-Login Actions add custom claims at issue time, once per login, with no per-request cost.
Custom claims must be URI-namespaced. Plain names like email will be silently stripped — Auth0 reserves those for OIDC. Use a URI prefix you control.
Configure the namespace, don't hardcode it. Drift between the Action and the backend is a high-impact misconfiguration; making it a single env var prevents it.
Always release distributed locks in finally. Relying on TTL cleanup means your happy path forces unnecessary contention on every other caller for the duration of the TTL.

Has your auth integration ever hidden a per-request external call you weren't aware of? Drop the story in the comments — these are always educational.

Building jo4.io — a URL shortener whose backend doesn't ask Auth0 the same question on every API call.

DigitalOcean Postgres Has a Firewall Trap. We Walked Right Into It.

Anand Rathnas — Wed, 19 Aug 2026 02:20:59 +0000

We added a database backup pipeline. The pipeline temporarily allowlists the runner IP, runs pg_dump, then revokes the IP. Standard pattern, used by basically every CI-driven DB job.

The first run did all of those things correctly. It also took down production.

Here's what nobody tells you about DigitalOcean managed Postgres.

The Setup

Our managed Postgres had no trusted-source rules at all. By default this means "open to the world" — or rather, open to anything that has the connection string and credentials. The droplet running our app talked to it from a public IP, no allowlist required.

The new pipeline added one rule:

doctl databases firewalls add "$DB_ID" --rule "ip_addr:${RUNNER_IP}"

The runner IP. Just the runner. We expected: "OK, the runner can also reach the DB now."

What actually happened: DigitalOcean flipped the database into enforcement mode the moment a single rule was added. From "anyone with credentials" to "only IPs on the allowlist." Our droplet wasn't on the allowlist — it had never needed to be. So the application instantly lost its database connection.

The backup pipeline was running. The dump succeeded. The app was 500-ing for every request the entire time.

The Quiet Bit

The semantics are documented somewhere — eventually we found a docs page that mentions trusted sources are enforced "when configured." That phrasing is doing a lot of work. "Configured" means "any rule exists." Adding the first rule is the moment your enforcement model changes from "off" to "on."

There is no warning in the dashboard. No prompt. No "are you sure?" You add a rule, the rule appears in the list, and your previously-public-but-credentialed database becomes private to that one rule. Everything else that wasn't on the list — including the production app — is now blocked.

If you remove all rules, the DB returns to open mode. So you can dig out of the hole. But during the window, all your traffic is dropped at the network layer, not at the auth layer. From the app's perspective, the database has vanished.

The Fix

Two parts: an immediate ops fix, and a permanent guard.

Immediate ops fix

Add the production droplet's IP to the allowlist before the pipeline runs again. Once your droplet is on the list, the pipeline adding another runner IP is fine — it's just one more entry in an existing allowlist, no enforcement-mode flip.

DROPLET_IP=$(doctl compute droplet list --format Name,PublicIPv4 --no-header \
  | awk -v n="${PROJECT_NAME}-server" '$1==n {print $2}')
doctl databases firewalls append "$DB_ID" --rule "ip_addr:${DROPLET_IP}"

append, not add — same effect for a single rule, but append makes it explicit you're not setting the entire firewall.

Permanent guard

The real fix is preflight in every workflow that touches the firewall. Refuse to run if the production droplet isn't already on the list:

- name: Verify droplet IP is in DB trusted-sources (preflight)
  run: |
    # Adding ANY trusted-source rule to a previously-empty list flips DO Postgres
    # into enforcement mode. If the droplet IP isn't already on the list, the prod
    # app loses DB access the moment we open the runner IP. Fail loudly here so we
    # never strand prod again — operator must add the droplet IP permanently.

    DB_ID="${{ steps.database.outputs.db_id }}"
    DROPLET_NAME="${PROJECT_NAME}-server"

    DROPLET_IP=$(doctl compute droplet list --format Name,PublicIPv4 --no-header \
      | awk -v n="$DROPLET_NAME" '$1==n {print $2}')

    if [ -z "$DROPLET_IP" ]; then
      echo "❌ Could not resolve public IP for droplet '$DROPLET_NAME'"
      exit 1
    fi

    if ! doctl databases firewalls list "$DB_ID" \
        | awk 'NR>1 && $3=="ip_addr" {print $4}' \
        | grep -qxF "$DROPLET_IP"; then
      echo "❌ Droplet IP $DROPLET_IP is NOT in DB trusted-sources for $DB_ID"
      echo ""
      echo "Add it once (operator) before re-running this workflow:"
      echo "  doctl databases firewalls append $DB_ID --rule \"ip_addr:$DROPLET_IP\""
      echo ""
      echo "Refusing to proceed — opening the runner IP without the droplet rule"
      echo "would lock prod out of the database."
      exit 1
    fi

    echo "✅ Droplet IP confirmed in trusted-sources"

This is a hard guard, not a hint. Without the droplet rule, the workflow refuses to run. The operator has to add the rule once, manually, with full knowledge of what they're doing. After that, every subsequent workflow run sails through the preflight.

We added the same preflight to every other workflow that touches the DB firewall — backup, seed, restore. One pattern, replicated everywhere we mutate trusted-sources.

The Footnote on `doctl`

While we were in there, another doctl quirk: doctl databases firewalls list doesn't support --no-header or --format. Both flags exist for other doctl subcommands, but not this one. We were parsing column 3 expecting (UUID, Type, Value) and getting back (UUID, ClusterUUID, Type, Value) because the un-suppressible header line tipped our awk indexing off by one.

# Wrong (silently parses nothing):
doctl databases firewalls list "$DB_ID" --no-header --format UUID,Type,Value \
  | awk -v ip="$RUNNER_IP" '$2=="ip_addr" && $3==ip {print $1}'

# Right:
doctl databases firewalls list "$DB_ID" \
  | awk -v ip="$RUNNER_IP" 'NR>1 && $3=="ip_addr" && $4==ip {print $1}'

Skip the header with NR>1, accept that columns are positional, and remember that doctl flag support is per-subcommand. There's no global contract.

Lessons Learned

Empty allowlist is not the same as no enforcement. A managed-DB firewall with zero rules behaves as "off." A firewall with one rule behaves as "deny by default." Adding the first rule is a state change, not an addition.
Test firewall changes against a non-prod database first if your platform doesn't have explicit "enforcement: on" / "enforcement: off" toggles. The blast radius of the first rule is the entire system.
Hard preflights beat warnings. A workflow comment that says "be sure the droplet IP is on the allowlist" is wishful thinking. A preflight that exits non-zero unless the droplet IP is on the allowlist is a control.
doctl flag support is per-subcommand. Don't assume --format works just because the next subcommand over supports it. Read the actual --help for the exact subcommand you're calling.
When ops touches network ACLs, fence both ends. Open the rule, run the work, close the rule, and preflight that the persistent rules required by prod are present before you do any of that.

Have you been bitten by an "off-by-default" cloud setting that turned on the moment you configured anything? Drop the story in the comments.

Building jo4.io — a URL shortener whose ops scripts now treat firewall rules like loaded weapons.

Why Your JPA Counter Column Keeps Resetting to Zero

Anand Rathnas — Mon, 17 Aug 2026 02:22:16 +0000

I'll cut to the chase: if you have a counter column, a @Modifying @Query that increments it, and a @Version field on the same entity, you almost certainly have a silent data-loss bug.

We had this exact bug. It cost us about two weeks of "huh, that referrer's click count looks low" before we figured out what was happening. Here's what we found and how we fixed it.

The Setup

We track referrer attribution. Every time someone clicks a referral short link, we increment a denormalized total_clicks counter on the referrer row. Doing this with a load-modify-save round-trip would be slow under fan-out, so we use a @Modifying @Query that increments at the SQL level:

@Modifying(clearAutomatically = true)
@Query("UPDATE ReferrerEntity r "
     + "SET r.totalClicks = COALESCE(r.totalClicks, 0) + 1, "
     + "    r.modifiedTime = :now "
     + "WHERE r.id = :referrerId AND r.deleted = false")
int incrementClickCount(@Param("referrerId") Long referrerId,
                        @Param("now") Long now);

Looks fine. Atomic at the DB level. Counts always go up. Until you also have this on the entity:

@Version
private Long version;

@Column(name = "total_clicks")
@Builder.Default
private Long totalClicks = 0L;

And somewhere else in the code path, an unrelated service does this:

ReferrerEntity referrer = referrerRepository.findById(id).orElseThrow();
referrer.setPayoutEmail(newEmail);
referrerRepository.save(referrer);  // <-- here be dragons

The Problem

Watch the timeline:

Step	Thread A (click handler)	Thread B (settings update)
1		`findById(id)` → loads referrer with `totalClicks=42, version=7`
2	`incrementClickCount(id)` → SQL `UPDATE`, row now `totalClicks=43, version=7`
3	`incrementClickCount(id)` → SQL `UPDATE`, row now `totalClicks=44, version=7`
4		`setPayoutEmail(...)`; `save()`

What does step 4 do? It writes back the entity Hibernate has in memory: totalClicks=42, version=7, payoutEmail=newEmail.

The @Version check passes — version is still 7, because our @Modifying query never bumped it. So Hibernate happily issues:

UPDATE referrer SET total_clicks = 42, payout_email = '...', version = 8 WHERE id = ? AND version = 7

Two clicks, gone. Silently. The version mechanism that's supposed to prevent exactly this lost-update did not prevent it, because the writes that bumped the counter never bumped the version.

The bug compounds under load. The hotter the counter, the more clicks get clobbered. Slow weeks the counter looks fine; viral campaigns it's wildly under-counted. Fun debugging.

The Two-Part Fix

Part 1: Mark counter columns `updatable = false`

The fundamental issue is that Hibernate's save() writes every column it tracks, including ones it has no business writing. A counter that's only ever incremented by @Modifying @Query is one of those columns. Tell Hibernate to leave it alone:

@Column(name = "total_clicks", updatable = false)
@Builder.Default
private Long totalClicks = 0L;

@Column(name = "total_conversions", updatable = false)
@Builder.Default
private Long totalConversions = 0L;

updatable = false tells Hibernate: this column is read-only from the entity's perspective. Don't include it in UPDATE statements. The only path that writes to it is the explicit SQL update. save() calls now leave the counter alone, no matter what value the in-memory entity has.

This is the rule we now apply across the codebase. Anywhere a column is mutated exclusively via @Modifying @Query — counters, denormalized stats, queue depths — that column gets updatable = false. Without exception.

Part 2: Bump the version inside the `@Modifying @Query`

updatable = false plugs the lost-update hole, but a different hole opens: now save() doesn't know the row changed, so optimistic-concurrency races against other fields silently overwrite each other.

Concretely: thread A bumps the counter via @Modifying. Thread B loads the entity (with old payoutEmail), updates it, saves. Without a version bump, B's save succeeds — but B was operating on a row that A had already mutated. If you care about A's mutation being a "real" change (and you should, for audit purposes), B should have hit OptimisticLockException and retried.

The fix is one extra clause in the SET:

@Modifying(clearAutomatically = true)
@Query("UPDATE ReferrerEntity r "
     + "SET r.totalClicks = COALESCE(r.totalClicks, 0) + 1, "
     + "    r.modifiedTime = :now, "
     + "    r.version = COALESCE(r.version, 0) + 1 "  // <-- this line
     + "WHERE r.id = :referrerId AND r.deleted = false")
int incrementClickCount(@Param("referrerId") Long referrerId,
                        @Param("now") Long now);

Apply this to every @Modifying @Query UPDATE on every entity that has a @Version field. We did the audit and bumped versions in around fifteen queries across ReferrerRepository, TeamRepository, UrlVariantRepository, and others. Easy fix, big footprint.

The Test That Would Have Caught It

We added a guard test that walks repository methods reflectively, finds every @Modifying @Query UPDATE, checks the JPQL string for version = in the SET clause, and fails if any are missing:

@Test
void everyModifyingQueryBumpsVersionWhenEntityHasVersion() {
    List<Method> queries = findAllModifyingQueriesOn(ReferrerRepository.class);
    for (Method m : queries) {
        Query q = m.getAnnotation(Query.class);
        String jpql = q.value();
        if (!isUpdate(jpql)) continue;
        if (entityHasVersion(targetEntity(m))) {
            assertTrue(
                jpql.replaceAll("\\s+", " ").contains("version = COALESCE("),
                m.getName() + " is missing version bump"
            );
        }
    }
}

This is the kind of test that will save you the next time someone adds a @Modifying @Query six months from now and forgets the version bump.

Lessons Learned

save() writes every column it tracks, including ones you wish it wouldn't. If a column is only mutated by direct SQL, mark it updatable = false to keep save() from silently overwriting it.
@Version only protects what the JPA layer knows changed. A @Modifying @Query that doesn't bump the version is invisible to the optimistic-lock check. Subsequent save() calls will overwrite and the version check will pass.
Add r.version = COALESCE(r.version, 0) + 1 to every @Modifying @Query UPDATE on a versioned entity. No exceptions.
Reflection-based guard tests are cheap insurance. A single test that walks repository methods catches the next regression before it merges. Worth the 100 lines.
Counter columns are a DSL of their own. They have one valid mutation pattern (atomic SQL increment) and one valid read pattern (load and trust). Anything else is a bug waiting for traffic.

Have you been bitten by JPA's "helpful" save behavior? What was the symptom that led you to it? Drop the war story in the comments.

Building jo4.io — a URL shortener whose click counters add up the same way every time.

Backups That You Don't Restore Aren't Backups: A Weekly Restore Drill

Anand Rathnas — Fri, 14 Aug 2026 03:26:42 +0000

Most teams have backups. Half of those teams have never restored one. The other half discovered the hard way that their dumps were corrupt, incomplete, or pointed at a database schema that no longer exists.

Backups you don't actively restore aren't backups — they're warm fuzzy feelings. Here's how we built a backup pipeline whose first-class citizen is the weekly restore, not the nightly dump.

The Pipeline at 30,000 Feet

Three GitHub Actions workflows:

Nightly backup — pg_dump from the production DB, gzip, upload to DigitalOcean Spaces. Runs at 03:00 UTC daily.
Restore test — pulls the latest backup, restores it into a Docker postgres:16 container in the runner, asserts table count and row counts. Runs Sunday 04:00 UTC.
Manual restore — parameterized one-click pipeline for the actual recovery flow. Same code path the restore test exercises, minus the assertions, plus the parameter for "which backup."

The clever bit isn't (1). The clever bit is (2) hard-failing if (1) silently broke.

The Backup Workflow

name: DigitalOcean-460-Backup

on:
  schedule:
    - cron: '0 3 * * *'   # 03:00 UTC daily
  workflow_dispatch:

concurrency:
  group: jo4-db-backup
  cancel-in-progress: false   # Don't cancel a running backup

env:
  DO_REGION: nyc3
  PROJECT_NAME: jo4
  SCHEMA_NAME: alertstage
  BACKUP_PREFIX: db-daily-backup

Two settings worth highlighting:

concurrency.cancel-in-progress: false — if workflow_dispatch fires while the cron is running, queue the new run, don't cancel the old one. Cancelling a half-uploaded dump leaves a corrupt object in the bucket and a confusing log trail.
Pin the pg_dump major version. PostgreSQL is forgiving about minor versions, but a pg_dump from a different major to the server is a bug factory:

- name: Install PostgreSQL client and verify version
  run: |
    sudo apt-get install -y -qq postgresql-client
    PG_DUMP_MAJOR=$(pg_dump --version | awk '{print $3}' | cut -d. -f1)
    if [ "$PG_DUMP_MAJOR" != "16" ]; then
      echo "❌ pg_dump must be 16 to match server, got: $(pg_dump --version)"
      exit 1
    fi

The Firewall Dance

Our managed Postgres is locked behind a trusted-source IP allowlist. The runner IP rotates every job. So every backup adds the runner's IP, runs the dump, and removes the IP — even if the dump fails:

- name: Open DB trusted-source for runner IP
  id: db_firewall
  run: |
    RUNNER_IP=$(curl -s https://api.ipify.org)
    DB_ID="${{ steps.database.outputs.db_id }}"
    doctl databases firewalls add "$DB_ID" --rule "ip_addr:${RUNNER_IP}"
    echo "runner_ip=$RUNNER_IP" >> $GITHUB_OUTPUT
    echo "db_id=$DB_ID"          >> $GITHUB_OUTPUT

- name: Close DB trusted-source (revoke runner IP)
  if: always()   # critical — runs even if the dump step failed
  run: |
    UUIDS=$(doctl databases firewalls list "$DB_ID" \
      | awk -v ip="$RUNNER_IP" 'NR>1 && $3=="ip_addr" && $4==ip {print $1}')
    echo "$UUIDS" | while read -r uuid; do
      [ -z "$uuid" ] && continue
      doctl databases firewalls remove "$DB_ID" "$uuid" || true
    done

if: always() is non-negotiable here. If the dump fails and we don't clean up, we leak ephemeral runner IPs into the trusted-sources list. After a few months you're looking at hundreds of stale entries.

The Dump Itself

- name: Run pg_dump
  run: |
    TS=$(date -u +%Y%m%d-%H%M%S)
    DUMP_FILE="/tmp/jo4db-${TS}.sql"

    PGPASSWORD="${{ steps.database.outputs.db_password }}" \
    PGSSLMODE=require \
    pg_dump \
      -h "${DB_HOST}" -p "${DB_PORT}" -U "${DB_USER}" -d "${DB_DATABASE}" \
      --schema=${SCHEMA_NAME} \
      --no-owner \
      --no-privileges \
      --quote-all-identifiers \
      -f "${DUMP_FILE}"

    gzip -9 "${DUMP_FILE}"

    SIZE_BYTES=$(stat -c%s "${DUMP_FILE}.gz")
    if [ "$SIZE_BYTES" -lt 1024 ]; then
      echo "❌ Dump suspiciously small ($SIZE_BYTES bytes) — aborting upload"
      exit 1
    fi

--no-owner --no-privileges — the dump is restorable on any Postgres 16+ instance, not just the one with our exact role names. Important for the restore test, which uses a fresh container.

--quote-all-identifiers — defends against case-sensitivity surprises if the restore target uses different default search paths.

The 1KB sanity floor — even an empty schema produces more than 1KB of dump (CREATE TABLE statements, comments, etc.). If the dump is smaller, something went wrong, abort before you upload garbage that overwrites yesterday's good backup.

The Lifecycle Rule (Idempotently)

We expire backups after 30 days via a bucket lifecycle rule. The catch: s3cmd setlifecycle replaces the entire bucket policy. We don't want to clobber other rules:

- name: Ensure 30-day lifecycle rule on backup prefix
  run: |
    BUCKET="${PROJECT_NAME}-assets"
    RULE_ID="expire-daily-backups"

    EXISTING=$(s3cmd getlifecycle "s3://${BUCKET}" 2>&1 || true)

    if echo "$EXISTING" | grep -qF "${RULE_ID}"; then
      echo "✅ Lifecycle rule already in place — skipping"
    elif echo "$EXISTING" | grep -q "<Rule>"; then
      echo "❌ Bucket has other rules but not '${RULE_ID}'."
      echo "   Refusing to overwrite. Merge manually."
      exit 1
    else
      echo "<LifecycleConfiguration><Rule><ID>${RULE_ID}</ID>..." > /tmp/lifecycle.xml
      s3cmd setlifecycle /tmp/lifecycle.xml "s3://${BUCKET}"
    fi

Three branches: ours is there (skip), foreign rules but not ours (abort, don't overwrite), nothing at all (apply ours). This is the only way to safely re-run lifecycle setup automatically.

The Restore Test — The Whole Point

Here's where most teams stop. Don't.

name: DigitalOcean-470-Restore-Test

on:
  schedule:
    - cron: '0 4 * * 0'   # Sunday 04:00 UTC

jobs:
  restore-test:
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: testadmin
          POSTGRES_PASSWORD: testpw_local_only
          POSTGRES_DB: jo4test
        ports: ['5432:5432']

    steps:
      - name: Sandbox banner & hard guard
        run: |
          # Hard guard #1 — target host MUST be localhost
          if [ "${TEST_PG_HOST}" != "localhost" ] \
             && [ "${TEST_PG_HOST}" != "127.0.0.1" ]; then
            echo "❌ Restore target is not localhost: ${TEST_PG_HOST}"
            exit 1
          fi

          # Hard guard #2 — must not look like a managed service
          if echo "${TEST_PG_HOST}" \
             | grep -qE "(ondigitalocean|amazonaws|gcp|azure)"; then
            echo "❌ Target appears to be cloud-managed: ${TEST_PG_HOST}"
            exit 1
          fi

The hard guards are paranoia, on purpose. The restore-test workflow shares code paths with the manual restore workflow. One env var typo — TEST_PG_HOST=jo4-db-do-user-...ondigitalocean.com — and you've just restored last week's backup over today's production database. The two grep -qE checks make that mistake un-shippable.

The rest of the restore test:

Download the most recent backup from Spaces.
Hard-fail if the most recent backup is older than 2 days. This is the silent-failure detector — if the nightly cron stops working, the next Sunday's restore test fails loudly within a week.
Pipe the gzipped SQL into psql against the localhost container.
Assert: minimum table count (catches half-applied schemas), positive row counts on key tables (catches empty dumps that passed the 1KB sanity floor).
Record end-to-end restore duration. This is your empirical RTO, not a number you guessed.

What This Buys You

A nightly backup that fails noisily if the dump is anomalously small or the upload doesn't complete.
A 30-day rolling window maintained by the bucket itself, not a cleanup script you'll forget about.
An automated weekly proof that backups are restorable, restored against a real Postgres of the same major version, with assertions on what came out.
An empirical RTO — every Sunday measures it, every quarter you can compare measurements against your commitment.
A manual restore workflow that's the same code path the test exercises — your worst day is not the day you debug the restore script.

Lessons Learned

Untested backups have a 50% failure rate. Industry data, our internal data, your data — same number. Test.
Failing-loud beats failing-silent every time. A workflow that hard-fails on stale backups, on small dumps, on cross-major pg_dump versions, on non-localhost restore targets — that workflow tells you when something's broken before you need it to work.
Pin the pg_dump major version. Cross-major dumps will seem to work and will be missing things you depended on.
Always use if: always() for cleanup steps that touch shared state (firewall rules, lock files, lifecycle policies). The cleanup needs to run on failure, otherwise failure leaks state.
Concurrency cancellation is the wrong default for backups. Queue them, don't cancel them. Cancelled pg_dump mid-upload is the kind of bug that masquerades as a corrupt restore six weeks later.

When did you last restore a backup? What broke when you tried? Drop it in the comments — the post-mortems are always educational.

Building jo4.io — a URL shortener with a documented RPO, RTO, and a weekly proof we can hit them.

How We Reset Monthly Quotas Without Race Conditions or Cron Drift

Anand Rathnas — Wed, 12 Aug 2026 03:26:05 +0000

If you're billing monthly via Stripe or RevenueCat, it's tempting to reset usage quotas inside your webhook handler. The customer pays → invoice fires → you zero out their urlCount. Clean. One source of truth.

It works for some customers. Two customer types it doesn't work for, and they're both yours:

Yearly subscribers. Their payment event fires once a year. Without a separate reset, they get the monthly cap (e.g., 500 URLs/month on Pro) and then exhaust it forever, because nothing zeroes the counter for the next 11 months.
FREE users. They never trigger a payment event. Their urlCount only moves on URL create/delete, so it grows linearly and never resets.

We needed a quota reset that's decoupled from billing cadence. Here's the Spring scheduler + Postgres advisory lock job we built.

The Shape of the Problem

Two things must be true:

The job runs once per calendar month, regardless of how many app instances are running.
Two instances waking up at the same moment must not both run it. (We're not idempotent — urlCount = 0 is, but the audit log isn't, and we want exactly one log line per reset.)

The naive solution is a global database flag, but you race on that too. The clean solution in Postgres is pg_try_advisory_xact_lock — an instance-level lock keyed on an integer that auto-releases at transaction end.

The Scheduler

@Slf4j
@Component
@EnableScheduling
@RequiredArgsConstructor
public class MonthlyQuotaResetScheduler {

    // Per-job lock ID. Distinct from other schedulers using advisory locks.
    static final long LOCK_ID = 1234567891L;

    private final UserRepository userRepository;
    private final TeamRepository teamRepository;
    private final EntityManager entityManager;
    private final TransactionTemplate transactionTemplate;

    /**
     * Reset quotas at 00:05 UTC on the 1st of each month.
     * Staggered 5 minutes off midnight to dodge the top-of-hour load spike.
     */
    @Scheduled(cron = "0 5 0 1 * *", zone = "UTC")
    public void resetMonthlyQuotas() {
        transactionTemplate.execute(status -> {
            Boolean acquired = (Boolean) entityManager
                .createNativeQuery("SELECT pg_try_advisory_xact_lock(:lockId)")
                .setParameter("lockId", LOCK_ID)
                .getSingleResult();

            if (!Boolean.TRUE.equals(acquired)) {
                log.debug("Monthly reset skipped — another instance has the lock");
                return null;
            }

            log.info("Acquired lock, running monthly quota reset...");
            long now = System.currentTimeMillis();
            int users = userRepository.resetAllUrlCounts(now);
            int teams = teamRepository.resetAllUrlCounts(now);
            log.info("Reset complete: users={}, teams={}", users, teams);
            return null;
        });
    }
}

A few choices worth calling out:

Cron 0 5 0 1 * *. Five minutes past midnight UTC on the first of the month. The 5-minute offset matters more than it looks — every other top-of-hour cron job in your infrastructure is also waking up at 0 0 0 1 * *. Stagger.

pg_try_advisory_xact_lock (not pg_advisory_xact_lock). The try_ variant returns false instead of blocking. If another instance has the lock, we return immediately and log a debug line. No piled-up workers waiting to do nothing.

Transaction-scoped lock (xact_lock, not lock). The lock auto-releases when the transaction commits or rolls back. We never have to remember to release it. If the JVM crashes mid-job, the connection drops, the transaction aborts, and the lock evaporates with it.

@EnableScheduling on the component. Spring needs this annotation somewhere in the application context. Putting it on the scheduler class itself makes the dependency local — drop the class, drop the annotation, no orphan config.

The Repository UPDATE

@Modifying
@Query("UPDATE UserEntity u SET u.urlCount = 0, u.modifiedTime = :now "
     + "WHERE u.deleted = false")
int resetAllUrlCounts(@Param("now") Long now);

Single SQL UPDATE across the whole table, inside the same transaction as the advisory lock. If you're tempted to fetch every user and save() each one — don't. That's O(n) round-trips and O(n) Hibernate dirty-checking. The bulk UPDATE is O(1) round-trips.

The Admin Override

Production rule of thumb: if the system can do it on a schedule, an admin will eventually need to do it for one customer right now. We added a single-user reset on the admin panel:

@Transactional
public UserEntity resetUrlCount(Long userId) {
    UserEntity user = userRepository.findById(userId)
        .filter(u -> !Boolean.TRUE.equals(u.getDeleted()))
        .orElseThrow(() -> new AppException(ErrorCode.USER_NOT_FOUND));

    int previous = user.getUrlCount() != null ? user.getUrlCount() : 0;
    userRepository.resetUrlCountForUser(userId, System.currentTimeMillis());
    log.info("Admin reset urlCount for user {}: {} -> 0", userId, previous);

    return userRepository.findById(userId).orElse(user);
}

Two details:

Filter on deleted = false before resetting. Don't unblock soft-deleted accounts.
Log the previous value. "Reset urlCount" with no context tells you nothing in an audit. "Reset urlCount for user 42: 487 → 0" tells you exactly what happened.

Bonus: The Sticky Override

Some customers need a permanent quota bump that survives plan changes, billing webhooks, and the monthly reset. We added a nullable urlLimitOverride column on the user entity:

public int getEffectiveUrlLimit() {
    // Admin override always wins.
    if (urlLimitOverride != null) {
        return urlLimitOverride;
    }
    // Defend against missed billing webhooks.
    if (!hasActiveSubscription() && subscriptionTier != FREE) {
        return FREE_TIER_URL_LIMIT;
    }
    return urlLimit;
}

Three rules of precedence:

Admin override wins. Settable only via an admin-only endpoint, never mapped from any user-facing DTO.
Lapsed subscription on a paid tier falls back to FREE limits. This is the safety net for missed webhooks.
Otherwise, the stored urlLimit (kept in sync with the plan config by the billing webhooks).

The override is sticky on purpose — billing webhooks don't touch it, the monthly reset doesn't touch it, the daily plan-config sync doesn't touch it. An admin sets it, and an admin clears it.

Pitfalls We Hit (So You Don't)

Don't put the reset inside billing webhooks. Yearly subscribers and FREE users will resent you for it.
Don't use an unbounded pg_advisory_lock without a try variant. Two instances will both wake up; one blocks for the other; the second one runs the job again immediately after the first commits.
Don't forget the audit log row. When a customer asks "why did my quota suddenly drop on the 1st?", the answer should be one query away.
Don't bundle the override into the user-facing profile DTO. Defense in depth — the only path to the column is the admin endpoint with requireAdmin() upstream.
Don't skip the idempotency test. Run the job twice in the same minute. The second run should be a no-op (the lock holder rejected it) or set urlCount from 0 to 0 (still a no-op). Both are fine. Anything else is a bug.

How do you decouple usage resets from billing cadence? What's your scheduler setup look like? Drop it in the comments.

Building jo4.io — a URL shortener where the quota math is the easy part of the platform.

Groq Returned Empty Content. The Bug Was Hiding in Reasoning Tokens.

Anand Rathnas — Mon, 10 Aug 2026 03:14:18 +0000

We use Groq's gpt-oss-safeguard model to classify pages behind freshly created short links. Most pages take a few hundred tokens to score. Some don't. And the ones that don't were silently failing — for weeks — until we noticed the symptom: a small but consistent stream of links stuck in "preview pending" forever.

Here's what we found.

The Problem

The classifier wraps a single Groq chat completion. Send page text, get back a JSON verdict (safe, unsafe, with category codes). For 95% of links, this works in well under a second.

For the other 5%, we'd see this in logs:

WARN  Empty content in Groq response
WARN  Classification failed for shortUrl=xyz123 — preview stays enabled

Empty content. Not a network error, not a rate limit, not malformed JSON. The API returned 200, the choices array had one entry, and choices[0].message.content was "".

What did those pages have in common? They weren't obvious spam. They weren't obvious safe. They were ambiguous — a wellness blog that mentioned medication dosages, a forum thread about firearms law, a satire site quoting violent rhetoric. The kind of content where a human reviewer would also pause.

The Wrong First Guess

Our first instinct: the model is rate-limited or degraded for hard inputs. We added retries. The empty-content rate didn't budge.

Second guess: we're hitting max_tokens. We had set it to 200. Maybe ambiguous pages produce longer verdicts. We bumped it to 400. Empty content rate didn't budge.

The clue we kept missing was sitting in the response body itself, in a field we weren't parsing.

The Root Cause

Groq's response includes a usage block, and usage.completion_tokens_details.reasoning_tokens was the smoking gun:

{
  "choices": [{
    "message": { "content": "" },
    "finish_reason": "length"
  }],
  "usage": {
    "completion_tokens": 200,
    "completion_tokens_details": {
      "reasoning_tokens": 200
    }
  }
}

gpt-oss-safeguard is a reasoning model. Before emitting a single character of content, it spends completion tokens on internal chain-of-thought. Easy pages spend a few dozen reasoning tokens, then emit a 50-token verdict. Ambiguous pages spend several hundred reasoning tokens — and on those, our 200-token budget was being exhausted inside the reasoning phase, leaving zero tokens for content.

The API obediently returned the response. choices[0].message.content was "" because there was nothing left in the budget to write into it. finish_reason was length, not stop — the model didn't decide it was done, the token budget cut it off mid-thought.

We were paying for full inference and getting empty strings.

The Fix

Three changes:

1. Switch from max_tokens to max_completion_tokens — max_tokens is deprecated for reasoning models. Use the correct parameter name so the API enforces the limit you mean.

2. Raise the budget with headroom. We profiled real ambiguous pages: worst case was ~550 reasoning tokens. We set the budget to 1024 — covers worst case plus content with margin to spare.

static final int MAX_COMPLETION_TOKENS = 1024;

String requestBody = objectMapper.writeValueAsString(Map.of(
    "model", modelName,
    "messages", List.of(
        Map.of("role", "system", "content", SAFETY_POLICY),
        Map.of("role", "user", "content", text)
    ),
    "max_completion_tokens", MAX_COMPLETION_TOKENS,
    "temperature", 0.0
));

3. Parse usage and alert when reasoning tokens approach the budget. This is the part that actually prevents the next regression:

if (reasoningTokens != null
    && reasoningTokens > MAX_COMPLETION_TOKENS * 0.8) {
    log.warn("Classifier reasoning tokens near budget: {}/{} — "
        + "consider raising max_completion_tokens",
        reasoningTokens, MAX_COMPLETION_TOKENS);
}

When reasoning crosses 80% of the budget, we log a warning. The next ambiguous page in that distribution is the one that will trip finish_reason=length and return empty content. We'd rather raise the budget before users see stuck previews, not after.

We also added the diagnostic to the empty-content branch:

if (rawContent == null || rawContent.isBlank()) {
    String finishReason = firstChoice.getFinishReason();
    log.warn("Empty content (finish_reason={} completion_tokens={} "
        + "reasoning_tokens={})",
        finishReason, completionTokens, reasoningTokens);
    return ClassificationResult.error(
        "Empty classifier output (finish_reason=" + finishReason + ")");
}

So if it ever happens again, the next person debugging it has the answer in the first log line, not after a week of squinting.

Lessons Learned

For reasoning models, max_tokens is a budget the model spends thinking and speaking. If the budget runs out mid-thought, you get a 200 response with empty content. There is no exception, no error code in the body — just a "" and a finish_reason: length that you have to parse to see.
Profile against the hardest inputs, not the average. Our 200-token budget worked fine on test fixtures because our fixtures were obvious. Ambiguity is what blows the budget.
finish_reason is the field that tells you the truth. stop = model is done. length = the model wanted to keep going and you didn't let it. Treat them as completely different outcomes.
completion_tokens_details.reasoning_tokens is the leading indicator. Don't wait for empty content to alert. Watch reasoning-token usage as a percentage of the budget, and alert before you cross the cliff.
Use the deprecated-API warnings. max_tokens was the wrong field name for reasoning models. The API silently honored it anyway, which made the bug subtler. The right field name is max_completion_tokens.

Have you been bitten by an LLM that "succeeded" with no output? What was your tell? Drop it in the comments.

Building jo4.io — a URL shortener with AI-backed content scanning that fails loudly, not silently.

DEV Community: Anand Rathnas

Grafana 11 Provisions Alerts Before Dashboards — and That Breaks folderUid

What We Expected

What Actually Happens

Why

The Fix

Verifying It Worked

Lessons Learned

Your sed -i Edit Isn't Reaching the Container: The Bind-Mount Inode Trap

The Symptom

Diagnosis: stat Tells the Truth

Why sed -i Is the Culprit

The Fix

Option A: edit in place for real

Option B: restart the container

Tools That Have the Same Trap

The Bulletproof Workflow

Lessons Learned

Why pgrep Can't Detect cloud-init's Apt Races on DigitalOcean

The Setup

The Problem

The Two Sources of Contention

Why the pgrep Loop Failed

The Fix

Lessons Learned

How to Plug Claude Routines Into Your SaaS With MCP

TL;DR

What I Thought I Was Building vs. What MCP Actually Required

Anatomy of the Shipped Code

The Four RFCs That Bit Me

1. RFC 7591 — Dynamic Client Registration

2. RFC 8414 — Authorization Server Metadata

3. RFC 9728 — Protected Resource Metadata

4. RFC 8707 — Resource Indicators

The MCP-Specific Bit (Finally)

The thread-local trap

How To Connect Your Own Client (The Tutorial Bit)

Step 1 — Discover

Step 2 — Register dynamically

Step 3 — PKCE authorization

Step 4 — Call the MCP server

Step 5 — Try a tool

Things I'd Tell My Past Self

What's Next

From Polling to SSE in RTK Query Without Breaking Reconnect Logic

The TL;DR

The Component

The Subscribe Function

The Sharp Edges

Cache Patching, Not Local State

What We Didn't Do

Lessons Learned

Server-Sent Events Across Multiple Pods: Redis Pub/Sub + Spring WebFlux

The Endpoint

The Per-User Sink

The Cross-Pod Problem

The Fallback to Local

The Transactional-Listener Bridge

Lessons Learned

We Were Hitting Auth0 /userinfo on Every Request. Here's the Fix.

The Default Auth0 Access Token

What That Looked Like

The Right Fix Is on the Auth0 Side

The Graceful Degradation

The Migration Window

Bonus: Lock Release

Lessons Learned

DigitalOcean Postgres Has a Firewall Trap. We Walked Right Into It.

The Setup

The Quiet Bit

The Fix

Immediate ops fix

Permanent guard

The Footnote on doctl

Lessons Learned

Why Your JPA Counter Column Keeps Resetting to Zero

The Setup

The Problem

The Two-Part Fix

Part 1: Mark counter columns updatable = false

Diagnosis: `stat` Tells the Truth

The Footnote on `doctl`

Part 1: Mark counter columns `updatable = false`

Part 2: Bump the version inside the `@Modifying @Query`