DEV Community: Anand Rathnas

Backups That You Don't Restore Aren't Backups: A Weekly Restore Drill

Anand Rathnas — Fri, 14 Aug 2026 03:26:42 +0000

Most teams have backups. Half of those teams have never restored one. The other half discovered the hard way that their dumps were corrupt, incomplete, or pointed at a database schema that no longer exists.

Backups you don't actively restore aren't backups — they're warm fuzzy feelings. Here's how we built a backup pipeline whose first-class citizen is the weekly restore, not the nightly dump.

The Pipeline at 30,000 Feet

Three GitHub Actions workflows:

Nightly backup — pg_dump from the production DB, gzip, upload to DigitalOcean Spaces. Runs at 03:00 UTC daily.
Restore test — pulls the latest backup, restores it into a Docker postgres:16 container in the runner, asserts table count and row counts. Runs Sunday 04:00 UTC.
Manual restore — parameterized one-click pipeline for the actual recovery flow. Same code path the restore test exercises, minus the assertions, plus the parameter for "which backup."

The clever bit isn't (1). The clever bit is (2) hard-failing if (1) silently broke.

The Backup Workflow

name: DigitalOcean-460-Backup

on:
  schedule:
    - cron: '0 3 * * *'   # 03:00 UTC daily
  workflow_dispatch:

concurrency:
  group: jo4-db-backup
  cancel-in-progress: false   # Don't cancel a running backup

env:
  DO_REGION: nyc3
  PROJECT_NAME: jo4
  SCHEMA_NAME: alertstage
  BACKUP_PREFIX: db-daily-backup

Two settings worth highlighting:

concurrency.cancel-in-progress: false — if workflow_dispatch fires while the cron is running, queue the new run, don't cancel the old one. Cancelling a half-uploaded dump leaves a corrupt object in the bucket and a confusing log trail.
Pin the pg_dump major version. PostgreSQL is forgiving about minor versions, but a pg_dump from a different major to the server is a bug factory:

- name: Install PostgreSQL client and verify version
  run: |
    sudo apt-get install -y -qq postgresql-client
    PG_DUMP_MAJOR=$(pg_dump --version | awk '{print $3}' | cut -d. -f1)
    if [ "$PG_DUMP_MAJOR" != "16" ]; then
      echo "❌ pg_dump must be 16 to match server, got: $(pg_dump --version)"
      exit 1
    fi

The Firewall Dance

Our managed Postgres is locked behind a trusted-source IP allowlist. The runner IP rotates every job. So every backup adds the runner's IP, runs the dump, and removes the IP — even if the dump fails:

- name: Open DB trusted-source for runner IP
  id: db_firewall
  run: |
    RUNNER_IP=$(curl -s https://api.ipify.org)
    DB_ID="${{ steps.database.outputs.db_id }}"
    doctl databases firewalls add "$DB_ID" --rule "ip_addr:${RUNNER_IP}"
    echo "runner_ip=$RUNNER_IP" >> $GITHUB_OUTPUT
    echo "db_id=$DB_ID"          >> $GITHUB_OUTPUT

- name: Close DB trusted-source (revoke runner IP)
  if: always()   # critical — runs even if the dump step failed
  run: |
    UUIDS=$(doctl databases firewalls list "$DB_ID" \
      | awk -v ip="$RUNNER_IP" 'NR>1 && $3=="ip_addr" && $4==ip {print $1}')
    echo "$UUIDS" | while read -r uuid; do
      [ -z "$uuid" ] && continue
      doctl databases firewalls remove "$DB_ID" "$uuid" || true
    done

if: always() is non-negotiable here. If the dump fails and we don't clean up, we leak ephemeral runner IPs into the trusted-sources list. After a few months you're looking at hundreds of stale entries.

The Dump Itself

- name: Run pg_dump
  run: |
    TS=$(date -u +%Y%m%d-%H%M%S)
    DUMP_FILE="/tmp/jo4db-${TS}.sql"

    PGPASSWORD="${{ steps.database.outputs.db_password }}" \
    PGSSLMODE=require \
    pg_dump \
      -h "${DB_HOST}" -p "${DB_PORT}" -U "${DB_USER}" -d "${DB_DATABASE}" \
      --schema=${SCHEMA_NAME} \
      --no-owner \
      --no-privileges \
      --quote-all-identifiers \
      -f "${DUMP_FILE}"

    gzip -9 "${DUMP_FILE}"

    SIZE_BYTES=$(stat -c%s "${DUMP_FILE}.gz")
    if [ "$SIZE_BYTES" -lt 1024 ]; then
      echo "❌ Dump suspiciously small ($SIZE_BYTES bytes) — aborting upload"
      exit 1
    fi

--no-owner --no-privileges — the dump is restorable on any Postgres 16+ instance, not just the one with our exact role names. Important for the restore test, which uses a fresh container.

--quote-all-identifiers — defends against case-sensitivity surprises if the restore target uses different default search paths.

The 1KB sanity floor — even an empty schema produces more than 1KB of dump (CREATE TABLE statements, comments, etc.). If the dump is smaller, something went wrong, abort before you upload garbage that overwrites yesterday's good backup.

The Lifecycle Rule (Idempotently)

We expire backups after 30 days via a bucket lifecycle rule. The catch: s3cmd setlifecycle replaces the entire bucket policy. We don't want to clobber other rules:

- name: Ensure 30-day lifecycle rule on backup prefix
  run: |
    BUCKET="${PROJECT_NAME}-assets"
    RULE_ID="expire-daily-backups"

    EXISTING=$(s3cmd getlifecycle "s3://${BUCKET}" 2>&1 || true)

    if echo "$EXISTING" | grep -qF "${RULE_ID}"; then
      echo "✅ Lifecycle rule already in place — skipping"
    elif echo "$EXISTING" | grep -q "<Rule>"; then
      echo "❌ Bucket has other rules but not '${RULE_ID}'."
      echo "   Refusing to overwrite. Merge manually."
      exit 1
    else
      echo "<LifecycleConfiguration><Rule><ID>${RULE_ID}</ID>..." > /tmp/lifecycle.xml
      s3cmd setlifecycle /tmp/lifecycle.xml "s3://${BUCKET}"
    fi

Three branches: ours is there (skip), foreign rules but not ours (abort, don't overwrite), nothing at all (apply ours). This is the only way to safely re-run lifecycle setup automatically.

The Restore Test — The Whole Point

Here's where most teams stop. Don't.

name: DigitalOcean-470-Restore-Test

on:
  schedule:
    - cron: '0 4 * * 0'   # Sunday 04:00 UTC

jobs:
  restore-test:
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: testadmin
          POSTGRES_PASSWORD: testpw_local_only
          POSTGRES_DB: jo4test
        ports: ['5432:5432']

    steps:
      - name: Sandbox banner & hard guard
        run: |
          # Hard guard #1 — target host MUST be localhost
          if [ "${TEST_PG_HOST}" != "localhost" ] \
             && [ "${TEST_PG_HOST}" != "127.0.0.1" ]; then
            echo "❌ Restore target is not localhost: ${TEST_PG_HOST}"
            exit 1
          fi

          # Hard guard #2 — must not look like a managed service
          if echo "${TEST_PG_HOST}" \
             | grep -qE "(ondigitalocean|amazonaws|gcp|azure)"; then
            echo "❌ Target appears to be cloud-managed: ${TEST_PG_HOST}"
            exit 1
          fi

The hard guards are paranoia, on purpose. The restore-test workflow shares code paths with the manual restore workflow. One env var typo — TEST_PG_HOST=jo4-db-do-user-...ondigitalocean.com — and you've just restored last week's backup over today's production database. The two grep -qE checks make that mistake un-shippable.

The rest of the restore test:

Download the most recent backup from Spaces.
Hard-fail if the most recent backup is older than 2 days. This is the silent-failure detector — if the nightly cron stops working, the next Sunday's restore test fails loudly within a week.
Pipe the gzipped SQL into psql against the localhost container.
Assert: minimum table count (catches half-applied schemas), positive row counts on key tables (catches empty dumps that passed the 1KB sanity floor).
Record end-to-end restore duration. This is your empirical RTO, not a number you guessed.

What This Buys You

A nightly backup that fails noisily if the dump is anomalously small or the upload doesn't complete.
A 30-day rolling window maintained by the bucket itself, not a cleanup script you'll forget about.
An automated weekly proof that backups are restorable, restored against a real Postgres of the same major version, with assertions on what came out.
An empirical RTO — every Sunday measures it, every quarter you can compare measurements against your commitment.
A manual restore workflow that's the same code path the test exercises — your worst day is not the day you debug the restore script.

Lessons Learned

Untested backups have a 50% failure rate. Industry data, our internal data, your data — same number. Test.
Failing-loud beats failing-silent every time. A workflow that hard-fails on stale backups, on small dumps, on cross-major pg_dump versions, on non-localhost restore targets — that workflow tells you when something's broken before you need it to work.
Pin the pg_dump major version. Cross-major dumps will seem to work and will be missing things you depended on.
Always use if: always() for cleanup steps that touch shared state (firewall rules, lock files, lifecycle policies). The cleanup needs to run on failure, otherwise failure leaks state.
Concurrency cancellation is the wrong default for backups. Queue them, don't cancel them. Cancelled pg_dump mid-upload is the kind of bug that masquerades as a corrupt restore six weeks later.

When did you last restore a backup? What broke when you tried? Drop it in the comments — the post-mortems are always educational.

Building jo4.io — a URL shortener with a documented RPO, RTO, and a weekly proof we can hit them.

How We Reset Monthly Quotas Without Race Conditions or Cron Drift

Anand Rathnas — Wed, 12 Aug 2026 03:26:05 +0000

If you're billing monthly via Stripe or RevenueCat, it's tempting to reset usage quotas inside your webhook handler. The customer pays → invoice fires → you zero out their urlCount. Clean. One source of truth.

It works for some customers. Two customer types it doesn't work for, and they're both yours:

Yearly subscribers. Their payment event fires once a year. Without a separate reset, they get the monthly cap (e.g., 500 URLs/month on Pro) and then exhaust it forever, because nothing zeroes the counter for the next 11 months.
FREE users. They never trigger a payment event. Their urlCount only moves on URL create/delete, so it grows linearly and never resets.

We needed a quota reset that's decoupled from billing cadence. Here's the Spring scheduler + Postgres advisory lock job we built.

The Shape of the Problem

Two things must be true:

The job runs once per calendar month, regardless of how many app instances are running.
Two instances waking up at the same moment must not both run it. (We're not idempotent — urlCount = 0 is, but the audit log isn't, and we want exactly one log line per reset.)

The naive solution is a global database flag, but you race on that too. The clean solution in Postgres is pg_try_advisory_xact_lock — an instance-level lock keyed on an integer that auto-releases at transaction end.

The Scheduler

@Slf4j
@Component
@EnableScheduling
@RequiredArgsConstructor
public class MonthlyQuotaResetScheduler {

    // Per-job lock ID. Distinct from other schedulers using advisory locks.
    static final long LOCK_ID = 1234567891L;

    private final UserRepository userRepository;
    private final TeamRepository teamRepository;
    private final EntityManager entityManager;
    private final TransactionTemplate transactionTemplate;

    /**
     * Reset quotas at 00:05 UTC on the 1st of each month.
     * Staggered 5 minutes off midnight to dodge the top-of-hour load spike.
     */
    @Scheduled(cron = "0 5 0 1 * *", zone = "UTC")
    public void resetMonthlyQuotas() {
        transactionTemplate.execute(status -> {
            Boolean acquired = (Boolean) entityManager
                .createNativeQuery("SELECT pg_try_advisory_xact_lock(:lockId)")
                .setParameter("lockId", LOCK_ID)
                .getSingleResult();

            if (!Boolean.TRUE.equals(acquired)) {
                log.debug("Monthly reset skipped — another instance has the lock");
                return null;
            }

            log.info("Acquired lock, running monthly quota reset...");
            long now = System.currentTimeMillis();
            int users = userRepository.resetAllUrlCounts(now);
            int teams = teamRepository.resetAllUrlCounts(now);
            log.info("Reset complete: users={}, teams={}", users, teams);
            return null;
        });
    }
}

A few choices worth calling out:

Cron 0 5 0 1 * *. Five minutes past midnight UTC on the first of the month. The 5-minute offset matters more than it looks — every other top-of-hour cron job in your infrastructure is also waking up at 0 0 0 1 * *. Stagger.

pg_try_advisory_xact_lock (not pg_advisory_xact_lock). The try_ variant returns false instead of blocking. If another instance has the lock, we return immediately and log a debug line. No piled-up workers waiting to do nothing.

Transaction-scoped lock (xact_lock, not lock). The lock auto-releases when the transaction commits or rolls back. We never have to remember to release it. If the JVM crashes mid-job, the connection drops, the transaction aborts, and the lock evaporates with it.

@EnableScheduling on the component. Spring needs this annotation somewhere in the application context. Putting it on the scheduler class itself makes the dependency local — drop the class, drop the annotation, no orphan config.

The Repository UPDATE

@Modifying
@Query("UPDATE UserEntity u SET u.urlCount = 0, u.modifiedTime = :now "
     + "WHERE u.deleted = false")
int resetAllUrlCounts(@Param("now") Long now);

Single SQL UPDATE across the whole table, inside the same transaction as the advisory lock. If you're tempted to fetch every user and save() each one — don't. That's O(n) round-trips and O(n) Hibernate dirty-checking. The bulk UPDATE is O(1) round-trips.

The Admin Override

Production rule of thumb: if the system can do it on a schedule, an admin will eventually need to do it for one customer right now. We added a single-user reset on the admin panel:

@Transactional
public UserEntity resetUrlCount(Long userId) {
    UserEntity user = userRepository.findById(userId)
        .filter(u -> !Boolean.TRUE.equals(u.getDeleted()))
        .orElseThrow(() -> new AppException(ErrorCode.USER_NOT_FOUND));

    int previous = user.getUrlCount() != null ? user.getUrlCount() : 0;
    userRepository.resetUrlCountForUser(userId, System.currentTimeMillis());
    log.info("Admin reset urlCount for user {}: {} -> 0", userId, previous);

    return userRepository.findById(userId).orElse(user);
}

Two details:

Filter on deleted = false before resetting. Don't unblock soft-deleted accounts.
Log the previous value. "Reset urlCount" with no context tells you nothing in an audit. "Reset urlCount for user 42: 487 → 0" tells you exactly what happened.

Bonus: The Sticky Override

Some customers need a permanent quota bump that survives plan changes, billing webhooks, and the monthly reset. We added a nullable urlLimitOverride column on the user entity:

public int getEffectiveUrlLimit() {
    // Admin override always wins.
    if (urlLimitOverride != null) {
        return urlLimitOverride;
    }
    // Defend against missed billing webhooks.
    if (!hasActiveSubscription() && subscriptionTier != FREE) {
        return FREE_TIER_URL_LIMIT;
    }
    return urlLimit;
}

Three rules of precedence:

Admin override wins. Settable only via an admin-only endpoint, never mapped from any user-facing DTO.
Lapsed subscription on a paid tier falls back to FREE limits. This is the safety net for missed webhooks.
Otherwise, the stored urlLimit (kept in sync with the plan config by the billing webhooks).

The override is sticky on purpose — billing webhooks don't touch it, the monthly reset doesn't touch it, the daily plan-config sync doesn't touch it. An admin sets it, and an admin clears it.

Pitfalls We Hit (So You Don't)

Don't put the reset inside billing webhooks. Yearly subscribers and FREE users will resent you for it.
Don't use an unbounded pg_advisory_lock without a try variant. Two instances will both wake up; one blocks for the other; the second one runs the job again immediately after the first commits.
Don't forget the audit log row. When a customer asks "why did my quota suddenly drop on the 1st?", the answer should be one query away.
Don't bundle the override into the user-facing profile DTO. Defense in depth — the only path to the column is the admin endpoint with requireAdmin() upstream.
Don't skip the idempotency test. Run the job twice in the same minute. The second run should be a no-op (the lock holder rejected it) or set urlCount from 0 to 0 (still a no-op). Both are fine. Anything else is a bug.

How do you decouple usage resets from billing cadence? What's your scheduler setup look like? Drop it in the comments.

Building jo4.io — a URL shortener where the quota math is the easy part of the platform.

Groq Returned Empty Content. The Bug Was Hiding in Reasoning Tokens.

Anand Rathnas — Mon, 10 Aug 2026 03:14:18 +0000

We use Groq's gpt-oss-safeguard model to classify pages behind freshly created short links. Most pages take a few hundred tokens to score. Some don't. And the ones that don't were silently failing — for weeks — until we noticed the symptom: a small but consistent stream of links stuck in "preview pending" forever.

Here's what we found.

The Problem

The classifier wraps a single Groq chat completion. Send page text, get back a JSON verdict (safe, unsafe, with category codes). For 95% of links, this works in well under a second.

For the other 5%, we'd see this in logs:

WARN  Empty content in Groq response
WARN  Classification failed for shortUrl=xyz123 — preview stays enabled

Empty content. Not a network error, not a rate limit, not malformed JSON. The API returned 200, the choices array had one entry, and choices[0].message.content was "".

What did those pages have in common? They weren't obvious spam. They weren't obvious safe. They were ambiguous — a wellness blog that mentioned medication dosages, a forum thread about firearms law, a satire site quoting violent rhetoric. The kind of content where a human reviewer would also pause.

The Wrong First Guess

Our first instinct: the model is rate-limited or degraded for hard inputs. We added retries. The empty-content rate didn't budge.

Second guess: we're hitting max_tokens. We had set it to 200. Maybe ambiguous pages produce longer verdicts. We bumped it to 400. Empty content rate didn't budge.

The clue we kept missing was sitting in the response body itself, in a field we weren't parsing.

The Root Cause

Groq's response includes a usage block, and usage.completion_tokens_details.reasoning_tokens was the smoking gun:

{
  "choices": [{
    "message": { "content": "" },
    "finish_reason": "length"
  }],
  "usage": {
    "completion_tokens": 200,
    "completion_tokens_details": {
      "reasoning_tokens": 200
    }
  }
}

gpt-oss-safeguard is a reasoning model. Before emitting a single character of content, it spends completion tokens on internal chain-of-thought. Easy pages spend a few dozen reasoning tokens, then emit a 50-token verdict. Ambiguous pages spend several hundred reasoning tokens — and on those, our 200-token budget was being exhausted inside the reasoning phase, leaving zero tokens for content.

The API obediently returned the response. choices[0].message.content was "" because there was nothing left in the budget to write into it. finish_reason was length, not stop — the model didn't decide it was done, the token budget cut it off mid-thought.

We were paying for full inference and getting empty strings.

The Fix

Three changes:

1. Switch from max_tokens to max_completion_tokens — max_tokens is deprecated for reasoning models. Use the correct parameter name so the API enforces the limit you mean.

2. Raise the budget with headroom. We profiled real ambiguous pages: worst case was ~550 reasoning tokens. We set the budget to 1024 — covers worst case plus content with margin to spare.

static final int MAX_COMPLETION_TOKENS = 1024;

String requestBody = objectMapper.writeValueAsString(Map.of(
    "model", modelName,
    "messages", List.of(
        Map.of("role", "system", "content", SAFETY_POLICY),
        Map.of("role", "user", "content", text)
    ),
    "max_completion_tokens", MAX_COMPLETION_TOKENS,
    "temperature", 0.0
));

3. Parse usage and alert when reasoning tokens approach the budget. This is the part that actually prevents the next regression:

if (reasoningTokens != null
    && reasoningTokens > MAX_COMPLETION_TOKENS * 0.8) {
    log.warn("Classifier reasoning tokens near budget: {}/{} — "
        + "consider raising max_completion_tokens",
        reasoningTokens, MAX_COMPLETION_TOKENS);
}

When reasoning crosses 80% of the budget, we log a warning. The next ambiguous page in that distribution is the one that will trip finish_reason=length and return empty content. We'd rather raise the budget before users see stuck previews, not after.

We also added the diagnostic to the empty-content branch:

if (rawContent == null || rawContent.isBlank()) {
    String finishReason = firstChoice.getFinishReason();
    log.warn("Empty content (finish_reason={} completion_tokens={} "
        + "reasoning_tokens={})",
        finishReason, completionTokens, reasoningTokens);
    return ClassificationResult.error(
        "Empty classifier output (finish_reason=" + finishReason + ")");
}

So if it ever happens again, the next person debugging it has the answer in the first log line, not after a week of squinting.

Lessons Learned

For reasoning models, max_tokens is a budget the model spends thinking and speaking. If the budget runs out mid-thought, you get a 200 response with empty content. There is no exception, no error code in the body — just a "" and a finish_reason: length that you have to parse to see.
Profile against the hardest inputs, not the average. Our 200-token budget worked fine on test fixtures because our fixtures were obvious. Ambiguity is what blows the budget.
finish_reason is the field that tells you the truth. stop = model is done. length = the model wanted to keep going and you didn't let it. Treat them as completely different outcomes.
completion_tokens_details.reasoning_tokens is the leading indicator. Don't wait for empty content to alert. Watch reasoning-token usage as a percentage of the budget, and alert before you cross the cliff.
Use the deprecated-API warnings. max_tokens was the wrong field name for reasoning models. The API silently honored it anyway, which made the bug subtler. The right field name is max_completion_tokens.

Have you been bitten by an LLM that "succeeded" with no output? What was your tell? Drop it in the comments.

Building jo4.io — a URL shortener with AI-backed content scanning that fails loudly, not silently.

Redis Sliding Window Rate Limiting with Lua: The 15-Line Script That Saved Our AI Budget

Anand Rathnas — Sat, 08 Aug 2026 02:57:25 +0000

We have a suggest-categories endpoint that calls Groq's API with Llama 4 Scout under the hood. Every call costs money. Not a lot per request, but a user hammering the endpoint 50 times in a minute adds up fast when you're an indie SaaS.

We needed rate limiting. Not the "use an API gateway" kind — the "I need per-user limits on one specific endpoint and I need it deployed by lunch" kind.

TL;DR: We built a sliding window rate limiter using a Redis sorted set and a 15-line Lua script. 5 requests per 60 seconds per user. The Lua script runs atomically in Redis — no race conditions, no distributed locks, no separate rate limit service. Fail-open on Redis unavailability, fail-closed on actual errors.

Why Not an Existing Library?

Fair question. Libraries like Bucket4j or Resilience4j exist. But we already had Redis, we already had a BidRateLimiterService using sorted sets for a different endpoint, and we wanted the same pattern for consistency. Rolling our own took less time than evaluating and configuring a library.

The existing BidRateLimiterService uses a dual-key approach (daily limit + per-brand limit). The category suggest endpoint only needs a single window. Different enough to warrant its own service, similar enough to follow the same Redis pattern.

The Lua Script

Here's the entire rate limiting logic. It runs atomically inside Redis — no round trips between your app and Redis during the check-and-record:

local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
local member = ARGV[4]

-- Remove entries outside the window
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)

-- Count current entries
local count = redis.call('ZCARD', key)

-- If at limit, return count without recording
if count >= limit then
    return count
end

-- Record this request
redis.call('ZADD', key, now, member)

-- Set TTL for auto-cleanup (window + 1 second buffer)
redis.call('PEXPIRE', key, window + 1000)

return count

Fifteen lines. Let me walk through what each block does.

Step 1: Prune Expired Entries

redis.call('ZREMRANGEBYSCORE', key, 0, now - window)

Redis sorted sets store members with scores. We use the timestamp (in milliseconds) as the score. ZREMRANGEBYSCORE removes every entry older than now - window. If your window is 60 seconds, this removes everything older than 60 seconds ago.

This is why it's a sliding window. Every time the script runs, it cleans up old entries relative to the current time. No fixed buckets, no boundary alignment issues.

Step 2: Count and Decide

local count = redis.call('ZCARD', key)
if count >= limit then
    return count
end

After pruning, ZCARD returns how many requests are in the current window. If we're at or over the limit, return the count immediately. We don't record the new request — we just reject it.

Returning the count (instead of a boolean) lets the caller log how far over the limit the user is. Useful for debugging.

Step 3: Record and Set TTL

redis.call('ZADD', key, now, member)
redis.call('PEXPIRE', key, window + 1000)

If we're under the limit, ZADD records the request with the current timestamp as the score. PEXPIRE sets a TTL on the entire key — if no new requests come in, Redis auto-deletes the key after the window expires. The extra 1-second buffer prevents the key from expiring between the prune and the check on a borderline request.

The Member Uniqueness Problem

The member value in ZADD needs to be unique. If two requests arrive in the same millisecond with the same member value, ZADD overwrites instead of adding — and your counter is wrong.

Our solution: timestamp:uuid-prefix.

String member = System.currentTimeMillis() + ":" + UUID.randomUUID().toString().substring(0, 8);

The timestamp makes it mostly unique. The 8-character UUID prefix handles the same-millisecond collision case. Could we use a full UUID? Sure. But sorted set members are stored in memory, and 8 characters of UUID gives us 2^32 combinations per millisecond. Good enough.

The Java Service

@Slf4j
@Service
public class CategorySuggestRateLimiterService {

    private static final int MAX_REQUESTS = 5;
    private static final int WINDOW_MILLIS = 60_000;

    private final StringRedisTemplate redisTemplate;
    private final DefaultRedisScript<Long> rateLimitScript;

    public CategorySuggestRateLimiterService(StringRedisTemplate redisTemplate) {
        this.redisTemplate = redisTemplate;
        this.rateLimitScript = new DefaultRedisScript<>();
        this.rateLimitScript.setScriptText(LUA_SCRIPT);
        this.rateLimitScript.setResultType(Long.class);
    }

    public boolean isAllowed(String tenantId, String userId) {
        String key = tenantId + ":ratelimit:category-suggest:" + userId;
        String member = System.currentTimeMillis() + ":"
            + UUID.randomUUID().toString().substring(0, 8);

        try {
            Long count = redisTemplate.execute(
                rateLimitScript,
                List.of(key),
                String.valueOf(System.currentTimeMillis()),
                String.valueOf(WINDOW_MILLIS),
                String.valueOf(MAX_REQUESTS),
                member
            );

            if (count == null) {
                // Redis returned null — fail open
                log.warn("Rate limiter returned null for user {}, allowing request", userId);
                return true;
            }

            return count < MAX_REQUESTS;
        } catch (Exception e) {
            // Hard Redis error — fail closed
            log.error("Rate limiter error for user {}", userId, e);
            throw e;
        }
    }
}

Static Final Script for Caching

private final DefaultRedisScript<Long> rateLimitScript;

DefaultRedisScript is initialized once in the constructor. Spring Data Redis caches the script's SHA1 hash after the first EVALSHA call. Subsequent executions use EVALSHA instead of EVAL, avoiding re-transmitting the script text on every request. If you created a new DefaultRedisScript per call, you'd re-register the SHA every time — pointless overhead.

Fail-Open vs. Fail-Closed

This was a deliberate design decision:

if (count == null) {
    // Fail open: allow the request
    return true;
}

If Redis returns null (network blip, script edge case), we allow the request. The user gets their category suggestion. We eat the LLM cost. This matches the pattern we use in BidRateLimiterService — rate limiting is a cost guard, not a security gate. Blocking a legitimate user because Redis hiccupped is worse than allowing an extra LLM call.

But a hard exception (Redis down, connection refused) propagates up:

catch (Exception e) {
    throw e; // fail closed on actual errors
}

If Redis is genuinely unreachable, something is wrong with infrastructure, and we want the request to fail loudly rather than silently burning through our AI budget with zero rate limiting.

Tenant-Scoped Keys

String key = tenantId + ":ratelimit:category-suggest:" + userId;

Our app is multi-tenant. Keys are scoped to {tenant}:ratelimit:category-suggest:{userId}. Each tenant's users have independent rate limits. The key structure also makes it trivial to inspect or flush limits for a specific tenant:

# Check a user's current window
redis-cli ZRANGE "acme:ratelimit:category-suggest:user123" 0 -1 WITHSCORES

# Clear a user's limit (support request)
redis-cli DEL "acme:ratelimit:category-suggest:user123"

Using It in the Controller

@PostMapping("/suggest-categories")
public ResponseEntity<?> suggestCategories(
        @AuthenticationPrincipal UserDetails user,
        @RequestBody SuggestCategoriesRequest request) {

    if (!rateLimiterService.isAllowed(user.getTenantId(), user.getUserId())) {
        return ResponseEntity.status(429)
            .body(Map.of("error", "Rate limit exceeded. Try again in 60 seconds."));
    }

    // Call Groq LLM
    List<String> categories = categoryService.suggest(request);
    return ResponseEntity.ok(categories);
}

Clean. The controller doesn't know about Redis, Lua, or sorted sets. It asks "is this allowed?" and gets a boolean.

Why Not a Token Bucket?

Token buckets are great for smoothing traffic. But for our use case — "has this user made more than 5 requests in the last 60 seconds?" — a sliding window is more intuitive and more precise.

Token bucket: refills at a fixed rate, allows bursts up to bucket size. A user could make 5 requests in 1 second, wait 59 seconds, and make 5 more.

Sliding window: counts actual requests in the last N seconds. At any point in time, the count reflects reality. No burst-then-wait gaming.

For protecting an AI budget, the sliding window gives us exactly the behavior we want: hard cap per user per minute, no exceptions.

What This Cost Us

Implementation time: About 2 hours, including tests.
Redis memory: Negligible. Each sorted set entry is ~50 bytes. 5 entries per user, auto-expiring. Even with 10,000 active users, that's 2.5 MB.
Latency: The Lua script executes in microseconds inside Redis. One network round trip. Faster than the LLM call it's guarding by three orders of magnitude.
AI budget saved: We caught a user running an automated script that was hitting suggest-categories 200+ times per hour. At our Groq pricing, that was adding up.

How do you rate-limit your AI endpoints? Curious whether people are using managed API gateways or rolling their own like us.

Building jo4.io — a URL shortener with analytics, bio pages, and an affiliate marketplace. Our AI features are now budget-friendly.

Adding Micrometer Metrics to Spring Boot Schedulers (Without Over-Engineering It)

Anand Rathnas — Thu, 06 Aug 2026 04:17:56 +0000

Our campaign enrichment scheduler had been running for months with zero observability. It processed batches every 15 minutes, ran a watchdog every 15 minutes, and cleaned up stale data daily at 4 AM. If something broke, we found out from user reports.

That's not a metrics strategy. That's hope.

TL;DR: We added spring-boot-starter-actuator to a Spring Boot 3.4 app, registered four Micrometer metrics (two counters, one more counter, one timer) in the constructor, and verified everything with SimpleMeterRegistry in unit tests. Total time: under an hour. Zero YAML configuration.

Starting Point: Zero Metrics Infrastructure

The app had no actuator dependency, no metrics endpoint, no Prometheus scraping. The scheduler looked like this:

@Slf4j
@Service
@RequiredArgsConstructor
public class CampaignEnrichmentScheduler {

    private final CampaignEnrichmentService service;

    @Scheduled(fixedRate = 900_000) // 15 minutes
    public void processBatch() {
        int processed = service.processBatch(50);
        log.info("Enriched {} campaigns", processed);
    }

    @Scheduled(fixedRate = 900_000)
    public void watchdog() {
        int recovered = service.recoverStuckCampaigns();
        if (recovered > 0) {
            log.warn("Recovered {} stuck campaigns", recovered);
        }
    }

    @Scheduled(cron = "0 0 4 * * *") // Daily 4 AM
    public void cleanup() {
        service.cleanupExpiredEnrichments();
    }
}

Logs only. If you wanted to know how many campaigns were processed in the last hour, you'd grep CloudWatch. Fun.

Step 1: Add the Dependency

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

That's it. Spring Boot auto-configures a MeterRegistry bean. No application.yml changes needed for basic metrics. If you're exporting to Prometheus, you'd add micrometer-registry-prometheus too, but that's a separate concern.

Step 2: Pick Your Metrics

Before writing code, decide what you actually need. We went with four:

Metric	Type	Why
`campaign.enrichment.processed`	Counter	Total campaigns enriched (throughput)
`campaign.enrichment.failed`	Counter	Total failures across batch + watchdog
`campaign.enrichment.recovered`	Counter	Campaigns rescued by watchdog (health signal)
`campaign.enrichment.batch.duration`	Timer	How long each batch takes (latency)

Four metrics. Not forty. You can always add more later.

Notice the shared failed counter across batch processing and the watchdog. At our scale, a single failure counter is fine. If we needed per-method failure rates, we'd add a method tag. But we don't need that yet, and unused tags are just noise in your dashboards.

Step 3: Replace @RequiredArgsConstructor with an Explicit Constructor

This is the part most tutorials skip. With @RequiredArgsConstructor, Lombok generates a constructor with all final fields. But Micrometer metrics are registered at construction time, and you need the MeterRegistry to create them. Counters and timers aren't fields you inject — they're fields you build from an injected registry.

@Slf4j
@Service
public class CampaignEnrichmentScheduler {

    private final CampaignEnrichmentService service;
    private final Counter processedCounter;
    private final Counter failedCounter;
    private final Counter recoveredCounter;
    private final Timer batchTimer;

    public CampaignEnrichmentScheduler(
            CampaignEnrichmentService service,
            MeterRegistry meterRegistry) {
        this.service = service;
        this.processedCounter = meterRegistry.counter("campaign.enrichment.processed");
        this.failedCounter = meterRegistry.counter("campaign.enrichment.failed");
        this.recoveredCounter = meterRegistry.counter("campaign.enrichment.recovered");
        this.batchTimer = meterRegistry.timer("campaign.enrichment.batch.duration");
    }
}

Why register in the constructor?

Metrics exist immediately — no lazy initialization surprises.
The MeterRegistry is the only injected dependency (besides your service). Clean.
The counter/timer fields are final. Immutable after construction.

Drop @RequiredArgsConstructor. An explicit constructor with two parameters is not boilerplate — it's clarity.

Step 4: Instrument the Methods

processBatch: Timer + Counter

Timer.record(Callable<T>) is the key API here. It wraps your callable, times it, and returns the result. No manual stopwatch:

@Scheduled(fixedRate = 900_000)
public void processBatch() {
    try {
        int processed = batchTimer.record(() -> service.processBatch(50));
        processedCounter.increment(processed);
        log.info("Enriched {} campaigns", processed);
    } catch (Exception e) {
        failedCounter.increment();
        log.error("Batch processing failed", e);
    }
}

A few things to notice:

batchTimer.record(() -> service.processBatch(50)) returns the lambda's return value. The timer records the duration. You get both in one call.
processedCounter.increment(processed) — Micrometer's Counter.increment() accepts a double. Internally, it's backed by a LongAdder, so it's thread-safe and performant. Passing an int works fine.
The catch block increments failedCounter. If batch processing throws, we count it and log it. The scheduler keeps running on the next tick.

watchdog: Counter Only

@Scheduled(fixedRate = 900_000)
public void watchdog() {
    try {
        int recovered = service.recoverStuckCampaigns();
        if (recovered > 0) {
            recoveredCounter.increment(recovered);
            log.warn("Recovered {} stuck campaigns", recovered);
        }
    } catch (Exception e) {
        failedCounter.increment();
        log.error("Watchdog failed", e);
    }
}

Same failedCounter as processBatch. At our volume, we don't need to distinguish where failures come from — we just need to know they're happening. If the counter spikes, we check logs.

cleanup: Just Logging

@Scheduled(cron = "0 0 4 * * *")
public void cleanup() {
    try {
        service.cleanupExpiredEnrichments();
        log.info("Cleanup completed");
    } catch (Exception e) {
        failedCounter.increment();
        log.error("Cleanup failed", e);
    }
}

No dedicated metric for cleanup. It runs once a day. If it fails, the shared counter catches it. Over-instrumenting a daily job that takes 2 seconds would be pure vanity metrics.

Step 5: Test It

This is where SimpleMeterRegistry shines. It's an in-memory registry — no Prometheus, no actuator endpoint, no running server. Just a plain object you construct in tests:

@ExtendWith(MockitoExtension.class)
class CampaignEnrichmentSchedulerTest {

    @Mock
    private CampaignEnrichmentService service;

    private MeterRegistry meterRegistry;
    private CampaignEnrichmentScheduler scheduler;

    @BeforeEach
    void setUp() {
        meterRegistry = new SimpleMeterRegistry();
        scheduler = new CampaignEnrichmentScheduler(service, meterRegistry);
    }

    @Test
    void processBatch_incrementsProcessedCounter() {
        when(service.processBatch(50)).thenReturn(10);

        scheduler.processBatch();

        assertThat(meterRegistry.counter("campaign.enrichment.processed").count())
            .isEqualTo(10.0);
    }

    @Test
    void processBatch_recordsBatchDuration() {
        when(service.processBatch(50)).thenReturn(5);

        scheduler.processBatch();

        Timer timer = meterRegistry.timer("campaign.enrichment.batch.duration");
        assertThat(timer.count()).isEqualTo(1);
        assertThat(timer.totalTime(TimeUnit.MILLISECONDS)).isGreaterThan(0);
    }

    @Test
    void processBatch_incrementsFailedCounterOnException() {
        when(service.processBatch(50)).thenThrow(new RuntimeException("db down"));

        scheduler.processBatch();

        assertThat(meterRegistry.counter("campaign.enrichment.failed").count())
            .isEqualTo(1.0);
    }

    @Test
    void watchdog_incrementsRecoveredCounter() {
        when(service.recoverStuckCampaigns()).thenReturn(3);

        scheduler.watchdog();

        assertThat(meterRegistry.counter("campaign.enrichment.recovered").count())
            .isEqualTo(3.0);
    }
}

Notice the explicit constructor: new CampaignEnrichmentScheduler(service, meterRegistry). This is why we dropped @RequiredArgsConstructor. We can inject a SimpleMeterRegistry without spinning up a Spring context. Pure unit tests, fast and deterministic.

The counter assertions use .count() which returns a double. Micrometer counters are always doubles internally, even when you increment by int values. So isEqualTo(10.0) not isEqualTo(10).

What We Deliberately Did Not Do

No custom tags. Tags like method=processBatch or status=success are useful at scale. We have one scheduler with three methods. Tags would add cardinality for no value.
No Gauge for "currently processing." Tempting, but our batches take seconds. A gauge that's 1 for 3 seconds and 0 for 897 seconds is useless.
No @Timed annotation. Micrometer provides @Timed for auto-instrumenting methods. It works, but it requires the TimedAspect bean, AOP configuration, and you lose control over what gets recorded on exceptions. Explicit Timer.record() is three lines and crystal clear.
No Prometheus/Grafana setup in this post. That's infrastructure, not instrumentation. Get the metrics right first.

The Complete Scheduler

@Slf4j
@Service
public class CampaignEnrichmentScheduler {

    private final CampaignEnrichmentService service;
    private final Counter processedCounter;
    private final Counter failedCounter;
    private final Counter recoveredCounter;
    private final Timer batchTimer;

    public CampaignEnrichmentScheduler(
            CampaignEnrichmentService service,
            MeterRegistry meterRegistry) {
        this.service = service;
        this.processedCounter = meterRegistry.counter("campaign.enrichment.processed");
        this.failedCounter = meterRegistry.counter("campaign.enrichment.failed");
        this.recoveredCounter = meterRegistry.counter("campaign.enrichment.recovered");
        this.batchTimer = meterRegistry.timer("campaign.enrichment.batch.duration");
    }

    @Scheduled(fixedRate = 900_000)
    public void processBatch() {
        try {
            int processed = batchTimer.record(() -> service.processBatch(50));
            processedCounter.increment(processed);
            log.info("Enriched {} campaigns", processed);
        } catch (Exception e) {
            failedCounter.increment();
            log.error("Batch processing failed", e);
        }
    }

    @Scheduled(fixedRate = 900_000)
    public void watchdog() {
        try {
            int recovered = service.recoverStuckCampaigns();
            if (recovered > 0) {
                recoveredCounter.increment(recovered);
                log.warn("Recovered {} stuck campaigns", recovered);
            }
        } catch (Exception e) {
            failedCounter.increment();
            log.error("Watchdog failed", e);
        }
    }

    @Scheduled(cron = "0 0 4 * * *")
    public void cleanup() {
        try {
            service.cleanupExpiredEnrichments();
            log.info("Cleanup completed");
        } catch (Exception e) {
            failedCounter.increment();
            log.error("Cleanup failed", e);
        }
    }
}

Four metrics. One explicit constructor. Tests that run in milliseconds. No YAML. No aspect magic. No over-engineering.

What metrics do you track on your scheduled tasks? If the answer is "just logs," I get it — we were there too.

Building jo4.io — a URL shortener with analytics and an affiliate marketplace. Our schedulers now have actual observability.

How a 60-Day Sliding Window Flooded Our Slack with 500 Duplicate Alerts

Anand Rathnas — Mon, 03 Aug 2026 04:35:30 +0000

I woke up to 47 Slack notifications. All from the same bot. All saying the same thing: "Duplicate on dev.to for [article] - already exists, skipping."

Every single CI run — push triggers, daily crons, all of them — was firing the same warnings for the same posts. Over and over. For weeks.

TL;DR: Our crosspost script had a 60-day sliding window for fetching dev.to articles. As posts aged out of that window, the script forgot they existed, tried to recreate them, got 422 duplicate errors, and Slacked us about it. Every. Single. Run. The fix was to remove the time window entirely and fetch all published articles with pagination.

The Architecture That Worked (For a While)

Our dev.to crosspost script had two time-based filters:

const CONFIG = {
  localMaxDays: parseInt(process.env.LOCAL_MAX_DAYS, 10) || 30,
  devtoMaxDays: parseInt(process.env.DEVTO_MAX_DAYS, 10) || 60,
};

The idea was simple:

LOCAL_MAX_DAYS (30 days): Only process local posts from the last 30 days. Don't waste time on old articles.
DEVTO_MAX_DAYS (60 days): Only fetch dev.to articles from the last 60 days. Don't paginate through years of content.

The 60-day window was intentionally wider than 30 days. A post from day 29 would still be found in the dev.to lookback window. Plenty of margin.

This worked perfectly for months.

The Invariant That Broke

We had a feature called publishAfter — a frontmatter field that lets you schedule posts for future publication. When a post's publishAfter date arrives, the script processes it regardless of LOCAL_MAX_DAYS.

See the problem? publishAfter bypasses the 30-day local filter. A post from 90 days ago with publishAfter set would still get processed. But the dev.to lookback only goes back 60 days.

Here's what happened:

Day 1:   Post published. publishAfter = "2026-01-15"
Day 1:   Script runs → creates article on dev.to
Day 60:  Article falls out of DEVTO_MAX_DAYS window
Day 61:  Script can't find it in dev.to cache
Day 61:  publishAfter bypasses LOCAL_MAX_DAYS → post is processed
Day 61:  Script tries to CREATE → dev.to returns 422 (duplicate)
Day 61:  Slack notification: "Duplicate on dev.to, skipping"
Day 62:  Same thing
Day 63:  Same thing
Day 400: Still the same thing

The 422 was handled gracefully — we caught it, logged it, sent a Slack warning, and moved on. The script didn't crash. It just spammed us relentlessly.

The Feedback Loop That Made It Worse

But wait, it gets better.

We had another "helpful" feature: when the script detected a post with a stale publishAfter date (older than DEVTO_MAX_DAYS), it would auto-update the date to today and commit the change to git.

if (this.isOlderThanDevtoMaxDays(publishAfter)) {
  // Update publishAfter to today
  return { shouldProcess: true, needsDateUpdate: true, newDate: today };
}

Here's the loop this created:

Old post has publishAfter: "2026-01-15"
Script detects it's older than 60 days
Script updates it to publishAfter: "2026-06-15" (today)
Script commits the change to git
Git push triggers CI
CI runs the script again
The post now has today's date, so it gets processed
Script can't find it on dev.to (still outside the 60-day window of the original publish date on dev.to)
Script tries to CREATE, gets 422, sends Slack warning
Repeat from step 2 next time the date gets stale again

The auto-update was supposed to prevent this exact problem. Instead, it created a commit-push-CI loop that amplified it.

Debugging: Following the Notifications Upstream

The Slack messages were the symptom. I started there:

[warn] Duplicate on dev.to for "Getting Started with Jo4" - already exists, skipping
[warn] Duplicate on dev.to for "Jo4 Origin Story" - already exists, skipping
[warn] Duplicate on dev.to for "Referral Tracking for Indie Hackers" - already exists, skipping

All old posts. All definitely already on dev.to. So why was the script trying to create them?

I checked the dev.to article cache. These posts weren't in it. Because they were published more than 60 days ago, and DEVTO_MAX_DAYS filtered them out.

Then I checked why the script was processing them at all. They were older than LOCAL_MAX_DAYS (30 days). Shouldn't they be skipped? No — publishAfter was set, and the publishAfter logic bypassed the LOCAL_MAX_DAYS check.

Two time windows. Two bypass mechanisms. One broken invariant.

The Fix: Kill the Time Window

The root cause wasn't the publishAfter bypass or the auto-update commit loop. Those were symptoms. The root cause was DEVTO_MAX_DAYS — a premature optimization that created a blind spot.

// Before: only fetch last 60 days
async fetchDevtoArticles() {
  const response = await fetch(
    `${DEVTO_API_URL}/me/published?per_page=100`,
    { headers: { 'api-key': this.apiKey } }
  );
  const articles = await response.json();
  // Filter to last DEVTO_MAX_DAYS...
}

// After: fetch ALL articles with pagination
async fetchDevtoArticles() {
  let page = 1;
  let articles;

  do {
    const response = await fetch(
      `${DEVTO_API_URL}/me/published?per_page=100&page=${page}`,
      { headers: { 'api-key': this.apiKey } }
    );

    if (!response.ok) {
      throw new Error(`dev.to API error: ${response.status}`);
    }

    articles = await response.json();

    for (const article of articles) {
      if (article.canonical_url) {
        this.devtoArticles.set(article.canonical_url, {
          id: article.id,
          url: article.url,
          title: "article.title,"
          edited_at: article.edited_at,
        });
      }
    }

    page++;
  } while (articles.length === 100);
}

Three changes:

Removed DEVTO_MAX_DAYS entirely. Fetch all published articles. With pagination, this is a few extra API calls at most. We have ~50 articles — one page.
Removed the publishAfter auto-update logic. No more auto-committing date changes. No more commit-push-CI loops. If a post is old, it's old. That's fine.
Added fail-fast on API errors. The old code silently returned partial data if a pagination request failed. Now it throws. Partial data is worse than no data — it creates the same blind spot we just fixed.

// Before: silent return with partial data
if (!response.ok) {
  console.error(`API error: ${response.status}`);
  return; // script continues with incomplete cache
}

// After: fail-fast
if (!response.ok) {
  throw new Error(`dev.to API error: ${response.status}`);
}

Why Not Just Increase DEVTO_MAX_DAYS?

Tempting. Set it to 365 days and call it a day. But that's patching the symptom:

It delays the problem by a year instead of fixing it.
The publishAfter bypass still violates the invariant.
The auto-update commit loop still exists.
You're betting that no post will ever be older than your magic number.

Fetching all articles is the correct solution. The performance cost is negligible — dev.to's API returns 100 articles per page, and pagination adds maybe 200ms per extra page. We'd need 1,000+ articles before this matters.

The Lesson: Time Windows Need Invariants

Two time windows that are "designed to align" will eventually drift. Especially when other features bypass one window but not the other.

If you have a cache with a time-based eviction window, ask yourself:

Can anything bypass the filter that feeds into this cache?
What happens when an item falls out of the cache but is still referenced?
Is partial data from this cache worse than no data?

For us, the answers were: yes, Slack spam, and absolutely.

The 60-day window saved us approximately zero performance. It cost us weeks of duplicate alerts and a debugging session that touched every part of the pipeline. Premature optimization, meet production consequences.

Have you been burned by time-based caching creating blind spots? I'd love to hear your war story.

Building jo4.io — a URL shortener with analytics, bio pages, and an affiliate marketplace. Our blog crosspost pipeline no longer floods Slack.

Embedding Stats Widgets Without Breaking CSP: iframes, CORS, and Content Security Policy

Anand Rathnas — Fri, 31 Jul 2026 04:27:32 +0000

I wanted to let users embed their link stats on their own websites. Should be simple — render some numbers in an iframe, done. Right?

Fourteen hundred lines of code later, I had a working embed system with layered CSP control, server-side sparklines, and per-entity framing permissions.

Here's everything I learned.

TL;DR

Embeddable widgets need layered framing control: a default CSP via a security filter, then surgical per-response overrides in the controller. Public embeds get frame-ancestors *; disabled or missing ones get frame-ancestors 'none' plus X-Frame-Options: DENY. Compute expensive visuals (like sparklines) server-side to keep the iframe lightweight.

What We're Building

A stats widget that users can drop into their site with a single iframe tag:

<iframe
  src="https://jo4.io/embed/stats/abc123?days=30"
  width="400"
  height="200"
  frameborder="0"
></iframe>

It shows click counts, a sparkline chart, and basic analytics for a shortened URL. The owner controls whether the embed is public or disabled via a toggle in their dashboard.

Step 1: The Backend — EmbedController

The controller is the brains of the operation. It handles the request, checks permissions, sets security headers, and returns the stats data.

Here's the core structure (simplified from 232 lines):

@RestController
@RequestMapping("/embed")
@RequiredArgsConstructor
public class EmbedController {

    private final UrlService urlService;
    private final EmbedStatsCacheService embedStatsCacheService;

    private static final String CSP_FRAME_ANCESTORS_SELF = "frame-ancestors 'self'";

    @GetMapping("/stats/{shortCode}")
    public ResponseEntity<EmbedStatsResponse> getStats(
            @PathVariable String shortCode,
            @RequestParam(defaultValue = "30") String daysParam,
            HttpServletResponse response) {

        int days = parseDaysSafely(daysParam);
        UrlEntity url = urlService.findByShortCode(shortCode);

        if (url == null || !url.isEnablePublicStats()) {
            setDenyFraming(response);
            return ResponseEntity.notFound().build();
        }

        setAllowFraming(response);

        EmbedStatsResponse stats = embedStatsCacheService
            .getStats(shortCode, days);

        return ResponseEntity.ok(stats);
    }
}

The key methods are setAllowFraming and setDenyFraming. This is where the layered CSP control happens.

Step 2: Layered Framing Control

Here's the architecture for controlling who can iframe your content:

Layer 1: Security filter sets a safe default. SecurityHeadersConfig adds frame-ancestors 'self' to all /embed/ paths. This means by default, embeds can only be framed by your own domain.

@Component
public class SecurityHeadersConfig implements Filter {

    @Override
    public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
            throws IOException, ServletException {

        HttpServletResponse response = (HttpServletResponse) res;
        HttpServletRequest request = (HttpServletRequest) req;

        if (request.getRequestURI().startsWith("/embed/")) {
            response.setHeader("Content-Security-Policy",
                "frame-ancestors 'self'");
        }

        chain.doFilter(req, res);
    }
}

Layer 2: Controller surgically replaces the directive. After the filter runs, the controller overrides the CSP based on the entity's permissions:

private void setAllowFraming(HttpServletResponse response) {
    // Replace the default 'self' with wildcard — allow anyone to embed
    String currentCsp = response.getHeader("Content-Security-Policy");
    if (currentCsp != null) {
        response.setHeader("Content-Security-Policy",
            currentCsp.replace(CSP_FRAME_ANCESTORS_SELF, "frame-ancestors *"));
    }
    // Remove X-Frame-Options entirely — it conflicts with frame-ancestors
    response.setHeader("X-Frame-Options", "");
}

private void setDenyFraming(HttpServletResponse response) {
    // Replace with 'none' — block all framing
    String currentCsp = response.getHeader("Content-Security-Policy");
    if (currentCsp != null) {
        response.setHeader("Content-Security-Policy",
            currentCsp.replace(CSP_FRAME_ANCESTORS_SELF, "frame-ancestors 'none'"));
    }
    // Belt and suspenders — also set X-Frame-Options for older browsers
    response.setHeader("X-Frame-Options", "DENY");
}

Why this layered approach instead of just setting CSP in the controller?

Defense in depth. If someone adds a new embed endpoint and forgets to set framing headers, the filter's default 'self' prevents open framing.
Single override point. The String.replace on CSP_FRAME_ANCESTORS_SELF ensures we're modifying a known value, not blindly appending directives.
No conflicts. One CSP header with one frame-ancestors directive, not multiple headers fighting each other.

Step 3: Caching with EmbedStatsCacheService

Stats queries are expensive — they aggregate click data over time ranges. You don't want to run that query every time someone loads a page with an embedded widget.

@Service
@RequiredArgsConstructor
public class EmbedStatsCacheService {

    private final StatsRepository statsRepository;
    private final Cache<String, EmbedStatsResponse> cache;

    public EmbedStatsResponse getStats(String shortCode, int days) {
        String cacheKey = shortCode + ":" + days;

        return cache.get(cacheKey, key -> {
            List<DailyClickCount> clicks = statsRepository
                .getDailyClicks(shortCode, days);

            return EmbedStatsResponse.builder()
                .totalClicks(clicks.stream()
                    .mapToLong(DailyClickCount::getCount).sum())
                .sparklineSvgPoints(computeSparkline(clicks))
                .days(days)
                .build();
        });
    }
}

The cache key is shortCode:days, so different time ranges are cached independently. The cache TTL is short enough (5 minutes) that stats stay reasonably fresh without hammering the database.

Step 4: Server-Side Sparklines

Here's one I'm proud of. The sparkline chart in the embed widget is computed entirely server-side as an SVG points string.

Why not use a JavaScript charting library in the iframe? Three reasons:

Bundle size. Even a lightweight charting library adds 30-50KB. For an iframe that shows one tiny sparkline, that's absurd.
Load time. The iframe should render instantly. No waiting for JavaScript bundles to download and parse.
CSP simplicity. No script-src directives to worry about in the iframe's CSP.

The server computes the SVG polyline points:

private String computeSparkline(List<DailyClickCount> clicks) {
    if (clicks.isEmpty()) return "";

    long maxCount = clicks.stream()
        .mapToLong(DailyClickCount::getCount)
        .max().orElse(1);

    double width = 200.0;
    double height = 40.0;

    StringBuilder points = new StringBuilder();
    for (int i = 0; i < clicks.size(); i++) {
        double x = (i * width) / (clicks.size() - 1);
        double y = height - (clicks.get(i).getCount() * height / maxCount);
        if (i > 0) points.append(" ");
        points.append(String.format("%.1f,%.1f", x, y));
    }

    return points.toString();
}

The frontend renders it with a single <polyline> element:

<svg viewBox="0 0 200 40" class="sparkline">
  <polyline
    points="${sparklineSvgPoints}"
    fill="none"
    stroke="#3b82f6"
    stroke-width="1.5"
  />
</svg>

No JavaScript. No charting library. Just an SVG string from the server and a few bytes of HTML.

Step 5: Frontend — Shared StatsWidget

On the React side, I originally had two separate components: BioStatsWidget.tsx and UrlStatsWidget.tsx. They were 80% identical. Classic copy-paste debt.

I refactored both into a single shared StatsWidget.tsx:

interface StatsWidgetProps {
  shortCode: string;
  days?: number;
  variant: 'bio' | 'url';
}

export function StatsWidget({ shortCode, days = 30, variant }: StatsWidgetProps) {
  const { data, isLoading } = useEmbedStats(shortCode, days);

  return (
    <div className={`stats-widget stats-widget--${variant}`}>
      <div className="stats-widget__count">
        {isLoading ? '...' : data?.totalClicks.toLocaleString()}
      </div>
      {data?.sparklineSvgPoints && (
        <svg viewBox="0 0 200 40" className="stats-widget__sparkline">
          <polyline
            points={data.sparklineSvgPoints}
            fill="none"
            stroke="currentColor"
            strokeWidth="1.5"
          />
        </svg>
      )}
    </div>
  );
}

The variant prop controls styling differences between bio page stats and URL stats. Same data, same logic, different visual treatment.

The Full Picture

Here's how all the pieces fit together:

User's website
  └── <iframe src="jo4.io/embed/stats/abc123">
        │
        ├── SecurityHeadersConfig: sets frame-ancestors 'self' (default)
        ├── EmbedController: overrides to frame-ancestors * (public embed)
        ├── EmbedStatsCacheService: returns cached stats
        ├── Sparkline: server-computed SVG points
        └── Response: JSON + correct CSP headers

The iframe loads fast (no JS charting library), respects CSP (no inline scripts), and has granular framing control (per-entity, not global).

Common Pitfalls

Setting both CSP frame-ancestors and X-Frame-Options with conflicting values. Modern browsers prefer frame-ancestors, but older ones fall back to X-Frame-Options. Make sure they agree — or remove X-Frame-Options when you want permissive framing.

Forgetting that CSP applies to the framed page, not the framing page. The frame-ancestors directive goes on the response from jo4.io/embed/stats/*, not on the parent page. This is the embed's way of saying "these domains are allowed to frame me."

Caching responses with per-entity headers. If you put a CDN in front of your embed endpoint, make sure the cache key includes the entity's framing permission. Otherwise, a cached public-embed response might get served for a disabled embed, or vice versa.

Have you built embeddable widgets before? What was the trickiest part — CSP, styling, or something else entirely?

Building jo4.io - a URL shortener with embeddable stats widgets that actually respect your security policy.

The Security Audit That Found 4 Issues in One Weekend

Anand Rathnas — Tue, 28 Jul 2026 04:15:25 +0000

I spent a weekend doing a security audit on jo4. I expected to find maybe one minor issue. I found four, and one of them was leaking stack traces to users.

Here's the full damage report.

TL;DR

Spring Security's default security headers are great — until they break your features. Blanket X-Frame-Options: DENY kills embeddable widgets, unchecked query parameters leak stack traces, and default CSP can conflict with your custom headers. The fix isn't disabling security — it's taking surgical control.

Issue 1: X-Frame-Options Nuking All Embeds

Spring Security auto-injects X-Frame-Options: DENY on every response. Every. Single. One.

For most endpoints, that's exactly what you want. You don't want someone embedding your login page in a malicious iframe. But jo4 has embed widgets — public stats dashboards that users can iframe into their own sites.

// What Spring Security was doing to ALL responses
X-Frame-Options: DENY

This meant every embed widget was broken. Browsers refused to render them in iframes. The feature existed, the code worked, but Spring Security was silently killing it at the HTTP header level.

The wrong fix would be to just disable X-Frame-Options globally. That opens you up to clickjacking on every page.

The right fix: disable Spring's blanket header and delegate the decision to the controller that actually knows what should be embeddable.

@Configuration
public class SecurityConfig {
    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        http
            .headers(h -> h.frameOptions(f -> f.disable()))
            // ... rest of security config
        ;
        return http.build();
    }
}

Then in EmbedController, set the header dynamically based on whether the entity has enablePublicStats turned on:

@GetMapping("/embed/stats/{shortCode}")
public ResponseEntity<EmbedResponse> getEmbedStats(
        @PathVariable String shortCode,
        HttpServletResponse response) {

    UrlEntity url = urlService.findByShortCode(shortCode);

    if (url != null && url.isEnablePublicStats()) {
        // Allow embedding anywhere
        response.setHeader("X-Frame-Options", "ALLOWALL");
    } else {
        // Lock it down
        response.setHeader("X-Frame-Options", "DENY");
    }

    // ... return embed data
}

Per-entity control. Public embeds work. Everything else stays locked. Spring Security isn't in the way anymore, but we didn't sacrifice anything.

Issue 2: Stack Trace Leak via Type Binding

The EmbedController had a days parameter for controlling the stats time range:

@GetMapping("/embed/stats/{shortCode}")
public ResponseEntity<EmbedResponse> getEmbedStats(
        @PathVariable String shortCode,
        @RequestParam int days) {  // <-- Problem

Typed as int. Seems fine. But what happens when someone hits:

/embed/stats/abc123?days=foo

Spring's type binding can't convert "foo" to int. It throws a TypeMismatchException. And by default, Spring returns a 400 with the full exception message — including internal class names, method signatures, and sometimes stack traces.

{
  "status": 400,
  "error": "Bad Request",
  "message": "Failed to convert value of type 'java.lang.String' to required type 'int'; For input string: \"foo\""
}

That's an information leak. An attacker now knows you're running Spring, using Java, and can start probing for known vulnerabilities in those versions.

The fix is simple — accept a String and parse it safely:

@GetMapping("/embed/stats/{shortCode}")
public ResponseEntity<EmbedResponse> getEmbedStats(
        @PathVariable String shortCode,
        @RequestParam(defaultValue = "30") String daysParam) {

    int days;
    try {
        days = Integer.parseInt(daysParam);
        if (days < 1 || days > 365) {
            days = 30;
        }
    } catch (NumberFormatException e) {
        days = 30; // Graceful default, no stack trace
    }

    // ... proceed with valid days value
}

No stack trace. No information leak. Bad input gets a sensible default. The user sees their stats for 30 days instead of an error page.

Issue 3: CSP Header Conflicts

Spring Security has its own Content Security Policy defaults. I also had custom security headers configured. The result: two CSP headers on the same response, and browsers were picking the more restrictive one.

The fix was consolidating all security headers into a single SecurityHeadersConfig instead of letting Spring inject defaults and then layering custom headers on top. One source of truth for CSP, not two fighting each other.

Issue 4: Duplicate Security Configs

This one was embarrassing. I had two security configuration classes — SecurityAuth0Config and SecurityConfig — both configuring HTTP security. Both needed the .headers(h -> h.frameOptions(f -> f.disable())) change.

I updated SecurityConfig and thought I was done. Embeds were still broken in production. Spent 30 minutes debugging before I realized there was a second security config class that was still injecting X-Frame-Options: DENY.

// SecurityAuth0Config.java — the one I forgot about
@Configuration
public class SecurityAuth0Config {
    @Bean
    public SecurityFilterChain auth0FilterChain(HttpSecurity http) throws Exception {
        http
            .headers(h -> h.frameOptions(f -> f.disable()))  // Need this here too!
            // ... auth0-specific config
        ;
        return http.build();
    }
}

The lesson: if you have multiple SecurityFilterChain beans, every single one independently configures headers. Missing one means that filter chain still uses Spring's defaults.

The Bigger Lesson

Framework security defaults are a double-edged sword. They protect you from things you haven't thought about. But they also apply blanket policies that don't understand your specific features.

The answer isn't "disable Spring Security defaults." The answer is:

Understand what the defaults do — read the response headers your app is actually sending
Take surgical control where defaults conflict with features
Test the edges — the weird query parameters, the iframe scenarios, the multi-config interactions
Don't assume one security config covers everything — especially with multiple filter chains

I found all four of these in one weekend because I finally sat down and curl -v'd my own endpoints instead of just clicking through the UI. The UI looked fine. The headers told a different story.

When was the last time you actually inspected the response headers your framework is sending? You might be surprised what's in there.

Building jo4.io - a URL shortener where the security headers actually make sense now.

When Your SPA Eats Your API Routes: Fixing the Cloudflare Worker vs React Router Conflict

Anand Rathnas — Sat, 25 Jul 2026 04:16:31 +0000

My affiliate tracking links were returning the entire React app instead of a 302 redirect.

Not a 404. Not a 500. A perfectly happy 200 with my SPA's HTML shell. As if the browser was saying: "Here's your single-page application! You wanted a redirect? Never heard of her."

TL;DR

If you serve a React SPA from Cloudflare/CloudFront as a static bundle, every URL that doesn't match a static file falls through to index.html. That includes your API-style routes like /go/*. Fix it by adding a routing branch in your Cloudflare Worker that intercepts those paths at the edge before the SPA bundle even loads.

The Setup

Jo4 has an affiliate tracking feature. When someone clicks a /go/partner-slug link, the server is supposed to:

Record the click (timestamp, referrer, etc.)
Return a 302 Found redirect to the partner's destination URL

Simple HTTP. The browser follows the redirect, the user lands on the partner's page, and we have a clean record of the click.

The backend API at jo4-api.jo4.io handles this perfectly. The endpoint receives the request, logs the event, and returns:

HTTP/1.1 302 Found
Location: https://partner-site.com/landing

The problem? Those requests never reached the backend.

What Was Actually Happening

Here's the request flow I expected:

Browser -> Cloudflare -> Backend API -> 302 redirect

Here's what was actually happening:

Browser -> Cloudflare -> Static SPA bundle -> 200 OK (index.html)

The SPA is deployed as a static bundle behind Cloudflare. Cloudflare's default behavior for SPAs is to serve index.html for any path that doesn't match a static file. That's the whole point of a SPA — React Router handles routing client-side.

But React Router had no route for /go/*. It didn't need one. These aren't client-side routes. They're server-side endpoints that need to hit the API.

So when someone clicked jo4.io/go/partner-slug:

Cloudflare received the request
No static file matched /go/partner-slug
Cloudflare served index.html (SPA fallback)
Browser got a 200 OK with HTML
React Router loaded, found no matching route
User saw... the app. Not a redirect. The app.

The affiliate click was never recorded. The partner never got their traffic. Nobody was happy.

The Fix: Edge Routing in the Cloudflare Worker

The key insight is embarrassingly simple once you see it: the routing decision needs to happen at the edge, before the SPA bundle is even considered.

Cloudflare Workers sit in front of everything. They see every request before it hits your static files, your origin, anything. That's where the routing belongs.

// Cloudflare Worker - simplified routing
async function handleRequest(request) {
  const url = new URL(request.url);

  // Affiliate tracking routes - proxy to backend API
  if (url.pathname.startsWith('/go/')) {
    const apiUrl = new URL(url.pathname, 'https://jo4-api.jo4.io');
    apiUrl.search = url.search;

    return fetch(apiUrl.toString(), {
      method: request.method,
      headers: request.headers,
      redirect: 'manual', // Don't follow the redirect — pass it through
    });
  }

  // Everything else — serve the SPA
  return fetch(request);
}

The redirect: 'manual' part is important. Without it, the Worker would follow the 302 itself and return the final page content. We want to pass the 302 straight through to the browser so it handles the redirect.

After the Fix

Browser -> Cloudflare Worker -> intercepts /go/* -> proxies to jo4-api.jo4.io -> 302 redirect
Browser -> Cloudflare Worker -> everything else -> serves SPA

The SPA never sees /go/* requests. They're handled and done before React even loads.

I also updated the AffiliateNotificationService URLs to use the new route structure, so notification emails and Slack messages all point to the correct /go/ paths.

Why This Bit Me

I've been building web apps for years. I know how SPA routing works. But here's the thing — when you're moving fast and deploying a new feature, you test the feature. You test the affiliate link creation, the click tracking endpoint, the redirect logic. You test it all through the API directly.

What you don't test is: "What happens when a real browser hits this URL through the full CDN/SPA stack?"

I was testing jo4-api.jo4.io/go/partner-slug directly. That worked fine. The bug only appeared when hitting jo4.io/go/partner-slug — through the SPA's Cloudflare distribution.

The Lesson

If you're running a SPA behind Cloudflare (or CloudFront, or any CDN with SPA fallback):

Your SPA will eat every URL that doesn't match a static file
Server-side routes need edge-level interception — don't rely on the SPA to "not handle" them
Test through the full stack, not just the API directly
Cloudflare Workers are the right place for this routing — they execute before anything else

The SPA fallback behavior is a feature, not a bug. It's doing exactly what it's supposed to do. The problem is assuming that "routes I didn't define in React Router" will somehow magically reach your backend. They won't. The CDN doesn't know the difference between a client-side route and a server-side route. You have to tell it.

Ever had your SPA swallow routes meant for your backend? What was your fix — Workers, nginx rules, or something else?

Building jo4.io - a URL shortener with affiliate tracking that actually redirects now.

Testing Cloudflare Workers Without Cloudflare: A Vitest Setup That Actually Works

Anand Rathnas — Thu, 23 Jul 2026 04:19:43 +0000

I added 3,059 lines of tests to our Cloudflare Worker. Not a single line uses miniflare. No Workers runtime simulation. Just Vitest, vi.stubGlobal('fetch', mockFetch), and focused assertions on the things that actually break in production.

Here's why that's the right call and how to set it up.

TL;DR

Cloudflare Workers are just JavaScript functions that receive a Request and return a Response. You don't need the Workers runtime to test most of the logic. Stub fetch, write unit tests for routing, caching headers, and redirect handling. Save integration tests for deployment verification. This approach is faster, simpler, and catches the bugs that miniflare would miss anyway.

The Setup

A Cloudflare Worker's entry point looks like this:

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);

    if (url.pathname.startsWith('/go/')) {
      return proxyTrackingRequest(request, env);
    }

    if (url.pathname.startsWith('/blog/')) {
      return proxyBlogRequest(request, env);
    }

    return proxyDefaultRequest(request, env);
  }
};

It's a function. It takes a Request. It returns a Response. You can test this without Cloudflare's runtime.

Here's the Vitest setup:

// vitest.config.js
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    environment: 'node',
    globals: true,
  },
});

No special Workers environment. No miniflare plugin. Just Node.

Stubbing Fetch

The Worker delegates to your backend via fetch. Stub it:

import { describe, it, expect, vi, beforeEach } from 'vitest';
import worker from '../src/index';

describe('Cloudflare Worker', () => {
  let mockFetch;

  beforeEach(() => {
    mockFetch = vi.fn();
    vi.stubGlobal('fetch', mockFetch);
  });

  afterEach(() => {
    vi.restoreAllMocks();
  });
});

vi.stubGlobal('fetch', mockFetch) replaces the global fetch that the Worker calls internally. Your tests control exactly what the backend returns.

The Test That Caught a Production Bug

This is the test I'm most proud of:

it('does NOT set Cache-Control on tracking routes - clicks must not be cached', async () => {
  mockFetch.mockResolvedValue(new Response('', {
    status: 302,
    headers: { 'Location': 'https://merchant.com/product' }
  }));

  const request = new Request('https://jo4.io/go/abc123');
  const response = await worker.fetch(request, mockEnv, mockCtx);

  // Verify no cache headers on tracking routes
  expect(response.headers.get('Cache-Control')).toBeNull();
  expect(response.headers.get('CDN-Cache-Control')).toBeNull();
});

Why does this matter? Our tracking links (/go/*) record clicks. Every hit to /go/abc123 creates a unique jo4_cid (click ID) and records the visit. If Cloudflare caches the 302 redirect, subsequent clicks skip the backend entirely. The publisher's click count flatlines. The brand sees zero conversions. Everyone loses money.

Blog routes get cached aggressively. Tracking routes must never be cached. One misplaced Cache-Control header and affiliate tracking silently breaks.

This test runs in 2ms. It would catch a regression the moment someone refactors the proxy logic and accidentally applies caching globally.

Testing Redirect Handling

Another critical test: verifying the Worker doesn't follow redirects.

it('passes redirect: manual to fetch for tracking routes', async () => {
  mockFetch.mockResolvedValue(new Response('', {
    status: 302,
    headers: { 'Location': 'https://merchant.com/product' }
  }));

  const request = new Request('https://jo4.io/go/abc123');
  await worker.fetch(request, mockEnv, mockCtx);

  // The Worker must NOT follow the redirect itself
  expect(mockFetch).toHaveBeenCalledWith(
    expect.any(String),
    expect.objectContaining({
      redirect: 'manual'
    })
  );
});

Without { redirect: 'manual' }, the Worker follows the 302 to the merchant's site. The fetch call resolves to the merchant's HTML page instead of the redirect response. The click event is recorded, but the Worker returns a 200 with the merchant's HTML instead of a 302 that the browser follows. The user sees the merchant page inside the Worker's response context. It's a mess.

redirect: 'manual' tells fetch to return the 302 as-is. The Worker passes the redirect response through to the browser. The browser follows it. The click is tracked AND the redirect works correctly.

Testing Cache Headers on Blog Routes

The inverse of the tracking test: blog routes SHOULD be cached.

it('sets Cache-Control on blog routes', async () => {
  mockFetch.mockResolvedValue(new Response('<html>Blog post</html>', {
    status: 200,
    headers: { 'Content-Type': 'text/html' }
  }));

  const request = new Request('https://jo4.io/blog/some-post/');
  const response = await worker.fetch(request, mockEnv, mockCtx);

  expect(response.headers.get('Cache-Control')).toBeTruthy();
});

Blog content is static. Caching it at the edge reduces origin load and improves page speed. The test ensures blog routes always get cache headers, even after refactoring.

Simplifying the Production Code

Writing these tests exposed unnecessary complexity in the Worker itself. The proxyRequest function had a conditional cache flag:

// Before: conditional caching logic
async function proxyRequest(request, env, shouldCache) {
  const response = await fetch(targetUrl, {
    redirect: shouldCache ? 'follow' : 'manual',
    // ... other options
  });

  if (shouldCache) {
    response.headers.set('Cache-Control', 'public, max-age=3600');
  }

  return response;
}

After writing the tests, I realized the caching decision is entirely determined by the route type. There's no scenario where a tracking route should be cached or a blog route shouldn't. The conditional flag was adding complexity without flexibility.

// After: separate functions, no conditionals
async function proxyTrackingRequest(request, env) {
  return fetch(targetUrl, { redirect: 'manual' });
  // No cache headers. Ever.
}

async function proxyBlogRequest(request, env) {
  const response = await fetch(targetUrl);
  response.headers.set('Cache-Control', 'public, max-age=3600');
  return response;
}

Simpler code. Easier to test. Impossible to accidentally cache a tracking route.

Why Not Miniflare?

Miniflare is excellent for integration testing. It simulates the Workers runtime, KV stores, Durable Objects, and other Cloudflare primitives. If your Worker uses KV heavily or relies on Durable Objects, miniflare is the right choice.

But for our Worker, the logic is straightforward: route requests, set headers, handle redirects. The bugs that bite us are:

Cache headers on the wrong routes
Missing redirect: 'manual' on tracking requests
Incorrect path matching

None of these require a simulated runtime. They're logic bugs. Unit tests with mocked fetch catch them instantly.

Miniflare adds startup time, dependencies, and configuration complexity. For 3,059 lines of tests, the faster feedback loop of pure Vitest matters.

The Numbers

Metric	Value
Lines of tests added	3,059
Test execution time	~1.2 seconds
Dependencies added	0 (Vitest was already installed)
Miniflare config files	0
Production bugs caught by cache header tests	1 (and counting)

The Pattern

If your Cloudflare Worker is primarily a proxy that routes, caches, and redirects:

Use vi.stubGlobal('fetch', mockFetch) to control backend responses
Test cache headers per route type explicitly
Test redirect handling with redirect: 'manual' assertions
Keep tests fast -- no runtime simulation needed
Save miniflare for Workers that use KV, D1, or Durable Objects

The goal isn't 100% simulation fidelity. It's catching the bugs that cost money. A cached tracking link is a silent revenue killer. A test that runs in 2ms and prevents it is worth more than a full miniflare integration suite that takes 30 seconds to boot.

How do you test your Cloudflare Workers? Miniflare, Vitest, or something else? I'm genuinely curious if others have gone the lightweight route.

Building jo4.io -- affiliate tracking links that are tested to never be cached.

Why Auth0 E2E Tests Fail After Login: The localStorage Race Condition

Anand Rathnas — Mon, 20 Jul 2026 04:40:09 +0000

Your Playwright test logs into Auth0 successfully. The redirect completes. The app loads. You snapshot storageState() for subsequent tests. Every test after that gets 401 Unauthorized on API calls.

The login worked. The token didn't persist.

TL;DR

Auth0 SPA SDK v2 writes two keys to localStorage inside handleRedirectCallback(): the user identity and the access token. Playwright's waitForLoadState('networkidle') fires before these writes complete because they're synchronous DOM operations, not network requests. Snapshotting too early captures the user key but misses the access token. Fix: use page.waitForFunction() to poll for the exact access token key.

The Symptom

Here's what the test looked like:

// global-setup.ts
const page = await browser.newPage();
await page.goto('/');

// Auth0 redirects to login page
await page.fill('[name="username"]', process.env.TEST_USER);
await page.fill('[name="password"]', process.env.TEST_PASSWORD);
await page.click('[type="submit"]');

// Wait for the app to load after Auth0 redirects back
await page.waitForLoadState('networkidle');

// Save auth state for all tests
await page.context().storageState({ path: 'auth.json' });

This works 60% of the time. The other 40%, every subsequent test fails with 401.

The auth.json file holds the answer. On a failing run, it contains one localStorage key:

{
  "origins": [{
    "localStorage": [{
      "name": "@@auth0spajs@@::user::@@user@@",
      "value": "{\"sub\":\"auth0|abc123\",\"email\":\"test@example.com\"...}"
    }]
  }]
}

On a passing run, it contains two:

{
  "origins": [{
    "localStorage": [{
      "name": "@@auth0spajs@@::user::@@user@@",
      "value": "{\"sub\":\"auth0|abc123\"...}"
    }, {
      "name": "@@auth0spajs@@::your-client-id::your-audience::openid profile email",
      "value": "{\"body\":{\"access_token\":\"eyJ...\",\"token_type\":\"Bearer\"...}}"
    }]
  }]
}

The second key -- the one with the access token -- is missing on failing runs. The first key (user identity) is present either way.

The Root Cause

Auth0 SPA SDK v2 handles the OAuth callback in this sequence:

handleRedirectCallback() makes a network request to Auth0's /oauth/token endpoint to exchange the authorization code
The token response comes back
The SDK writes the user identity to localStorage (synchronous DOM operation)
The SDK writes the access token to localStorage (synchronous DOM operation)

Steps 3 and 4 are synchronous DOM writes. They're not network requests. Playwright's waitForLoadState('networkidle') triggers when there are no pending network requests for 500ms. That condition is met after step 2 completes -- the /oauth/token response arrived and there's nothing else in flight.

But steps 3 and 4 haven't executed yet. They're queued in the JavaScript event loop, waiting for the current microtask/macrotask to finish. networkidle fires, your test proceeds to storageState(), and you capture a snapshot mid-write.

Sometimes you catch both keys. Sometimes you only catch the first. It depends on exactly when the event loop gets to those writes relative to when Playwright reads localStorage. Classic race condition.

The Fix

14 lines:

async function waitForAuth0Token(page) {
  await page.waitForFunction(
    (clientId) => {
      const keys = Object.keys(localStorage);
      return keys.some(key =>
        key.startsWith('@@auth0spajs@@') &&
        key.includes(clientId) &&
        !key.includes('@@user@@')
      );
    },
    process.env.AUTH0_CLIENT_ID,
    { timeout: 30000 }
  );
}

This polls localStorage until the access token key appears. Not the user key (which arrives first and is useless on its own), but the key that contains the client ID and audience -- the one with the actual access token.

The updated setup:

// global-setup.ts
await page.click('[type="submit"]');
await page.waitForLoadState('networkidle');
await waitForAuth0Token(page);  // Wait for the ACTUAL token
await page.context().storageState({ path: 'auth.json' });

We keep waitForLoadState('networkidle') as a coarse wait for the page to settle, then add the precise wait for the specific localStorage key we need.

The waitForFunction Gotcha

One thing that cost me 30 minutes: the function signature.

// WRONG - opts is passed as the function's argument
await page.waitForFunction(fn, { timeout: 30000 });

// RIGHT - null is the function argument, opts is the third param
await page.waitForFunction(fn, null, { timeout: 30000 });

page.waitForFunction() takes three arguments: the function, the argument to pass to that function, and the options object. If you pass two arguments, Playwright treats the second as the function argument, not as options. Your timeout never gets applied, and you get the default 30-second timeout anyway (lucky coincidence in this case) or a confusing type error.

When you need to pass an actual argument and options:

await page.waitForFunction(
  (clientId) => { /* use clientId */ },
  process.env.AUTH0_CLIENT_ID,  // Passed to the function as clientId
  { timeout: 30000 }            // Playwright options
);

Why Not Just Add a Delay?

You could do this:

await page.waitForLoadState('networkidle');
await page.waitForTimeout(2000);  // "Should be enough"

It works until it doesn't. On a slow CI runner, 2 seconds isn't enough. On a fast local machine, you're wasting 2 seconds per run for no reason. Fixed delays in tests are a code smell that guarantees flakiness at scale.

The waitForFunction approach is deterministic. It resolves the moment the token appears, whether that's 50ms or 5 seconds after networkidle.

The Key Pattern

The Auth0 SPA SDK localStorage key follows this pattern:

@@auth0spajs@@::{clientId}::{audience}::{scope}

For example:

@@auth0spajs@@::abc123def456::https://api.jo4.io::openid profile email

The @@user@@ key follows a different pattern and only contains identity claims (sub, email, name). It does NOT contain the access token. If your test only needs to verify the user is logged in, the user key is sufficient. If your test makes API calls, you need the access token key.

Summary

What	Fires When	Good Enough?
`waitForLoadState('networkidle')`	No network requests for 500ms	No -- misses localStorage writes
`waitForTimeout(2000)`	After 2 seconds	Flaky -- arbitrary delay
`waitForFunction(checkLocalStorage)`	Token key exists in localStorage	Yes -- deterministic

The race condition exists because networkidle is a network-level signal, and localStorage writes are a DOM-level operation. They live in different layers. You need to wait at the layer where the state actually lives.

Hit this exact issue? Let me know in the comments -- I'm curious how many teams have independently discovered this race condition.

Building jo4.io -- E2E tested from Auth0 login to payout settlement.

Setting Up E2E Tests with Playwright + Auth0: The 5 Things That Took Longer Than the Tests Themselves

Anand Rathnas — Fri, 17 Jul 2026 04:12:12 +0000

We deleted approximately 10,000 lines of old EC2 Terraform infrastructure and replaced it with 1,637 lines of Playwright E2E tests that prove our entire "publisher gets paid" flow works -- from signup to settlement -- in seconds instead of months.

The tests themselves? Straightforward. The setup? Five separate rabbit holes, each deeper than the last.

TL;DR

Auth0 SPA SDK writes tokens to localStorage asynchronously after handleRedirectCallback(). Playwright's waitForLoadState('networkidle') fires before those writes complete. You need page.waitForFunction() to poll for the exact localStorage key. Also, you need a test-only controller gated by @ConditionalOnProperty to bypass time-based business logic. Here are all five things that cost me more time than writing the test assertions.

1. Auth0 Tokens and the localStorage Race

This was the biggest time sink. After Auth0 redirects back to your app, the SPA SDK calls handleRedirectCallback() which exchanges the authorization code for tokens. Those tokens get written to localStorage.

Playwright's waitForLoadState('networkidle') seems like the right thing to wait for. The token exchange is a network request, after all. But the localStorage writes that happen after the token exchange are synchronous DOM operations, not network requests. networkidle fires before they complete.

If you snapshot storageState() at this point, you capture a half-written auth state. The user key exists but the access token doesn't. Subsequent tests authenticate (they have the user identity) but get 401 on every API call (no access token).

The fix: poll for the exact key.

await page.waitForFunction(
  () => {
    const keys = Object.keys(localStorage);
    return keys.some(key =>
      key.startsWith('@@auth0spajs@@') &&
      key.includes('your-client-id') &&
      key.includes('your-audience')
    );
  },
  null,  // Must pass null as second arg
  { timeout: 30000 }
);

That three-argument form matters. waitForFunction(fn, opts) treats opts as the argument passed to the function. You need waitForFunction(fn, null, opts) to pass options to Playwright, not to your function.

I wrote a separate deep-dive on this localStorage race condition because it deserves its own post.

2. Bypassing the 14-Day Commission Hold

Jo4 has a 14-day hold on commissions before they're eligible for approval. This is a fraud prevention measure -- it gives brands time to detect invalid conversions. After approval, settlements run on a monthly cron.

For E2E tests, you can't wait 14 days. And you definitely can't wait for the first of next month.

The solution: a test-only Spring controller that's excluded from production builds at compile time.

@RestController
@RequestMapping("/api/v1/internal/test")
@ConditionalOnProperty(name = "jo4.e2e-test-mode", havingValue = "true")
public class TestSchedulerController {

    private final CommissionService commissionService;
    private final SettlementService settlementService;

    @PostMapping("/auto-approve-commissions")
    public ResponseEntity<Void> autoApproveCommissions() {
        commissionService.bulkAutoApproveExpiredPending(Long.MAX_VALUE);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/trigger-settlement")
    public ResponseEntity<Void> triggerSettlement() {
        settlementService.processMonthlySettlements();
        return ResponseEntity.ok().build();
    }
}

@ConditionalOnProperty("jo4.e2e-test-mode") means this controller doesn't even get instantiated unless the property is explicitly set to true. It's not just gated at runtime -- Spring doesn't register the bean at all. No risk of it existing in production.

The bulkAutoApproveExpiredPending(Long.MAX_VALUE) call is the clever bit. The method normally checks if commissions are older than 14 days. By passing Long.MAX_VALUE as the threshold, every commission qualifies as "expired" and gets auto-approved immediately.

3. The Full Flow Test

With the Auth0 race condition solved and the time-bypass controller in place, the E2E test proves the entire monetization pipeline:

test('publisher gets paid - full flow', async ({ page }) => {
  // 1. Publisher signs up and creates a tracking link
  await loginAsPublisher(page);
  const trackingLink = await createTrackingLink(page, campaignId);

  // 2. Simulate a click and conversion
  await page.goto(trackingLink);
  await simulateConversion(page, trackingLink);

  // 3. Bypass the 14-day hold
  await fetch(`${API_URL}/api/v1/internal/test/auto-approve-commissions`, {
    method: 'POST',
    headers: authHeaders
  });

  // 4. Trigger settlement
  await fetch(`${API_URL}/api/v1/internal/test/trigger-settlement`, {
    method: 'POST',
    headers: authHeaders
  });

  // 5. Verify publisher sees the payout
  await page.goto('/dashboard/earnings');
  await expect(page.getByTestId('settlement-amount')).toBeVisible();
});

This runs in seconds. The real flow takes 14 days + waiting for the monthly cron. The E2E test proves every step works without the calendar being involved.

4. Deleting 10,000 Lines of Terraform

Before the Playwright setup, we had old EC2-based Terraform infrastructure for running Selenium tests. It provisioned EC2 instances, installed Chrome, managed security groups, and had its own deployment pipeline.

All of that infrastructure existed to run browser tests that were flaky, slow, and required dedicated compute.

Playwright runs locally or in CI with zero infrastructure. No EC2 instances. No security groups. No AMIs to maintain. We deleted every line of that Terraform and replaced it with a playwright.config.ts and a package.json script.

The 10,000 lines deleted versus 1,637 lines added ratio tells the story. Modern testing tools have made dedicated test infrastructure obsolete for most teams.

5. Keeping Test State Clean

The last surprise: Auth0 rate limits on the /oauth/token endpoint. If every test logs in fresh through the Auth0 UI, you'll hit rate limits fast in CI.

The fix is Playwright's storageState pattern:

// global-setup.ts — runs once before all tests
async function globalSetup() {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  await loginThroughAuth0UI(page);

  // Wait for the localStorage key (see problem #1)
  await waitForAuth0Token(page);

  // Save the authenticated state
  await page.context().storageState({ path: AUTH_STATE_PATH });
  await browser.close();
}

// playwright.config.ts
export default defineConfig({
  use: {
    storageState: AUTH_STATE_PATH,  // Every test starts authenticated
  },
});

One real login. Every test reuses the saved cookies and localStorage tokens. No rate limits. Tests run faster because they skip the Auth0 redirect dance.

What I'd Do Differently

If I were starting over, I'd set up the TestSchedulerController and the waitForFunction localStorage polling on day one, before writing a single test. Those two pieces are the foundation everything else depends on. I spent hours debugging test failures that were actually setup failures.

Also: don't bother with networkidle. For Auth0 flows, it's a trap. Use explicit waits for the exact state you need.

The Numbers

Deleted: ~10,000 lines of Terraform EC2 test infra
Added: 1,637 lines of Playwright tests
Full flow test time: ~12 seconds
Real flow time: 14 days + monthly cron
Auth0 logins per CI run: 1 (thanks to storageState)

Have you set up E2E tests with Auth0 or another identity provider? What was your biggest pain point? I'd bet it was the token storage timing.

Building jo4.io -- an affiliate marketplace tested end-to-end, from click to payout.