How I Made 140 Background Jobs Idempotent With Claude Code: 5 Lessons

TL;DR

A retried background job charged 37 customers twice, and that one incident sent me on a two-week audit of all 140 jobs in our Python/Celery stack. I used Claude Code to classify every job's side effects, add idempotency keys, and build a "run it twice" test harness. The AI was excellent at the boring 80% and surprisingly bad at one specific thing, which I'll get to.

The Problem

Here's the incident in one paragraph.

A worker picked up a charge_subscription_renewal job, called the payment provider, got a successful charge back... and then the pod got OOM-killed before it acknowledged the message. The broker did exactly what it's supposed to do with an unacknowledged message: it redelivered it. A second worker ran the job from scratch and charged the card again. 37 customers, one bad Tuesday morning, a lot of refund emails.

The root cause wasn't the OOM. It was that we had been quietly assuming exactly-once delivery in a system that only ever promised at-least-once. Every queue I've worked with (Celery on RabbitMQ, SQS, Sidekiq, Cloud Tasks) has the same fine print: your job may run more than once. Retries, visibility timeouts, worker crashes, deploys that kill in-flight tasks. Any of them can do it.

So the real question became: which of our 140 jobs are safe to run twice, and which ones are time bombs?

Nobody on the team knew. The jobs had been written over six years by maybe 20 different people. Some were obviously harmless (rebuild a search index), some were obviously dangerous (charge a card), and about 90 were somewhere in the murky middle.

Stack, for context:

• Python 3.12, Django 5.1
• Celery 5.4 on RabbitMQ, acks_late=True on most tasks
• PostgreSQL 16
• Claude Code (CLI) as the coding agent, October 2026 build

How I Solved It

I broke the work into four phases. The key decision was to make the AI produce a structured inventory first, before touching any code.

Phase 1: Inventory every side effect

I asked Claude Code to walk every task and fill out a fixed schema. Not a summary, a schema. Free-form summaries are where hallucinations hide; a rigid table forces the model to commit to something you can check.

# one entry per task, generated by the agent, reviewed by me
- task: billing.tasks.charge_subscription_renewal
  side_effects:
    - kind: external_api
      target: payment_provider.charge
      reversible: false
    - kind: db_write
      target: Invoice
      operation: insert
  natural_idempotency: none
  retry_policy: autoretry_for=(Exception,), max_retries=5
  risk: critical
  evidence: "billing/tasks.py L88-L141"

The evidence field turned out to be the most important line. Every claim had to point at a file and a line range, so I could verify it in seconds instead of re-reading the whole task.

The prompt was roughly:

For each Celery task in the repo, produce one YAML entry with this exact schema.
Follow calls at least 3 levels deep into helpers.
If you are not sure whether a call has a side effect, mark it `unknown`,
never guess `none`. Cite file + line range for every side effect.

That "mark it unknown, never guess none" line did a lot of work. It flipped the model's default from optimistic to paranoid, which is exactly what you want in a safety audit.

The result after one long session:

Risk level	Count	Example
Critical (money / external, irreversible)	9	charge card, send payout
High (user-visible duplicates)	23	send email, push notification
Medium (internal duplicates, fixable)	38	insert audit row, increment counter
Safe (naturally idempotent)	61	rebuild cache, recompute stats
Unknown	9	deep helper chains

I manually verified all 9 critical, all 9 unknown, and a random sample of 20 from the rest. The agent's classification held up in 17 of the 20 sampled. The 3 misses were all "medium" jobs that were actually "high", and I'll come back to why.

Phase 2: One idempotency primitive, not 140 bespoke fixes

The tempting move is to let the agent patch each task individually. Don't. You end up with 140 slightly different home-grown dedup schemes.

Instead I wrote a single primitive by hand (this is the part I didn't want an AI to improvise) and had the agent apply it everywhere.

# Idempotency guard: claim a key in Postgres before doing side effects.
from django.db import IntegrityError, transaction

class AlreadyProcessed(Exception):
    pass

def claim_once(key: str, task_name: str) -> None:
    try:
        with transaction.atomic():
            IdempotencyRecord.objects.create(key=key, task_name=task_name)
    except IntegrityError:
        raise AlreadyProcessed(key)

class IdempotencyRecord(models.Model):
    key = models.CharField(max_length=200, unique=True)
    task_name = models.CharField(max_length=200)
    created_at = models.DateTimeField(auto_now_add=True)

The unique constraint does the real work. Two workers racing on the same key can't both win, because Postgres won't let them.

For the payment job specifically, a local guard isn't enough, because the crash can happen after the charge but before our DB commit. So we also pass a deterministic idempotency key to the payment provider itself:

def charge_subscription_renewal(subscription_id: int, period: str):
    key = f"renewal:{subscription_id}:{period}"
    provider.charge(
        amount=sub.price,
        customer=sub.customer_id,
        idempotency_key=key,  # provider dedups server-side
    )

💡 The key must be derived from the business event, not generated at runtime. uuid4() inside the task is useless, since every retry gets a fresh UUID. renewal:{subscription_id}:{period} is the same no matter how many times the job runs.

Phase 3: Let the agent roll it out, in batches

With the primitive in place, the agent's job was mechanical: for each non-safe task, pick the business-event key, wrap the side effects, and handle AlreadyProcessed as a success (not a failure, or the retry logic will keep hammering it).

I ran it in batches of about 10 tasks, one PR per batch, risk-ordered: critical first, then high, then medium.

flowchart LR
    A[Task received] --> B{claim_once key}
    B -- new key --> C[Run side effects]
    B -- IntegrityError --> D[Log + ack as success]
    C --> E[Commit + ack]
    C -- crash --> F[Broker redelivers]
    F --> A

Phase 4: The "run it twice" harness

This is the part I'd steal if I were you. I had Claude Code generate a pytest fixture that runs every task twice in a row with identical arguments and asserts that the observable state after run 2 equals the state after run 1.

import pytest

@pytest.mark.parametrize("task, args", TASK_CASES)
def test_task_is_idempotent(task, args, snapshot_state):
    task.apply(args=args).get()
    after_first = snapshot_state()

    task.apply(args=args).get()
    after_second = snapshot_state()

    assert after_second == after_first, f"{task.name} is not idempotent"

snapshot_state() captures row counts of the relevant tables plus every call recorded by our mocked email, push, and payment clients. If run 2 sends one more email or inserts one more row, the test fails.

On the first run of this harness, 14 tasks failed that the inventory had marked as fixed or safe. That's the number that made me glad I built it.

Lessons Learned

1. Structured output beats "please audit this"

Asking an agent to "check these jobs for idempotency issues" gets you a confident essay. Asking it to fill a schema with evidence line ranges gets you something you can actually verify. The schema is your review interface. ✅

2. The AI is great at following calls, bad at implicit side effects

Here's the specific weakness I promised. All 3 misclassifications in my sample, and most of the 14 harness failures, came from side effects that weren't visible as a function call:

• Django post_save signals that sent a webhook
• A model's save() override that bumped a counter
• A database trigger (!) that inserted an audit row

The agent followed explicit calls three levels deep without breaking a sweat. But "this .save() also fires a signal registered in a different app" is spooky action at a distance, and it missed it repeatedly. ⚠️ If your codebase leans on signals or triggers, tell the agent explicitly to enumerate them first.

3. Write the safety primitive yourself

I let the agent write roughly 90% of the diff, but claim_once and the model behind it were mine. The core of a correctness guarantee should be small, boring, and authored by someone who will be paged when it breaks. Let the AI do the rollout, not the invention.

4. "Already processed" is a success, not an error

Our first batch treated AlreadyProcessed as an exception, so Celery retried it five times and then dumped it in the dead-letter queue. That generated a flood of false alerts. Catch it, log it at INFO, and ack the message.

5. Tests that run things twice are cheap insurance

The harness took the agent about 20 minutes to scaffold and me about two hours to tune the state snapshot. It now runs in CI on every PR that touches a task. New jobs get caught before they ship, which is the only way this audit doesn't rot in six months. 🚀

What's Next

• Cleanup policy for IdempotencyRecord. It grows forever right now. I'm adding a TTL sweep keyed to each task's maximum retry window.
• Signal inventory. Since implicit side effects were the agent's blind spot, I'm having it build a static map of every signal and trigger, and feeding that map into future audits as context.
• Chaos test in staging. Randomly kill workers mid-task for a day and confirm nothing double-fires. Trust, but verify.

Wrap-up

If you run background jobs, assume they will run twice, because eventually they will. An AI coding agent can make the audit dramatically faster, but only if you give it a rigid schema, write the core guard yourself, and back everything with a test that literally runs each job twice.

👉 If this was useful, follow me here on Dev.to for more build logs on shipping real systems with Claude Code. And I'm curious: what's the worst duplicate-job bug you've seen? Drop it in the comments. I'll read every one.