DEV Community

delay.run
delay.run

Posted on

Case study (hypothetical): the month-end usage billing job that charged 300 customers for the wrong month

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

This is a hypothetical case study. The company and numbers are invented; the bugs are the kind that show up on the first of the month at companies with usage-based pricing.

The setup

An API company bills customers monthly for usage: number of requests, plus a base fee. On the 1st of each month, a scheduled job:

  1. Selects every active customer.
  2. For each one, sums last month's usage from a usage_events table.
  3. Creates an invoice with the payment provider, with one line item for the base fee and one for usage.
  4. Finalises the invoice, which charges the card.

The job is a cron entry on a worker server: 5 0 1 * *. The worker server is set to UTC. It's been running for a year.

The incident

1 December, 00:05 UTC. The job starts. The query for "last month" looks like this:

SELECT customer_id, count(*)
FROM usage_events
WHERE created_at >= date_trunc('month', now() - interval '1 month')
  AND created_at <  date_trunc('month', now())
GROUP BY customer_id;
Enter fullscreen mode Exit fullscreen mode

It looks right. It isn't, in two ways nobody had noticed.

Problem 1: the database session's time zone. created_at was a timestamptz column, but date_trunc('month', now()) truncates in the session's time zone. A recent change to the connection pool configuration had set the session time zone to America/New_York, the company's head office, so dashboards would show local times. At 00:05 UTC on 1 December it was 19:05 on 30 November in New York. date_trunc('month', now()) returned 1 November, and "last month" became October.

For every customer, the job billed October's usage again.

Problem 2: a retry that wasn't safe. At 00:40 UTC, the job hit the payment provider's rate limit on invoice creation, got a few errors and crashed. The cron wrapper retried the whole job at 01:05. The job had no record of which customers it had already invoiced, so it started from the top. 120 customers got a second invoice, for October again.

1 December, 09:30 UTC. Support has 40 tickets. Customers who'd had a quiet November and a busy October got large, wrong invoices. Some got two.

Fixing the immediate damage

The team voided the unpaid duplicate invoices, refunded the paid ones, and reran November's billing with a corrected job. They emailed every affected customer with the correct invoice and a short explanation. About 300 customers were affected in total; 120 of them twice.

A billing job should be safe to retry, and you should be able to see exactly what each run did. delay.run runs the HTTP job on schedule, retries a failed run with backoff, and keeps a log of every attempt, so the retry is visible and your endpoint can make it a no-op.

What actually went wrong

"Last month" was computed implicitly. The query relied on now() and the session's time zone to decide which month to bill. Both are environmental: they depend on when the job runs and on connection settings that someone else can change.

The billing period wasn't a first-class value. Nothing in the system said "this run is billing November 2026". The month was inferred inside a query, so nobody could see it in logs, and a retry couldn't check whether it had already billed it.

The job wasn't idempotent per customer. A crash halfway through left the system in a state where re-running started over.

The fixes

1. Pass the billing period explicitly. The scheduler now triggers the job with a parameter: period=2026-11. The job computes the boundaries in UTC, in code, and logs them:

start = datetime(2026, 11, 1, tzinfo=timezone.utc)
end   = datetime(2026, 12, 1, tzinfo=timezone.utc)
Enter fullscreen mode Exit fullscreen mode
SELECT customer_id, count(*)
FROM usage_events
WHERE created_at >= %(start)s AND created_at < %(end)s
GROUP BY customer_id;
Enter fullscreen mode Exit fullscreen mode

No now(), no date_trunc, no dependency on the session's time zone.

2. Pin the session time zone for jobs. The worker's database connections set SET TIME ZONE 'UTC' on connect. Dashboards can use their own pool.

3. One invoice per customer per period, enforced by the database.

CREATE TABLE billing_runs (
  customer_id  bigint NOT NULL,
  period       text   NOT NULL,     -- '2026-11'
  invoice_id   text,
  PRIMARY KEY (customer_id, period)
);
Enter fullscreen mode Exit fullscreen mode

Before creating an invoice, the job inserts a row; a conflict means "already billed, skip". The invoice creation call to the provider uses an idempotency key built from customer_id and period, so even a crash between the API call and the database update can't create two invoices.

4. Process customers as separate jobs. Instead of one giant loop, the month-end trigger enqueues one job per customer. A rate-limit error now delays one customer's invoice, with backoff, instead of crashing the run.

5. A pre-flight check. Before finalising anything, the job compares the total usage it's about to bill with the previous month's total. A difference of more than 50% pauses the run and alerts a human. October's usage billed as November's would have tripped this immediately.

What I'd take from it

  • Billing periods are data, not something you compute from now().
  • Never let a session or server time zone decide a financial boundary. Use explicit UTC instants (or the customer's own zone, explicitly, if your terms say so).
  • Idempotency key per customer per period, in your database and at the provider.
  • Small jobs fail small. One job per customer beats one job for everyone.
  • Sanity-check totals before charging. A two-line comparison would have stopped this.

Related reading

Have you had a month-end job go wrong? I'd love to hear whether it was the time zone, the retry, or something else entirely.

Top comments (1)

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

The session timezone detail is easy to miss until it charges the wrong month. Storing the period in UTC and deriving the label from it would have caught this earlier.