A paid source ran out of credit and our own code hid it from us for nine days

Nakodo leans on a handful of external sources, and two of them are paid, prepaid, and can therefore simply stop. On 12 October one of them ran out of balance. The work that depended on it stopped that day and did not start again for nine days. Nothing crashed, no alert fired, and the only record anywhere was a row in a counters table with a timestamp in it.

Two separate mistakes made that possible, and they are worth separating, because only one of them is the obvious one.

Mistake one: an exhaustion flag that renewed itself

Every external source has a daily row, and a refusal from the source writes exhaustedAt on it. Every request first asks whether the row is usable:

const usable = sql`${providerDays.exhaustedAt} is null`;

That is correct for a source whose budget is a calendar day. A daily API quota resets at a fixed hour, so "it refused us, stop asking until the day turns over" is exactly right, and it saves a few thousand pointless requests.

It is wrong for a prepaid balance, and wrong in a way that is almost funny. The day turns over. The first job of the new day asks the source. The source, still unpaid, refuses. The new day's row is marked exhausted, and every other request that day is skipped locally. Each midnight bought one request, which was spent proving the outage would continue. Nine days of that.

The fix is a per-source retry window rather than a global rule:

export const retryAfter = (counter: Counter): Date =>
  RETRY_AFTER_MS[counter] === undefined
    ? nextUtcMidnight()
    : new Date(Date.now() + RETRY_AFTER_MS[counter]);

The map has two entries, both one hour, for the two sources whose balance can be topped up during the day. Every other source is absent from it and keeps the old behaviour, which for a daily quota is the honest model of what the provider actually does.

The window lives inside the same statement that spends the allowance, so there is no cron and no second round trip:

const usable =
  retryAfterMs === undefined
    ? sql`${providerDays.exhaustedAt} is null`
    : sql`(${providerDays.exhaustedAt} is null
           or ${providerDays.exhaustedAt} < now() - make_interval(secs => ${retryAfterMs / 1000}))`;

await db.insert(providerDays).values({ provider, day, requestsUsed: n })
  .onConflictDoUpdate({
    target: [providerDays.provider, providerDays.day],
    set: { requestsUsed: sql`${providerDays.requestsUsed} + ${n}` },
    setWhere: limit === null ? usable : sql`${providerDays.requestsUsed} + ${n} <= ${limit} and ${usable}`,
  })
  .returning({ used: providerDays.requestsUsed });

An upsert whose WHERE refuses to update returns no rows, so "may I spend one request" and "is this source worth asking again yet" are one question with one answer, decided by the database. Jobs that get a no are deferred rather than failed, with the retry time attached, so the queue holds them and nothing is lost.

Mistake two: nobody was told

The deeper problem is that a source refusing us is not a bug anyone will find in the logs. It is not an exception. Both of ours answer with an HTTP 402 and a sentence of JSON explaining themselves, handled politely by code written to handle it politely. The product keeps working. It just quietly does less.

So refusals now email the operator, and the only interesting part is making sure that happens once:

export async function noteBlock(source: NotifiedSource): Promise<boolean> {
  const [row] = await db
    .insert(providerState)
    .values({ provider: source, blockNotifiedAt: sql`now()` })
    .onConflictDoUpdate({
      target: providerState.provider,
      set: { blockNotifiedAt: sql`coalesce(${providerState.blockNotifiedAt}, now())` },
    })
    .returning({ first: sql<boolean>`${providerState.blockNotifiedAt} = now()` });
  return row.first;
}

coalesce(blockNotifiedAt, now()) is a write that only writes when the column is null: an existing notice keeps its original timestamp. Then the returning clause asks whether the value that is now in the row equals now(), which inside a single statement is a fixed instant, so the comparison means "I am the request that set this". One statement answers both "latch it" and "was I first", with no read, no transaction and no race between two server instances arriving together.

The caller does nothing clever with the answer except send one email:

const paragraphs = [
  `${outage.what} Nakodo noticed at ${utc(new Date())}.`,
  outage.effect,
  "Nakodo tries the source again about an hour after each refusal, so the work picks up by itself once it can go through. The jobs waiting on it stay in the queue rather than failing.",
  "You get this email once. The next one comes only after a request has worked again.",
];

Each caller passes an Outage with three fields, and the types are the whole design of the alert:

export type Outage = {
  what: string;                              // "the balance is used up"
  effect: string;                            // what stops, and what carries on
  link: { href: string; label: string };     // where to fix it
};

An alert that says a source is down is a notification. An alert that says what stopped, what is still running, and which page takes a card is a thing you can act on from a phone. The whole function is wrapped in a try that logs and swallows, because the caller is already on its way to failing a request and an email service having a bad minute must not turn one outage into two.

Forgetting is the other half

A latch you cannot clear is a latch that fires once per deployment and then never again. The clearing side is the mirror image:

await db.update(providerState)
  .set({ blockNotifiedAt: null })
  .where(and(eq(providerState.provider, source), isNotNull(providerState.blockNotifiedAt)));

The isNotNull in the WHERE means the statement is a no-op when there was no notice, which is what lets every successful request call it without thinking. And because a busy invocation might make hundreds of successful requests, there is a module-level set in front of it:

const healthy = new Set<NotifiedSource>();

export async function noteSourceWorking(source: NotifiedSource): Promise<void> {
  if (healthy.has(source)) return;
  healthy.add(source);
  // ... clear the day's refusal and the email notice
}

The set is a hint, not a lock. It is per instance and dies with the process, the database statements are idempotent anyway, and a failure removes the source from the set so the next request tries again. A later outage takes the source back out.

What the table looks like now

export const providerState = pgTable("provider_state", {
  provider: text().$type<NotifiedSource>().primaryKey(),
  nextRequestAt: timestamp({ withTimezone: true }).notNull().defaultNow(),
  blockNotifiedAt: timestamp({ withTimezone: true }),
  updatedAt: updatedAt(),
}).enableRLS();

One row per source, keyed by its name, holding two facts: when the next request may start, and whether we have already said it stopped working. Both are things every server instance has to agree on, which is why neither lives in memory.

The thing I would take from the nine days is not the SQL. It is that the condition had no symptom. The product's plans are metered by what it does for you, and in that incident it did less for everybody while looking entirely healthy. Code that handles a dependency's failure gracefully, with no alert attached to the graceful path, is how an outage gets to be nine days long.