DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Eight things our app refuses to do when a dependency is missing, and six it does anyway

Munchable depends on more things than it should. Postgres, Redis, an auth provider, Stripe, a mail provider, a model API, a cron scheduler. On any given day one of them is slower than it was yesterday, and roughly once a quarter one of them is simply not there.

There is no resilience.ts in this repo and no circuit breaker library. What there is instead is 168 catch blocks and .catch() calls outside the tests, each written at the moment I was holding the problem. I went through them looking for a policy, and found that there is one, and that it is not the policy I would have written down in advance.

Here are the two that got me to it. They guard two API routes, they are the same five lines, and they do opposite things.

// app/api/product-search/route.ts
try {
  decision = await enforceParallel([
    { limiter: lim.searchDev, key: user.id },
    { limiter: lim.searchIp, key: getClientIdentifier(request) },
  ]);
} catch {
  decision = { success: true };
}
Enter fullscreen mode Exit fullscreen mode
// app/api/ocr/route.ts
try {
  decision = await enforce([
    { limiter: lim.ocrDevMin, key: user.id },
    { limiter: lim.ocrDevDay, key: user.id },
    { limiter: lim.ocrIp, key: ip },
  ]);
} catch {
  return NextResponse.json({ error: 'rate_limiter_unavailable' }, { status: 503 });
}
Enter fullscreen mode Exit fullscreen mode

Same dependency, same failure, same shape. One says "the limiter is broken, let them through". The other says "the limiter is broken, nobody gets in".

The difference is what being wrong costs

Searching the catalogue by name is a read. It hits an index, it returns rows, it is condition-blind and keyed to nobody. If the limiter is down and an abusive client gets 10,000 searches through instead of 300, the bill is some database time and I find out from the dashboard.

Reading a label is a model call. It costs money per request, every time, immediately, to whoever is pointing a camera at it. If the limiter is down and the ceiling disappears, the failure mode is not a slow afternoon, it is an invoice. So that route would rather be unavailable than unmetered, and it says so in the error code: not 503 service_unavailable but rate_limiter_unavailable, which is a different operational problem and deserves a different word in the log.

Once I had those two side by side, the rest of the codebase sorted itself.

The eight refusals

Cron without its secret. Every scheduled route wants Authorization: Bearer <CRON_SECRET>, compared in constant time. If the secret is not configured at all, the route answers 503 and does not run. The tempting alternative is to run unauthenticated when there is nothing to authenticate against, which is how a public URL ends up triggering a billing job. You can watch the other half of this from outside:

$ curl -s https://munchable.app/api/cron/indexnow
{"error":"unauthorized"}
Enter fullscreen mode Exit fullscreen mode

Cron without Redis. The jobs take a Redis lock so that a slow run still going when the next tick fires is skipped rather than doubled. No Redis means no lock, and no lock means the job does not run. These jobs spend money and write curated data, and running two of them at once is worse than running none.

A model call without a budget counter. There is one daily cap on inline curation calls across every account, and a smaller per-account one. The reservation is an incr against Redis, and its catch block is one line:

} catch {
  return false;
}
Enter fullscreen mode Exit fullscreen mode

No counter, no call. A cost circuit breaker that stops working has to stop the thing it was breaking.

Erasure without the key that completes it. Deleting an account cancels the subscription, deletes every row keyed to the user, anonymises their catalogue contributions and deletes the identity record. That last step needs the service-role key. Without it the route returns 501 before touching anything, because the alternative is a half-deletion: rows gone, identity still there, and a user who has been told their account is deleted. A refusal is recoverable. A partial erasure is not, and it breaks a specific promise in our privacy notice.

A production app build with no backend configured. The mobile app requires an account whenever a backend is configured, and unconditionally in a production build:

export const REQUIRE_ACCOUNT =
  isSupabaseConfigured || process.env.EXPO_PUBLIC_ENV === 'production';
Enter fullscreen mode Exit fullscreen mode

The second clause is the interesting one. A production binary built with the keys missing is a mistake, and the failure mode without that clause is the worst available: the app opens, works, gates nothing, and nobody notices until someone asks why the free tier is not metered. With it, a mis-built release fails closed to the sign-in screen, which is impossible to miss within about four seconds of installing it.

A contribution without a receipt. Captures carry a signed receipt issued by the route that read the label. If the signing secret is not configured, production refuses every capture with a 501 and development saves them without crediting anything. There is deliberately no third branch where a missing secret means "skip the check", because that is the branch an attacker would be aiming for.

The two metered routes above, which is the pair I started with.

Everything, while the maintenance switch is on. One committed boolean puts every API route behind a 503 with a Retry-After, refuses sign in without calling the auth provider at all, and leaves the marketing pages up, because they render from the repo and have nothing to be down about.

The six that carry on

A webhook with no way to de-duplicate. This one is the mirror image of the cron lock, with the same missing dependency.

if (!redis) {
  console.error('[webhook-idempotency] Redis not configured, cannot de-duplicate.');
  return { isFirstDelivery: true, degraded: true };
}
Enter fullscreen mode Exit fullscreen mode

Mail providers retry and occasionally deliver twice. Without Redis there is no way to know whether this delivery is the first, and the two available wrong answers are not equivalent: treating a real message as a duplicate drops somebody's support request for ever, and treating a duplicate as real forwards one email twice. So it claims first delivery, and sets degraded so the log says which branch it took.

The same absent Redis makes a scheduled job refuse to start and makes this handler proceed. Both are right, and the reason they differ is that the job can be run again later and the email cannot.

A spam classification that threw. Inbound mail goes through a check before it opens a ticket or gets forwarded. Any error in that check means "handle as normal mail", and there are two tests asserting exactly that. A classifier that fails closed is a classifier that silently bins a customer's first email to us on the day its dependency has a bad hour.

A Stripe call during erasure. Cancelling the subscription is best effort, wrapped in a catch that does nothing. The user asked to be deleted; the deletion must not hang on a payment provider. A subscription left running against a deleted account is a billing problem I can find and fix. A failed deletion is a legal one.

Three light reads. Entitlement, contributions and the catalogue search let a request through when their limiter is unreachable, for the reason at the top: being wrong costs database time.

A cache write on the phone. Every AsyncStorage write in the app is wrapped and swallowed. A failed cache write costs a re-lookup.

The local wipe after an account is deleted. It never throws. By the time it runs the server side has already succeeded, and there is no useful error to show somebody who no longer has an account.

The rule, written down after the fact

Refuse when being wrong creates something that cannot be taken back: a charge, a duplicate write, a half-deletion, an unmetered endpoint, a release with the gate missing. Carry on when being wrong creates something that can: a duplicate message, a few extra reads, a cache miss, an orphaned subscription row.

It is an asymmetry about reversibility, not about severity, which is why "important" is useless as a guide. Erasure is the most important operation in the app and it refuses rather than half-finish. Forwarding an email is one of the least important and it proceeds rather than risk dropping one. Both of those follow from the same sentence.

The part I would recommend to anybody auditing their own catch blocks: read them in pairs. A single catch {} always looks defensible on its own, and no test will tell you otherwise, because a swallowed error passes either way. It is only next to its twin, the one that does the opposite thing for the same reason, that you find out whether you have a policy or a habit. The pair at the top of this post is the only reason I can state the rule at all, and I did not state it when I wrote either of them.

If you want to poke at the refusing half yourself, every route above answers from the live deployment. The privacy notice is the document all of this is in service of, and the support page is where the one fail-open path that handles human beings ends up.

Top comments (0)