One panic shouldn't kill the scan: isolating rules with catch_unwind

Cloud Cost Analyzer runs 92 rules against your cloud account in a single scan. A rule is a real unit of work: it pulls a resource's config and metrics, evaluates them against a cost model, and maybe emits a finding. They're independent, and every one of them leans on libraries I don't control: cloud SDKs, response parsers, date and math crates.

Any of that can meet an input it didn't anticipate and panic: an unwrap() that couldn't fail until some resource shape I'd never seen, an index out of bounds deep in a dependency, an overflow on a field that's normally small. Usually nobody wrote bad code. A library just hit an edge case nobody dependency-audited for.

The question isn't whether a rule will panic. It's what happens to the other 91 when rule

57 does. A cost report that dies two-thirds of the way through because one rule met one weird EBS volume isn't much use to anyone.

"So handle it with Result"

The obvious objection: this is Rust, so model the failure as a Result and there's nothing to catch. The rules do exactly that for every failure they can see coming. Can't reach the metrics API? Result. Missing a field the check needs? Result. Expected failures travel the normal error path and always have.

A panic is the failure I didn't foresee. You can't write ? for a bug you don't know exists, and you certainly can't for one buried three crates deep in a dependency that panics on input its author never tested. Result handles the errors you anticipated. catch_unwind covers the ones you couldn't, and it only exists in my code because Result can't reach into code I didn't write.

std::panic::catch_unwind runs a closure and hands you back a Result instead of letting the panic tear through your stack. Here's the real dispatch, one rule at a time:

let mut eval = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
    rule.evaluate(resource, metrics, costs)
}))
.unwrap_or_else(|_| {
    tracing::error!(
        rule_id = rule_id,
        resource_id = %resource.resource_id,
        "rule panicked during evaluate; skipping its finding"
    );
    None
})?;

A rule panics, we log which rule and which resource, return None for that one finding, and the loop moves on to the next rule. One panic costs one finding.

What AssertUnwindSafe actually promises

The compiler demands that wrapper, and it's worth understanding why before reaching for it.

catch_unwind only accepts a closure that's UnwindSafe. The idea: if a panic unwinds through the middle of your closure, any state it shared with the outside could be left half-updated, and UnwindSafe is the marker that says there's no such state to observe. My closure captures &rule, &resource, and friends by reference, so the compiler can't prove that on its own and refuses to compile.

AssertUnwindSafe is me telling the compiler I've checked, and that catching this panic won't let anyone observe a broken invariant. That holds here for a concrete reason: on panic I throw the result away. I return None and never read a partially-built finding. There's no torn state to see because I don't look. That's the bar for reaching for AssertUnwindSafe. You should be able to point at why the post-panic state is unobservable.

The setting that silently turns all of this off

catch_unwind only works if panics unwind, which is why it isn't a try/catch. Rust has two panic strategies, and you choose per build:

# Cargo.toml — a size-optimized profile
# Inherits all release settings, adds panic=abort to remove unwind tables
panic = "abort"

Under panic = "abort", a panic doesn't unwind the stack. It calls the abort handler and kills the process immediately, so catch_unwind never gets a chance to run. Every one of those careful per-rule guards becomes a no-op, and rule #57 takes the whole process down with it. That happens in the exact build you ship to production, because panic = "abort" is something you turn on to drop unwind tables and shrink the binary.

So panic isolation is a whole-binary decision rather than a library feature you add. If you want one bad rule to cost one finding, you have to keep the unwinding runtime, and eat the binary size, on purpose. No assert or catch_unwind call can buy that back for you. It's the trade you make once, in the profile, and it's invisible in the code that depends on it.

One more edge: poisoned locks

If a rule panics while holding a Mutex or RwLock, that lock is poisoned and the next rule to touch it inherits the panic. The isolation above holds partly because the shared data these rules read is immutable and lock-free, so there's no lock to poison. If yours mutates shared state under a lock, catching the panic isn't enough. You have to decide what a poisoned lock means.

Where it landed

Rules and the second-stage analyzers both run inside that catch_unwind, under an unwinding profile. When a rule, or a library it leans on, meets an input nobody anticipated, the cost is one missing finding and a log line naming the culprit. The isolation works because the binary is built to allow it, and I can say exactly why the post-panic state is safe to ignore: I never read it.

---

This is part of a series on building Cloud Cost Analyzer in Rust.