OK, DEGRADED, ABSTAINED, FAILED: Designing a Typed Outcome Model for LLM Calls

Most LLM-calling code I've seen treats an LLM step like an HTTP request: try/catch, retry a couple times on failure, throw if it still doesn't work. That model breaks down fast once an LLM call is one step inside a larger workflow, because "it didn't work" isn't actually one outcome, it's at least four, and they need different handling:

• The call succeeded and produced a usable result.
• The call never happened, because you could tell in advance there was nothing for the model to do.
• The call failed, but you have a reasonable fallback and the workflow should keep going with degraded output.
• The call failed and there is no reasonable fallback; the workflow step genuinely failed.

If your code only has try/catch, all four of these get flattened into "threw" or "didn't throw," and whoever calls your function has to reverse-engineer which case they're in from whatever's left in the catch block. I ran into this building a document-processing pipeline (Mastra workflow steps backed by LLM calls, orchestrated through Inngest), and the fix was to stop treating "did the LLM call work" as a boolean and make it a first-class typed result with exactly four outcomes: OK, DEGRADED, ABSTAINED, FAILED.

This post walks through the design, the non-obvious bugs that showed up in review, and why I ended up thinking about it less like error handling and more like a state machine.

The core type

export type LlmStepFailureClass = 'timeout' | 'parse' | 'schema' | 'upstream-5xx' | 'upstream-4xx'

export type LlmStepEnvelope = {
  stepId: string
  attempts: number
  failureClass?: LlmStepFailureClass
  modelVersion?: string
  promptBundleSha?: string
  latencyMs: number
  tokenUsage?: LlmStepTokenUsage
}

export type LlmStepResult<T> =
  | (LlmStepEnvelope & { outcome: 'OK'; value: T })
  | (LlmStepEnvelope & { outcome: 'DEGRADED'; value: T })
  | (LlmStepEnvelope & { outcome: 'ABSTAINED'; value: T })
  | (LlmStepEnvelope & { outcome: 'FAILED' })

Two design choices here are doing most of the work.

First, OK, DEGRADED, and ABSTAINED all carry a value: T, but FAILED doesn't. That's not a stylistic accident; it's the type system enforcing the actual contract. A caller who wants to keep the pipeline moving through a degraded or abstained result can pattern-match on outcome !== 'FAILED' and get a value back with zero unwrapping ceremony. A caller who tries to read .value off a FAILED result gets a compile error, not a runtime undefined. The discriminated union is carrying real information: "did this step produce something usable" and "why not" are two different questions, and the type keeps them separate.

Second, the envelope deliberately does not include the step's output. Every field in LlmStepEnvelope is a scalar or a small nested record; attempt count, failure class, model version, a hash of the prompt template, latency, token usage. None of it is the actual content the LLM produced. That's intentional: the envelope is safe to log to Datadog wholesale, faceted on outcome and failureClass, without a separate PII/content allowlist step. If the actual document content lived in the same object as the metrics, you'd either have to strip it before every log call (easy to forget) or accept that your observability pipeline now contains customer document content (not acceptable). Splitting them means the "safe to log" boundary is a type boundary, not a discipline you have to remember to apply.

Why four states and not two

ABSTAINED was the one that took me longest to justify as its own state rather than just an early return. The distinction is about why nothing happened:

abstain?: { on: () => boolean; to: () => T }

abstain.on() is checked once, before any LLM call is attempted. If it returns true, the step returns ABSTAINED with abstain.to()'s value, and no LLM call happens at all. This is for the case where the step can tell in advance that there's nothing to do, zero tables to classify, zero content blocks to map. That's meaningfully different from DEGRADED, which means "we tried, it didn't work, here's a fallback." Collapsing those into one state would mean losing the distinction between "there was no work" and "the work failed"; which matters a lot when you're looking at a dashboard trying to figure out whether a spike in non-OK outcomes is benign (empty documents) or a real regression (the model's failing on documents that have content).

degradeTo is the mirror image, checked only after retries are exhausted with no successful attempt:

if (degradeTo) {
  return { ...envelope, outcome: 'DEGRADED', value: degradeTo() }
}
return { ...envelope, outcome: 'FAILED' }

Some steps have a reasonable degrade path (classify every table as content rather than picking a specific type; proceed with the algorithmic graph instead of the LLM's corrections) and some don't (a step whose entire job is the LLM call has nothing to degrade to, so it just fails, matching whatever the legacy behavior was before this executor existed). Making degradeTo optional rather than mandatory means a step that can't sensibly degrade doesn't have to fake one.

Classifying failures without instanceof gymnastics

The retry decision hinges on why an attempt failed, and I wanted that decision made once, centrally, rather than re-implemented in every step. classifyThrownError maps whatever the AI SDK throws into one of the five failure classes:

export function classifyThrownError(error: unknown): LlmStepFailureClass {
  if (error instanceof Error && error.name === 'AbortError') return 'timeout'
  if (APICallError.isInstance(error)) {
    if (typeof error.statusCode === 'number' && error.statusCode >= 400 && error.statusCode < 500) {
      return 'upstream-4xx'
    }
    return 'upstream-5xx'
  }
  return 'parse'
}

Note the default: a missing status code on an APICallError is treated as upstream-5xx (retryable), not silently dropped into some catch-all. If the AI SDK ever throws an APICallError without a status code, I want that treated as "probably transient, worth retrying" rather than swallowed. And 'schema' a validated-but-wrong-shape response, deliberately isn't produced by this classifier at all; it's expected to come from the attempt function's own explicit result, because "the JSON didn't parse" and "the JSON parsed but violated the schema" are different failure modes discovered at different stages, and conflating them here would erase that distinction right where it's most useful.

Only three of the five classes retry by default:

const RETRYABLE_FAILURE_CLASSES: ReadonlySet<LlmStepFailureClass> = new Set(['timeout', 'parse', 'upstream-5xx'])

upstream-4xx and schema don't retry, because retrying a genuine 4xx (bad request, auth failure) or a validated-but-wrong-shape response won't produce a different outcome without a different prompt or model, you'd just be burning latency and budget on a guaranteed repeat failure.

The bug review caught: a 429 is a 4xx that should retry

This is the part of the story that's more interesting than the initial design, because it's a real bug that shipped in the first review pass. A 429 (rate limited) is, by HTTP semantics, a 4xx; so the naive classifier put it in upstream-4xx, and upstream-4xx doesn't retry. Which means: every rate-limited call failed permanently, on the first attempt, in exactly the situation a retry exists to handle.

The fix keeps the Datadog-facing classification honest (a 429 still shows up as upstream-4xx, not relabeled as something misleading) while overriding the retry decision independently:

export type LlmStepAttemptResult<T> =
  | { ok: true; value: T; modelVersion?: string; tokenUsage?: LlmStepTokenUsage }
  | { ok: false; failureClass: LlmStepFailureClass; retryable?: boolean }

const isRateLimited = APICallError.isInstance(error) && error.statusCode === RATE_LIMIT_STATUS_CODE
return { ok: false, failureClass, ...(isRateLimited ? { retryable: true } : {}) }

const isRetryable = result.retryable ?? RETRYABLE_FAILURE_CLASSES.has(result.failureClass)
if (!isRetryable) break

retryable is an optional per-attempt override that, when present, wins over the default per-failureClass policy. This is a small piece of API surface, but it's carrying a real distinction: what happened (classification, for observability) and what to do about it (retry decision) are independently variable, and forcing them to be the same field would have reproduced the exact bug that got caught.

The bug an automated reviewer caught: timeouts that don't actually time out

The second round of fixes came from an automated review pass, and the sharpest one was this: firing controller.abort() on an AbortSignal only signals the attempt, it doesn't force the await to resolve. If the caller's attemptFn doesn't check the signal (or ignores it), the call just keeps running past perAttemptMs, silently, with no timeout actually enforced despite the code looking like it enforces one.

The fix is to stop trusting the signal to end the attempt, and race it against a timer promise instead:

async function runAttempt<T>(
  attemptFn: (signal: AbortSignal) => Promise<LlmStepAttemptResult<T>>,
  perAttemptMs: number,
): Promise<LlmStepAttemptResult<T>> {
  const controller = new AbortController()

  let resolveTimeout: (result: LlmStepAttemptResult<T>) => void
  const timeoutPromise = new Promise<LlmStepAttemptResult<T>>((resolve) => {
    resolveTimeout = resolve
  })
  const timer = setTimeout(() => {
    controller.abort()
    resolveTimeout({ ok: false, failureClass: 'timeout' })
  }, perAttemptMs)

  try {
    return await Promise.race([
      attemptFn(controller.signal).catch((error): LlmStepAttemptResult<T> => {
        const failureClass = classifyThrownError(error)
        const isRateLimited = APICallError.isInstance(error) && error.statusCode === RATE_LIMIT_STATUS_CODE
        return { ok: false, failureClass, ...(isRateLimited ? { retryable: true } : {}) }
      }),
      timeoutPromise,
    ])
  } finally {
    clearTimeout(timer)
  }
}

Promise.race guarantees the caller gets an answer at perAttemptMs regardless of whether attemptFn respects the abort signal; the timer promise wins the race either way. controller.signal is still passed through and still fired, so a well-behaved attemptFn gets a real chance to cancel its underlying request; the race is the backstop for the ones that don't. This is a pattern worth internalizing generally: an AbortSignal is a request for cooperation, not a guarantee; if a timeout has to be enforced regardless of the callee's behavior, race it against something you control.

The same review pass caught a related issue in the backoff delay: without a cap, a jittered exponential backoff can compute a delay longer than the time budget that's left, causing the step to blow past its total time budget on the wait alone.

function backoffDelay(attemptNumber: number, maxDelayMs: number): Promise<void> {
  const exponentialMs = RETRY_BASE_DELAY_MS * 2 ** (attemptNumber - 1)
  const jitteredMs = exponentialMs * (0.5 + Math.random() * 0.5)
  const cappedMs = Math.max(0, Math.min(jitteredMs, maxDelayMs))
  return new Promise((resolve) => setTimeout(resolve, cappedMs))
}

maxDelayMs is the remaining wall-clock budget, computed fresh at the call site (budget.totalMs - elapsedMs), not a static constant, so the cap tightens as the budget gets consumed, and the last retry's backoff can never itself exceed however much time is left.

Combining abstainOn/abstainTo into one object, on purpose

The last fix is smaller but worth calling out because it's a pattern I now reach for by default: two parameters that only make sense together should be one parameter, not two independently optional ones. The API originally had abstainOn and abstainTo as separate optional callbacks. That shape lets a caller supply one without the other, which compiles fine and blows up at runtime the first time abstain.on() returns true with no to() to call.

abstain?: { on: () => boolean; to: () => T }

Grouping them into a single optional object makes the mis-pairing a type error instead of a runtime throw. There's no runtime check needed to enforce "if you provide one, provide both"; TypeScript enforces it for free, because there's only one optional thing to provide.

What the whole loop looks like end to end

export async function runLlmStep<T>(params: RunLlmStepParams<T>): Promise<LlmStepResult<T>> {
  const { stepId, attemptFn, budget, degradeTo, abstain, promptBundleSha } = params
  const startTime = performance.now()

  if (abstain?.on()) {
    return {
      outcome: 'ABSTAINED',
      value: abstain.to(),
      stepId,
      attempts: 0,
      latencyMs: performance.now() - startTime,
      promptBundleSha,
    }
  }

  let lastFailureClass: LlmStepFailureClass | undefined
  let attemptCount = 0

  while (attemptCount < budget.attempts) {
    const elapsedMs = performance.now() - startTime
    if (elapsedMs >= budget.totalMs) break

    attemptCount += 1
    const remainingMs = budget.totalMs - elapsedMs
    const perAttemptMs = Math.min(budget.perAttemptMs, remainingMs)

    const result = await runAttempt(attemptFn, perAttemptMs)

    if (result.ok) {
      return {
        outcome: 'OK',
        value: result.value,
        stepId,
        attempts: attemptCount,
        latencyMs: performance.now() - startTime,
        modelVersion: result.modelVersion,
        tokenUsage: result.tokenUsage,
        promptBundleSha,
      }
    }

    lastFailureClass = result.failureClass
    const isRetryable = result.retryable ?? RETRYABLE_FAILURE_CLASSES.has(result.failureClass)
    if (!isRetryable) break

    const remainingBudgetMs = budget.totalMs - (performance.now() - startTime)
    const willRetry = attemptCount < budget.attempts && remainingBudgetMs > 0
    if (willRetry) await backoffDelay(attemptCount, remainingBudgetMs)
  }

  const envelope: LlmStepEnvelope = {
    stepId,
    attempts: attemptCount,
    failureClass: lastFailureClass,
    latencyMs: performance.now() - startTime,
    promptBundleSha,
  }

  return degradeTo ? { ...envelope, outcome: 'DEGRADED', value: degradeTo() } : { ...envelope, outcome: 'FAILED' }
}

Every exit point in this function produces exactly one of the four outcomes, with an envelope, in one place. A caller elsewhere in the workflow doesn't need to know anything about retries, backoff, or timeout races; it switches on outcome and moves on.

What I'd generalize from this

The specific types here are tied to Mastra workflow steps and the AI SDK's error shapes, but the underlying idea isn't LLM-specific at all:

• When "it didn't work" has more than one meaning, give each meaning its own state, and let the type system make the states mutually exclusive rather than relying on convention (a null check here, an error code there).
• Separate "what happened" from "what to do about it." Classification and retry policy look like the same decision until a rate limit shows up wearing a 4xx status code, and then they very much aren't.
• Don't trust a cancellation signal to be a guarantee. If a timeout has to hold regardless of what the callee does with the signal, race it against a timer you own.
• Cap resource budgets against what's actually left, not a static constant, a backoff delay computed against the original budget can itself blow through what's remaining.
• When two optional parameters only make sense together, make them one optional object. It turns a runtime footgun into a compile error, for free.

None of this required anything exotic, it's a discriminated union, a Promise.race, and a bit of care about which fields belong together. What made it worth doing carefully is that this executor now sits underneath every LLM-backed step in the pipeline; get the four states and the failure classification right once, and every step that uses it inherits correct retry, timeout, and degrade behavior without having to reason about any of it itself.

Comments

[Max Quimby]

The FAILED-carries-no-value encoding is the quiet hero here — it's "make illegal states unrepresentable" applied to LLM steps, and it kills the entire class of undefined-on-a-failed-result bugs at compile time instead of 2am. Worth the whole post on its own.

The distinction that saved us the most grief was exactly your ABSTAINED vs FAILED — a step that correctly decided there was nothing to do is not an error, and flattening it into the catch block means your retry logic starts retrying no-ops. But the one that later bit us, and that I'd flag as the sequel: DEGRADED silently propagating. A step three hops upstream returns DEGRADED, downstream steps pattern-match outcome !== 'FAILED', unwrap the value, and treat it as clean — so by the end of the workflow you've lost the fact that the output rests on a degraded foundation. We ended up carrying a monotonic "worst outcome seen" up the chain so the final result can't claim OK if any contributing step degraded. How are you aggregating outcomes across a multi-step Mastra/Inngest run — does the workflow-level result surface the min across steps, or does each step stand alone?

[David Shibley]

That ABSTAINED vs FAILED distinction was exactly the one I was most glad I made explicit. And you're right that collapsing it into the catch block means retry logic starts firing on no-ops, which is genuinely worse than the original failure.

On the DEGRADED propagation question: I have to be honest, in the current implementation each step stands alone. outcome from runLlmStep gets logged to Datadog per-step (faceted, so you can see the distribution across a run), but it doesn't thread through the step output schemas to downstream steps. A step three hops upstream returning DEGRADED is invisible to anything after it; every downstream step pattern-matches on outcome !== 'FAILED', unwraps the value, and treats it as clean. You've found the gap. We have plans to address this soon.

The monotonic "worst outcome seen" approach you landed on is the right shape. The implementation question is where it lives? Whether that's a field carried explicitly in the inter-step output schemas (which would force every step definition to declare and forward it, but makes the provenance visible to TypeScript), or a parallel accumulator in requestContext that any step can write to and the final step reads (cheaper to add, but writes are invisible to the type system and easy to forget). We haven't made that call yet. The Datadog faceting buys partial visibility, you can see post-hoc that a run had a DEGRADED step, but you're right that a final result that claims OK when any upstream step degraded is a silent lie, and that's exactly where the post-hoc log stops being sufficient.

The sequel you're describing is real and currently unbuilt. Curious how you ended up threading the monotonic accumulator. Did it live in the step output type, or somewhere outside the per-step schema?