A Reliable LLM System Is a Workflow, Not a Model Endpoint

An LLM request can fail after the provider accepted it, return syntactically valid but unusable output, or trigger a tool whose result is unknown when the connection times out. These are workflow problems. A more capable model does not remove them.

For production systems, make the workflow durable and treat model output as one untrusted result inside it.

Model the operation as states

Suppose the product reviews an uploaded contract and creates follow-up tasks. Represent the operation explicitly:

received
  → authorized
  → extracting
  → retrieving
  → generating
  → validating
  → awaiting_confirmation
  → executing
  → completed

Any active state → failed | cancelled | expired

Persist state transitions with an operation ID. A worker can resume after a crash, and the UI can distinguish “still running” from “the connection disappeared.” Terminal states should be written exactly once, even if messages are delivered more than once.

Assume at-least-once delivery

Many practical queues provide at-least-once delivery. A message may be processed again after a visibility timeout, worker crash, or acknowledgment failure. Design handlers to tolerate it.

For each stage, store an input version and result reference under a uniqueness constraint:

operation_id + stage_name + stage_version

If the same stage message arrives twice, return the existing result or safely continue the in-progress lease. Do not issue another model call or tool action merely because the queue redelivered a message.

Idempotency is especially important across external actions. Give each intended action a stable key and reconcile ambiguous outcomes before retrying.

Use deadlines, not stacked timeouts

An operation has an end-to-end deadline. Each stage receives the remaining budget:

remaining = operation_deadline - now - completion_margin

Retrieval, generation, repair, and tool calls cannot each claim the original full timeout. If too little budget remains to finish safely, expire the operation and present a recovery path.

Propagate cancellation too. A cancelled browser request should eventually stop queue work and provider calls where supported. Cancellation is a state transition, not just a closed socket.

Isolate failure domains

Keep these components independently observable and degradable:

ingress and authorization
document extraction
retrieval/index
model provider adapter
output validation
tool executor
notification/UI stream

A retrieval outage should not look like a model error. A notification failure should not rerun a completed tool action. A malformed model response should not poison the queue for unrelated tenants.

Use separate concurrency limits where one dependency can consume shared capacity. Large document extraction and long generation calls may need different worker pools. Apply per-tenant admission control so a single workload cannot occupy every slot.

Version every behavioral input

A reproducible operation record references:

• application release;
• task and prompt version;
• model alias and provider-reported model ID when available;
• retrieval index or corpus version;
• output schema version;
• tool policy version;
• evaluation suite version.

Do not assume a model name completely identifies behavior. Provider aliases and hosted systems can change. Capture the identifiers exposed by the service and keep your own routing configuration versioned.

This record does not require storing every prompt forever. Content retention should follow data policy; behavioral configuration can be stored separately from user content.

Validate before side effects

Generated structured output is untrusted. Parse and validate it against the application schema, then enforce domain rules:

• referenced records exist and belong to the authorized tenant;
• enum values are allowed;
• amounts and dates fall within permitted ranges;
• required evidence exists;
• the number of proposed actions stays under a limit;
• links use approved schemes and destinations.

The validator returns typed failures. A format failure may allow one bounded repair attempt. A permission or domain-rule failure should not be “prompted away.”

Place human confirmation after validation and before consequential tools. The confirmation should display the exact action set that the executor will receive.

Separate release evaluation from runtime monitoring

Before changing a prompt, model, or retrieval configuration, run a versioned evaluation set with:

• representative normal cases;
• empty, contradictory, and oversized input;
• permission and tenant-boundary cases;
• known severe errors;
• malformed tool arguments;
• cancellation and timeout scenarios.

Use multiple measures rather than one average: schema validity, task-specific rubric scores, severe-failure count, latency distribution, and cost per successful operation.

Runtime monitoring then detects drift in the live workload: validation failures, fallback rate, tool rejection, queue age, deadline expiration, user correction, and sampled quality review where policy permits. Production monitoring cannot replace a release gate because users should not be the first evaluation dataset for a severe regression.

Design recovery before success

Every stage needs a recovery decision:

Stage	Recovery question
Extraction	Can the user replace or re-upload the file?
Retrieval	Is a no-context response allowed, or must the task stop?
Generation	Is the call safe to retry within the deadline?
Validation	Can a bounded repair preserve the same intent?
Confirmation	How long can the proposal remain valid?
Tool execution	Can the system reconcile an unknown outcome?
Notification	Can it retry without replaying the operation?

Persist enough stage evidence to answer these questions after a worker restart.

Amazon’s Builders’ Library article on making retries safe with idempotent APIs explains how stable client request identifiers help services reconcile repeated requests. The OpenTelemetry generative AI conventions provide shared names for tracing model operations. Neither substitutes for a workflow model, but both help make its behavior observable.

The model endpoint is one unreliable boundary among several. Reliability comes from durable states, bounded work, stable identities, validation before effects, and the ability to resume or stop without repeating what already happened.