DEV Community: Hitarth Desai

Dead-Letter Queues for LLM Extraction Failures: Capture, Triage, and Replay Without Losing Trust

Hitarth Desai — Fri, 24 Jul 2026 06:20:16 +0000

A validation failure is not an exception to hide. It is a record your system does not yet know how to trust.

That distinction matters in LLM extraction pipelines. A malformed invoice, an unexpected OCR layout, a model response that violates the schema, and a semantically impossible value may all reach the same line of validation code. If the only outcomes are “retry” or “drop,” the pipeline will either waste money repeating the same failure or silently lose work.

The production answer is a dead-letter path: a durable place for failed records to wait with enough evidence to explain, triage, and safely replay them.

The queue itself is the easy part. The hard part is designing the failure contract around it.

Validation is where routing begins

Constrained decoding and post-hoc validation solve different problems. Even with both, some records should fail. Real documents are messy, schemas change, OCR corrupts values, and models sometimes return plausible nonsense.

A robust validation boundary should produce more than true or false. It should emit a reason the rest of the pipeline can act on:

which schema and model versions were used
which fields failed and why
whether the payload was malformed, incomplete, or semantically invalid
the confidence signal attached to the extraction
whether a retry is likely to change the result

That result becomes a routing decision.

High-confidence, valid records can flow forward. Recoverable transport failures can use a bounded retry. Ambiguous or invalid records belong in review or a dead-letter queue. Confidence-based routing is useful precisely because “trust everything” and “review everything” are both bad operating models.

A dead-letter record needs evidence, not just payload

Putting the original input on another queue is not enough. Without context, the team investigating the failure has to reconstruct the run from scattered logs—if those logs still exist.

I would store a dead-letter envelope containing:

a stable record ID and idempotency key
a reference to the immutable source document, with access controls appropriate to its sensitivity
the extracted payload, including the raw model response only when retention policy permits it
schema, prompt, model, OCR, and pipeline versions
validation errors in a machine-readable form
confidence score and routing threshold
attempt count and timestamps
correlation or trace ID
the explicit reason the record entered the dead-letter path

This is less about collecting every possible field and more about preserving the decision. Six days later, an engineer or reviewer should be able to answer: what did the system see, what did it produce, which contract rejected it, and can it be replayed safely?

Do not turn the dead-letter queue into a shadow database. Store references when the source of truth already exists, define retention and deletion rules, and avoid copying sensitive document contents into systems with weaker controls.

Classify failures before you retry them

Not every failure deserves another model call.

I separate failures into a few broad classes:

Transient infrastructure failure. A timeout, rate limit, or unavailable dependency may succeed later. Retry it with exponential backoff, jitter, and a strict budget.
Deterministic contract failure. The same payload violates the same schema every time. Repeating the call without changing an input, prompt, model, or schema is usually just paying to reproduce the failure.
Ambiguous source data. The document itself does not contain enough evidence. Route it to human review rather than asking the model to invent certainty.
Version mismatch or drift. A new document format or schema version broke an assumption. Quarantine the affected cohort and fix the system, not each record individually.
Policy failure. The record must not be processed automatically because of sensitivity, jurisdiction, or business rules. This needs a controlled workflow, not a clever retry.

The retry policy should be based on that classification. Retry budgets and backpressure matter because a provider incident can otherwise turn one failed request into a storm of expensive duplicates.

Replay must be idempotent

A dead-letter queue is only useful if records can leave it safely.

The dangerous replay implementation simply sends the record back to the start. If earlier attempts already wrote partial state, emitted events, or triggered downstream actions, replay can create duplicate invoices, duplicate notifications, or inconsistent audit trails.

Safe replay needs an idempotency boundary. Give each logical extraction a stable key. Make downstream writes upsert or compare against a known processing version. Record which stages completed. Re-run only the stages affected by the fix when possible.

I also want replay to name the change that justifies it:

new schema version
corrected OCR output
revised prompt or constrained-output definition
new model version
reviewer-supplied correction
repaired upstream document

“Try again” is not a remediation strategy. “Replay against schema v4 after fixing the currency parser” is.

Human review should produce reusable signal

Human-in-the-loop systems often fail in a quieter way: they create a review screen, ask an operator to correct a value, and throw away the reason.

A useful review workflow captures structured outcomes:

corrected field values
reason code for the correction
whether the source was ambiguous or the model was wrong
whether the schema or extraction logic needs to change
reviewer identity and timestamp for the audit trail

Those outcomes improve more than the one record. They reveal recurring document formats, brittle fields, bad thresholds, and failure cohorts worth fixing upstream. They can also become curated evaluation examples, provided privacy and data-governance rules allow it.

The goal is not to keep humans in the loop forever. It is to spend human attention where risk is high and turn repeated review work into engineering feedback.

The dead-letter queue is an observability surface

A DLQ with no metrics is an archive of surprises.

At minimum, I would track:

dead-letter rate as a percentage of processed records
failure count by reason, document type, customer or source cohort, schema version, and model version
age of the oldest unresolved record
time from failure to review or remediation
replay success rate
records that exceed retention or review SLOs
estimated token or provider cost consumed by failed attempts

Watch the rate, not only the count. Traffic growth can make the raw count rise while reliability improves. A sudden failure-rate spike after a schema, prompt, OCR, or model change is a much cleaner drift signal.

This connects the dead-letter path to the broader observability story: traces explain an individual failure; aggregate metrics show whether the system is becoming less trustworthy. Schema drift often appears first as a change in validation failures by field or document cohort.

A practical control loop

The architecture I keep coming back to is:

Constrain generation where the stack supports it.
Apply deterministic repair only where it is safe.
Validate the typed result and attach a confidence signal.
Route valid, high-confidence records forward.
Retry transient failures within a budget.
Send ambiguous or invalid records to review or a durable dead-letter path.
Fix the underlying cause, then replay with an idempotency key and explicit processing version.
Feed failure and review outcomes back into schemas, evals, prompts, and pipeline monitoring.

This is the operational layer around tools such as confident-extract. The library is published on PyPI and focuses on deterministic structured extraction, validation, and confidence. The queueing, review, retention, and replay design belong to the application around that boundary.

That separation is important. A useful open-source component should make its boundary sharper, not claim to be the entire production system.

Failure handling is part of the product

Teams often design the happy path first and treat failed records as an operations problem to solve later.

In an LLM system, the failure path is part of the normal path. Probabilistic components, messy source data, and changing contracts guarantee that some records will need a different decision. The system earns trust by making that decision explicit, durable, observable, and reversible.

Do not drop the record. Do not retry it forever. Preserve the evidence, route it by risk, and make replay a controlled engineering action.

That is what turns “the model failed” from an incident into a workflow.

Written by Hitarth Desai (hitarthbuilds), an AI Systems Engineer building reliable LLM extraction and MLOps pipelines. His open-source confident-extract package is available on PyPI. promptcrucible remains in active development.

Originally published by Hitarth Desai (hitarthbuilds) at https://hitarthdesai.com/blog/dead-letter-queues-llm-extraction-pipelines/

Constrained Decoding vs Post-hoc Validation: Production LLM Extraction Needs Both

Hitarth Desai — Sun, 19 Jul 2026 07:42:34 +0000

Constrained decoding and post-hoc validation solve different problems.

Constrained decoding is generation-time control:

fewer malformed payloads
less wrapper text
better adherence to schema/tool shape

Post-hoc validation is the trust boundary:

coerce what is safe
reject what is not
attach confidence where routing matters

A payload can be perfectly valid JSON and still be wrong for the workflow.

That is why production extraction systems need both.

This is also the systems instinct behind confident-extract, which I publish as Hitarth Desai under the GitHub handle hitarthbuilds.

My rule of thumb:

Constrain early. Validate hard. Trust late.

Beyond Prompt Engineering: The AI Systems Layer Production LLM Apps Need

Hitarth Desai — Sun, 21 Jun 2026 07:30:06 +0000

Prompt engineering starts the demo. Contracts, validation, observability, and failure handling are what make LLM products survive production.

Most LLM products start with a prompt.

That is a sensible place to start. The prompt is where the first prototype happens, where the first internal demo lands, and where the first stakeholder starts to believe the product might work.

But the prompt is not where production reliability comes from.

The moment an LLM feature touches real inputs, downstream systems, or customer-facing workflows, the real engineering problem changes. The question is no longer only "can the model do the task?" It becomes:

Can the output be shaped into a contract another service can trust?
Can failures be detected instead of silently accepted?
Can the same run be inspected and explained later?
Can cost stay bounded when usage spikes?
Can the product say what it measured, what it skipped, and where the result came from?
Can the workflow survive model drift, schema drift, and ugly real-world inputs?

That is the layer I care about: the AI systems layer after prompt engineering.

Prompt engineering is necessary, not sufficient

Prompt engineering matters. Better instructions, better examples, and tighter task framing can improve output quality quickly.

The mistake is treating the prompt as the reliability boundary.

A prompt can ask a model to return JSON. It cannot guarantee the JSON is valid, typed, semantically correct, complete, or safe for downstream use. A prompt can ask for citations. It cannot guarantee the product actually inspected and tracked the underlying sources. A prompt can ask for consistency. It cannot replace schema versioning, retry budgets, validation telemetry, or replayable traces.

In production, the model is one probabilistic component inside a system that has to behave predictably anyway.

That is the distinction I care about:

Prompt engineering improves model behavior.
AI systems engineering makes the product reliable even when the model is imperfect.

The output contract is the real product boundary

If a model is writing prose for a human, the human can interpret the ambiguity.

If a model is returning data to software, the software needs a contract.

In extraction systems, that contract might be an invoice schema, a claims schema, or a lead-enrichment object. In agent systems, it might be a tool-call contract with typed arguments and bounded action shapes. In AI visibility products, it might be a ranked report with source metadata, scoring, provider coverage, and explicit failure states.

The exact schema changes by product. The principle does not:

Model output should not cross into the rest of the system until it has been constrained, validated, and made observable.

That is why I built confident-extract, an open-source Python library for deterministic, schema-constrained extraction from LLMs. It packages a boundary I kept needing in real systems: schema in, constrained generation, strict validation, typed output out.

The specific library is less important than the pattern. Every serious LLM product eventually needs a hard boundary between probabilistic generation and deterministic system behavior.

Reliability is mostly unglamorous work

The parts that make AI systems dependable rarely look exciting in a demo.

They look like:

versioned schemas
field-level validation
retry budgets
idempotent jobs
dead-letter paths
source coverage metadata
model-version tracking
cache keys built from the right inputs
cost guards
validation failure dashboards
replayable inputs for debugging

This is the work that turns "the model worked once" into "the system can be trusted under load."

It is also where real product defensibility often lives. Competitors can imitate a prompt much faster than they can replicate a production-grade control layer around the workflow.

`confident-extract` as shipped proof

I care about this topic enough to have built and shipped part of that reliability layer.

confident-extract exists because I kept rebuilding the same boundary in extraction systems: define the contract, constrain the generation path, validate hard, and only then hand typed data to the rest of the pipeline. The library is intentionally small and opinionated because reliability work benefits from sharp interfaces, not vague abstractions.

For me, open source matters here because it is visible proof of engineering taste. It shows the kind of boundary I believe should exist in production LLM systems.

Its sibling project, promptcrucible, is still unpublished and in active development. I am not treating it as public proof yet because it has not earned that status.

AnswerRank AI as product evidence

AnswerRank AI shows the same systems thinking in a different product shape.

The surface-level prompt for that product sounds easy: analyze how visible a product is inside AI-generated buying answers.

The actual product is not just a prompt.

It needs page extraction, answer generation, mention parsing, competitor comparison, scoring, source metadata, caching, and a usable explanation of what happened during the run. The result should not be a mysterious number. It should tell the user what was measured, which providers were used, and what can be improved next.

That is the same engineering principle again: uncertainty should be surfaced, bounded, and made inspectable instead of hidden behind a confident interface.

The hiring signal I want this to send

The industry does not just need people who can make a model say something interesting on a hand-picked input.

It needs engineers who can build the layer around the model:

the contract
the validation boundary
the workflow
the observability
the cost controls
the product logic
the failure handling
the honest explanation of limits

That is the kind of work I want to be known for.

I am most interested in AI systems and founding-engineer roles where the hard problem is not merely adding an LLM, but making AI reliable enough to become part of the product itself.

Prompt engineering starts the conversation.

AI systems engineering is what ships.

Originally published at https://hitarthdesai.com/blog/beyond-prompt-engineering-ai-systems-layer/