A vendor changes a field type without bumping the API version. Nobody tells you. Your schema validation starts rejecting every inbound order, and within twenty minutes 5,000 webhooks are sitting in an SQS dead letter queue with a retention clock ticking four days by default, fourteen at the outside.
The concrete version of this: Shopify's order payload carries both a numeric id (820982911946154500) and a human facing name (#1001). A middleware layer starts mapping name into the field your service reads as order_id, or an upstream client switches to Shopify's GraphQL global IDs (gid://shopify/Order/820982911946154500). Either way your integer parse blows up, and 5,000 inventory decrements never happen.
The standard remediation is a senior engineer's afternoon. Page, export the queue, poke at three sample payloads until the diff is obvious, write a one-off Python script, test it against a copy, redrive. It works. It is also the least interesting use of your most expensive engineer, and it happens on a Friday roughly as often as it doesn't.
There's a version of this that runs without the human. Not because the model is smart enough to fix your data because the diff is the hard part, and the diff is one sample payload wide.
Use the model to write the fix, not to apply it
The tempting design is to hand all 5,000 broken payloads to an LLM and ask for clean ones back. Don't.
Run the numbers: 5,000 payloads at roughly 1,000 tokens in and 1,000 out is 5M input and 5M output tokens. On Claude Sonnet 4.6 via Bedrock ($3/$15 per million), that's about $90 per incident, and it scales linearly with queue depth a 50,000 message backlog costs $900. You also get non-deterministic output on data you need to be byte-exact, and no way to review the transformation, because there isn't one. There are 5,000 independent guesses.
The alternative is to treat the model as a code generator. Send it one redacted sample payload plus the validation error, get back a deterministic transformation, then run that transformation on ordinary compute. Two thousand input tokens instead of five million. One artifact a human can read in fifteen seconds.
On the data-boundary question: Bedrock inference is genuinely better than a third-party API here AWS states that prompts and completions aren't stored, aren't shared with model providers, and aren't used to train models, and traffic stays inside the AWS network (or a PrivateLink endpoint if you configure one). But that's not a reason to be careless. Your own CloudWatch logging of the Bedrock call will persist whatever you sent, cross-region inference can move payloads out of the region your DPA covers, and one sample payload is a much smaller compliance conversation than five thousand. Redact the sample before it leaves your VPC and the question mostly stops being a question.
The architecture
1. The trap. Messages exceed maxReceiveCount on the source queue and SQS moves them to the DLQ. Nothing clever here; this is just SQS doing its job.
2. The signal. A CloudWatch alarm on ApproximateNumberOfMessagesVisible for the DLQ crosses its threshold. Worth knowing: an alarm cannot invoke Step Functions directly. Alarm actions are limited to SNS, EC2 actions, Auto Scaling, Systems Manager, and since December 2023 Lambda. To reach a state machine you match the CloudWatch Alarm State Change event with an EventBridge rule and target Step Functions from there. Plenty of architecture diagrams draw a straight line from alarm to state machine. It doesn't exist.
3. The orchestrator. Step Functions is the safety harness. It owns the retries, the timeouts, the audit log of every state transition, and critically the approval gate. The AI is a single task inside a workflow you control, not a thing with an IAM role and opinions.
4. The generator. Step Functions passes one redacted sample and the captured validation error (type mismatch: expected integer at $.order_id, received string) to Bedrock, with a prompt along the lines of: given this payload and this error, emit a transformation that maps the payload to the expected schema. For a job this narrow, Claude Haiku 4.5 ($1/$5 per million) is usually enough; reach for Sonnet 4.6 when the schema is genuinely gnarly. Skip Claude 3.5 Sonnet it moved to Bedrock's Public Extended Access tier in December 2025 and now bills at $6/$30, double the current-generation rate for worse results.
5. The mutator. A Lambda function receives the transformation and applies it across the DLQ in batches of ten. It runs well inside the 15-minute ceiling for a backlog this size.
Note that SQS's native DLQ redrive (StartMessageMoveTask) can't help with the transformation step it moves messages verbatim, and it only works for messages that arrived via a redrive policy, not ones written to the DLQ directly with SendMessage. Verbatim replay of a payload your validator already rejected just refills the DLQ. That's the entire reason the mutator exists.
6. The replay. Cleaned payloads go back to the source queue, or onto an EventBridge bus if you want the fan-out. Either is fine. The source queue is simpler and one fewer service in the blast radius.
7. The receipt. Step Functions posts to Slack via Amazon Q Developer in chat applications the service formerly called AWS Chatbot, renamed in February 2025, same APIs and IAM permissions underneath:
5,000 Shopify webhooks failed validation. Root cause:
order_idtype change. Generated a transformation, cleaned the payloads, replayed 5,000 of 5,000. [View generated code]
| Component | Cost |
|---|---|
| Bedrock — Sonnet 4.6, ~2K in / ~600 out | $0.015 |
| Lambda — 512 MB, ~4 min | $0.002 |
| SQS — ~1,000 batched API calls | $0.0004 |
| EventBridge — 5,000 custom events | $0.005 |
| Step Functions — ~20 state transitions | $0.0005 |
| Total | ~$0.02 |
Drop EventBridge and replay straight to the source queue and it's closer to $0.018. Either way, the standing CloudWatch alarm at $0.10 per month costs more than five repairs.
The human comparison: two to three hours of senior engineering time at a $100/hour loaded rate is $200–$300, plus whatever that engineer was actually supposed to ship that day. So roughly four orders of magnitude, and the naive send-everything-to-the-model approach sits between them at $90.
The economics are not really the point, though. Twenty cents versus three hundred dollars is a rounding error against your Bedrock bill for anything else. The point is the four-day retention window, and that the window doesn't care whether your on-call engineer is asleep.
Where this breaks
Running model-generated code in a production Lambda is remote code execution with extra steps. If you build this, build these too.
Don't exec() arbitrary Python if you can avoid it. The stronger design has the model emit a constrained transformation spec a JSONata expression, a JMESPath mapping, a small declarative DSL you wrote and can validate which your Lambda interprets. Now you're parsing data, not evaluating code, and the failure mode is a rejected spec instead of a shell. If you do need real Python, run it through an AST allowlist before it executes and put the whole thing in a separate AWS account.
Understand what the sandbox actually buys you. Restricting the mutator's execution role to sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes and sqs:SendMessage on two specific queue ARNs is correct and necessary. It is not containment. sqs:DeleteMessage alone lets bad code destroy 5,000 customer events, and any egress-capable permission you grant events:PutEvents, a log write is an exfiltration path for whatever the code can read. The role is the security boundary; the Lambda isn't.
Also: cutting internet access requires attaching the function to private VPC subnets with no NAT gateway and adding VPC endpoints for SQS and Bedrock. A Lambda with no VPC configuration has full outbound internet access by default. "No NAT gateway" on its own means nothing.
Gate the replay on a human for anything that moves money. Step Functions' .waitForTaskToken callback pattern is built for this. Generate the transformation, post it to Slack with approve and deny buttons, hold the execution open until someone clicks. Reviewing thirty lines of generated Python takes fifteen seconds; the alternative is an autonomous system with write access to your payment events. For inventory counts, auto-approve. For Stripe payouts, don't.
Idempotency isn't optional, and it never was. SQS standard queues are at-least-once delivery, so your consumer already needed to handle duplicates before any of this existed. Replay just makes the consequences of skipping it arrive on a schedule. If a double-processed inventory update decrements twice, fix that first this architecture will find the bug for you, at the worst possible moment.
The part worth keeping
Dead letter queues are where good data goes to die, usually taking someone's afternoon with it. The failure is almost never hard. It's a type coercion, a renamed field, a null that used to be an empty string. What makes it expensive is that a human has to be awake, paged, and holding context to notice.
Separating the cognitive work reading a diff, writing a transformation from the mechanical work of applying it to thousands of records is what makes the automation tractable. The model does one small thing, once, on one sample. Everything downstream is boring deterministic compute with an IAM role you wrote.
Which is roughly the right shape for putting an LLM anywhere near production.
Top comments (0)