DEV Community: Avneet bansal

Textract vs Bedrock Data Automation vs Vision LLMs on Invoice Photos: 140 Images and a 6-Page Invoice, Real Calls

Avneet bansal — Mon, 09 Nov 2026 15:04:19 +0000

Invoices from the field rarely arrive as clean PDFs. They arrive as phone photos: the invoice lying on a stack of other papers, a coffee receipt on top of it, or a sheet held against a car dashboard. I wanted to know which extraction setup holds up in those conditions, what goes wrong, and what a team can do about it in a few lines of code.

This post is a small comparison. Every number comes from real API calls, and every raw response was saved so the scoring could be re-run without calling AWS again. The code snippets below are the code I used, trimmed to the parts that matter. I ran each one against the saved responses to check that it reproduces the numbers in this post. The last section explains how the benchmark is built, so you can rebuild it for your own documents.

TL;DR

Out of the box, no tool got the invoice total right on more than 83% of the 40 photos. For Textract and BDA, the usual failure was not reading the text. It was picking the wrong amount: "Amount paid", "Amount due" or a previous balance.

Small configuration changes moved totals as much as the choice of tool. A label filter plus a confidence-gated crop took Textract from 42.5% to 85% on totals. A custom BDA blueprint took BDA from 67.5% to 100%.

A subtotal + tax = total check catches most OCR mistakes but few vision-model mistakes. Nova 2 Lite passed that check on 35 of 40 photos, and 20 of those had a wrong field, mostly the date or the invoice number.

Using Textract for the values and Nova 2 Lite as a cross-check auto-approved 22 of 40 invoices with zero silent errors, for about $11 per 1,000.

On photographed pages of a 6-page invoice, checking each page against its own carried-forward amount flagged every page with a real error and missed none.

On 95 real receipts with a labelled total, Textract read it right 97.9% of the time. BDA and Nova often read "31.000" (Indonesian rupiah) as 31.

What was tested

Data. Three sets.

The first set is 5 synthetic invoices in 2 layouts, each photographed in 8 scenes: flat, angled, blur and glare, folded, low light, a stack of papers, a receipt lying on the invoice, and a dark dashboard-like background with a fuel receipt and a parking stub. That makes 40 images. The invoices are generated, so the answer key is exact, and every background paper has known values too. They contain traps that come up in accounts payable work: "Amount paid" and "Balance due" next to "Total", a "Previous balance" line, separate "Invoice to" and "Deliver to" blocks, and DD/MM/YYYY dates.

The second set is the CORD v2 test split: 100 real phone photos of receipts from Indonesia (CC BY 4.0). 95 of them have a labelled total.

The third set is one synthetic 6-page invoice with 149 line items. Each page ends with "Page subtotal (carried forward)" and the next page starts with "Brought forward". The header is only on page 1 and the totals are only on page 6. It was tested twice: as a clean PDF, and with each page photographed in a different scene and combined into one PDF, the way a phone scanner app would.

Tools. All run in us-west-2.

Amazon Textract AnalyzeExpense (synchronous for single images, StartExpenseAnalysis for the 6-page PDF)
Amazon Bedrock Data Automation (BDA) with the AWS public Invoice and Receipt blueprints, and with a custom blueprint
Claude Sonnet 4.5 and Amazon Nova 2 Lite on Amazon Bedrock, through the us. inference profiles, with one JSON prompt that asks for each value plus a 0 to 1000 bounding box (the prompt follows the pattern in aws-samples/sample-document-information-localization), temperature 0
Pre-processing before Textract: Apple Vision document detection with perspective correction, and PaddleOCR's DocPreprocessor (orientation plus UVDoc unwarping)
A spatial check after Textract: textract-field-memory, an open-source library I wrote that learns where fields sit on a known layout (earlier post)

Two modes for Textract and BDA. Out of the box means defaults and the highest-confidence answer. Configured means the change a team would make after reading the output once. The vision models ran with one prompt and no tuning. The configured changes are shown as code below, so you can judge whether they are fair.

Result 1: accuracy on 40 photos

Brackets are 95% intervals. With 40 images they are wide, and that is the honest size of this study.

Setup	Mode	Invoice total right	All 6 key fields right	Median latency	Cost / 1,000
Textract AnalyzeExpense	default	42.5% (29 to 58)	42.5%	3.7 s	$10.00
BDA, public Invoice blueprint	default	67.5% (52 to 80)	65.0%	25.9 s	$40.00
Claude Sonnet 4.5	default	70.0% (55 to 82)	50.0%	6.8 s	$10.70
Amazon Nova 2 Lite	default	82.5% (68 to 91)	37.5%	3.7 s	$1.07
Textract + label filter + confident crop	configured	85.0% (71 to 93)	82.5%	4.0 s	$10.00
BDA, custom blueprint	configured	100.0% (91 to 100)	67.5%	25.2 s	$40.00

The six key fields are vendor, invoice number, invoice date, subtotal, tax and total. BDA latency includes the S3 upload and polling for the asynchronous job.

Textract picks the wrong amount

Textract read the text well. The problem was which amount it labelled TOTAL. On Layout A it took the "Amount paid" line, which is printed as a negative number such as -$100.00, on 7 of 24 photos. On Layout B it was worse: 14 of 16 photos returned the amount due or the previous balance.

Every candidate in the AnalyzeExpense response carries the label printed next to it, so the fix is a few lines:

import re

FIELDS = {"VENDOR_NAME": "vendor_name", "INVOICE_RECEIPT_ID": "invoice_number",
          "INVOICE_RECEIPT_DATE": "invoice_date", "DUE_DATE": "due_date",
          "RECEIVER_NAME": "customer_name", "SUBTOTAL": "subtotal", "TAX": "tax", "TOTAL": "total"}
NOT_THE_TOTAL = re.compile(r"paid|balance|previous|due", re.I)
SKIP_LABEL = {"TOTAL": NOT_THE_TOTAL, "SUBTOTAL": NOT_THE_TOTAL, "TAX": NOT_THE_TOTAL,
              "RECEIVER_NAME": re.compile(r"deliver|ship", re.I)}
MONEY = {"SUBTOTAL", "TAX", "TOTAL"}


def pick_fields(response, configured=True):
    """Highest-confidence candidate per field, skipping amounts that are not the invoice total."""
    best = {}
    for doc in response["ExpenseDocuments"]:
        for f in doc["SummaryFields"]:
            kind = f["Type"]["Text"]
            value = (f.get("ValueDetection") or {}).get("Text", "")
            label = (f.get("LabelDetection") or {}).get("Text", "")
            if kind not in FIELDS or not value:
                continue
            if configured and kind in MONEY and value.strip().startswith("-"):
                continue  # "Amount paid" is printed as a negative number
            if configured and kind in SKIP_LABEL and SKIP_LABEL[kind].search(label):
                continue
            rank = (f["Type"].get("Confidence", 0), f["ValueDetection"].get("Confidence", 0))
            if kind not in best or rank > best[kind][0]:
                best[kind] = (rank, value)
    return {FIELDS[k]: v for k, (_, v) in best.items()}

On real receipts the same filter cost 3.2 points (97.9% to 94.7%), so it is a per-document-type choice, not a free win.

BDA's public blueprint defines TOTAL as the amount due

The public Invoice blueprint defines TOTAL as "total amount due including subtotal and tax". On the two invoices that print a previous balance, it returned the amount due on 13 of 16 photos.

A custom blueprint that asks for the two amounts separately got every total right. This is the full schema (9 fields):

import json
import boto3

S = lambda t, i: {"type": t, "inferenceType": "explicit", "instruction": i}

SCHEMA = {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "description": "A single invoice photographed on a desk or dashboard. "
                   "Other papers may be visible; use only the invoice.",
    "class": "Invoice",
    "type": "object",
    "properties": {
        "VENDOR_NAME": S("string", "Name of the company that issued the invoice, from the invoice header"),
        "INVOICE_NUMBER": S("string", "The invoice number exactly as printed"),
        "INVOICE_DATE": S("string", "Invoice issue date as YYYY-MM-DD. If the invoice says dates "
                                    "are DD/MM/YYYY, convert accordingly"),
        "DUE_DATE": S("string", "Payment due date as YYYY-MM-DD, same date-format rule as INVOICE_DATE"),
        "CUSTOMER_NAME": S("string", "Name in the Bill to or Invoice to block. Not the Deliver to or "
                                     "Ship to block"),
        "SUBTOTAL": S("number", "Subtotal or net amount before tax"),
        "TAX": S("number", "Total tax or VAT amount on this invoice"),
        "INVOICE_TOTAL": S("number", "Total of this invoice: subtotal plus tax. Not the balance due, "
                                     "amount due, amount paid or previous balance"),
        "AMOUNT_DUE": S("number", "Amount due or balance due, if printed separately from the "
                                  "invoice total"),
    },
}

bda = boto3.client("bedrock-data-automation", region_name="us-west-2")
bda.create_blueprint(blueprintName="invoice-benchmark-total-vs-due", type="DOCUMENT",
                     blueprintStage="LIVE", schema=json.dumps(SCHEMA))

One date problem remained. Both Layout B invoices state "Dates are shown as DD/MM/YYYY". The custom blueprint converted 07/10/2026 correctly on all 8 photos of one invoice, and read 11/08/2026 as November 8 on all 8 photos of the other. Instructions guide the model; they do not guarantee the result.

Cropping helps on clutter, fails on the dashboard

Apple Vision's document detection helped on stacks of paper (Textract fields went from 60% to 87.5% right on that scene) and failed on the dashboard scene, where it detected the wrong region with a confidence of about 0.41.

Using the crop only when the detector is confident kept the gain and removed the loss:

CROP_MIN_CONFIDENCE = 0.8

def image_to_send(original, cropped, crop_confidence):
    return cropped if crop_confidence >= CROP_MIN_CONFIDENCE else original

In this set every dashboard crop scored about 0.41 and every other crop scored 0.855 or more, so the threshold was not tested on borderline cases.

PaddleOCR's unwarping helped Textract on the cluttered scenes (stack 60% to 70%, receipt on top 72.5% to 82.5%, dashboard 40% to 50% of fields) and made no difference on angled photos. Its orientation step turned 5 upright pages by 90 degrees, and totals stayed at 42.5%.

textract-field-memory did not help on uncropped photos: no learned template matched any of them. My reading is that the page sits in a different spot in every photo, but I did not test that directly. After cropping, it flagged 46 of 63 wrong fields for review, and also flagged 43 of 257 correct ones. That makes it a review signal, not an extractor.

Result 2: what this means in production

Accuracy does not tell you how much work is left. Two numbers do: how many invoices can be approved without a person, and how many errors slip through.

Rule	Auto-approved	Silent errors	Cost / 1,000
Nova 2 Lite, validation only	35 of 40	20	$1.07
Claude Sonnet 4.5, validation only	34 of 40	14	$10.70
BDA custom blueprint, validation only	35 of 40	8	$40.00
Textract configured, validation only	34 of 40	1	$10.00
Textract configured values, Nova 2 Lite as cross-check	22 of 40	0	$11.07
Textract configured values, BDA custom as cross-check	30 of 40	1	$50.00

A silent error is an approved invoice with any of the six key fields wrong. Validation and the cross-check look like this:

from datetime import datetime


def money(text):
    """'$1,141.60' -> 1141.6, '-$100.00' -> -100.0, 412.68 -> 412.68."""
    if text is None:
        return None
    if isinstance(text, (int, float)):
        return round(float(text), 2)
    m = re.search(r"(-?)[$£€]?(\d+(?:\.\d+)?)", str(text).replace(",", "").replace(" ", ""))
    return round(float(m.group(1) + m.group(2)), 2) if m else None


def validates(f):
    """Key fields present and subtotal + tax = total, to the cent."""
    s, t, total = money(f.get("subtotal")), money(f.get("tax")), money(f.get("total"))
    if None in (s, t, total):
        return False
    if any(not str(f.get(k) or "").strip() for k in ("vendor_name", "invoice_number", "invoice_date")):
        return False
    return abs(s + t - total) <= 0.011


def iso_date(value):
    """Only unambiguous formats. 07/10/2026 could be July or October, so it returns None."""
    for fmt in ("%Y-%m-%d", "%b %d, %Y"):
        try:
            return datetime.strptime(str(value or "").strip(), fmt).date().isoformat()
        except ValueError:
            pass
    return None


def same(a, b, field):
    if field == "invoice_date":
        da, db = iso_date(a), iso_date(b)
        return True if None in (da, db) else da == db
    if field in ("subtotal", "tax", "total"):
        return money(a) is not None and money(a) == money(b)
    norm = lambda x: re.sub(r"[^a-z0-9]", "", str(x or "").lower())
    return bool(norm(a)) and norm(a) == norm(b)


def auto_approve(values, cross_check):
    """Approve with the first tool's values only when both tools pass validation and agree."""
    return (validates(values) and validates(cross_check) and
            all(same(values.get(k), cross_check.get(k), k)
                for k in ("invoice_number", "invoice_date", "subtotal", "tax", "total")))

The arithmetic check works well for Textract: none of the 23 Textract photos with a wrong amount passed it. It works poorly for the vision models, for two reasons. Most of their silent errors are in fields the check never looks at: of Nova's 20, the date was wrong on 15 and the invoice number on 7. Some of their money errors also add up. Sonnet read 381.23 + 31.45 = 412.68 on a blurred photo as 391.23 + 31.40 = 422.63, and 6 of the 12 Sonnet photos with a wrong amount passed the check this way.

Agreement routing works, and the order matters. With Textract supplying the values and Nova checking them, 22 invoices were approved and none were wrong. Swapping the roles (Nova values, Textract check) approved the same 22 with 9 silent errors, all of them dates. Textract returns the date as printed, so a DD/MM/YYYY date cannot be compared safely, and Nova's conversion of those dates was its weak spot.

All 8 of BDA custom's silent errors are the 11/08/2026 date from Result 1.

Result 3: real receipts

95 of the 100 CORD receipts have a labelled total.

Setup	Total right	Read 31.000 as 31
Textract AnalyzeExpense	97.9% (93 to 99)	0
Claude Sonnet 4.5	91.6% (84 to 96)	1
Amazon Nova 2 Lite	83.2% (74 to 89)	12
BDA, public Receipt blueprint	76.8% (67 to 84)	20

If your documents use a dot as the thousands separator, test that case first. It accounts for 20 of BDA's 22 misses and 12 of Nova's 16.

Result 4: a 6-page invoice with 149 line items

Long invoices add problems that single pages do not have: the header and totals sit on different pages, every page has a running subtotal that looks like a total, and a model has to return about 150 rows without dropping or repeating one.

Textract and BDA read the whole PDF natively, so they need no chunking. For the vision models I tested two ways. The first sends all 6 pages in one call. The second is structure-aware chunking: page 1 is read first, then pages 2 to 6 in parallel, each told the vendor, invoice number and date that page 1 returned. Each page is then checked against its own printed running totals, and each page's brought-forward amount must equal the previous page's carried-forward amount:

from decimal import Decimal


def cents(x):
    try:
        return Decimal(str(x)).quantize(Decimal("0.01"))
    except Exception:
        return None


def pages_to_review(pages, subtotal):
    """pages: the model's JSON for each page, in order, with brought_forward, carried_forward and
    line_items. subtotal: from the totals block on the last page. Returns the page numbers to review."""
    flagged, prev = [], None
    for n, p in enumerate(pages, 1):
        if not p:
            flagged.append(n)
            prev = None
            continue
        lines = sum((cents(li.get("amount")) or Decimal(0)) for li in p.get("line_items") or [])
        start = cents(p.get("brought_forward")) or Decimal(0)
        end = cents(p.get("carried_forward"))
        if end is None and n == len(pages):
            end = subtotal
        chained = prev is None or start == prev
        if end is None or not chained or lines + start != end:
            flagged.append(n)
        prev = cents(p.get("carried_forward"))
    return flagged

A flagged page is read once more by the other model at 1.5x size. Code, not the model, merges the pages and checks that all lines add up to the subtotal.

For Textract, the fix is different. StartExpenseAnalysis returns results per page and labels each page's "Page subtotal (carried forward)" as TOTAL with a confidence above 99.9. Take totals from the last page only:

def last_page_only(response):
    def page(doc):
        return max((f.get("PageNumber", 0) for f in doc["SummaryFields"]), default=0)
    last = max(page(d) for d in response["ExpenseDocuments"])
    return {"ExpenseDocuments": [d for d in response["ExpenseDocuments"] if page(d) == last]}


def pick_fields_multipage(response):
    fields = pick_fields(response)
    totals = pick_fields(last_page_only(response))
    fields.update({k: totals.get(k) for k in ("subtotal", "tax", "total")})
    return fields

Clean PDF:

Setup	Total	Line items right	Latency	Cost
Textract async, default	wrong (balance due)	149 of 149, plus 5 running-total rows taken as items	18 s	$0.06
Textract async, label filter only	wrong (a page 3 carried-forward amount)	149 of 149	18 s	$0.06
Textract async, label filter + totals from last page	right	149 of 149	18 s	$0.06
BDA, public Invoice blueprint	right	107 of 149	135 s	$0.24
BDA, custom blueprint	right	not in schema	37 s	$0.24
Sonnet 4.5, whole document	right	149 of 149	87 s	$0.16
Sonnet 4.5, page chunks + checksum	right	149 of 149	48 s	$0.18
Nova 2 Lite, whole document	right	149 of 149	38 s	$0.02
Nova 2 Lite, page chunks + checksum	right	149 of 149	21 s	$0.03

BDA's public blueprint got every line amount right but returned the wrong quantity on 42 of them, mostly the quantity of a nearby row. Those amounts still add up to the subtotal, so a sum check cannot catch it. A quantity times unit price check per row would. On the clean PDF, chunking did not change accuracy, and it roughly halved latency because the pages run in parallel.

Photographed pages (flat, angled, blur, folded, low light, and page 6 on a stack of other papers):

| Setup | Total | Line items right | Lines add up to subtotal |
|---|---|---|---|
| Textract async, label filter + totals from last page | right | 144 of 149 | no |
| BDA, public Invoice blueprint | right | 105 of 149 | yes |
| Sonnet 4.5, whole document | wrong (tax misread) | 129 of 149 | no |
| Sonnet 4.5, page chunks + checksum | right | 123 of 149 | no |
| Nova 2 Lite, whole document | right | 127 of 149 | no |
| Nova 2 Lite, page chunks + checksum | right | 136 of 149 | no |

No setup got every line right on the photos. That is the real result, and it is why the checks matter. Textract and every vision-model setup failed the final "lines add up to subtotal" check, so none of them would be approved silently. BDA's public blueprint passed it while 44 rows did not match on amount and quantity.

The page checksum is the part I would reuse. It flagged every page that had a wrong line, for both models, and missed none:

Model	Pages with a real error	Flagged	Flagged but fine	Missed
Sonnet 4.5	3	3	1	0
Nova 2 Lite	4	4	1	0

The one false flag per model was page 4. Its lines were right, but the carried-forward amount on the blurred photo of page 3 was misread, so the chain check failed on the next page. The retry was less useful: reading a flagged page again with the other model at 1.5x size fixed 1 of the 7 pages with real errors. On photos, chunking tells you which pages a person should look at. It does not repair them. One more thing to watch: Sonnet's chunked run returned the customer as "Eastside Manufacturing Co.", a name that is not on the invoice ("Example Manufacturing Co." is). The whole-document call read it correctly.

Other things worth knowing

Run-to-run variance. At temperature 0, both Bedrock models gave the same answer in all 3 runs for about 94% of fields. Total accuracy moved by up to 2.5 points between runs.
Bounding boxes from vision models. Nova 2 Lite's boxes overlapped the printed value 91.6% of the time. Sonnet 4.5's boxes overlapped 0% as returned. Its y values ran about 1.27 times too large, and dividing y by 4/3 brought 64.4% into overlap. If you need boxes for an audit trail, verify them.
Malformed output. Sonnet returned JSON with an extra closing brace on 2 of 100 receipts and 1 of 120 invoice calls. Nova returned a malformed reply on 1 receipt and 2 invoice calls. Decode the first JSON object and log when you had to:

import json


def first_json(text):
    try:
        return json.JSONDecoder().raw_decode(text[text.find("{"):])[0]
    except ValueError:
        return None

Line items on single pages. The BDA public blueprint got every line item amount and quantity right on all 40 photos. Textract found 82% of them (88% with the confident crop).

Which setup for which job

If you need	Start with
Lowest cost, with a person reviewing	Nova 2 Lite, around $1 per 1,000. Do not auto-approve on an arithmetic check alone
Auto-approval with few silent errors	Textract with a label filter and confident crop, cross-checked by a cheap vision model
Line items on single-page invoices	BDA public Invoice blueprint. On long invoices, check quantity times unit price per row
Line items on long or multi-page invoices	Textract async with totals from the last page had the most lines right here. Add a page-level running-total checksum to whichever tool you use, so a person reviews only the flagged pages
Bounding boxes you can show an auditor	Textract and BDA return a box for every value. I did not measure the accuracy of either; verify any vision-model box
Photos with lots of other papers	A document crop gated on the detector's confidence

Cost

Prices are on-demand list prices in us-west-2, checked on the Textract and Bedrock pricing pages: Textract AnalyzeExpense $0.010 per page, BDA custom output $0.040 per page (the rate for blueprints with up to 30 fields; the public Invoice blueprint has 19, the Receipt blueprint 16 and my custom blueprint 9), Claude Sonnet 4.5 $3.30 and $16.50 per million input and output tokens, and Nova 2 Lite $0.33 and $2.75. The model prices are for the us. profiles used here. The global. profiles are about 10% cheaper ($3 and $15, $0.30 and $2.50).

The whole study, including the 6-page runs and three runs I later replaced, cost about $14.18 at list price.

Limits

This is a small study: 5 invoice designs in 2 layouts photographed 40 ways, 100 real receipts from one country, and one 6-page invoice run once per setup. The photo scenes are generated; the dashboard is a drawn background, not a car. ML Kit and VisionKit phone scanners were not tested. The vision models used one prompt with no tuning, while Textract and BDA got one round of configuration.

How the benchmark is built

If you want to run the same comparison on your own documents, this is the structure. It has four stages, and each stage writes files the next one reads.

Stage	What it does	Output
1. Generate	Render each invoice from HTML with headless Chrome, and write the exact answer key from the same data	Invoice PNG and PDF, `answer_key.json`
2. Photograph	Warp each invoice onto a background with Pillow (perspective, blur, glare, folds, noise, other papers on top or behind) and record which background papers are in each shot	40 JPEGs, a per-photo key with the background values
3. Call	Send each image to each tool once and save the full raw response with its measured latency	One JSON file per image per tool
4. Score	Read the saved responses, apply the configured rules, compare with the answer key	Tables, per-field outcomes, failure images

The answer key carries the traps. Generating the invoices means the key is exact, and it can list every other amount printed on the page, so a wrong answer can be classified instead of just counted:

{
  "inv-01": {
    "layout": "A",
    "vendor_name": "Bluefin Office Supply Co.",
    "invoice_number": "BOS-24817",
    "invoice_date": "2026-10-14",
    "due_date": "2026-11-13",
    "customer_name": "Example Corp",
    "currency": "USD",
    "subtotal": "381.23",
    "tax": "31.45",
    "total": "412.68",
    "line_items": [{"description": "Copy paper, A4, 80 gsm (box of 5 reams)",
                    "quantity": 4, "unit_price": "38.50", "amount": "154.00"}],
    "traps": {"amount_paid": "100.00", "balance_due": "312.68"}
  }
}

The background papers (another invoice, a bank statement, receipts, a parking stub) get their own key, and a check at generation time makes sure none of their amounts equals an amount on a main invoice. Otherwise a background value could not be told apart from a trap.

Every call is made once and saved. Scoring then runs from disk, so you can change a rule and re-score in seconds without paying again:

import json
import time
from pathlib import Path


def cached_call(folder, case_id, fn):
    """Run fn() once and save {'latency_s', 'called_at', 'response'}; later runs read the file."""
    path = Path("results/raw") / folder / (case_id.replace("/", "__") + ".json")
    path.parent.mkdir(parents=True, exist_ok=True)
    if path.exists():
        return json.loads(path.read_text())
    start = time.time()
    response = fn()
    record = {"latency_s": round(time.time() - start, 3),
              "called_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
              "response": response}
    path.write_text(json.dumps(record, indent=1, default=str))
    return record

Each money field gets one of five outcomes. This is what turns "79% accurate" into "it picked the amount-paid line":

def classify_money(pred, truth, traps, background_values):
    """correct, trap (another amount on the same invoice), background (an amount on another
    paper in the photo), wrong, or missing."""
    value = money(pred)
    if value is None:
        return "missing"
    if value == money(truth):
        return "correct"
    # "Amount paid" is printed as a negative number, so compare magnitudes for traps.
    if abs(value) in {abs(money(t)) for t in traps.values()}:
        return "trap"
    if value in background_values:
        return "background"
    return "wrong"

Dates follow the tool's format. A tool that returns ISO dates must convert correctly, and swapping day and month counts as a trap. A tool that returns the printed text must match the printed text. Line items match only when amount and quantity are both right.

The real receipts come from the CORD v2 test split on Hugging Face (naver-clova-ix/cord-v2, CC BY 4.0). Parse the labels and the predictions with the same function, because the labels mix formats ("60.000", "63,000.00", "9.500,00").

If you run invoices through any of these tools, I would like to hear which failure you hit most: the wrong amount, the wrong date, or the wrong row.

Spatial Memory vs Correction Memory: I Benchmarked Both Against a Plain LLM

Avneet bansal — Thu, 05 Nov 2026 04:16:24 +0000

Earlier this year I wrote about two small open-source libraries for document extraction. The first, Give Your Document OCR Pipeline a Memory, introduced textract-field-memory, which remembers where fields sit on a recurring layout. The second, Your LLM Keeps Making the Same Extraction Mistake, introduced sample-prompt-correction-memory, which turns a human correction into something the pipeline keeps.

Each post made its own case. Neither one answered the question readers kept asking me: if you can only add one, which one, and what does each actually buy you against a plain LLM?

So I built a small benchmark that runs both mechanisms against the same document stream, next to a no-memory baseline, and measured them on the same five metrics. The short answer is that they are not competitors. One is a cost lever and the other is a quality lever.

TL;DR. On 120 synthetic invoices and forms with Claude Haiku 4.5 on Amazon Bedrock: spatial memory cut LLM calls from 600 to 30 (about 95%) and p50 latency from 3.3 s to a memory read. Correction memory left the call count alone but cut the repeated-error rate from 0.185 to 0.012. Both raised field accuracy from 0.815 to about 0.99.

The setup

Three arms process the identical seeded stream, in the same order.

Baseline. Plain LLM extraction on every field of every document. No memory. This is the reference for cost and for how often a mistake comes back.

+Spatial memory. Models textract-field-memory. For the first three documents of a type, it calls the model and records where each field sits. Once a field's position is stable within tolerance, it reads the value from the learned region and skips the model call.

+Correction memory. Models sample-prompt-correction-memory. It always calls the model. The first time a field comes back wrong, a simulated reviewer corrects it, and the arm stores the correction as a generalizing transform (for example, "undo the day/month swap"), not as a frozen value. Later documents of that type get the transform applied.

The dataset is 120 synthetic documents, 60 invoices and 60 forms, five fields each (invoice number, date, vendor, total, tax). Two fields per type carry a deterministic, systematic error, the kind a stateless pipeline repeats forever: a date with day and month swapped, and an amount with a stray leading digit (12.34 read as 112.34).

The five metrics were written down in a test plan before any run: field accuracy, LLM calls, token cost, latency, and repeated-error rate. That last one is the share of later occurrences of a field that still show the same error after it was first corrected.

The results

Real-model run: Amazon Bedrock, us.anthropic.claude-haiku-4-5-20251001-v1:0, us-west-2, seed 42.

Arm	Field accuracy	LLM calls	Calls/doc	Token cost (modeled)	p50 latency	p95 latency	Repeated-error rate
Baseline	0.815	600	5.00	$0.0556	3,331 ms	5,592 ms	0.185
+Spatial memory	0.990	30	0.25	$0.0028	~0 ms	156 ms	0.007
+Correction memory	0.988	600	5.00	$0.0556	3,218 ms	4,243 ms	0.012

By document type, the baseline scored 0.80 on invoices and 0.83 on forms. Spatial memory reached 0.99 on both. Correction memory reached 1.00 on invoices and 0.98 on forms.

The cost column is small because the run is small. The ratio is what matters: spatial memory ran at about 5% of the baseline's modeled token cost.

What each number is telling you

Spatial memory is the cost story. The 30 calls it made are exactly the warmup: 3 documents, times 5 fields, times 2 document types. After that, every field on every document was a lookup. That is why p50 latency collapses to a memory read and p95 drops to 156 ms (the tail is the warmup documents). On a stable, high-volume layout, the model is doing work the layout already answered.

Correction memory is the quality story. It does not save a single call, because it still runs the model on every field. What it changes is what happens after the first mistake. In the baseline, the date swap and the stray digit come back on every later document, which is where the 0.185 repeated-error rate comes from. With correction memory, each systematic error is paid for once, by one review, and then it stops. The rate drops to 0.012.

They fix different failure modes. Spatial memory removes redundant calls on fields the layout pins down. Correction memory removes errors the model keeps making on calls that still happen. A pipeline that has both problems needs both layers, and a combined arm is the obvious next configuration to measure.

How this connects the two earlier posts

The OCR memory post ended with a direct ask: the stress tests there were operational numbers, with no labeled benchmark and no comparison against a baseline. This benchmark is a first step on that. It has ground-truth values, a fixed baseline, and pre-registered metrics.

The correction memory post described a three-tier design: graduated rules at zero cost, a confidence-gated LLM tier, and few-shot self-healing for low-confidence fields. This benchmark isolates one property of that design, that a correction applied once stops the same mistake from recurring, and measures it on a stream.

Put together, the two posts describe two separate places a pipeline wastes effort. This post puts numbers on both, side by side.

What this benchmark does not show

These caveats matter more than the headline, so read them before quoting a number.

The arms model the libraries, they do not import them. Each arm is a self-contained reimplementation of the library's core mechanism, so the benchmark runs with no install and no network. The published libraries do more (IoU and center-distance scoring, template identification, drift detection, rule graduation, few-shot retry). Swapping the real libraries into the arms is the next step.

The spatial arm's accuracy is close to an upper bound. When spatial confidence is high, the arm returns the value at the learned position, and on synthetic data that value is the ground truth. In a real pipeline, that read still depends on the OCR being right in that box. The defensible claim from this arm is the 95% call reduction. The accuracy gain says that on a stable layout, a positional read avoids the model's occasional misread, assuming the OCR is clean.

The correction arm knows two error patterns. Its transform learner recognizes the day/month swap and the stray leading digit, because those are the errors the generator injects. A real correction log is messier, and the first post covers how the library handles that with few-shot retrieval instead of hand-written transforms.

The errors are synthetic. In Bedrock mode, the model reads a pre-noised OCR span, so the baseline's errors come from the injected noise model, not from Haiku misreading a clean document. The accuracy and repeated-error deltas are properties of that noise model on this set. Treat them as directional.

Cost is modeled. Measured input and output tokens from the Bedrock usage block, times the published Haiku 4.5 rate ($1.00 per 1M input tokens, $5.00 per 1M output tokens). It is not a bill.

It is small. 120 documents, two types, one seed for the real-model run.

The two mechanisms, in code

The harness is small enough that the core of each arm fits on screen. This is the decision the spatial arm makes for every field:

def confident(memory, key, bbox, warmup=3, tol=0.02):
    """True once a (doc_type, field) has a stable learned position."""
    if key not in memory:
        return False
    learned_bbox, seen = memory[key]
    if seen < warmup:
        return False
    return all(abs(a - b) <= tol for a, b in zip(learned_bbox, bbox))

# per field:
if confident(memory, (doc_type, field_name), field_bbox):
    value = read_region(field_bbox)        # no LLM call
else:
    value = call_llm(field)                # warmup or position moved
observe(memory, (doc_type, field_name), field_bbox)

And this is the correction arm. The important part is that the stored correction is a transform that inverts the systematic error, not the corrected value itself, so it applies to the next document regardless of its specific date or amount:

def learn_transform(raw, truth):
    # Day/month swap: 2026-07-03 read as 2026-03-07
    if raw.count("-") == 2 and truth.count("-") == 2:
        ry, rm, rd = raw.split("-")
        ty, tm, td = truth.split("-")
        if ry == ty and rm == td and rd == tm:
            return lambda v: "-".join([v.split("-")[0], v.split("-")[2], v.split("-")[1]])
    # Stray leading digit: 12.34 read as 112.34
    if "." in raw and raw[1:] == truth:
        return lambda v: v[1:]
    return lambda v: v   # not systematic, do not transform

# per field:
raw = call_llm(field)
if key in corrections:
    value = corrections[key](raw)          # apply what review taught us
elif raw != truth and not reviewed[key]:
    corrections[key] = learn_transform(raw, truth)   # one human review
    reviewed[key] = True
    value = truth
else:
    value = raw

The Bedrock call itself is a single-field Converse request at temperature 0 with maxTokens 64, asking for the value only. Token counts come from the response usage block, which is what drives the cost column.

Rebuilding it yourself

The harness is not public, but nothing in it is hard to rebuild, and the method is the useful part. Generate a few hundred synthetic documents from two or three fixed templates, with ground-truth values and bounding boxes per field. Inject one or two systematic errors per type (a date swap and a stray digit are enough to start). Run every arm over the same seeded stream in the same order, log every field, and compute the metrics from the logs afterwards rather than inside the arms. That last step keeps the arms simple and makes each number auditable against the raw records.

I also ran the same harness with a deterministic stub in place of the model, with no AWS account, to check the metric math. It showed the same structure as the Bedrock run: the baseline at 0.803 accuracy and a 0.197 repeated-error rate, spatial memory at 30 calls and 0.990 accuracy, and correction memory at 1.000 accuracy with no repeated errors. When the stub and the real model agree on the shape, the shape comes from the mechanism and not from one model's behavior.

Where this goes next

Three things are on the list. A combined arm, spatial memory first and correction memory on whatever still reaches the model. The real libraries swapped in for the reimplemented mechanisms. And document types where layouts drift, which is where spatial memory should hand back to the model and where textract-field-memory's drift detection earns its place.

If you run document pipelines and have a layout-heavy, correction-heavy workload, I would like to hear which of the two problems costs you more. That answer decides which layer to add first.

Libraries: textract-field-memory and sample-prompt-correction-memory

From Writing the Spec to Approving It: What AI-DLC Looks Like on a Real Build

Avneet bansal — Fri, 30 Oct 2026 10:19:29 +0000

Most AI-assisted development still follows the old split. A human writes the spec, a human writes the code, and the AI helps with autocomplete or generates tests at the end. AI-DLC (the AI-Driven Development Life Cycle) reverses that. The AI drafts the requirements, the design, and the code. The human answers short questions and approves each stage before the next one starts.

To see what that looks like in practice, I built a small command-line tool with it and then read the record the workflow left behind. This post walks through that record, folder by folder.

TL;DR

AI-DLC is public and open source: awslabs/aidlc-workflows. The method comes from this AWS DevOps blog post.

My only written input was one sentence. Everything else I did was pick options and approve gates.

The run produced requirements, four architecture decisions, a unit plan, a traceability file, and 12 tests that passed when I ran them, in about 44 minutes from start to the code approval.

The human role moves from author to decision-maker. The decisions are still yours, and the record shows where they changed the outcome.

The app

The tool is a Spooky Date Fact Generator. It picks one of 10 spooky dates (or takes --date MM-DD) and prints a spooky event for that day from Wikipedia's "On this day" feed. It is small on purpose.

The whole request was this sentence:

/aidlc build random date generator with fact around that number

The flip, in one table

	Traditional with AI help	AI-DLC
Requirements	Human writes them	AI drafts them from multiple-choice answers
Design	Human sketches it	AI proposes, human picks between options
Code	Human writes, AI assists	AI writes against an approved plan
Tests	Human or AI at the end	AI writes them to the stated requirements
Human job	Author	Decide, validate, approve
Evidence	Pull request diff	Audit log, decisions, traceability file

Where everything was written

Two locations matter. The generated app sits in its own folder, and the workflow record sits in an aidlc/ folder.

app/                                  <- the generated tool
  src/spooky-facts.ts      core: dates, fetch, keyword filter, nearest-date search
  src/cli.ts               command line: options, output, exit codes
  test/                    12 tests, offline, with a fake fetch
  package.json, biome.json, tsconfig.json, README.md

aidlc/
  codekb/                             <- 9 docs about the existing repo
  intents/<date>-date-fact-generator/ <- the record for this build
    aidlc-state.md                    stage checklist and settings
    audit/                            every event, decision, and approval
    inception/
      reverse-engineering/            scan of the existing repo
      requirements-analysis/          questions + requirements.md
      domain-design/                  questions + components.md + decisions.md
      units-generation/               unit-of-work files
    construction/u1-spooky-date-cli/code-generation/
      code-generation-plan.md, code-summary.md,
      unit-test-instructions.md, traceability.json

Each stage writes a -questions.md file, and your answers sit in the same file as the AI's questions. That makes the decision trail readable without any tooling.

Stage by stage: what it did and why it helps

Initialization. The workflow scaffolded the record folder, scanned the workspace, and classified it as brownfield TypeScript on Bun. This is automatic. It helps because every later stage starts from facts about the repo instead of guesses.

Reverse Engineering. A developer agent scanned the existing repo and wrote nine knowledge-base documents (architecture, technology stack, code structure and others). This matters most in a brownfield repo. The later code followed the repo's Bun, TypeScript, and Biome conventions because the AI had read them first.

Requirements Analysis. The AI asked six questions, such as where facts come from, what to do when a date has no spooky event, and whether this should be a web page or a command line. Each had lettered options. I answered, and one answer included a free-text note: generate dates that have spooky facts, from 10 such scenarios. The result is requirements.md with 15 functional and 5 non-functional requirements, each with acceptance criteria and a tag pointing back to the answer that produced it, such as [Q5]. This stage turns one vague sentence into testable statements before any code exists.

Domain Design. Four more questions produced components.md and four architecture decision records. The AI recommended splitting the tool into four parts. I chose two. The decision record states that the four-part split was rejected as more structure than needed. This is the clearest example of the validator role, because the human overrode the AI's recommendation and the reason is recorded.

Units Generation. One question: build as one unit or two. I chose one, approved the plan, and got unit-of-work.md with boundaries, responsibilities, and constraints. It fixes the blast radius. The unit is limited to apps/date-fact-generator/, and nothing in the framework folders may change.

Code Generation. The AI wrote a plan, then the code, then the tests. It asked one thing here: Bun was not installed, so should it install Bun or switch to Node. I chose Bun. The code-summary.md lists files created, decisions made, and deviations from the plan. One deviation worth noting: the AI added word forms such as "dies" and "sank" to the keyword list, because whole-word matching would otherwise miss "Houdini dies". The summary also states that some planned checks were grouped into fewer tests to match the 12-test count.

Build and Test. In the record I read, this stage was still marked in progress, so its summary was not written yet. I ran the checks myself from the app folder: 12 tests passed with 93 assertions, and lint and typecheck were clean.

What the human actually did

Reading the audit log, the human work was this:

Typed one sentence to start.
Approved a recomposed plan. The workflow's planner found the default express plan was missing inputs for later stages, and proposed adding Domain Design and Units Generation while dropping deployment and monitoring.
Answered 11 multiple-choice questions across three stages, plus two more decisions: build inside the existing repo, and install Bun.
Approved 5 stage gates: reverse engineering, requirements, domain design, units, and code generation.

That is the flip. Nothing in that list is writing a spec or writing code. All of it is deciding and checking.

Why this holds up better than a prompt

Two details make the validation real instead of a rubber stamp.

First, traceability. traceability.json maps every requirement ID (FR1.1 through NFR5, 20 in total) to the file that satisfies it. If a requirement has no target, it shows up.

Second, the record is complete. The audit log holds the timestamp of each question, answer, and approval. A reviewer six months later can see why the design has two components and not four.

What this run does not show

This was an express-scope build of a 658-line app, with plan approval, sensors, and learnings switched off by the scope. It shows the shape of the method, not how it behaves on a large system with many units and teams. The Build and Test stage had not completed in the record, and the tests are offline, so only one manual live smoke run, recorded in the code summary, exercised the real Wikipedia feed.

The questions are only as good as the options offered. A weak option list can steer a hurried approver, so reading the alternatives still matters.

Takeaway

The useful change is not that the AI writes more code. It is that the human's time moves to the points where judgment counts: what the tool should do, which design to accept, and whether the result meets the requirements. If you want to try it, start with a small brownfield repo and read the -questions.md files first. They show the method faster than any description.

Organization-Wide EC2 DNS Automation with EventBridge and AWS Config Aggregator: One Rule Per Account

Avneet bansal — Mon, 26 Oct 2026 14:24:48 +0000

Originally published on Medium.

How to detect every EC2 launch and termination across an entire AWS Organization, and turn it into automated DNS registration, without deploying repeated infrastructure in every account.

I spent longer than I would like to admit staring at this problem before the answer got simple. So if you are wiring up DNS across an AWS Organization right now, maybe this saves you a weekend.

Here is the short version of what I landed on. I needed near real-time DNS registration for every EC2 instance across a multi-account, multi-region org, feeding an on-prem enterprise DDI system (think Infoblox). The two obvious ways to do it both annoyed me: one puts infrastructure in every account, the other leans on a fragile cross-account API-scraping job.

What I used instead is a hub-and-spoke event setup:

Each spoke account gets exactly one EventBridge rule. That is the whole footprint.
One hub account does everything else: a custom event bus, an SQS queue, one Lambda.
The Lambda fills in the missing details by querying the AWS Config Aggregator, which, and this is the part I almost missed, already has every instance's IP, tags, and state for the entire org. No new data pipeline required.

Onboarding a new account came down to a one-line deploy. Below is the whole thing: the architecture, why I made the calls I made, and the code.

The problem (why the single-account version does not scale)

In one account this is a five-minute job. EC2 fires a state-change event, you point an EventBridge rule at a Lambda, done. I have written that Lambda a dozen times.

Across an organization it falls apart for a few reasons:

Instances come up in dozens of accounts and several regions.
The people who run DNS are not the people who own the workload accounts, so "just add it to the account" is not really on the table.
The EC2 state-change event does not include the IP address. You get an instance ID and a state. That is it. Useless for DNS on its own.
Nobody wants to deploy, version, and babysit a Lambda plus queue plus IAM role in every account just to catch these events.

The usual reason you end up here is a move to an enterprise DDI platform. Every instance that boots needs a DNS record, every instance that dies needs its record cleaned up, org-wide, in near real-time, with as little junk in each account as you can manage.

So two things really drove the design:

Adding an account should be almost free.
The IP is not in the event, so I have to look it up, and that lookup cannot be flaky.

The approaches I threw out

Option A: put the whole stack in every account. Rule, queue, Lambda, IAM role, shipped everywhere with StackSets. It works. But now you own N copies of the same code, and every bug fix or IAM tweak is an N-account rollout. I have maintained fleets like that. No thanks.

Option B: a central poller that assumes a role into the source account and calls DescribeInstances for the IP. Logic is centralized, which is nice, but you need an assume-role trust in every account and you are paying a cross-account API hop on every single event. More IAM surface, more things to break.

Option C: hub-and-spoke, enriched from the Config Aggregator. This is the one. All the logic sits in the hub. Spokes just forward. And instead of building anything to fetch the IP, the hub asks the Config Aggregator, which is already sitting there aggregating EC2 config (IP, tags, state) from every account and region.

Option C won for a boring but decisive reason: the hard part, the enrichment data, already existed. I did not have to build it. I just had to notice it was there.

How it fits together

Spokes forward EC2 state changes to the hub's custom bus; the hub enriches via the org Config Aggregator, then calls the DDI system. Spokes are dumb, the hub is smart, and the data already exists. That is the whole idea in one line.

That second hub rule is easy to forget, by the way. If you only listen on the custom bus, the hub account's own instances never get DNS records, because they never leave home to reach the custom bus. Ask me how I know.

Filling in the blanks

Most single-account tutorials skip this bit, which is exactly the bit that matters at scale. Here is what the event actually gives you:

{
  "source": "aws.ec2",
  "detail-type": "EC2 Instance State-change Notification",
  "detail": {
    "instance-id": "i-1234567890",
    "state": "running"
  }
}

An ID and a state. No IP, no tags, no hostname. So the Lambda goes and gets the rest, from the aggregator, not from the source account.

The nice thing: the aggregator is org-level, so one query resolves an instance no matter which account or region it lives in. No assume-role, no per-account permissions to manage.

The code

Everything here is vendor-neutral and anonymized. Swap the Infoblox call for whatever you actually talk to: Route 53, a CMDB, an IPAM API.

The spoke's one and only rule (Terraform)

This is the entire per-account footprint. Ship it with StackSets or a Terraform module.

# --- Deployed in EVERY spoke account/region ---

resource "aws_cloudwatch_event_rule" "forward_ec2_state" {
  name           = "forward-ec2-state-to-hub"
  description    = "Forward EC2 running/terminated events to the hub account"
  event_bus_name = "default"

  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail = {
      state = ["running", "terminated"]
    }
  })
}

resource "aws_cloudwatch_event_target" "to_hub_bus" {
  rule           = aws_cloudwatch_event_rule.forward_ec2_state.name
  event_bus_name = "default"
  arn            = "arn:aws:events:${var.region}:${var.hub_account_id}:event-bus/ec2-dns-hub-bus"
  role_arn       = aws_iam_role.eventbridge_cross_account.arn
}

# Minimal role that lets EventBridge PutEvents to the hub bus
resource "aws_iam_role" "eventbridge_cross_account" {
  name = "eventbridge-forward-to-hub"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "events.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "put_events" {
  role = aws_iam_role.eventbridge_cross_account.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "events:PutEvents"
      Resource = "arn:aws:events:${var.region}:${var.hub_account_id}:event-bus/ec2-dns-hub-bus"
    }]
  })
}

The hub's custom bus, with an org-scoped resource policy

This policy is the thing that makes "one rule per spoke" actually work: any account in the org can publish, and the hub needs zero per-account config.

# --- Deployed ONCE in the hub account ---

resource "aws_cloudwatch_event_bus" "hub" {
  name = "ec2-dns-hub-bus"
}

resource "aws_cloudwatch_event_bus_policy" "allow_org" {
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "AllowOrgAccountsToPutEvents"
      Effect    = "Allow"
      Principal = "*"
      Action    = "events:PutEvents"
      Resource  = aws_cloudwatch_event_bus.hub.arn
      Condition = {
        StringEquals = {
          "aws:PrincipalOrgID" = var.aws_org_id
        }
      }
    }]
  })
}

Do not skip the aws:PrincipalOrgID condition. It scopes that Principal = "*" down to your organization instead of, you know, the entire internet. It is the one line standing between "elegant pattern" and "open event bus."

Rules, queue, and Lambda wiring (hub)

resource "aws_sqs_queue" "events" {
  name                       = "ec2-dns-events"
  visibility_timeout_seconds = 120
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 5
  })
}

resource "aws_sqs_queue" "dlq" {
  name = "ec2-dns-events-dlq"
}

# Rule on the custom bus (events from spokes)
resource "aws_cloudwatch_event_rule" "hub_custom" {
  name           = "ec2-state-custom-bus"
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail        = { state = ["running", "terminated"] }
  })
}

resource "aws_cloudwatch_event_target" "custom_to_sqs" {
  rule           = aws_cloudwatch_event_rule.hub_custom.name
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  arn            = aws_sqs_queue.events.arn
}

# Second rule on the default bus catches the hub account's OWN instances
resource "aws_cloudwatch_event_rule" "hub_default" {
  name           = "ec2-state-default-bus"
  event_bus_name = "default"
  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail        = { state = ["running", "terminated"] }
  })
}

resource "aws_cloudwatch_event_target" "default_to_sqs" {
  rule = aws_cloudwatch_event_rule.hub_default.name
  arn  = aws_sqs_queue.events.arn
}

resource "aws_lambda_event_source_mapping" "sqs_to_lambda" {
  event_source_arn = aws_sqs_queue.events.arn
  function_name    = aws_lambda_function.dns_registrar.arn
  batch_size       = 10
}

The enrichment Lambda (Python)

This is the heart of it: resolve the instance against the Config Aggregator, then act on the state.

import json
import boto3

config = boto3.client("config")

AGGREGATOR_NAME = "OrganizationConfigAggregator"

def get_instance_details(instance_id: str) -> dict | None:
    """Resolve IP, tags, account, and region from the org Config Aggregator."""
    expression = (
        "SELECT resourceId, accountId, awsRegion, "
        "configuration.privateIpAddress, "
        "configuration.publicIpAddress, tags "
        f"WHERE resourceId = '{instance_id}' "
        "AND resourceType = 'AWS::EC2::Instance'"
    )
    resp = config.select_aggregate_resource_config(
        Expression=expression,
        ConfigurationAggregatorName=AGGREGATOR_NAME,
        Limit=1,
    )
    results = resp.get("Results", [])
    return json.loads(results[0]) if results else None


def handler(event, _context):
    for record in event["Records"]:
        body = json.loads(record["body"])
        detail = body["detail"]
        instance_id = detail["instance-id"]
        state = detail["state"]

        details = get_instance_details(instance_id)
        if not details:
            # Terminated instances may already be gone from the aggregator;
            # fall back to event data for deregistration.
            if state == "terminated":
                deregister_dns(instance_id)
            continue

        ip = (
            details.get("configuration", {}).get("privateIpAddress")
            or details.get("configuration", {}).get("publicIpAddress")
        )

        if state == "running" and ip:
            register_dns(instance_id, ip, details.get("tags", []))
        elif state == "terminated":
            deregister_dns(instance_id)


def register_dns(instance_id, ip, tags):
    # Replace with your Infoblox / Route 53 / IPAM call.
    # Infoblox example: POST to the WAPI record:a endpoint.
    print(f"REGISTER {instance_id} -> {ip}")


def deregister_dns(instance_id):
    print(f"DEREGISTER {instance_id}")

One thing worth calling out: that SELECT against the aggregator resolves a resource across every account and region in a single call. That is the capability that collapses the whole cross-account enrichment headache into a one-liner. I keep coming back to it for other problems now.

And notice the fallback in the handler. A terminated instance has often already dropped out of the aggregator by the time you look, so you cannot rely on the lookup for deregistration. You fall back to the instance ID from the event. Small thing, but it will bite you in testing if you miss it.

Why there is an SQS queue in the middle

You can target the Lambda straight from the rule, and for a demo I would. In production I put SQS in between on purpose:

A bulk launch, a scale-out, a StackSet rollout, can throw hundreds of events at you at once. SQS soaks that up and lets the Lambda drain at its own pace instead of getting hammered.
When Infoblox or the aggregator has a bad minute, messages retry and eventually land in a DLQ you can actually look at. Nothing quietly disappears.
Batching. Ten events per invocation, fewer cold starts, lower cost.

Decoupling ingest from processing is most of what separates "worked in the demo" from "still working at 2am during a mass launch."

What actually came out of it

Detection and DNS updates happened within seconds, across the whole org.
Adding an account or region was a single forwarding rule. No Lambda, no queue, no Config setup per account.
Registration and cleanup were fully automated across the instance lifecycle, nobody touching DNS by hand.
Cost tracked instance activity, not the clock, because there is no idle compute.
The shape is reusable. Same hub-spoke plus aggregator-enrichment works for CMDB sync, security tagging, IPAM updates, anything org-wide and event-driven.

The part worth stealing

Take Infoblox out of the picture and you are left with a pattern I now reach for often:

Fan org-wide events into one hub, then enrich them from a data source the organization already has, instead of building a new one.

Two moves do the work:

A custom event bus with an aws:PrincipalOrgID resource policy gives you cross-account fan-in and a one-rule spoke footprint.
The Config Aggregator, used as an enrichment lookup, resolves any resource in any account or region without an assume-role fan-out.

So the next time you are about to copy the same Lambda into 40 accounts, stop and ask whether a fan-in plus an aggregator you already run could do it with one rule per account instead. Usually it can.

Wrapping up

Honestly, none of the individual pieces here are clever. Cross-account EventBridge is documented. Config aggregators are documented. The win was in composing them so the per-account cost dropped to near zero and the enrichment data was something we already had lying around. That is where most of the good architecture calls seem to live, not in the shiny new component, but in the one you talked yourself out of building.

If you end up adapting the enrichment step to Route 53, a CMDB, or your own IPAM, I am happy to talk through it in the comments.

Kubernetes Security Is Three Problems, Not One: Admission, Identity, Runtime

Avneet bansal — Thu, 22 Oct 2026 03:11:34 +0000

Kubernetes security failures usually do not look like network intrusions. An attacker who gets code execution inside a container, through an application bug, a poisoned dependency, or a policy gap that let a privileged pod schedule, is already inside the perimeter the firewall was watching. From there the firewall verified nothing: which secrets the process can reach, which syscalls it can make, whether the running image is even the one you signed.

That is the useful reframe. Kubernetes security is not one problem at the boundary. It is three problems inside it, and each one needs its own control that keeps working when the others fail:

Admission decides what is allowed to deploy, and with what configuration.
Identity decides what each running pod is allowed to reach.
Runtime watches what a pod actually does, which is often not what its manifest claimed.

None of the individual tools below are new, and none of this is a new technique. What is worth sharing is how the layers fit together, so a failure in any one of them does not open an exploitable gap. The model maps cleanly onto NIST SP 800-207 (zero trust) and NIST SP 800-190 (container security), and you can hold it to the CIS Kubernetes Benchmark as a measurable floor.

Layer 1: admission, decide what gets in

Start with what Kubernetes gives you for free. Pod Security Admission (PSA) enforces security profiles per namespace with no external tooling. The Restricted profile blocks host namespace sharing, privilege escalation, root contexts, and unsafe seccomp settings. A practical split is Restricted for externally facing namespaces and Baseline for internal platform namespaces with documented exceptions.

PSA is a floor, not a scalpel. It enforces at the namespace level, so it cannot say "this specific Deployment must run as UID 1000 with this seccomp profile." That is where Kyverno comes in. Its policies are plain YAML, they live in the same GitOps repo as your manifests, and they go through the same pull request review as any other change. Four rule types cover most needs:

validate rejects non-compliant resources (for example, require every image to come from an approved ECR registry).
mutate corrects omissions automatically, which turns "you forgot a security context" from an admission failure into a silent fix.
generate creates dependent resources on creation.
verifyImages enforces signature checks at admission time (more on that below).

For constraint logic that depends on the state of other cluster resources, or cross-namespace rules, OPA Gatekeeper and Rego express things Kyverno's YAML cannot. Running both is a reasonable choice: Kyverno for the common case, Gatekeeper where you need Rego's relational power.

One rollout lesson worth stating plainly: both tools have an audit mode, and you should use it first. Measure violations with enforcement off, remediate systematically, then switch to enforce. Flipping straight to enforcement is how a security rollout turns into a production incident.

Layer 2: identity, kill the static credential

A static AWS key in a ConfigMap or an environment variable is a persistent risk with a wide blast radius. IRSA (IAM Roles for Service Accounts) removes the category entirely. The AWS SDK in a pod annotated with an IRSA service account exchanges a short-lived, audience-scoped JWT for temporary IAM credentials through STS. Nothing static is created, stored, or rotated. The IAM trust policy validates the token's subject claim against a specific service account and namespace, so a compromised pod in one namespace cannot assume another namespace's role.

The payoff is blast-radius reduction you can describe concretely. Without IRSA, a compromised pod inherits whatever the node's instance profile can reach. With IRSA, it is limited to that one pod's scoped role.

Application secrets need a parallel mechanism, and that is the ExternalSecrets Operator. It syncs values from AWS Secrets Manager or Parameter Store into Kubernetes Secrets and refreshes them on the source's rotation schedule. When Secrets Manager rotates a database password, the new value propagates without a deployment. IRSA plus ExternalSecrets together removes every static credential: cloud access through IRSA, application secrets through ESO, nothing hardcoded in a Helm values file.

Scaling per-workload IAM roles across hundreds of workloads is the part teams tend to defer. Attribute-based access control with IAM permission boundaries makes it tractable: a role that expresses "this service account, in this namespace, for this environment" can be generated from workload metadata instead of hand-authored each time.

Layer 3: runtime, watch what actually happens

Admission decides what can deploy. Identity decides what it can reach. Neither sees what a pod does after it is running. Falco does, using eBPF probes at the kernel level to observe every syscall from every container on the node. A shell spawned in a container whose image has no shell in its approved process list raises an alert, regardless of how the compromise happened or whether the attacker edited the deployment manifest. Because eBPF sits below the container runtime, it keeps working even if the runtime is tampered with, and it needs no change to the image or the application.

Two practical notes. First, uniform rules across every workload generate false positives that erode trust in the alerts. A data-processing pod that legitimately reads many files will trip rules tuned for an application server. Per-workload rule sets, stored in Git next to the manifests, let you tighten baselines for payment or secret-handling services and loosen them for CI runners. Second, route Falco alerts into a SIEM so individual detections become correlated events. A shell-spawn alert followed within seconds by an outbound connection to an unknown IP is a pattern a correlation engine can act on, including automated network-policy quarantine of the affected pod.

Plan for a calibration period. Default Falco rules produce false-positive volume that takes iterative tuning, with both security and application knowledge, before the layer runs at production-grade sensitivity.

The layer most teams skip: supply chain

An image that passes admission is trusted to be what its manifest says, but only if the reference is cryptographically verified. Tags are mutable. Someone can overwrite the content behind a trusted tag and the cluster will deploy it under the same name, which bypasses every admission policy that checks the image's properties rather than its identity.

Cosign signs images at the registry, and Kyverno verifyImages rules reject anything at admission that is not signed by an approved identity. That closes the tag-mutability gap. SLSA provenance is the next step up: signing proves an image came from your build system, provenance proves what went into that build (source commit, build platform, resolved dependencies). A Software Bill of Materials attached as an attestation then lets you answer "which running images contain this newly disclosed CVE?" by querying the SBOM store instead of inspecting containers one by one.

Putting it together

The point of three independent layers is that no single failure is fatal. A policy gap that lets a bad pod through is still caught by scoped identity and runtime monitoring. A runtime compromise still cannot reach credentials the pod's IAM role never granted. A swapped image is rejected before it schedules.

If you are starting from scratch, a reasonable order is:

Turn on Pod Security Admission at Baseline or Restricted. Free, immediate.
Add Kyverno in audit mode, learn your violations, then enforce.
Move workloads to IRSA and ExternalSecrets to delete static credentials.
Deploy Falco, then budget real time to tune per-workload rules.
Add Cosign signing and verifyImages, then SLSA provenance and SBOMs.

The honest tradeoff is maintenance. Policy-as-code needs upkeep as Kubernetes APIs and workloads change, and Falco rule tuning is ongoing work. The reason to pay that cost is coverage no single control gives you: admission, identity, and runtime failures are different problems, and defending only one of them leaves the other two open.

A Field Guide to Beating Amazon Bedrock Throttling

Avneet bansal — Tue, 20 Oct 2026 20:45:01 +0000

If you run real LLM traffic through Amazon Bedrock, throttling is not an edge case. It is the normal operating condition at scale. A ThrottlingException or TooManyRequestsException is Bedrock telling you that your request rate or token rate crossed a quota for that model in that region. The fix is rarely one setting. It is a layered defense: understand the quota, retry intelligently, cap concurrency, degrade gracefully, handle timeouts, and know what to escalate to AWS.

This is the set of strategies I run across production LLM workloads, written as a generic guide. No workload specifics, just the patterns and when each one earns its place.

TL;DR

Know your quota first: check Service Quotas, learn which limits are adjustable (soft) and which are not (hard).

Retry with exponential backoff and full jitter on throttling errors. This is the floor, not the ceiling.

Cap concurrency so you stop generating load the quota cannot absorb.

Add a circuit breaker so a throttle storm backs off instead of hammering.

Handle timeouts separately: shrink the batch for model timeouts, tune and retry for client read timeouts.

Use cross-region inference profiles to spread token demand across regions.

Reserve provisioned throughput only when on-demand plus the above is not enough.

Step one: know the quota you are hitting

Before writing a line of retry code, find the actual limit. Bedrock quotas are per model, per region, and split by type: tokens per minute (TPM), requests per minute (RPM), and for some models separate input/output token quotas.

Quotas for Amazon Bedrock — the overview, including that default quotas vary by account age, usage history, and region.
Quotas for the bedrock-runtime endpoint — the runtime invoke limits you hit at inference time. View them in the Service Quotas console under Amazon Bedrock.
Scaling and throughput best practices — how per-model TPM and RPM quotas interact.

Soft limits vs hard limits

This distinction decides your whole strategy.

Soft limits (adjustable). In the Amazon Bedrock endpoints and quotas table these are marked Adjustable: Yes. You can raise them. Request an increase through Service Quotas; smaller increases often auto-approve, larger ones route to AWS Support for review.
Hard limits (not adjustable). Marked Adjustable: No. No amount of asking moves these. Your only lever is architectural: reduce demand, spread it across regions, or change the access pattern.

The practical rule: if the limit is soft and you are consistently near it, file the increase and also build the defenses below, because the increase takes time and never makes throttling impossible. If the limit is hard, skip the request entirely and go straight to architecture.

One often-missed detail: Bedrock reserves TPM quota at the start of a request based on input tokens plus maxTokens. If you leave maxTokens unset, it can default to the model maximum and reserve far more quota than your call actually needs, so you throttle earlier than your real token use suggests. Set maxTokens to a realistic ceiling.

Layer one: retry with backoff and jitter

The baseline defense. Catch the throttling error codes (ThrottlingException, TooManyRequestsException), wait, and try again. Two details separate a good retry from a harmful one:

Exponential backoff. Each attempt waits longer: base * 2^attempt, capped at a ceiling. This gives the quota window time to refill.
Full jitter. Randomize the delay inside that window. Without jitter, every throttled caller retries at the same instant and you get a synchronized stampede that throttles again. With jitter, retries spread out.

A compact pattern:

import random, time
from botocore.exceptions import ClientError

THROTTLE = {"ThrottlingException", "TooManyRequestsException"}

def call_with_backoff(fn, *, base=2.0, cap=15.0, max_retries=5):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except ClientError as exc:
            code = exc.response.get("Error", {}).get("Code", "")
            if code not in THROTTLE or attempt == max_retries:
                raise
            delay = min(base * 2 ** attempt, cap)
            time.sleep(random.uniform(0, delay))  # full jitter

A note on the boto3 client's own retries. Boto3 has a built-in retry mode (standard or adaptive) you set via botocore.config.Config(retries=...). You can lean on it, or disable it (max_attempts=0) and own the retry loop yourself. Owning it is worth the effort when you want the backoff, jitter, and error classification to match the rest of your system rather than two retry layers fighting each other.

The same retry in a Step Functions workflow

If your Bedrock calls run inside a Step Functions state machine, you do not write the backoff loop by hand. You declare it on the task's Retry block and the service applies exponential backoff with jitter for you. Step Functions error handling added a JitterStrategy field so wait times are spread instead of synchronized, the same reason full jitter matters in code.

"InvokeBedrock": {
  "Type": "Task",
  "Resource": "arn:aws:states:::bedrock:invokeModel",
  "Retry": [
    {
      "ErrorEquals": ["ThrottlingException", "TooManyRequestsException"],
      "IntervalSeconds": 2,
      "MaxAttempts": 15,
      "BackoffRate": 2.0,
      "MaxDelaySeconds": 30,
      "JitterStrategy": "FULL"
    }
  ],
  "Catch": [
    { "ErrorEquals": ["States.ALL"], "Next": "HumanReview" }
  ],
  "Next": "NextStep"
}

The fields map onto the code version: IntervalSeconds is the base delay, BackoffRate is the exponential multiplier, MaxDelaySeconds is the cap, MaxAttempts is the retry count, and JitterStrategy: FULL randomizes each wait. The Catch routes to a fallback once retries are exhausted, for example a human-review step, so an exhausted retry degrades gracefully instead of failing the whole execution.

Layer two: cap concurrency

Retries handle the request that already failed. Concurrency caps stop you from generating more load than the quota can absorb in the first place. If 50 workers all call the same model at once, you will throttle no matter how clever the backoff is.

Put a ceiling on in-flight Bedrock calls. In async Python a semaphore does it:

import asyncio

semaphore = asyncio.Semaphore(10)  # max concurrent Bedrock calls

async def bounded_call(fn):
    async with semaphore:
        return await fn()

Upstream of the application, cap concurrency at the infrastructure layer too. If documents arrive as events, put them on an SQS queue and let the SQS-to-Lambda event source mapping enforce a maximum_concurrency. Work waits in the queue until a slot frees, with no polling and no custom counters:

resource "aws_lambda_event_source_mapping" "sqs_to_worker" {
  event_source_arn = aws_sqs_queue.work.arn
  function_name    = aws_lambda_function.worker.arn
  batch_size       = 1
  scaling_config {
    maximum_concurrency = 5
  }
}

Match the concurrency ceiling to the quota, not to how many workers you could theoretically run. The right number is the one that keeps steady-state token rate under the TPM limit.

Layer three: a circuit breaker for throttle storms

Backoff and concurrency caps handle normal pressure. A sustained throttle storm needs something that steps back harder. A circuit breaker watches the throttle rate and, once it crosses a threshold inside a time window, pauses all calls for a cooldown, then resumes at reduced concurrency before restoring full throughput.

The state machine:

Closed (normal): calls flow, throttles are counted.
Open (tripped): after N throttles in the window, pause all calls for a cooldown and halve the concurrency limit.
Half-open (recovering): after the cooldown, resume at the reduced limit; if clean for a while, restore full concurrency; if it throttles again, reopen.

This converts a feedback loop that makes throttling worse (retry, throttle, retry, throttle) into one that actively sheds load until Bedrock recovers. It pairs with the concurrency limiter: the breaker is what turns the limit down and back up.

Layer four: handle timeouts, they are not throttling

Throttling rarely travels alone, and the loudest neighbor is the timeout. There are two distinct kinds, and they want opposite fixes.

Model timeout: shrink the work

A ModelTimeoutException means the request itself was too large to finish in the model's time budget. It is not a rate problem, so retrying the same payload will time it out again. The fix is to reduce the work: halve the batch or chunk the input, then retry smaller.

async def call_shrinking(fn, items, *, max_retries=3):
    batch = list(items)
    for attempt in range(max_retries + 1):
        try:
            return await fn(batch)
        except Exception as exc:
            code = getattr(exc, "response", {}).get("Error", {}).get("Code", "")
            if code != "ModelTimeoutException" or attempt == max_retries:
                raise
            batch = batch[: max(1, len(batch) // 2)]  # halve and retry

Client timeout: tune the budget, then retry the transient ones

A client-side ReadTimeoutError or ConnectTimeoutError is different again: the SDK gave up waiting before the model answered. Large prompts and long generations routinely exceed the boto3 default read timeout, which produces timeouts that look like failures but are really an impatient client.

Two moves:

Set a realistic read_timeout on the boto3 client so a slow-but-valid response is not cut off.
Treat the transient connection timeouts as retryable, alongside throttling, rather than letting them bubble up as hard failures.

from botocore.config import Config

client = boto3.client(
    "bedrock-runtime",
    config=Config(read_timeout=300, retries={"max_attempts": 0}),
)

RETRYABLE = {"ThrottlingException", "TooManyRequestsException"}
# also retry ReadTimeoutError / ConnectTimeoutError by exception class name

Classify before reacting. A throttle wants backoff, a model timeout wants a smaller batch, a client timeout wants a longer budget plus a retry, and a ValidationException wants no retry at all.

Layer five: spread demand with cross-region inference profiles

Quotas are per region. A cross-region inference profile lets a single on-demand call be served from one of several regions, so your token demand is spread across each region's quota instead of piling onto one. For on-demand workloads this is often the highest-leverage change: no capacity to buy, and the effective ceiling goes up because you are drawing from multiple regional pools.

It is not infinite headroom, and it does not change a hard per-model limit in a single region, but for spiky on-demand traffic it buys real room.

Layer six: provisioned throughput, only when you need it

If on-demand plus everything above still throttles under predictable heavy load, Bedrock offers Provisioned Throughput: reserved capacity (model units) with a guaranteed token rate, bought by commitment. It removes throttling for that reserved slice because the capacity is yours.

Treat it as the last layer, not the first. It costs whether or not you use it, so it only makes sense for steady, predictable volume. For bursty or unpredictable traffic, retries, concurrency caps, and cross-region profiles on on-demand are usually the better economics.

Tuning in practice

The layers above are the toolbox. How hard you turn each knob depends on the workload, and a couple of patterns are worth stating plainly.

Match the aggressiveness of retries to the cost of a lost call. Heavy, expensive calls justify more retry attempts, a higher backoff ceiling, and tighter concurrency, because losing one is costly. High-volume, cheaper work is better served by an upstream queue that caps concurrency and a workflow-level self-retry that ends in a manual-review fallback, keeping the per-call retries light. There is no single correct number; the right setting is the one that keeps steady-state token rate under the quota for that workload.

Two caveats that are easy to get wrong:

A "fallback model" can quietly resolve to the same model. If both the primary and fallback point at the same model id through shared config, the fallback functions as extra retries against one model, not as model diversity. Calling it a fallback overstates what it does until the config points at a genuinely different model.
Retry, backoff, and concurrency are not the same as reserved capacity. A defense built entirely from retries, backoff, concurrency caps, and on-demand cross-region inference profiles has no reserved-capacity floor under a true spike. That can be a deliberate on-demand posture, but name it as one, because on-demand plus retries still has a ceiling.

How to decide what to reach for

A short decision order when you see throttling:

Check the quota. Is the limit soft or hard? If soft and you are near it, file the increase now, in parallel with the rest.
Confirm retries have backoff and jitter. If not, fix that first. It is the cheapest win.
Cap concurrency to keep steady-state rate under the TPM limit, at both the app and the infrastructure layer.
Add a circuit breaker if throttle storms are recurring rather than occasional.
Separate timeouts from throttles. Shrink the batch for model timeouts, tune the read timeout and retry for client timeouts.
Spread across regions with a cross-region inference profile for on-demand traffic.
Buy provisioned throughput only for steady, predictable, high volume the above cannot cover.

Throttling is not a wall you hit once and clear. It is a pressure you manage continuously. The systems that stay up are the ones that treat the quota as a first-class design constraint, not an exception to catch.

I Moved My Prompts Into DynamoDB and the Throttling Went Away

Avneet bansal — Sun, 18 Oct 2026 23:03:50 +0000

Prompts feel like configuration, so it is tempting to park them in whatever managed prompt-management service your platform hands you. You get versioning, a console, and a clean API, and for a while that is the right call. The trouble shows up later, under load, when prompt retrieval sits on the hot path of every inference call and the service starts pushing back.

That is the wall I hit. The prompt-management layer was throttling requests. Because every call had to fetch its prompt before doing any real work, that throttle became a ceiling on the whole pipeline. The model was not the bottleneck. Prompt lookup was.

I moved prompt serving into DynamoDB and kept the managed service as the authoring source of truth. The throttling went away, and the design has been running well since.

Why a prompt-management service becomes the bottleneck

Prompt-management services are built for authoring and governance, not for being read thousands of times a minute on a latency-sensitive path. They come with request quotas, and those quotas are usually shared across the account rather than scaled to one workload's read traffic. When read volume climbs, you hit the quota long before you hit any limit that has to do with the prompts themselves.

The tell is subtle at first. You start seeing throttling and too-many-requests errors on prompt retrieval, so you add retries with exponential backoff and maybe a fallback for when the service says no. That is reasonable hygiene, but once you are wrapping a plain config lookup in retry logic, the store has quietly become the throughput limit of the application. You can optimize the model, batch the work, and tune everything downstream, and none of it matters, because the request is already stalled waiting for its prompt.

Why DynamoDB fits this access pattern

A prompt lookup is about the simplest access pattern there is. You have a key, and you want the item back. That is a single-item read by primary key, which is exactly what DynamoDB is good at.

Read throughput scales with the table, not a fixed service quota. On-demand capacity absorbs spikes and provisioned capacity with auto scaling handles steady load. The shared quota that was throttling me no longer sat on the path.
The access pattern is a key lookup. Partition on the prompt identifier and every fetch is a single GetItem. No scans, no filtering, predictable single-digit-millisecond reads.
Reads and writes separate cleanly. High-volume reads hit DynamoDB. The rare authoring writes stay in the managed service where review and governance already live, and a sync step pushes changes down into DynamoDB.

The part that makes it safe: checksum-based sync

Caching prompts is easy. Keeping the cache honest is the part worth getting right.

The managed service stays the source of truth for prompt content. DynamoDB holds the serving copy that the hot path reads. To keep them in agreement without re-uploading everything on every deploy, I sync on a checksum.

The sync is write-through. When it detects a change at the source, it writes the new prompt into DynamoDB first, and readers pick it up from there afterward. The flow is simple:

For each prompt in the source of truth, compute a checksum of its content.
Compare that checksum against what is already stored for that prompt.
If they match, do nothing. If they differ, the prompt changed, so write the new content into DynamoDB and record the new checksum.

This turns every sync into a cheap compare instead of a blind rewrite. Unchanged prompts cost a comparison and no write. Only a genuine edit at the source triggers a write. Because the write lands before any read needs it, a new prompt propagates cleanly: the serving copy is already updated by the time the hot path asks for it.

The nice property is that the checksum is the change signal. You do not need a separate "is this dirty" flag or a manual version bump. If the content is identical, the checksum is identical, and the write is skipped. If someone edits a prompt, the checksum moves and the new text is written on the next sync.

If you need rollback or audit: versioning in DynamoDB

DynamoDB does not hand you versioning the way a prompt-management service does, but you can model it, and it is worth saying so plainly because "DynamoDB has no versioning" is a common misread.

If a single current copy per prompt is all you need, the checksum layout above is enough. If you want history, rollback, or an audit trail, add a version to the key:

Partition key: the prompt identifier.
Sort key: the version. A monotonic integer, an ISO timestamp, or the checksum itself all work.
A current pointer: one extra item per prompt (for example sort key CURRENT) that names the active version.

The hot path reads the current pointer by key, so serving stays a single keyed read. History stays queryable with a Query on the partition key. The checksum still decides when a new version row gets written, so you are not stacking a new version for an unchanged prompt. If concurrent syncs are possible, a conditional write (ConditionExpression on the stored checksum) keeps a stale sync from overwriting a newer one.

The tradeoff is simple: the single-copy layout is less to manage, the versioned layout buys rollback and audit. Both are key reads on the hot path, so neither reintroduces the throttling problem.

The shape of the change

The migration itself was small, which is part of why it was the right call.

Model the table around the lookup. Partition key on the prompt identifier. That is the whole schema, because the access pattern was known up front.
Read by key on the hot path. Each inference call does a single keyed read. No scan runs against this table in production, so cost stays proportional to the reads you actually make.
Sync by checksum, not on a timer or on every deploy. The source of truth drives updates, and only real changes cross into DynamoDB.

The result is that prompt retrieval stopped being a shared, rate-limited dependency and became a key read against a store that scales with the traffic hitting it.

What I would tell someone before they copy this

This is not an argument against prompt-management services. They earn their place for authoring, review, and governance. The mistake is letting that same service sit on a high-volume read path it was never sized for.

A few things worth keeping in mind:

The symptom is a throughput ceiling that tracks request volume, not model latency. If you are wrapping prompt retrieval in throttling retries, that is the tell. No downstream tuning will move it.
Design the table around the access pattern first. A key lookup wants a key schema. Pick the partition key from the query you actually run, and keep scans out of production.
Make the cache self-correcting. A checksum compare against the source of truth means the write-through sync updates the serving copy only when content actually changes, without manual version juggling and without rewriting unchanged data.
Model versioning if you need it, do not assume it is missing. A sort-key version plus a current pointer gives you rollback and audit while keeping the hot path a single keyed read.

Moving prompts into DynamoDB was a small change with a disproportionate payoff. The throttling that was capping the pipeline is gone, retrieval is a clean key lookup, and the checksum sync keeps the serving copy honest against the source of truth. It has held up in production since.

The GitOps Bridge: Keep Terraform and ArgoCD Out of Each Other's Way

Avneet bansal — Mon, 12 Oct 2026 04:18:15 +0000

Two tools, two clocks. Terraform provisions the cluster, the node groups, the networking, the IAM. It changes rarely and under heavy approval. ArgoCD deploys the applications onto that cluster. It changes many times a day under light approval. The trouble starts when you let them reach into each other, because then a routine app deploy can be blocked behind an infrastructure change, and an infrastructure change can take applications down with it. The fix is not a better tool. It is a thin contract between the two, so each stays in its own lane. I used this to compress a three-environment EKS migration from a planned six months to about three. Here is the contract, and what it actually bought.

TL;DR

Infrastructure (Terraform) and applications (ArgoCD/Flux) have different change frequencies, approval flows, and blast radii. Coupling them is the root problem.

The GitOps Bridge is a narrow handoff: Terraform writes cluster facts into a well-defined interface (ConfigMap/Secret labels and annotations), and GitOps controllers read that interface to deploy.

Neither side imports the other's state. The interface is the only shared surface.

On a real migration this let three environments come up reproducibly, with the customer executing the IaC themselves under a zero-access constraint, and cut the timeline roughly in half.

The coupling you do not notice until it hurts

The common anti-pattern is to let Terraform deploy applications (via the Helm or kubectl provider) or to let application manifests carry infrastructure facts hard-coded inside them. Both feel convenient. Both create coupling that only shows its cost later.

When Terraform deploys apps, every application change rides the Terraform lifecycle: a terraform apply that should only touch a node group now also reconciles your workloads, and a drift in one app can fail a plan that was supposed to be about infrastructure. When app manifests hard-code infrastructure facts (the cluster's OIDC issuer, a role ARN, a VPC ID, the region), those manifests are no longer portable across environments, and every environment promotion becomes a hand-edit. The blast radius of each change grows to cover both layers, and the audit trail blurs, because you can no longer tell from the change history whether a given commit was an infrastructure decision or an application one.

The two layers genuinely have different natures. Infrastructure changes are infrequent, high-risk, and want slow, careful approval. Application changes are frequent, lower-risk, and want to flow. Forcing them through one pipeline makes the fast side as slow as the slow side, and the slow side as risky as the fast side.

The contract

The bridge is deliberately small. Terraform's job ends at provisioning the cluster and then publishing a set of facts about it into a known location. In practice that location is a ConfigMap and a Secret in a known namespace, carrying labels and annotations that describe the cluster: its name, region, the OIDC provider URL, role ARNs for workloads, networking identifiers, and any environment-specific values the apps will need. Terraform owns writing these. It never installs an application.

ArgoCD's job begins by reading that interface. A root application points at the known ConfigMap/Secret, pulls the cluster facts, and uses them to parameterize everything it deploys. The app manifests themselves carry no hard-coded infrastructure values. They carry references into the bridge. ArgoCD owns reading these. It never provisions infrastructure.

The important property is what is absent. Terraform does not read ArgoCD's state, and ArgoCD does not read Terraform's state. There is no shared state file, no circular dependency where each waits on the other. The only thing they share is the interface, and the interface is data, not control. Terraform writes facts; ArgoCD reads facts. That one-directional, data-only handoff is the whole design, and it is what lets each side keep its own clock.

Why this made a hard migration tractable

The migration that proved it out had a constraint that is more common than people admit: I could not directly touch the target environments. The work had to be delivered as infrastructure-as-code that the customer ran themselves, across three environments, an internal lab, a sandbox, and production, with no interactive access to any of them.

That constraint is exactly where the bridge pays off. Because the apps consume cluster facts through the interface rather than embedding them, the same application definitions deploy unchanged to all three environments. Each environment's Terraform writes its own values into its own bridge, and the GitOps layer adapts automatically. There was no per-environment hand-editing of manifests, which is where this kind of migration usually loses weeks and acquires drift. The customer could run the IaC in each environment and get a reproducible result, because the only thing that varied between environments lived in the bridge, generated by that environment's own provisioning.

For one customer, the combination (Terraform plus the bridge plus GitOps, with Karpenter handling node provisioning on the cluster side) took a migration planned for about six months down to roughly three. The halving did not come from any single tool being fast. It came from removing the coupling that would otherwise have forced serial, hand-held, per-environment work. When each environment comes up the same way from the same definitions, you stop paying the integration tax three times.

How to adopt it

Start by drawing the line. List every value your application manifests currently hard-code that is really an infrastructure fact: cluster name, region, OIDC issuer, IAM role ARNs, subnet or security-group IDs, DNS zone names. Those are the contents of your bridge.

Have Terraform write them. After it provisions the cluster, have it create a ConfigMap (for non-secret facts) and a Secret (for anything sensitive) in a known namespace, populated from its outputs. Decide the key names once and treat them as the contract; renaming a bridge key is now a breaking change, which is the right level of ceremony for it.

Have GitOps read them. Point a root ArgoCD (or Flux) application at that namespace, and parameterize your app set from the bridge values rather than from literals. Each environment gets its own bridge from its own Terraform, and the same app definitions follow.

A minimal, generic version of both sides looks like this. First, Terraform publishes the facts after it provisions the cluster (the values are illustrative placeholders, not from any real environment):

# Terraform: write the bridge after the cluster exists.
# The labels are what ArgoCD's ApplicationSet selects on; the data is the facts.
resource "kubernetes_config_map" "cluster_bridge" {
  metadata {
    name      = "cluster-bridge"
    namespace = "argocd"
    labels = {
      "bridge.example.com/environment" = var.environment   # dev | staging | prod
    }
  }

  data = {
    cluster_name   = module.eks.cluster_name
    region         = var.region
    oidc_issuer    = module.eks.oidc_provider          # for IRSA
    account_id     = data.aws_caller_identity.current.account_id
    # a workload's IAM role ARN, resolved per environment:
    app_irsa_role  = module.app_irsa.iam_role_arn
  }
}

Then an ArgoCD ApplicationSet reads that ConfigMap and templates every app from it, so the app definitions themselves carry references, never literals:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-addons
  namespace: argocd
spec:
  generators:
    - clusters:                      # or a configMap/plugin generator reading the bridge
        selector:
          matchLabels:
            bridge.example.com/environment: prod
  template:
    metadata:
      name: '{{name}}-addons'
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/platform-addons
        targetRevision: main
        path: addons
        helm:
          # bridge facts flow in as values; no hard-coded ARNs or issuers in the chart
          parameters:
            - name: region
              value: '{{metadata.annotations.region}}'
            - name: oidcIssuer
              value: '{{metadata.annotations.oidc_issuer}}'
            - name: appIrsaRole
              value: '{{metadata.annotations.app_irsa_role}}'
      destination:
        server: '{{server}}'
        namespace: kube-system

The exact generator you use (cluster, configMap, or a plugin generator) depends on your setup; the invariant is the one that matters: Terraform owns the left side and writes facts, ArgoCD owns the right side and reads them, and the app charts reference bridge values instead of embedding them. Swap prod for dev or staging and nothing in the app definitions changes, only the bridge each environment's Terraform wrote.

Hold the boundary. The one discipline that keeps this working is refusing to cross the line later under deadline pressure: no Helm-provider app installs sneaking into Terraform, no infrastructure ARNs sneaking back into manifests. The value is entirely in the separation, so the separation is the thing to protect.

The reward is that your fast layer stays fast and your slow layer stays safe. An application deploy no longer waits on an infrastructure plan. An infrastructure change no longer reconciles your workloads as a side effect. And a new environment is a new bridge, not a new round of manual edits. Keep the two clocks separate, and let a thin, one-way, data-only contract be the only thing between them.

If you run Terraform and a GitOps controller together, it is worth asking which infrastructure facts your app manifests still hard-code. Those are the ones the bridge is for.

Your Readiness Probe Is Lying to You

Avneet bansal — Sat, 10 Oct 2026 21:14:36 +0000

A readiness probe is a promise. It tells Kubernetes "route traffic to me, I can handle it." Most of the time the promise is kept because the probe checks the thing that actually matters. The failures worth writing about are the ones where the probe keeps saying Ready while the component has quietly lost the one connection it needs to do any useful work. The traffic keeps arriving. None of it can be served. Nothing looks broken on the dashboard.

I hit exactly this in KEDA, the CNCF event-driven autoscaler, and the fix merged as kedacore/keda #8226. The bug is a good teaching case on its own. The part I did not expect was the second bug hiding inside the fix, which a maintainer caught in review.

TL;DR

KEDA's Metrics Server reported Ready even when its gRPC connection to the keda-operator was down.

In that state the HPA silently stops receiving scale signals, while the replica still looks healthy.

The fix makes the readiness probe observe the real gRPC connection state and report NotReady when the connection is unusable.

A naive version of that fix has a trap: once a replica goes NotReady it stops receiving RPCs, so an Idle connection never gets woken and the replica can stay NotReady forever.

The merged fix nudges an Idle connection to reconnect from inside the readiness check itself.

The setup: who talks to whom

KEDA has two moving parts here. The keda-operator does the actual work of talking to scalers and computing metrics. The Metrics Server (a custom metrics API server) is what the Kubernetes HPA queries when it wants to know "how many of these should be running." The Metrics Server does not compute metrics itself. It forwards the request to the operator over a gRPC connection and relays the answer.

That gRPC connection is the Metrics Server's entire reason to exist. If it is down, the Metrics Server is a shell. It can accept an HPA query and has nothing to answer it with.

The lie

The Metrics Server's readiness probe did not check that connection. It reported Ready as long as the process was up and its HTTP server was listening. So when the gRPC connection to the operator dropped, the following happened, all at once and all invisible: the replica stayed Ready, Kubernetes kept it in the APIService endpoints, the HPA kept sending it metric queries, and every one of those queries failed or hung because there was no live operator connection behind them. The autoscaler stops scaling, and the one signal you would look at, pod readiness, is green.

This is the specific shape of outage that is worst to debug. A crash is loud. A readiness probe that passes while the component cannot serve is silent, and it points you away from the real cause, because the first thing anyone checks is "are the pods ready," and they are.

The fix, version one

The fix is conceptually simple. Make readiness reflect the dependency. gRPC already tracks connection state through grpc.ClientConn.GetState(), which returns one of Idle, Connecting, Ready, TransientFailure, or Shutdown. Crucially, GetState() is a pure observer. It reports the current state and never initiates a connection, so it is safe to call from a probe with no side effects.

So version one added a small accessor on the metrics-service client that returned the connection state, and wired a readyz check in the adapter that reported NotReady unless the connection was Ready. When the operator connection drops, the probe fails, Kubernetes pulls the replica from the APIService endpoints, and the HPA stops sending it queries it cannot answer. When the connection recovers, the probe passes again. That is the behavior you want.

The second bug, caught in review

Here is the part I like. A maintainer looked at version one and pointed out a failure mode that only exists because the fix works.

Think through what happens after the probe correctly reports NotReady. The replica gets removed from the APIService endpoints. Now no RPCs flow through that client at all, because nothing is routed to it. And gRPC, by default, parks a connection in the Idle state after a period with no RPCs (GRPC_CLIENT_IDLE_TIMEOUT_MS, 30 minutes by default). An idle gRPC connection does not reconnect on its own. It waits for someone to use it.

So the naive fix can deadlock the recovery. The connection goes bad, the probe reports NotReady, the replica stops getting RPCs, the connection drifts to Idle, and nothing ever wakes it, because the only thing that would wake it is an RPC, and RPCs only resume once the probe reports Ready again, which requires the connection to be awake. The replica can sit NotReady permanently, needing a manual restart to recover. The fix for a silent hang would have introduced a silent hang.

The resolution is to make the readiness check active, not passive, in exactly one narrow case. When the accessor observes an Idle connection, it triggers a non-blocking reconnect before returning:

// GetConnectionState returns the current connectivity state of the underlying
// gRPC connection to the KEDA metrics service.
//
// When the connection is Idle it also triggers a non-blocking reconnection
// attempt (moving it towards Connecting) before returning. Once a non-Ready
// state removes the replica from the APIService endpoints, no RPCs flow through
// this client, so nothing else would ever wake an Idle connection.
func (c *GrpcClient) GetConnectionState() connectivity.State {
    state := c.connection.GetState()
    if state == connectivity.Idle {
        c.connection.Connect()
    }
    return state
}

Connect() is non-blocking and a no-op when the connection is not Idle, so it is safe to call from a probe. The probe still returns the pre-nudge state (so a just-nudged Idle connection reports NotReady that cycle), but the connection is now moving toward Connecting and then Ready on its own. The replica recovers without a human. The probe went from a pure observer to an observer-with-one-side-effect, and that single side effect is what closes the recovery loop.

Reproduce it yourself

The regression test does not need a cluster or a running operator. It exercises the connection-state logic directly with an in-process gRPC client, so you can run it from a clean checkout:

git clone --depth 20 https://github.com/kedacore/keda.git
cd keda
go test ./pkg/metricsservice/... -run 'TestGetConnectionState' -v

Two tests cover the two behaviors above. One asserts that a freshly created client (gRPC does not dial until first use) is not Ready, so a probe built on it correctly reports NotReady for a replica that has not connected yet. The other asserts the review fix: calling the accessor on an Idle connection moves it off Idle within a couple of seconds without any RPC being issued, so a sidelined replica can recover on its own.

Expected: both subtests pass. (If you are on a network that cannot reach the Go module proxy, the clone's large dependency tree may need go clean -modcache or a working GOPROXY first.)

How to adopt the pattern

The KEDA-specific takeaway is small: upgrade to a release that includes #8226 and your Metrics Server will stop advertising readiness while blind to the operator.

The portable takeaway is the one worth internalizing. A readiness probe should check the dependency that the component cannot work without, not just that the process is up. For a forwarding service, that dependency is the downstream connection. And when you make a probe gate traffic on a connection's health, check whether removing traffic from an unhealthy replica also removes the thing that would let it recover. If recovery depends on activity, and your probe stops activity, you need the probe itself to drive recovery, carefully and without blocking. Check the real dependency, and make sure the healthy path back exists.

A probe that only ever reports the process is alive is answering a question nobody asked. The question Kubernetes is actually asking is "can you serve." Answer that one.

If you run forwarding or adapter services behind readiness probes, it is worth auditing what they actually check. The gap between "process is up" and "can serve" is where the quiet outages live.

A Prompt-Injection Firewall for AWS: What I'd Build, and What I Measured

Avneet bansal — Sat, 10 Oct 2026 20:10:28 +0000

Prompt injection has sat at the top of the OWASP Top 10 for LLM Applications since the list began in 2023, and it is still number one in the 2025 and 2026 editions. That is unusual. Most vulnerability classes get a real fix and slide down the list. SQL injection had parameterized queries. XSS had output encoding and CSP. Prompt injection has no equivalent, because the flaw is architectural: an LLM reads the system prompt, the user's input, and any retrieved documents or tool output as one undifferentiated stream of text, with no built-in boundary between trusted instructions and untrusted data.

I work on LLM systems on AWS, and this is the problem I keep coming back to. This post is not a product announcement. It is the design I would build if I were building a prompt-injection firewall today, grounded in what the research actually says and an honest account of what already exists. I will be clear throughout about which parts are established fact and which are my proposed approach.

TL;DR

Prompt injection is OWASP's #1 LLM risk across every edition, and a 2026 proof shows pure input-wrapper defenses face a hard trilemma: they cannot be continuous, utility-preserving, and complete at the same time.

The space is no longer empty. NeuralTrust ships a commercial "Generative Application Firewall," Check Point acquired Lakera, Cloudflare has Firewall for AI, and open-source projects exist. The honest gap is a cloud-native, AWS-reference, agent-aware design, not "nobody has done this."

My proposal: a multi-layer ensemble with a weighted decision engine, output and tool-call enforcement, and a feedback loop, built the trilemma into the design rather than pretending to beat it.

I built the offline core (heuristic + vector detectors + decision engine) and benchmarked it on 285 labeled prompts. The measured trade-off is stark: at the threshold that catches 92% of attacks, 25% of legitimate prompts get flagged; tighten it until false positives are near zero and detection falls to 39%. No setting gives both. That is the trilemma, measured, and the argument for the semantic layer.

Why this problem resists a clean fix

A 2026 paper, "Why Prompt Injection Defense Wrappers Fail?", formalizes something practitioners had felt for a while. It argues that a defense that wraps the input faces a trilemma of three properties that cannot all hold at once:

Continuity — a small change to the input should not flip the defense decision.
Utility preservation — legitimate prompts must still get through and work.
Completeness — every injection attempt must be caught.

You can get two. Not three. The paper is explicit that this bounds input-wrapper style defenses specifically, and that it does not rule out training-time alignment, architectural changes, or defenses that deliberately trade away some utility. That nuance matters, and I will come back to it, because it is the single most important design constraint.

The practical consequence is that any honest firewall should stop promising completeness. It should instead be explicit about where on the continuity/utility/completeness surface it chooses to sit, and make that choice configurable.

What already exists (the part the hype usually skips)

When I first sketched this idea, my note to myself said "no one has built this." That was wrong, and it is worth correcting in public, because anyone who works in LLM security will know the landscape.

Lakera Guard — commercial, API-first prompt-injection and jailbreak detection. Check Point acquired Lakera in September 2025 and folded it into its platform. Its headline accuracy and sub-50ms latency are the vendor's own figures, not something I measured.
NeuralTrust — ships a commercial product it literally calls a Generative Application Firewall, an inline enforcement layer for LLM apps and agents. So the "GAF" concept is not paper-only; it is a shipping product.
Cloudflare Firewall for AI — a commercial layer that blocks unsafe prompts and addresses several OWASP LLM risks at the edge.
Open-source — several GitHub projects named prompt-shield (and the one-word promptshield) implement detector ensembles, reverse-proxy enforcement, and threat scoring. LLM Guard is an active rule-based scanner set, and NeMo Guardrails (NVIDIA) handles conversation-flow control. Rebuff, an earlier multi-layer project, is archived.

The "GAF" framing itself comes from a 2026 paper, "Introducing the Generative Application Firewall (GAF)", which proposes a single enforcement point that unifies prompt filters, output validators, and data masking, maintains session context, and covers agents and their tool calls, by analogy to a WAF for web traffic.

So the honest gap is narrower and more specific than "nobody has built a firewall." It is: an open, cloud-native reference architecture on AWS primitives that is agent-aware (intercepts tool calls, not just prompts), uses a multi-detector ensemble rather than a single technique, and is explicit about the trilemma trade-off it makes. That is a real and useful gap. It is not a claim of inventing the category.

The architecture I'd build

The design is a pipeline: detect, decide, enforce, learn. Each stage maps cleanly onto AWS primitives, which is the point, it should be deployable without inventing new infrastructure.

          ┌───────────────────────────────────────────────┐
 request  │  1. DETECTION  (ensemble, run in parallel)      │
 ───────► │   heuristic · ML classifier · LLM-as-judge      │
          │   vector similarity to known attacks            │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  2. DECISION  (weighted vote + session context) │
          │   configurable threshold → the trilemma dial    │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  3. ENFORCE   allow / block / sanitize / review │
          │   also: scan the OUTPUT, gate TOOL CALLS         │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  4. LEARN   log verdicts, feed back FPs,         │
          │   grow the attack-vector store                   │
          └───────────────────────────────────────────────┘

On AWS, I'd map it like this: API Gateway as the entry point; the detector ensemble as Lambda functions, with the ML classifier on SageMaker and the LLM-as-judge on Bedrock; the attack-vector store on OpenSearch; the decision engine as a small Step Functions or Lambda orchestration reading session history from DynamoDB; and CloudWatch plus S3 for the audit trail. None of that is exotic, which is deliberate.

Why an ensemble, and what each layer is actually good at

No single detector is enough, and the reasons are well documented. The CAITLYN paper puts the trade-off bluntly: deterministic signature and regex filters run at sub-millisecond speed with zero token cost but fall to trivial obfuscation and semantic rewrites. So the ensemble is not padding; each layer covers a different failure mode of the others.

A heuristic layer is the cheap first pass. Illustrative, not a complete ruleset:

SUSPICIOUS = [
    r"ignore (all )?previous instructions",
    r"disregard the (system|above) prompt",
    r"you are now [a-z ]+",          # role-override attempts
    r"reveal (your )?(system )?prompt",
]

def heuristic_score(text: str) -> float:
    hits = sum(bool(re.search(p, text, re.I)) for p in SUSPICIOUS)
    return min(hits / 2, 1.0)   # saturate; this is a signal, not a verdict

That catches the lazy attacks for free and catches nothing clever, which is exactly why it only contributes a signal. A semantic layer (an ML classifier, or an LLM-as-judge on Bedrock) catches the paraphrased and indirect attacks the regex misses, at real token cost and latency. A vector-similarity layer compares the input against an embedding store of known attack patterns, so a variant of something you have seen scores high even if the wording is new.

The decision engine is where the trilemma becomes a dial

This is the heart of the honest version. Instead of a single detector shouting "block," the engine combines the signals into one score and compares it to a threshold you set per tenant or per use case:

WEIGHTS = {"heuristic": 0.15, "classifier": 0.35, "judge": 0.35, "vector": 0.15}

def decide(signals: dict[str, float], threshold: float) -> str:
    score = sum(WEIGHTS[k] * v for k, v in signals.items())
    if score >= threshold:            return "block"
    if score >= threshold * 0.6:      return "flag_for_review"
    return "allow"

The threshold is the trilemma choice, made explicit. A bank's internal agent can run it high and tolerate more false positives, because a missed injection is expensive. A low-stakes public chatbot runs it lower to protect utility. The firewall does not pretend to be complete; it hands you the knob and logs where you set it. That framing is the design taking the arXiv result seriously instead of marketing past it.

The weights above show the full four-detector design. The benchmark in the next section only exercises the two detectors I actually built offline (heuristic and vector), so it runs with two weights rather than four. The semantic detectors in this snippet are the layer I have not built yet.

I built the offline core and measured it

Design claims are cheap, so I implemented the two offline layers (the heuristic detector and the vector-similarity detector) plus the decision engine, and ran them against a labeled corpus of 285 prompts: 192 injection and 93 benign. The injection set is deliberately held out, paraphrases, obfuscations (spaced-out and hyphenated triggers), and indirect-via-document payloads that are not the examples the vector detector was fitted on, so the catch rate reflects generalisation rather than memorisation. The benign set includes hard negatives that mention "instructions," "system prompt," "override," and "ignore" innocently. Everything is pure standard-library Python with a fixed seed, so the run is deterministic and reproducible.

Here is the actual output, sweeping the decision threshold (weights: heuristic 0.4, vector 0.6):

 thresh  detect%    FP%  precision     F1
--------------------------------------------
   0.15      98%    52%       0.80   0.88
   0.18      92%    25%       0.88   0.90
   0.20      84%    14%       0.93   0.88
   0.25      60%     8%       0.94   0.74
   0.30      39%     1%       0.99   0.55
   0.40      10%     0%       1.00   0.18
   0.60       5%     0%       1.00   0.09

Read that top to bottom and the trilemma stops being an abstraction. At a threshold of 0.18 the firewall catches 92% of the injections, but it also flags a quarter of legitimate prompts, the hard benign negatives about writing system prompts and documenting team instructions get swept up. Tighten the threshold to 0.30 and false positives drop to 1% (precision 0.99), but detection collapses to 39%: most attacks walk straight through. There is no row where detection is high and false positives are low at the same time. The best F1 (0.90) sits near the aggressive end, where you have already accepted a real false-positive cost.

That is not a tuning failure I can polish away with better weights. It is the shape the 2604.06436 result predicts for this class of defense, and watching it fall out of real numbers is more convincing than any single headline accuracy figure. The honest reading is specific: a cheap offline ensemble reliably catches the obvious and the lexically similar, and genuinely struggles to separate a subtle paraphrase of an attack from a legitimate request that happens to talk about prompts. That overlap is exactly the region a semantic layer, an ML classifier or an LLM-as-judge on Bedrock, is supposed to resolve, and it is why I treat that layer as the next build step rather than a nice-to-have.

A note on honesty, because it is the whole point of this post: the corpus is synthetic and template-generated, so its diversity is bounded by the templates; the numbers are operational rather than a formal benchmark against published baselines; and the implementation is not open-sourced. I would not quote these figures without having run the code, and I would not dress them up as production-grade. They are a measured illustration of a trade-off, nothing more and nothing less.

The two things that make it agent-aware

Most of the field still treats this as "scan the user's prompt." For modern agentic systems that is half the surface. Two additions matter:

Output scanning. The model's response can carry an injected instruction or leak a system prompt. Scanning the output, not just the input, is where several designs (including NeuralTrust's and the GAF paper's) converge, and I'd treat it as mandatory rather than optional.

Tool-call gating. This is the one I care most about, and where indirect injection lives. When an agent decides to call a tool, that decision can be the payload: a poisoned document retrieved via RAG tells the agent to call send_email or delete_record. Gating the tool call, checking the proposed action and arguments against policy before execution, is a different and arguably more important checkpoint than filtering the prompt:

def gate_tool_call(tool: str, args: dict, policy) -> str:
    if tool in policy.denied:                       return "block"
    if tool in policy.requires_human and args_risky(args, policy):
        return "human_review"
    return "allow"

The "Firewalls to Secure Dynamic LLM Agentic Networks" work takes this further, projecting cross-agent messages onto a structured, task-scoped protocol so that persuasive framing and embedded instructions are stripped by construction. That is a stronger idea than scoring, and it is the direction I would want the tool-call layer to grow toward.

Honest limitations, up front

Because a firewall that oversells itself is worse than none:

It cannot be complete. The 2604.06436 result makes that a theorem for input-wrapper defenses, not a matter of effort. Anyone claiming 100% is wrong or redefining the problem.
The semantic layers cost latency and tokens. The sub-50ms figures you see are vendor numbers for narrower, single-layer detectors; a multi-layer ensemble with an LLM-judge step will cost more, and I would not quote a latency number I had not measured.
It is defense in depth, not a boundary. The durable lesson from the evaluation literature is that security boundaries belong in application code and permissions. A firewall lowers the hit rate; least-privilege tool scoping and human approval on destructive actions are what contain the ones that get through.

On the compliance angle

It is tempting to attach urgency to regulation, so here is the accurate version. The EU AI Act's high-risk obligations were pushed back by the 2026 Digital Omnibus to December 2027 and August 2028, not August 2026 as early drafts implied. What did take effect on August 2, 2026 is the transparency regime (Article 50) and penalty powers for general-purpose models. So the regulatory pressure is real but phased, and the honest pitch is that security controls for LLM apps are becoming table stakes over the next two years, not that a deadline hits tomorrow.

The path forward

The offline core and the trade-off measurement above are done. From here the order would be:

Add the semantic layer (an ML classifier, or an LLM-as-judge on Bedrock) specifically to attack the overlap region the offline ensemble could not separate, and re-measure whether it buys real detection without pushing false positives back up. That is the hypothesis the current numbers set up.
Harden output scanning and tool-call gating, and benchmark against indirect-injection scenarios (poisoned RAG content, malicious tool results), since that is the underserved part.
Publish it as an AWS reference architecture with the numbers attached, honest about where it sits relative to NeuralTrust, Lakera, and the OSS projects, rather than claiming to replace them.
Close the loop: feed false positives back to retune thresholds and grow the attack-vector store, and treat that feedback data as the real moat over time.

That is a build I would stand behind, because every claim in it is either cited or measured. The gap it fills is specific: an open, agent-aware, AWS-native reference design that treats the defense trilemma as a dial instead of a thing to lie about.

If you are working on LLM security, I'd genuinely like to hear which layer you think earns its keep, and whether tool-call gating or output scanning has mattered more in your own systems.

Sources

OWASP Top 10 for LLM Applications (2025, 2026) — prompt injection ranked #1 across editions. https://genai.owasp.org/llm-top-10/
arXiv 2604.06436 — "Why Prompt Injection Defense Wrappers Fail?" (the defense trilemma)
arXiv 2601.15824 — "Introducing the Generative Application Firewall (GAF)"
arXiv 2608.27990 — "CAITLYN: Can LLM Agents Autonomously Synthesize Defenses against Emerging Injection Attacks?"
arXiv 2502.01822 — "Firewalls to Secure Dynamic LLM Agentic Networks"
Check Point acquires Lakera (Sept 2025) — csoonline.com
NeuralTrust Generative Application Firewall; Cloudflare Firewall for AI (vendor sources)
EU AI Act timeline after the 2026 Digital Omnibus (high-risk obligations deferred to Dec 2027 / Aug 2028)

The benchmark figures come from my own offline reference implementation (pure standard-library Python: regex heuristic + character tri-gram cosine detectors + a weighted decision engine), run against a synthetic, template-generated labeled corpus of 192 injection and 93 benign prompts (285 total), with seed attacks held disjoint from the test set. The implementation is not open-sourced; the numbers are operational, not a formal benchmark.

Cut LLM Document-Extraction Cost by 85% Without Losing Accuracy

Avneet bansal — Wed, 07 Oct 2026 02:23:09 +0000

Most production LLM extraction pipelines route every field through a frontier model, every time. It works, and for a while nobody questions the bill. Then volume grows, or someone runs the per-document math, and the real question shows up. How much of this spend is buying accuracy, and how much is just habit.

In practice, most of it is habit. The majority of fields in a typical extraction job do not need a frontier model at all. A large share of the model calls answer questions the document never asked. A surprising fraction of calls re-extract data that has not changed since the last run. None of that is a model problem. It is an architecture problem, and architecture problems are the good kind, because you can fix them without waiting on anyone's roadmap.

I build cost-optimized inference for document-extraction workloads with 100+ fields. This is the practitioner's version of what actually moves the number. The moves below, applied in sequence, take per-document cost down by 85 to 93 percent with negligible accuracy impact. The interesting part is not any single technique. It is why they compound instead of overlap.

TL;DR

Set a trusted baseline with the expensive model first, then prove a cheaper model can match it.

Skip fields a document type cannot contain (adaptive selection).

Route each field to the cheapest model that can do the job (tiered routing + distillation).

Stop re-extracting unchanged fields (field-level caching).

Stop paying for idle (serverless scale-to-zero).

The savings multiply, not add, because each move attacks a different kind of waste.

🧭 Set the baseline with the expensive model first

Before optimizing anything, run the workload on a frontier model and treat those results as the baseline. Early on the goal is not low cost. It is output you trust and a confidence signal you believe. Once that baseline exists, it becomes the yardstick for everything cheaper.

Then, field by field, test whether a smaller and cheaper model can match it. Run the cheap model over the same documents, compare against the frontier baseline, and keep the fields where agreement is high. The fields that match move to the cheaper tier. The fields that do not stay on the frontier model.

This is how the tier percentages below get decided in practice. You are not guessing which fields a lightweight model can handle. You are measuring it against output you already trust.

The same comparison runs later as the quality-floor monitor at the end of this post. It starts here, as the step that earns each downgrade.

✂️ Start by not doing unnecessary work: adaptive field selection

The cheapest extraction call is the one you never make.

A contract schema might define 100+ fields, but a given document, say a simple service agreement, only contains 40 to 50 of them. Extracting the rest produces "not found" responses that cost tokens and return nothing. Classify the document type first, then extract only the fields that can actually appear in it.

This one move eliminates 30 to 40 percent of calls per document. It is the right thing to do first, because every later optimization then operates on a smaller base.

🎯 Right-size the model to the field: tiered routing

Not every field needs the same horsepower. In a typical extraction job:

~65 to 70 percent of fields are structurally simple: dates, amounts, party names, reference numbers, categorical flags. Predictable formats, consistent locations. A lightweight model handles these at a fraction of frontier cost with equivalent accuracy.
~20 percent need moderate reasoning: interpreting indirect phrasing, resolving ambiguous references, synthesizing across sections. A mid-tier model (for example a Haiku or Nova Lite class model) is plenty.
~10 percent genuinely need a frontier model: obligation summaries, risk assessment, termination-condition analysis, cross-document context.

Route each field to the cheapest tier that can do the job. The routing decision is nearly free. It is a field-type to tier lookup table plus a confidence-based fallback, so a tier that returns low confidence escalates to the next one. No ML is needed for routing, because the mapping is stable for a given document corpus. Blended per-document cost drops 60 to 75 percent, and quality holds because the hard fields still get frontier attention. The split above is not assumed. It comes straight out of the baseline comparison from the previous section.

Distillation: earn your way to a cheaper Tier 1

Tiered routing gets better when Tier 1 is a model you distilled for your own documents. Run a frontier model (the teacher) over 200 to 500 representative documents to produce gold-standard extractions, then train a smaller student model to reproduce that behavior on your specific fields. Managed services such as Amazon Bedrock Model Distillation handle the training without ML expertise.

The student model learns your narrow task, the dates, amounts, party names, and term lengths you see every day, and it runs much cheaper and faster with a small accuracy gap (about 2 percent) on the tasks it was trained for. Fields outside its training distribution fall through to the mid-tier or frontier model.

The bootstrap problem is obvious. You do not have 200 documents on day one, so distill progressively. Months one and two run routing-only on general-purpose models (40 to 50 percent savings) while storing every validated result as future training data. In month three, once you have accumulated enough examples, distillation runs and Tier 1 becomes your student model (another 30 to 40 percent off). Each quarter you re-distill with more data, and the share of fields handled by the cheapest tier climbs from about 70 percent toward 85 percent. The cost curve bends down the longer the system runs.

♻️ Stop paying twice: field-level caching

Documents get reprocessed when schemas are updated, prompts improve, or a quality review flags systematic errors. Without caching, reprocessing 200 documents costs exactly what processing them the first time did.

Key each result by the document-section SHA-256, the field name, the prompt version, and the model version. If all four match, return the cached value with zero inference. When you update a prompt for one field type, only that field's cache entries invalidate, and everything else serves from cache. A prompt update typically touches three to five field types, so 85 to 90 percent of fields serve from cache on reprocessing. For a 200-document corpus reprocessed after a four-field prompt change, that is 800 extractions instead of 7,000, an 88 percent reduction on that cycle.

🛑 Stop paying for idle: serverless scale-to-zero

A 200-document-per-month workload is idle most of the time. Documents land in object storage, a lightweight function writes metadata that triggers a stream event, and a containerized extraction agent runs only when work exists. During idle periods the system costs nothing beyond storage. For bursty, low-volume workloads this removes the fixed infrastructure floor that usually makes small deployments uneconomical. KEDA is a clean way to get scale-to-zero and event-driven scaling for this on Kubernetes.

🧮 Why these compound instead of add

A tempting mistake is to add the savings together. Distillation saves 75 percent, adaptive selection 35 percent, caching 30 percent, so the total must be 140 percent. That is impossible. The techniques do not each operate on the original cost. Each operates on the residual the previous stage left behind. They address different sources of waste, so they multiply:

Stage	Technique	Operates on	Residual
1	Adaptive field selection	Total field count	60 to 70 percent of baseline
2	Distillation + tiered routing	Remaining fields	20 to 25 percent of baseline
3	Field-level caching	Reprocessed documents	10 to 15 percent on reprocessing
4	Serverless scale-to-zero	Infrastructure	Zero idle cost
Result	Compound effect	End to end	5 to 15 percent of naive baseline

Roughly (1 - 0.35) x (1 - 0.65) x (1 - 0.25), which lands around 0.07 to 0.15 of the original, an 85 to 93 percent reduction. Each technique matters most at a different stage. Adaptive selection eliminates unnecessary work, distillation reduces the cost of necessary work, caching eliminates repeated work, and serverless eliminates idle overhead. No one of them gets you there alone.

🛡️ Making it safe: the quality-floor monitor

Aggressive cost optimization earns an obvious objection. If 70 percent of fields go to a cheaper model, you need a way to know it is not silently getting worse, or drifting as document patterns change.

Build the check in. Each week, randomly re-run 5 to 10 percent of Tier-1 fields through the frontier model as a shadow extraction and compare. This is the same baseline comparison from the start of the post, now running continuously. Agreement at or above 98 percent means the floor is holding. Below 95 percent raises an alert. Below 90 percent auto-triggers re-distillation on the last three months of data. Shadow validation adds only about 3 to 5 percent to total cost, and it means the system monitors itself, with no dedicated QA team required.

The same monitor also spots promotions. If the mid-tier and frontier models consistently agree on a field currently routed to Tier 2, that field is a candidate to fold into the next distillation cycle, so the system keeps shifting work to the cheapest tier on its own.

💡 The payoff beyond the bill

The reason this is worth doing well is not only the savings on an existing workload. Every percentage point off per-unit cost widens the set of use cases where automation is worth building at all. A workload that only penciled out for one of your five document types at naive pricing might clear the bar for three or four once the architecture is right. Cost optimization is not a late-stage cleanup. Treat it as a first-class design requirement alongside accuracy, latency, and reliability, and more of what you want to automate becomes viable.

Five moves, five kinds of waste:

Expensive-model baseline, then prove the cheap model turns every downgrade into a measured decision.
Adaptive field selection removes unnecessary work.
Tiered routing + distillation reduces the cost of necessary work.
Field-level caching removes repeated work.
Serverless scale-to-zero removes idle overhead.

Apply them in sequence, guard them with a self-checking quality floor, and you get frontier-grade extraction at a small fraction of the naive cost, without betting accuracy on it.

Two patterns I open-sourced

Two of the ideas above are available as small, dependency-light libraries under AWS Samples (MIT-0), if you want a reference to build from:

aws-samples / sample-textract-field-memory

Spatial field location memory for document processing pipelines. Learns field positions, validates extractions, identifies document types by layout, detects drift, and monitors template health. Zero dependencies, pure Python.

textract-field-memory

Disclaimer

This is sample code, for non-production usage. You should work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment.

Note: When processing documents containing PII (e.g. SSNs), PHI, or payment data, ensure your implementation meets applicable compliance requirements (HIPAA, PCI-DSS, GDPR, etc.). This library stores only field names and bounding-box coordinates — never field values or document content — but you remain responsible for securing the underlying documents.

The Problem

Every system that processes recurring structured data starts from scratch each time. It has no memory of what it saw yesterday.

                    WITHOUT spatial memory
┌──────────────┐
│  Document 1  │──► OCR/Extract ──► "Employee Name" at (0.05, 0.10) ──► ✓ stored nowhere
│  Document 2  │──► OCR/Extract ──► "Employee Name" at (0.05, 0.10) ──► ✓ re-discovered
│  Document 3  │──► OCR/Extract ──► "Employee Name" at (0.05, 0.10) ──► ✓ re-discovered again
│

…

View on GitHub

aws-samples / sample-prompt-correction-memory

Self-improving LLM document extraction on AWS. Each human QA correction improves future extractions via few-shot self-healing and deterministic rule graduation — no retraining, no redeployment. Costs decrease over time.

sample-prompt-correction-memory

A sample project showing automated prompt self-correction for LLM document extraction on AWS. Each human correction improves future extractions — no retraining, no redeployment.

Disclaimer: This is sample code, for non-production usage. You should work with your security and legal teams to meet your organizational security, regulatory and compliance requirements before deployment. Do not use this sample with personal data, health data or other regulated data.

The Problem

Every LLM-based extraction pipeline makes the same mistakes repeatedly. Your QA team corrects an error today, but the model makes the identical error on tomorrow's document. The correction evaporates.

                    WITHOUT correction memory
┌──────────────┐
│  Document 1  │──► LLM Extract ──► "quarterly" for payment_terms  ──► QA corrects to "30"
│  Document 2  │──► LLM Extract ──► "quarterly" for payment_terms  ──► QA corrects AGAIN
│  Document 3  │──► LLM Extract ──► "quarterly" for payment_terms  ──► QA corrects AGAIN
│     ...      │
└──────────────┘
    Same

…

View on GitHub

If you are optimizing an extraction pipeline, I would be curious where your biggest cost sink turned out to be: unnecessary calls, over-powered models, reprocessing, or idle. Drop a comment.

Your LLM Keeps Making the Same Extraction Mistake. Here's How to Make It Learn.

Avneet bansal — Sat, 03 Oct 2026 01:32:58 +0000

> TL;DR — I open-sourced an AWS sample that makes LLM document extraction learn from human corrections. Each QA fix is captured once and reused, so the pipeline gets more accurate and cheaper over time — with no retraining and no redeployment. Repo: github.com/aws-samples/sample-prompt-correction-memory. You can run the whole self-healing loop locally in 30 seconds, no AWS account needed.

The problem: corrections that evaporate

If you've built an LLM extraction pipeline, you know this pain.

Your QA analyst opens today's invoice. The model extracted payment_terms as "quarterly." That's wrong — "quarterly" is the billing frequency; the actual payment term is Net 30. The analyst corrects it. Done.

Tomorrow, an identical invoice from the same vendor arrives. The model extracts... "quarterly" again. Same mistake. The analyst corrects it again.

The correction evaporated. It fixed one document and taught the system nothing.

Multiply that across thousands of documents and dozens of recurring error patterns, and you get the two options most teams settle for:

Pay analysts to re-fix the same patterns forever (QA cost never goes down), or
Fine-tune a model (expensive, slow, needs ML ops, and you redo it every time the patterns shift).

There's a third way that needs neither.

The idea: treat corrections as durable memory

What if every human correction became a permanent, reusable asset?

That's the pattern behind sample-prompt-correction-memory. Instead of throwing corrections away, it stores them and feeds them back into future extractions — so the same mistake is never made twice.

It works as three tiers, and the magic is that traffic gradually shifts from the expensive tier to the free one:

🟢 Tier 1 — Graduated rules ($0, no LLM call)

When the same correction pattern recurs enough times, the system synthesizes a deterministic rule (a regex, a lookup, a normalization) that handles that field with zero LLM cost and sub-millisecond latency. "Net 30" → 30 becomes a rule; it never needs a model again.

🔵 Tier 2 — Cloud LLM extraction (confidence-gated)

Fields with no matching rule go to Amazon Bedrock (Claude Haiku 4.5). Every field comes back with a confidence score. If confidence clears the threshold, accept it and move on.

🟠 Tier 3 — Self-healing (only when needed)

If confidence is below threshold, the system retrieves the most relevant past corrections, formats them as few-shot examples, and re-extracts with a stronger model (Claude Sonnet 4.5). The model learns from the specific mistakes it made before — in-context, no training.

Every QA correction flows into a correction log (DynamoDB) that powers both the self-healing retrieval and the rule graduation. So the more the system is used, the smarter and cheaper it gets.

Why this matters (the practical payoff)

No ML ops. No retraining, no GPUs, no deployment cycles. A correction takes effect on the very next document.
Cost drops as quality rises. Normally you trade one for the other. Here, recurring fixes graduate to $0 rules — so accuracy climbs while per-document cost falls.
QA effort compounds instead of repeating. Fix it once → fixed forever.
It's a QA action, not an ML project. Improving the pipeline no longer requires the ML team. An analyst uploads a correction; the system improves.

The clearest fit is high-repetition, back-office document processing — accounts payable, contracts, compliance filings, supply-chain docs — where the same field errors recur and analyst time is the real expense.

The architecture

Fully serverless, deployed with AWS SAM:

Document ──► S3 ──► EventBridge ──► SQS ──► Lambda (extract)
                                              │
                          ┌───────────────────┼───────────────────┐
                          ▼                    ▼                   ▼
                   Tier 1: Rules       Tier 2: Bedrock      Tier 3: Self-heal
                   (deterministic)     (Claude Haiku 4.5)   (Claude Sonnet 4.5)
                          ▲                                        │
                          │                                        ▼
                          └──────── Correction Log (DynamoDB) ◄────┘
                                    (grows from QA feedback)

QA corrections are uploaded as JSON to an S3 corrections/ prefix; a second Lambda validates and ingests them into the correction log. Everything is encrypted with a customer-managed KMS key, and IAM is scoped to the specific Bedrock model ARNs the sample uses.

See it in 30 seconds (no AWS account)

The quickstart runs the entire self-healing loop with a mocked Bedrock client, so you can watch the mechanism without deploying anything:

git clone https://github.com/aws-samples/sample-prompt-correction-memory
cd sample-prompt-correction-memory
pip install -e ".[dev]"
python examples/quickstart.py

You'll see output like this:

Step 1: Initial extraction (no correction memory)
  Field:      effective_date
  Value:      March 2024        Confidence: 0.55   ❌ NO

Step 2: Self-healing triggers (0.55 < threshold 0.70)
  Retrieving corrections for 'effective_date'...
  Found: "March 2024" → "2024-03-01"

Step 3: Re-extraction with correction memory
  Value:       2024-03-01       Confidence: 0.95   ✓ YES
  Self-Healed: True

Low-confidence extraction → retrieve past correction → re-extract → correct answer. No retraining. No redeployment.

Deploying to AWS

With the AWS CLI, SAM CLI, and Bedrock model access for Claude Haiku 4.5 + Sonnet 4.5:

make deploy    # S3, DynamoDB, Lambda, EventBridge, SQS, KMS
make seed      # load sample corrections into the correction log
make trigger   # upload a sample document → triggers real extraction
make verify    # read back results (fields, confidence, self_healed)
make destroy   # empty buckets + delete the stack

make verify prints each extracted field with its confidence and whether self-healing kicked in — your proof it works end-to-end.

A companion: spatial memory

This sample provides semantic correction memory (what was extracted wrong, and why). Its companion, sample-textract-field-memory, provides spatial memory (where fields appear on a document layout). Together they form a dual memory for document pipelines: use the cheap spatial lookup when you're confident where a field is, and fall back to self-healing extraction when you're not.

Honest limitations

Because it's more useful when it's used well:

It's sample/reference code, not a production library. Harden IAM, add monitoring, and put a human in the loop for high-stakes fields before real use.
It helps most where the same field errors recur. For wildly heterogeneous, one-off documents, there's less pattern to learn.
Graduated rules currently take effect without a human approval step — in production you'd add a review gate before a synthesized rule starts answering on its own.
Don't use it with regulated data (PII/PHI/PCI) as-is; that path needs a proper compliance review.

Try it and tell me

The repo is open source under aws-samples, MIT-0 licensed, with a benchmark suite, an interactive dashboard, and an offline test harness.

👉 github.com/aws-samples/sample-prompt-correction-memory

What repetitive extraction error is your team still fixing by hand? I'd love to hear which patterns you'd want a system like this to learn.

This is an open-source AWS sample intended for demonstration and non-production use.