DEV Community: Avneet bansal

A Field Guide to Beating Amazon Bedrock Throttling

Avneet bansal — Tue, 20 Oct 2026 20:45:01 +0000

If you run real LLM traffic through Amazon Bedrock, throttling is not an edge case. It is the normal operating condition at scale. A ThrottlingException or TooManyRequestsException is Bedrock telling you that your request rate or token rate crossed a quota for that model in that region. The fix is rarely one setting. It is a layered defense: understand the quota, retry intelligently, cap concurrency, degrade gracefully, handle timeouts, and know what to escalate to AWS.

This is the set of strategies I run across production LLM workloads, written as a generic guide. No workload specifics, just the patterns and when each one earns its place.

TL;DR

Know your quota first: check Service Quotas, learn which limits are adjustable (soft) and which are not (hard).

Retry with exponential backoff and full jitter on throttling errors. This is the floor, not the ceiling.

Cap concurrency so you stop generating load the quota cannot absorb.

Add a circuit breaker so a throttle storm backs off instead of hammering.

Handle timeouts separately: shrink the batch for model timeouts, tune and retry for client read timeouts.

Use cross-region inference profiles to spread token demand across regions.

Reserve provisioned throughput only when on-demand plus the above is not enough.

Step one: know the quota you are hitting

Before writing a line of retry code, find the actual limit. Bedrock quotas are per model, per region, and split by type: tokens per minute (TPM), requests per minute (RPM), and for some models separate input/output token quotas.

Quotas for Amazon Bedrock — the overview, including that default quotas vary by account age, usage history, and region.
Quotas for the bedrock-runtime endpoint — the runtime invoke limits you hit at inference time. View them in the Service Quotas console under Amazon Bedrock.
Scaling and throughput best practices — how per-model TPM and RPM quotas interact.

Soft limits vs hard limits

This distinction decides your whole strategy.

Soft limits (adjustable). In the Amazon Bedrock endpoints and quotas table these are marked Adjustable: Yes. You can raise them. Request an increase through Service Quotas; smaller increases often auto-approve, larger ones route to AWS Support for review.
Hard limits (not adjustable). Marked Adjustable: No. No amount of asking moves these. Your only lever is architectural: reduce demand, spread it across regions, or change the access pattern.

The practical rule: if the limit is soft and you are consistently near it, file the increase and also build the defenses below, because the increase takes time and never makes throttling impossible. If the limit is hard, skip the request entirely and go straight to architecture.

One often-missed detail: Bedrock reserves TPM quota at the start of a request based on input tokens plus maxTokens. If you leave maxTokens unset, it can default to the model maximum and reserve far more quota than your call actually needs, so you throttle earlier than your real token use suggests. Set maxTokens to a realistic ceiling.

Layer one: retry with backoff and jitter

The baseline defense. Catch the throttling error codes (ThrottlingException, TooManyRequestsException), wait, and try again. Two details separate a good retry from a harmful one:

Exponential backoff. Each attempt waits longer: base * 2^attempt, capped at a ceiling. This gives the quota window time to refill.
Full jitter. Randomize the delay inside that window. Without jitter, every throttled caller retries at the same instant and you get a synchronized stampede that throttles again. With jitter, retries spread out.

A compact pattern:

import random, time
from botocore.exceptions import ClientError

THROTTLE = {"ThrottlingException", "TooManyRequestsException"}

def call_with_backoff(fn, *, base=2.0, cap=15.0, max_retries=5):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except ClientError as exc:
            code = exc.response.get("Error", {}).get("Code", "")
            if code not in THROTTLE or attempt == max_retries:
                raise
            delay = min(base * 2 ** attempt, cap)
            time.sleep(random.uniform(0, delay))  # full jitter

A note on the boto3 client's own retries. Boto3 has a built-in retry mode (standard or adaptive) you set via botocore.config.Config(retries=...). You can lean on it, or disable it (max_attempts=0) and own the retry loop yourself. Owning it is worth the effort when you want the backoff, jitter, and error classification to match the rest of your system rather than two retry layers fighting each other.

The same retry in a Step Functions workflow

If your Bedrock calls run inside a Step Functions state machine, you do not write the backoff loop by hand. You declare it on the task's Retry block and the service applies exponential backoff with jitter for you. Step Functions error handling added a JitterStrategy field so wait times are spread instead of synchronized, the same reason full jitter matters in code.

"InvokeBedrock": {
  "Type": "Task",
  "Resource": "arn:aws:states:::bedrock:invokeModel",
  "Retry": [
    {
      "ErrorEquals": ["ThrottlingException", "TooManyRequestsException"],
      "IntervalSeconds": 2,
      "MaxAttempts": 15,
      "BackoffRate": 2.0,
      "MaxDelaySeconds": 30,
      "JitterStrategy": "FULL"
    }
  ],
  "Catch": [
    { "ErrorEquals": ["States.ALL"], "Next": "HumanReview" }
  ],
  "Next": "NextStep"
}

The fields map onto the code version: IntervalSeconds is the base delay, BackoffRate is the exponential multiplier, MaxDelaySeconds is the cap, MaxAttempts is the retry count, and JitterStrategy: FULL randomizes each wait. The Catch routes to a fallback once retries are exhausted, for example a human-review step, so an exhausted retry degrades gracefully instead of failing the whole execution.

Layer two: cap concurrency

Retries handle the request that already failed. Concurrency caps stop you from generating more load than the quota can absorb in the first place. If 50 workers all call the same model at once, you will throttle no matter how clever the backoff is.

Put a ceiling on in-flight Bedrock calls. In async Python a semaphore does it:

import asyncio

semaphore = asyncio.Semaphore(10)  # max concurrent Bedrock calls

async def bounded_call(fn):
    async with semaphore:
        return await fn()

Upstream of the application, cap concurrency at the infrastructure layer too. If documents arrive as events, put them on an SQS queue and let the SQS-to-Lambda event source mapping enforce a maximum_concurrency. Work waits in the queue until a slot frees, with no polling and no custom counters:

resource "aws_lambda_event_source_mapping" "sqs_to_worker" {
  event_source_arn = aws_sqs_queue.work.arn
  function_name    = aws_lambda_function.worker.arn
  batch_size       = 1
  scaling_config {
    maximum_concurrency = 5
  }
}

Match the concurrency ceiling to the quota, not to how many workers you could theoretically run. The right number is the one that keeps steady-state token rate under the TPM limit.

Layer three: a circuit breaker for throttle storms

Backoff and concurrency caps handle normal pressure. A sustained throttle storm needs something that steps back harder. A circuit breaker watches the throttle rate and, once it crosses a threshold inside a time window, pauses all calls for a cooldown, then resumes at reduced concurrency before restoring full throughput.

The state machine:

Closed (normal): calls flow, throttles are counted.
Open (tripped): after N throttles in the window, pause all calls for a cooldown and halve the concurrency limit.
Half-open (recovering): after the cooldown, resume at the reduced limit; if clean for a while, restore full concurrency; if it throttles again, reopen.

This converts a feedback loop that makes throttling worse (retry, throttle, retry, throttle) into one that actively sheds load until Bedrock recovers. It pairs with the concurrency limiter: the breaker is what turns the limit down and back up.

Layer four: handle timeouts, they are not throttling

Throttling rarely travels alone, and the loudest neighbor is the timeout. There are two distinct kinds, and they want opposite fixes.

Model timeout: shrink the work

A ModelTimeoutException means the request itself was too large to finish in the model's time budget. It is not a rate problem, so retrying the same payload will time it out again. The fix is to reduce the work: halve the batch or chunk the input, then retry smaller.

async def call_shrinking(fn, items, *, max_retries=3):
    batch = list(items)
    for attempt in range(max_retries + 1):
        try:
            return await fn(batch)
        except Exception as exc:
            code = getattr(exc, "response", {}).get("Error", {}).get("Code", "")
            if code != "ModelTimeoutException" or attempt == max_retries:
                raise
            batch = batch[: max(1, len(batch) // 2)]  # halve and retry

Client timeout: tune the budget, then retry the transient ones

A client-side ReadTimeoutError or ConnectTimeoutError is different again: the SDK gave up waiting before the model answered. Large prompts and long generations routinely exceed the boto3 default read timeout, which produces timeouts that look like failures but are really an impatient client.

Two moves:

Set a realistic read_timeout on the boto3 client so a slow-but-valid response is not cut off.
Treat the transient connection timeouts as retryable, alongside throttling, rather than letting them bubble up as hard failures.

from botocore.config import Config

client = boto3.client(
    "bedrock-runtime",
    config=Config(read_timeout=300, retries={"max_attempts": 0}),
)

RETRYABLE = {"ThrottlingException", "TooManyRequestsException"}
# also retry ReadTimeoutError / ConnectTimeoutError by exception class name

Classify before reacting. A throttle wants backoff, a model timeout wants a smaller batch, a client timeout wants a longer budget plus a retry, and a ValidationException wants no retry at all.

Layer five: spread demand with cross-region inference profiles

Quotas are per region. A cross-region inference profile lets a single on-demand call be served from one of several regions, so your token demand is spread across each region's quota instead of piling onto one. For on-demand workloads this is often the highest-leverage change: no capacity to buy, and the effective ceiling goes up because you are drawing from multiple regional pools.

It is not infinite headroom, and it does not change a hard per-model limit in a single region, but for spiky on-demand traffic it buys real room.

Layer six: provisioned throughput, only when you need it

If on-demand plus everything above still throttles under predictable heavy load, Bedrock offers Provisioned Throughput: reserved capacity (model units) with a guaranteed token rate, bought by commitment. It removes throttling for that reserved slice because the capacity is yours.

Treat it as the last layer, not the first. It costs whether or not you use it, so it only makes sense for steady, predictable volume. For bursty or unpredictable traffic, retries, concurrency caps, and cross-region profiles on on-demand are usually the better economics.

Tuning in practice

The layers above are the toolbox. How hard you turn each knob depends on the workload, and a couple of patterns are worth stating plainly.

Match the aggressiveness of retries to the cost of a lost call. Heavy, expensive calls justify more retry attempts, a higher backoff ceiling, and tighter concurrency, because losing one is costly. High-volume, cheaper work is better served by an upstream queue that caps concurrency and a workflow-level self-retry that ends in a manual-review fallback, keeping the per-call retries light. There is no single correct number; the right setting is the one that keeps steady-state token rate under the quota for that workload.

Two caveats that are easy to get wrong:

A "fallback model" can quietly resolve to the same model. If both the primary and fallback point at the same model id through shared config, the fallback functions as extra retries against one model, not as model diversity. Calling it a fallback overstates what it does until the config points at a genuinely different model.
Retry, backoff, and concurrency are not the same as reserved capacity. A defense built entirely from retries, backoff, concurrency caps, and on-demand cross-region inference profiles has no reserved-capacity floor under a true spike. That can be a deliberate on-demand posture, but name it as one, because on-demand plus retries still has a ceiling.

How to decide what to reach for

A short decision order when you see throttling:

Check the quota. Is the limit soft or hard? If soft and you are near it, file the increase now, in parallel with the rest.
Confirm retries have backoff and jitter. If not, fix that first. It is the cheapest win.
Cap concurrency to keep steady-state rate under the TPM limit, at both the app and the infrastructure layer.
Add a circuit breaker if throttle storms are recurring rather than occasional.
Separate timeouts from throttles. Shrink the batch for model timeouts, tune the read timeout and retry for client timeouts.
Spread across regions with a cross-region inference profile for on-demand traffic.
Buy provisioned throughput only for steady, predictable, high volume the above cannot cover.

Throttling is not a wall you hit once and clear. It is a pressure you manage continuously. The systems that stay up are the ones that treat the quota as a first-class design constraint, not an exception to catch.

I Moved My Prompts Into DynamoDB and the Throttling Went Away

Avneet bansal — Sun, 18 Oct 2026 23:03:50 +0000

Prompts feel like configuration, so it is tempting to park them in whatever managed prompt-management service your platform hands you. You get versioning, a console, and a clean API, and for a while that is the right call. The trouble shows up later, under load, when prompt retrieval sits on the hot path of every inference call and the service starts pushing back.

That is the wall I hit. The prompt-management layer was throttling requests. Because every call had to fetch its prompt before doing any real work, that throttle became a ceiling on the whole pipeline. The model was not the bottleneck. Prompt lookup was.

I moved prompt serving into DynamoDB and kept the managed service as the authoring source of truth. The throttling went away, and the design has been running well since.

Why a prompt-management service becomes the bottleneck

Prompt-management services are built for authoring and governance, not for being read thousands of times a minute on a latency-sensitive path. They come with request quotas, and those quotas are usually shared across the account rather than scaled to one workload's read traffic. When read volume climbs, you hit the quota long before you hit any limit that has to do with the prompts themselves.

The tell is subtle at first. You start seeing throttling and too-many-requests errors on prompt retrieval, so you add retries with exponential backoff and maybe a fallback for when the service says no. That is reasonable hygiene, but once you are wrapping a plain config lookup in retry logic, the store has quietly become the throughput limit of the application. You can optimize the model, batch the work, and tune everything downstream, and none of it matters, because the request is already stalled waiting for its prompt.

Why DynamoDB fits this access pattern

A prompt lookup is about the simplest access pattern there is. You have a key, and you want the item back. That is a single-item read by primary key, which is exactly what DynamoDB is good at.

Read throughput scales with the table, not a fixed service quota. On-demand capacity absorbs spikes and provisioned capacity with auto scaling handles steady load. The shared quota that was throttling me no longer sat on the path.
The access pattern is a key lookup. Partition on the prompt identifier and every fetch is a single GetItem. No scans, no filtering, predictable single-digit-millisecond reads.
Reads and writes separate cleanly. High-volume reads hit DynamoDB. The rare authoring writes stay in the managed service where review and governance already live, and a sync step pushes changes down into DynamoDB.

The part that makes it safe: checksum-based sync

Caching prompts is easy. Keeping the cache honest is the part worth getting right.

The managed service stays the source of truth for prompt content. DynamoDB holds the serving copy that the hot path reads. To keep them in agreement without re-uploading everything on every deploy, I sync on a checksum.

The sync is write-through. When it detects a change at the source, it writes the new prompt into DynamoDB first, and readers pick it up from there afterward. The flow is simple:

For each prompt in the source of truth, compute a checksum of its content.
Compare that checksum against what is already stored for that prompt.
If they match, do nothing. If they differ, the prompt changed, so write the new content into DynamoDB and record the new checksum.

This turns every sync into a cheap compare instead of a blind rewrite. Unchanged prompts cost a comparison and no write. Only a genuine edit at the source triggers a write. Because the write lands before any read needs it, a new prompt propagates cleanly: the serving copy is already updated by the time the hot path asks for it.

The nice property is that the checksum is the change signal. You do not need a separate "is this dirty" flag or a manual version bump. If the content is identical, the checksum is identical, and the write is skipped. If someone edits a prompt, the checksum moves and the new text is written on the next sync.

If you need rollback or audit: versioning in DynamoDB

DynamoDB does not hand you versioning the way a prompt-management service does, but you can model it, and it is worth saying so plainly because "DynamoDB has no versioning" is a common misread.

If a single current copy per prompt is all you need, the checksum layout above is enough. If you want history, rollback, or an audit trail, add a version to the key:

Partition key: the prompt identifier.
Sort key: the version. A monotonic integer, an ISO timestamp, or the checksum itself all work.
A current pointer: one extra item per prompt (for example sort key CURRENT) that names the active version.

The hot path reads the current pointer by key, so serving stays a single keyed read. History stays queryable with a Query on the partition key. The checksum still decides when a new version row gets written, so you are not stacking a new version for an unchanged prompt. If concurrent syncs are possible, a conditional write (ConditionExpression on the stored checksum) keeps a stale sync from overwriting a newer one.

The tradeoff is simple: the single-copy layout is less to manage, the versioned layout buys rollback and audit. Both are key reads on the hot path, so neither reintroduces the throttling problem.

The shape of the change

The migration itself was small, which is part of why it was the right call.

Model the table around the lookup. Partition key on the prompt identifier. That is the whole schema, because the access pattern was known up front.
Read by key on the hot path. Each inference call does a single keyed read. No scan runs against this table in production, so cost stays proportional to the reads you actually make.
Sync by checksum, not on a timer or on every deploy. The source of truth drives updates, and only real changes cross into DynamoDB.

The result is that prompt retrieval stopped being a shared, rate-limited dependency and became a key read against a store that scales with the traffic hitting it.

What I would tell someone before they copy this

This is not an argument against prompt-management services. They earn their place for authoring, review, and governance. The mistake is letting that same service sit on a high-volume read path it was never sized for.

A few things worth keeping in mind:

The symptom is a throughput ceiling that tracks request volume, not model latency. If you are wrapping prompt retrieval in throttling retries, that is the tell. No downstream tuning will move it.
Design the table around the access pattern first. A key lookup wants a key schema. Pick the partition key from the query you actually run, and keep scans out of production.
Make the cache self-correcting. A checksum compare against the source of truth means the write-through sync updates the serving copy only when content actually changes, without manual version juggling and without rewriting unchanged data.
Model versioning if you need it, do not assume it is missing. A sort-key version plus a current pointer gives you rollback and audit while keeping the hot path a single keyed read.

Moving prompts into DynamoDB was a small change with a disproportionate payoff. The throttling that was capping the pipeline is gone, retrieval is a clean key lookup, and the checksum sync keeps the serving copy honest against the source of truth. It has held up in production since.

The GitOps Bridge: Keep Terraform and ArgoCD Out of Each Other's Way

Avneet bansal — Mon, 12 Oct 2026 04:18:15 +0000

Two tools, two clocks. Terraform provisions the cluster, the node groups, the networking, the IAM. It changes rarely and under heavy approval. ArgoCD deploys the applications onto that cluster. It changes many times a day under light approval. The trouble starts when you let them reach into each other, because then a routine app deploy can be blocked behind an infrastructure change, and an infrastructure change can take applications down with it. The fix is not a better tool. It is a thin contract between the two, so each stays in its own lane. I used this to compress a three-environment EKS migration from a planned six months to about three. Here is the contract, and what it actually bought.

TL;DR

Infrastructure (Terraform) and applications (ArgoCD/Flux) have different change frequencies, approval flows, and blast radii. Coupling them is the root problem.

The GitOps Bridge is a narrow handoff: Terraform writes cluster facts into a well-defined interface (ConfigMap/Secret labels and annotations), and GitOps controllers read that interface to deploy.

Neither side imports the other's state. The interface is the only shared surface.

On a real migration this let three environments come up reproducibly, with the customer executing the IaC themselves under a zero-access constraint, and cut the timeline roughly in half.

The coupling you do not notice until it hurts

The common anti-pattern is to let Terraform deploy applications (via the Helm or kubectl provider) or to let application manifests carry infrastructure facts hard-coded inside them. Both feel convenient. Both create coupling that only shows its cost later.

When Terraform deploys apps, every application change rides the Terraform lifecycle: a terraform apply that should only touch a node group now also reconciles your workloads, and a drift in one app can fail a plan that was supposed to be about infrastructure. When app manifests hard-code infrastructure facts (the cluster's OIDC issuer, a role ARN, a VPC ID, the region), those manifests are no longer portable across environments, and every environment promotion becomes a hand-edit. The blast radius of each change grows to cover both layers, and the audit trail blurs, because you can no longer tell from the change history whether a given commit was an infrastructure decision or an application one.

The two layers genuinely have different natures. Infrastructure changes are infrequent, high-risk, and want slow, careful approval. Application changes are frequent, lower-risk, and want to flow. Forcing them through one pipeline makes the fast side as slow as the slow side, and the slow side as risky as the fast side.

The contract

The bridge is deliberately small. Terraform's job ends at provisioning the cluster and then publishing a set of facts about it into a known location. In practice that location is a ConfigMap and a Secret in a known namespace, carrying labels and annotations that describe the cluster: its name, region, the OIDC provider URL, role ARNs for workloads, networking identifiers, and any environment-specific values the apps will need. Terraform owns writing these. It never installs an application.

ArgoCD's job begins by reading that interface. A root application points at the known ConfigMap/Secret, pulls the cluster facts, and uses them to parameterize everything it deploys. The app manifests themselves carry no hard-coded infrastructure values. They carry references into the bridge. ArgoCD owns reading these. It never provisions infrastructure.

The important property is what is absent. Terraform does not read ArgoCD's state, and ArgoCD does not read Terraform's state. There is no shared state file, no circular dependency where each waits on the other. The only thing they share is the interface, and the interface is data, not control. Terraform writes facts; ArgoCD reads facts. That one-directional, data-only handoff is the whole design, and it is what lets each side keep its own clock.

Why this made a hard migration tractable

The migration that proved it out had a constraint that is more common than people admit: I could not directly touch the target environments. The work had to be delivered as infrastructure-as-code that the customer ran themselves, across three environments, an internal lab, a sandbox, and production, with no interactive access to any of them.

That constraint is exactly where the bridge pays off. Because the apps consume cluster facts through the interface rather than embedding them, the same application definitions deploy unchanged to all three environments. Each environment's Terraform writes its own values into its own bridge, and the GitOps layer adapts automatically. There was no per-environment hand-editing of manifests, which is where this kind of migration usually loses weeks and acquires drift. The customer could run the IaC in each environment and get a reproducible result, because the only thing that varied between environments lived in the bridge, generated by that environment's own provisioning.

For one customer, the combination (Terraform plus the bridge plus GitOps, with Karpenter handling node provisioning on the cluster side) took a migration planned for about six months down to roughly three. The halving did not come from any single tool being fast. It came from removing the coupling that would otherwise have forced serial, hand-held, per-environment work. When each environment comes up the same way from the same definitions, you stop paying the integration tax three times.

How to adopt it

Start by drawing the line. List every value your application manifests currently hard-code that is really an infrastructure fact: cluster name, region, OIDC issuer, IAM role ARNs, subnet or security-group IDs, DNS zone names. Those are the contents of your bridge.

Have Terraform write them. After it provisions the cluster, have it create a ConfigMap (for non-secret facts) and a Secret (for anything sensitive) in a known namespace, populated from its outputs. Decide the key names once and treat them as the contract; renaming a bridge key is now a breaking change, which is the right level of ceremony for it.

Have GitOps read them. Point a root ArgoCD (or Flux) application at that namespace, and parameterize your app set from the bridge values rather than from literals. Each environment gets its own bridge from its own Terraform, and the same app definitions follow.

A minimal, generic version of both sides looks like this. First, Terraform publishes the facts after it provisions the cluster (the values are illustrative placeholders, not from any real environment):

# Terraform: write the bridge after the cluster exists.
# The labels are what ArgoCD's ApplicationSet selects on; the data is the facts.
resource "kubernetes_config_map" "cluster_bridge" {
  metadata {
    name      = "cluster-bridge"
    namespace = "argocd"
    labels = {
      "bridge.example.com/environment" = var.environment   # dev | staging | prod
    }
  }

  data = {
    cluster_name   = module.eks.cluster_name
    region         = var.region
    oidc_issuer    = module.eks.oidc_provider          # for IRSA
    account_id     = data.aws_caller_identity.current.account_id
    # a workload's IAM role ARN, resolved per environment:
    app_irsa_role  = module.app_irsa.iam_role_arn
  }
}

Then an ArgoCD ApplicationSet reads that ConfigMap and templates every app from it, so the app definitions themselves carry references, never literals:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-addons
  namespace: argocd
spec:
  generators:
    - clusters:                      # or a configMap/plugin generator reading the bridge
        selector:
          matchLabels:
            bridge.example.com/environment: prod
  template:
    metadata:
      name: '{{name}}-addons'
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/platform-addons
        targetRevision: main
        path: addons
        helm:
          # bridge facts flow in as values; no hard-coded ARNs or issuers in the chart
          parameters:
            - name: region
              value: '{{metadata.annotations.region}}'
            - name: oidcIssuer
              value: '{{metadata.annotations.oidc_issuer}}'
            - name: appIrsaRole
              value: '{{metadata.annotations.app_irsa_role}}'
      destination:
        server: '{{server}}'
        namespace: kube-system

The exact generator you use (cluster, configMap, or a plugin generator) depends on your setup; the invariant is the one that matters: Terraform owns the left side and writes facts, ArgoCD owns the right side and reads them, and the app charts reference bridge values instead of embedding them. Swap prod for dev or staging and nothing in the app definitions changes, only the bridge each environment's Terraform wrote.

Hold the boundary. The one discipline that keeps this working is refusing to cross the line later under deadline pressure: no Helm-provider app installs sneaking into Terraform, no infrastructure ARNs sneaking back into manifests. The value is entirely in the separation, so the separation is the thing to protect.

The reward is that your fast layer stays fast and your slow layer stays safe. An application deploy no longer waits on an infrastructure plan. An infrastructure change no longer reconciles your workloads as a side effect. And a new environment is a new bridge, not a new round of manual edits. Keep the two clocks separate, and let a thin, one-way, data-only contract be the only thing between them.

If you run Terraform and a GitOps controller together, it is worth asking which infrastructure facts your app manifests still hard-code. Those are the ones the bridge is for.

Your Readiness Probe Is Lying to You

Avneet bansal — Sat, 10 Oct 2026 21:14:36 +0000

A readiness probe is a promise. It tells Kubernetes "route traffic to me, I can handle it." Most of the time the promise is kept because the probe checks the thing that actually matters. The failures worth writing about are the ones where the probe keeps saying Ready while the component has quietly lost the one connection it needs to do any useful work. The traffic keeps arriving. None of it can be served. Nothing looks broken on the dashboard.

I hit exactly this in KEDA, the CNCF event-driven autoscaler, and the fix merged as kedacore/keda #8226. The bug is a good teaching case on its own. The part I did not expect was the second bug hiding inside the fix, which a maintainer caught in review.

TL;DR

KEDA's Metrics Server reported Ready even when its gRPC connection to the keda-operator was down.

In that state the HPA silently stops receiving scale signals, while the replica still looks healthy.

The fix makes the readiness probe observe the real gRPC connection state and report NotReady when the connection is unusable.

A naive version of that fix has a trap: once a replica goes NotReady it stops receiving RPCs, so an Idle connection never gets woken and the replica can stay NotReady forever.

The merged fix nudges an Idle connection to reconnect from inside the readiness check itself.

The setup: who talks to whom

KEDA has two moving parts here. The keda-operator does the actual work of talking to scalers and computing metrics. The Metrics Server (a custom metrics API server) is what the Kubernetes HPA queries when it wants to know "how many of these should be running." The Metrics Server does not compute metrics itself. It forwards the request to the operator over a gRPC connection and relays the answer.

That gRPC connection is the Metrics Server's entire reason to exist. If it is down, the Metrics Server is a shell. It can accept an HPA query and has nothing to answer it with.

The lie

The Metrics Server's readiness probe did not check that connection. It reported Ready as long as the process was up and its HTTP server was listening. So when the gRPC connection to the operator dropped, the following happened, all at once and all invisible: the replica stayed Ready, Kubernetes kept it in the APIService endpoints, the HPA kept sending it metric queries, and every one of those queries failed or hung because there was no live operator connection behind them. The autoscaler stops scaling, and the one signal you would look at, pod readiness, is green.

This is the specific shape of outage that is worst to debug. A crash is loud. A readiness probe that passes while the component cannot serve is silent, and it points you away from the real cause, because the first thing anyone checks is "are the pods ready," and they are.

The fix, version one

The fix is conceptually simple. Make readiness reflect the dependency. gRPC already tracks connection state through grpc.ClientConn.GetState(), which returns one of Idle, Connecting, Ready, TransientFailure, or Shutdown. Crucially, GetState() is a pure observer. It reports the current state and never initiates a connection, so it is safe to call from a probe with no side effects.

So version one added a small accessor on the metrics-service client that returned the connection state, and wired a readyz check in the adapter that reported NotReady unless the connection was Ready. When the operator connection drops, the probe fails, Kubernetes pulls the replica from the APIService endpoints, and the HPA stops sending it queries it cannot answer. When the connection recovers, the probe passes again. That is the behavior you want.

The second bug, caught in review

Here is the part I like. A maintainer looked at version one and pointed out a failure mode that only exists because the fix works.

Think through what happens after the probe correctly reports NotReady. The replica gets removed from the APIService endpoints. Now no RPCs flow through that client at all, because nothing is routed to it. And gRPC, by default, parks a connection in the Idle state after a period with no RPCs (GRPC_CLIENT_IDLE_TIMEOUT_MS, 30 minutes by default). An idle gRPC connection does not reconnect on its own. It waits for someone to use it.

So the naive fix can deadlock the recovery. The connection goes bad, the probe reports NotReady, the replica stops getting RPCs, the connection drifts to Idle, and nothing ever wakes it, because the only thing that would wake it is an RPC, and RPCs only resume once the probe reports Ready again, which requires the connection to be awake. The replica can sit NotReady permanently, needing a manual restart to recover. The fix for a silent hang would have introduced a silent hang.

The resolution is to make the readiness check active, not passive, in exactly one narrow case. When the accessor observes an Idle connection, it triggers a non-blocking reconnect before returning:

// GetConnectionState returns the current connectivity state of the underlying
// gRPC connection to the KEDA metrics service.
//
// When the connection is Idle it also triggers a non-blocking reconnection
// attempt (moving it towards Connecting) before returning. Once a non-Ready
// state removes the replica from the APIService endpoints, no RPCs flow through
// this client, so nothing else would ever wake an Idle connection.
func (c *GrpcClient) GetConnectionState() connectivity.State {
    state := c.connection.GetState()
    if state == connectivity.Idle {
        c.connection.Connect()
    }
    return state
}

Connect() is non-blocking and a no-op when the connection is not Idle, so it is safe to call from a probe. The probe still returns the pre-nudge state (so a just-nudged Idle connection reports NotReady that cycle), but the connection is now moving toward Connecting and then Ready on its own. The replica recovers without a human. The probe went from a pure observer to an observer-with-one-side-effect, and that single side effect is what closes the recovery loop.

Reproduce it yourself

The regression test does not need a cluster or a running operator. It exercises the connection-state logic directly with an in-process gRPC client, so you can run it from a clean checkout:

git clone --depth 20 https://github.com/kedacore/keda.git
cd keda
go test ./pkg/metricsservice/... -run 'TestGetConnectionState' -v

Two tests cover the two behaviors above. One asserts that a freshly created client (gRPC does not dial until first use) is not Ready, so a probe built on it correctly reports NotReady for a replica that has not connected yet. The other asserts the review fix: calling the accessor on an Idle connection moves it off Idle within a couple of seconds without any RPC being issued, so a sidelined replica can recover on its own.

Expected: both subtests pass. (If you are on a network that cannot reach the Go module proxy, the clone's large dependency tree may need go clean -modcache or a working GOPROXY first.)

How to adopt the pattern

The KEDA-specific takeaway is small: upgrade to a release that includes #8226 and your Metrics Server will stop advertising readiness while blind to the operator.

The portable takeaway is the one worth internalizing. A readiness probe should check the dependency that the component cannot work without, not just that the process is up. For a forwarding service, that dependency is the downstream connection. And when you make a probe gate traffic on a connection's health, check whether removing traffic from an unhealthy replica also removes the thing that would let it recover. If recovery depends on activity, and your probe stops activity, you need the probe itself to drive recovery, carefully and without blocking. Check the real dependency, and make sure the healthy path back exists.

A probe that only ever reports the process is alive is answering a question nobody asked. The question Kubernetes is actually asking is "can you serve." Answer that one.

If you run forwarding or adapter services behind readiness probes, it is worth auditing what they actually check. The gap between "process is up" and "can serve" is where the quiet outages live.

A Prompt-Injection Firewall for AWS: What I'd Build, and What I Measured

Avneet bansal — Sat, 10 Oct 2026 20:10:28 +0000

Prompt injection has sat at the top of the OWASP Top 10 for LLM Applications since the list began in 2023, and it is still number one in the 2025 and 2026 editions. That is unusual. Most vulnerability classes get a real fix and slide down the list. SQL injection had parameterized queries. XSS had output encoding and CSP. Prompt injection has no equivalent, because the flaw is architectural: an LLM reads the system prompt, the user's input, and any retrieved documents or tool output as one undifferentiated stream of text, with no built-in boundary between trusted instructions and untrusted data.

I work on LLM systems on AWS, and this is the problem I keep coming back to. This post is not a product announcement. It is the design I would build if I were building a prompt-injection firewall today, grounded in what the research actually says and an honest account of what already exists. I will be clear throughout about which parts are established fact and which are my proposed approach.

TL;DR

Prompt injection is OWASP's #1 LLM risk across every edition, and a 2026 proof shows pure input-wrapper defenses face a hard trilemma: they cannot be continuous, utility-preserving, and complete at the same time.

The space is no longer empty. NeuralTrust ships a commercial "Generative Application Firewall," Check Point acquired Lakera, Cloudflare has Firewall for AI, and open-source projects exist. The honest gap is a cloud-native, AWS-reference, agent-aware design, not "nobody has done this."

My proposal: a multi-layer ensemble with a weighted decision engine, output and tool-call enforcement, and a feedback loop, built the trilemma into the design rather than pretending to beat it.

I built the offline core (heuristic + vector detectors + decision engine) and benchmarked it on 285 labeled prompts. The measured trade-off is stark: at the threshold that catches 92% of attacks, 25% of legitimate prompts get flagged; tighten it until false positives are near zero and detection falls to 39%. No setting gives both. That is the trilemma, measured, and the argument for the semantic layer.

Why this problem resists a clean fix

A 2026 paper, "Why Prompt Injection Defense Wrappers Fail?", formalizes something practitioners had felt for a while. It argues that a defense that wraps the input faces a trilemma of three properties that cannot all hold at once:

Continuity — a small change to the input should not flip the defense decision.
Utility preservation — legitimate prompts must still get through and work.
Completeness — every injection attempt must be caught.

You can get two. Not three. The paper is explicit that this bounds input-wrapper style defenses specifically, and that it does not rule out training-time alignment, architectural changes, or defenses that deliberately trade away some utility. That nuance matters, and I will come back to it, because it is the single most important design constraint.

The practical consequence is that any honest firewall should stop promising completeness. It should instead be explicit about where on the continuity/utility/completeness surface it chooses to sit, and make that choice configurable.

What already exists (the part the hype usually skips)

When I first sketched this idea, my note to myself said "no one has built this." That was wrong, and it is worth correcting in public, because anyone who works in LLM security will know the landscape.

Lakera Guard — commercial, API-first prompt-injection and jailbreak detection. Check Point acquired Lakera in September 2025 and folded it into its platform. Its headline accuracy and sub-50ms latency are the vendor's own figures, not something I measured.
NeuralTrust — ships a commercial product it literally calls a Generative Application Firewall, an inline enforcement layer for LLM apps and agents. So the "GAF" concept is not paper-only; it is a shipping product.
Cloudflare Firewall for AI — a commercial layer that blocks unsafe prompts and addresses several OWASP LLM risks at the edge.
Open-source — several GitHub projects named prompt-shield (and the one-word promptshield) implement detector ensembles, reverse-proxy enforcement, and threat scoring. LLM Guard is an active rule-based scanner set, and NeMo Guardrails (NVIDIA) handles conversation-flow control. Rebuff, an earlier multi-layer project, is archived.

The "GAF" framing itself comes from a 2026 paper, "Introducing the Generative Application Firewall (GAF)", which proposes a single enforcement point that unifies prompt filters, output validators, and data masking, maintains session context, and covers agents and their tool calls, by analogy to a WAF for web traffic.

So the honest gap is narrower and more specific than "nobody has built a firewall." It is: an open, cloud-native reference architecture on AWS primitives that is agent-aware (intercepts tool calls, not just prompts), uses a multi-detector ensemble rather than a single technique, and is explicit about the trilemma trade-off it makes. That is a real and useful gap. It is not a claim of inventing the category.

The architecture I'd build

The design is a pipeline: detect, decide, enforce, learn. Each stage maps cleanly onto AWS primitives, which is the point, it should be deployable without inventing new infrastructure.

          ┌───────────────────────────────────────────────┐
 request  │  1. DETECTION  (ensemble, run in parallel)      │
 ───────► │   heuristic · ML classifier · LLM-as-judge      │
          │   vector similarity to known attacks            │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  2. DECISION  (weighted vote + session context) │
          │   configurable threshold → the trilemma dial    │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  3. ENFORCE   allow / block / sanitize / review │
          │   also: scan the OUTPUT, gate TOOL CALLS         │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  4. LEARN   log verdicts, feed back FPs,         │
          │   grow the attack-vector store                   │
          └───────────────────────────────────────────────┘

On AWS, I'd map it like this: API Gateway as the entry point; the detector ensemble as Lambda functions, with the ML classifier on SageMaker and the LLM-as-judge on Bedrock; the attack-vector store on OpenSearch; the decision engine as a small Step Functions or Lambda orchestration reading session history from DynamoDB; and CloudWatch plus S3 for the audit trail. None of that is exotic, which is deliberate.

Why an ensemble, and what each layer is actually good at

No single detector is enough, and the reasons are well documented. The CAITLYN paper puts the trade-off bluntly: deterministic signature and regex filters run at sub-millisecond speed with zero token cost but fall to trivial obfuscation and semantic rewrites. So the ensemble is not padding; each layer covers a different failure mode of the others.

A heuristic layer is the cheap first pass. Illustrative, not a complete ruleset:

SUSPICIOUS = [
    r"ignore (all )?previous instructions",
    r"disregard the (system|above) prompt",
    r"you are now [a-z ]+",          # role-override attempts
    r"reveal (your )?(system )?prompt",
]

def heuristic_score(text: str) -> float:
    hits = sum(bool(re.search(p, text, re.I)) for p in SUSPICIOUS)
    return min(hits / 2, 1.0)   # saturate; this is a signal, not a verdict

That catches the lazy attacks for free and catches nothing clever, which is exactly why it only contributes a signal. A semantic layer (an ML classifier, or an LLM-as-judge on Bedrock) catches the paraphrased and indirect attacks the regex misses, at real token cost and latency. A vector-similarity layer compares the input against an embedding store of known attack patterns, so a variant of something you have seen scores high even if the wording is new.

The decision engine is where the trilemma becomes a dial

This is the heart of the honest version. Instead of a single detector shouting "block," the engine combines the signals into one score and compares it to a threshold you set per tenant or per use case:

WEIGHTS = {"heuristic": 0.15, "classifier": 0.35, "judge": 0.35, "vector": 0.15}

def decide(signals: dict[str, float], threshold: float) -> str:
    score = sum(WEIGHTS[k] * v for k, v in signals.items())
    if score >= threshold:            return "block"
    if score >= threshold * 0.6:      return "flag_for_review"
    return "allow"

The threshold is the trilemma choice, made explicit. A bank's internal agent can run it high and tolerate more false positives, because a missed injection is expensive. A low-stakes public chatbot runs it lower to protect utility. The firewall does not pretend to be complete; it hands you the knob and logs where you set it. That framing is the design taking the arXiv result seriously instead of marketing past it.

The weights above show the full four-detector design. The benchmark in the next section only exercises the two detectors I actually built offline (heuristic and vector), so it runs with two weights rather than four. The semantic detectors in this snippet are the layer I have not built yet.

I built the offline core and measured it

Design claims are cheap, so I implemented the two offline layers (the heuristic detector and the vector-similarity detector) plus the decision engine, and ran them against a labeled corpus of 285 prompts: 192 injection and 93 benign. The injection set is deliberately held out, paraphrases, obfuscations (spaced-out and hyphenated triggers), and indirect-via-document payloads that are not the examples the vector detector was fitted on, so the catch rate reflects generalisation rather than memorisation. The benign set includes hard negatives that mention "instructions," "system prompt," "override," and "ignore" innocently. Everything is pure standard-library Python with a fixed seed, so the run is deterministic and reproducible.

Here is the actual output, sweeping the decision threshold (weights: heuristic 0.4, vector 0.6):

 thresh  detect%    FP%  precision     F1
--------------------------------------------
   0.15      98%    52%       0.80   0.88
   0.18      92%    25%       0.88   0.90
   0.20      84%    14%       0.93   0.88
   0.25      60%     8%       0.94   0.74
   0.30      39%     1%       0.99   0.55
   0.40      10%     0%       1.00   0.18
   0.60       5%     0%       1.00   0.09

Read that top to bottom and the trilemma stops being an abstraction. At a threshold of 0.18 the firewall catches 92% of the injections, but it also flags a quarter of legitimate prompts, the hard benign negatives about writing system prompts and documenting team instructions get swept up. Tighten the threshold to 0.30 and false positives drop to 1% (precision 0.99), but detection collapses to 39%: most attacks walk straight through. There is no row where detection is high and false positives are low at the same time. The best F1 (0.90) sits near the aggressive end, where you have already accepted a real false-positive cost.

That is not a tuning failure I can polish away with better weights. It is the shape the 2604.06436 result predicts for this class of defense, and watching it fall out of real numbers is more convincing than any single headline accuracy figure. The honest reading is specific: a cheap offline ensemble reliably catches the obvious and the lexically similar, and genuinely struggles to separate a subtle paraphrase of an attack from a legitimate request that happens to talk about prompts. That overlap is exactly the region a semantic layer, an ML classifier or an LLM-as-judge on Bedrock, is supposed to resolve, and it is why I treat that layer as the next build step rather than a nice-to-have.

A note on honesty, because it is the whole point of this post: the corpus is synthetic and template-generated, so its diversity is bounded by the templates; the numbers are operational rather than a formal benchmark against published baselines; and the implementation is not open-sourced. I would not quote these figures without having run the code, and I would not dress them up as production-grade. They are a measured illustration of a trade-off, nothing more and nothing less.

The two things that make it agent-aware

Most of the field still treats this as "scan the user's prompt." For modern agentic systems that is half the surface. Two additions matter:

Output scanning. The model's response can carry an injected instruction or leak a system prompt. Scanning the output, not just the input, is where several designs (including NeuralTrust's and the GAF paper's) converge, and I'd treat it as mandatory rather than optional.

Tool-call gating. This is the one I care most about, and where indirect injection lives. When an agent decides to call a tool, that decision can be the payload: a poisoned document retrieved via RAG tells the agent to call send_email or delete_record. Gating the tool call, checking the proposed action and arguments against policy before execution, is a different and arguably more important checkpoint than filtering the prompt:

def gate_tool_call(tool: str, args: dict, policy) -> str:
    if tool in policy.denied:                       return "block"
    if tool in policy.requires_human and args_risky(args, policy):
        return "human_review"
    return "allow"

The "Firewalls to Secure Dynamic LLM Agentic Networks" work takes this further, projecting cross-agent messages onto a structured, task-scoped protocol so that persuasive framing and embedded instructions are stripped by construction. That is a stronger idea than scoring, and it is the direction I would want the tool-call layer to grow toward.

Honest limitations, up front

Because a firewall that oversells itself is worse than none:

It cannot be complete. The 2604.06436 result makes that a theorem for input-wrapper defenses, not a matter of effort. Anyone claiming 100% is wrong or redefining the problem.
The semantic layers cost latency and tokens. The sub-50ms figures you see are vendor numbers for narrower, single-layer detectors; a multi-layer ensemble with an LLM-judge step will cost more, and I would not quote a latency number I had not measured.
It is defense in depth, not a boundary. The durable lesson from the evaluation literature is that security boundaries belong in application code and permissions. A firewall lowers the hit rate; least-privilege tool scoping and human approval on destructive actions are what contain the ones that get through.

On the compliance angle

It is tempting to attach urgency to regulation, so here is the accurate version. The EU AI Act's high-risk obligations were pushed back by the 2026 Digital Omnibus to December 2027 and August 2028, not August 2026 as early drafts implied. What did take effect on August 2, 2026 is the transparency regime (Article 50) and penalty powers for general-purpose models. So the regulatory pressure is real but phased, and the honest pitch is that security controls for LLM apps are becoming table stakes over the next two years, not that a deadline hits tomorrow.

The path forward

The offline core and the trade-off measurement above are done. From here the order would be:

Add the semantic layer (an ML classifier, or an LLM-as-judge on Bedrock) specifically to attack the overlap region the offline ensemble could not separate, and re-measure whether it buys real detection without pushing false positives back up. That is the hypothesis the current numbers set up.
Harden output scanning and tool-call gating, and benchmark against indirect-injection scenarios (poisoned RAG content, malicious tool results), since that is the underserved part.
Publish it as an AWS reference architecture with the numbers attached, honest about where it sits relative to NeuralTrust, Lakera, and the OSS projects, rather than claiming to replace them.
Close the loop: feed false positives back to retune thresholds and grow the attack-vector store, and treat that feedback data as the real moat over time.

That is a build I would stand behind, because every claim in it is either cited or measured. The gap it fills is specific: an open, agent-aware, AWS-native reference design that treats the defense trilemma as a dial instead of a thing to lie about.

If you are working on LLM security, I'd genuinely like to hear which layer you think earns its keep, and whether tool-call gating or output scanning has mattered more in your own systems.

Sources

OWASP Top 10 for LLM Applications (2025, 2026) — prompt injection ranked #1 across editions. https://genai.owasp.org/llm-top-10/
arXiv 2604.06436 — "Why Prompt Injection Defense Wrappers Fail?" (the defense trilemma)
arXiv 2601.15824 — "Introducing the Generative Application Firewall (GAF)"
arXiv 2608.27990 — "CAITLYN: Can LLM Agents Autonomously Synthesize Defenses against Emerging Injection Attacks?"
arXiv 2502.01822 — "Firewalls to Secure Dynamic LLM Agentic Networks"
Check Point acquires Lakera (Sept 2025) — csoonline.com
NeuralTrust Generative Application Firewall; Cloudflare Firewall for AI (vendor sources)
EU AI Act timeline after the 2026 Digital Omnibus (high-risk obligations deferred to Dec 2027 / Aug 2028)

The benchmark figures come from my own offline reference implementation (pure standard-library Python: regex heuristic + character tri-gram cosine detectors + a weighted decision engine), run against a synthetic, template-generated labeled corpus of 192 injection and 93 benign prompts (285 total), with seed attacks held disjoint from the test set. The implementation is not open-sourced; the numbers are operational, not a formal benchmark.

Cut LLM Document-Extraction Cost by 85% Without Losing Accuracy

Avneet bansal — Wed, 07 Oct 2026 02:23:09 +0000

Most production LLM extraction pipelines route every field through a frontier model, every time. It works, and for a while nobody questions the bill. Then volume grows, or someone runs the per-document math, and the real question shows up. How much of this spend is buying accuracy, and how much is just habit.

In practice, most of it is habit. The majority of fields in a typical extraction job do not need a frontier model at all. A large share of the model calls answer questions the document never asked. A surprising fraction of calls re-extract data that has not changed since the last run. None of that is a model problem. It is an architecture problem, and architecture problems are the good kind, because you can fix them without waiting on anyone's roadmap.

I build cost-optimized inference for document-extraction workloads with 100+ fields. This is the practitioner's version of what actually moves the number. The moves below, applied in sequence, take per-document cost down by 85 to 93 percent with negligible accuracy impact. The interesting part is not any single technique. It is why they compound instead of overlap.

TL;DR

Set a trusted baseline with the expensive model first, then prove a cheaper model can match it.

Skip fields a document type cannot contain (adaptive selection).

Route each field to the cheapest model that can do the job (tiered routing + distillation).

Stop re-extracting unchanged fields (field-level caching).

Stop paying for idle (serverless scale-to-zero).

The savings multiply, not add, because each move attacks a different kind of waste.

🧭 Set the baseline with the expensive model first

Before optimizing anything, run the workload on a frontier model and treat those results as the baseline. Early on the goal is not low cost. It is output you trust and a confidence signal you believe. Once that baseline exists, it becomes the yardstick for everything cheaper.

Then, field by field, test whether a smaller and cheaper model can match it. Run the cheap model over the same documents, compare against the frontier baseline, and keep the fields where agreement is high. The fields that match move to the cheaper tier. The fields that do not stay on the frontier model.

This is how the tier percentages below get decided in practice. You are not guessing which fields a lightweight model can handle. You are measuring it against output you already trust.

The same comparison runs later as the quality-floor monitor at the end of this post. It starts here, as the step that earns each downgrade.

✂️ Start by not doing unnecessary work: adaptive field selection

The cheapest extraction call is the one you never make.

A contract schema might define 100+ fields, but a given document, say a simple service agreement, only contains 40 to 50 of them. Extracting the rest produces "not found" responses that cost tokens and return nothing. Classify the document type first, then extract only the fields that can actually appear in it.

This one move eliminates 30 to 40 percent of calls per document. It is the right thing to do first, because every later optimization then operates on a smaller base.

🎯 Right-size the model to the field: tiered routing

Not every field needs the same horsepower. In a typical extraction job:

~65 to 70 percent of fields are structurally simple: dates, amounts, party names, reference numbers, categorical flags. Predictable formats, consistent locations. A lightweight model handles these at a fraction of frontier cost with equivalent accuracy.
~20 percent need moderate reasoning: interpreting indirect phrasing, resolving ambiguous references, synthesizing across sections. A mid-tier model (for example a Haiku or Nova Lite class model) is plenty.
~10 percent genuinely need a frontier model: obligation summaries, risk assessment, termination-condition analysis, cross-document context.

Route each field to the cheapest tier that can do the job. The routing decision is nearly free. It is a field-type to tier lookup table plus a confidence-based fallback, so a tier that returns low confidence escalates to the next one. No ML is needed for routing, because the mapping is stable for a given document corpus. Blended per-document cost drops 60 to 75 percent, and quality holds because the hard fields still get frontier attention. The split above is not assumed. It comes straight out of the baseline comparison from the previous section.

Distillation: earn your way to a cheaper Tier 1

Tiered routing gets better when Tier 1 is a model you distilled for your own documents. Run a frontier model (the teacher) over 200 to 500 representative documents to produce gold-standard extractions, then train a smaller student model to reproduce that behavior on your specific fields. Managed services such as Amazon Bedrock Model Distillation handle the training without ML expertise.

The student model learns your narrow task, the dates, amounts, party names, and term lengths you see every day, and it runs much cheaper and faster with a small accuracy gap (about 2 percent) on the tasks it was trained for. Fields outside its training distribution fall through to the mid-tier or frontier model.

The bootstrap problem is obvious. You do not have 200 documents on day one, so distill progressively. Months one and two run routing-only on general-purpose models (40 to 50 percent savings) while storing every validated result as future training data. In month three, once you have accumulated enough examples, distillation runs and Tier 1 becomes your student model (another 30 to 40 percent off). Each quarter you re-distill with more data, and the share of fields handled by the cheapest tier climbs from about 70 percent toward 85 percent. The cost curve bends down the longer the system runs.

♻️ Stop paying twice: field-level caching

Documents get reprocessed when schemas are updated, prompts improve, or a quality review flags systematic errors. Without caching, reprocessing 200 documents costs exactly what processing them the first time did.

Key each result by the document-section SHA-256, the field name, the prompt version, and the model version. If all four match, return the cached value with zero inference. When you update a prompt for one field type, only that field's cache entries invalidate, and everything else serves from cache. A prompt update typically touches three to five field types, so 85 to 90 percent of fields serve from cache on reprocessing. For a 200-document corpus reprocessed after a four-field prompt change, that is 800 extractions instead of 7,000, an 88 percent reduction on that cycle.

🛑 Stop paying for idle: serverless scale-to-zero

A 200-document-per-month workload is idle most of the time. Documents land in object storage, a lightweight function writes metadata that triggers a stream event, and a containerized extraction agent runs only when work exists. During idle periods the system costs nothing beyond storage. For bursty, low-volume workloads this removes the fixed infrastructure floor that usually makes small deployments uneconomical. KEDA is a clean way to get scale-to-zero and event-driven scaling for this on Kubernetes.

🧮 Why these compound instead of add

A tempting mistake is to add the savings together. Distillation saves 75 percent, adaptive selection 35 percent, caching 30 percent, so the total must be 140 percent. That is impossible. The techniques do not each operate on the original cost. Each operates on the residual the previous stage left behind. They address different sources of waste, so they multiply:

Stage	Technique	Operates on	Residual
1	Adaptive field selection	Total field count	60 to 70 percent of baseline
2	Distillation + tiered routing	Remaining fields	20 to 25 percent of baseline
3	Field-level caching	Reprocessed documents	10 to 15 percent on reprocessing
4	Serverless scale-to-zero	Infrastructure	Zero idle cost
Result	Compound effect	End to end	5 to 15 percent of naive baseline

Roughly (1 - 0.35) x (1 - 0.65) x (1 - 0.25), which lands around 0.07 to 0.15 of the original, an 85 to 93 percent reduction. Each technique matters most at a different stage. Adaptive selection eliminates unnecessary work, distillation reduces the cost of necessary work, caching eliminates repeated work, and serverless eliminates idle overhead. No one of them gets you there alone.

🛡️ Making it safe: the quality-floor monitor

Aggressive cost optimization earns an obvious objection. If 70 percent of fields go to a cheaper model, you need a way to know it is not silently getting worse, or drifting as document patterns change.

Build the check in. Each week, randomly re-run 5 to 10 percent of Tier-1 fields through the frontier model as a shadow extraction and compare. This is the same baseline comparison from the start of the post, now running continuously. Agreement at or above 98 percent means the floor is holding. Below 95 percent raises an alert. Below 90 percent auto-triggers re-distillation on the last three months of data. Shadow validation adds only about 3 to 5 percent to total cost, and it means the system monitors itself, with no dedicated QA team required.

The same monitor also spots promotions. If the mid-tier and frontier models consistently agree on a field currently routed to Tier 2, that field is a candidate to fold into the next distillation cycle, so the system keeps shifting work to the cheapest tier on its own.

💡 The payoff beyond the bill

The reason this is worth doing well is not only the savings on an existing workload. Every percentage point off per-unit cost widens the set of use cases where automation is worth building at all. A workload that only penciled out for one of your five document types at naive pricing might clear the bar for three or four once the architecture is right. Cost optimization is not a late-stage cleanup. Treat it as a first-class design requirement alongside accuracy, latency, and reliability, and more of what you want to automate becomes viable.

Five moves, five kinds of waste:

Expensive-model baseline, then prove the cheap model turns every downgrade into a measured decision.
Adaptive field selection removes unnecessary work.
Tiered routing + distillation reduces the cost of necessary work.
Field-level caching removes repeated work.
Serverless scale-to-zero removes idle overhead.

Apply them in sequence, guard them with a self-checking quality floor, and you get frontier-grade extraction at a small fraction of the naive cost, without betting accuracy on it.

Two patterns I open-sourced

Two of the ideas above are available as small, dependency-light libraries under AWS Samples (MIT-0), if you want a reference to build from:

aws-samples / sample-textract-field-memory

Spatial field location memory for document processing pipelines. Learns field positions, validates extractions, identifies document types by layout, detects drift, and monitors template health. Zero dependencies, pure Python.

textract-field-memory

Disclaimer

This is sample code, for non-production usage. You should work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment.

Note: When processing documents containing PII (e.g. SSNs), PHI, or payment data, ensure your implementation meets applicable compliance requirements (HIPAA, PCI-DSS, GDPR, etc.). This library stores only field names and bounding-box coordinates — never field values or document content — but you remain responsible for securing the underlying documents.

The Problem

Every system that processes recurring structured data starts from scratch each time. It has no memory of what it saw yesterday.

                    WITHOUT spatial memory
┌──────────────┐
│  Document 1  │──► OCR/Extract ──► "Employee Name" at (0.05, 0.10) ──► ✓ stored nowhere
│  Document 2  │──► OCR/Extract ──► "Employee Name" at (0.05, 0.10) ──► ✓ re-discovered
│  Document 3  │──► OCR/Extract ──► "Employee Name" at (0.05, 0.10) ──► ✓ re-discovered again
│

…

View on GitHub

aws-samples / sample-prompt-correction-memory

Self-improving LLM document extraction on AWS. Each human QA correction improves future extractions via few-shot self-healing and deterministic rule graduation — no retraining, no redeployment. Costs decrease over time.

sample-prompt-correction-memory

A sample project showing automated prompt self-correction for LLM document extraction on AWS. Each human correction improves future extractions — no retraining, no redeployment.

Disclaimer: This is sample code, for non-production usage. You should work with your security and legal teams to meet your organizational security, regulatory and compliance requirements before deployment. Do not use this sample with personal data, health data or other regulated data.

The Problem

Every LLM-based extraction pipeline makes the same mistakes repeatedly. Your QA team corrects an error today, but the model makes the identical error on tomorrow's document. The correction evaporates.

                    WITHOUT correction memory
┌──────────────┐
│  Document 1  │──► LLM Extract ──► "quarterly" for payment_terms  ──► QA corrects to "30"
│  Document 2  │──► LLM Extract ──► "quarterly" for payment_terms  ──► QA corrects AGAIN
│  Document 3  │──► LLM Extract ──► "quarterly" for payment_terms  ──► QA corrects AGAIN
│     ...      │
└──────────────┘
    Same

…

View on GitHub

If you are optimizing an extraction pipeline, I would be curious where your biggest cost sink turned out to be: unnecessary calls, over-powered models, reprocessing, or idle. Drop a comment.

Your LLM Keeps Making the Same Extraction Mistake. Here's How to Make It Learn.

Avneet bansal — Sat, 03 Oct 2026 01:32:58 +0000

> TL;DR — I open-sourced an AWS sample that makes LLM document extraction learn from human corrections. Each QA fix is captured once and reused, so the pipeline gets more accurate and cheaper over time — with no retraining and no redeployment. Repo: github.com/aws-samples/sample-prompt-correction-memory. You can run the whole self-healing loop locally in 30 seconds, no AWS account needed.

The problem: corrections that evaporate

If you've built an LLM extraction pipeline, you know this pain.

Your QA analyst opens today's invoice. The model extracted payment_terms as "quarterly." That's wrong — "quarterly" is the billing frequency; the actual payment term is Net 30. The analyst corrects it. Done.

Tomorrow, an identical invoice from the same vendor arrives. The model extracts... "quarterly" again. Same mistake. The analyst corrects it again.

The correction evaporated. It fixed one document and taught the system nothing.

Multiply that across thousands of documents and dozens of recurring error patterns, and you get the two options most teams settle for:

Pay analysts to re-fix the same patterns forever (QA cost never goes down), or
Fine-tune a model (expensive, slow, needs ML ops, and you redo it every time the patterns shift).

There's a third way that needs neither.

The idea: treat corrections as durable memory

What if every human correction became a permanent, reusable asset?

That's the pattern behind sample-prompt-correction-memory. Instead of throwing corrections away, it stores them and feeds them back into future extractions — so the same mistake is never made twice.

It works as three tiers, and the magic is that traffic gradually shifts from the expensive tier to the free one:

🟢 Tier 1 — Graduated rules ($0, no LLM call)

When the same correction pattern recurs enough times, the system synthesizes a deterministic rule (a regex, a lookup, a normalization) that handles that field with zero LLM cost and sub-millisecond latency. "Net 30" → 30 becomes a rule; it never needs a model again.

🔵 Tier 2 — Cloud LLM extraction (confidence-gated)

Fields with no matching rule go to Amazon Bedrock (Claude Haiku 4.5). Every field comes back with a confidence score. If confidence clears the threshold, accept it and move on.

🟠 Tier 3 — Self-healing (only when needed)

If confidence is below threshold, the system retrieves the most relevant past corrections, formats them as few-shot examples, and re-extracts with a stronger model (Claude Sonnet 4.5). The model learns from the specific mistakes it made before — in-context, no training.

Every QA correction flows into a correction log (DynamoDB) that powers both the self-healing retrieval and the rule graduation. So the more the system is used, the smarter and cheaper it gets.

Why this matters (the practical payoff)

No ML ops. No retraining, no GPUs, no deployment cycles. A correction takes effect on the very next document.
Cost drops as quality rises. Normally you trade one for the other. Here, recurring fixes graduate to $0 rules — so accuracy climbs while per-document cost falls.
QA effort compounds instead of repeating. Fix it once → fixed forever.
It's a QA action, not an ML project. Improving the pipeline no longer requires the ML team. An analyst uploads a correction; the system improves.

The clearest fit is high-repetition, back-office document processing — accounts payable, contracts, compliance filings, supply-chain docs — where the same field errors recur and analyst time is the real expense.

The architecture

Fully serverless, deployed with AWS SAM:

Document ──► S3 ──► EventBridge ──► SQS ──► Lambda (extract)
                                              │
                          ┌───────────────────┼───────────────────┐
                          ▼                    ▼                   ▼
                   Tier 1: Rules       Tier 2: Bedrock      Tier 3: Self-heal
                   (deterministic)     (Claude Haiku 4.5)   (Claude Sonnet 4.5)
                          ▲                                        │
                          │                                        ▼
                          └──────── Correction Log (DynamoDB) ◄────┘
                                    (grows from QA feedback)

QA corrections are uploaded as JSON to an S3 corrections/ prefix; a second Lambda validates and ingests them into the correction log. Everything is encrypted with a customer-managed KMS key, and IAM is scoped to the specific Bedrock model ARNs the sample uses.

See it in 30 seconds (no AWS account)

The quickstart runs the entire self-healing loop with a mocked Bedrock client, so you can watch the mechanism without deploying anything:

git clone https://github.com/aws-samples/sample-prompt-correction-memory
cd sample-prompt-correction-memory
pip install -e ".[dev]"
python examples/quickstart.py

You'll see output like this:

Step 1: Initial extraction (no correction memory)
  Field:      effective_date
  Value:      March 2024        Confidence: 0.55   ❌ NO

Step 2: Self-healing triggers (0.55 < threshold 0.70)
  Retrieving corrections for 'effective_date'...
  Found: "March 2024" → "2024-03-01"

Step 3: Re-extraction with correction memory
  Value:       2024-03-01       Confidence: 0.95   ✓ YES
  Self-Healed: True

Low-confidence extraction → retrieve past correction → re-extract → correct answer. No retraining. No redeployment.

Deploying to AWS

With the AWS CLI, SAM CLI, and Bedrock model access for Claude Haiku 4.5 + Sonnet 4.5:

make deploy    # S3, DynamoDB, Lambda, EventBridge, SQS, KMS
make seed      # load sample corrections into the correction log
make trigger   # upload a sample document → triggers real extraction
make verify    # read back results (fields, confidence, self_healed)
make destroy   # empty buckets + delete the stack

make verify prints each extracted field with its confidence and whether self-healing kicked in — your proof it works end-to-end.

A companion: spatial memory

This sample provides semantic correction memory (what was extracted wrong, and why). Its companion, sample-textract-field-memory, provides spatial memory (where fields appear on a document layout). Together they form a dual memory for document pipelines: use the cheap spatial lookup when you're confident where a field is, and fall back to self-healing extraction when you're not.

Honest limitations

Because it's more useful when it's used well:

It's sample/reference code, not a production library. Harden IAM, add monitoring, and put a human in the loop for high-stakes fields before real use.
It helps most where the same field errors recur. For wildly heterogeneous, one-off documents, there's less pattern to learn.
Graduated rules currently take effect without a human approval step — in production you'd add a review gate before a synthesized rule starts answering on its own.
Don't use it with regulated data (PII/PHI/PCI) as-is; that path needs a proper compliance review.

Try it and tell me

The repo is open source under aws-samples, MIT-0 licensed, with a benchmark suite, an interactive dashboard, and an offline test harness.

👉 github.com/aws-samples/sample-prompt-correction-memory

What repetitive extraction error is your team still fixing by hand? I'd love to hear which patterns you'd want a system like this to learn.

This is an open-source AWS sample intended for demonstration and non-production use.

LLM Serving on Kubernetes in 2026: What's Solved and What's Still Open

Avneet bansal — Sat, 03 Oct 2026 01:11:35 +0000

A field survey of the Kubernetes + LLM inference stack in 2026 — the problems that now have serious players, and the edges that are still genuinely underserved.

If you run large language models on Kubernetes, you've probably noticed the ground shifting fast. A year ago most of the hard problems were open. In 2026 a lot of them have serious, well-funded players — which changes the calculus if you're deciding where to build. Competing with a GA project is very different from pioneering an empty space.

This is a map of the territory: what's crowded, and where I think the real gaps still are. I'm deliberately not pitching any of these as "my novel architecture" — several have academic papers or shipping projects attached. The point is to show where the open problems are so you can decide where your effort pays off.

What's already crowded

These are the areas where the obvious pain points now have mature answers. Building here means competing, not pioneering.

KV-cache-aware routing / model-aware gateways. Largely solved. The Gateway API Inference Extension (a kubernetes-sigs project) defines a standard InferencePool routing contract, and llm-d, NVIDIA Dynamo, and GKE Inference Gateway (which went GA in 2026) all do prefix-cache affinity and cache-aware routing on top of it.

Prefill/decode disaggregation. NVIDIA Dynamo and llm-d cover this well.

Fractional GPU sharing (MIG / MPS / time-slicing). Run:ai, cast.ai, the NVIDIA GPU Operator, and vCluster all address it.

GPU scheduling primitives (DRA, gang scheduling). KAI Scheduler, Grove, Kueue, and Volcano are all in play, with Dynamic Resource Allocation graduating in core Kubernetes. Crowded, though still messy at the edges.

If your idea lives in one of those buckets, go in with eyes open: you're entering a race.

Where the real gaps still are

Three edges still look genuinely underserved, and each is shaped like a Kubernetes-native problem — a CRD plus a controller plus some node-level machinery.

1. Model-weight distribution as a first-class Kubernetes primitive

This is the one I find most interesting. Model weights are often 100+ GB, and today they're still delivered largely by ad-hoc downloads from object storage — without the pull-caching, digest addressing, and content verification that Kubernetes already gives you for container images. That gap is a major driver of the two-to-ten-minute cold starts that make scale-to-zero economically painful for inference, since every new replica pays that cost before serving its first request.

The literature backs up the problem. An empirical study (arXiv 2607.16596, "Cold-Start Model Delivery in Kubernetes Inference Serving") measures the delivery paths — modelcar sidecars, native image volumes, object-storage download — on artifacts sized to 1B-, 7B-, and 70B-class weights, and a companion paper (arXiv 2609.20874) frames the same two-to-ten-minute startup as the reason reactive autoscaling is structurally late. So the pain is validated; what's missing is a first-class primitive for it. Note that these papers study and characterize the problem — they don't ship the CRD-shaped solution below, and neither do I claim to have invented it.

The interesting shape of a solution: a ModelArtifact / ModelCache CRD, a controller, and a node-level DaemonSet that treats weights like OCI-addressable, content-verified, node-locally-cached blobs — with topology-aware peer-to-peer prewarming (pull from a neighbor node's cache over the fast fabric instead of re-pulling from object storage), plus scheduler hints so pods land where the weights already live. NVIDIA Dynamo's ModelExpress already does some of this (one worker publishes weights, others pull over NIXL/RDMA), so even here you'd be refining a direction others have started, not working in a vacuum.

Why it's a good build target: it's infrastructure-shaped, CRD-heavy, and not yet a solved commodity. You can prototype the caching, P2P, and scheduler-hint logic and measure cold-start reduction on modest hardware — or even with simulated weights — without a large GPU budget.

2. SLO-driven inference autoscaling that understands request heterogeneity

A caution up front: "SLO-aware inference autoscaling" as a headline is already a shipped commodity — AWS, KEDA, and academia all have versions of it. So this is only interesting if you go narrower than the headline.

Current autoscaling scales on queue depth or KV-cache utilization. But the Cascade paper (arXiv 2608.06557, "Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving") shows that requests under the same SLO can differ in urgency by a wide margin — a short chat versus a long agentic reasoning trace — and that you can direct queueing and data-movement overhead toward the requests that can absorb it. Related systems like Scorpio (arXiv 2505.23022) exploit the same SLO heterogeneity for admission and scheduling, so this is an active research line, not an empty one. The defensible wedge is per-request-class latency budgeting plus admission control: an InferenceSLO CRD where you declare per-class latency budgets, and a controller that combines a batch-latency predictor with HPA/KEDA to make admission and scaling decisions on predicted goodput rather than raw queue length.

This one is more research-flavored and genuinely publishable — but only if you build and benchmark it against the existing SLO-aware baselines. Without numbers, it reads as a restatement of what's already shipping.

3. Multi-LoRA adapter lifecycle as a CRD

Serving hundreds of tenant-specific LoRA adapters over one base model is now a real production pattern (arXiv 2511.22880), and vLLM and Dynamo support runtime adapter loading. What's still hand-rolled is lifecycle management: which adapters are hot, cache eviction by usage, routing requests to workers that already hold the adapter, and VRAM budgeting. Dynamo has some declarative Kubernetes support here, but it's early.

The shape: a LoRAAdapter + AdapterPool CRD that manages adapter placement, usage-based eviction, and adapter-affinity routing hints — essentially "KV-cache-aware routing," but for adapters.

How I'd choose

If you want a portfolio or OSS operator project: idea #1. It's clearly the most infrastructure-shaped, there's a fresh paper validating the problem, and it's demonstrable without a big GPU budget.

If you'd rather write something paper-worthy: idea #2, with the per-request-class framing and real benchmarks.

Idea #3 is a reasonable middle ground.

A closing note on honesty, because it matters in this space: none of these are empty green fields. Each sits next to active prior art, and dev.to readers who work on inference will know it. The value here isn't a claim of novelty — it's a clear-eyed map of where the race is already crowded and where there's still room to build something useful.

Sources

Gateway API Inference Extension — kubernetes-sigs/gateway-api-inference-extension (github.com))
NVIDIA Dynamo; ModelExpress weight distribution (docs.nvidia.com)
Run:ai, cast.ai, NVIDIA GPU Operator, vCluster (GPU sharing)
KAI Scheduler, Grove, Kueue, Volcano; Dynamic Resource Allocation in Kubernetes
arXiv 2607.16596 — "Cold-Start Model Delivery in Kubernetes Inference Serving: An Empirical Study of OCI-Based Distribution and Its Integrity" (G. Kliukovkin)
arXiv 2609.20874 — "Decomposing Predictive Kubernetes Autoscaling for LLM Serving Under Long Startup Delays"
arXiv 2608.06557 — "Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving"
arXiv 2505.23022 — "Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference"
arXiv 2511.22880 — "Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems"

Give Your Document OCR Pipeline a Memory

Avneet bansal — Tue, 08 Sep 2026 22:30:40 +0000

After OCR, most pipelines throw the layout away and re-discover the same form from scratch every time. textract-field-memory is a small, zero-dependency Python layer that remembers where fields sit, checks they landed in the right place, tells you which layout a document is, and warns you when a form quietly changes. It doesn't do OCR. It makes the OCR you already run a lot more useful.

Where this started

I kept running into the same annoying gap on document pipelines. You send a form through Textract (or Tesseract, or whatever), you get back field names and bounding boxes, you pull the values, and then you throw all the positional information away. The next copy of that exact same form shows up an hour later and the pipeline treats it like it's never seen anything like it before.

That bugged me for a few reasons.

Nothing checks that a field came from the right spot on the page. If "Total" gets read out of the shipping box instead of the amount box, the value still looks like a number, so it sails downstream and nobody blinks.

Nothing notices when a form changes. A vendor nudges their invoice template, fields move a centimeter, and you find out three weeks later when a downstream job starts choking on garbage.

And there's no cheap way to ask "have I seen this layout before?" So every document, familiar or not, takes the same expensive road.

So I wrote a little library that sits after OCR and keeps the memory the pipeline was throwing out.

What it is, and what it definitely isn't

It's a spatial memory layer. You hand it field names plus normalized bounding boxes (x, y, width, height between 0 and 1). It learns roughly where each field tends to appear on a given template, and from then on it can validate positions, identify which template a document belongs to, and watch for drift.

It is not an OCR engine and it never touches pixels. Something upstream has to extract the fields first. This library just remembers where they ended up and reasons about that.

Under the hood there's nothing exotic going on, which I think is a feature. Spatial scoring is intersection-over-union plus the distance between box centers. Name matching is a normalized Levenshtein distance. Template identification is Jaccard overlap of the field-name sets combined with the spatial score. No model, no training step, no network calls. It's pure standard-library Python with zero dependencies, and templates are just JSON files on disk.

A quick look at the API

from field_memory import TemplateMemory

memory = TemplateMemory(store_path="./my_templates")

# 1. Learn — feed documents after OCR extraction
memory.record(document, template_id="employment-form")

# 2. Locate — spatial field lookup with a confidence score
matches = memory.locate(new_doc, "Employee Name")
# → FieldMatch(combined_score=0.95, spatial_score=0.93, within_expected_region=True)

# 3. Identify — which known layout is this?
match = memory.identify_template(new_doc)
# → TemplateMatch(template_id="employment-form", similarity_score=0.87)

# 4. Detect drift — has the layout shifted?
drift = memory.detect_drift(new_doc, "employment-form")
# → DriftReport(is_drifting=False, overall_drift_score=0.02)

Clustering comes for free. If you call record() and leave out the template_id, it either merges the document into a template it recognizes or spins up a new one. Run that for a while and you end up with a map of every distinct layout flowing through your pipeline without ever configuring one by hand.

Why there's no LLM inside

For a layout you've already seen, answering "where's this field?" is basically a dictionary lookup. Sub-millisecond, and free. The same question routed through an LLM is half a second to two seconds and costs you per field. Paying that toll on forms you've processed a thousand times is silly.

So the design leans into being complementary instead of competitive:

def smart_extract(document, field_name):
    matches = memory.locate(document, field_name)
    if matches and matches[0].within_expected_region and matches[0].combined_score >= 0.85:
        return matches[0].key_value, "spatial"   # free, instant
    return call_llm(document, field_name), "llm"  # pay only when needed

Think of it as the cheap first pass that decides whether you even need the expensive one. The 0.85 threshold is just a starting point. Crank it up if a wrong answer is costly and you'd rather fall back to the LLM more often.

Where it actually helps

Domain	What it remembers	What it catches
Invoice processing	Where fields sit per vendor layout	Vendor template changes, wrong-position reads
Insurance claims	Field positions across many form types	Unknown forms that need routing for review
Healthcare intake	Where fields land on clinic forms	Layout revisions before they break downstream
Manufacturing QA	Component positions on boards	Parts that shifted from baseline

The only real requirement is that your data has named things with bounding boxes. That's it.

Some numbers, and the caveats that go with them

I put it through a stress test using real Textract (FORMS) on generated PDFs plus scanned, deliberately degraded versions of the same documents.

Layout identification came out at 100% across five vendors whose invoices look nothing alike, trained on just three documents each. On dense tax returns with 33 fields crammed into a tight three-column grid, roughly 0.03 units apart, it located every field. Drift detection behaved the way I hoped: as I pushed forms further off their baseline, the drift scores climbed smoothly from 0.012 up to 0.047 and tripped the flag once they crossed the threshold. On injected anomalies (shifted, scattered, reversed, wrong-form) it caught four of five. And because it's all local computation, lookups stay under a millisecond.

Now the honest part.

The benchmark runs on synthetic PDFs generated with ReportLab and their scanned variants, not a big real-world labeled corpus. Sample sizes are small, around 20 forms per scenario.

There's no proper precision/recall/F1 yet because I don't have a ground-truth label set built. The numbers above are operational rather than a formal comparison against baselines, and I'd treat them that way.

The one anomaly it missed was the "same layout, just fewer fields" case. The fields that remain are still sitting in the right places, so spatially nothing looks wrong. You could argue that's correct behavior, but it's worth knowing going in.

Handwritten forms are a dead end, because Textract FORMS gives back nothing to learn from. That's an upstream limitation, not something this library can paper over.

I'd rather you hit these on the page than in production.

Try it

Clone it and install from source. This works right now, nothing to sign up for:

git clone https://github.com/aws-samples/sample-textract-field-memory
cd sample-textract-field-memory
pip install -e .
python examples/demo.py   # runs on synthetic data, no AWS needed

demo.py walks through template learning, spatial matching, anomaly detection, and drift analysis on synthetic data, so you can watch the whole thing work without credentials or config.

Repo: github.com/aws-samples/sample-textract-field-memory

It lives under aws-samples but there's no AWS dependency in it. Pure standard-library Python, works completely offline with any OCR source that gives you field names and boxes. MIT-0 licensed.

If you run document pipelines, the thing I'd most like feedback on is turning these stress tests into a real labeled benchmark. And if the "remember it, stop re-discovering it" idea resonates, you've got the whole pitch.