DEV Community: Raj Mohamed

AI System Design for Production

Raj Mohamed — Wed, 21 Oct 2026 13:11:59 +0000

Your Chatbot Demo Works. That Doesn't Mean It's Ready for 10,000 Users.

An architect's walkthrough of what has to exist around the LLM before you ship an AI assistant to real customers

The LLM call is the visible part. Almost everything that decides whether you survive launch week sits below it.

Picture this. An engineer at an e-commerce company — let's call it BlipKart — has built a chatbot. It answers questions, looks up products and even walks a customer through placing an order. The demo goes well. The CEO is impressed and asks for one thing: put it in front of 10,000 users by Monday.

It's a familiar moment, and it hides a trap. A demo proves that a model can produce a good answer when a friendly person asks a friendly question. Production asks a different question: can the system keep producing safe, correct, affordable answers when thousands of strangers use it at once, some of them trying to break it, while your model provider has a bad day?

Calling an LLM API is the easy part. Running a dependable product around it is the job. After fifteen years designing payments, cards and banking platforms — where "it worked in the demo" has never once been an acceptable sign-off — I find the same disciplines apply almost one-for-one to AI systems. Here is how I'd take BlipKart from demo to production.

1. Start with requirements you can measure

Before drawing a single box, separate two conversations that teams constantly blur.

High-level design is the map: which major components exist and how they talk. For BlipKart that's an API gateway, an intent router, a cache, a retrieval-augmented generation (RAG) pipeline, a model gateway and the commerce APIs.

Low-level design is the inside of each box: vector-store replicas and regions, chunk sizes, schemas, prompt templates, the classes that implement them. Mixing the two levels in one review is how a team spends an hour debating chunk size while nobody notices there is no authentication layer.

Then write the requirements down in two lists.

Functional — what the bot must do: answer FAQs, track orders, process refunds, hand certain conversations to a human.

Non-functional — how well it must do it, with numbers stakeholders sign off on:

P95 latency under 3 seconds
Monthly spend within ₹3 lakh
Zero PII leaks
99.9% availability

If those numbers aren't written down, every later trade-off becomes an argument about opinions instead of a check against a target.

Do the capacity maths — and don't trust the average

10,000 users × 5 messages a day = 50,000 messages. Spread evenly across 86,400 seconds, that's roughly 0.58 requests per second. It sounds trivial. It's also misleading.

Same 50,000 messages, very different load. Even a gentle evening peak is more than double the daily average — and a sale campaign or push notification can be far worse.

Traffic clusters: lunch breaks, evenings, the minute a promotional push lands. Then add retries, multi-turn conversations and tool calls that each fan out to several model invocations. Size for the peak minute, not the day, and convert messages into tokens per month so you can talk to your provider about rate limits and reserved capacity before launch, not during an incident.

2. Protect customers at the boundary

A reference shape for the system. Everything inside the trust boundary runs before a single token is generated.

Authenticate before any account action

"Refund my last order" means nothing until the system knows whose last order. Identity has to be established with proper standards — OIDC, OAuth-based sign-in, SSO — using an identity provider such as Keycloak (or whatever your enterprise already runs).

One point I'd push harder than most write-ups do: authorisation must live in the tools, not in the prompt. The order-lookup tool should derive the customer ID from the verified token, never from text the model passes in. If the model can be talked into asking for "order 4471 for customer 9902", the only thing that should stop it is the API refusing — not the model's good manners.

Let the gateway absorb abuse

An API gateway is the right place to enforce per-user request limits and daily or monthly token budgets, before traffic reaches either your application or the LLM. Pair it with a web application firewall and load balancing. Cloudflare, NGINX, Kong, Apigee or your cloud provider's native gateway can all do this. Tune the limits to protect the provider and your bill without punishing a genuine customer who asks five follow-up questions.

Check what goes in — and what comes out

Prompt injection is not theoretical. There's a widely reported case of a car dealership's chatbot appearing to agree to sell a vehicle worth around $70,000 for $1 after a user instructed it to agree with everything. Whatever the legal outcome of such cases, the design lesson is clear: define, in advance, what the bot is never allowed to promise.

Input guardrails flag disallowed or manipulative requests.
Output guardrails catch PII, prices or commitments the bot has no authority to make before the response reaches the customer.

Legal and compliance should own the list of prohibited responses. Engineering owns enforcing it.

3. Route every request to the right place

Not every message deserves a large language model. A greeting, an order-status question, a refund request and a policy question are four different workloads.

An intent router sits in the middle of the design and picks the path:

Greetings and thanks → a fixed response
Order status, prices, refunds → commerce APIs and tools
Policy and how-to questions → RAG
Ambiguous sales or support questions → a classifier, and only then an LLM

Cache what's stable. Fetch what's live.

Semantic caching is a cost win for stable answers and a correctness bug for anything live or personal.

Semantic caching matches questions by meaning, so "how do I pay with UPI?" and "can I use UPI to pay?" can share one answer. That saves real money on repetitive FAQs.

It is also a sharp tool. Two rules I'd treat as non-negotiable:

Never cache live facts or account actions. Prices come from the pricing API. Order status comes from the order service. A return request triggers a tool, not a remembered answer.
Never let a cache key cross a customer boundary. A cached answer that contains anything personal is a PII leak waiting to happen.

Give every cached entry a TTL and an explicit invalidation trigger.

Keep RAG content honest

Policy PDFs and troubleshooting guides get chunked, embedded and indexed in a vector store — Qdrant, Chroma, Pinecone, pgvector and others. The failure mode isn't retrieval quality on day one; it's staleness on day ninety, when the return policy changed and the bot is still quoting the old one.

Wire change data capture into the content source. When a document changes, re-index only the affected chunks and evict the cache entries built on them. No nightly "re-index everything" job, and no stale answers lingering for a week.

Know when to hand over to a human

A customer disputing a charge or about to cancel a membership needs judgement and empathy, not a generated paragraph. Decide up front which intents the AI must never handle alone. And if the bot is allowed to offer something with commercial value — a retention discount, a goodwill credit — write explicit rules for when, how much and how often.

4. Control cost without lowering quality

Minimum sufficient intelligence

Climb only as high as the task needs. Every rung up costs more and behaves less predictably.

This is the principle I'd put on the wall: use the least complex mechanism that meets the quality bar.

"Thanks!" needs a rule, not a model.
"Where's my order?" needs an API call, not a model.
Sorting a known set of request types is a job for a traditional ML classifier.
A standard return-policy answer is fine on a small, cheap LLM.
A split-shipment refund with a partial return and a voucher? That may genuinely need a larger reasoning model.

Rules are cheap, fast and predictable. Models are flexible. Spend flexibility only where you need it.

Route models by task, and prove it with evals

Model routing sends each request to the model best suited to it. The choice shouldn't be made on brand or parameter count — build an evaluation set from real BlipKart conversations and score candidates on accuracy, latency, cost and policy compliance. Re-run it every time you change a prompt or a model version.

Make spend visible before it becomes a surprise

If the monthly bill quietly drifts from ₹3 lakh to ₹6 lakh, you want to find out on day eight, not in the invoice. Set token budgets per user or account, track spend per model and per intent, and alert as you approach thresholds. Review regularly which intents actually justify the expensive model.

5. Build for spikes and outages

Queues absorb bursts; the model gateway hides provider failures behind a tested fallback.

Put a model gateway between you and your providers

An AI/model gateway keeps provider keys out of application code, centralises rate limits and token budgets, and can fail over to a second provider when the primary returns errors or throttles you. LiteLLM and Vercel AI Gateway are two examples.

The catch: a fallback you haven't tested is not a fallback. Prompts tuned for one model often behave differently on another — different formatting, different refusal behaviour, different tool-calling quirks. Run your evaluation suite against the fallback provider too, and load-test it at the traffic it would actually receive during a failover.

Queue, throttle, scale

When requests arrive faster than you can serve them, queues smooth the curve. Backpressure and throttling slow intake before you overwhelm the provider. Kafka or RabbitMQ can hand long-running work to background workers so the chat API doesn't block. Load balancing and autoscaling add and remove compute as traffic moves.

Streaming responses over server-sent events shows text as it's generated. It doesn't make the total answer faster — but it dramatically reduces the time a customer stares at a spinner, which is what they actually perceive.

Make releases and recovery boring

Infrastructure as code (Terraform) and CI/CD (GitHub Actions, GitLab, Jenkins) make deployments repeatable and reviewable. For AI systems I'd add one more gate: an automated evaluation run that blocks the deploy if quality drops. Every release needs a tested rollback path, and the team should rehearse recovery before a real outage forces them to improvise.

6. Measure health — and correctness

A green uptime dashboard doesn't mean the bot is right. A chatbot can be fast, available and confidently wrong.

Track the operational signals — P95 latency, token usage, spend, error rates — and the behavioural ones: did it call the right tool? Did the refund flow complete? Did the answer match current policy? Tools like Datadog, New Relic, LangSmith and MLflow each cover parts of this.

Set alerts on the targets you agreed in step one: latency above 3 seconds, spend crossing budget, guardrail trigger rates spiking. Route them by severity — email for trends, a page for an outage. Keep end-to-end traces so an engineer can see exactly which hop failed. And keep re-checking behaviour after every change to prompts, models, policies or integrations, because each one can shift answers in ways no unit test will catch.

The production-readiness checklist

Twelve questions to answer before opening access to real customers.

BlipKart's prototype is a good start. But production readiness was never about the model — it's about the system around it.

Before you open the doors to 10,000 users, confirm the requirements and the peak-traffic maths. Check authentication, abuse controls, guardrails, data freshness, human handoffs, cost limits, fallbacks, scaling, monitoring and recovery. Then run a load test at the expected peak and, just as importantly, prove the system fails safely — that when something breaks, the customer gets a polite handoff, not a wrong refund.

Every one of these choices is a trade-off between risk, quality, latency and budget. There's no universal stack. But there is a universal question worth asking before Monday:

If this goes wrong at 8 pm on launch day, what happens next — and who finds out first: you, or your customers?

I'm Raj Mohamad Sabibullah, a Senior Solution Architect working across payments, cards, banking platforms and applied AI. If you're taking an AI assistant from prototype to production, I'd like to hear what tripped you up — leave a response below or connect with me on LinkedIn.

Tags: Artificial Intelligence · System Design · LLM · Software Architecture · Generative AI

# Building a STRIDE-Aware Fraud-Pattern Review Agent on Amazon Bedrock

Raj Mohamed — Wed, 21 Oct 2026 06:08:13 +0000

A step-by-step build: from model access to a deployed agent, with every console screen along the way.

Raj Mohamad Sabibullah · Senior Solution Architect

I spend most of my week in the unglamorous middle layer of a digital bank build: a card management system on one side, a core banking system on the other, Kafka topics carrying every authorization and reversal in between, and a security team that wants a threat model for anything that so much as reads a transaction record. It is not the part of the stack that gets a conference talk. But it is exactly the part where agentic AI turned out to be more useful than I expected — not as a chatbot bolted onto a mobile app, but as a colleague sitting inside the fraud-ops workflow, reading runbooks faster than any analyst can, and never seeing a live PAN.

This post is the build log I wish I'd had going in: every step from requesting model access to a working CLI invocation, in order, with the console screen at each step. It's long on purpose — a previous, shorter version of this post got fair feedback that it skipped the "how," so this one doesn't skip anything. Account IDs, transaction references and internal system names below are illustrative, not real; the steps, screens and configuration are the real sequence I used.

What we're building An agent that takes a flagged transaction reference from a cards-ops analyst, classifies it against a STRIDE-based fraud taxonomy, cites the PCI DSS 4.0 control it relied on, and returns a risk rating — without ever seeing or returning an unmasked card number.

On this page

Prerequisites
Request model access
Stage the knowledge source in S3
Create the Knowledge Base
Create the IAM execution role
Write the Lambda action groups
Create the agent
Attach the action group schema
Create the guardrail
Prepare and test the agent
Create an alias, promote a version
Codify in CDK and deploy
Invoke from the CLI / your app
Reference architecture
Mapping STRIDE to cards
What broke, what I'd change

0 Prerequisites

Before touching the console, make sure you have:

An AWS account with Amazon Bedrock available in your target region (I used eu-central-1 for data-residency reasons — check what your own compliance scope requires before picking a region).
IAM permissions to create roles, Lambda functions, S3 buckets, an OpenSearch Serverless collection, and Bedrock resources (agents, knowledge bases, guardrails). In a real bank environment this is almost never "AdministratorAccess" — I worked with a scoped permission set our cloud platform team provisioned for the Bedrock build.
The AWS CLI v2 and the AWS CDK v2 installed locally, both authenticated against a non-production account/profile.
A read-only API in front of your core-banking system (CMS, Core Banking System, or equivalent) that the agent's action group can call. The agent never talks to the core-banking systems directly — it always goes through this proxy, which is the enforcement point for "read-only, no PII."
Your fraud runbooks and control-mapping documents, as PDFs or text files, ready to upload as the knowledge source.

With that in place, the build runs in the order below — each step depends on the one before it.

1 Request model access

Bedrock model access is granted per account, per region, and it is not on by default. This trips up almost everyone the first time — an agent creation will fail silently referencing a model you haven't been granted.

Go to Amazon Bedrock → Model access and request access to the models you need. For this build that's Claude 3.7 Sonnet (the agent's reasoning model) and Titan Text Embeddings V2 (for the knowledge base).

Model access is a one-time, per-region grant. Approval for Anthropic models is usually instant; some providers require a use-case form.

Access typically activates within a few minutes for Anthropic and Amazon models. Confirm the status shows Access granted before moving on — every later step assumes this is done.

2 Stage the knowledge source in S3

The agent's domain knowledge — STRIDE runbooks, the PCI DSS control mapping, prior incident write-ups — needs to live somewhere Bedrock can ingest it from. Create a dedicated S3 bucket and upload the source documents.

bash

aws s3 mb s3://stride-runbooks-kb-src-eucentral1 --region eu-central-1
aws s3 cp ./runbooks/ s3://stride-runbooks-kb-src-eucentral1/ --recursive

One file per STRIDE category kept retrieval clean later — a runbook that blends two categories produces a knowledge-base chunk that blends two answers.

A few things that mattered here:

One document per STRIDE category, not one giant "fraud handbook" PDF. Chunking works section-by-section; a single 40-page PDF covering everything produces chunks that straddle unrelated topics.
Enable versioning on the bucket if your governance process requires an audit trail of what the agent was trained on at a point in time — ours does, since a control mapping can change between PCI DSS revisions.
Keep this bucket in the same region as the knowledge base you create next; cross-region ingestion adds complexity you don't need for a first build.

3 Create the Bedrock Knowledge Base

This is where the runbooks become something the agent can retrieve from. Go to Amazon Bedrock → Knowledge bases → Create knowledge base and work through the wizard.

The first two steps of the wizard (name/description, then pointing at the S3 data source) are straightforward. The step worth slowing down on is vector store configuration:

Chunk size was the single highest-leverage tuning knob in this whole build — more than model choice or prompt wording.

Configuration choices that mattered:

Vector store: "Quick create a new vector store" provisions an Amazon OpenSearch Serverless collection for you. For a first build this is the right choice — you can migrate to a self-managed collection or Aurora pgvector later if you need finer control over capacity or cost.
Embeddings model: Titan Text Embeddings V2 at 1024 dimensions. Cohere and other embedding models are available; Titan was the default and there was no reason to move off it for this use case.
Chunking strategy: fixed-size, 150 tokens, 20% overlap. The Bedrock default is 300 tokens. Our runbook sections are short and dense (one STRIDE category, one control, one example), and at the 300-token default the agent would occasionally retrieve a chunk that blended two adjacent categories. Dropping to 150 tokens fixed this more reliably than any prompt-level instruction did.

After the wizard finishes, sync the data source (Knowledge base → Data source → Sync) so the documents in S3 are actually embedded and indexed — creating the knowledge base does not automatically ingest the files. Sync typically takes a few minutes for a handful of PDFs; watch the sync history for failures before assuming it worked.

4 Create the IAM execution role

Every Bedrock agent runs under an IAM role that Bedrock assumes to invoke the model, retrieve from the knowledge base, and call your action-group Lambdas. This is the control that turns "an LLM agent" into "an LLM agent that provably cannot write to your core-banking system," and it's worth building before you build the agent itself.

The permissions boundary is the layer that matters most: even if the Lambda code changes next year to add a write path, the boundary denies it at the IAM layer regardless of what the code tries to do.

Two design decisions here, not just defaults:

A permissions boundary, not just attached policies. Attached policies say what the role can do; the boundary is a hard ceiling on what any policy attached to this role — now or in the future — is allowed to grant. We set ours to explicitly deny any write action against the core-banking proxy, so a future code change that tries to add a write call fails at deploy time, not in production.
One policy per Lambda, scoped to that Lambda's ARN, rather than a single lambda:InvokeFunction on *. This is more setup work and it is the difference between "the agent can call three specific, reviewed functions" and "the agent can call any Lambda in the account," which is not a sentence I wanted to say to our security team.

The trust relationship restricts sts:AssumeRole to the bedrock.amazonaws.com service principal, scoped further with an aws:SourceAccount condition — standard practice, worth double-checking it's actually there rather than assumed.

5 Write the Lambda action-group functions

The agent doesn't touch any real system directly — every action it can take is a Lambda function behind an action group. We built three:

Function	Purpose	Access
`cms-read-proxy-fn`	Looks up a transaction by reference ID, returns a masked record	Read-only, scoped IAM role into the CMS read replica
`stride-classifier-fn`	Scores the transaction against STRIDE categories using rules + recent MSK velocity features	Reads from MSK consumer group, no write access anywhere
`sns-notify-fn`	Publishes an alert to the cards-ops on-call topic when risk is HIGH	`sns:Publish` on one topic only

A trimmed version of the transaction-lookup handler, showing the masking that happens before anything reaches the model:

python · cms-read-proxy-fn

import json
import boto3

def lambda_handler(event, context):
    reference_id = event["parameters"][0]["value"]
    record = fetch_from_cms_readonly(reference_id)  # internal read-only client

    # Mask before this ever leaves the function — the agent and the
    # guardrail are a second line of defense, not the first.
    masked_pan = "•••• •••• •••• " + record["pan"][-4:]

    response_body = {
        "referenceId": reference_id,
        "amount": record["amount"],
        "currency": record["currency"],
        "maskedPan": masked_pan,
        "deviceFingerprint": record["device_fingerprint"],
        "authCount90s": record["velocity_90s"],
        "merchantCategoryCode": record["mcc"],
    }

    return {
        "messageVersion": "1.0",
        "response": {
            "actionGroup": event["actionGroup"],
            "apiPath": event["apiPath"],
            "httpMethod": event["httpMethod"],
            "httpStatusCode": 200,
            "responseBody": {"application/json": {"body": json.dumps(response_body)}},
        },
    }

The masking happening inside the Lambda — not left to the model or the guardrail to catch — is deliberate. Guardrails and instructions are the second and third layers; the function itself should never emit a raw PAN in the first place.

Deploy these three functions however your team normally deploys Lambdas (console, SAM, CDK — we moved to CDK for this in step 11). Note each function's ARN; you'll need them for the action group and the IAM role above.

6 Create the agent

With the knowledge base synced, the execution role scoped, and the Lambdas deployed, go to Amazon Bedrock → Agents → Create agent.

Everything the agent can reason with or act on attaches to this one resource: model, instructions, action groups, knowledge base, and guardrail.

Fill in, in order:

Agent name and description — fraud-pattern-review-agent.
IAM role — select the execution role created in step 4 (or let the console generate one, then immediately tighten it to match — I'd recommend building it yourself first, since the auto-generated role is intentionally broad).
Foundation model — Claude 3.7 Sonnet.
Agent instructions — this is the single highest-leverage text in the whole build. Ours:

agent instructions

You are a fraud-pattern review assistant for the XY Bank Cards Workstream.
Given a transaction reference, cross-check STRIDE threat categories against the
knowledge base of card-fraud runbooks, summarize risk indicators in plain
language, cite the matching control from PCI DSS 4.0 / CBE regulation where
applicable, and NEVER return raw PAN, CVV, or cardholder PII in your response.
Always call the transaction-lookup action before concluding a risk rating.

Three lessons baked into that wording:

State the negative constraint as its own capitalized sentence, early. Buried as a passive aside, it gets ignored more often than as a standalone line.

Force the tool call explicitly. Without "always call the transaction-lookup action before concluding," the agent would occasionally answer from a prior turn's context instead of looking the transaction up fresh.

Name the citation format you want. Asking for "the matching control from PCI DSS 4.0 / CBE regulation where applicable" produces consistent, checkable citations — paired with contextual grounding in the guardrail (step 8), which stops the model from inventing a plausible-sounding control number it didn't actually retrieve.

Save the agent. It exists now, but it can't do anything yet — no action groups, no knowledge base link, no guardrail. Those are the next three steps.

7 Attach the action group and its OpenAPI schema

An action group tells the agent what it can call and how — Bedrock uses an OpenAPI schema to generate the function-calling interface, and a Lambda function to actually execute the call.

Go to the agent's Action groups tab and create one per Lambda (transaction-lookup, stride-classifier, analyst-notify). For each, select "Define with API schemas," point it at the corresponding Lambda ARN, and provide the schema.

The OpenAPI schema is what lets the model decide, on its own, when to call this action and with what parameters — get the field descriptions right and the model's tool-use gets noticeably more reliable.

The schema for transaction-lookup:

yaml · openapi schema

openapi: 3.0.0
info:
  title: Transaction Lookup API
  version: 1.0.0
paths:
  /transaction/{referenceId}:
    get:
      summary: Retrieve a masked transaction record by reference ID
      operationId: getTransaction
      parameters:
        - name: referenceId
          in: path
          required: true
          schema: { type: string }
      responses:
        '200':
          description: Masked transaction record
          content:
            application/json:
              schema:
                type: object
                properties:
                  referenceId: { type: string }
                  amount: { type: number }
                  currency: { type: string }
                  maskedPan: { type: string, example: "•••• •••• •••• 4471" }
                  deviceFingerprint: { type: string }
                  authCount90s: { type: integer }
                  merchantCategoryCode: { type: string }

Two practical notes: the summary field on each operation matters more than you'd expect — it's what the model reads to decide when to call this tool, so write it as a clear instruction, not a terse label. And repeat this for stride-classifier and analyst-notify with their own schemas before moving on — an agent with only one of three action groups attached will happily answer without ever invoking the other two.

Then, on the agent's main page, link the knowledge base created in step 3 under the Knowledge base section, and give it its own retrieval instructions (a short sentence telling the agent when to consult it — e.g., "Use this knowledge base to look up STRIDE categories and their matching compliance controls").

8 Create the guardrail

This is the step I'd tell anyone building something similar to budget the most time for — it took longer than the agent and action groups combined, mostly in review cycles with our security team, not console clicks.

Go to Amazon Bedrock → Guardrails → Create guardrail.

A guardrail is a separate, versioned resource from the agent — you can update the guardrail and ship a stricter policy without redeploying the agent itself.

Three configuration sections did most of the work:

Sensitive information filters — add CREDIT_DEBIT_CARD_NUMBER and CREDIT_DEBIT_CARD_CVV as built-in PII types, action set to Block (not just mask), plus a regex-based filter for our local national ID format. Blocking rather than masking here means a masked PAN can't even partially leak across a multi-turn conversation.
Denied topics — phrased as topics ("bypassing 3-D Secure or step-up authentication," "requesting core-banking service account credentials"), not keyword lists. Keyword blocklists are trivially rephrased around; topic-level denial catches the intent.
Contextual grounding checks — grounding threshold 0.75, relevance threshold 0.70. This is the control that rejects a generated answer when it isn't well-supported by what was actually retrieved from the knowledge base — it's what stops the model from citing a PCI DSS requirement number it half-remembers instead of one it retrieved.

Once created, attach the guardrail to the agent from the agent's main configuration page (visible in the step 6 screenshot above, bottom right). Guardrails are versioned independently of the agent, so tightening a filter later doesn't require touching the agent resource at all — useful when a security review asks for a stricter policy mid-project.

9 Prepare and test the agent

With instructions, action groups, knowledge base, and guardrail all attached, click Prepare on the agent (top right of the Agent builder screen) — this compiles the current draft configuration into something invocable. Then open the Test window to run it end to end before anything touches a real alias.

A full round trip in the test window: two action-group calls, a STRIDE classification, a cited control, and a masked card number — all before this reaches a real alias.

What to check here, beyond "did it answer":

Did it call the action group at all, or answer from the knowledge base alone? The trace panel (expand any response in the test window) shows every tool invocation — use it, don't just read the final text.
Is the PAN actually masked in the response, or did the guardrail have to intervene? You want the masking to happen at the Lambda layer (step 5) and never rely on the guardrail as the only line of defense.
Is the cited control real? Cross-check a few citations against your actual PCI DSS mapping document. If grounding is working, these should be reliably accurate; if they aren't, revisit chunk size before touching the guardrail thresholds.

Iterate on the instructions and chunking here — this loop is fast (no redeploy needed) and it's where most of the actual tuning happens.

10 Create an alias and promote a version

The agent's DRAFT state is for testing only. To integrate it into anything real, create a version (an immutable snapshot of the prepared configuration) and point an alias at it — your application code calls the alias, not the draft, so you can promote a new version without changing any calling code.

prod-v1 stays pinned to a reviewed, immutable version. Testing a new instruction wording happens against staging first.

Go to Aliases → Create alias, name it (prod-v1), and either create a new version from the current draft or point it at an existing one. Keep a staging alias around pointed at whatever draft you're actively iterating on — it lets you validate a change against the same integration path production uses, without touching prod-v1 until you're ready.

11 Codify everything in CDK and deploy

Once the console configuration behaves the way you want, capture it as infrastructure-as-code so it ships through the same pipeline as everything else in the workstream, with a clean audit trail for the security sign-off — no console drift between what was reviewed and what's running.

typescript · cdk stack

const guardrail = new bedrock.CfnGuardrail(this, 'CardsPiiGuardrail', {
  name: 'cards-pii-guardrail-v3',
  blockedInputMessaging: 'This request cannot be processed under current policy.',
  blockedOutputsMessaging: 'This response was withheld by the cards data guardrail.',
  sensitiveInformationPolicyConfig: {
    piiEntitiesConfig: [
      { type: 'CREDIT_DEBIT_CARD_NUMBER', action: 'BLOCK' },
      { type: 'CREDIT_DEBIT_CARD_CVV', action: 'BLOCK' },
    ],
  },
  contextualGroundingPolicyConfig: {
    filtersConfig: [
      { type: 'GROUNDING', threshold: 0.75 },
      { type: 'RELEVANCE', threshold: 0.70 },
    ],
  },
});

const agent = new bedrock.CfnAgent(this, 'FraudPatternAgent', {
  agentName: 'fraud-pattern-review-agent',
  foundationModel: 'anthropic.claude-3-7-sonnet-20250219-v1:0',
  instruction: fs.readFileSync('./agent-instructions.txt', 'utf-8'),
  guardrailConfiguration: {
    guardrailIdentifier: guardrail.attrGuardrailId,
    guardrailVersion: guardrail.attrVersion,
  },
  agentResourceRoleArn: agentExecutionRole.roleArn,
});

const alias = new bedrock.CfnAgentAlias(this, 'ProdAlias', {
  agentId: agent.attrAgentId,
  agentAliasName: 'prod-v1',
});

Bootstrap once per account/region if you haven't already, then deploy:

bash

cdk bootstrap aws://104829xxxxxx/eu-central-1 --profile xybank-nonprod
cdk deploy FraudPatternAgentStack --profile xybank-nonprod

Same agent, same configuration, now deployed through CDK — IAM role, Lambda action groups, guardrail, knowledge base and alias all in one stack.

The stack outputs (AgentId, AgentAliasId, KnowledgeBaseId) are what your calling application needs — capture them in your deployment pipeline's output rather than hardcoding them, since a version promotion can change the alias ID.

12 Invoke the agent from the CLI / your app

With an alias live, the agent is callable via the bedrock-agent-runtime API — from the CLI for a smoke test, or from your cards-ops console backend for the real integration.

bash · smoke test

aws bedrock-agent-runtime invoke-agent \
  --agent-id FRAUD7X2QK9 \
  --agent-alias-id TSTALIASID \
  --session-id test-001 \
  --input-text "Review txn XYB-2026-88213910" \
  --profile xybank-nonprod

For the actual application integration, the same call goes through the AWS SDK from the Lambda orchestrator behind API Gateway (see the architecture diagram below) — the CLI call above is purely for verifying the deployed alias works before wiring up the real request path.

python · orchestrator

import boto3

client = boto3.client("bedrock-agent-runtime", region_name="eu-central-1")

response = client.invoke_agent(
    agentId="FRAUD7X2QK9",
    agentAliasId="PRODALIASID",
    sessionId=session_id,
    inputText=analyst_query,
)

for event in response["completion"]:
    if "chunk" in event:
        print(event["chunk"]["bytes"].decode())

That's the full path — model access, knowledge base, IAM boundary, action groups, agent, guardrail, alias, IaC, invocation. Everything from here is tuning.

13 Reference architecture

Zooming back out, here's how all the pieces built above fit together:

The agent orchestrates through Lambda action groups; nothing writes back to the core-banking system, and PII never leaves the guardrail boundary.

API Gateway and the Lambda orchestrator in front of the agent (left side of the diagram) handle request-level auth against our existing identity-provider session before anything reaches Bedrock — that layer isn't part of the Bedrock build itself, but it's what makes "who called this agent and were they authorized" answerable, which matters as much to a security review as anything inside Bedrock does.

14 Mapping STRIDE onto a cards workflow

STRIDE was built for software threat modeling, not fraud triage, so we adapted rather than adopted it:

STRIDE category	What it looks like in a card transaction	Primary control we cite
Spoofing	Card-not-present transaction from a new device fingerprint, no prior history with the payee	PCI DSS 4.0 Req 8.4.2 — MFA for CNP
Tampering	Amount or MCC mismatch between authorization and clearing message	ISO 8583 field-level integrity checks
Repudiation	Multiple rapid authorizations with no matching customer-initiated event in the channel logs	Non-repudiation logging, regulatory circular on transaction logging
Information disclosure	Any flow where the agent itself would need to return unmasked PAN/CVV	Bedrock Guardrail sensitive-info filter (this is the one the guardrail exists to stop)
Denial of service	Velocity spikes consistent with card-testing rather than genuine usage	Rate-limiting at API Gateway + card-testing runbook
Elevation of privilege	Any request pattern that resembles probing the core-banking proxy for scopes beyond transaction read	IAM least privilege + CloudWatch anomaly alarms

The knowledge base holds one runbook section per row, which is why the chunking decision in step 3 mattered as much as it did — a chunk that blends "spoofing" and "repudiation" guidance produces exactly the kind of blended, half-right answer that erodes an analyst's trust in the tool after two or three uses.

15 What broke, and what I'd do differently

A few honest notes, since most write-ups skip this part:

The first version of the instructions let the agent answer from memory when knowledge base retrieval came back thin. Turning on contextual grounding in the guardrail (step 8) fixed this more reliably than any amount of prompt tuning — treat grounding as a guardrail-layer control, not a prompt-layer one.

Chunk size was the single highest-leverage tuning knob, above model choice, above temperature, above instruction wording. If your retrieval quality is inconsistent, look at chunking (step 3) before you look at the model.

IAM scoping (step 4) took longer than every other piece of this build combined. Getting the security team comfortable with "an LLM agent has a role that can invoke this Lambda" required the permissions boundary, a CloudTrail-backed review process, and a tabletop walkthrough of what happens if the model is prompt-injected through a transaction memo field. Budget for that conversation; it's not a formality in a cards environment.

I'd add a human-in-the-loop confirmation step before sns-notify-fn fires in production, even though it's a read-only-adjacent notification. An agent that pages someone at 3 a.m. on a false positive burns trust fast.

None of this is exotic. It's the same discipline any of us would apply to a human-facing fraud process — least privilege, grounded evidence, a clear audit trail — applied to an agent instead of a person. The interesting part isn't that Bedrock can do agentic orchestration; it's that the hard 80% of shipping it in a bank was identical to the hard 80% of shipping anything else in a bank.

Where I'd take this next

The natural extension is multi-agent collaboration — a supervisor agent that routes between this fraud-pattern reviewer and a separate compliance-documentation agent for SAR (suspicious activity report) drafting, which Bedrock now supports natively. I haven't shipped that part yet, so I'll leave it for a follow-up post once it's been through a security review rather than write about it speculatively here.

If you're building something similar on a legacy cards or core-banking stack, I'd genuinely like to compare notes — particularly on how you scoped IAM for action groups touching a system of record, and how your security team handled the guardrail sign-off. Drop a comment or find me on LinkedIn.