Ciao Amici 👋
Every hiking trail in the Alps has a map at the trailhead. A good one shows you the huts, the peaks and the river crossings. What it never shows you is the stretch where the path is a loose scree slope, or the hut that closes in October, or the signpost that points you towards the wrong valley. For that, you need someone who has walked it.
A few days ago a roadmap called "How to Master AWS for AI?" landed in my feed, drawn by Umang Chaudhary (@mlwithumang). It is a lovely trailhead map: one centre (Amazon Bedrock), six branches (Foundations, Data and Compute, Amazon Bedrock, AI Agents, Knowledge and AI Services, Ops and Governance) and a suggested five-step learning path along the bottom. I have built on several of these services myself, so I did the obvious thing. I walked the map against my own scars.
This is a long article on purpose. It follows the five steps in order, and for each one it gives you three things: what is verifiably true today (as of 30 October 2026), code you can run, and the places where I would move the signposts. Every code sample in this article was executed before I wrote it down, and none of it needs an AWS account or costs a cent. I will tell you exactly how I tested it, and exactly what I did not test, at the end.
What is in here
- The map, and my three changes to it
- Step 0: money and blast radius (budgets, token ceilings, least privilege, logging)
- Step 1: foundations (reading an AccessDenied properly)
- Step 2: models and prompts (the Converse API, token accounting, stop reasons)
- Step 3: grounding and RAG (S3 Vectors, DynamoDB vectors, and a retrieval test you can run offline)
- Step 4: agents (Strands, AgentCore, and the two rules of distributed systems that break agents)
- Step 5: evaluate and govern (guardrails at the tool boundary, evaluation, audit)
- A failure cheat sheet, a four-week plan, and what I did not verify
The map in one table
Here is the learning path from the roadmap, in my words:
| Step | The roadmap says | What it really means |
|---|---|---|
| 1 | Foundations: IAM, KMS, CDK | Know who is allowed to do what, and keep secrets secret |
| 2 | Models and Prompts: Bedrock, Nova | Call a model and shape what it says |
| 3 | Grounding and RAG: Knowledge Bases, S3 Vectors | Make the model answer from your data, not from its imagination |
| 4 | Agents: AgentCore, Strands | Let the model take actions, safely |
| 5 | Evaluate and Govern: Guardrails, Evaluations | Prove it works, and stop it doing what it should not |
It is a sound order for a first pass. My notes are about what a map cannot show, because it is a map and not a diary. Here is the same path with my three changes highlighted. I will explain each one as we go:
Step 0: money and blast radius
The roadmap lists Budgets and Cost Explorer under Core Setup, which I love, but I would pull it out of the pile and put it before everything else. AI services bill by usage, and usage is exactly what an eager experiment or a runaway loop produces. There are four layers of protection, and they do different jobs.
Layer 1: an alert (slow, but it reaches you). AWS Budgets can email you when actual spend passes a percentage of a monthly limit. Here is a budget with two alerts, at 50 and 80 percent of 10 US dollars. I did not guess the structure: I validated it against the SDK's own model for create_budget (more on that in the testing section).
{
"Budget": {
"BudgetName": "ai-experiments-monthly",
"BudgetType": "COST",
"TimeUnit": "MONTHLY",
"BudgetLimit": { "Amount": "10", "Unit": "USD" }
},
"NotificationsWithSubscribers": [
{
"Notification": {
"NotificationType": "ACTUAL",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 50,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "you@example.com" }]
},
{
"Notification": {
"NotificationType": "ACTUAL",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 80,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "you@example.com" }]
}
]
}
Create it with client.create_budget(AccountId=..., **json.load(open("budget.json"))) using boto3 and the budgets client. Be honest about what this layer is: an alert, not a brake. It reacts to billing data, which arrives with a delay, so it tells you that you have a problem rather than stopping the problem.
Layer 2: a brake inside your code. The cheapest protection is a ceiling you enforce yourself. This tiny wrapper counts the tokens from every Converse call, refuses to call again once you pass a limit, and writes a ledger line per call so you can see where the tokens went:
# code/token_budget.py
"""A hard token ceiling around Bedrock Converse, with a local usage ledger."""
import json
import time
class BudgetExceeded(Exception):
pass
class TokenBudget:
def __init__(self, max_tokens: int, ledger_path: str = "usage.jsonl"):
self.max_tokens = max_tokens
self.used = 0
self.ledger_path = ledger_path
def converse(self, client, **kwargs):
if self.used >= self.max_tokens:
raise BudgetExceeded(f"{self.used} of {self.max_tokens} tokens used")
response = client.converse(**kwargs)
usage = response["usage"]
self.used += usage["totalTokens"]
with open(self.ledger_path, "a", encoding="utf-8") as f:
f.write(json.dumps({
"ts": time.time(),
"model": kwargs.get("modelId"),
"input": usage["inputTokens"],
"output": usage["outputTokens"],
"total_used": self.used,
}) + "\n")
return response
Note that the check happens before the call, so one call can overshoot the ceiling by up to one response. That is fine for a safety net. The test that proves it stubs Bedrock locally, makes two calls of 100 tokens against a ceiling of 150, and confirms that the third call is refused and the ledger shows two lines.
Layer 3: least privilege. Calling Converse requires the bedrock:InvokeModel permission, and ConverseStream requires bedrock:InvokeModelWithResponseStream. So a role that may only call one cheap model looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "InvokeOneModelOnly",
"Effect": "Allow",
"Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
"Resource": "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-micro-v1:0"
}
]
}
An agent, a notebook or a Lambda that runs under this role cannot suddenly start calling an expensive model, because the policy does not let it.
Layer 4: know what you are logging. Bedrock model invocation logging is off by default. When you turn it on it records the full request and response bodies, token counts and the caller identity for supported calls, and it can write to S3 and to CloudWatch Logs. That is excellent for debugging and for audit, and it also means your log bucket now contains your users' prompts. Encrypt it, restrict access, and set your own retention, because the logs are kept until you delete the configuration. You can tag calls with requestMetadata in the Converse request so you can filter the logs later.
One more money fact that surprises people: cross-Region inference profiles let a request be served by compute in several Regions. AWS states there is no extra routing cost and that the price is based on the Region from which you call the profile. Geographic profiles use standard pricing and global profiles are described as approximately 10 percent cheaper. They also have a governance side: if your organisation uses Service Control Policies to restrict Regions, the destination Regions of the profile must be allowed, and global profiles need an allowance for aws:RequestedRegion set to unspecified. Do not discover this on the day you deploy.
Rule I follow: if I cannot explain where the money stops, I do not press deploy.
Step 1: foundations, and how to read an AccessDenied
IAM, KMS, Secrets Manager, VPC endpoints, the CLI and CDK. Nobody gets excited about this branch, and everybody who skips it pays for it later.
I once wrote a whole article about the fact that the same AWS error can have two different answers, and that guessing which one you are facing is how an afternoon disappears. With AI workloads you meet that early. A model call fails, an agent cannot reach a tool, a Lambda cannot read a bucket, and the message is the same polite "access denied". The skill is to stop guessing and walk a short list, in order:
The commands behind that list are boring and worth memorising:
aws sts get-caller-identity # who am I, really? (wrong role is the most common cause)
aws iam simulate-principal-policy \
--policy-source-arn <role-arn> \
--action-names bedrock:InvokeModel \
--resource-arns arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-micro-v1:0
The simulator answers the identity-policy question quickly. If it says allowed and the call still fails, the refusal is coming from somewhere else in the chain: a Service Control Policy or permissions boundary, a resource policy, a KMS key policy, or the Region and model you are actually calling. CloudTrail records the call with its error code and message, which is the receipt you want when two causes look identical.
On KMS, one design rule from the vector services is worth knowing now. When you create an S3 Vectors bucket you choose its encryption type (SSE-S3 or a KMS key) at creation time, and you cannot change it afterwards. Choosing encryption is a day-one decision, not a later hardening step.
Step 2: models and prompts
Amazon Bedrock gives you one API over many model families: Amazon Nova, Anthropic Claude, Meta, Mistral and others. For new code, the Converse API is the one to learn, because it has a consistent request and response shape across models that support messages. Here is the minimum:
import boto3
client = boto3.client("bedrock-runtime", region_name="us-east-1")
response = client.converse(
modelId="amazon.nova-micro-v1:0", # or an inference profile ID or ARN
system=[{"text": "You are a concise AWS tutor."}],
messages=[{"role": "user", "content": [{"text": "What is idempotency, in two sentences?"}]}],
inferenceConfig={"maxTokens": 300, "temperature": 0.2},
)
text = response["output"]["message"]["content"][0]["text"]
usage = response["usage"] # inputTokens, outputTokens, totalTokens
stop = response["stopReason"] # do not ignore this
Three habits are hiding in those lines.
Read the stop reason. The SDK model lists these values: end_turn, tool_use, max_tokens, stop_sequence, guardrail_intervened, content_filtered, malformed_model_output, malformed_tool_use and model_context_window_exceeded. Only the first one means "the model finished its thought". A truncated answer (max_tokens) that your code treats as complete is a bug that looks like a bad model.
Count tokens on every call. The usage block is your cost meter. Input tokens include everything you send: the system prompt, the conversation history and, later in Step 3, the retrieved passages. Output tokens are what the model writes. That is why a chatty prompt and a large retrieval depth both show up on the bill, and why the ledger from Step 0 pays for itself.
Use the knobs that exist. Beyond inferenceConfig, Converse accepts additionalModelRequestFields for model-specific parameters, requestMetadata for tagging calls in your logs, guardrailConfig for attaching a guardrail, and serviceTier, whose documented values are reserved, priority, default and flex. Prompt caching works by adding cachePoint blocks, and which blocks a model accepts depends on the model. A detail from the docs that bites people: if you call a prompt from Prompt management, you cannot also pass system, inferenceConfig, toolConfig or additionalModelRequestFields, because the managed prompt owns them.
Tool use, which Step 4 builds on, is part of the same API. You describe tools in toolConfig, the model answers with stopReason set to tool_use and a toolUse block (an id, a name and an input), you run the tool, and you send the outcome back as a toolResult for that id with a status of success or error. Every agent framework is, underneath, a loop around that exchange.
The thing I would add here is a habit, not a service. A model that sounds confident is not a model that is right. I built a small game, gaslight-bench, to find out whether a model can be talked into agreeing with false statements about AWS. The game is playful, but the question behind it is serious: if a model can be pushed off the facts by a confident voice, its own tone can never be your evidence. And the first piece of evidence you can build is tiny: five questions you know the answers to, run every time you change a prompt or a model. That is where evaluation starts, and it starts here, not in Step 5. We will turn it into a real harness in Step 3.
Step 3: grounding and RAG
If you want the model to answer from your documents, the system has two halves, and it helps to keep them separate in your head:
The ingest half runs when your documents change: split them into chunks, turn each chunk into an embedding (a list of numbers), and store the embedding with the chunk's text and some metadata. The query half runs on every question: embed the question, fetch the nearest chunks, put them in the prompt, and ask the model to answer from them. Most RAG problems are in the middle: the right chunk never comes back, so the model never sees the answer, and then it improvises.
The roadmap lists four places to keep embeddings: S3 and S3 Vectors, OpenSearch Serverless, DynamoDB vector search and Aurora pgvector. Two of those have changed a lot recently, so here is what I could verify.
Amazon S3 Vectors became generally available on 2 December 2025. AWS states up to 2 billion vectors per index and up to 10,000 indexes per vector bucket, with queries around 100 ms for frequent use and under a second for infrequent ones, and claims up to 90 percent lower total cost for uploading, storing and querying vectors. It plugs into Bedrock Knowledge Bases, and into OpenSearch for hybrid search. The SDK model tells you the practical constraints: the only data type is float32, the distance metric is cosine or euclidean, and the index name, dimension, distance metric and non-filterable metadata keys are fixed when you create the index. That last point is the one to plan around: if you change your embedding model later, the dimension may change, and you will re-embed and re-index everything into a new index.
Amazon DynamoDB vector search became generally available on 5 August 2026. You store embeddings next to your normal item attributes, define a vector index (at table creation through VectorIndexes, or later through VectorIndexUpdates), and search with the SearchVectors operation, which takes the index name, the query vector, a TopK and an optional condition expression to filter on item attributes. The distance functions are COSINE, DOT_PRODUCT and EUCLIDEAN, and AWS states a target of single-digit millisecond latency at 99 percent plus recall. One detail trips people up: DynamoDB does not create the embeddings for you. You generate them with a model of your choice (a Bedrock model works) and store them. Write-ups of the launch also say that vector indexes require on-demand capacity and cap the number of results returned per search, so read the current limits page before you design around either.
Here is the same embedding stored both ways. The calls are shaped exactly to the SDK's current models (I validated every parameter offline, see the testing section), and the embed function is the only part that spends money:
# code/vector_stores.py
"""Two ways to store the same embedding on AWS. Calls are shaped to the current SDK models.
Not run against a live account in my tests: they are validated offline (see test_vectors.py).
"""
import boto3
EMBED_MODEL = "amazon.titan-embed-text-v2:0" # check the model card for the request body you use
def embed(text: str, dims: int = 1024) -> list:
"""Real embedding call (costs a little per token). Swap in any embedding model."""
import json
bedrock = boto3.client("bedrock-runtime")
resp = bedrock.invoke_model(modelId=EMBED_MODEL, body=json.dumps({"inputText": text, "dimensions": dims, "normalize": True}))
return json.loads(resp["body"].read())["embedding"]
# ---- S3 Vectors ----------------------------------------------------------------
def s3_vectors_setup(s3v, bucket: str, index: str, dims: int = 1024):
s3v.create_vector_bucket(vectorBucketName=bucket)
s3v.create_index(
vectorBucketName=bucket, indexName=index,
dataType="float32", dimension=dims, distanceMetric="cosine",
# Big text you only want back, never filter on: mark it non-filterable (immutable later).
metadataConfiguration={"nonFilterableMetadataKeys": ["source_text"]},
)
def s3_vectors_put(s3v, bucket, index, items):
"""items: [(key, embedding, {"source_text": ..., "topic": ...}), ...]"""
s3v.put_vectors(
vectorBucketName=bucket, indexName=index,
vectors=[{"key": k, "data": {"float32": e}, "metadata": md} for k, e, md in items],
)
def s3_vectors_query(s3v, bucket, index, query_embedding, top_k=3, topic=None):
kw = {"filter": {"topic": topic}} if topic else {}
resp = s3v.query_vectors(
vectorBucketName=bucket, indexName=index, topK=top_k,
queryVector={"float32": query_embedding}, returnMetadata=True, returnDistance=True, **kw,
)
return resp["vectors"] # each has key, distance, metadata
# ---- DynamoDB vector search ----------------------------------------------------
def ddb_create_table_with_vector_index(ddb, table: str, dims: int = 1024):
ddb.create_table(
TableName=table,
BillingMode="PAY_PER_REQUEST", # vector indexes need on-demand capacity (per AWS write-ups; check docs)
AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}],
KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}],
VectorIndexes=[{
"IndexName": "by-embedding",
"VectorAttribute": {"AttributeName": "embedding"},
"Dimensions": dims,
"DistanceFunction": "COSINE",
"Projection": {"ProjectionType": "ALL"},
}],
)
def ddb_search(ddb, table: str, query_embedding, top_k=3):
resp = ddb.search_vectors(
TableName=table, IndexName="by-embedding", TopK=top_k,
SearchVector=[{"N": repr(float(x))} for x in query_embedding],
)
return resp["SearchResults"]
How do I choose? Check the current documentation for limits and pricing, then pick the boring option that matches the shape of your data:
| If your situation is... | I would start with |
|---|---|
| Lots of documents, queries are occasional, cost matters most | S3 Vectors |
| Your app already lives in DynamoDB and the vectors belong to the same items (agent memory, product similarity) | DynamoDB vector search |
| You need rich search: hybrid keyword plus vector, ranking, analytics | OpenSearch |
| Your data is relational and you already run Postgres | Aurora with pgvector |
The test that matters more than the database
Choosing a store is the easy decision. The hard question is "does the right chunk come back?", and you can answer it with a page of Python, offline, before you spend anything. A retrieval test needs a golden set (questions with the chunk that should answer each one) and two numbers: recall@k (how often the right chunk is in the top k) and MRR (how high it ranks, on average).
Here is the whole harness. The embedding in it is a deliberately simple hashed bag of words, a stand-in so the harness runs anywhere; swap embed() for a real model and keep everything else:
# code/retrieval_eval.py
"""A retrieval test you can run with no AWS account: recall@k and MRR over a golden set.
The embedding here is a hashed bag of words, a stand-in so the harness runs anywhere.
Swap embed() for a real model call (see the Bedrock version in the article) and keep the rest.
"""
import re
import zlib
import numpy as np
CHUNKS = {
"c1": "S3 event notifications are delivered at least once so handlers must tolerate duplicates",
"c2": "Lambda retries failed asynchronous invocations up to two more times by default",
"c3": "The sequencer field in an S3 event orders events for the same object key",
"c4": "Bedrock Guardrails check model input and model output but not tool parameters by default",
"c5": "AgentCore Gateway turns APIs Lambda functions and MCP servers into tools for agents",
"c6": "AgentCore Memory keeps short term session events and long term extracted records",
"c7": "S3 Vectors stores embeddings in vector buckets and queries them with cosine or euclidean distance",
"c8": "DynamoDB vector search stores embeddings next to item attributes and you generate them yourself",
"c9": "Model invocation logging is off by default and records prompts and responses when enabled",
"c10": "AWS Budgets can alert you when actual spend passes a percentage of your monthly limit",
"c11": "Cross Region inference profiles route requests to compute in several Regions",
"c12": "Strands hooks run before a tool call after a tool call and before a model invocation",
}
GOLDEN = [ # (question, id of the chunk that should come back)
("how many times does lambda retry an async invocation", "c2"),
("can an s3 notification arrive twice", "c1"),
("how do i order s3 events for one key", "c3"),
("do guardrails inspect tool parameters", "c4"),
("which agentcore piece exposes lambda functions as tools", "c5"),
("where do agents keep long term memory", "c6"),
("which distance metrics does s3 vectors support", "c7"),
("who creates the embeddings for dynamodb vector search", "c8"),
("is bedrock prompt logging enabled by default", "c9"),
("how do i get an alert at 80 percent of my monthly spend", "c10"),
("what runs just before a tool call in strands", "c12"),
("send my traffic to another region automatically", "c11"),
("stop paying for something i forgot to turn off", "c10"), # real paraphrase: no shared words, so a lexical embedder should miss
]
STOP = {"a", "an", "the", "of", "in", "to", "is", "are", "do", "does", "how", "i", "my", "for", "and", "or", "on", "at", "it", "by", "up", "you", "can", "which", "what", "where", "who", "than", "so", "but", "not", "when"}
DIM = 512
def embed(text: str) -> np.ndarray:
v = np.zeros(DIM)
for tok in re.findall(r"[a-z0-9]+", text.lower()):
if tok not in STOP:
v[zlib.crc32(tok.encode()) % DIM] += 1.0
n = np.linalg.norm(v)
return v / n if n else v
def rank(question: str, ids, matrix):
scores = matrix @ embed(question)
return [ids[i] for i in np.argsort(-scores)]
def evaluate(k_values=(1, 3)):
ids = list(CHUNKS)
matrix = np.stack([embed(CHUNKS[i]) for i in ids])
hits = {k: 0 for k in k_values}
rr, misses = 0.0, []
for q, gold in GOLDEN:
order = rank(q, ids, matrix)
pos = order.index(gold) + 1
rr += 1 / pos
for k in k_values:
hits[k] += pos <= k
if pos > 3:
misses.append((q, gold, order[:3]))
n = len(GOLDEN)
return {f"recall@{k}": hits[k] / n for k in k_values} | {"mrr": rr / n}, misses
if __name__ == "__main__":
metrics, misses = evaluate()
for name, value in metrics.items():
print(f"{name:10s} {value:.2f}")
for q, gold, got in misses:
print(f"MISS {q!r} wanted {gold} got {got}")
Running it prints this:
recall@1 0.85
recall@3 0.92
mrr 0.89
MISS 'stop paying for something i forgot to turn off' wanted c10 got ['c9', 'c1', 'c3']
Look at the miss, because the miss is the point. The question "stop paying for something I forgot to turn off" is about AWS Budgets, but it shares no words with the chunk about budgets, so a lexical embedder cannot find it. A real embedding model closes much of that gap, and hybrid search (keyword plus vector) closes more, but you will not know how much until you measure. Without the golden set you would have shipped a system that sounds right and silently misses a whole class of questions. With it, the same twelve lines of output tell you whether a new chunk size, a new embedding model or a new filter made things better or worse. Treat it like a unit test for retrieval, and put it in CI.
Two practical notes. First, chunk size is a trade: small chunks are precise but lose context, large chunks carry context but dilute the match and cost more input tokens when you stuff them in the prompt. Test two or three sizes against your golden set instead of copying someone's default. Second, use metadata filters (S3 Vectors takes a filter on query, DynamoDB takes a condition expression) to narrow by tenant, language or document type before similarity does its work. It is cheaper and more accurate than hoping the nearest neighbour is also the right tenant.
Step 4: agents, and where reality arrives
This is the branch I know best, because I lived it. The Amazon Bedrock AgentCore family went generally available in October 2025 and covers the pieces you would otherwise build yourself:
Runtime hosts your agent code in a serverless environment with each session isolated. Memory keeps short-term session events and long-term extracted records. Gateway turns APIs, Lambda functions and MCP servers into tools behind one endpoint. Identity gives agents workload identities and a token vault for OAuth flows. Observability sends metrics, traces and logs to CloudWatch with OpenTelemetry export. Code Interpreter and Browser run sandboxed code and managed browser sessions. Policy checks each tool call at the Gateway against Cedar rules. Around it sit the builders the roadmap lists: the Strands Agents SDK, Kiro and Amazon Quick.
An agent in a dozen lines, and what the framework is doing
With the Strands Agents SDK, a first agent is genuinely small. And because Strands has a provider for Ollama (pip install 'strands-agents[ollama]'), you can build and test the whole loop on a local model with no AWS bill at all:
from strands import Agent, tool
from strands.models.ollama import OllamaModel
@tool
def get_invoice_total(invoice_id: str) -> str:
"""Return the total for an invoice."""
return {"INV-1": "42.00", "INV-2": "17.50"}.get(invoice_id, "not found")
local = OllamaModel(host="http://localhost:11434", model_id="llama3.1") # ollama pull llama3.1 first
agent = Agent(model=local, tools=[get_invoice_total])
agent("What is the total of invoice INV-2?")
Behind that one call is the loop from Step 2: the model asks for get_invoice_total, the framework runs it, feeds the result back, and the model writes the answer. When the model is good enough, you change one line to use a Bedrock model.
Hosting it: the AgentCore CLI changed under me
One thing I would have been bitten by if I had copied an older tutorial: the Python starter toolkit CLI for AgentCore (the agentcore configure and agentcore launch flow) now prints a notice that it is no longer supported and points to a new AgentCore CLI that you install from npm (npm install -g @aws/agentcore). I installed the new one (version 1.0.1) and read its help. It scaffolds a project from templates (for example agent-python-strands, a minimal Python template, LangChain, TypeScript and MCP server templates), runs agents locally with agentcore dev, and ships with deploy, invoke, status, logs and traces commands, plus an eval command group that we will meet in Step 5. The workflow is therefore:
npm install -g @aws/agentcore
agentcore create --name robot-demo --template agent-python-strands
agentcore dev # runs the agent locally, with local traces
# agentcore deploy # this one builds and deploys to AWS: it is where the bill starts
Notice where the money starts. Everything up to dev is local. deploy creates real resources in your account, so keep it behind the budget and the least-privilege role from Step 0, and delete what you deploy when the experiment is over.
If you want to see the runtime contract for yourself without deploying, the SDK's app class is small. This entrypoint, tested through a local test client, answers POST /invocations with your return value and GET /ping with a health status:
from bedrock_agentcore.runtime import BedrockAgentCoreApp
app = BedrockAgentCoreApp()
@app.entrypoint
def invoke(payload):
prompt = payload.get("prompt", "Hello")
return {"result": f"echo: {prompt}"} # replace with: agent(prompt)
if __name__ == "__main__":
app.run()
When I posted {"prompt": "ping"} to /invocations I got {"result": "echo: ping"} with a 200, and /ping answered {"status":"Healthy", ...}. That is the whole contract your agent code has to honour.
The two rules that break agents
An agent is a distributed system with a language model in the middle, and distributed systems have rules that do not care how clever your prompt is. Two of them cost me the most, and the roadmap does not mention either.
Rule 1: triggers deliver at least once. S3 event notifications are designed to be delivered at least once, and Lambda retries failed asynchronous invocations up to two more times by default. So if your tool charges a customer, writes to a ledger or sends an email, it will eventually do it twice unless you made it safe to repeat.
The fix is an old bookkeeping idea: write down a key before you act, and check it first. A naive version (insert the key, then act) has a hole that readers pointed out after my first idempotency article: what happens when the first worker crashes after claiming the key but before finishing? The key says "in progress" forever, and the work never completes. The robust version gives each claim a lease and a fencing check. Here it is as one small module on top of a single DynamoDB table:
# code/idempotency.py
"""Idempotent execution with a lease, backed by one DynamoDB table.
Table: partition key "pk" (string). Enable TTL on "expires_at" to clean up.
States: INPROGRESS (has an owner token and a lease) and DONE (has a result).
"""
import time
import uuid
from botocore.exceptions import ClientError
class InProgress(Exception):
"""Another worker holds a live lease on this key. Let the retry come back later."""
class LostLease(Exception):
"""Our lease expired and someone else took over. Do not record our result."""
def _failed_condition(err: ClientError) -> bool:
return err.response["Error"]["Code"] == "ConditionalCheckFailedException"
class IdempotencyStore:
def __init__(self, table, lease_seconds=60, retention_seconds=86400, clock=time.time):
self.table = table
self.lease = lease_seconds
self.retention = retention_seconds
self.clock = clock
def claim(self, key):
"""Return ("run", token) if we may do the work, or ("done", result) if it is already done."""
now = int(self.clock())
token = uuid.uuid4().hex
try:
self.table.put_item(
Item={
"pk": key,
"status": "INPROGRESS",
"owner": token,
"lease_until": now + self.lease,
"expires_at": now + self.retention,
},
# New key, or an abandoned claim whose lease has run out.
ConditionExpression="attribute_not_exists(pk) OR (#s = :inprog AND lease_until < :now)",
ExpressionAttributeNames={"#s": "status"},
ExpressionAttributeValues={":inprog": "INPROGRESS", ":now": now},
)
return "run", token
except ClientError as err:
if not _failed_condition(err):
raise
item = self.table.get_item(Key={"pk": key}, ConsistentRead=True).get("Item")
if item and item["status"] == "DONE":
return "done", item.get("result")
raise InProgress(key)
def complete(self, key, token, result):
"""Record the result, but only if we still own the claim (the fencing check)."""
try:
self.table.update_item(
Key={"pk": key},
UpdateExpression="SET #s = :done, #r = :r",
ConditionExpression="#o = :t AND #s = :inprog",
ExpressionAttributeNames={"#s": "status", "#r": "result", "#o": "owner"},
ExpressionAttributeValues={":done": "DONE", ":inprog": "INPROGRESS", ":t": token, ":r": result},
)
except ClientError as err:
if _failed_condition(err):
raise LostLease(key) from err
raise
def release(self, key, token):
"""Give the key back after a failure so the next retry can run immediately."""
try:
self.table.delete_item(
Key={"pk": key},
ConditionExpression="#o = :t",
ExpressionAttributeNames={"#o": "owner"},
ExpressionAttributeValues={":t": token},
)
except ClientError as err:
if not _failed_condition(err):
raise
def run_once(store: IdempotencyStore, key: str, fn):
"""Run fn() at most once per key. fn must return a string (store JSON if you need more)."""
state, value = store.claim(key)
if state == "done":
return value
token = value
try:
result = fn()
except Exception:
store.release(key, token)
raise
store.complete(key, token, result)
return result
Walk through the four situations, because all four are tested. (1) The same key twice runs the side effect once, and the second call returns the stored result. (2) A second worker arriving while the first holds a live lease gets InProgress, so it backs off and the retry comes later. (3) If the first worker stalls past its lease, a second worker can take over, and when the stalled worker finally wakes up and tries to record its result, the conditional write refuses it with LostLease. That refusal is the fencing check. (4) If the work fails, the key is released, so the retry can run immediately instead of waiting for the lease to expire.
Now the honest limit, which I would rather you hear from me than discover in production. The fencing check protects your record. It does not magically undo a side effect that the stalled worker already performed in the outside world. If the stalled worker already charged the card, the new worker may charge it again. The only complete answer is to pass the idempotency key to the system that performs the side effect (a payment provider's idempotency key, for example), so the duplicate is rejected there too. Think of the lease as protection for your own ledger, and the downstream key as protection for the world.
Rule 2: events do not arrive in order. Amazon S3's own documentation says event notifications are not guaranteed to arrive in the order the events occurred. For object creation and deletion events it gives you a tool: the sequencer field, a hexadecimal string that you can compare for events on the same object key (and only the same key). The documented comparison has a trap: the strings can differ in length, so you left-pad the shorter one with zeros and then compare lexicographically. Compare them raw and you get a wrong answer: as text, 10 sorts before F, although in hex 10 (16) is later than F (15). Here is a guard that only applies an event if it is newer than the one you stored, using a conditional write so it is safe even when two events race:
# code/ordering.py
"""Drop stale events: only apply an S3 event if its sequencer is newer than the one we stored.
AWS guidance: the sequencer is hex, only comparable for the same object key, and values can
differ in length, so left-pad the shorter one with zeros and compare as strings.
"""
from botocore.exceptions import ClientError
WIDTH = 64 # comfortably longer than real sequencers
def normalise(sequencer: str) -> str:
return sequencer.upper().zfill(WIDTH)
def is_later(a: str, b: str) -> bool:
return normalise(a) > normalise(b)
def apply_if_newer(table, object_key: str, sequencer: str, payload: str) -> bool:
"""Write payload only when this event is newer. Returns False for a stale event."""
seq = normalise(sequencer)
try:
table.update_item(
Key={"pk": object_key},
UpdateExpression="SET seq = :s, payload = :p",
ConditionExpression="attribute_not_exists(seq) OR seq < :s",
ExpressionAttributeValues={":s": seq, ":p": payload},
)
return True
except ClientError as err:
if err.response["Error"]["Code"] == "ConditionalCheckFailedException":
return False
raise
The test applies an event, then a newer one, then a stale one that arrives late, and confirms that the stale one is dropped and the stored payload is still the newest. It also asserts the 10 versus F trap, so the padding cannot be removed by accident.
If I could add one signpost to Step 4 it would read: before you give an agent tools, make the tools safe to repeat and safe to reorder.
Step 5: evaluate and govern
The last step has two halves, and they are different jobs.
Governing: guardrails stop at the model call
Bedrock Guardrails check what goes into a model and what comes out of it. An important nuance, which AWS spells out in its own security guidance: that boundary is the model call. Tool parameters, external data and traffic between agents sit outside it.
The Strands Agents SDK lets you extend the check to those moments with hooks. In the version I installed (strands-agents 1.62.0), a BeforeToolCallEvent carries the tool_use (name and input) and a cancel_tool field: set it to a string and the tool is skipped and the string goes back to the model. An AfterToolCallEvent carries the result, which you may replace. And the ApplyGuardrail operation lets you run a guardrail on any text yourself, with source set to INPUT or OUTPUT, returning an action of NONE or GUARDRAIL_INTERVENED. Put those together and you get a guardrail on tools:
# code/guardrail_hook.py
"""Extend Bedrock Guardrails to tool calls in a Strands agent."""
import json
from strands.hooks import AfterToolCallEvent, BeforeToolCallEvent, HookProvider, HookRegistry
class GuardrailHook(HookProvider):
def __init__(self, bedrock_runtime, guardrail_id: str, guardrail_version: str = "DRAFT"):
self.client = bedrock_runtime
self.gid = guardrail_id
self.gver = guardrail_version
def register_hooks(self, registry: HookRegistry) -> None:
registry.add_callback(BeforeToolCallEvent, self.before_tool)
registry.add_callback(AfterToolCallEvent, self.after_tool)
def _blocked(self, source: str, text: str) -> bool:
resp = self.client.apply_guardrail(
guardrailIdentifier=self.gid,
guardrailVersion=self.gver,
source=source, # "INPUT" or "OUTPUT"
content=[{"text": {"text": text}}],
)
return resp["action"] == "GUARDRAIL_INTERVENED"
def before_tool(self, event: BeforeToolCallEvent) -> None:
if self._blocked("INPUT", json.dumps(event.tool_use["input"])):
# Setting cancel_tool skips the tool; the string goes back to the model.
event.cancel_tool = "Blocked by guardrail. Do not retry this call."
def after_tool(self, event: AfterToolCallEvent) -> None:
text = " ".join(b.get("text", "") for b in event.result.get("content", []))
if text and self._blocked("OUTPUT", text):
event.result = {
"toolUseId": event.result["toolUseId"],
"status": "error",
"content": [{"text": "Tool result withheld by guardrail."}],
}
# Usage:
# from strands import Agent
# agent = Agent(tools=[...], hooks=[GuardrailHook(boto3.client("bedrock-runtime"), "gr-id", "1")])
The test registers the hook with a Strands hook registry, then feeds it fake events: a tool input containing a sensitive marker gets cancelled, a clean one passes, and a tool result containing the marker is replaced by an error result. It uses a stub in place of Bedrock, so it proves the wiring, not the guardrail's judgement. The judgement, what your guardrail actually blocks, is the part you must test with your own data.
Three other pieces belong in the governing kit. AgentCore Policy checks each tool call at the Gateway against Cedar rules, which is the right tool when a rule must be a hard rule rather than a hopeful instruction in a prompt. Automated Reasoning checks, part of Guardrails, check model answers against policies you define, which suits rule-heavy domains. And for the paper trail, CloudTrail records who called what, while the invocation logging from Step 0 records what was said.
Evaluating: evidence, not vibes
Bedrock Evaluations and AgentCore Observability are on the roadmap. AgentCore Evaluations was announced in preview in December 2025, and the new AgentCore CLI I installed today has an eval command group with subcommands for evaluators, online evaluation configs, datasets, batch evaluations, on-demand evaluation of existing sessions, and A/B tests. Preview and general availability change, so check the current status before you plan around it. The shape of the idea is stable, though, and you can practise it for free:
A golden set of cases, a runner that executes them, graders that score each result, and a gate that fails the build when a score drops. You have already seen the smallest version: the retrieval harness in Step 3. The tests in this article are the same idea applied to reliability: the idempotency test replays the same event twice, the ordering test delivers events backwards, and the budget test tries to overspend. For agents, add a red-team list of nasty inputs. A good one covers instructions hidden in tool output, requests to reveal the system prompt, malformed arguments and attempts to talk the agent into skipping a check. I once spent a whole article attacking my own agent with chaos tests and injected instructions, and the lesson was simple: attack your work before someone else does, and do not wait for the last step of a learning path to start.
A failure cheat sheet
| Symptom | Likely cause | First thing to check |
|---|---|---|
| Customer charged twice, logs spotless | At-least-once delivery or an async retry | An idempotency key checked before the side effect |
| Ledger shows an old value after a newer one | Events arrived out of order | Sequencer compare, conditional write |
| Work never finishes after a crash | A claim with no lease | Lease expiry on the claim |
| Answer sounds right but cites nothing real | The right chunk was never retrieved | recall@k on a golden set |
| Answer cut off mid-sentence |
max_tokens stop reason treated as complete |
stopReason on every call |
| Bill higher than expected | Large history or retrieval depth, a loop | The token ledger, then usage per call |
| AccessDenied that policy simulator says is allowed | SCP, resource policy, KMS, Region or model | CloudTrail error message and the Step 1 list |
| Tool does something it should not | Guardrails stop at the model call | A hook on tool calls, or Policy at the Gateway |
Where I would move the signposts
Here is my honest list of changes to the map. They are small, and they are all about order.
- Start evaluating in Step 2, not Step 5. The day you call a model, write down five questions you know the answer to and check them. That tiny habit grows into an evaluation suite. Waiting until the end means evaluating a finished building.
- Put cost controls at step zero. A budget, a token ceiling and a one-model role are cheaper than any lesson.
- Add a reliability step between 3 and 4. Idempotency, leases, retries and ordering. It is not glamorous and it is where production breaks.
- Treat the map as a menu, not a checklist. Textract, Polly, Comprehend and Rekognition are excellent, and you do not need them to learn AI on AWS. Learn them when a real problem asks for them.
A four week plan that respects your wallet
If I were starting again, this is how I would spend four weeks:
| Week | Focus | The small proof you build |
|---|---|---|
| 1 | IAM, budgets, one model call | A budget alert, and a script that calls one Bedrock model through the token ceiling |
| 2 | Retrieval | A tiny knowledge base over twenty of your own documents, and five questions you can grade with recall@k |
| 3 | One agent with two tools | Tools that are safe to repeat and reorder, and a test that replays the same event twice |
| 4 | Guardrails and evaluation | A guardrail on tool calls, plus a red-team list of ten nasty inputs wired into the build |
Notice what the table does not contain: a big deployment. Each week ends with something small, local where possible, and provable. If you want a "done when" for each week, it is this: week 1 is done when the alert exists and the ceiling stops a loop; week 2 when a miss in your golden set makes you change something; week 3 when the replay test passes; week 4 when a red-team input fails the build until you fix it.
How I tested this, and what I did not
I would rather tell you the limits of my testing than let you assume them.
What I ran. I installed boto3 1.43.122, moto 5.2.5 (a local stand-in for DynamoDB), numpy, strands-agents 1.62.0, bedrock-agentcore 1.25.2 and the new AgentCore CLI 1.0.1, and ran: the idempotency lease (four situations), the ordering guard, the token ceiling (against a stubbed Bedrock client), the budget JSON (validated through the SDK's request validation), the retrieval harness, the guardrail hook (against a stubbed Bedrock client and a real Strands hook registry), the vector store calls (parameter validation against the SDK's models for S3 Vectors and DynamoDB), and the AgentCore app routes (through a local test client). They all pass, and they are in the companion folder with a README and a single command to run them.
What I did not run. None of this touched a real AWS account, because I do not spend money on experiments I can prove locally. So I have not measured real Bedrock latency or cost, I have not run embed() against a real embedding model (check the model card for the exact request body of the model you pick), I have not exercised a real guardrail's judgement, and I have not deployed anything with agentcore deploy. Facts about services (limits, dates, availability) come from AWS announcements and documentation, and a few details (the on-demand capacity requirement and result cap for DynamoDB vector search) come from third-party write-ups of AWS's launch, so I flagged them where they appear. Services change: verify limits and pricing against the current documentation before you build on them.
Closing thoughts
The roadmap is a very good map. Maps are made by people who stand back far enough to see the whole mountain, and that is a gift. What I tried to add here is the view from the path: the scree, the closed hut, the sign that points the wrong way, and a few tools you can pick up for free. Use the map to know where you are going, and use your own mistakes (cheaply, on a small budget) to learn how to get there.
If you want to see the terrain I was talking about, here are the stories behind the field notes:
- Moving the Robot Accountant to AgentCore: Dev.to | AWS Builder Center
- The IAM error that has two answers: AWS Builder Center
- Out-of-order events and AI agents: AWS Builder Center
- Breaking my own agent on purpose: AWS Builder Center
- The ledger that charged twice: Dev.to | AWS Builder Center
- Can you gaslight an AI about AWS?: Dev.to | GitHub
All the code from this article, with its tests, is in the companion repository: https://github.com/mursalfk/aws-for-ai-field-guide.
What is the one step of this path you wish somebody had warned you about?
Ciao! 👋
References
- Amazon S3 Vectors generally available: https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-s3-vectors-generally-available/
- Amazon DynamoDB vector search generally available: https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-dynamodb-vector-search/
- AgentCore Policy and Evaluations (preview announcement): https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-bedrock-agentcore-policy-evaluations-preview
- Extending Bedrock Guardrails to tool interactions with Strands: https://aws.amazon.com/blogs/security/extend-amazon-bedrock-guardrails-to-tool-interactions-using-the-strands-agents-sdk/
- Automated Reasoning checks in Bedrock Guardrails: https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-automated-reasoning-checks.html
- Converse API and its permissions: https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html
- Cross-Region inference and inference profiles: https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html
- Model invocation logging: https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html
- Amazon S3 event message structure (the sequencer field): https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html
- Strands Agents hooks: https://strandsagents.com/latest/documentation/docs/user-guide/concepts/agents/hooks/
- Strands Ollama model provider: https://strandsagents.com/latest/documentation/docs/user-guide/concepts/model-providers/ollama/
Top comments (0)