DEV Community: AWS Community Builders

[AWS] Should you return results or answers? The difference between Web Search and Web Answer [AgentCore]

Nao San — Sun, 08 Nov 2026 06:49:55 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

【AWS】結果を返すか、答えを返すか。Web SearchとWeb Answerの使い分け【AgentCore】 #AIエージェント - Qiita

はじめに AIエージェントに最新のウェブ情報を渡したい場面は多いです。これまでは、検索APIを呼び、ページを取得し、要約して、出典を付ける、というパイプラインを自分で組む必要がありました。 Amazon Bedrock AgentCoreには、この部分をマネージドで提供す...

qiita.com

Introduction

There are many situations where you want to give an AI agent up-to-date web information. Until now, you had to build the pipeline yourself: call a search API, fetch pages, summarize them, and attach sources.

Amazon Bedrock AgentCore offers two managed features for this part.

Web Search is a tool that returns ranked search results. It was introduced in June 2026
Web Answer is a tool that returns a finished answer with citations. It became generally available (GA) on November 5, 2026

Both are added as connectors on AgentCore Gateway and called via MCP. AWS describes them as "complementary", but when you actually have to choose, questions like these come up.

If the only difference is what comes back, can't I do the same things with either one?
If I want to control which domains are referenced, how far can each tool go?
Which one can I use in the Tokyo Region? (As of 2026-11-08. Supported Regions may increase.)

This article organizes the specifications from the official documentation in tables, adds the results I confirmed on a real environment, and summarizes criteria for choosing between them.

So that readers who are new to Web Answer can follow along, the first half compares the two in tables, and the second half shows the verification results.

Key differences between Web Search and Web Answer

Here are the key points as a table first. The "Confirmed on a real environment" column shows what I confirmed in us-east-1 on 2026-11-08.

Criterion	Web Search	Web Answer	Confirmed on a real environment
What it returns	Ranked results (text, url, title, publishedDate)	A synthesized answer with citations	Confirmed (Verification 1). Search has no `structuredContent`, and `publishedDate` is in a custom format or `unknown`. Answer's citation markers map to the order of first appearance after removing duplicates
Who composes the answer	The model of your own agent	The Web Answer side	Confirmed (Verification 1). Web Search returned only results (snippets), and Web Answer returned an answer with citations
Price (tool side)	$7 / 1,000 queries	$10 / 1,000 invocations	Official documentation (see below)
Regions (as of 2026-11-08)	7 Regions including Tokyo	us-east-1 only (the blog says "more AWS Regions coming soon")	Official documentation
Domain control	Two layers: administrator and request (100 entries each)	Two layers: administrator and request (50 entries each)	Confirmed (Verification 3). Both kept to the administrator's boundary, but violations look different (Answer returns an error; Search narrows the results or returns an empty result)

The choice is almost entirely decided by two questions: "Do you want to handle the results with your own model, or do you want a finished answer?" and "Do you need to keep everything in Tokyo?" For domain control, both could be fixed on the administrator side, but they behave differently when a request goes outside the boundary. Note that Web Search returns an empty result without an error.

Overview

First, here is the architecture as a diagram. The calling agent goes through the Gateway and calls two targets (connectors). The web index is operated by Amazon and sits outside your own account. The answer from Web Answer is written by a model on the Amazon side, and you cannot choose that model. The results from Web Search are turned into an answer by your own model, which you choose on the calling side.

Specification comparison by aspect

No.	Aspect	Web Search	Web Answer
1	What it returns	Ranked results	A synthesized answer with citations
2	connectorId	`web-search`	`web-answer`
3	Query length	Up to 200 characters	1 to 30,000 characters
4	Result count control	`maxResults` 1 to 25 (default 10)	None
5	Domain lists	100 each (request side)	50 each (100 for exclude after merging)
6	Date filter	Yes (1.2.0 or later)	Yes (1.1.0 or later)
7	Location	None	`userLocation.country`
8	Regions (as of 2026-11-08)	7 Regions (including Tokyo)	us-east-1 only
9	Price	$7 / 1,000 queries	$10 / 1,000 invocations
10	IAM permission on the service role (independent per tool)	`InvokeWebSearch`	`InvokeWebAnswer`

Detailed explanation of each aspect

Aspect 1: What each tool returns

AWS positions Web Search as follows.

"Web Search Tool on Amazon Bedrock AgentCore is different — it is backed by a web index that Amazon builds and operates directly."
—— Web Search Tool (AgentCore Developer Guide)

Web Answer is described as searching the web, reading pages, and answering with citations.

"searches the web, reads relevant pages to ground its answer, and includes citations."
—— Web Answer (AgentCore Developer Guide)

Here is the difference in response format, field by field.

Web Search (content[0].text is a JSON string)

{
  "id": "824f89d0",
  "results": [
    {
      "text": "...",
      "publishedDate": "2024-10-07",
      "url": "...",
      "title": "..."
    }
  ]
}

Web Answer (content[0].text is Markdown, and structuredContent holds the list of citations)

{
  "isError": false,
  "content": [{"type": "text", "text": "... [[1](https://example.com/basel-iii)]..."}],
  "structuredContent": {
    "answer": [
      {
        "text": "...",
        "citations": [{"domain": "example.com", "url": "https://example.com/basel-iii"}]
      }
    ]
  },
  "_meta": {"latencyMs": 5210}
}

Both of the above are examples from the official documentation. When I checked on a real environment (us-east-1, 2026-11-08, connectors Search 1.3.0 / Answer 1.1.0), I found the following differences (details in Verification 1).

Web Search results have no structuredContent; only the JSON string in content[0].text is returned
Web Search's publishedDate is not ISO-8601. It is a string such as 05:00PM, Wednesday, September 09 2026, PDT, or unknown
Web Answer's citation marker [[N]] did not correspond to citations[N-1]. It corresponded to the Nth URL in order of first appearance after removing duplicates (looking up by index mismatched in 7 of 9 runs, while the deduplicated order matched in all 9)

Aspects 2 to 4: Differences from the caller's point of view

Web Search is meant to be given short search keywords (up to 200 characters), and maxResults lets you limit the number of results
Web Answer accepts a longer natural-language question (up to 30,000 characters). There is no way to specify the number of results
Tool names have the form <target name>___<tool name> (three underscores). On the real environment they were web-answer-tool___WebAnswer and web-search-tool___WebSearch. On a Gateway with semantic search enabled, tools/list can also include another tool (x_amz_bedrock_agentcore_search), so you need to pick by the end of the name (the Gateway I used for verification had semantic search disabled, so it was not included)
tools/list and tools/call succeeded when I sent SigV4-signed HTTP requests without initialize. Web Search's tools/list includes title and annotations (readOnlyHint: true, openWorldHint: true) (connector 1.3.0)

Aspects 5 and 6: Domain and date filters

Both have two layers: a boundary that the administrator sets on the target, and conditions that the agent specifies on each call.

"Request-level filters cannot override the target-level exclude list or expand results beyond the target-level include list."
—— Web Search Tool (AgentCore Developer Guide)

Rule	Web Search	Web Answer
Request-side list limit	100 each	50 each
Handling of exclude	The administrator's exclude and the request's exclude are merged, and domains in the merged list are excluded	Same as left. Up to 100 entries after merging
When both sides have include	Only the intersection is used. An empty intersection is an error	Officially "only the intersection is used; an empty intersection is an error". In my tests, even with a non-empty intersection, it returned an error if even one non-permitted domain was included
Date specification	ISO-8601, UTC, both ends inclusive	Date only, or date-time (`Z` or `+HH:mm` allowed)
Required connector version	Filters from 1.2.0	Filters from 1.1.0

In practice, if a regulated-industry customer wants the agent to reference only trusted domains, you can pin that with the administrator-side include. In Verification 3, I confirmed that domains outside the boundary specified from the request side, and domains that hit the administrator's exclude when included on the request side, did not appear in the results (one observation per pattern).

Aspect 8: Regions and data handling

Item	Web Search	Web Answer
Supported Regions (as of 2026-11-08)	us-east-1, us-east-2, us-west-2, eu-west-1, ap-southeast-2, ap-northeast-1 (Tokyo), us-gov-west-1	us-east-1 only
Scope of processing	Amazon-operated web index	Amazon-operated web index and knowledge graph. Answers are generated by a model on Bedrock using geographic cross-Region inference

The supported Regions above are as of 2026-11-08. "us-east-1 only" for Web Answer describes the current situation, and the blog says "more AWS Regions coming soon". Because the supported Regions may increase, check the latest Region table before use. The verification in this article was done in us-east-1, which Web Answer supports.

If you want to keep everything within the Tokyo Region, Web Search is the only choice at present (2026-11-08). If you use Web Answer in us-east-1, you need to check the customer's requirements, because data processing involves a US Region.

"Zero data egress" means that queries and results do not leave the AWS environment. The wording differs between the blog (queries and results stay within the AWS environment) and the product page (they can also leave via a VPC or PrivateLink). I did not verify which wording matches the actual behavior. If the data exit point matters for a customer's requirements, check the official documentation or ask AWS.

Aspect 9: Price

Item	Web Search	Web Answer
Tool unit price	$7.00 / 1,000 queries ($0.007 per call)	$10 / 1,000 invocations ($0.010 per call)
Gateway API	$0.005 / 1,000 calls	Same as left (because calls go through the Gateway)

Gateway API charges depend on the number of ListTools, InvokeTool, and Ping calls. The unit prices are listed on the Amazon Bedrock AgentCore pricing page.

Aspect 10: Separation of IAM responsibilities

Target	Permission
Gateway service role (Web Search)	`bedrock-agentcore:InvokeWebSearch` (resource: `arn:aws:bedrock-agentcore:us-east-1:aws:tool/web-search.v1`)
Gateway service role (Web Answer)	`bedrock-agentcore:InvokeWebAnswer` (resource: `arn:aws:bedrock-agentcore:us-east-1:aws:tool/web-answer.v1`)
Caller (execution role of the agent or harness)	`bedrock-agentcore:InvokeGateway` (Gateway ARN)

Verification 2 showed that InvokeGateway is required on the caller and not on the service role. InvokeWebAnswer and InvokeWebSearch on the service role are each required independently per tool (details in Verification 2).

What I confirmed on a real environment

The following is what I actually confirmed in us-east-1 on 2026-11-08. Each verification is written in this order: what I checked, the measurements, and what can be concluded.

Verification 1: Sending the same question to both and comparing the response formats

I checked the difference in response format and how citations are attached, using actual responses. I sent the same three questions to both Web Search and Web Answer. I fixed the questions to English because Web Answer supported English at launch.

Question	Type	Question text
Q1	Fact check	What is the maximum execution timeout of an AWS Lambda function?
Q2	Regulation / policy	What is the minimum Common Equity Tier 1 capital ratio required under Basel III?
Q3	Latest information	What are the recent updates to Amazon Bedrock AgentCore?

I sent each question to each tool three times, for 18 calls in total. All returned HTTP 200 with isError false. Web Search used maxResults: 5. I recorded the response format, whether and where citations appear, whether publishedDate is present, and latency. For latency, I recorded both _meta.latencyMs and client-side timing for Web Answer, and only client-side timing for Web Search because it returns no _meta.

Latency (milliseconds; average of 3 runs; client side is the time of the whole HTTP request)

Question	Search, client side	Answer, client side	Answer, server side (`_meta.latencyMs`)
Q1 (fact check)	1,101	3,122	2,292
Q2 (regulation)	1,136	3,011	2,215
Q3 (latest information)	1,259	6,555	5,744

Differences in the response

Aspect	Web Search	Web Answer
Top-level keys	`isError`, `content`	`isError`, `content`, `structuredContent`, `_meta`
Body	`content[0].text` is a JSON string of the form `{"id": ..., "results": [...]}`, and `results` has `maxResults` entries (5)	`content[0].text` is Markdown. Citations are `[[N](url)]`
Contents of one item	`text` (snippet, about 700 to 5,000 characters), `title`, `url`, `publishedDate`	The answer body and a list of citations (`structuredContent.answer` is one block)
Dates	`publishedDate` is in the form `05:00PM, Wednesday, September 09 2026, PDT`, or `unknown` (0 to 2 of 5 items were `unknown`)	No date is attached
When the same question is sent 3 times	The set of result URLs was the same all 3 times. The whole response was not fully identical, though (the length was the same in some runs)	The body length changes between runs (whole-response character counts for Q1: 1,947 / 1,485 / 2,313)
Characters in the whole response	About 9,600 to 14,100	About 1,400 to 2,500 for Q1 and Q2, about 10,000 to 12,700 for Q3

Actual content returned (Q1 "What is the maximum execution timeout of an AWS Lambda function?", first run)

Web Answer's answer (full content[0].text, 627 characters).

The standard maximum timeout for an AWS Lambda function is 900 seconds (15 minutes). [[1](https://docs.aws.amazon.com/lambda/latest/dg/configuration-timeout.html)]

There is one exception: for functions on Lambda Managed Instances, asynchronous and event source mapping invocations can be set to up to 5,400 seconds (90 minutes). [[2](https://aws.amazon.com/blogs/compute/announcing-90-minute-function-timeout-on-aws-lambda-managed-instances/)] Synchronous invocations on those instances still cap at 15 minutes. [[2](https://aws.amazon.com/blogs/compute/announcing-90-minute-function-timeout-on-aws-lambda-managed-instances/)]

Web Search results (maxResults: 5. The 5 snippets total 8,227 characters. The full text is in the collapsible section below).

#	Title	URL	publishedDate	Snippet characters
1	Configure Lambda Event Function timeout	docs.aws.amazon.com/lambda/latest/dg/configuration-timeout.html	05:00PM, Sunday, October 30 2016, PDT	1,164
2	Troubleshoot configuration issues in Lambda	docs.aws.amazon.com/lambda/latest/operatorguide/configurations.html	unknown	2,312
3	AWS SDK for Kotlin	docs.aws.amazon.com/sdk-for-kotlin/api/latest/lambda/.../timeout.html	unknown	1,365
4	Announcing 90-minute function timeout on AWS Lambda Managed Instances	aws.amazon.com/blogs/compute/announcing-90-minute-function-timeout-on-aws-lambda-managed-instances/	12:08PM, Wednesday, September 09 2026, PDT	2,023
5	AWS Lambda vs Step Functions vs EventBridge	jayendrapatil.com/aws-lambda-vs-step-functions-vs-eventbridge/	05:00PM, Wednesday, June 24 2026, PDT	1,363

The full content[0].text of Web Search (Q1, first run, 9,595 characters) follows. Reading it side by side with Web Answer's answer (627 characters) shows how the citations differ.

Full response of Web Search (click to expand)

{
  "id": "dafbd346",
  "results": [
    {
      "publishedDate": "05:00PM, Sunday, October 30 2016, PDT",
      "text": "Configure Lambda Event Function timeout Lambda runs your code for a set amount of time before timing out. Timeout is the maximum amount of time in seconds that a Lambda function can run. The default value for this setting is 3 seconds, but you can adjust this in increments of 1 second up to a maximum value \n\nof 900 seconds (15 minutes). For functions using AWS Lambda Managed Instances, asynchronous invocations and event source mapping invocations (except Amazon MQ and Amazon DocumentDB) support a maximum allowed value of 5,400 seconds (90 minutes). \n\nConfiguring timeout (AWS SAM) Determining the appropriate timeout value for a Lambda function If the timeout value is close to the average duration of a function, there is a higher risk that the function times out unexpectedly. The duration of a function can vary \n\nUnder General configuration , choose Edit . For Timeout , set a value between 1 and 900 seconds (15 minutes). Choose Save . Configuring timeout (AWS CLI) You can use the update-function-configuration command to configure the timeout value, in seconds. The following example command increases the function timeout to 120 seconds (2 minutes). ",
      "title": "Configure Lambda Event Function timeout",
      "url": "https://docs.aws.amazon.com/lambda/latest/dg/configuration-timeout.html"
    },
    {
      "publishedDate": "unknown",
      "text": "The following topics provide troubleshooting advice for common issues that you might encounter related to Lambda function configuration settings. Memory configurations You can configure a Lambda function to use between 128 MB and 10,240 MB of memory. By default, any function created in the console is assigned the smallest amount of memory. Many Lambda functions are performant at this lowest setting. However, if you are importing large code libraries or completing memory intensive tasks, 128 MB is not sufficient. \n\nTimeouts Timeouts for Lambda functions can be set between 1 and 900 seconds (15 minutes). For functions using AWS Lambda Managed Instances, asynchronous invocations and event source mapping invocations (except Amazon MQ and Amazon DocumentDB) support a maximum allowed value of 5,400 seconds (90 minutes). By default, the Lambda console sets this to 3 seconds. The timeout value is a safety valve that ensures functions do not run indefinitely. After the timeout value is reached, Lambda stops the function invocation. \n\nIf a timeout value is set close to the average duration of a function, this increases the risk that the function times out unexpectedly. The duration of a function can vary based on the amount of data transfer and processing, and the latency of any services the function interacts with. Common causes of timeout include: \n\n2. Duration – Before the function runs out of memory, it finishes invocations at a steady double-digit millisecond rate. As paging occurs, the duration takes an order of magnitude longer. 3. Error count – As the memory leak exceeds allocated memory, eventually the function errors due to the computation exceeding the timeout, or the execution environment stops the function. \n\nIn JavaScript, asynchronous callbacks are handled with an event loop. Other runtimes use different mechanisms to handle concurrency. When the function's execution environment ends, Lambda freezes the environment until the next invocation. \n\n1. The code calls the asynchronous doWork function using the await keyword in the handler. 2. The doWork function takes some period of time to complete before resolving the promise. 3. The function times out because doWork takes longer than the timeout limit allows and the callback result is not returned in a later invocation. ",
      "title": "Troubleshoot configuration issues in Lambda",
      "url": "https://docs.aws.amazon.com/lambda/latest/operatorguide/configurations.html"
    },
    {
      "publishedDate": "unknown",
      "text": "If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose â€œAcceptâ€ or â€œDecline.â€ To make more detailed choices, choose â€œCustomize.â€ \n\nTo opt out of our use of cookies or similar technologies to engage in these activities, select â€œOpt out of cross-context behavioral adsâ€ and â€œSave preferencesâ€ below. If you clear your browser cookies or visit this site from a different device or browser, you will need to make your selection again. For more information about cookies and how we use them, read ourÂ Cookie Notice. \n\nTo opt out of the use of other identifiers, such as contact information, for these activities, fill out the formÂ here. For more information about how AWS handles your information, read theÂ AWS Privacy Notice. \n\nThe amount of time (in seconds) that Lambda allows a function to run before stopping it. The default is 3 seconds, and the maximum allowed value is 900 seconds. For functions using Lambda Managed Instances, asynchronous invocations and event source mapping invocations (except Amazon MQ and Amazon DocumentDB) support a maximum allowed value of 5,400 seconds (90 minutes). For more information, see Lambda execution environment. ",
      "title": "AWS SDK for Kotlin",
      "url": "https://docs.aws.amazon.com/sdk-for-kotlin/api/latest/lambda/aws.sdk.kotlin.services.lambda.model/-update-function-configuration-request/-builder/timeout.html"
    },
    {
      "publishedDate": "12:08PM, Wednesday, September 09 2026, PDT",
      "text": "Activating 90-minute function timeout You can now configure any Lambda function running on a Managed Instance with a timeout of up to 90 minutes (5,400 seconds) for async and ESM invocations. Synchronous invocations retain the existing 15-minute maximum. The function executes exactly as before: same runtime, same handler, same IAM execution role, same virtual private cloud (VPC) configuration. The only difference is that your function now supports longer continuous execution. Your initialization code (Init phase) is still limited to 15 minutes on Lambda Managed Instances. \n\nThere is no additional charge for using the 90-minute timeout. Standard Lambda Managed Instances pricing applies. 90-minute timeout and durable functions The 90-minute function timeout and durable functions are complementary. The function timeout (--timeout) controls how long each individual invocation can run, while the durable execution timeout (ExecutionTimeout in --durable-config) controls the total elapsed time from execution start to completion. Durable functions use checkpoints to track progress and automatically recover from failures through replay, re-executing from the beginning while skipping completed work. With today’s launch, each asynchronous invocation in a durable function running on a Managed Instance can now execute for up to 90 minutes continuously, while the corresponding durable execution can run for up to 1 year. For synchronous and event source mapping invocations, both the invocation and the corresponding durable execution are limited to 90 minutes. \n\nNote that invocations for Amazon MQ ESM and Amazon DocumentDB (with MongoDB compatibility) ESM remain limited to 15 minutes. Synchronous invocations (15 minutes maximum): Synchronous invocations retain the existing 15-minute maximum timeout. If you set your function timeout to greater than 15 minutes and invoke it synchronously, Lambda continues to apply the 15-minute timeout. The GetFunctionConfiguration API reports the configured timeout value. ",
      "title": "Announcing 90-minute function timeout on AWS Lambda Managed Instances",
      "url": "https://aws.amazon.com/blogs/compute/announcing-90-minute-function-timeout-on-aws-lambda-managed-instances/"
    },
    {
      "publishedDate": "05:00PM, Wednesday, June 24 2026, PDT",
      "text": "| event_date | 2026-08-11 |\n| title | AWS Lambda vs Step Functions vs EventBridge |\n| original_url | https://jayendrapatil.com/aws-lambda-vs-step-functions-vs-eventbridge/ |\n| source_domain | jayendrapatil.com |\n| search_queries_hash | 187d207b814cb47441a22825e67d4c83 |\n### search_queries (Level 0)\n#### search_queries - Item 1\n- What architecture pattern is best for decoupling microservices that react to a single event? : Amazon EventBridge with a custom event bus and content-based routing rules provides the most decoupled architecture, allowing producers to remain unaware of consumers.\n#### search_queries - Item 2\n- How much does AWS Step Functions Standard Workflow cost? : AWS Step Functions Standard Workflow costs $25.00 per 1 million state transitions.\n#### search_queries - Item 3\n- What is the difference between AWS Lambda, Step Functions, and EventBridge? : Lambda handles compute (executing code), Step Functions handles orchestration (coordinating multiple services into workflows), and EventBridge handles event routing (connecting producers and consumers) — they are complementary services often used together.\n#### search_queries - Item 4\n- What is the maximum execution duration for AWS Lambda? : Standard Lambda functions have a 15-minute timeout, MicroVMs support up to 8 hours, and Lambda Durable Functions can suspend for up to 1 year.",
      "title": "AWS Lambda vs Step Functions vs EventBridge",
      "url": "https://jayendrapatil.com/aws-lambda-vs-step-functions-vs-eventbridge/"
    }
  ]
}

Comparing the two, you can see the following.

Citations are attached differently. Web Answer attaches the source URL to each sentence of the answer (the [[1](...)] and [[2](...)] above). Web Search only attaches url and title to each result, and does not tell you which sentence is based on which result. When you compose an answer, your own model has to add the citations
The first Web Search snippet already contains the answer (up to 900 seconds, and 5,400 seconds on Managed Instances). However, the answer sits in the middle of explanatory text about how to configure the setting, and extracting it is the job of your own model
The results also include content that is only weakly related to the answer. The third result contains the text of a cookie consent banner with garbled characters, and the fifth is a list of search queries from a third-party blog, which includes a statement that none of the other four contain: "MicroVMs support up to 8 hours". Deciding which to use is also the job of your own model. Web Answer's answer cited only two sources: the official documentation and an AWS blog
Web Answer summarized the same content in a 627-character answer and returned it with a citation on each sentence. The numbers in the answer (900 seconds and 5,400 seconds) are the same as those in the first snippet
The amount returned is 8,227 characters (5 results) for Web Search and 627 characters (Q1) for Web Answer, a difference of about 13 times

The difference in length comes from the difference in what each tool returns.

Aspect	Web Search	Web Answer
What it returns	Excerpts (snippets) of the pages it found, returned without summarizing. Raw material	Only the answer to the question, written after reading the pages, plus citations. The page bodies are not returned
What determines the length	The number of results (`maxResults`) and the length of each snippet (about 700 to 5,000 characters this time). The difference between questions is small (about 9,600 to 14,100 characters for Q1 to Q3)	The content of the question. Short for questions that can be answered briefly (about 1,400 to 2,500 characters for Q1 and Q2), and long for questions with a lot to answer (about 10,000 to 12,700 characters for Q3)
Who extracts the answer	Your own model	Web Answer

In other words, Web Answer was short because the questions could be answered briefly. For Q3 (recent updates to AgentCore), Web Answer's answer was also about 10,000 characters, the same as Web Search's results. The essential difference is not length, but "whether it returns material or an answer". The following diagram summarizes what comes back for the same question (Q1).

For Q2 (the CET1 ratio under Basel III), Web Answer also answered "a minimum of 4.5%, plus a 2.5% buffer, so effectively 7%", citing three sources. Web Search returned five results, including IMF, BIS, Investopedia, and Wikipedia.

How to read Web Answer's citations

According to the official description, marker N corresponds to citations[N-1] of that block. In my measurements, the same URL appeared multiple times in citations, so the number of entries was larger than the number of distinct markers (for example, the first run of Q1 had 4 entries but only 2 unique URLs). Looking up by citations[N-1] mismatched the marker's URL in 7 of 9 runs, while looking up by the Nth URL in order of first appearance after removing duplicates matched in all 9 runs.

import re

PAT = re.compile(r"\[\[(\d+)\]\((https?://[^)]+)\)\]")

def unique_urls(citations):
    seen = []
    for c in citations:
        if c["url"] not in seen:
            seen.append(c["url"])
    return seen
# URL of marker N = unique_urls(citations)[N-1]

Note that the sample is only 3 questions × 3 runs, in us-east-1, on a single day. There is no guarantee that this mapping always holds, and it may change with specification changes. If you process citations programmatically, it is safer to use the URL in the marker itself.

The results of Verification 1 affect the choice as follows.

If you want to reason over the results with your own model, Web Search is faster (about 1.1 seconds versus about 3.1 seconds for Q1), and maxResults lets you limit the number of results. However, dates have string variations and unknown is mixed in, so if you use dates as a filtering condition, it is safer to use the request-side publishedDateFilter (Verification 4)
If you want a finished answer, Web Answer returns it with citations, but it took about 6.6 seconds for the latest-information question (Q3). If latency matters, measure by question type before choosing
When you use Web Answer's citations for display, use the URL in the marker, not the index into citations

Verification 2: Separation of IAM responsibilities

To establish the minimum required permissions, I checked the behavior with the service role's InvokeWebAnswer / InvokeWebSearch present and absent, and with InvokeGateway attached to different roles.

Configuration	Web Search	Web Answer
2-1 Baseline: service role has only the dedicated permissions, caller has `InvokeGateway`	Success	Success
2-2 Remove `InvokeWebAnswer` from the service role	Success	Failure (HTTP 200, JSON-RPC error `-32002`: `Execution role is not authorized for connector web-answer`)
2-3 Restore `InvokeWebAnswer` and remove `InvokeWebSearch`	Failure (HTTP 200, JSON-RPC error `-32002`: `Execution role is not authorized for connector web-search`)	Success
2-4 Add `InvokeGateway` to the service role (with both dedicated permissions)	Success	Success
2-5 Remove `InvokeGateway` from the caller (service role as in 2-4)	Failure (HTTP 403, JSON-RPC error `-32002`: `Authorization error - Insufficient permissions`)	Failure (same as left)

I made the calls by assuming a dedicated verification IAM role (wa-verify-caller) with assume-role. After each permission change I waited 20 seconds before calling, and every configuration gave the expected result from the first call (no retries were needed to wait for propagation).

The results of Verification 2 show the following.

InvokeGateway is required on the caller and not on the service role (in 2-1 it succeeded without it on the service role; in 2-4 adding it changed nothing; in 2-5 it failed without it on the caller)
The dedicated permissions on the service role are independent per tool. Removing only Web Answer's permission left Web Search working, and vice versa. You can remove the permissions for tools you don't use from the service role
Errors appear in three different ways depending on the cause. This can also help you narrow down a permission problem

Cause	HTTP status	How it is returned
The service role lacks the tool's permission	200	JSON-RPC `error` (`-32002`, `Execution role is not authorized for connector ...`)
The caller lacks `InvokeGateway`	403	JSON-RPC `error` (`-32002`, `Authorization error - Insufficient permissions`)
Invalid filter or date specification (Verifications 3 and 4)	200	`isError: true` inside `result` (`DependencyFailedException`)

Caller code will miss failures unless it checks all three places: the HTTP status, the JSON-RPC error, and result.isError.

Verification 3: Combination rules of the domain filters

To check the specification that "the agent cannot widen the boundary", I varied the combinations of include and exclude on the administrator side and the request side, and checked the results and errors.

Pattern	Administrator side	Request side	Expected (official)	Measured
Empty intersection	include: docs.aws.amazon.com	include: bis.org	Web Answer: validation error	Both tools returned an error. Answer: `include contains domains that are not permitted for this tool: bis.org`. Search: `Runtime domainFilter.include has no overlap with admin domainFilter.include`
Mixing in a domain outside the boundary	include: docs.aws.amazon.com	include: docs.aws.amazon.com, wikipedia.org	Domains outside the boundary are not used	Search: no error, and all 5 results were from docs.aws.amazon.com (only the intersection was used). Answer: error (rejected because `wikipedia.org` is not permitted; it fails even when there is an intersection)
Merging excludes	exclude: investopedia.com	exclude: imf.org	Both excluded	The two domains that had appeared in the baseline (Q2 of V1) no longer appeared in either tool (Search: bis.org, fraser.stlouisfed.org, debevoise.com, en.wikipedia.org, and others. Answer: bis.org, fraser.stlouisfed.org, mayerbrown.com)
Can the administrator's exclude be bypassed?	exclude: investopedia.com	include: investopedia.com	Cannot be overridden	Answer: error (`include contains domains that are excluded, fully or partly, for this tool: investopedia.com. Remove them from include.`). Search: HTTP 200, `isError: false`, 0 results (an empty result, not an error)

Errors caused by filters were returned with HTTP status 200, isError: true, and DependencyFailedException - ... in the body. You cannot detect the failure from the HTTP status alone.

Verification 3 showed three things.

The boundary set by the administrator was actually respected by both tools. Domains outside the boundary and domains the administrator excluded did not appear in the results (one observation per pattern)
The way a violation looks is different. Web Answer reports an error if even one non-permitted or excluded domain is included in the request. Web Search quietly narrows down to the intersection, and when the request collides with an exclude, it returns an empty result without an error
An agent that uses Web Search may interpret an empty result as "nothing was found". The calling side needs to decide how to handle zero results

Verification 4: Date filter

To see whether the date filter can be used for freshness requirements, I varied the range of publishedDateFilter and checked whether the results changed. I also checked the error when from is later than to.

Condition	Web Search	Web Answer
Limit to the last month (`from: 2026-10-08`, `to: 2026-11-08`, question Q3)	All 10 results were in range (`publishedDate` from 2026-10-11 to 2026-11-06). Before filtering (V1), results from April to September 2026 were mixed in	The cited sources were mainly pages from October to November 2026 (the Web Answer announcement page, October AgentCore announcements, and so on). Because the dates of the cited sources are not attached to the result, you can only judge whether they are within the range from the text
`from` > `to` (`from: 2026-11-08`, `to: 2026-10-08`)	Error: `publishedDate.from (2026-11-08) must not be after publishedDate.to (2026-10-08)`	Error: `filters.publishedDateFilter.from must not be after to.`

Verification 4 shows how to use it for freshness requirements.

For use cases that need freshness, both can be narrowed with the request-side publishedDateFilter. However, only Web Search lets you check the dates of the results yourself (Web Answer's citations carry no dates)
Errors in the date filter are also returned with HTTP 200 and isError: true (same as Verification 3)
Web Search's publishedDate was a string containing PDT or PST. If you want to sort by date, you need to parse it yourself
Web Answer is given today's date when generating an answer, and this answer said "current date: November 7, 2026" (apparently the date in Pacific Time)

Criteria for choosing

Based on the results of Verifications 1 to 4, here is a summary by criterion. The "Basis" column shows which verification in this article it rests on.

Criterion	Choose Web Search	Choose Web Answer	Basis
You want to reason over the returned results with your own model	Suitable (url, title, snippet, and date are returned)	Not suitable (a finished answer is returned)	Verification 1
You want a finished answer to a sub-question	You need to synthesize it yourself	Suitable (returned with citations)	Verification 1
Response speed matters	Fast (average about 1.1 to 1.3 seconds)	Varies by question (about 3.0 seconds; about 6.6 seconds for latest-information questions)	Verification 1
You want to keep everything within the Tokyo Region	Suitable	Not possible as of 2026-11-08 (may change)	Region table
You want to pin the referenced domains on the administrator side	Possible with both. Outside the boundary, results are narrowed or empty	Possible with both. Specifying outside the boundary returns an error	Verification 3
You want failures to be easy to notice	An empty result can be returned without an error	Invalid specifications are returned as errors	Verification 3
You want to filter by date	Possible. You can check the result's `publishedDate` (custom format)	Possible. The dates of cited sources are not attached to the result	Verification 4
You want to guarantee citations are displayed in a machine-checkable way	Display the result's url yourself	The answer comes back with citations. Use the URL in the marker	Verification 1

Here is a rough guide for choosing.

If you want to control how the results are handled by your own model, or you want to use Tokyo, choose Web Search
If you want to use the finished answer and citations as they are, and us-east-1 is acceptable, choose Web Answer
You can also use both. AWS also calls them "complementary", and you can split by question type, for example Web Answer for fact checks and Web Search for preliminary research

As for citations, for both tools it is a condition of use to keep and display the cited sources and links. Bulk extraction and building a competing index are prohibited.

How to verify it yourself

This shows a flow that readers can try in their own environment. The following is a shortened version of the code I actually used in the verification. I did the verification with a dedicated role and Gateway, and deleted everything after finishing.

You need AWS CLI v2, Python 3.10 or later, boto3 (1.43.91 at the time of verification), and requests. The Region is us-east-1.

1. Create the service role

The trust policy (trust.json). Replace <ACCOUNT_ID> with your own account ID.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "bedrock-agentcore.amazonaws.com"},
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {"aws:SourceAccount": "<ACCOUNT_ID>"},
        "ArnLike": {"aws:SourceArn": "arn:aws:bedrock-agentcore:us-east-1:<ACCOUNT_ID>:gateway/*"}
      }
    }
  ]
}

The permissions policy (policy.json). It is required independently per tool.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "bedrock-agentcore:InvokeWebAnswer",
      "Resource": "arn:aws:bedrock-agentcore:us-east-1:aws:tool/web-answer.v1"
    },
    {
      "Effect": "Allow",
      "Action": "bedrock-agentcore:InvokeWebSearch",
      "Resource": "arn:aws:bedrock-agentcore:us-east-1:aws:tool/web-search.v1"
    }
  ]
}

aws iam create-role --role-name wa-verify-gateway-role \
  --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name wa-verify-gateway-role \
  --policy-name wa-verify-base --policy-document file://policy.json

The inline policy of the created role looks like this in the console.

2. Create the Gateway and targets with Python

import time
import boto3

control = boto3.client("bedrock-agentcore-control", region_name="us-east-1")

gw = control.create_gateway(
    name="wa-verify-gateway",
    roleArn="arn:aws:iam::<ACCOUNT_ID>:role/wa-verify-gateway-role",
    protocolType="MCP",
    authorizerType="AWS_IAM",
)
gid = gw["gatewayId"]
while control.get_gateway(gatewayIdentifier=gid)["status"] != "READY":
    time.sleep(3)

def add_target(name, connector_id, config_name):
    t = control.create_gateway_target(
        name=name,
        gatewayIdentifier=gid,
        targetConfiguration={"mcp": {"connector": {
            "source": {"connectorId": connector_id},
            "configurations": [{"name": config_name, "parameterValues": {}}],
        }}},
        credentialProviderConfigurations=[{"credentialProviderType": "GATEWAY_IAM_ROLE"}],
    )
    while control.get_gateway_target(gatewayIdentifier=gid, targetId=t["targetId"])["status"] != "READY":
        time.sleep(3)

add_target("web-answer-tool", "web-answer", "WebAnswer")
add_target("web-search-tool", "web-search", "WebSearch")

Right after creating the role, it is safer to wait about 10 seconds for propagation before creating the Gateway. If you do not specify a version, the current default is used (at the time of verification, 1.1.0 for Web Answer and 1.3.0 for Web Search).

The Gateway reached the Ready status about 10 seconds after creation. Both targets also became Ready within a few seconds.

In the target list, the target types are displayed as "ウェブ回答ツール" (Web Answer tool) and "ウェブ検索ツール" (Web Search tool).

Settings screens in the console

You can make the same settings from the console. In step 3, "Add target", of the Gateway creation wizard, choose as follows. The console in the screenshots is displayed in Japanese, so the Japanese labels are shown in parentheses.

Leave the target protocol as "MCP target" (MCP ターゲット)
For the target type, choose "Connector - New" (コネクター - 新規). The other options are MCP server, Lambda ARN, REST API, and API Gateway
"Pre-configured targets" (事前設定済みのターゲット) lists knowledge base (KB), Web Search tool (ウェブ検索ツール), Web Answer tool (ウェブ回答ツール), Step Functions, and other integrations. Choose "Web Answer tool" here
When you choose "Web Answer tool", a blue notice appears. It says what the developer guide says: because it uses geographic cross-Region inference, data processing stays in commercial Regions within the specified geographic boundary (such as the US, EU, or APAC)
For the outbound authentication setting, only "IAM role" (IAM ロール) can be selected (OAuth client, API key, and no authorization cannot be selected)

The correspondence with the code is as follows.

Console item	Code item
Target type "Connector", pre-configured target "Web Answer tool"	`connector.source.connectorId: "web-answer"`
Pre-configured target "Web Search tool"	`connectorId: "web-search"`
Outbound authentication setting "IAM role"	`credentialProviderType: "GATEWAY_IAM_ROLE"`
"Domain filtering" (ドメインフィルタリング) under additional settings (included / excluded)	`parameterValues.domainFilter.include` / `exclude` (the "administrator-side" boundary in Verification 3)

When you open "Additional settings - Optional" (追加設定 - オプション), there is a "Domain filtering" item. Its description is "Set the domains to include in and exclude from this target", the current counts are shown as "Included: 0 · Excluded: 0", and you edit them with the "View and manage domains" button. In other words, the boundary that the administrator sets on the target, which Verification 3 dealt with, is set in the target's additional settings in the console.

On this screen (with additional settings opened), I did not see an item for choosing the connector version. If you want to pin the version, use source.version, as in the code in this article (I did not check whether it can be pinned in the console).

3. Call MCP with SigV4 signing

tools/list and tools/call worked without initialize.

import json
import boto3, requests
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest

REGION = "us-east-1"
URL = "https://<GATEWAY_ID>.gateway.bedrock-agentcore.us-east-1.amazonaws.com/mcp"  # <GATEWAY_ID> is the gatewayId returned by create_gateway (e.g. name-alphanumerics)

def rpc(method, params=None, session=None):
    body = json.dumps({"jsonrpc": "2.0", "id": 1, "method": method, "params": params or {}})
    creds = (session or boto3.Session()).get_credentials().get_frozen_credentials()
    req = AWSRequest(method="POST", url=URL, data=body,
                     headers={"Content-Type": "application/json",
                              "Accept": "application/json, text/event-stream"})
    SigV4Auth(creds, "bedrock-agentcore", REGION).add_auth(req)
    r = requests.post(URL, data=body, headers=dict(req.headers), timeout=120)
    return r.status_code, r.text

print(rpc("tools/list"))
print(rpc("tools/call", {"name": "web-answer-tool___WebAnswer",
                         "arguments": {"query": "What is the maximum execution timeout of an AWS Lambda function?"}}))
print(rpc("tools/call", {"name": "web-search-tool___WebSearch",
                         "arguments": {"query": "What is the maximum execution timeout of an AWS Lambda function?",
                                       "maxResults": 5}}))

The caller's IAM needs bedrock-agentcore:InvokeGateway (targeting the Gateway ARN). Check for failures in three places: the HTTP status, the JSON-RPC error, and result.isError (see Verifications 2 and 3).

4. Try the domain filters (optional)

Change the administrator-side boundary with update_gateway_target. The required arguments are gatewayIdentifier, targetId, and targetConfiguration.

control.update_gateway_target(
    gatewayIdentifier=gid,
    targetId="<TARGET_ID>",
    name="web-search-tool",
    targetConfiguration={"mcp": {"connector": {
        "source": {"connectorId": "web-search"},
        "configurations": [{"name": "WebSearch", "parameterValues": {
            "domainFilter": {"include": ["docs.aws.amazon.com"]}
        }}],
    }}},
    credentialProviderConfigurations=[{"credentialProviderType": "GATEWAY_IAM_ROLE"}],
)

On the request side, add filters to arguments.

{"query": "...", "filters": {"domainFilter": {"include": ["bis.org"]}}}
{"query": "...", "filters": {"publishedDateFilter": {"from": "2026-10-08", "to": "2026-11-08"}}}

5. Clean up

Delete the targets, the Gateway, and the IAM role, in that order.

for t in control.list_gateway_targets(gatewayIdentifier=gid)["items"]:
    control.delete_gateway_target(gatewayIdentifier=gid, targetId=t["targetId"])
# Wait for the target deletions to complete
control.delete_gateway(gatewayIdentifier=gid)

aws iam delete-role-policy --role-name wa-verify-gateway-role --policy-name wa-verify-base
aws iam delete-role --role-name wa-verify-gateway-role

Conclusion

I started out with the understanding that Web Answer is "something where a model reads web search results and answers for you". After actually running both and comparing them, I found that the difference, where Web Search returns material in the form of search results and Web Answer returns an answer with citations, becomes the axis for choosing between them, and it even affects how errors look and which Regions you can use. If you want to use a finished answer and its citations as they are, choose Web Answer. If you want to handle the results with your own model, choose Web Search.
As of 2026-11-08, Web Answer is available only in us-east-1, but the blog says more Regions are planned. Connector versions and tool names may also change in the future.

References

From ETL Jobs to Agentic Orchestrators

kienly — Sun, 08 Nov 2026 05:42:29 +0000

I have spent most of my career on the boring, important part of data work: the pipelines that run at 2 a.m. and that nobody thinks about until they break. And I have always liked pipelines for one simple reason. They do not improvise. Give a workflow engine a schedule, a schema, and a known sequence of steps, and it will run the same plan every single day, without getting tired or distracted or "creative."

That predictability is the whole point. It is also the reason I was skeptical when people started talking about putting AI agents in charge of data operations.

This post is about where I landed. Short version: an agent is a great front door to a pipeline, but it should never be the thing that decides how the pipeline runs. The pipeline should keep that job. Here is how to set it up that way with AWS Step Functions and Amazon Bedrock AgentCore Gateway.

Why the boring part is still the hard part

Pipelines are predictable on paper. In practice, the operational work around them is anything but.

Late files arrive in Amazon S3 hours after the batch window closed. An API throttles during a busy week. Someone in finance messages the on-call channel: "Can you re-run yesterday's partition? We fixed a reference table." The pipeline itself is automated, but the control plane around it is still manual. Someone reads the message, checks the S3 key, opens the console or the CLI, builds a JSON payload, starts the run, watches it, and reports back.

It is repetitive, low-risk work that eats time. So it is natural to want an AI agent to handle it.

The trap is what that usually looks like. The first instinct is to hand the agent the keys to the cloud. Give it permission to call glue:StartJobRun, redshift-data:ExecuteStatement, lambda:InvokeFunction, and s3:CopyObject, and you have quietly asked a language model to act as a distributed transaction coordinator. I would not trust a person to do that job from a chat window at midnight, and I do not think a model should either. If the agent gets one parameter wrong on step three of a five-step load, or times out while a Spark cluster is starting, or retries a load that was not idempotent, you are left with a half-written table and no single place that says what happened.

A better split: the agent asks, the workflow does

AWS recently made Step Functions workflows available as tools for AI agents through Amazon Bedrock AgentCore Gateway, using the Model Context Protocol (MCP). It was announced alongside OpenAI GPT-6.1 Terra on Bedrock, which is covered in the AWS Weekly Roundup. The useful part for us: you can expose a workflow you already trust as a set of MCP tools, without writing a wrapper Lambda or any glue code.

The design rule I keep coming back to is this:

The agent decides when to act and with what input. Step Functions decides how the workflow runs.

Everything else in this post follows from that split.

1. The architecture, in one picture

First, a quick note on MCP. It is an open protocol, built on JSON-RPC, that lets an AI application discover tools (tools/list) and call them (tools/call). The appeal is that you expose a tool server once, and any compliant agent can use it, without custom adapters for every model or framework.

Amazon Bedrock AgentCore Gateway acts as that managed MCP boundary. It ships with a built-in Step Functions connector, connectorId: "states". When you register a Step Functions target, the Gateway generates MCP-compliant schemas for the state machine operations you choose, and it calls Step Functions with its own IAM service role.

Here is the whole flow:

Each layer has a narrow job:

The operator and the agent. You ask in plain language. The agent, running on a Bedrock model such as GPT-6.1 Terra, turns that into structured parameters, picks the right tool, checks the status, and tells you what happened. It is good at the translation. It is not in charge of the run.
AgentCore Gateway. This is the protocol and security boundary. It exposes the Step Functions operations you selected, generates their input schemas, injects the values you pinned (such as stateMachineArn), and authenticates with a tightly scoped role.
Step Functions. This is where the real work is decided: state transitions, payload validation, error handling, retries with backoff, and failure notifications. The workflow is the source of truth for what happens next. Step Functions also integrates with a long list of AWS services, so the orchestration logic does not have to be rewritten to become agent-accessible.
The data services. S3, Glue, DynamoDB and SNS do the heavy lifting: storage, Spark processing, audit records, and alerts. They only accept requests from the Step Functions execution role. The agent never talks to them directly.

Why this beats giving the agent direct access

Compare that with handing an agent tool definitions for s3:HeadObject, glue:StartJobRun, redshift-data:ExecuteStatement, and sns:Publish. In that setup, the plan for the pipeline lives inside the model's context window. If the agent process dies while a 20-minute Glue job is running, nothing starts the downstream load or sends the failure alert. An agent with direct Glue or warehouse permissions could also be talked into running something it should not, through prompt injection hidden in a ticket or a filename.

With Step Functions in the middle, the agent's reach shrinks to one entry point: the workflow. It cannot reorder the steps, skip validation, turn off alerts, or quietly widen its own permissions.

2. The scenario: a corrected sales extract at an odd hour

Let's make this concrete. An upstream e-commerce platform uploads a corrected daily sales file to S3, well outside the midnight batch. An engineer (or a Slack ops bot) asks the agent:

"Start the sales ingestion pipeline for raw/sales/2026-11-05/sales.parquet and tell me if it succeeds."

No shell scripts, no console clicking. The agent works through four steps, using the AWS Step Functions and AgentCore Gateway integration:

Discover the tools and pull out the input. The agent lists the MCP tools on the Gateway target, finds StartExecution, and puts raw/sales/2026-11-05/sales.parquet into the sourceKey field. It never needs the state machine ARN, because that is pinned in the Gateway configuration.
Run the workflow. The Gateway calls states:StartExecution on the pinned Standard workflow. The run goes through these states:
- ValidateInput: a Choice state that checks that sourceKey is present and matches raw/sales/*.parquet before any compute is started.
- StartGlueJob: a synchronous task (arn:aws:states:::glue:startJobRun.sync) that starts the PySpark job with the source key and waits for it to finish. Transient failures are retried with exponential backoff.
- RecordSuccess: writes an audit record (execution ARN, source key, completion time, status) to a DynamoDB table.
- NotifyFailure and WorkflowFailed: if validation or the Glue job fails after its retries, a Catch routes to an SNS notification and ends in an explicit Fail state.
Watch it. When StartExecution returns an executionArn, the agent calls DescribeExecution until the status leaves RUNNING. If something fails, or you want the details, it calls GetExecutionHistory and points to the exact state where things went wrong.
Summarize. Once the run is finished, the agent turns the status, duration, and audit record into a short, readable answer.

Why Standard workflows, not Express

Step Functions has two workflow types. For agent-driven pipelines, I would use Standard every time, for two reasons:

Runs can be long. Glue and Redshift jobs often take longer than the five-minute limit on Express workflows. Standard workflows can run for up to a year.
You need the history. Standard workflows keep a full execution history that DescribeExecution and GetExecutionHistory can query. Express workflows send logs to CloudWatch asynchronously and do not support these calls in the same way, which is a problem when the agent needs to answer "what happened?"

3. The implementation, piece by piece

The repository is deployed with AWS SAM (Serverless Application Model). The full, copy-paste version lives in the demo repository. Four design choices took me the most time to get right, and they are the ones I would look at first.

1. The state machine: validate before you spend money

The first defense against a bad agent call is an explicit ValidateInput state in the Amazon States Language definition (state-machine/data-pipeline.asl.json). StartExecution takes its input as a serialized JSON string, and the Gateway's schema cannot check the contents of that string. So the workflow has to. The Choice state below rejects bad input right away, which costs next to nothing compared with starting a Glue job:

"ValidateInput": {
  "Type": "Choice",
  "Comment": "Ensure sourceKey is provided and targets the raw/sales/ prefix with a .parquet extension",
  "Choices": [
    {
      "And": [
        {
          "Variable": "$.sourceKey",
          "IsPresent": true
        },
        {
          "Variable": "$.sourceKey",
          "IsString": true
        },
        {
          "Variable": "$.sourceKey",
          "StringMatches": "raw/sales/*.parquet"
        }
      ],
      "Next": "StartGlueJob"
    }
  ],
  "Default": "FormatValidationFailure"
}

StartGlueJob uses the .sync integration pattern (arn:aws:states:::glue:startJobRun.sync). Step Functions starts the job, waits, and only moves to RecordSuccess once Spark has finished. You do not write any polling code.

One detail that has bitten people, including me in early drafts: if you route the error path to NotifyFailure and that state ends with "End": true, Step Functions considers the error handled and marks the whole execution SUCCEEDED. The agent then reports a success that never happened. The fix is to send the error path to a "Type": "Fail" state.

2. The Gateway target: pin what the agent should never choose

The Gateway target is defined in gateway/target.json. It sets "connectorId": "states" and lists the operations to expose. The important part is pinning:

State-machine operations (StartExecution, ListExecutions, DescribeStateMachine) work on a stateMachineArn. We pin that value in the target. The Gateway removes the parameter from the schema the agent sees and fills it in at call time. The agent has no way to pick a different state machine.
Execution operations (DescribeExecution, GetExecutionHistory, StopExecution) work on an executionArn, which is different for every run. It cannot be pinned, so the agent passes the ARN it got back from StartExecution. Access is controlled by IAM resource scoping instead.

3. Two tiers of IAM, because Step Functions has two ARN formats

This one trips people up. State-machine actions and execution actions are checked against different resource ARNs:

State Machine ARN format:
arn:aws:states:<region>:<account-id>:stateMachine:<state-machine-name>

Execution ARN format:
arn:aws:states:<region>:<account-id>:execution:<state-machine-name>:<execution-name>

In template.yaml, the AgentCoreGatewayServiceRole splits these into two statements:

Policies:
  - PolicyName: StepFunctionsMCPConnectorLeastPrivilege
    PolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Sid: StateMachineLevelOperations
          Effect: Allow
          Action:
            - states:StartExecution
            - states:ListExecutions
            - states:DescribeStateMachine
          Resource: !GetAtt SalesIngestionStateMachine.Arn
        - Sid: ExecutionLevelOperations
          Effect: Allow
          Action:
            - states:DescribeExecution
            - states:GetExecutionHistory
            - states:StopExecution
          Resource: !Sub "arn:aws:states:${AWS::Region}:${AWS::AccountId}:execution:${SalesIngestionStateMachine.Name}:*"

Because the execution pattern includes the state machine name, the role can only inspect or stop executions of this one pipeline. If someone tricks the agent into stopping a payroll run in the same account, IAM denies it at the API.

4. Calling the Gateway over MCP from Python

The client, agent/invoke_agent.py, speaks standard MCP JSON-RPC over SigV4-signed HTTPS. It first calls tools/list to check the generated schemas, then starts the run with tools/call:

mcp_start_request = {
    "jsonrpc": "2.0",
    "id": "req-start-1",
    "method": "tools/call",
    "params": {
        "name": "states_StartExecution",
        "arguments": {
            "name": f"sales-ingest-{uuid.uuid4().hex[:8]}",
            "input": json.dumps({"sourceKey": source_key})
        }
    }
}

After that, the client polls states_DescribeExecution with backoff until the status is no longer RUNNING, then pulls states_GetExecutionHistory for the step-level timing.

4. Guardrails, and the mistakes I would avoid

I want to be direct about this part. A system prompt that says "never run any workflow except the sales pipeline, and never pass a bad S3 key" is a wish, not a control. Prompts can be injected, context gets confused, and models change. The protections that actually hold are the ones enforced outside the model: IAM policies, Gateway pinning, and input validation in the state machine.

Here is how I think about the main risks:

Operational risk	What it looks like in practice	Deterministic mitigation
Prompt injection	A malicious string in a ticket or filename tells the agent to start a cleanup workflow or pass odd arguments to Glue.	Pin `stateMachineArn` in the Gateway target. Enforce the `raw/sales/*.parquet` pattern in the `ValidateInput` Choice state.
Unauthorized execution	The agent tries to find and start a finance or HR workflow in the same account.	Pin the ARN in the Gateway, and scope `states:StartExecution`, `ListExecutions`, and `DescribeStateMachine` in IAM to that one ARN. Never grant `states:` or `Resource: ""`.
Stopping the wrong run	The agent is asked to cancel a stuck job and picks up an execution ARN that belongs to something important.	Scope `StopExecution`, `DescribeExecution`, and `GetExecutionHistory` to `execution:PinnedStateMachineName:*`.
Runaway retries	A schema error makes the agent call `StartExecution` over and over, burning compute.	Keep retries inside the state machine with `MaxAttempts` and `BackoffRate`. Use the `name` parameter for idempotency, and cap tool calls per session.
No audit trail	A pipeline changed warehouse tables overnight, and nobody can tell whether a person, a schedule, or an agent started it.	Use Standard workflows, turn on CloudTrail for Step Functions and Gateway calls, and prefix agent-started execution names (for example `agent-run-*`).
Long runs	A 45-minute job outlasts the agent's conversation turn.	Start the run, return the `executionArn` straight away, and poll in later turns (or use a callback).

Four mistakes I have seen (and made)

Using the state machine ARN for execution actions. The deploy works, StartExecution works, and then the agent gets an AccessDeniedException when it checks status. Execution APIs need the :execution:Name:* form.
Leaving stateMachineArn unpinned during testing. The Gateway then makes it a required argument. Now you are hardcoding account IDs and regions into the prompt, and inviting the model to make up ARNs.
Using an Express workflow for agent monitoring. DescribeExecution and GetExecutionHistory are designed around Standard workflows. The tools will fail or give you nothing useful.
Swallowing errors. If the failure path ends with "End": true, the execution reports SUCCEEDED. Your agent will happily tell the user the pipeline worked just as the on-call pager goes off.

5. Where GPT-6.1 Terra fits

The Gateway is half of an agent. The other half is the model that reads the request, plans the tool calls, and writes the summary. AWS announced in the weekly roundup that OpenAI GPT-6.1 Terra is generally available on Amazon Bedrock. AWS describes it as a fast, cost-efficient model for everyday enterprise work.

One honest note: any speed, latency, or price figures from AWS or OpenAI are vendor-reported. Measure tool-call accuracy, latency, and cost per run against your own payloads before you put a number in an SLA.

With that caveat, a fast, cheap model suits this kind of work well. Here are four patterns I find useful:

Short tool loops. A typical request means StartExecution, then DescribeExecution, then maybe GetExecutionHistory: three to five calls. A fast model keeps that feeling responsive, which is what makes an ops bot pleasant to use.
Triage with context. When a run fails, GetExecutionHistory returns the state transitions and the error. A model can read that, check a knowledge base of runbooks and table contracts, and reply with the cause and the likely fix. A Glue error caused by a column type change is a good example.
Read-only assistants. Analysts often want to know whether today's partition has loaded before a dashboard goes out. A Gateway target that exposes only ListExecutions, DescribeExecution, and GetExecutionHistory lets a model answer status questions in Slack, with no start or stop permissions.
Backfill coordination. Over a multi-day backfill, an agent can walk through verified partitions, start runs with a concurrency limit, check each one, and stop as soon as a data-quality check fails.

When I would not use this pattern

Not every workflow belongs behind an agent. I would skip it in three cases:

Scheduled or event-driven jobs. A nightly run at 02:00 UTC, or a run triggered by an S3 upload, should be an EventBridge rule that starts Step Functions. Adding a model there only adds cost, latency, and another way to fail.
High-volume, low-latency streams. Thousands of events per second with tight latency targets belong in Kinesis, Kafka, Lambda, or Express workflows with deterministic routing. Agent tool calls are built for control-plane decisions at human speed.
Simple read-only questions. "What were total sales in the Northeast yesterday?" is a query, not a workflow. Send it to a text-to-SQL tool on Athena, Redshift, or a semantic layer.

6. Before you go to production

Before this goes from a sandbox into a real account, I would check these:

[ ] Standard workflows for anything an agent can reach, so the history is durable and queryable.
[ ] Pin stateMachineArn in the Gateway target for every operation that takes it.
[ ] Two-tier IAM. State-machine ARN for state-machine actions, the :execution:Name:* pattern for execution actions. No wildcard states:*.
[ ] Timeouts, retries, and catches in the ASL. Use TimeoutSeconds, Retry with backoff for transient errors, and a Catch that alerts and ends in a Fail state.
[ ] Validate inputs at the start of the state machine, before any expensive compute.
[ ] Log every MCP call on the Gateway, with the caller and the payload.
[ ] CloudTrail and CloudWatch Logs enabled for Step Functions, for forensics later.
[ ] Separate environments. Separate Gateway targets and roles for dev, staging, and production, so a staging agent cannot touch production.
[ ] Rate limits. Guard Glue and warehouse capacity against runaway loops with concurrency checks, idempotent names, or throttling.
[ ] Human approval for risky work. Truncations, full reloads, and expensive cluster starts should wait on a waitForTaskToken callback that a person approves in Slack or through a signed link.

7. Closing thoughts

The instinct, when building agents, has been to give models direct access to cloud infrastructure. I understand the appeal. But a language model makes a poor distributed transaction coordinator. When an agent touches raw connectors, retries, idempotency, and audit trails turn into text in a prompt, and that is hard to debug at 3 a.m.

Putting Step Functions Standard workflows behind AgentCore Gateway gives you a cleaner split:

The agent decides when and what. It reads the request and fills in the parameters. Step Functions decides how: retries, error handling, and state history.
Boundaries beat instructions. A prompt is a request. IAM scoping, ARN pinning, and validation in the state machine are rules.
You can see what happened. Standard workflows keep a full history, so the agent can answer accurately without anyone digging through log streams.

AI agents do not remove the need for data engineering discipline. If anything, they make it more important. Our job is to design the state machines and the security boundaries that let an agent run a production platform safely, and to be honest with ourselves about where the agent should stop.

Resources & Demo Repository

The complete deployable project, including the SAM template, least-privilege IAM roles, the PySpark Glue job, and the MCP client, is in this repository:

State machine definition: state-machine/data-pipeline.asl.json
AgentCore Gateway config: gateway/target.json
SAM template: template.yaml
Python agent client: agent/invoke_agent.py
Setup guide: docs/env-setup.md

Preparing for Re:Invent 2026

Nadtakan Futhoem (Nad) — Sun, 08 Nov 2026 00:32:40 +0000

It's one of those times again! There is one big conference hosted by AWS that I always look forward to. Yep, I'm talking about AWS Re:Invent.

Every year, I'm excited to see what new services, tools, and announcements AWS has in store.

Out of curiosity, I did some research and discovered that the conference started back in 2012. That took me back to 2018, when I was just getting started with cloud computing. Fast forward to today, 14 years since the first one, and the conference is still going strong.

I attended for the first time last year, and it was an incredible experience!

Being surrounded by thousands of people, all sharing the same excitement, ambition, and curiosity about technology, was something else. I have to admit, I felt a little overwhelmed at first, but in the best possible way.

There was so much to see, learn, and explore. After experiencing it firsthand, I learned that attending is one thing, but preparing for it beforehand is another.

So, if you are planning to attend AWS Re:Invent for the first time, or even if you've been before, I'd love to share a few things I learned from my experience to help you prepare and make the most of your time there.

Along with all the excitement about attending sessions, workshops, and exploring what's new, I have one more thing on my agenda.

I am excited about something I've been planning for a while. I have one mission: run a 5K.

It's happening on Wednesday at 6 AM. Yes, you read that right. I'm swapping my usual conference morning routine for a 5K race!

I'm pretty excited, and it's definitely one of the things I'm looking forward to most.

I went ahead and created a training plan, of course, using ChatGPT. I set an ambitious goal: run a 5K in 30 minutes. I even wanted to run every day. Sounds ambitious, but I thought it was possible!

My training plan included easy runs, tempo runs to improve speed, and long runs to build endurance. I ended up running three times a week, adding strength training and taking only one rest day per week.

A month went by, and I felt great. I'd run during my lunch break or in the evening. I kept tracking my time and pace. Always looking for ways to improve.

But then I started noticing something wasn't quite right. My body was getting tired, and I was not giving it enough time to recover. I was pushing myself too hard and had not accounted for how much recovery my body actually needed.

I started reflecting on what I was doing wrong. My body was telling me something, and I needed to listen.

I stopped running completely for two weeks. Instead, I started doing yoga and gentle stretching. I also stopped lifting weights for a while. I wanted to be gentle to my body. I also spent time in the sauna and hot tub to relax and unwind.

Looking back, I realized I had overlooked a few things. I was not warming up before my runs or cooling down afterward. I was not stretching either. I had been so focused on running faster, and I forgot about taking care of my body along the way.

As of today, I feel that taking a step back was the right decision.

Now, I'm planning my training again, this time with a more balanced approach and the time I have left before race day.

My new plan includes two running days, one strength training day, two swimming days, and two rest days each week. I want to focus on quality rather than simply doing more.

There are a few things I am paying more attention to this time.

Warm up before running. A dynamic warm-up is important to prepare my muscles and joints before hitting the run.
Paying attention to my running form. Engage my core, maintain a comfortable stride, and be mindful of how my feet land. I also want to improve my balance and lower body strength because running involves repeatedly supporting my body on one leg at a time.
Make strength training count. Focus on strengthening my lower body and improving stability and balance.
Add swimming. It's a great way to get more low-impact, full-body exercise while giving my legs a break from running.
Fuel my body properly. Make sure I'm eating enough, especially getting enough carbohydrates and protein to support my training.
Get enough sleep. Recovery doesn't happen only at the gym or on the running trail. I had a great run, as always, after getting good sleep last night, so a good night's sleep matters too.
Stay hydrated. Drink enough water and consider electrolytes(coconut water) when needed, especially after longer or sweatier workouts.

If you've made it this far, I hope you've found something useful in my little training journey.

But most importantly, don't forget to listen to your body. It's easy to get caught up in chasing a goal and forget that recovery or rest is just as important as training.

And if you are running the 5K at AWS Re:Invent, I'll see you at the starting line. Come say hi!

See ya!

Kubernetes CPU Requests in Amazon EKS Auto Mode

Jesus Aguirre — Sat, 07 Nov 2026 15:51:40 +0000

A Pod can remain Pending even when an Amazon EKS cluster has healthy nodes. One common reason is that no eligible node has enough allocatable CPU to satisfy the Pod's request.

In this hands-on experiment, we first deploy a Pod that requests 500m CPU. Kubernetes schedules it on an existing c6a.large node. We then recreate the Pod with a request of 4 CPUs. The existing node cannot accommodate that request, so the Pod becomes unschedulable and EKS Auto Mode provisions additional capacity.

The experiment demonstrates how a Kubernetes resource request can influence node provisioning. It does not demonstrate scaling based on actual CPU utilization: the container runs sleep 3600, which does not consume four CPUs.

What we will observe

The experiment follows these steps:

Deploy a Pod requesting 500m CPU and 128Mi memory.
Confirm that it runs on an existing c6a.large instance.
Recreate the Pod with a 4 CPU request, leaving memory unchanged.
Observe the FailedScheduling event with reason Insufficient cpu.
Inspect the NodeClaim created by EKS Auto Mode.
Verify that a new c6a.2xlarge instance joins the cluster and reaches Ready.
Confirm that the Pod is scheduled on the new node.

The c6a.2xlarge is the instance type observed in this run, not a guaranteed selection. EKS Auto Mode chooses capacity that satisfies the workload and the constraints of the applicable NodePool, subject to available offerings and capacity.

How scheduling leads to provisioning

Kubernetes scheduling and infrastructure provisioning are related, but they are separate operations.

The Kubernetes Scheduler assigns a Pod to a Node. To make that decision, it checks whether the node has enough remaining allocatable resources to satisfy the Pod's requests and whether the Pod meets the node's other scheduling constraints.

A simplified CPU accounting model is:

CPU remaining = CPU allocatable - Total CPU requests of scheduled Pods

This is a conceptual model, not the complete scheduling algorithm. The Scheduler also considers memory, ephemeral storage, Pod count, node selectors, affinity, taints and tolerations, topology, volumes, and other constraints.

Three node values are useful when investigating a scheduling decision:

Capacity is the total resource capacity reported for the node.
Allocatable is the amount Kubernetes makes available to Pods after node-level reservations.
Allocated resources in kubectl describe node summarizes the requests and limits of Pods assigned to that node.

An EC2 instance's advertised vCPU count is not the same as the CPU available to new Pods. System and kubelet reservations reduce allocatable capacity, and existing Pod requests consume part of that capacity from the Scheduler's perspective. Actual CPU utilization is not a substitute for request accounting.

When no eligible node can satisfy a Pod, the Scheduler reports a FailedScheduling event. Insufficient cpu means the CPU fit check failed for the eligible nodes at that time. It does not mean that EC2 failed to launch an instance, nor does it describe how busy the CPUs are.

EKS Auto Mode can respond to an unschedulable Pod by provisioning more compute capacity when an enabled NodePool can satisfy the Pod's requirements. The provisioning system evaluates the workload against NodePool constraints and available infrastructure; Kubernetes does not request a particular EC2 instance type directly.

A NodePool defines constraints and disruption settings for nodes that can be provisioned. A NodeClaim is the Kubernetes resource representing a request for a node. After capacity is launched, the instance must register as a Kubernetes Node and become ready before the Scheduler can place the pending Pod on it.

How EKS Auto Mode Provisions New Compute Capacity

Experiment environment

The original experiment used this environment:

Component	Recorded configuration
AWS Region	`us-east-1`
Amazon EKS	Auto Mode enabled
Kubernetes version	`1.37`
NodePools	`general-purpose`, `system`
Workload Pod	`cpu-test`
Container image	`public.ecr.aws/docker/library/busybox:1.36`
Container command	`sleep 3600`
EC2 capacity observed	On-Demand

The handoff records the Kubernetes version as 1.37, but does not include the cluster's exact EKS platform version. It also does not preserve the full live NodePool and NodeClass YAML, Availability Zone, architecture metadata, NodePool limits, or every effective instance constraint. Inspect these values in your own cluster rather than assuming they match the original run.

Prerequisites

You need:

An existing EKS cluster with Auto Mode enabled.
AWS CLI and kubectl configured for the cluster.
Kubernetes permissions to create and delete Pods and inspect Pods, Events, Nodes, NodePools, and NodeClaims.
AWS IAM permission ec2:DescribeInstances if you want to query the EC2 instance from the CLI.

The identity used by the operator is separate from the roles used by EKS and the nodes. The cluster IAM role grants permissions to cluster-managed components; the node IAM role is assumed by EC2 nodes. EKS access entries and Kubernetes RBAC determine which IAM principals can access Kubernetes resources. The operator's AWS CLI identity separately needs permission to call EC2 APIs.

Set the cluster details and connect

export AWS_REGION=us-east-1
export CLUSTER_NAME="your-cluster-name"

aws eks update-kubeconfig \
  --region "$AWS_REGION" \
  --name "$CLUSTER_NAME"

kubectl cluster-info
kubectl get nodes

Check that the expected Auto Mode NodePools exist

kubectl get nodepools
kubectl describe nodepool general-purpose

The experiment used the built-in general-purpose and system pools. AWS documents the built-in pools as managed configurations; their constraints influence which instance offerings are eligible. The original handoff does not include the complete effective configuration at the time of the experiment, so we cannot attribute the c6a.2xlarge selection to a specific individual constraint.

Step 1: Schedule a Pod with a `500m` CPU request

Create cpu-test-500m.yaml

apiVersion: v1
kind: Pod
metadata:
  name: cpu-test
spec:
  restartPolicy: Never
  containers:
    - name: busybox
      image: public.ecr.aws/docker/library/busybox:1.36
      command:
        - sleep
        - "3600"
      resources:
        requests:
          cpu: "500m"
          memory: "128Mi"

Apply the manifest and inspect the Pod

kubectl apply -f cpu-test-500m.yaml
kubectl get pod cpu-test -o wide
kubectl describe pod cpu-test

Expected outcome: the Pod reaches Running and is assigned to a node with enough allocatable CPU and memory.

Observed in the experiment: the Pod ran on an EC2 c6a.large instance. The original terminal transcript was not preserved, so the instance type is reported as a documented observation rather than a verbatim command output.

Record the assigned Kubernetes node

NODE_NAME=$(kubectl get pod cpu-test \
  -o jsonpath='{.spec.nodeName}')

echo "$NODE_NAME"

Inspect its resource accounting

kubectl describe node "$NODE_NAME"

Review the Capacity, Allocatable, and Allocated resources sections. The first two show the node's reported total and Pod-assignable resources; the last shows the requests and limits of the Pods already assigned to it.

The original experiment did not preserve the numeric output of kubectl describe node. We therefore do not infer the exact allocatable CPU on the c6a.large from its EC2 vCPU specification.

Step 2: Increase the CPU request to four CPUs

Now change only the CPU request. Keep the memory request, image, and command the same.

Create cpu-test-4cpu.yaml

apiVersion: v1
kind: Pod
metadata:
  name: cpu-test
spec:
  restartPolicy: Never
  containers:
    - name: busybox
      image: public.ecr.aws/docker/library/busybox:1.36
      command:
        - sleep
        - "3600"
      resources:
        requests:
          cpu: "4"
          memory: "128Mi"

Delete the first Pod and create the new one

kubectl delete pod cpu-test
kubectl apply -f cpu-test-4cpu.yaml

kubectl get pod cpu-test -o wide
kubectl describe pod cpu-test

Observed in the experiment: the Pod remained Pending. Its events recorded FailedScheduling with reason Insufficient cpu.

The original complete event message was not retained; this is a summary of the recorded observation, not a verbatim terminal transcript.

Expected outcome: while no eligible node can satisfy the Pod's request and other scheduling constraints, the Pod remains pending. Exact event wording and counts can vary.

The request changed from 500m (half a CPU) to 4 CPUs, while memory remained at 128Mi. This makes CPU request the workload variable changed in the experiment.

The key point is that a request is used for scheduling even when the container is not consuming that amount of CPU at runtime. The sleep 3600 process does not generate four CPUs of load.

The experiment is about requested resources and scheduling feasibility, not utilization-based scaling.

Step 3: Inspect the NodeClaim created by EKS Auto Mode

When a Pod is unschedulable, EKS Auto Mode can provision additional capacity if an applicable NodePool can meet the Pod's requirements and provisioning is not blocked by constraints or infrastructure availability.

List the NodeClaims

kubectl get nodeclaims -o wide

Inspect the NodeClaim created during the experiment

kubectl describe nodeclaim <nodeclaim-name>

Observed in the experiment: EKS Auto Mode created a NodeClaim in the general-purpose pool. Its generated name was general-purpose-rf7fr. This name is lab evidence only; NodeClaim names are generated and should not be hard-coded.

A NodeClaim provides the Kubernetes-side record of the capacity request and its provisioning lifecycle. Inspect its conditions and, when available, its status.providerID. A NodeClaim being present does not by itself mean that the EC2 instance has registered as a Ready Kubernetes Node.

The experiment also recorded that the original node entered termination with reason Empty. Preserve that as a separate lifecycle observation: the available evidence does not establish that the change to the CPU request directly caused the old node's termination.

Step 4: Verify the new node and the Pod placement

List the nodes

kubectl get nodes -o wide

Observed in the experiment: EKS Auto Mode provisioned a new c6a.2xlarge EC2 instance.

The instance was On-Demand and the corresponding Kubernetes Node reached Ready. The Pod was then scheduled successfully.

A c6a.2xlarge has eight advertised vCPUs, but those eight vCPUs should not be treated as eight CPUs of Pod-allocatable capacity.

Check the registered node's Kubernetes resource values:

NODE_NAME=$(kubectl get pod cpu-test \
  -o jsonpath='{.spec.nodeName}')

kubectl describe node "$NODE_NAME"
kubectl get pod cpu-test -o wide

Expected outcome: the Pod reaches Running and its assigned node is the newly provisioned node. The exact instance type is not guaranteed; another compatible offering may be selected in a different run.

Correlate the Pod, Node, NodeClaim, and EC2 instance

The Kubernetes Node's spec.providerID is the link to the cloud instance. For an AWS-backed node, the final component of the provider ID is the EC2 instance ID.

First, get the node assigned to the Pod and its provider ID

NODE_NAME=$(kubectl get pod cpu-test \
  -o jsonpath='{.spec.nodeName}')

PROVIDER_ID=$(kubectl get node "$NODE_NAME" \
  -o jsonpath='{.spec.providerID}')

INSTANCE_ID="${PROVIDER_ID##*/}"

printf 'Node: %s\nProvider ID: %s\nInstance ID: %s\n' \
  "$NODE_NAME" "$PROVIDER_ID" "$INSTANCE_ID"

Find the NodeClaim with the matching provider ID. This command uses jq

NODECLAIM_NAME=$(kubectl get nodeclaims -o json | \
  jq -r --arg provider_id "$PROVIDER_ID" \
  '.items[] | select(.status.providerID == $provider_id) | .metadata.name')

printf 'NodeClaim: %s\n' "$NODECLAIM_NAME"
kubectl describe nodeclaim "$NODECLAIM_NAME"

If the result is empty, the NodeClaim's provider ID may not yet be populated. Inspect kubectl get nodeclaims -o wide and retry after the resource progresses.

Finally, query EC2

aws ec2 describe-instances \
  --region "$AWS_REGION" \
  --instance-ids "$INSTANCE_ID" \
  --query 'Reservations[].Instances[].{
    InstanceId:InstanceId,
    InstanceType:InstanceType,
    State:State.Name,
    Architecture:Architecture,
    AvailabilityZone:Placement.AvailabilityZone
  }'

This command requires the operator's AWS identity to have ec2:DescribeInstances. The Kubernetes commands require read access to Pods, Nodes, and NodeClaims. These are separate permission systems.

The experiment recorded the new instance as c6a.2xlarge with On-Demand capacity. The instance ID and complete original AWS CLI output were not retained in the article handoff, so the reusable commands discover the ID dynamically rather than embedding a lab-specific value.

Mapping Kubernetes Resources to EC2 Instances

Using a custom NodePool for a workload class

The experiment used the built-in general-purpose NodePool. If you need to express workload-specific constraints, you can create a custom NodePool and give its nodes a label.

For example, this NodePool labels its nodes with workload-class: cpu-oriented

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: cpu-oriented
spec:
  template:
    metadata:
      labels:
        workload-class: cpu-oriented
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: eks.amazonaws.com/instance-category
          operator: In
          values:
            - c
        - key: eks.amazonaws.com/instance-cpu
          operator: In
          values:
            - "8"
            - "16"
            - "32"
        - key: kubernetes.io/arch
          operator: In
          values:
            - amd64
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - on-demand

This is an illustrative custom configuration, not the NodePool used in the original experiment. It assumes the cluster has the Auto Mode default NodeClass. Check the target cluster's available NodeClass and supported requirements before applying it.

How NodePool Constraints Shape Eligible Compute Capacity

A Pod can target nodes from this pool using a nodeSelector

spec:
  nodeSelector:
    workload-class: cpu-oriented

The selector value must match the label defined under spec.template.metadata.labels in the NodePool. If the values do not match, the Pod will not be eligible for nodes from this pool.

The NodePool constrains the set of eligible capacity; it does not guarantee that a particular EC2 size will be selected. Even with these requirements, the chosen instance depends on the Pod's resource requests, all applicable constraints, and available capacity.

What the experiment teaches us

The two configurations produced the following documented results

	Initial Pod	Updated Pod
CPU request	`500m`	`4`
Memory request	`128Mi`	`128Mi`
Observed EC2 instance	`c6a.large`	`c6a.2xlarge`
Advertised vCPUs	2	8
Scheduling result	Scheduled on existing node	Initially `FailedScheduling`; scheduled after new node became Ready

The important change was not an increase in measured CPU consumption. It was an increase in the CPU request that the Scheduler had to accommodate.

The sequence demonstrates the interaction between:

Pod requests, which describe the resources Kubernetes must account for when scheduling.
The Scheduler, which determines whether an eligible node can accept the Pod.
NodePools, which constrain the kinds of nodes the provisioning system can create.
NodeClaims, which represent requested node capacity.
EC2, which supplies the compute instance.
Node registration and readiness, which make the new capacity available to Kubernetes.

The old node's Empty termination reason and the transient InvalidDiskCapacity warnings observed during node startup are additional lab observations. Both nodes subsequently became Ready; the available evidence does not establish a more specific cause for either observation, so neither should be treated as proof of a persistent issue or as a direct consequence of the CPU request change.

Practical considerations

CPU requests affect scheduling and can therefore affect the infrastructure required to run a workload. A request that is much higher than the workload needs can lead to inefficient placement or larger infrastructure. A request that is too low can increase packing density but may contribute to contention when workloads need more CPU concurrently.

Use application measurements and realistic workload testing to set requests. For production, also consider replicas, disruption budgets, topology spread, graceful termination, and the cost implications of the eligible instance types.

The experiment used a single Pod to isolate scheduling behavior. It does not demonstrate high availability, performance under load, or the minimum instance size required for a four-CPU request.

Clean up

Delete the workload:

kubectl delete pod cpu-test

If you created a custom NodePool for this exercise, remove it when it is no longer needed:

kubectl delete nodepool cpu-oriented

Only delete the EKS cluster if it was created exclusively for this experiment and you have confirmed that deletion is intended. Review related AWS resources and billing independently; deleting a cluster should not be assumed to remove every shared networking or IAM resource.

The original lab cleanup removed the Pod and EKS cluster. The observed EC2 instances reached terminated; shared networking resources and IAM roles and policies were preserved. The handoff does not establish that every auxiliary resource, such as ENIs or security groups, was removed.

Conclusion

A Kubernetes CPU request is more than a description of an application's desired resources: it is an input to scheduling. When a Pod's request changes from 500m to 4 CPUs, a node that previously accommodated it may no longer be eligible.

In this experiment, the 500m Pod ran on an existing c6a.large. After the request increased to 4, the Pod became pending with FailedScheduling and Insufficient cpu. EKS Auto Mode created a NodeClaim, provisioned a new c6a.2xlarge instance, and made the node Ready. The Pod was then scheduled successfully.

The instance type is specific to this run, not a fixed outcome. The reusable lesson is to trace the entire path: inspect the Pod's requests and scheduling events, review the matching NodePool, follow the NodeClaim, correlate its provider ID with the Kubernetes Node and EC2 instance, and confirm that the node is Ready before concluding that the workload has usable capacity.

NexOps: Build CloudOps Agent Using Strands+Bedrock+Terraform

Anirban Das — Sat, 07 Nov 2026 12:26:15 +0000

Overview of NexOps

NexOps is a cloud operations assistant that lets users ask questions about AWS infrastructure in plain language. It combines a lightweight web interface with an AI agent that can inspect selected AWS services, gather operational evidence, and explain findings in a clear and practical way.

The backend uses the Strands Agents framework and Amazon Bedrock. For each chat request, it creates a fresh agent instance and gives it a set of service-specific tools implemented with Boto3. The application is designed to support operations teams with investigation and routine administration while keeping the scope of available actions explicit.

Ask questions about infrastructure, health, security posture, and costs.
Gather details from supported AWS services through dedicated tools.
Receive summaries that distinguish observed facts from possible causes.
Use gated EC2 lifecycle and Systems Manager operations when the deployment is configured to allow them.

NexOps is not an unrestricted AWS command runner. Tool coverage depends on the AWS account, region, IAM permissions, and service prerequisites available to the running application.

Business Problems

Cloud teams often need to inspect several consoles and services to answer a basic operational question. During an incident, switching between instance state, monitoring data, logs, alarms, and recent changes can slow down diagnosis and make it harder to share a concise summary with the rest of the team.

NexOps brings selected checks together behind a conversational interface. It can help reduce repetitive lookup work and make operational information more accessible, while leaving decisions and approvals with the people responsible for the environment.

Scattered operational data: Relevant details may live in EC2, CloudWatch, CloudTrail, AWS Config, IAM, and other service views.
Slow first-pass investigation: Engineers may repeat the same inventory and health checks before they can focus on likely causes.
Hard-to-summarize findings: Raw service responses can be lengthy and difficult to turn into a useful incident update.
Security review overhead: Teams need a convenient way to inspect selected IAM, encryption, access, and network controls.
Cost visibility: Service costs and low-utilization candidates can be difficult to spot during routine operations.

The application supports these workflows by collecting targeted evidence and presenting it in natural language. Findings such as low CPU, aged access keys, or broad policy statements are review signals; they are not automatic judgments or remediation decisions.

Architecture

NexOps separates the user interface, request routing, and AWS operations into deployable components. The browser loads a static frontend served by NGINX on ECS Fargate. Chat requests pass through Amazon API Gateway and a private integration to the backend Application Load Balancer, which forwards them to the FastAPI service running in ECS Fargate.

For each chat request, FastAPI validates the input and optional region, constructs a Strands agent backed by Amazon Bedrock, and returns the agent's response. The agent selects from registered Python tools; those tools use Boto3 to call supported AWS APIs. The AWS identity available to the task, typically through its ECS task role, determines which calls succeed.

Frontend: Static HTML, CSS, and JavaScript served by NGINX in a Fargate task.
Request entry: Public frontend Application Load Balancer; API Gateway HTTP API for chat traffic.
Private backend path: API Gateway VPC Link to an internal Application Load Balancer and backend ECS service.
Agent service: FastAPI endpoints, Strands orchestration, and service-specific Boto3 tools.
Model: Amazon Bedrock, using the configured model ID (Nova Micro by default in the application).
Infrastructure: Terraform definitions for ECR, ECS, load balancers, API Gateway, logging, and supporting resources.

The backend creates a new agent for every request, so chat history is not persisted across requests. The frontend and API deployment settings, including API URL, CORS origins, authentication, and network access, must be aligned for a production deployment.

What This Application Does

NexOps translates an operator's question into one or more targeted tool calls and then summarizes the returned evidence. The registered tools cover selected parts of AWS rather than every service or every resource in an account.

Inventory: Inspect supported EC2, EBS, VPC, security group, S3, Lambda, RDS, and ECS resources.
Monitoring and troubleshooting: Query CloudWatch metrics, alarms, log groups, and logs; inspect EC2 status and console output; and gather recent CloudTrail changes.
Security and compliance review: Inspect selected IAM users, groups, roles, policies, MFA and access-key metadata, security group rules, encryption settings, and AWS Config results.
Cost analysis: Review Cost Explorer service costs, supported resource-level EC2 costs, and low-CPU candidates.
Approved operations: Start, stop, or reboot EC2 instances and invoke restricted Systems Manager actions when the relevant application gates, allowlists, IAM permissions, and AWS prerequisites are satisfied.
Result verification: Where supported, inspect the resulting state after an operation and report whether it completed, failed, or remains in progress.

The agent is instructed to use evidence for live AWS claims, report the region and scope it checked, and identify uncertainty. It does not provide a generic AWS API executor, IAM write tools, resource deletion, or arbitrary shell access. A tool being available also does not mean that the task role has permission to use it.

Functional Workflow

The workflow begins when a user enters a question in the web application. The frontend sends the message and, optionally, a region to the API Gateway endpoint. API Gateway routes the request over its VPC Link to the backend service.

FastAPI validates the request and applies the configured API-token behavior. It then creates a Strands agent using the configured Bedrock model and gives the agent the supported tool set and operational system prompt. The agent chooses relevant read-only checks first for investigative requests, calls AWS through Boto3, and uses the returned results to produce a concise response.

The user submits a natural-language question and optional AWS region.
The frontend posts the request to the API Gateway chat route.
API Gateway forwards it through the VPC Link and internal load balancer to FastAPI.
The backend validates the input and creates a request-scoped agent.
The agent selects the narrowest relevant AWS tools and gathers results.
For a clearly authorized state change, the agent may call a gated operation if deployment policy permits it.
The agent summarizes evidence, limitations, errors, and action status for the frontend.

The health endpoint provides a liveness response and does not contact AWS or Bedrock. Chat calls do not share persisted conversation history. An accepted AWS operation is reported as submitted until a verification check confirms its resulting state.

AWS Components

The infrastructure uses managed AWS services to host the frontend and backend, expose their HTTP entry points, and connect the agent to AWS operations and model inference.

Amazon ECS on AWS Fargate: Runs the NGINX frontend container and FastAPI backend container without managing EC2 hosts.
Amazon ECR: Stores the frontend and backend container images; the Terraform configuration enables image scanning on push.
Elastic Load Balancing (Application Load Balancer): A public-facing frontend ALB serves the web tier, while an internal ALB receives backend traffic from API Gateway.
Amazon API Gateway HTTP API: Exposes the chat route and connects privately to the backend through a VPC Link.
Amazon Bedrock: Provides model inference for the Strands agent. The model ID and Bedrock region are runtime configuration.
AWS IAM: The ECS task role supplies the application's AWS permissions; the execution role supports ECS task startup integrations such as image pulls and log delivery.
Amazon CloudWatch Logs: Receives container logs and supports the application's monitoring and log-search tools.
AWS KMS: Supplies keys referenced by ECS execute-command logging configuration.
Amazon VPC: Places ECS tasks and the internal backend path in private subnets, with security groups controlling connectivity.
Amazon S3: Stores Terraform state through the configured remote backend; the state bucket must already exist and be accessible.
AWS Systems Manager, AWS Config, AWS CloudTrail, and Cost Explorer: Supply operational, compliance, change-history, and cost data when enabled and permitted.

Security Groups

Public Frontend ALB Security Group:

Name: Public-CloudOps-Webapp-LB
Inbound: HTTPS from Anywhere
Outbound: Default

Frontend ECS Security Group:

Name: CloudOps-Web-App
Inbound: TCP at 8080 port from Public-CloudOps-Webapp-LB
Outbound: Default

VPC Link Security Group:

Name: cloudops-vpc-link-sg
Inbound: HTTPS from CloudOps-Web-App
Outbound: HTTPS to ALBSG (Internal ALB)

Internal ALB Security Group:

Name: InternalALBSG
Inbound: HTTPS from cloudops-vpc-link-sg
Outbound: Default

Backend ECS Security Group:

Name: StrandsAgentSG
Inbound: TCP at 8000 port from InternalALBSG
Outbound: Default

Some resources and IAM roles are looked up by Terraform rather than created by these modules. Operators must review the configured role policies, network rules, model access, and service prerequisites for their own AWS environment.

Implementation Steps

NexOps is implemented as two application components: a Python backend and a static web frontend. The backend builds a request-scoped Strands agent and registers the AWS tools it can use. The frontend is served by NGINX and proxies chat requests to the API Gateway endpoint configured for its container.

Complete Repository Folder Structure

The tree below shows the files tracked in the repository and how the frontend, backend, AWS tools, Terraform infrastructure, and CI pipelines are organized.

strands-cloudops-agent/
├── .gitlab-ci.yml
├── .gitlab-ci-backend.yml
├── .gitlab-ci-frontend.yml
├── Backend/
│   ├── .dockerignore
│   ├── .env.example
│   ├── .gitignore
│   ├── Dockerfile.backend
│   ├── README.md
│   ├── app.py
│   ├── agent/
│   │   ├── orchestrator.py
│   │   └── system_prompt.py
│   ├── services/
│   │   ├── __init__.py
│   │   ├── aws_clients.py
│   │   ├── policy.py
│   │   └── result_formatter.py
│   ├── tools/
│   │   ├── __init__.py
│   │   ├── cloudtrail_tools.py
│   │   ├── cloudwatch_alarms.py
│   │   ├── cloudwatch_logs.py
│   │   ├── cloudwatch_metrics.py
│   │   ├── cloudwatch_tools.py
│   │   ├── config_tools.py
│   │   ├── cost_optimization.py
│   │   ├── ebs_tools.py
│   │   ├── ec2_inventory.py
│   │   ├── ec2_lifecycle.py
│   │   ├── ec2_tools.py
│   │   ├── ecs_tools.py
│   │   ├── iam_tools.py
│   │   ├── lambda_tools.py
│   │   ├── network_tools.py
│   │   ├── policy_checks.py
│   │   ├── rds_tools.py
│   │   ├── remediation.py
│   │   ├── s3_tools.py
│   │   └── ssm_operations.py
│   ├── Backend_Infra/
│   │   ├── alb.tf
│   │   ├── apigateway.tf
│   │   ├── data.tf
│   │   ├── ecr.tf
│   │   ├── ecs.tf
│   │   ├── output.tf
│   │   ├── provider.tf
│   │   ├── terraform.tfvars
│   │   └── variable.tf
│   └── requirements.txt
├── Frontend/
│   ├── .dockerignore
│   ├── .gitignore
│   ├── Dockerfile.frontend
│   ├── README.md
│   ├── cloudopsweb/
│   │   ├── README.md
│   │   ├── app.js
│   │   ├── index.html
│   │   ├── nginx.conf
│   │   └── styles.css
│   └── Frontend_Infra/
│       ├── alb.tf
│       ├── data.tf
│       ├── ecr.tf
│       ├── ecs.tf
│       ├── output.tf
│       ├── provider.tf
│       ├── terraform.tfvars
│       └── variable.tf
├── PROJECT_BLOG.md
└── __pycache__/
    └── app.cpython-313.pyc

1. Configure the AWS environment: Choose the target account and region, confirm Bedrock model access, and prepare the VPC, subnets, security groups, Terraform state bucket, and required IAM roles.
2. Configure the backend: Set the Bedrock model and region, default AWS region, and any account or region restrictions. Grant the ECS task role only the AWS API permissions required by the tools to be enabled. Backend/agent/orchestrator.py constructs the Bedrock model and passes it, the registered tools, and system prompt to Strands:

model_id = os.getenv("BEDROCK_MODEL_ID", "us.amazon.nova-micro-v1:0").strip()
bedrock_region = os.getenv(
    "BEDROCK_REGION", os.getenv("AWS_REGION", "us-east-1")
).strip()
model = BedrockModel(
    model_id=model_id,
    region_name=bedrock_region,
    max_tokens=int(os.getenv("BEDROCK_MAX_TOKENS", "2048")),
)
return Agent(
    model=model,
    tools=TOOLS,
    system_prompt=SYSTEM_PROMPT,
    name="cloudops-agent",
)

3. Package the backend: Backend/Dockerfile.backend installs Python dependencies and runs the FastAPI service with Uvicorn on port 8000:

FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt ./requirements.txt
RUN python -m pip install --no-cache-dir -r requirements.txt

COPY . .
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]

4. Package the frontend: Frontend/Dockerfile.frontend uses NGINX to serve the browser UI. NGINX proxies /api/chat to the API Gateway endpoint provided to the frontend task as API_GATEWAY_URL:

FROM nginx:alpine

COPY cloudopsweb/nginx.conf /etc/nginx/templates/default.conf.template
COPY cloudopsweb/index.html /usr/share/nginx/html/
COPY cloudopsweb/styles.css /usr/share/nginx/html/
COPY cloudopsweb/app.js /usr/share/nginx/html/

EXPOSE 8080
CMD ["nginx", "-g", "daemon off;"]

5. Configure operational safeguards: Keep mutations disabled unless they are deliberately required. When enabled, restrict them to approved account IDs and regions; separately allowlist reviewed SSM documents. Configure API authentication and production mode before exposing chat to untrusted clients.
6. Build and publish images: Build the backend and frontend containers from their respective Dockerfiles, then push both images to their ECR repositories.
7. Deploy infrastructure: Review and apply Terraform in Backend/Backend_Infra and Frontend/Frontend_Infra. Confirm the looked-up roles, network resources, image names, frontend API endpoint, CORS origins, and load balancer health checks.
8. Verify the application path: Check backend /health, confirm ECS tasks become healthy, load the frontend through its ALB, and submit a chat request through API Gateway in a non-production environment.
9. Validate tool access and operations: Test read-only inventory and monitoring calls first. If lifecycle or SSM actions are needed, validate the gates, exact allowlists, IAM permissions, target state, and post-action verification behavior.
10. Operate and improve: Monitor container logs and service health, review IAM access periodically, and update model, infrastructure, and tool configuration as requirements change.

Registered Agent Tools

These are the functions registered in Backend/agent/orchestrator.py. The agent selects tools based on each request; it does not call every tool for every chat. Access also depends on IAM permissions and AWS service configuration.

EC2 and resource inventory: list_ec2_instances, get_ec2_instance_details, get_ec2_instance_state, get_ec2_instance_status, get_ec2_console_output, list_aws_resources, get_resource_tags.
CloudWatch: get_cloudwatch_metrics, list_cloudwatch_alarms, get_cloudwatch_alarm_history, search_cloudwatch_logs, list_cloudwatch_log_groups, get_cloudwatch_dashboard.
CloudTrail and AWS Config: get_recent_account_events, get_resource_event_history, list_config_rules, get_config_resource_history, get_config_compliance.
IAM inspection: get_iam_role_details, list_iam_role_policies, simulate_iam_policy, list_iam_users, list_iam_groups, get_iam_user_details, get_iam_group_details, get_group_members, list_user_groups, get_user_attached_policies, get_group_attached_policies, get_user_inline_policies, get_group_inline_policies, list_user_access_keys, list_user_mfa_devices, analyze_group_permissions, identify_empty_iam_groups, check_users_without_mfa, identify_groups_with_admin_access, check_user_access_key_age.
VPC and networking: list_vpcs, list_subnets, list_security_groups, list_route_tables, list_network_acls, check_security_group_rules.
EBS: list_ebs_volumes, get_ebs_volume_details, check_ebs_encryption, list_ebs_snapshots.
RDS: list_rds_instances, get_rds_instance_details, check_rds_encryption, get_rds_events.
S3: list_s3_buckets, get_s3_bucket_details, check_s3_encryption, check_s3_public_access, list_s3_lifecycle_rules.
Lambda: list_lambda_functions, get_lambda_function_details, get_lambda_function_errors, list_lambda_event_sources.
ECS: list_ecs_clusters, list_ecs_services, list_ecs_tasks, get_ecs_service_events.
Diagnosis and change verification: diagnose_ec2_health, analyze_cloudwatch_metrics, analyze_instance_logs, get_recent_resource_changes, verify_remediation_result.
Systems Manager: check_ssm_managed_status, check_patch_compliance, get_ssm_command_invocation, get_ssm_automation_execution, run_ssm_command, run_ssm_automation, restart_application_service.
EC2 lifecycle: start_ec2_instance, stop_ec2_instance, reboot_ec2_instance.
Security policy checks: check_iam_resource_policy, check_resource_encryption.
Cost analysis: get_service_costs, get_resource_costs, find_idle_resources, estimate_ec2_cost_impact.

The repository documents several deployment settings that need attention before production exposure. In particular, the backend Terraform task definition currently enables the mutation gate for a specific account and region, while it does not configure production authentication; the API Gateway CORS origin is set to http://localhost. Review and replace these values with environment-appropriate settings before deployment to a shared or public environment.

This is what frontend UI should look where users need to provide prompts -

Example: If we want to get list of running of instances name and instance types

Repository Link: https://github.com/dasanirban834/strands-cloudops-agent

Benefits

NexOps brings selected AWS operational checks into one conversational workflow. This can help engineers get to relevant evidence faster, especially when a question spans multiple services or requires a concise update for teammates.

Faster first-pass investigations: Operators can ask a question in plain language and have the agent gather relevant inventory, monitoring, log, and change-history details from supported tools.
Clearer operational summaries: Responses can turn service output into readable findings while calling out the region, scope, errors, and uncertainty involved.
Broader access to operational information: A conversational interface can make routine checks easier for team members who do not work in every AWS console each day.
Consistent review signals: Teams can use the same supported checks to spot items such as missing encryption, broad network rules, aged access keys, or low utilization for follow-up.
Controlled operational actions: Where enabled, lifecycle and Systems Manager actions are bounded by application gates, allowlists, IAM permissions, and AWS prerequisites, with verification available for supported operations.
Deployable, modular foundation: The separate frontend, backend, tools, and Terraform components provide a basis for adapting the application to an organization's account boundaries and operational practices.

These benefits depend on accurate IAM scoping, suitable AWS configuration, and human review. NexOps helps collect and explain evidence; it does not replace incident judgment, security approval, or environment-specific production controls.

Conclusion

NexOps demonstrates how a Strands-based agent can connect a natural-language interface to a focused set of AWS operations. It combines evidence gathering across supported services with readable summaries, while keeping state-changing capabilities explicitly gated.

The repository provides the core application and infrastructure needed to deploy and extend this approach. Before using it in a shared or production environment, teams should review authentication, CORS, IAM permissions, network access, Bedrock model availability, and mutation settings. With those controls configured for the target environment, NexOps can serve as a practical assistant for routine cloud operations and initial investigation.

How to Master AWS for AI?

Mursal Furqan Kumbhar — Sat, 07 Nov 2026 11:48:28 +0000

Ciao Amici 👋

Every hiking trail in the Alps has a map at the trailhead. A good one shows you the huts, the peaks and the river crossings. What it never shows you is the stretch where the path is a loose scree slope, or the hut that closes in October, or the signpost that points you towards the wrong valley. For that, you need someone who has walked it.

A few days ago a roadmap called "How to Master AWS for AI?" landed in my feed, drawn by Umang Chaudhary (@mlwithumang). It is a lovely trailhead map: one centre (Amazon Bedrock), six branches (Foundations, Data and Compute, Amazon Bedrock, AI Agents, Knowledge and AI Services, Ops and Governance) and a suggested five-step learning path along the bottom. I have built on several of these services myself, so I did the obvious thing. I walked the map against my own scars.

This is a long article on purpose. It follows the five steps in order, and for each one it gives you three things: what is verifiably true today (as of 30 October 2026), code you can run, and the places where I would move the signposts. Every code sample in this article was executed before I wrote it down, and none of it needs an AWS account or costs a cent. I will tell you exactly how I tested it, and exactly what I did not test, at the end.

What is in here

The map, and my three changes to it
Step 0: money and blast radius (budgets, token ceilings, least privilege, logging)
Step 1: foundations (reading an AccessDenied properly)
Step 2: models and prompts (the Converse API, token accounting, stop reasons)
Step 3: grounding and RAG (S3 Vectors, DynamoDB vectors, and a retrieval test you can run offline)
Step 4: agents (Strands, AgentCore, and the two rules of distributed systems that break agents)
Step 5: evaluate and govern (guardrails at the tool boundary, evaluation, audit)
A failure cheat sheet, a four-week plan, and what I did not verify

The map in one table

Here is the learning path from the roadmap, in my words:

Step	The roadmap says	What it really means
1	Foundations: IAM, KMS, CDK	Know who is allowed to do what, and keep secrets secret
2	Models and Prompts: Bedrock, Nova	Call a model and shape what it says
3	Grounding and RAG: Knowledge Bases, S3 Vectors	Make the model answer from your data, not from its imagination
4	Agents: AgentCore, Strands	Let the model take actions, safely
5	Evaluate and Govern: Guardrails, Evaluations	Prove it works, and stop it doing what it should not

It is a sound order for a first pass. My notes are about what a map cannot show, because it is a map and not a diary. Here is the same path with my three changes highlighted. I will explain each one as we go:

Step 0: money and blast radius

The roadmap lists Budgets and Cost Explorer under Core Setup, which I love, but I would pull it out of the pile and put it before everything else. AI services bill by usage, and usage is exactly what an eager experiment or a runaway loop produces. There are four layers of protection, and they do different jobs.

Layer 1: an alert (slow, but it reaches you). AWS Budgets can email you when actual spend passes a percentage of a monthly limit. Here is a budget with two alerts, at 50 and 80 percent of 10 US dollars. I did not guess the structure: I validated it against the SDK's own model for create_budget (more on that in the testing section).

{
  "Budget": {
    "BudgetName": "ai-experiments-monthly",
    "BudgetType": "COST",
    "TimeUnit": "MONTHLY",
    "BudgetLimit": { "Amount": "10", "Unit": "USD" }
  },
  "NotificationsWithSubscribers": [
    {
      "Notification": {
        "NotificationType": "ACTUAL",
        "ComparisonOperator": "GREATER_THAN",
        "Threshold": 50,
        "ThresholdType": "PERCENTAGE"
      },
      "Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "you@example.com" }]
    },
    {
      "Notification": {
        "NotificationType": "ACTUAL",
        "ComparisonOperator": "GREATER_THAN",
        "Threshold": 80,
        "ThresholdType": "PERCENTAGE"
      },
      "Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "you@example.com" }]
    }
  ]
}

Create it with client.create_budget(AccountId=..., **json.load(open("budget.json"))) using boto3 and the budgets client. Be honest about what this layer is: an alert, not a brake. It reacts to billing data, which arrives with a delay, so it tells you that you have a problem rather than stopping the problem.

Layer 2: a brake inside your code. The cheapest protection is a ceiling you enforce yourself. This tiny wrapper counts the tokens from every Converse call, refuses to call again once you pass a limit, and writes a ledger line per call so you can see where the tokens went:

# code/token_budget.py
"""A hard token ceiling around Bedrock Converse, with a local usage ledger."""
import json
import time


class BudgetExceeded(Exception):
    pass


class TokenBudget:
    def __init__(self, max_tokens: int, ledger_path: str = "usage.jsonl"):
        self.max_tokens = max_tokens
        self.used = 0
        self.ledger_path = ledger_path

    def converse(self, client, **kwargs):
        if self.used >= self.max_tokens:
            raise BudgetExceeded(f"{self.used} of {self.max_tokens} tokens used")
        response = client.converse(**kwargs)
        usage = response["usage"]
        self.used += usage["totalTokens"]
        with open(self.ledger_path, "a", encoding="utf-8") as f:
            f.write(json.dumps({
                "ts": time.time(),
                "model": kwargs.get("modelId"),
                "input": usage["inputTokens"],
                "output": usage["outputTokens"],
                "total_used": self.used,
            }) + "\n")
        return response

Note that the check happens before the call, so one call can overshoot the ceiling by up to one response. That is fine for a safety net. The test that proves it stubs Bedrock locally, makes two calls of 100 tokens against a ceiling of 150, and confirms that the third call is refused and the ledger shows two lines.

Layer 3: least privilege. Calling Converse requires the bedrock:InvokeModel permission, and ConverseStream requires bedrock:InvokeModelWithResponseStream. So a role that may only call one cheap model looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InvokeOneModelOnly",
      "Effect": "Allow",
      "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
      "Resource": "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-micro-v1:0"
    }
  ]
}

An agent, a notebook or a Lambda that runs under this role cannot suddenly start calling an expensive model, because the policy does not let it.

Layer 4: know what you are logging. Bedrock model invocation logging is off by default. When you turn it on it records the full request and response bodies, token counts and the caller identity for supported calls, and it can write to S3 and to CloudWatch Logs. That is excellent for debugging and for audit, and it also means your log bucket now contains your users' prompts. Encrypt it, restrict access, and set your own retention, because the logs are kept until you delete the configuration. You can tag calls with requestMetadata in the Converse request so you can filter the logs later.

One more money fact that surprises people: cross-Region inference profiles let a request be served by compute in several Regions. AWS states there is no extra routing cost and that the price is based on the Region from which you call the profile. Geographic profiles use standard pricing and global profiles are described as approximately 10 percent cheaper. They also have a governance side: if your organisation uses Service Control Policies to restrict Regions, the destination Regions of the profile must be allowed, and global profiles need an allowance for aws:RequestedRegion set to unspecified. Do not discover this on the day you deploy.

Rule I follow: if I cannot explain where the money stops, I do not press deploy.

Step 1: foundations, and how to read an AccessDenied

IAM, KMS, Secrets Manager, VPC endpoints, the CLI and CDK. Nobody gets excited about this branch, and everybody who skips it pays for it later.

I once wrote a whole article about the fact that the same AWS error can have two different answers, and that guessing which one you are facing is how an afternoon disappears. With AI workloads you meet that early. A model call fails, an agent cannot reach a tool, a Lambda cannot read a bucket, and the message is the same polite "access denied". The skill is to stop guessing and walk a short list, in order:

The commands behind that list are boring and worth memorising:

aws sts get-caller-identity                      # who am I, really? (wrong role is the most common cause)
aws iam simulate-principal-policy \
  --policy-source-arn <role-arn> \
  --action-names bedrock:InvokeModel \
  --resource-arns arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-micro-v1:0

The simulator answers the identity-policy question quickly. If it says allowed and the call still fails, the refusal is coming from somewhere else in the chain: a Service Control Policy or permissions boundary, a resource policy, a KMS key policy, or the Region and model you are actually calling. CloudTrail records the call with its error code and message, which is the receipt you want when two causes look identical.

On KMS, one design rule from the vector services is worth knowing now. When you create an S3 Vectors bucket you choose its encryption type (SSE-S3 or a KMS key) at creation time, and you cannot change it afterwards. Choosing encryption is a day-one decision, not a later hardening step.

Step 2: models and prompts

Amazon Bedrock gives you one API over many model families: Amazon Nova, Anthropic Claude, Meta, Mistral and others. For new code, the Converse API is the one to learn, because it has a consistent request and response shape across models that support messages. Here is the minimum:

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

response = client.converse(
    modelId="amazon.nova-micro-v1:0",          # or an inference profile ID or ARN
    system=[{"text": "You are a concise AWS tutor."}],
    messages=[{"role": "user", "content": [{"text": "What is idempotency, in two sentences?"}]}],
    inferenceConfig={"maxTokens": 300, "temperature": 0.2},
)

text = response["output"]["message"]["content"][0]["text"]
usage = response["usage"]                       # inputTokens, outputTokens, totalTokens
stop = response["stopReason"]                   # do not ignore this

Three habits are hiding in those lines.

Read the stop reason. The SDK model lists these values: end_turn, tool_use, max_tokens, stop_sequence, guardrail_intervened, content_filtered, malformed_model_output, malformed_tool_use and model_context_window_exceeded. Only the first one means "the model finished its thought". A truncated answer (max_tokens) that your code treats as complete is a bug that looks like a bad model.

Count tokens on every call. The usage block is your cost meter. Input tokens include everything you send: the system prompt, the conversation history and, later in Step 3, the retrieved passages. Output tokens are what the model writes. That is why a chatty prompt and a large retrieval depth both show up on the bill, and why the ledger from Step 0 pays for itself.

Use the knobs that exist. Beyond inferenceConfig, Converse accepts additionalModelRequestFields for model-specific parameters, requestMetadata for tagging calls in your logs, guardrailConfig for attaching a guardrail, and serviceTier, whose documented values are reserved, priority, default and flex. Prompt caching works by adding cachePoint blocks, and which blocks a model accepts depends on the model. A detail from the docs that bites people: if you call a prompt from Prompt management, you cannot also pass system, inferenceConfig, toolConfig or additionalModelRequestFields, because the managed prompt owns them.

Tool use, which Step 4 builds on, is part of the same API. You describe tools in toolConfig, the model answers with stopReason set to tool_use and a toolUse block (an id, a name and an input), you run the tool, and you send the outcome back as a toolResult for that id with a status of success or error. Every agent framework is, underneath, a loop around that exchange.

The thing I would add here is a habit, not a service. A model that sounds confident is not a model that is right. I built a small game, gaslight-bench, to find out whether a model can be talked into agreeing with false statements about AWS. The game is playful, but the question behind it is serious: if a model can be pushed off the facts by a confident voice, its own tone can never be your evidence. And the first piece of evidence you can build is tiny: five questions you know the answers to, run every time you change a prompt or a model. That is where evaluation starts, and it starts here, not in Step 5. We will turn it into a real harness in Step 3.

Step 3: grounding and RAG

If you want the model to answer from your documents, the system has two halves, and it helps to keep them separate in your head:

The ingest half runs when your documents change: split them into chunks, turn each chunk into an embedding (a list of numbers), and store the embedding with the chunk's text and some metadata. The query half runs on every question: embed the question, fetch the nearest chunks, put them in the prompt, and ask the model to answer from them. Most RAG problems are in the middle: the right chunk never comes back, so the model never sees the answer, and then it improvises.

The roadmap lists four places to keep embeddings: S3 and S3 Vectors, OpenSearch Serverless, DynamoDB vector search and Aurora pgvector. Two of those have changed a lot recently, so here is what I could verify.

Amazon S3 Vectors became generally available on 2 December 2025. AWS states up to 2 billion vectors per index and up to 10,000 indexes per vector bucket, with queries around 100 ms for frequent use and under a second for infrequent ones, and claims up to 90 percent lower total cost for uploading, storing and querying vectors. It plugs into Bedrock Knowledge Bases, and into OpenSearch for hybrid search. The SDK model tells you the practical constraints: the only data type is float32, the distance metric is cosine or euclidean, and the index name, dimension, distance metric and non-filterable metadata keys are fixed when you create the index. That last point is the one to plan around: if you change your embedding model later, the dimension may change, and you will re-embed and re-index everything into a new index.

Amazon DynamoDB vector search became generally available on 5 August 2026. You store embeddings next to your normal item attributes, define a vector index (at table creation through VectorIndexes, or later through VectorIndexUpdates), and search with the SearchVectors operation, which takes the index name, the query vector, a TopK and an optional condition expression to filter on item attributes. The distance functions are COSINE, DOT_PRODUCT and EUCLIDEAN, and AWS states a target of single-digit millisecond latency at 99 percent plus recall. One detail trips people up: DynamoDB does not create the embeddings for you. You generate them with a model of your choice (a Bedrock model works) and store them. Write-ups of the launch also say that vector indexes require on-demand capacity and cap the number of results returned per search, so read the current limits page before you design around either.

Here is the same embedding stored both ways. The calls are shaped exactly to the SDK's current models (I validated every parameter offline, see the testing section), and the embed function is the only part that spends money:

# code/vector_stores.py
"""Two ways to store the same embedding on AWS. Calls are shaped to the current SDK models.

Not run against a live account in my tests: they are validated offline (see test_vectors.py).
"""
import boto3

EMBED_MODEL = "amazon.titan-embed-text-v2:0"  # check the model card for the request body you use


def embed(text: str, dims: int = 1024) -> list:
    """Real embedding call (costs a little per token). Swap in any embedding model."""
    import json
    bedrock = boto3.client("bedrock-runtime")
    resp = bedrock.invoke_model(modelId=EMBED_MODEL, body=json.dumps({"inputText": text, "dimensions": dims, "normalize": True}))
    return json.loads(resp["body"].read())["embedding"]


# ---- S3 Vectors ----------------------------------------------------------------
def s3_vectors_setup(s3v, bucket: str, index: str, dims: int = 1024):
    s3v.create_vector_bucket(vectorBucketName=bucket)
    s3v.create_index(
        vectorBucketName=bucket, indexName=index,
        dataType="float32", dimension=dims, distanceMetric="cosine",
        # Big text you only want back, never filter on: mark it non-filterable (immutable later).
        metadataConfiguration={"nonFilterableMetadataKeys": ["source_text"]},
    )


def s3_vectors_put(s3v, bucket, index, items):
    """items: [(key, embedding, {"source_text": ..., "topic": ...}), ...]"""
    s3v.put_vectors(
        vectorBucketName=bucket, indexName=index,
        vectors=[{"key": k, "data": {"float32": e}, "metadata": md} for k, e, md in items],
    )


def s3_vectors_query(s3v, bucket, index, query_embedding, top_k=3, topic=None):
    kw = {"filter": {"topic": topic}} if topic else {}
    resp = s3v.query_vectors(
        vectorBucketName=bucket, indexName=index, topK=top_k,
        queryVector={"float32": query_embedding}, returnMetadata=True, returnDistance=True, **kw,
    )
    return resp["vectors"]  # each has key, distance, metadata


# ---- DynamoDB vector search ----------------------------------------------------
def ddb_create_table_with_vector_index(ddb, table: str, dims: int = 1024):
    ddb.create_table(
        TableName=table,
        BillingMode="PAY_PER_REQUEST",  # vector indexes need on-demand capacity (per AWS write-ups; check docs)
        AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}],
        KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}],
        VectorIndexes=[{
            "IndexName": "by-embedding",
            "VectorAttribute": {"AttributeName": "embedding"},
            "Dimensions": dims,
            "DistanceFunction": "COSINE",
            "Projection": {"ProjectionType": "ALL"},
        }],
    )


def ddb_search(ddb, table: str, query_embedding, top_k=3):
    resp = ddb.search_vectors(
        TableName=table, IndexName="by-embedding", TopK=top_k,
        SearchVector=[{"N": repr(float(x))} for x in query_embedding],
    )
    return resp["SearchResults"]

How do I choose? Check the current documentation for limits and pricing, then pick the boring option that matches the shape of your data:

If your situation is...	I would start with
Lots of documents, queries are occasional, cost matters most	S3 Vectors
Your app already lives in DynamoDB and the vectors belong to the same items (agent memory, product similarity)	DynamoDB vector search
You need rich search: hybrid keyword plus vector, ranking, analytics	OpenSearch
Your data is relational and you already run Postgres	Aurora with pgvector

The test that matters more than the database

Choosing a store is the easy decision. The hard question is "does the right chunk come back?", and you can answer it with a page of Python, offline, before you spend anything. A retrieval test needs a golden set (questions with the chunk that should answer each one) and two numbers: recall@k (how often the right chunk is in the top k) and MRR (how high it ranks, on average).

Here is the whole harness. The embedding in it is a deliberately simple hashed bag of words, a stand-in so the harness runs anywhere; swap embed() for a real model and keep everything else:

# code/retrieval_eval.py
"""A retrieval test you can run with no AWS account: recall@k and MRR over a golden set.

The embedding here is a hashed bag of words, a stand-in so the harness runs anywhere.
Swap embed() for a real model call (see the Bedrock version in the article) and keep the rest.
"""
import re
import zlib

import numpy as np

CHUNKS = {
    "c1": "S3 event notifications are delivered at least once so handlers must tolerate duplicates",
    "c2": "Lambda retries failed asynchronous invocations up to two more times by default",
    "c3": "The sequencer field in an S3 event orders events for the same object key",
    "c4": "Bedrock Guardrails check model input and model output but not tool parameters by default",
    "c5": "AgentCore Gateway turns APIs Lambda functions and MCP servers into tools for agents",
    "c6": "AgentCore Memory keeps short term session events and long term extracted records",
    "c7": "S3 Vectors stores embeddings in vector buckets and queries them with cosine or euclidean distance",
    "c8": "DynamoDB vector search stores embeddings next to item attributes and you generate them yourself",
    "c9": "Model invocation logging is off by default and records prompts and responses when enabled",
    "c10": "AWS Budgets can alert you when actual spend passes a percentage of your monthly limit",
    "c11": "Cross Region inference profiles route requests to compute in several Regions",
    "c12": "Strands hooks run before a tool call after a tool call and before a model invocation",
}

GOLDEN = [  # (question, id of the chunk that should come back)
    ("how many times does lambda retry an async invocation", "c2"),
    ("can an s3 notification arrive twice", "c1"),
    ("how do i order s3 events for one key", "c3"),
    ("do guardrails inspect tool parameters", "c4"),
    ("which agentcore piece exposes lambda functions as tools", "c5"),
    ("where do agents keep long term memory", "c6"),
    ("which distance metrics does s3 vectors support", "c7"),
    ("who creates the embeddings for dynamodb vector search", "c8"),
    ("is bedrock prompt logging enabled by default", "c9"),
    ("how do i get an alert at 80 percent of my monthly spend", "c10"),
    ("what runs just before a tool call in strands", "c12"),
    ("send my traffic to another region automatically", "c11"),
    ("stop paying for something i forgot to turn off", "c10"),  # real paraphrase: no shared words, so a lexical embedder should miss
]

STOP = {"a", "an", "the", "of", "in", "to", "is", "are", "do", "does", "how", "i", "my", "for", "and", "or", "on", "at", "it", "by", "up", "you", "can", "which", "what", "where", "who", "than", "so", "but", "not", "when"}
DIM = 512


def embed(text: str) -> np.ndarray:
    v = np.zeros(DIM)
    for tok in re.findall(r"[a-z0-9]+", text.lower()):
        if tok not in STOP:
            v[zlib.crc32(tok.encode()) % DIM] += 1.0
    n = np.linalg.norm(v)
    return v / n if n else v


def rank(question: str, ids, matrix):
    scores = matrix @ embed(question)
    return [ids[i] for i in np.argsort(-scores)]


def evaluate(k_values=(1, 3)):
    ids = list(CHUNKS)
    matrix = np.stack([embed(CHUNKS[i]) for i in ids])
    hits = {k: 0 for k in k_values}
    rr, misses = 0.0, []
    for q, gold in GOLDEN:
        order = rank(q, ids, matrix)
        pos = order.index(gold) + 1
        rr += 1 / pos
        for k in k_values:
            hits[k] += pos <= k
        if pos > 3:
            misses.append((q, gold, order[:3]))
    n = len(GOLDEN)
    return {f"recall@{k}": hits[k] / n for k in k_values} | {"mrr": rr / n}, misses


if __name__ == "__main__":
    metrics, misses = evaluate()
    for name, value in metrics.items():
        print(f"{name:10s} {value:.2f}")
    for q, gold, got in misses:
        print(f"MISS  {q!r}  wanted {gold}  got {got}")

Running it prints this:

recall@1   0.85
recall@3   0.92
mrr        0.89
MISS  'stop paying for something i forgot to turn off'  wanted c10  got ['c9', 'c1', 'c3']

Look at the miss, because the miss is the point. The question "stop paying for something I forgot to turn off" is about AWS Budgets, but it shares no words with the chunk about budgets, so a lexical embedder cannot find it. A real embedding model closes much of that gap, and hybrid search (keyword plus vector) closes more, but you will not know how much until you measure. Without the golden set you would have shipped a system that sounds right and silently misses a whole class of questions. With it, the same twelve lines of output tell you whether a new chunk size, a new embedding model or a new filter made things better or worse. Treat it like a unit test for retrieval, and put it in CI.

Two practical notes. First, chunk size is a trade: small chunks are precise but lose context, large chunks carry context but dilute the match and cost more input tokens when you stuff them in the prompt. Test two or three sizes against your golden set instead of copying someone's default. Second, use metadata filters (S3 Vectors takes a filter on query, DynamoDB takes a condition expression) to narrow by tenant, language or document type before similarity does its work. It is cheaper and more accurate than hoping the nearest neighbour is also the right tenant.

Step 4: agents, and where reality arrives

This is the branch I know best, because I lived it. The Amazon Bedrock AgentCore family went generally available in October 2025 and covers the pieces you would otherwise build yourself:

Runtime hosts your agent code in a serverless environment with each session isolated. Memory keeps short-term session events and long-term extracted records. Gateway turns APIs, Lambda functions and MCP servers into tools behind one endpoint. Identity gives agents workload identities and a token vault for OAuth flows. Observability sends metrics, traces and logs to CloudWatch with OpenTelemetry export. Code Interpreter and Browser run sandboxed code and managed browser sessions. Policy checks each tool call at the Gateway against Cedar rules. Around it sit the builders the roadmap lists: the Strands Agents SDK, Kiro and Amazon Quick.

An agent in a dozen lines, and what the framework is doing

With the Strands Agents SDK, a first agent is genuinely small. And because Strands has a provider for Ollama (pip install 'strands-agents[ollama]'), you can build and test the whole loop on a local model with no AWS bill at all:

from strands import Agent, tool
from strands.models.ollama import OllamaModel

@tool
def get_invoice_total(invoice_id: str) -> str:
    """Return the total for an invoice."""
    return {"INV-1": "42.00", "INV-2": "17.50"}.get(invoice_id, "not found")

local = OllamaModel(host="http://localhost:11434", model_id="llama3.1")   # ollama pull llama3.1 first
agent = Agent(model=local, tools=[get_invoice_total])
agent("What is the total of invoice INV-2?")

Behind that one call is the loop from Step 2: the model asks for get_invoice_total, the framework runs it, feeds the result back, and the model writes the answer. When the model is good enough, you change one line to use a Bedrock model.

Hosting it: the AgentCore CLI changed under me

One thing I would have been bitten by if I had copied an older tutorial: the Python starter toolkit CLI for AgentCore (the agentcore configure and agentcore launch flow) now prints a notice that it is no longer supported and points to a new AgentCore CLI that you install from npm (npm install -g @aws/agentcore). I installed the new one (version 1.0.1) and read its help. It scaffolds a project from templates (for example agent-python-strands, a minimal Python template, LangChain, TypeScript and MCP server templates), runs agents locally with agentcore dev, and ships with deploy, invoke, status, logs and traces commands, plus an eval command group that we will meet in Step 5. The workflow is therefore:

npm install -g @aws/agentcore
agentcore create --name robot-demo --template agent-python-strands
agentcore dev          # runs the agent locally, with local traces
# agentcore deploy     # this one builds and deploys to AWS: it is where the bill starts

Notice where the money starts. Everything up to dev is local. deploy creates real resources in your account, so keep it behind the budget and the least-privilege role from Step 0, and delete what you deploy when the experiment is over.

If you want to see the runtime contract for yourself without deploying, the SDK's app class is small. This entrypoint, tested through a local test client, answers POST /invocations with your return value and GET /ping with a health status:

from bedrock_agentcore.runtime import BedrockAgentCoreApp

app = BedrockAgentCoreApp()

@app.entrypoint
def invoke(payload):
    prompt = payload.get("prompt", "Hello")
    return {"result": f"echo: {prompt}"}      # replace with: agent(prompt)

if __name__ == "__main__":
    app.run()

When I posted {"prompt": "ping"} to /invocations I got {"result": "echo: ping"} with a 200, and /ping answered {"status":"Healthy", ...}. That is the whole contract your agent code has to honour.

The two rules that break agents

An agent is a distributed system with a language model in the middle, and distributed systems have rules that do not care how clever your prompt is. Two of them cost me the most, and the roadmap does not mention either.

Rule 1: triggers deliver at least once. S3 event notifications are designed to be delivered at least once, and Lambda retries failed asynchronous invocations up to two more times by default. So if your tool charges a customer, writes to a ledger or sends an email, it will eventually do it twice unless you made it safe to repeat.

The fix is an old bookkeeping idea: write down a key before you act, and check it first. A naive version (insert the key, then act) has a hole that readers pointed out after my first idempotency article: what happens when the first worker crashes after claiming the key but before finishing? The key says "in progress" forever, and the work never completes. The robust version gives each claim a lease and a fencing check. Here it is as one small module on top of a single DynamoDB table:

# code/idempotency.py
"""Idempotent execution with a lease, backed by one DynamoDB table.

Table: partition key "pk" (string). Enable TTL on "expires_at" to clean up.
States: INPROGRESS (has an owner token and a lease) and DONE (has a result).
"""
import time
import uuid

from botocore.exceptions import ClientError


class InProgress(Exception):
    """Another worker holds a live lease on this key. Let the retry come back later."""


class LostLease(Exception):
    """Our lease expired and someone else took over. Do not record our result."""


def _failed_condition(err: ClientError) -> bool:
    return err.response["Error"]["Code"] == "ConditionalCheckFailedException"


class IdempotencyStore:
    def __init__(self, table, lease_seconds=60, retention_seconds=86400, clock=time.time):
        self.table = table
        self.lease = lease_seconds
        self.retention = retention_seconds
        self.clock = clock

    def claim(self, key):
        """Return ("run", token) if we may do the work, or ("done", result) if it is already done."""
        now = int(self.clock())
        token = uuid.uuid4().hex
        try:
            self.table.put_item(
                Item={
                    "pk": key,
                    "status": "INPROGRESS",
                    "owner": token,
                    "lease_until": now + self.lease,
                    "expires_at": now + self.retention,
                },
                # New key, or an abandoned claim whose lease has run out.
                ConditionExpression="attribute_not_exists(pk) OR (#s = :inprog AND lease_until < :now)",
                ExpressionAttributeNames={"#s": "status"},
                ExpressionAttributeValues={":inprog": "INPROGRESS", ":now": now},
            )
            return "run", token
        except ClientError as err:
            if not _failed_condition(err):
                raise
        item = self.table.get_item(Key={"pk": key}, ConsistentRead=True).get("Item")
        if item and item["status"] == "DONE":
            return "done", item.get("result")
        raise InProgress(key)

    def complete(self, key, token, result):
        """Record the result, but only if we still own the claim (the fencing check)."""
        try:
            self.table.update_item(
                Key={"pk": key},
                UpdateExpression="SET #s = :done, #r = :r",
                ConditionExpression="#o = :t AND #s = :inprog",
                ExpressionAttributeNames={"#s": "status", "#r": "result", "#o": "owner"},
                ExpressionAttributeValues={":done": "DONE", ":inprog": "INPROGRESS", ":t": token, ":r": result},
            )
        except ClientError as err:
            if _failed_condition(err):
                raise LostLease(key) from err
            raise

    def release(self, key, token):
        """Give the key back after a failure so the next retry can run immediately."""
        try:
            self.table.delete_item(
                Key={"pk": key},
                ConditionExpression="#o = :t",
                ExpressionAttributeNames={"#o": "owner"},
                ExpressionAttributeValues={":t": token},
            )
        except ClientError as err:
            if not _failed_condition(err):
                raise


def run_once(store: IdempotencyStore, key: str, fn):
    """Run fn() at most once per key. fn must return a string (store JSON if you need more)."""
    state, value = store.claim(key)
    if state == "done":
        return value
    token = value
    try:
        result = fn()
    except Exception:
        store.release(key, token)
        raise
    store.complete(key, token, result)
    return result

Walk through the four situations, because all four are tested. (1) The same key twice runs the side effect once, and the second call returns the stored result. (2) A second worker arriving while the first holds a live lease gets InProgress, so it backs off and the retry comes later. (3) If the first worker stalls past its lease, a second worker can take over, and when the stalled worker finally wakes up and tries to record its result, the conditional write refuses it with LostLease. That refusal is the fencing check. (4) If the work fails, the key is released, so the retry can run immediately instead of waiting for the lease to expire.

Now the honest limit, which I would rather you hear from me than discover in production. The fencing check protects your record. It does not magically undo a side effect that the stalled worker already performed in the outside world. If the stalled worker already charged the card, the new worker may charge it again. The only complete answer is to pass the idempotency key to the system that performs the side effect (a payment provider's idempotency key, for example), so the duplicate is rejected there too. Think of the lease as protection for your own ledger, and the downstream key as protection for the world.

Rule 2: events do not arrive in order. Amazon S3's own documentation says event notifications are not guaranteed to arrive in the order the events occurred. For object creation and deletion events it gives you a tool: the sequencer field, a hexadecimal string that you can compare for events on the same object key (and only the same key). The documented comparison has a trap: the strings can differ in length, so you left-pad the shorter one with zeros and then compare lexicographically. Compare them raw and you get a wrong answer: as text, 10 sorts before F, although in hex 10 (16) is later than F (15). Here is a guard that only applies an event if it is newer than the one you stored, using a conditional write so it is safe even when two events race:

# code/ordering.py
"""Drop stale events: only apply an S3 event if its sequencer is newer than the one we stored.

AWS guidance: the sequencer is hex, only comparable for the same object key, and values can
differ in length, so left-pad the shorter one with zeros and compare as strings.
"""
from botocore.exceptions import ClientError

WIDTH = 64  # comfortably longer than real sequencers


def normalise(sequencer: str) -> str:
    return sequencer.upper().zfill(WIDTH)


def is_later(a: str, b: str) -> bool:
    return normalise(a) > normalise(b)


def apply_if_newer(table, object_key: str, sequencer: str, payload: str) -> bool:
    """Write payload only when this event is newer. Returns False for a stale event."""
    seq = normalise(sequencer)
    try:
        table.update_item(
            Key={"pk": object_key},
            UpdateExpression="SET seq = :s, payload = :p",
            ConditionExpression="attribute_not_exists(seq) OR seq < :s",
            ExpressionAttributeValues={":s": seq, ":p": payload},
        )
        return True
    except ClientError as err:
        if err.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

The test applies an event, then a newer one, then a stale one that arrives late, and confirms that the stale one is dropped and the stored payload is still the newest. It also asserts the 10 versus F trap, so the padding cannot be removed by accident.

If I could add one signpost to Step 4 it would read: before you give an agent tools, make the tools safe to repeat and safe to reorder.

Step 5: evaluate and govern

The last step has two halves, and they are different jobs.

Governing: guardrails stop at the model call

Bedrock Guardrails check what goes into a model and what comes out of it. An important nuance, which AWS spells out in its own security guidance: that boundary is the model call. Tool parameters, external data and traffic between agents sit outside it.

The Strands Agents SDK lets you extend the check to those moments with hooks. In the version I installed (strands-agents 1.62.0), a BeforeToolCallEvent carries the tool_use (name and input) and a cancel_tool field: set it to a string and the tool is skipped and the string goes back to the model. An AfterToolCallEvent carries the result, which you may replace. And the ApplyGuardrail operation lets you run a guardrail on any text yourself, with source set to INPUT or OUTPUT, returning an action of NONE or GUARDRAIL_INTERVENED. Put those together and you get a guardrail on tools:

# code/guardrail_hook.py
"""Extend Bedrock Guardrails to tool calls in a Strands agent."""
import json

from strands.hooks import AfterToolCallEvent, BeforeToolCallEvent, HookProvider, HookRegistry


class GuardrailHook(HookProvider):
    def __init__(self, bedrock_runtime, guardrail_id: str, guardrail_version: str = "DRAFT"):
        self.client = bedrock_runtime
        self.gid = guardrail_id
        self.gver = guardrail_version

    def register_hooks(self, registry: HookRegistry) -> None:
        registry.add_callback(BeforeToolCallEvent, self.before_tool)
        registry.add_callback(AfterToolCallEvent, self.after_tool)

    def _blocked(self, source: str, text: str) -> bool:
        resp = self.client.apply_guardrail(
            guardrailIdentifier=self.gid,
            guardrailVersion=self.gver,
            source=source,  # "INPUT" or "OUTPUT"
            content=[{"text": {"text": text}}],
        )
        return resp["action"] == "GUARDRAIL_INTERVENED"

    def before_tool(self, event: BeforeToolCallEvent) -> None:
        if self._blocked("INPUT", json.dumps(event.tool_use["input"])):
            # Setting cancel_tool skips the tool; the string goes back to the model.
            event.cancel_tool = "Blocked by guardrail. Do not retry this call."

    def after_tool(self, event: AfterToolCallEvent) -> None:
        text = " ".join(b.get("text", "") for b in event.result.get("content", []))
        if text and self._blocked("OUTPUT", text):
            event.result = {
                "toolUseId": event.result["toolUseId"],
                "status": "error",
                "content": [{"text": "Tool result withheld by guardrail."}],
            }

# Usage:
#   from strands import Agent
#   agent = Agent(tools=[...], hooks=[GuardrailHook(boto3.client("bedrock-runtime"), "gr-id", "1")])

The test registers the hook with a Strands hook registry, then feeds it fake events: a tool input containing a sensitive marker gets cancelled, a clean one passes, and a tool result containing the marker is replaced by an error result. It uses a stub in place of Bedrock, so it proves the wiring, not the guardrail's judgement. The judgement, what your guardrail actually blocks, is the part you must test with your own data.

Three other pieces belong in the governing kit. AgentCore Policy checks each tool call at the Gateway against Cedar rules, which is the right tool when a rule must be a hard rule rather than a hopeful instruction in a prompt. Automated Reasoning checks, part of Guardrails, check model answers against policies you define, which suits rule-heavy domains. And for the paper trail, CloudTrail records who called what, while the invocation logging from Step 0 records what was said.

Evaluating: evidence, not vibes

Bedrock Evaluations and AgentCore Observability are on the roadmap. AgentCore Evaluations was announced in preview in December 2025, and the new AgentCore CLI I installed today has an eval command group with subcommands for evaluators, online evaluation configs, datasets, batch evaluations, on-demand evaluation of existing sessions, and A/B tests. Preview and general availability change, so check the current status before you plan around it. The shape of the idea is stable, though, and you can practise it for free:

A golden set of cases, a runner that executes them, graders that score each result, and a gate that fails the build when a score drops. You have already seen the smallest version: the retrieval harness in Step 3. The tests in this article are the same idea applied to reliability: the idempotency test replays the same event twice, the ordering test delivers events backwards, and the budget test tries to overspend. For agents, add a red-team list of nasty inputs. A good one covers instructions hidden in tool output, requests to reveal the system prompt, malformed arguments and attempts to talk the agent into skipping a check. I once spent a whole article attacking my own agent with chaos tests and injected instructions, and the lesson was simple: attack your work before someone else does, and do not wait for the last step of a learning path to start.

A failure cheat sheet

Symptom	Likely cause	First thing to check
Customer charged twice, logs spotless	At-least-once delivery or an async retry	An idempotency key checked before the side effect
Ledger shows an old value after a newer one	Events arrived out of order	Sequencer compare, conditional write
Work never finishes after a crash	A claim with no lease	Lease expiry on the claim
Answer sounds right but cites nothing real	The right chunk was never retrieved	recall@k on a golden set
Answer cut off mid-sentence	`max_tokens` stop reason treated as complete	`stopReason` on every call
Bill higher than expected	Large history or retrieval depth, a loop	The token ledger, then `usage` per call
AccessDenied that policy simulator says is allowed	SCP, resource policy, KMS, Region or model	CloudTrail error message and the Step 1 list
Tool does something it should not	Guardrails stop at the model call	A hook on tool calls, or Policy at the Gateway

Where I would move the signposts

Here is my honest list of changes to the map. They are small, and they are all about order.

Start evaluating in Step 2, not Step 5. The day you call a model, write down five questions you know the answer to and check them. That tiny habit grows into an evaluation suite. Waiting until the end means evaluating a finished building.
Put cost controls at step zero. A budget, a token ceiling and a one-model role are cheaper than any lesson.
Add a reliability step between 3 and 4. Idempotency, leases, retries and ordering. It is not glamorous and it is where production breaks.
Treat the map as a menu, not a checklist. Textract, Polly, Comprehend and Rekognition are excellent, and you do not need them to learn AI on AWS. Learn them when a real problem asks for them.

A four week plan that respects your wallet

If I were starting again, this is how I would spend four weeks:

Week	Focus	The small proof you build
1	IAM, budgets, one model call	A budget alert, and a script that calls one Bedrock model through the token ceiling
2	Retrieval	A tiny knowledge base over twenty of your own documents, and five questions you can grade with recall@k
3	One agent with two tools	Tools that are safe to repeat and reorder, and a test that replays the same event twice
4	Guardrails and evaluation	A guardrail on tool calls, plus a red-team list of ten nasty inputs wired into the build

Notice what the table does not contain: a big deployment. Each week ends with something small, local where possible, and provable. If you want a "done when" for each week, it is this: week 1 is done when the alert exists and the ceiling stops a loop; week 2 when a miss in your golden set makes you change something; week 3 when the replay test passes; week 4 when a red-team input fails the build until you fix it.

How I tested this, and what I did not

I would rather tell you the limits of my testing than let you assume them.

What I ran. I installed boto3 1.43.122, moto 5.2.5 (a local stand-in for DynamoDB), numpy, strands-agents 1.62.0, bedrock-agentcore 1.25.2 and the new AgentCore CLI 1.0.1, and ran: the idempotency lease (four situations), the ordering guard, the token ceiling (against a stubbed Bedrock client), the budget JSON (validated through the SDK's request validation), the retrieval harness, the guardrail hook (against a stubbed Bedrock client and a real Strands hook registry), the vector store calls (parameter validation against the SDK's models for S3 Vectors and DynamoDB), and the AgentCore app routes (through a local test client). They all pass, and they are in the companion folder with a README and a single command to run them.

What I did not run. None of this touched a real AWS account, because I do not spend money on experiments I can prove locally. So I have not measured real Bedrock latency or cost, I have not run embed() against a real embedding model (check the model card for the exact request body of the model you pick), I have not exercised a real guardrail's judgement, and I have not deployed anything with agentcore deploy. Facts about services (limits, dates, availability) come from AWS announcements and documentation, and a few details (the on-demand capacity requirement and result cap for DynamoDB vector search) come from third-party write-ups of AWS's launch, so I flagged them where they appear. Services change: verify limits and pricing against the current documentation before you build on them.

Closing thoughts

The roadmap is a very good map. Maps are made by people who stand back far enough to see the whole mountain, and that is a gift. What I tried to add here is the view from the path: the scree, the closed hut, the sign that points the wrong way, and a few tools you can pick up for free. Use the map to know where you are going, and use your own mistakes (cheaply, on a small budget) to learn how to get there.

If you want to see the terrain I was talking about, here are the stories behind the field notes:

Moving the Robot Accountant to AgentCore: Dev.to | AWS Builder Center
The IAM error that has two answers: AWS Builder Center
Out-of-order events and AI agents: AWS Builder Center
Breaking my own agent on purpose: AWS Builder Center
The ledger that charged twice: Dev.to | AWS Builder Center
Can you gaslight an AI about AWS?: Dev.to | GitHub

All the code from this article, with its tests, is in the companion repository: https://github.com/mursalfk/aws-for-ai-field-guide.

What is the one step of this path you wish somebody had warned you about?

Ciao! 👋

References

Amazon S3 Vectors generally available: https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-s3-vectors-generally-available/
Amazon DynamoDB vector search generally available: https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-dynamodb-vector-search/
AgentCore Policy and Evaluations (preview announcement): https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-bedrock-agentcore-policy-evaluations-preview
Extending Bedrock Guardrails to tool interactions with Strands: https://aws.amazon.com/blogs/security/extend-amazon-bedrock-guardrails-to-tool-interactions-using-the-strands-agents-sdk/
Automated Reasoning checks in Bedrock Guardrails: https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-automated-reasoning-checks.html
Converse API and its permissions: https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html
Cross-Region inference and inference profiles: https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html
Model invocation logging: https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html
Amazon S3 event message structure (the sequencer field): https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html
Strands Agents hooks: https://strandsagents.com/latest/documentation/docs/user-guide/concepts/agents/hooks/
Strands Ollama model provider: https://strandsagents.com/latest/documentation/docs/user-guide/concepts/model-providers/ollama/

It Remembers Across Restarts — and Forgets Mid-Conversation

Pin Xiong — Sat, 07 Nov 2026 01:50:54 +0000

Verified on 2026-11-07 against strands-agents 1.63.0, Python 3.11 and Bedrock in us-east-1. Every number below came out of a run, and the setup steps live in README.md. The model is amazon.nova-pro-v1:0 on Amazon Bedrock.

An agent can remember your name across a restart and still lose it seven turns later in the same conversation. Those are two separate failures, and fixing one does nothing for the other.

Article 01 established that an agent's state is one list of messages and that the conversation is the only channel into the model. Article 03 bounded a single call. This post follows that list through a restart, a window and a summary, and measures where each one fails.

It still knows you after a restart

The simplest memory test runs as two separate OS processes.

from strands import Agent
from strands.models import BedrockModel
from strands.session import FileSessionManager

model = BedrockModel(model_id="amazon.nova-pro-v1:0", region_name="us-east-1")
session_manager = FileSessionManager(
    session_id="restart-demo", storage_dir="./sessions_tmp"
)
agent = Agent(model=model, session_manager=session_manager, callback_handler=None)
agent("My name is Pin and my favorite number is 42. Just acknowledge, one short sentence.")

The first process answers "Hello Pin, it's great to know that 42 is your favorite number!" and exits. Nothing stays in memory. The second process shares nothing with the first except that directory, and it repeats the same imports and the same three setup statements with the same session_id. The repository keeps both phases in one file and branches on sys.argv[1], which is why the figure runs the script twice. If you copy the block above into a second file, keep all four statements; the call alone raises NameError.

print(agent("What is my name and my favorite number?"))

It answers "Your name is Pin and your favorite number is 42." and then adds a caveat about not storing personal information. No variable carried the fact across the process boundary. A file did.

The cost is visible on disk. After the first run the storage directory holds four files: a session.json of 165 bytes, an agent.json of 447 bytes, and one JSON file per message, message_0.json at 390 bytes and message_1.json at 594 bytes. After the second run there are six. The message count and the file count move together, one file per message, which is the layout the class docstring draws at file_session_manager.py:31.

That layout makes a session cheap to append to, and it makes the transcript readable without the SDK. When an agent misbehaves at 3 a.m., cat agents/agent_default/messages/message_17.json answers more questions than a debugger. The snippets in this post use ./sessions_tmp for readability; the scripts derive the directory from their own location, so on disk it is src/sessions_tmp.

The three things living in agent.json

The 447-byte file from the restart run holds an empty state, because that script never sets any. Open the file a state test writes instead, 528 bytes including two timestamps, and the word memory starts to come apart.

{
  "agent_id": "default",
  "state": {
    "visits": 1,
    "user": { "name": "Pin", "plan": "pro" }
  },
  "conversation_manager_state": {
    "__name__": "SlidingWindowConversationManager",
    "removed_message_count": 0,
    "model_call_count": 0
  },
  "_internal_state": {
    "interrupt_state": { "interrupts": {}, "context": {}, "activated": false },
    "model_state": {}
  }
}

That file arrived after two agent.state.set calls and one turn, and it holds three kinds of memory side by side. The state block is your key-value store, written by application code. The conversation_manager_state block is bookkeeping owned by the window manager, including a counter of messages it has already removed. The _internal_state block belongs to pause and resume handling; for a plain chat agent it stays empty, and it appears here only because it shares the same restore path. All three travel together, because restoring the transcript without the window counters would make the agent re-trim history it had already trimmed.

These three blocks are not the three lanes drawn later. The lanes are the whole session on disk, the window in memory and the state store; inside agent.json, the state store is one block among three.

From here on, state means the agent.state key-value store, not the message list. The SDK uses the same word for both objects, and keeping them apart is half of this article.

set and get take a string key, and values must survive a round trip through JSON. The check happens at write time, which turns a corrupt-file bug into a local error, and the message names the permitted types:

ValueError: Value is not JSON serializable: object. Only JSON-compatible types (str, int, float, bool, list, dict, None) are allowed

The type behind the store is JSONSerializableDict, so an arbitrary Python object cannot enter it in the first place. A set() call only mutates memory, though. State reaches disk when the session manager flushes, which happens on the next agent invocation. A process that sets state and exits before calling the model persists nothing.

The distinction worth keeping is who reads which store. Messages are for the model. State is for your code. A plan id, a retry budget, the last tool result you care about, none of that needs to occupy the window, and none of it should be reconstructed by asking the model to remember.

One file instead of six

The message log is one way to put a session on disk, and the SDK ships a second one that answers the same restart test differently.

from strands.session import SnapshotSessionManager
from strands.storage import LocalFileStorage

session_manager = SnapshotSessionManager(
    session_id="snapshot-demo",
    storage=LocalFileStorage(base_dir="./snapshots_tmp"),
    save_latest_on="message",
)
agent = Agent(model=model, session_manager=session_manager, callback_handler=None)

The same two-process experiment gives the same recall and a different disk: one file, session/snapshot-demo/scopes/agent/default/snapshots/snapshot_latest.json, at 878 bytes after the first run and 1774 after the second. That session/ prefix comes from the manager wrapping the storage in a namespaced view, so the same prefix appears on S3 as well.

Where the message log accumulates history as a directory of small files, the snapshot overwrites a single blob that carries transcript, state and manager bookkeeping together. One write is atomic and restores in one read, so the transcript and the removed-message counter can never disagree. The cost is granularity. A crash leaves the most recently saved snapshot, which with save_latest_on="message" is at most one message behind.

The snapshot docstring at snapshot_session_manager.py:1 calls this "the recommended path for new agents" and draws the line to the older managers that persist each message individually. Read that as a direction of travel, not a deprecation notice: the examples I checked still reach for FileSessionManager, and no session manager is the SDK default, since session_manager defaults to None and an agent without one persists nothing.

The window the model actually sees

Now the second question.

SlidingWindowConversationManager defaults to window_size=40. By default it runs once at the end of each invocation (per_turn=False); set per_turn=True and it runs before every model call instead. The check itself is one comparison in apply_management at sliding_window_conversation_manager.py:150: if len(messages) is at or below the window, return, otherwise trim. Two limits share the word window here. window_size counts messages and belongs to the conversation manager. The context window counts tokens and belongs to the model. They are unrelated, and only the first one is in play in this section.

I set the window to 6 to make the gap observable within seven turns, wiring it the same way as any other manager, imported from strands.agent.conversation_manager and passed to Agent(...).

from strands.agent.conversation_manager import SlidingWindowConversationManager

agent = Agent(
    model=model,
    session_manager=session_manager,
    conversation_manager=SlidingWindowConversationManager(window_size=6),
    callback_handler=None,
)

Turn one plants a codename, five filler turns follow, and turn seven asks for the codename back. Two counters print after every turn: the length of agent.messages, which is what the model can see, and the number of message files on disk, which is what the session stored.

Two counters diverge at turn four and never meet again. The arithmetic is worth stating, because it explains the exact turn: every turn appends a user message and an assistant message, so a window of six holds exactly three turns, and turn four is the first one that overflows. The trim then cuts back to window_size itself, which is why the count pins at six rather than drifting to five or seven. By the end the session holds fourteen message files and the model sees six.

One nuance the figure cannot show: with FileSessionManager no message file is deleted. The transcript keeps everything and agent.json rewrites removed_message_count to record what the window dropped. A snapshot manager behaves differently, because it saves the trimmed list.

The model did not say "I do not know your codename." It produced a refusal about private information, as if it had been asked something improper:

Sorry, I can't respond in a way that might provide private and sensitive information.
This is to maintain the privacy and security of the individuals.

A forgotten fact does not surface as an absence. It surfaces as a confident answer built from whatever is left in the window. If you have ever watched an assistant deny something it was told earlier in the same conversation, this is the mechanism.

Two memories so far, and they disagree. A session is durable but invisible. A window is visible but ephemeral. Add the third lane and the map is complete.

State does not inject itself

The third lane has a gap of its own. In 1.63.0, agent.state is written to agent.json and restored on the next invocation, but nothing in the prompt path ever reads it. FileSessionManager hands it to SessionAgent.from_agent (state=agent.state.get(), types/session.py:145) and reloads it at repository_session_manager.py:262; SnapshotSessionManager goes through take_snapshot and load_snapshot instead (agent/agent.py:2262, :2304). And system_prompt is a value, typed str | list[SystemContentBlock] | None (agent/agent.py:243), not a callable. So a fact in state is durable and invisible at the same time: the same failure the window produced, out of a different store.

If the model has to see it, the injection is yours to write. The smallest working version is a BeforeInvocationEvent hook (hooks/events.py:39) plus the system_prompt setter (agent/agent.py:812):

BASE_PROMPT = "You are a concise assistant."

def inject_manifest(event: BeforeInvocationEvent) -> None:
    manifest = json.dumps(agent.state.get(), sort_keys=True)
    agent.system_prompt = f"{BASE_PROMPT}\n\n<manifest>\n{manifest}\n</manifest>"

agent.add_hook(inject_manifest, BeforeInvocationEvent)

Same setup as the sliding-window run, with one addition: the codename goes into both the first user message and agent.state, and the hook rewrites the system prompt on every invocation. Turn seven now answers with the codename, even though it left the window at turn four:

ORION_IN_MESSAGES_BEFORE_ASK: False
TURN 7 ask     | msgs= 6 stored=14 | ORION
ORION_IN_SYSTEM_PROMPT: True

The hook fires before every invocation, not once at construction, so the manifest tracks state as it changes. The channel is not free either: it repeats the serialized state in the prompt on every call, which is a fair price for an invariant and a bad one for a log. Which restates the original rule with the machinery filled in. Messages are for the model, state is for your code, and the bridge between them is a hook you write yourself.

Summarizing, but only at the cliff

The obvious fix is to summarize the oldest messages instead of dropping them, and my first attempt at demonstrating it failed in an instructive way.

I ran the same seven turns with summary_ratio=0.3, preserve_recent_messages=2 and proactive compression armed at 0.001, about 300 tokens on this model. At the end there were 14 messages, no summary anywhere, and the codename still sitting in the raw first message. The manager had never fired.

THRESHOLD   : 0.001
TURN 7 ask     | msgs=14 | Your codename is ORION, as you previously stated. Is there
                 anything else I can assist you with?
ROLES_IN_WINDOW: ['user', 'assistant', 'user', 'assistant', ...]
ORION_VISIBLE_TO_MODEL: True

The reason is in the code, not the name. This manager has no window size at all, and its apply_management leaves history untouched. Summarization happens on exactly two triggers: a real context overflow, or the base class's proactive check. Neither fired, because fourteen short messages occupy a rounding error of a 300,000-token context window. The manager was not ignoring the conversation; it was waiting for a cliff that a demo conversation never reaches.

Making the cliff reachable

Proactive compression is the knob that makes the cliff reachable. It runs before every model call at conversation_manager.py:132, where the check divides projected input tokens by the context window and compares the ratio against a threshold, 0.7 by default and with no floor.

from strands.agent.conversation_manager import SummarizingConversationManager

session_manager = FileSessionManager(
    session_id="summary-demo", storage_dir="./sessions_tmp"
)
conversation_manager = SummarizingConversationManager(
    summary_ratio=0.3,
    preserve_recent_messages=2,
    proactive_compression={"compression_threshold": 0.00005},
)
agent = Agent(
    model=model,
    session_manager=session_manager,
    conversation_manager=conversation_manager,
    callback_handler=None,
)

On a 300,000-token window, 0.00005 works out to fifteen tokens, small enough that the conversation crosses it immediately. The same seven turns then behave differently.

The count rose by two per turn until the compression engaged, then by one. Each firing selects max(1, int(len(messages) * summary_ratio)) messages, capped by preserve_recent_messages, and replaces them with a single summary, so two or three old messages collapse into one and the net gain per turn drops to a single entry. Turn seven is the exception: nine stays nine, because the run ends with removed_message_count: 6 and one summary the manager re-injects, and 14 − 6 + 1 = 9. Nine is where this transcript settled, not a fixed ceiling.

What did not change is the role. The window still opens with two user entries, because the summary itself is injected as a user message. What replaced the original turn is its content, not its role.

## Conversation Summary
* Topic 1: Request for conversation summary
  * The user, identified by the codename ORION, requested a summary of the conversation.

The codename survived, and it survived as a line inside a model-written summary, which is a weaker guarantee than a file on disk. A summary is lossy and the summarizer is the same model whose memory you are trying to protect. This run also logged a warning worth reading twice: "Cannot summarize: only the previous summary is old enough".

With a short window and summary_ratio=0.3, the selection rounds down to one message, and that one message is the summary it wrote last time. The manager declines, logs, and continues rather than raising, which is right for a context-pressure path and easy to miss in production.

What surprised me

The first surprise is the shape of forgetting. I expected a missing codename to produce "I do not know". Instead the model built a plausible refusal out of the six messages it could still see. Memory failure in an agent looks like hallucination, and that makes it much harder to catch in a log review than a raised exception.

The second is that SummarizingConversationManager does not summarize on message count. The name suggests a window; the implementation waits for context pressure. Anyone who configures it expecting sliding-window behavior, and never crosses the threshold, gets an agent that grows without bound while the configuration looks correct.

The third is how small the distance is between working and not. The mechanism never changed between the two runs above. Only the ratio did: 0.001 was invisible, 0.00005 was decisive, on the same model and the same seven turns.

When not to use this

If every invocation is a single turn with no follow-up, skip sessions entirely. A session manager writes files you will never read and restores state you will never reuse, and the cost is paid on every call.

Do not use agent.state as a database. It is a small JSON dict that travels with the agent snapshot, which makes it right for a plan id or a retry counter and wrong for anything large, queryable or shared across agents.

And do not treat a summary as a record. If a fact has to survive compression, put it in state and inject it as above, or write it into the system prompt directly, not in the hope that the summarizer keeps it. The codename survived here because it was the most salient thing in seven turns. A detail mentioned once inside a long tool result will not be so lucky.

Try it yourself

git clone https://github.com/pinxiong/strands-in-action.git
cd strands-in-action/usecases/04-sessions-state-memory
uv venv --python 3.11
VIRTUAL_ENV=.venv uv pip install strands-agents==1.63.0

Run with AWS credentials that can call amazon.nova-pro-v1:0 in us-east-1. Delete src/sessions_tmp/ and src/snapshots_tmp/ before every rerun: the session ids are hard-coded, so a second first appends to the previous transcript and the file counts above stop matching.

rm -rf src/sessions_tmp src/snapshots_tmp
python src/session_restart.py first    # plant a name, list the files
python src/session_restart.py second   # fresh process, same session id
python src/agent_state.py first        # state set/get, JSON rejection
python src/agent_state.py second       # state after a restart
python src/sliding_window.py           # fourteen stored, six visible
python src/state_manifest.py           # same window, codename returns via the manifest
python src/summarizing_window.py       # summary keeps the codename
rm -rf src/sessions_tmp/session_summary-demo
COMPRESSION_THRESHOLD=0.001 python src/summarizing_window.py   # nothing happens
python src/snapshot_session.py first   # one snapshot blob
python src/snapshot_session.py second  # recall from that blob

The two summarizing runs share session_id="summary-demo", which is why the rm -rf sits between them. Without it the second run restores the nine messages the first one left behind, and the counts above will not repeat.

Takeaways

Sessions are files, windows are lists in memory. The first survives a restart, the second survives nothing.
agent.json carries three kinds of memory at once: application state, window bookkeeping and interrupt state, restored together.
agent.state refuses a non-JSON value the moment you set it, and reaches disk only on the next invocation.
Stored is not surfaced: agent.state reaches the model only if you inject it, and a BeforeInvocationEvent hook rewriting the system prompt is the smallest bridge.
A window of six holds three turns, so turn four is where visible and stored history part ways.
A forgotten fact comes back as a confident refusal, so overflow looks like hallucination rather than amnesia.
Summarizing is triggered by context pressure, not message count; below the threshold it never runs at all.
The threshold is a ratio with no floor, which is what makes compression testable in a demo.
A summary message keeps the user role and loses detail, so it is a weaker record than state.
One snapshot file trades a readable audit trail for consistent restores, at most one message behind when saving on every message.

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4

xbill — Fri, 06 Nov 2026 18:41:59 +0000

This article provides a step by step comparison of six inference serving engines on one Amazon EC2 G6 instance: llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and Google's LiteRT-LM, each serving Gemma 4 E2B on Google's QAT 4-bit weights from one NVIDIA L4.

https://github.com/xbill9/gemma4-dev

The question is which engine to put on a G6 when one small model has the whole GPU. Each engine gets the same weights, the same prompts, the same 1-to-64-request load sweep and the same 1,319 GSM8K and 400 BFCL questions, and each is measured with and without Google's E2B drafter wherever the engine supports one. Every engine ran on one instance, one at a time, with the card to itself.

The Results in Brief

Answer quality barely moves between engines. GSM8K lands between 0.8954 and 0.9014 on the five CUDA engines with the drafter on or off, and every paired difference against llama.cpp is within the run-to-run range. LiteRT-LM scores level on GSM8K and lower on BFCL.

For one request, five engines are within 8% of each other without a drafter (134.7 to 144.6 tokens per second end to end) and LiteRT-LM runs at 0.62 to 0.73x llama.cpp. The drafter decides the order: it gives vLLM 1.96x, llama.cpp 1.53x, SGLang 1.39x and LiteRT-LM 1.15x, and it slows mistral.rs to 0.95x.

Under load the batching engines separate. At 64 requests of 128 tokens in and 512 out, vLLM reaches 3,716 tokens per second without a drafter and 4,516 with it, SGLang 2,967, mistral.rs 1,825 and llama.cpp 1,056. PyTorch and LiteRT-LM serve one request at a time.

With each engine at its best setting, vLLM is the fastest at every level from 1 to 64 requests, on both shapes, and its drafter never loses. At the g6.4xlarge on-demand price that is $0.08 per million output tokens at full load.

What Changes and What Stays

Engine	Version	Weights	Installed with
llama.cpp	`7182ef71e`	exact Q4_0 GGUF	CMake, CUDA 13.1, sm_89
vLLM	0.33.0	W4A16, int4 tables	pip, its own `torch 2.14.0+cu130`
SGLang	0.5.23	W4A16, int4 tables	pip, cp313 wheel on Python 3.14, two patches
mistral.rs	v0.9.5	exact Q4_0 GGUF	`cargo install`, CUDA 13.1, sm_89
PyTorch	this repo's int4 engine	W4A16	the DLAMI's torch 2.13
LiteRT-LM	0.19.0	exact Q4_0, and Google's int2/4/8 file	pip, GPU through WebGPU on Vulkan

The same for all six: one g6.4xlarge on-demand in us-east-1d (16 vCPUs, 64 GiB, one NVIDIA L4 with 23,034 MiB, driver 595.112.07), the PyTorch 2.13 / Ubuntu 26.04 Deep Learning AMI, Gemma 4 E2B on Google's QAT Q4_0 grid in every container, greedy decoding, and the system Python 3.14 with no virtual environment. The exact Q4_0 GGUF and the W4A16 checkpoint carry the same 4-bit values: xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf for llama.cpp and mistral.rs, xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 for vLLM and SGLang.

At This Point You Should Have…

An AWS account with G-family vCPU quota in us-east-1 and an instance profile with SSM access
An S3 bucket for moving files to and from the instance
The system python3 with boto3, and the MCP server in gpu-pytorch-g6-2b-q4w4a16/
A clone: git clone https://github.com/xbill9/gemma4-dev

Step 1 — Find a G6 With Capacity

The rigs name a g6.2xlarge (8 vCPUs, one L4). On 2026-11-05 it had no spot or on-demand capacity in any us-east-1 zone, and neither did g6.4xlarge spot. The MCP tool create_g6_instance walks every subnet, spot first and then on-demand:

subnet-0c2872fe4182b9ec1 spot=True: ❌ AWS InsufficientInstanceCapacity: There is no Spot capacity available that matches your request.
subnet-09e0f13c0a5b43092 spot=True: ❌ AWS InsufficientInstanceCapacity: There is no Spot capacity available that matches your request.
subnet-0c2872fe4182b9ec1 spot=False: ❌ AWS InsufficientInstanceCapacity: We currently do not have sufficient g6.4xlarge capacity in the Availability Zone you requested (us-east-
subnet-0d2b2038965e8d0c0 spot=False: ✅ Launching `i-040ed957b67c2d514` (g6.4xlarge, on-demand, 1x L4) in `us-east-1`.
AMI: `ami-0f5139ed9ed52b805`

Both sizes carry the same L4. The g6.4xlarge doubles the vCPUs and memory, which only llama.cpp's prompt threads and the build steps use.

🔎 Tip: all access is SSM and S3. The instance has no SSH key and its role has no S3 access. Commands go through SSM Run Command, which returns at most 24,000 characters, so files larger than that move through presigned S3 URLs generated on the workstation.

Step 2 — One Rig Directory per Engine

Each engine has its own directory with a tpu.env that decides every serving flag, a Makefile run on the instance, and its measurements under benchmarks/runs/:

gpu-llamacpp-g6-2b-q4_0/
gpu-vllm-g6-2b-q4w4a16emb4/
gpu-sglang-g6-2b-q4w4a16emb4/
gpu-mistralrs-g6-2b-q4_0/
gpu-pytorch-g6-2b-q4w4a16/
gpu-litertlm-g6-2b-int248/
gpu-litertlm-g6-2b-q4w4a16emb4/

gpu-vllm-g6-2b-q4w4a16emb4/flight/setup.sh installs every engine in order and logs each step, so a failed step reruns alone:

GEMMA4_DEV=/opt/gemma4-dev MODEL_S3_URI=s3://... bash setup.sh llamacpp vllm sglang mistralrs litert
18:48:44 START llamacpp
18:56:56 END llamacpp
18:56:56 START vllm
18:59:18 END vllm
18:59:18 START sglang
18:59:57 END sglang

Step 3 — What the Deep Learning AMI Needs

The AMI ships the NVIDIA driver and a PyTorch 2.13 environment, and no CUDA toolkit. Four things differ from a desktop install:

vLLM brings its own torch. The system Python has no torch, so pip install vllm==0.33.0 pulls its pinned wheel from PyPI, built for CUDA 13.0, and it runs on driver 595.112.07:

torch 2.14.0+cu130 cuda 13.0 NVIDIA L4 (8, 9)

llama.cpp and mistral.rs build against CUDA 13.1 from NVIDIA's apt repository, for sm_89. mistral.rs also links cuRAND, which is a separate package: libcurand-dev-13-1.
LiteRT-LM finds the L4 through Vulkan. The AMI lists the L4 as a Vulkan device, and LiteRT-LM's WebGPU backend serves on it.
pip puts scripts in ~/.local/bin. SSM runs commands as root with HOME=/root, and that directory is off its PATH, so vllm, litert-lm and SGLang's ninja are found only once it is added.

🔎 Tip: SGLang 0.5.23 needs two patches for this checkpoint. Its Gemma 4 attention shim runs only for multimodal configs, and it leaves the int4 embed_tokens and per-layer table as uninitialized bf16. The rig's patches/ keep both tables packed on the card.

Step 4 — Size Each Engine for the Whole Card

The card serves one engine at a time and nothing else, so each engine gets the most the L4 will hold:

vLLM reserves 85% of the card (--gpu-memory-utilization 0.85), 64 running requests, 16,384 tokens each.
SGLang reserves 80% (--mem-fraction-static 0.80), 64 running requests, decode CUDA graphs up to 64.
llama.cpp runs 64 slots over a 65,536-token cache for the load sweep, and one slot of 131,072 for accuracy.
mistral.rs runs 32 requests over a fixed 16,384-token cache.

🔎 Tip: mistral.rs plans activation memory per running request. It sets aside about 395 MiB for each, so 64 requests do not load on the L4's 23 GB:

Error: Primary device cuda[0] needs 28292 MiB (2976 MiB of fixed model components, 25316 MiB of activations, 0 MiB of post-load reservations), but only 21192 MiB is usable under the memory goal.

At 32 requests a 65,536-token cache leaves too little room; --max-seqs 32 --kv-cache 16384tokens --max-num-batched-tokens 4096 serves in 5,286 MiB.

Step 5 — One Request at a Time

flight/engines.py sends six chat prompts, 256 tokens each, three times per engine, and takes the geometric mean of each prompt's median, end to end. Then 4 requests at once:

arm              e2e tok/s vs llama  decode TTFT ms 4 at once  x one    memory ready s repeat
llama                135.7    1.000   136.9      29     437.6   3.23  2930 MiB     6.1   6/6
llama-d2             207.7    1.531   211.4      30     373.2   1.80  3004 MiB     4.1   6/6
mistralrs            144.6    1.066   144.4      11     500.0   3.46  4664 MiB    24.1   6/6
pytorch              134.7    0.993       -       -     134.5   1.00  3248 MiB   236.2   6/6
litert-int248         98.4    0.725   125.2     512     100.5   1.02  1489 MiB     9.4   6/6
vllm                 140.4    1.035   140.7      16     537.2   3.83 18228 MiB   167.3   6/6
vllm-d4              275.3    2.030   281.2      19     711.3   2.58 18918 MiB   132.5   6/6
sglang               136.5    1.006   139.7      50     475.4   3.48 18345 MiB   307.2   6/6
sglang-d4            189.6    1.398   198.1      55     508.3   2.68 15191 MiB    61.1   6/6

Engine	No drafter	Best drafter	Drafter gain
vLLM	140.4	🥇 275.3 (4 tokens)	1.96x
llama.cpp	135.7	207.7 (2 tokens)	1.53x
SGLang	136.5	189.6 (4 tokens)	1.39x
mistral.rs	144.6	137.6 (1 token)	0.95x
PyTorch	134.7	—	—
LiteRT-LM	98.4	112.8 (its own)	1.15x

LiteRT-LM's exact Q4_0 build runs at 84.4 tokens per second, and LiteRT-LM takes about half a second to its first token against 11 to 55 ms for the others. Every engine repeats its own greedy text on all six prompts.

Step 6 — 1 to 64 Requests

sweep.py runs two shapes, 512 tokens in / 128 out and 128 in / 512 out, at 1 to 64 requests at once, five repeats, a unique prompt per request so no prefix cache can hit, output timed from the stream. flight/suite.py runs every sweep and every accuracy job one server at a time, and marks each finished job so a rerun skips it:

python3 suite.py /opt/flight-out/2026-11-06-suite-g6 sweep:vllm,sweep:vllm-d1,sweep:vllm-d4,sweep:sglang,...
2026-11-06 04:06:20 START sweep:vllm ready_s=39.1 gpu=52, 2040 MHz, 19296 MiB
2026-11-06 04:07:33   sweep:vllm in512-out128 rc=0
2026-11-06 04:10:07   sweep:vllm in128-out512 rc=0
2026-11-06 04:10:08 END sweep:vllm ok=True

Aggregate output tokens per second without a drafter, 128 in / 512 out:

Requests	vLLM	SGLang	mistral.rs	llama.cpp
1	138.8	134.6	133.9	133.8
8	991.0	967.4	626.5	597.5
64	🥇 3,715.7	2,967.1	1,825.2	1,056.3

512 in / 128 out:

Requests	vLLM	SGLang	mistral.rs	llama.cpp
1	132.8	129.6	133.1	125.4
8	738.3	708.3	591.2	428.1
64	🥇 1,593.7	1,373.0	1,132.0	630.8

All 126 levels completed. vLLM's output at 64 requests is 26.8x its one-request rate on long replies. SGLang reaches 0.80 to 0.86x vLLM at 64, mistral.rs 0.49 to 0.71x and llama.cpp 0.28 to 0.40x.

Step 7 — Each Engine at Its Best

best_per_level.py picks each engine's fastest measured setting at each level, 128 in / 512 out:

Requests	vLLM	SGLang	llama.cpp	mistral.rs
1	🥇 453 (4 drafted)	283 (4 drafted)	271 (2 drafted)	134 (none)
2	🥇 856 (4 drafted)	552 (4 drafted)	430 (2 drafted)	254 (none)
4	🥇 1,483 (4 drafted)	875 (4 drafted)	596 (2 drafted)	468 (none)
8	🥇 2,399 (4 drafted)	1,223 (4 drafted)	791 (2 drafted)	627 (none)
16	🥇 2,823 (4 drafted)	1,673 (none)	1,021 (2 drafted)	1,339 (none)
32	🥇 3,665 (1 drafted)	2,257 (none)	1,132 (2 drafted)	1,843 (none)
64	🥇 4,516 (4 drafted)	2,967 (none)	1,170 (2 drafted)	1,825 (none)

512 in / 128 out:

Requests	vLLM	SGLang	llama.cpp	mistral.rs
1	🥇 345 (4 drafted)	214 (4 drafted)	224 (2 drafted)	133 (none)
8	🥇 1,217 (4 drafted)	711 (4 drafted)	494 (2 drafted)	591 (none)
16	🥇 1,389 (1 drafted)	1,026 (none)	572 (2 drafted)	1,121 (none)
64	🥇 1,710 (1 drafted)	1,373 (none)	631 (none)	1,132 (none)

vLLM's drafter pays at every level: with 4 drafted tokens 2.60x / 3.26x no drafter for one request, still 1.02x / 1.22x at 64. SGLang's drafter pays up to 4 to 8 requests and loses from 16 (0.73x / 0.69x at 64). llama.cpp's 2-token drafter pays at every level on long replies (1.11x at 64) and up to 16 requests on long prompts.

Step 8 — Answer Quality

gen_eval.py asks each engine all 1,319 GSM8K test questions (2,048-token limit) and the 400 BFCL simple function-calling questions (512), one request at a time, greedy, thinking off:

Engine	GSM8K	BFCL	GSM8K with drafter	BFCL with drafter
llama.cpp	0.8984	0.9250	0.8976	0.9200
vLLM	0.8969	0.9225	0.8992	0.9225
SGLang	0.8954	0.9150	0.8992	0.9150
mistral.rs	0.9014	0.9200	—	—
PyTorch	0.8961	0.9225	—	—
LiteRT-LM, exact Q4_0	0.8999	0.8600	—	—
LiteRT-LM, Google's file	0.8908	0.8250	—	—

Paired question by question, every engine is level with llama.cpp on GSM8K, and the drafter changes no score. On BFCL, SGLang is 4 questions behind llama.cpp, a gap bigger than the run-to-run range, and LiteRT-LM is 26 and 40 behind.

The drafter cuts the time to finish GSM8K:

Engine	No drafter	With drafter	Speedup
vLLM	3,918 s	🥇 1,270 s	3.08x
SGLang	4,035 s	1,956 s	2.06x
llama.cpp	3,971 s	2,007 s	1.98x
mistral.rs	3,918 s	—	—
PyTorch	3,991 s	—	—
LiteRT-LM	5,018 s	—	—

🔎 Tip: send thinking off explicitly to SGLang and mistral.rs. "chat_template_kwargs": {"enable_thinking": false} keeps replies short on every engine; mistral.rs thinks by default on Gemma 4.

Step 9 — Cost

At the g6.4xlarge on-demand price, $1.3232 an hour in us-east-1 from the AWS Price List API, the measured throughput works out to (arithmetic, from cost.py):

Setting	128 in / 512 out	512 in / 128 out
vLLM + drafter, 1 request	$0.81 per M tokens	$1.06 per M tokens
vLLM + drafter, 64 requests	🥇 $0.08	🥇 $0.21 (1 drafted)
SGLang, 64 requests	$0.12	$0.27
mistral.rs, 64 requests	$0.20	$0.32
llama.cpp + drafter, 64 requests	$0.31	$0.59

The whole sweep and accuracy suite ran 12.2 hours, $16.14 on demand. A g6.2xlarge carries the same L4 at $0.9776 an hour; it was not measured here.

Step 10 — Tear Down

Results come off the instance as one tarball through a presigned S3 PUT, and then the instance is terminated:

python3 -c "import boto3; print(boto3.client('ec2', region_name='us-east-1').terminate_instances(InstanceIds=['i-040ed957b67c2d514'])['TerminatingInstances'][0]['CurrentState']['Name'])"
shutting-down

Compare and Contrast

	vLLM	SGLang	llama.cpp	mistral.rs
1 request, best	🥇 275.3	189.6	207.7	144.6
64 requests, 128/512, best	🥇 4,516	2,967	1,170	1,825
Drafter under load	🟢 pays at every level	❌ 0.69x at 64	🟢 to 16–64	❌ slower
GSM8K level	🟢 yes	🟢 yes	🟢 yes	🟢 yes
BFCL level	🟢 yes	⚠️ 4 behind	🟢 yes	🟢 yes
Runs as installed	🟢 yes	❌ two patches	🟢 yes	⚠️ needs cuRAND, fixed cache

PyTorch and LiteRT-LM serve one request at a time:

	PyTorch	LiteRT-LM
1 request, best	134.7	112.8
GSM8K level	🟢 yes	🟢 yes
BFCL level	🟢 yes	❌ 26–40 behind
Runs as installed	🟢 yes	⚠️ Vulkan

So, Which One?

On a G6 serving one small model, run vLLM with Google's W4A16 drafter at 4 tokens: it is the fastest engine for one request (275.3 tokens per second) and at every load to 64 requests, it finishes GSM8K 3.08x sooner than without the drafter at the same score, and at full load it costs $0.08 per million output tokens on demand. SGLang without a drafter reaches 0.66x of that at 64 requests and needs two patches. llama.cpp with its 2-token drafter is the simplest to build and serves one user at 207.7 tokens per second, but tops out near 1,170 tokens per second. mistral.rs, PyTorch and LiteRT-LM suit one user at a time.

Summary

The goal of this article was to find the inference serving engine to run for Gemma 4 E2B on an Amazon EC2 G6. The key was holding everything else fixed: one instance, the same 4-bit values in every container, the same prompts, load sweep and question sets, and each engine sized for the whole L4. The six-engine results were:

🟢 All six engines give the same GSM8K answers (0.8908 to 0.9014) with the drafter on or off.
🟢 vLLM with its drafter is the fastest engine at every level from 1 to 64 requests: 275.3 tokens per second for one request and 4,516 at 64 (128 in / 512 out).
🟢 vLLM's drafter pays at every load on the L4, and finishes GSM8K 3.08x sooner.
🟢 At full load vLLM serves a million output tokens for $0.08 on demand.
⚠️ The drafter's gain depends on the engine: 1.96x vLLM, 1.53x llama.cpp, 1.39x SGLang, 1.15x LiteRT-LM, 0.95x mistral.rs.
⚠️ g6.2xlarge had no capacity in us-east-1 on the day; g6.4xlarge on-demand carries the same L4.
❌ mistral.rs loads at most 32 requests on the L4's 23 GB.
❌ SGLang's drafter cuts output to 0.69 to 0.73x of running without it at 64 requests.

Scope: one g6.4xlarge on-demand in us-east-1d, one NVIDIA L4, the Deep Learning AMI with driver 595.112.07, Gemma 4 E2B on Google's QAT Q4_0 grid, one engine at a time, greedy decoding. One-request speed is six prompts of 256 tokens, three repeats; load is two synthetic shapes at 1 to 64 requests, five repeats each; accuracy is one pass of GSM8K (1,319) and BFCL simple (400). The one-request runs were on 2026-11-05 and the sweeps and accuracy on 2026-11-06, on the same instance; costs are arithmetic from the on-demand price.

The strategy for comparing inference serving engines on Amazon EC2 G6 was validated with an incremental step by step approach.

References

[Part 3] Your first steps in Computer Vision on AWS: deploying models for an ASL recognition app

Darya Petrashka — Fri, 06 Nov 2026 15:54:57 +0000

This is the third and final part of the series "Your first steps in Computer Vision on AWS". If you are just starting, this tutorial will guide you step by step from the initial dataset to a deployed app.

You can read the first part on ASL dataset + preprocessing and the second part on a custom CNN training.

What we are trying to build

While most ML tutorials end with model training and validation, we will take things one step further and deploy our models in an app. This is the most interesting and exciting part! We will make our models live!

Here is what the AWS architecture will look like:

Here's a quick overview of the solution:

The best way to test models is to take photos using a mobile phone. A simple static, serverless ASL Alphabet Recognition web application is hosted on S3 and accessed via a CloudFront URL. To call the models for predictions, we will use API Gateway. A Lambda event function will preprocess the uploaded photo and send it to a serverless SageMaker inference endpoint. The endpoint will be created using a container stored in ECR.

Most of the AWS infrastructure will be created automatically using CloudFormation (not shown on the architecture image). This will make the setup reproducible. However, we will first need to create the ECR repositories, build and push the SageMaker and Lambda container images, and upload the model archive to S3. The website files will also be uploaded to S3 manually after the stack is deployed.

❓ Why serverless endpoint?
Our use case will have idle periods between traffic spurts and can tolerate cold starts. Serverless endpoints automatically launch compute resources and scale them in and out (even down to 0) depending on traffic, eliminating the need to choose instance types or manage scaling policies. With a pay-per-use model, Serverless Inference is a cost-effective option in our infrequent traffic pattern. The serverless endpoint supports a maximum memory size of 6144 MB (6 GB), which is more than enough for our three models.

❓ What would be the app cost?
I spent less than $5 on the application. The serverless endpoint was running for around a week, but the cost was minimal. I spent more on the ml.g5.xlarge instance in the previous part.

Prepare models for inference

⚠️ For this step, you can use your local environment or a SageMaker notebook instance (ml.t3.medium is more than enough).

Models trained in the previous part are ready: put raw.keras, filtered.keras, and edges.keras into models/
You will also need to add the class_names.json file to the models/ directory (copy/download it from the GitHub repo).
It contains the exact class mapping used during training.

From inside models/ directory run the command:

tar -czvf asl-models.tar.gz \
    raw.keras \
    filtered.keras \
    edges.keras \
    class_names.json

You will get the archive asl-models.tar.gz. Upload it to S3:

aws s3 cp asl-models.tar.gz \
    s3://YOUR-BUCKET/asl-app/models/asl-models.tar.gz

⚠️ Don't forget to replace the placeholder YOUR-BUCKET and put your S3 bucket name!

Once the archive has been uploaded to S3, we will use its S3 URI in CloudFormation.

s3://YOUR-BUCKET/asl-app/models/asl-models.tar.gz

Build and push SageMaker inference container to ECR

⚠️ For this step, you can use your local environment with AWS CLI (How to install/update AWS CLI?) or a SageMaker notebook instance (ml.t3.medium is more than enough).

⚠️ Copy/download inference.py, requirements.txt, and Dockerfile files from the GitHub repo. They should sit inside the sagemaker directory (same structure as in the repo).

The code in inference.py is pretty simple: its main job is to handle inference for all three models.

Now, create an ECR repository:

aws ecr create-repository \
    --repository-name asl-sagemaker-inference

Get your account ID:

aws sts get-caller-identity \
    --query Account \
    --output text

Set variables:

export AWS_REGION=YOUR-REGION
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity \
    --query Account \
    --output text)

export ECR_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

aws ecr get-login-password \
    --region $AWS_REGION \
    | docker login \
    --username AWS \
    --password-stdin $ECR_URI

Build and tag:

cd sagemaker

docker build \
    -t asl-sagemaker-inference:latest .

docker tag \
    asl-sagemaker-inference:latest \
    $ECR_URI/asl-sagemaker-inference:latest

And finally push:

docker push \
    $ECR_URI/asl-sagemaker-inference:latest

⚠️ Make sure that your role or identity has the required permissions to do the job. Required actions: ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, ecr:BatchCheckLayerAvailability, and ecr:PutImage.

You will use in CloudFormation:
ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/asl-sagemaker-inference:latest

Build Lambda image

The main task of the Lambda function will be to accept an image from a phone, crop it to the center at 200×200 pixels, do the required preprocessing for the Gaussian-filtered model and Canny edge model and finally obtain predictions from all three models by invoking the SageMaker endpoint. For each model, it will get the top 3 predictions with their probabilities.

⚠️ For this step, you can use your local environment with AWS CLI (How to install/update AWS CLI?) or a SageMaker notebook instance (ml.t3.medium is more than enough).

⚠️ Copy/download lambda_function.py, requirements.txt, and Dockerfile files from the GitHub repo. They should sit inside the lambda directory (same structure as in the repo).

Then, create a new ECR repository:

aws ecr create-repository \
    --repository-name asl-preprocessing-lambda

Get your account ID and set variables:

aws sts get-caller-identity \
    --query Account \
    --output text

export AWS_REGION=YOUR-REGION
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity \
    --query Account \
    --output text)

export ECR_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

aws ecr get-login-password \
    --region $AWS_REGION \
    | docker login \
    --username AWS \
    --password-stdin $ECR_URI

Build, tag, and push:

cd lambda

docker buildx build \
    --platform linux/amd64 \
    --provenance=false \
    -t asl-preprocessing-lambda:latest \
    .

docker tag asl-preprocessing-lambda:latest \
    $ECR_URI/asl-preprocessing-lambda:latest

docker push $ECR_URI/asl-preprocessing-lambda:latest

You will use the Lambda image in CloudFormation:
ACCOUNT.dkr.ecr.REGION.amazonaws.com/asl-preprocessing-lambda:latest

Deploy the CloudFormation stack

⚠️ For this step, you can use your local environment with AWS CLI (How to install/update AWS CLI?) or a SageMaker notebook instance (ml.t3.medium is more than enough).

⚠️ Copy/download the cloudformation.yaml file from the GitHub repo.

This CloudFormation stack will create all the resources required for the serverless ASL Alphabet Recognition web application: CloudFront, S3 bucket, API Gateway, Lambda event function, SageMaker Serverless endpoint. As shown in the architecture diagram, CloudFront will route requests to /api/* to API Gateway, giving you a single public URL for the application.
Before deploying the stack, you will need to retrieve three values:

Type	Value
Model	`s3://YOUR-BUCKET/asl-app/models/asl-models.tar.gz`
SageMaker image	`ACCOUNT.dkr.ecr.REGION.amazonaws.com/asl-sagemaker-inference:latest`
Lambda image	`ACCOUNT.dkr.ecr.REGION.amazonaws.com/asl-preprocessing-lambda:latest`

Let's deploy the stack:

aws cloudformation deploy \
    --template-file cloudformation.yaml \
    --stack-name asl-recognizer \
    --capabilities CAPABILITY_IAM \
    --parameter-overrides \
        ModelS3Uri=s3://YOUR-BUCKET/asl-app/models/asl-models.tar.gz \
        SageMakerImageUri=ACCOUNT.dkr.ecr.REGION.amazonaws.com/asl-sagemaker-inference:latest \
        LambdaImageUri=ACCOUNT.dkr.ecr.REGION.amazonaws.com/asl-preprocessing-lambda:latest

⚠️ Make sure that your role or identity has the required permissions to successfully deploy your CloudFormation stack. Required actions: cloudformation:DescribeStacks, s3:CreateBucket, s3:ListAllMyBuckets, s3:GetBucketLocation, and apigateway:POST.

It will take around 10-15 mins to complete. A good thing is that you can monitor stack events in the AWS CloudFormation Management Console:

Upload the website

⚠️ For this step, you can use your local environment with AWS CLI (How to install/update AWS CLI?) or a SageMaker notebook instance (ml.t3.medium is more than enough).

⚠️ Copy/download app.js, index.html, and styles.css files from the GitHub repo. They should sit inside the frontend directory (same structure as in the repo).

After CloudFormation finishes, retrieve the bucket:

aws cloudformation describe-stacks \
    --stack-name asl-recognizer \
    --query \
    "Stacks[0].Outputs[?OutputKey=='FrontendBucket'].OutputValue" \
    --output text

The command will return a bucket name similar to asl-recognizer-frontend-12000000. Use this value to upload your website pages from the frontend directory to S3:

aws s3 sync frontend/ s3://asl-recognizer-frontend-12000000/

After this, retrieve the CloudFront URL:

aws cloudformation describe-stacks \
    --stack-name asl-recognizer \
    --query \
    "Stacks[0].Outputs[?OutputKey=='WebsiteURL'].OutputValue" \
    --output text

You'll get the CloudFront URL (e.g. https://d1234567890abc.cloudfront.net). Paste it in the browser to access your application.
As you can see, the web application's UI is very simple. Its main purpose is to let us test the models and get predictions:

Again, the provided GitHub repo has all the required files for models inference preparation, SageMaker inference container creation, Lambda image creation, as well as website pages and the CloudFormation stack deployment. The repo structure is:

asl-web-app/
│
├── models/
│   ├── raw.keras
│   ├── filtered.keras
│   ├── edges.keras
│   └── class_names.json
│
├── sagemaker/
│   ├── inference.py
│   ├── Dockerfile
│   └── requirements.txt
│
├── lambda/
│   ├── lambda_function.py
│   ├── Dockerfile
│   └── requirements.txt
│
├── frontend/
│   ├── index.html
│   ├── app.js
│   └── styles.css
│
└── cloudformation.yaml

Test the app

The most exciting part begins! Use your phone to take photos and make predictions.
First, I tried several images from the asl_alphabet_test folder, which contains additional images provided alongside the training dataset.
All three models performed reasonably well, producing the correct class predictions:

However, you may notice that the models' predictions on your own photos are not always correct. I tried to show a 'W': the model trained on raw RGB returned 'R', Gaussian-filtered and Canny edge models predicted 'K':

All our models achieved more than 90% accuracy on our held-out test split. They also performed well on additional images from Kaggle's separate asl_alphabet_test folder, which were not included in the training, validation, or test splits. But when facing real-world images, models started to act weird.

This reveals an important lesson about ML models and their applications outside the collected dataset. There is a gap between a toy dataset and real images; and we didn't fully account for it. If you look closer at training images, they all share a similar background, same skin tone, same hand size. Lightning and focus do vary, but not drastically. What if the person signing in ASL has a darker skin tone? What if we use a green/blue/orange background?

In this project, we got too excited and proceeded with training using the data we had. And we are not alone. If you check the Kaggle dataset notebooks, many practitioners achieved almost 100% accuracy. The model works perfectly on the provided images, but its ability to generalize to real-world scenarios is not guaranteed.

📖 For the future, we need to be aware of data cascades: issues in data are 'cascaded' down the line, and the key strategy is to detect such data issues and work on them early on, before any model is trained. (Useful read: "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI).

📖 It is very possible that our models relied on shortcut learning (read Shortcut learning in deep neural networks). One way to investigate this is to use explainable AI methods, such as Grad-CAM. It can help visualize which parts of an image influence a CNN's predictions.

Conclusion

In this part, we deployed three CNN models to a single serverless SageMaker inference endpoint. We also used CloudFormation to provision the AWS resources needed to build a simple ASL Alphabet Recognition web application. Next, we tested the app with the test dataset images, as well as with our custom images. Finally, we found a gap between a toy Kaggle dataset and real-world conditions. The ability to overcome this gap is what distinguishes a simple proof of concept (POC) from a production-ready ML solution.

Beyond the Collector: How the Open-Source Observability Stack Fits Together

hridyesh bisht — Fri, 06 Nov 2026 07:13:59 +0000

In Understanding Observability with OpenTelemetry and Coffee, I introduced the fundamentals of observability. In Native OpenTelemetry, I explored how OpenTelemetry generates and processes telemetry, how the OpenTelemetry Collector manages telemetry pipelines, and why understanding those pipelines matters when troubleshooting production issues.

But what happens after telemetry leaves the Collector? Which tools store and analyze it? How do you connect these tools, and what should you consider before operating them in production?

This article explores the open-source observability ecosystem, from the responsibilities of individual tools to the engineering decisions involved in running them together.

OpenTelemetry isn't your observability backend

OpenTelemetry is an open-source, vendor-neutral observability framework. It provides APIs, SDKs, instrumentation libraries, and tools for generating, collecting, and exporting telemetry such as metrics, logs, and traces.

It also defines common data formats and semantic conventions that help applications describe telemetry consistently. However, OpenTelemetry isn't an observability backend. It doesn't replace the storage, query interfaces, and dashboards that teams use to investigate production issues. Those capabilities come from other tools.

Think of OpenTelemetry as the system that standardizes how information leaves each part of the coffee shop. The observability tools help you examine that information and understand what happened.

Let's look at what each tool contributes to an observability stack.

Prometheus

Prometheus commonly scrapes metrics endpoints exposed by applications and infrastructure. This pull-based model lets Prometheus collect measurements at configured intervals and discover targets through supported service-discovery mechanisms.

Prometheus is useful when you need to answer questions such as:

Is request latency increasing?
Are error rates rising after a deployment?
Is an application approaching its CPU or memory limits?
Does the system meet its latency or availability targets?

Note: You can also use alerting rules to detect conditions that require investigation.

When integrating Prometheus with OpenTelemetry, decide how Prometheus will receive metrics. A common approach is to scrape an endpoint exposed by an application or the OpenTelemetry Collector's Prometheus exporter.

Resources

Documentation: Prometheus documentation
Community: Prometheus community channels, including the #prometheus channel on CNCF Slack.
Discussions: Prometheus user forum

Jaeger

Jaeger provides a platform for exploring distributed traces and understanding how requests move through services.

Jaeger is useful when metrics tell you that a service is slow but don't explain which operation causes the delay. You can inspect individual traces to investigate service dependencies, slow operations, and errors.

When integrating Jaeger with OpenTelemetry, configure a compatible trace ingestion path, such as an appropriate OTLP endpoint. Also plan for trace storage and retention based on your deployment requirements. Jaeger uses a configured storage backend, so you must account for that backend in your architecture.

Resources

Documentation: Jaeger documentation
Community: Jaeger community channels, including the #jaeger channel on CNCF Slack.
Discussions: Jaeger GitHub Discussions

OpenSearch

OpenSearch provides search and analytics capabilities for observability data, including application logs. Structured fields make it easier to filter records by service, environment, timestamp, or trace ID. For example, a log record might contain:

{
"service.name": "payment-service",

"severity": "ERROR",

"message": "Payment provider timed out",

"trace_id": "example-trace-id"

}

Fields such as service.name and trace_id can help you narrow a search and connect a log entry to a related request. That connection depends on your instrumentation, log format, ingestion configuration, and the way you query the data.

OpenSearch provides flexible search and analytics, but you need to configure ingestion, indexing, storage, and retention. For example, OpenSearch Data Prepper can process telemetry before sending it to OpenSearch.

Resources

Documentation: OpenSearch observability documentation
Community: Connect with the OpenSearch community, including its public Slack workspace.
Discussions: OpenSearch community forum

Perses

Perses is an open-source dashboarding tool with an open dashboard specification and support for managing dashboards as code.

Perses doesn't replace the systems that store telemetry. It queries configured data sources and displays their results. Its plugins determine which backends it can query. This separation lets teams manage dashboards independently of telemetry storage. However, dashboard definitions still depend on compatible data sources, queries, and configuration.

For example, a team might configure Prometheus as a Perses data source to display request rates, error rates, and latency measurements. Other data sources depend on the available plugins and configuration.

Resources

Documentation: Perses documentation
Community: Perses contact and community channels, including the #perses-dev channel on CNCF Slack.
Discussions: Perses GitHub Discussions

How do these tools fit together?

An observability stack isn't one application. Installing these components doesn't automatically create a working observability stack. You must decide how data reaches each backend, which components process it, and how engineers access the results.

The following diagram illustrates one possible architecture. It distinguishes the Prometheus scrape process from the trace and log ingestion paths.

This architecture has three important parts.

Ingestion: Choose how each backend receives data. The Collector supports multiple receivers and exporters, but not every backend uses the same ingestion method.
Storage and analysis: Decide which systems handle each data set. You might store metrics in Prometheus, configure a storage backend for Jaeger, and store logs in OpenSearch.
- Routing data to multiple destinations is also possible when your requirements justify the additional processing and storage.
Visualization: Decide where engineers will explore the data. Jaeger and OpenSearch provide their own interfaces. Perses supports reusable dashboards, but you must configure compatible data sources.

OpenTelemetry provides a foundation for telemetry transport. It doesn't automatically configure every backend, unify their interfaces, or correlate all the data they contain. Correlation also depends on consistent attributes, trace context, and compatible ingestion and query capabilities.

What happens when an incident occurs?

Consider a production service whose response latency suddenly increases. An engineer might follow this workflow:

Metrics help you detect and scope the problem. Traces help you locate the delay, and logs provide events that may explain it. After applying a fix, check the relevant telemetry to verify that the system has recovered.

Start with one use case

You don't need to deploy every tool at once. Start with one observability question and build the smallest setup that can help you answer it.

Define the question. For example, determine whether a service's request latency is increasing.
Choose a backend. Select a tool that supports the type of data you need to investigate.
Send test telemetry. Configure the required ingestion path and generate a small amount of test data.
Verify the results. Query the data and confirm that the expected measurements and attributes appear.
Document what you learned. Record the configuration, useful queries, and any limitations you encountered.

For example, you could start with Prometheus and query request latency for one service. Once you need to investigate individual requests, add tracing. If you also need to search detailed application events, introduce a log backend.

This approach helps you validate each component before expanding the stack. It also makes integration problems easier to identify while the system is still small.

What should you consider before choosing your stack?

A successful experiment doesn't guarantee that a stack is ready for production. Before deploying it across services and environments, evaluate the integration work and operational responsibilities.

Integration and query models

Prometheus uses PromQL. OpenSearch supports its own query and analytics capabilities. Jaeger provides an interface for exploring traces, while Perses uses configured data sources and their respective query languages.

Engineers may need to switch between tools to investigate the same incident. You also need to configure data sources, align resource attributes, and preserve trace context in logs when you want to connect related events.

Reliability, storage, and security

Each deployed component introduces operational requirements. Plan for:

Storage and retention: Estimate how data volume will grow and define retention policies for each signal.
Availability: Decide how ingestion and investigation should behave when a backend becomes unavailable.
Scaling: Determine how to handle increasing telemetry volume and query load.
Security: Control access to telemetry and protect sensitive information.
Maintenance: Manage upgrades, configuration changes, backups, and recovery.

These concerns become more important as the number of services, environments, and engineering teams grows.

Licensing and long-term flexibility

Open-source projects provide different levels of freedom depending on their licenses. Some products follow an open-core model, offering an open-source base while reserving some capabilities for commercial editions.

Before adopting a tool, review its license, governance, community activity, commercial feature boundaries, and data export options. Consider how difficult it would be to replace the component if your requirements change.

The goal isn't to choose open-source software at any cost. It's to make an informed decision about flexibility, operational effort, and long-term sustainability.

Self-managed stack or unified observability platform?

You can assemble an observability stack from individual open-source tools or use a platform that brings telemetry together in one place. The right choice depends on your team's requirements, desired control, and capacity to operate the infrastructure.

Consideration	Self-managed stack	Unified observability platform
Control	Choose and configure individual components.	Use the capabilities and configuration options provided by the platform.
Integration	Your team connects and maintains the components.	The platform provides an integrated experience for supported signals.
Operations	Your team manages the infrastructure, storage, upgrades, and maintenance.	The platform can reduce the work required to operate separate backends.
Flexibility	Customize components to meet specific requirements.	Evaluate supported integrations, export options, and platform limitations.
Best fit	Teams that need direct control and have the capacity to maintain the stack.	Teams that prioritize a unified investigation experience and reduced operational work.

Note: Neither approach is inherently better. A self-managed stack gives you control over individual components, while a unified platform can reduce integration and maintenance work, which allows you to focus on your business logic.

Dash0: Explore telemetry in one platform

Dash0 is an OpenTelemetry-native observability platform that brings metrics, logs, traces, and resource information together for analysis and investigation.

For example, when a service's latency increases, you can examine metrics to understand the impact, inspect traces to identify slow operations, and investigate related logs to find potential causes within the same platform.

Dash0 supports PromQL and uses the Perses dashboard specification. These open standards help teams reuse familiar query capabilities and dashboard workflows. Dash0 isn't a connector that automatically combines an existing Prometheus, Jaeger, OpenSearch, and Perses deployment. It's an alternative approach that can reduce the need to operate these components separately, depending on your requirements.

Choose a self-managed stack when you need control over individual components and can maintain the infrastructure. Consider Dash0 when you want a unified place to investigate telemetry with less operational overhead.

Learn more in the Dash0 documentation.

Conclusion

An open-source observability stack is an architecture you design, not a set of tools that automatically work together.

Prometheus, Jaeger, OpenSearch, and Perses address different needs. OpenTelemetry provides a foundation for instrumentation and telemetry transport, but you must configure the data paths, storage, integrations, and investigation workflows.

You can assemble and maintain those components yourself, or evaluate a platform such as Dash0 to bring telemetry into a common observability experience. The best choice is the one that meets your requirements without introducing more operational complexity than your team can manage.

The Limits That Don't Limit

Pin Xiong — Fri, 06 Nov 2026 00:43:06 +0000

Verified on 2026-11-05 against strands-agents 1.63.0, Python 3.11 and Bedrock in us-east-1. Every number below came out of a run, and the setup steps live in README.md. The model is amazon.nova-pro-v1:0 on Amazon Bedrock.

Article 01 walked one cycle and counted thirteen stop reasons. Four of them start with limit_: limit_turns, limit_output_tokens, limit_total_tokens and limit_structured_output_attempts. That post treated them as outcomes. This one is about how to cause them, and about how little each boundary guarantees once you look at what it measures. Read 01 first; everything here starts one level up, at the configuration you pass to a single call.

The agent that never stops on its own

Here is a tool that asks to be called again. It records one tick of a batch job, then reports how many remain.

from strands import Agent, tool
from strands.models import BedrockModel

@tool
def tick(progress: int) -> str:
    """Record one tick of a batch job.

    Args:
        progress: how many ticks were already recorded before this call.

    Returns:
        The total recorded so far and how many remain in the batch.
    """
    done = min(progress + 1, 8)
    remaining = 8 - done
    if remaining > 0:
        return f"Recorded {done}/8 ticks. {remaining} remain. Call tick again with progress={done}."
    return f"Recorded {done}/8 ticks. Batch complete."

Run an agent on it with no limits and the loop finishes after 9 turns and 8 tool calls, at end_turn. The ninth turn is the final answer, the one with no tool call in it. The reason it stopped is the last line of the tool output: the batch was complete. Nothing in the SDK decided that. If the tool had kept asking for another call, the loop would have kept serving it.

That is the shape of the problem. An agent is a loop whose exit condition is a sentence the model reads, which is why four walls exist to replace that sentence with something you control. Two of them come from keys in the limits dict below: turns and the two token budgets. Two live outside it: a deadline is a manual agent.cancel() call, and a quota is a hook. structured_output_attempts is a fourth key in the same dict. It caps how many failed attempts at a structured response a request may accumulate (types/agent.py:185), so it stays out of this article.

agent(prompt, limits={"turns": 3, "total_tokens": 20000})

The rest of this article is one section per boundary, because each one bounds something different from what its name suggests.

What a turn actually counts

The type definition answers this in one line: a turn is "one model call plus any tool execution that follows" (types/agent.py:177-178). So a turn limit counts model calls. It does not count tool calls, and the gap between the two is decided by the model, not by you.

Two runs make the gap visible. Same tool, same eight calls, two prompts. A prompt that asks for one call at a time produces nine turns over eight calls. A prompt that permits several calls in one response produces three turns over the same eight calls.

A turns cap of 3 would have allowed the second run to finish its work and cut the first one off after three of them. The cap was identical. What it protected was not.

So the cap itself is exact when it fires. With turns=3, the loop stops with limit_turns after three turns and three calls. With turns=1 it stops after one turn and one call, and the run ends without a summary. A cycle runs in a fixed order: check the limits, call the model, run the tools it asked for. The check sits at the top of that cycle, before the model call. That placement is what lets the source promise something useful. A cap that fires at the next cycle's top fires after the previous turn's tools completed. agent.messages also stays valid for another call, with no model reply left pointing at tool calls that never ran (types/agent.py:168-169, event_loop/event_loop.py:217).

In short, the practical reading is uncomfortable. A turn limit is a budget on model calls. If your cost driver is tool invocations, this wall is in the wrong place.

The token cap that never sees the answer

output_tokens and total_tokens look like the answer to that problem. They are documented as soft limits, and the docstring is blunt about why: a single oversized response can overshoot, more so when nested calls run in parallel (types/agent.py:179-186).

The softness is larger than that sentence suggests, because both limits are checked in the same place as the turn cap, at a cycle boundary. A call that answers in one turn never reaches a boundary after the model speaks, so the check never sees the tokens it produced.

One probe makes it concrete. An eighty-token output cap on a single-turn answer. The run produces 255 output tokens and stops with end_turn, because a plain answer ends the loop before the check comes round again. The cap was not enforced. Its only comparison happened before the model spoke, against zero tokens.

Put a hundred-token cap on a loop and it binds, late but visibly. That cap stops the tick loop with limit_output_tokens at 109 tokens, nine past the line, on the second turn. That overshoot is one part of one response, which is why the price of checking between turns is paid inside them.

Total tokens compound faster, because every model call re-reads the whole conversation. In the same tick loop the input per cycle climbs 434, 559, 657, 755. Four cycles spend 2,648 tokens against a 2,600 cap and stop with limit_total_tokens. That total is 2,405 input plus 243 output. The cap was set on total spend and the growth curve is what pushed through it.

In other words, if you need a hard ceiling on one response, the limits dict you pass to agent() is the wrong tool. The model's own max_tokens (a BedrockModel parameter) caps a single call. Those limits check a conversation, and they check it one boundary late.

The deadline that waits

Wall-clock time is the boundary people assume is absolute. It is the most cooperative of the four. There is also no time key in the limits dict: a deadline is a timer thread you own that calls agent.cancel(). Here that thread is a watchdog.

Two probes, same agent.cancel(), different phases of the loop. In the first, the watchdog cancels three seconds into a long answer. Streaming is a cancellation safe point, so the call returns in 3.0 seconds with stop_reason set to cancelled and the cancel message as the final message from the model.

In the second, the watchdog fires five seconds into a tool that sleeps twenty seconds. The tool keeps sleeping. The call returns after 21.9 seconds, still cancelled, still carrying the message. The deadline was five seconds. The observed latency was more than four times that.

The source lists the safe points: during model response streaming, before tool execution, during MCP tool execution, after tool execution and before the next model call (agent/agent.py:686-690). A running synchronous tool is not on that list. Cancellation there means the agent stops as soon as control comes back.

Nor does cancelled undo what already ran. The loop appends the tool result to the messages and only then observes the cancellation (event_loop/event_loop.py:1135, then :1166), so the second probe above ends cancelled with [success] report ready: all checks green still sitting in agent.messages. Cancellation is a statement about the loop, not about the side effect. If the tool can charge a card, stop_reason alone will not tell you what happened: the retry path needs an idempotency key and a read-back of the effect before it runs again.

So a deadline is only as good as the code under it. Record the deadline when the call starts and have the tool check time.monotonic() against it or hand it to the timeout parameter of the HTTP client, database driver or subprocess. Without that, the wall you built has a door in it. The cancel message is worth using too: it becomes the text the caller sees, which beats a generic timeout error.

The tool also has a cooperative way out. ToolContext.cancel_signal is set by the loop on cancel, and the SDK documents the contract: poll it between steps or forward it to an API that accepts one, and "a tool that ignores it runs to completion" (types/tools.py:191-195). The decorator has to opt in — write @tool(context=True), or the SDK rejects the signature with a ValueError (tools/decorator.py:183):

@tool(context=True)
def slow_report(tool_context: ToolContext) -> str:
    if tool_context.cancel_signal.wait(20):
        return "aborted before doing any work: cancel signal received"
    return "report ready: all checks green"

Same watchdog, same five-second deadline. The unaware tool holds the call for 21.7 seconds and still reports success; the aware one returns in 5.0 and the caller reads the abort note instead:

=== A: cancel-unaware tool (time.sleep) ===
ELAPSED_S   : 21.7
STOP_REASON : cancelled
  [success] report ready: all checks green
=== B: cancel-aware tool (cancel_signal.wait) ===
ELAPSED_S   : 5.0
STOP_REASON : cancelled
  [success] aborted before doing any work: cancel signal received

The quota the loop cannot enforce

Taken together, none of the three boundaries above can enforce a count on how often one tool runs inside a single turn. The turn cap counts model calls, the token caps count tokens, and the deadline only stops tools that have not started yet: one already running keeps the loop waiting until it returns. A tool with side effects needs its own wall.

The SDK exposes one through hooks. A BeforeToolCallEvent callback fires before each tool call and can set cancel_tool, which turns that call into a tool result with error status (hooks/events.py:221-247). Count the calls yourself and stop at your number.

from collections import defaultdict
from strands.hooks import BeforeToolCallEvent

attempted = defaultdict(int)  # one dict per request: reset it if you reuse the agent

def quota(event: BeforeToolCallEvent) -> None:
    name = event.tool_use["name"]
    attempted[name] += 1
    if attempted[name] > 2:
        event.cancel_tool = (
            f"quota exceeded: {name} allows 2 calls per request, "
            f"this was attempt {attempted[name]}"
        )

agent.add_hook(quota, BeforeToolCallEvent)

With a quota of two on the tick tool, the run makes three attempts: two execute, one is blocked, and the model reads the error and changes its answer. It reports two ticks recorded and explains the limit back to the caller. The loop then ends with end_turn.

That last detail matters for how you place this wall, because the quota stops the tool, not the invocation. The model keeps its turn, sees a failure and decides what to do. For a payment API or a shell command, that is the right shape: the model gets a clear refusal it can reason about, and the loop is not killed mid-thought.

One more boundary on this wall: the counter lives in one process. attempted is a closure dict on one agent instance, so two workers pulling the same job each hold their own count and two plus two attempts all pass. A per-process quota bounds one run; it does not bound a job. If the job needs a ceiling, the count has to live in a shared store: reserve with a conditional write before the attempt, carry the attempt id as the idempotency key into the side effect, and release the reservation only after the effect is confirmed. Given what cancelled does not undo, "the run was cancelled" is not grounds for releasing it.

Four walls, one priority

Because all four limits checks run at the same cycle boundary, reaching several at once is the normal case rather than a rare one. The stop reason then follows a fixed order: turns first, then total_tokens, then output_tokens, then structured_output_attempts (agent/invocation_context.py:229-262). Cross both turns and total_tokens in one cycle and the run reports limit_turns only. Each check maps to its own limit_* reason in the stop set (types/event_loop.py:42-74).

Misconfiguration is caught before any of this runs. The limits payload is validated when the request starts: unknown keys raise TypeError with the accepted key list in the message, while zero, boolean and non-integer values raise TypeError with the offending key and value (agent/invocation_context.py:160-181). Three payloads, three errors, zero model calls:

TypeError: limits keys ['tuns'] are not recognized limits, expected one of 'turns', 'output_tokens', 'total_tokens', 'structured_output_attempts'
TypeError: limits['turns'] must be a positive int, got 0
TypeError: limits['turns'] must be a positive int, got True

A typo in a limit name is the worst kind of boundary bug when nothing validates it, because the call succeeds and the wall is simply absent. Validation closes that hole.

One operational footnote. Budgets restart on every top-level call: "Usage starts from zero on each top-level call; pass result.invocation_context to a later call to continue on the same limits and usage" (types/agent.py:165-166). Resume after a cancel or a limit stop with the same context and the budget continues; call again without it and the run starts fresh. And a budget inherited while already spent stops the call before the model runs: passing a context whose turn budget was exhausted returned limit_turns with zero new tokens in usage (src/limits_reset.py).

What surprised me

The official production guide links its tool quota advice to a "Limit Tool Counts" section of the hooks page. That anchor does not exist. The page has tool interception but no quota example, so the mechanism is real while the documented pattern is missing. The hook in the section above is built from BeforeToolCallEvent.cancel_tool and verified by a run.

My first pass at these experiments ran on deepseek.v3.2, the model from Articles 01 and 02. It failed in a way worth reporting. Some runs printed the tool call as literal text, complete with the model's internal markup, instead of requesting a tool. The loop read a plain answer and ended. A turn cap cannot demonstrate anything on a run that stops at turn three for the wrong reason, so every experiment here is pinned to amazon.nova-pro-v1:0. Boundary experiments are only as stable as the model's tool calling.

The last surprise is granularity. Five identical runs, one call per turn, 0.9 calls per turn every time. Change one sentence of the prompt and the same eight calls fit into three turns. The number your turn cap protects against is partly a property of your prompt.

When not to use this

A single-turn question and answer needs no turn cap. Adding one changes nothing that a max_tokens setting would not fix better. The limits budgets pay off in loops: retrieval agents, repair agents, anything that can call tools repeatedly.

In contrast, a quota hook is for tools with cost or side effects. Putting one on a pure function adds a failure mode for no gain. A deadline is only worth setting where you can also propagate it downstream, because inside a blocking tool it is a suggestion.

And do not treat these four as a checklist to fill in. Treat them as four different instruments, each with a known blind spot. Pick the one that covers the failure you actually fear.

Try it yourself

git clone https://github.com/pinxiong/strands-in-action.git
cd strands-in-action/usecases/03-execution-limits
uv venv --python 3.11
VIRTUAL_ENV=.venv uv pip install strands-agents==1.63.0

Run with AWS credentials that can call amazon.nova-pro-v1:0 in us-east-1:

python src/turn_limit.py          # turn cap, three probes
python src/turn_batching.py       # same calls, two prompts
python src/turn_granularity.py    # five identical runs
python src/token_budget.py        # single turn against a loop
python src/total_token_budget.py  # compounding spend
python src/cancel_watchdog.py     # cancellation latencies
python src/cancel_signal.py       # cooperative cancel, no rollback
python src/limits_reset.py        # budgets restart per top-level call
python src/tool_quota.py          # quota hook
python src/limits_validation.py   # no model calls needed

Takeaways

A turn is one model call plus its tools, so a turn cap bounds model calls, not tool calls.
The same eight tool calls took 9 turns under one prompt and 3 under another.
Token caps are checked at turn boundaries: a single-turn answer never reaches the check.
A token cap can overshoot by part of one response, and total_tokens compounds every turn.
Cancellation is cooperative. A five-second deadline took 21.9 seconds inside a sync tool; a tool that waits on tool_context.cancel_signal exits at the deadline instead.
A cancelled run can still hold the tool's success result. Cancellation never rolls the side effect back.
A quota hook stops one tool call, returns an error the model can read and leaves the loop alive. Its counter bounds one process, not a job shared across workers.
Limit budgets restart per top-level call; pass result.invocation_context to continue them.
Unknown limit keys fail loudly at request start, which is the only failure mode worth having.

Gen AI Concepts for Bedrock

عبدالله عياد | Abdullah Ayad — Thu, 05 Nov 2026 20:46:47 +0000

1. Tokenization

Definition:

Tokenization is the process of breaking down text into smaller units, called tokens, which can be words, subwords, or even individual characters. These tokens serve as the building blocks that AI models use to process and generate language.

How It Works:

A sentence like "Artificial Intelligence is powerful."
The model tokenizes it into smaller units:

Word-level tokenization:
["Artificial", "Intelligence", "is", "powerful", "."]
Subword tokenization (e.g., Byte-Pair Encoding):
["Artifi", "cial", "Intelli", "gence", "is", "power", "ful", "."]
Character-level tokenization:
["A", "r", "t", "i", "f", "i", "c", "i", "a", "l", ...]

Why It Matters:

Helps models understand and generate text efficiently.
Handles out-of-vocabulary words by breaking them into subwords.
Reduces complexity compared to processing raw text.

2. Context Window

Definition:

The context window refers to the maximum number of tokens an AI model can process at once when generating responses. It determines how much prior text the model "remembers" during inference.

How It Works:

If a model has a context window of 4,096 tokens, it can only consider the last 4,096 tokens when generating a response.
Older parts of a conversation or document may be forgotten unless explicitly included in the prompt.

Why It Matters:

Short context windows (e.g., 512 tokens) limit long-form coherence.
Larger context windows (e.g., 32,000+ tokens) improve memory and allow models to process entire documents in a single prompt.
Optimizing context windows is crucial for applications like chatbots, legal analysis, and summarization.

3. Embeddings

Definition:

Embeddings are numerical representations of words, sentences, or documents in a high-dimensional space, allowing AI models to understand their relationships based on meaning.

How It Works:

Instead of treating words as separate entities, embeddings represent them as vectors (lists of numbers) that capture their meanings.
Similar words have embeddings that are closer together in space.

Example:

"King" → [0.21, -1.32, 0.88, ...]
"Queen" → [0.19, -1.29, 0.87, ...]
"Dog" → [-0.75, 0.42, 0.33, ...]

Why It Matters:

Helps AI understand synonyms, context, and relationships between words.
Used in search engines, recommendation systems, and NLP tasks to improve accuracy.
Enables semantic search, where users can find relevant results even if they don’t use exact keywords.

GitHub
LinkedIn
Facebook
Medium

DEV Community: AWS Community Builders

[AWS] Should you return results or answers? The difference between Web Search and Web Answer [AgentCore]

【AWS】結果を返すか、答えを返すか。Web SearchとWeb Answerの使い分け【AgentCore】 #AIエージェント - Qiita

Introduction

Key differences between Web Search and Web Answer

Overview

Specification comparison by aspect

Detailed explanation of each aspect

Aspect 1: What each tool returns

Aspects 2 to 4: Differences from the caller's point of view

Aspects 5 and 6: Domain and date filters

Aspect 8: Regions and data handling

Aspect 9: Price

Aspect 10: Separation of IAM responsibilities

What I confirmed on a real environment

Verification 1: Sending the same question to both and comparing the response formats

Verification 2: Separation of IAM responsibilities

Verification 3: Combination rules of the domain filters

Verification 4: Date filter

Criteria for choosing

How to verify it yourself

1. Create the service role

2. Create the Gateway and targets with Python

Settings screens in the console

3. Call MCP with SigV4 signing

4. Try the domain filters (optional)

5. Clean up

Conclusion

References

From ETL Jobs to Agentic Orchestrators

Why the boring part is still the hard part

A better split: the agent asks, the workflow does

1. The architecture, in one picture

Why this beats giving the agent direct access

2. The scenario: a corrected sales extract at an odd hour

Why Standard workflows, not Express

3. The implementation, piece by piece

1. The state machine: validate before you spend money

2. The Gateway target: pin what the agent should never choose

3. Two tiers of IAM, because Step Functions has two ARN formats

4. Calling the Gateway over MCP from Python

4. Guardrails, and the mistakes I would avoid

Four mistakes I have seen (and made)

5. Where GPT-6.1 Terra fits

When I would not use this pattern

6. Before you go to production

7. Closing thoughts

Resources & Demo Repository

Preparing for Re:Invent 2026

Kubernetes CPU Requests in Amazon EKS Auto Mode

What we will observe

How scheduling leads to provisioning

Experiment environment

Prerequisites

Step 1: Schedule a Pod with a 500m CPU request

Step 2: Increase the CPU request to four CPUs

Step 3: Inspect the NodeClaim created by EKS Auto Mode

Step 4: Verify the new node and the Pod placement

Correlate the Pod, Node, NodeClaim, and EC2 instance

Using a custom NodePool for a workload class

What the experiment teaches us

Practical considerations

Clean up

Conclusion

NexOps: Build CloudOps Agent Using Strands+Bedrock+Terraform

Overview of NexOps

Business Problems

Architecture

What This Application Does

Functional Workflow

AWS Components

Security Groups

Implementation Steps

Complete Repository Folder Structure

Registered Agent Tools

Benefits

Conclusion

How to Master AWS for AI?

What is in here

The map in one table

Step 1: Schedule a Pod with a `500m` CPU request