DEV Community: Chigozirim Eke

Can we actually build the thing now?

Chigozirim Eke — Sun, 18 Oct 2026 04:58:27 +0000

Part 1 ended with "Now we can actually build it." So in Part 2 we build it on both frameworks, and give each one the same request to compare the courses they make.

The first iteration built by yours truly Claude Opus 5.5 had 235 tests, and every single one of them passed. But when I started poking (ok, so this has been my huge limitation with Claude. It doesn't surface gaps until I start poking), I realized not one of those tests had ever talked to a real model 🙃

When I pointed it at a real model, it failed on the very first call. And when it finally did produce a Terraform course, the quiz answers sat right under the questions and every exercise came with the finished files.

So I deleted the entire folder and started from scratch. The rebuild came out much better, and it's the one we'll walk through in this post.

Rant Aside. Because accuracy is super matters to me, whenever I'm learning something new, I search for "the most up to date resource", and more than once I've bought a course only to realise a few lessons in that it's already out of date. Case in point: I started Part 1 a couple of weeks ago, and Strands has already shipped a whole agent harness.

We'll start with what using it looks like, then build it up side by side the way we did in Part 1.

Everything here ran between 27 and 29 September 2026, on the same versions as Part 1 (strands-agents 1.55.1 and google-adk 2.9.0), and the code is in the companion repo. Both frameworks are actively developed, and strands-agents 1.61.1 and google-adk 2.12.0 were out by the time this went up, so some APIs may have changed since.

What should a learning system actually do?

When I started over, I wrote down what I actually want from this thing:

I say what I want to learn, and it asks a couple of questions to get the course right.
It covers what people actually do with it at work. For example, pinning a shared module's version.
It's as current as the docs on the day it's built.
I learn by doing, and there are tests that verify my work.
I have to try before I see the answer.
It doesn't cost a fortune to build, and it tells me what it spent.

What does using it look like?

You'll need uv. Then install it once, straight from the repo:

uv tool install "git+https://github.com/coolchigi/strands-vs-adk#subdirectory=part2-learning-system"

and start a course from anywhere:

learn "I want to learn TypeScript so I can build a small web app"

The first time, it asks for a Gemini API key and saves it, so you never set it up again. That key is for the ADK build, which runs on Gemini and is the default. To build with Strands, which runs Claude on Amazon Bedrock, add --engine strands. You'll need AWS credentials and access to Claude in Bedrock, and the Strands Python quickstart walks through setting both up:

learn --engine strands "I want to learn TypeScript so I can build a small web app"

If you already have AWS credentials and no Gemini key, it picks Strands on its own and doesn't ask for a key.

Every course has a spending cap, $5 on Gemini and $10 on Claude by default, and we'll get to how it works in the bill section.

Then it asks what it needs to know, and you answer right there:

A few questions first, so I build the right course:
  What is your background with programming, particularly JavaScript?
> I know JavaScript well but have never used TypeScript
  What kind of web app are you looking to build, and do you have a specific framework in mind (e.g., React, Next.js, Node.js)?
> a small web app with a backend API

Got it. Building your TypeScript course in ~/courses/typescript
This usually takes 30 to 90 minutes. You can stop it any time, and running the same command
again picks up where it left off.

  Looking for the official TypeScript documentation
  ✓ Found it: TypeScript, version 7.0
  ✓ Worked out 15 things you'll need to be able to do
  ✓ Checked 126 facts, word for word, against 17 official pages
  ✓ Planned 5 units, 12 lessons
  Writing lesson 1 of 12: Compiler CLI and Strict Type Validation
      ✓ passed its checks
  ...
Your course is ready: ~/courses/typescript
12 lessons, and it cost about $2.94 to build.

That's a trimmed output of the real run. Once the build finishes, you get a folder with a README.md for each lesson. Each one has the explanation, where every fact links to the page it came from, then a worked example, then an exercise with files for you to change.

You work through it with learn:

cd ~/courses/typescript
learn next      # where you are, and what's next
learn check     # run the exercise's tests
learn hint      # one hint at a time
learn solution  # the answer, when you want it
learn quiz      # answer first, then see the answer
learn review    # questions you missed come back after 1, 3, 7 and 21 days

Lesson 1's exercise gives you a formatter.ts with two TODO comments, which mark the code you have to write. One of them is to give a parameter a type. Here's what learn check says if you fix the code but skip the type:

$ learn check
Checking units/u1-strict-compiler-configuration-and-compilation/u1-l1-compiler-cli-and-strict-type-validation/exercise ...

formatter.ts(1,28): error TS7006: Parameter 'input' implicitly has an 'any' type.

Not yet. Read the messages above, change your files, and check again.
Stuck? learn hint

Add the type and it passes, then points you to learn quiz.

That error comes from the TypeScript compiler itself, because learn check runs each subject's real tools. So what you need installed depends on what you're learning. A Terraform course needs Terraform, a Python course needs Python 3.9 or newer, and a JavaScript or TypeScript course needs Node. Anything else has no tests, so you compare your work with the solution.

How is it built?

Behind those learn commands is the system we sketched in Part 1, with one new agent at the front:

learner ─▶ intake ─▶ researcher ─▶ curriculum builder ─▶ teacher ─▶ your course
                         ▲                  ▲                ▲
                         └──── the feedback agent reviews each stage, sends it back with reasons

The intake asks what it needs to know, like where you're starting from and what you want to build.
The researcher reads the official docs and makes a list of what the course has to teach (i.e. its objectives). It only keeps facts from the live pages.
The curriculum builder turns that list into units and lessons, and picks an exercise or a quiz to check each one.
The teacher writes each lesson, including the tests that check your exercise.
The feedback agent checks each stage's work and sends it back if something's wrong. Code checks what it can, like running every exercise, and a model reviews the rest.

Most of the system, like fetching pages and running tests, is plain Python that both sides share. The frameworks handle the parts this post compares: reaching the docs, getting typed answers back, running the feedback loop and keeping the bill down.

Both frameworks plug in through one method:

class Brain(Protocol):
    def ask(self, name: str, instruction: str, prompt: str, schema: type[T], research: bool = False) -> T:
        """One agent call, returning `schema`. With research=True the agent can search and read."""

Each framework implements ask its own way and runs the stages in its own graph. The Strands side uses Claude Sonnet 4.6 on Amazon Bedrock, which is Strands' default, and the ADK side uses Gemini 3.8 Flash. So when the two sides behave differently, it can be the framework or the model, and we'll say which one it was each time.

How does the researcher reach the docs?

The researcher needs tools to find the right pages and read them. Some of those tools come from MCP servers. MCP (Model Context Protocol) is an open source standard for connecting AI apps to outside tools and data, so the same server works with both Strands and ADK.

Here's how they both do it:

Strands	ADK
No search in the package	`google_search`, Google Search built into Gemini only
MCP servers go straight into `tools`	MCP servers come in through `McpToolset`
`web_fetch` reads a page live	`load_web_page`, which I couldn't use

Both sides also get the same two extra tools. One is AWS's documentation server, which searches and reads AWS's docs with no key. The other is list_docs_pages, a small tool we wrote that lists every page a docs site has, from the site's own sitemap.

Strands

Strands ships tools for fetching pages and none for searching, so I used an MCP server. Here we create an MCPClient and pass it straight into tools, and Strands opens and closes the connection for us. This is what connecting to the AWS docs server looks like:

from strands import Agent
from strands.tools.mcp import MCPClient

aws_docs = MCPClient(url="https://knowledge-mcp.global.api.aws")
researcher = Agent(system_prompt="Answer from the AWS documentation, in one sentence.",
                   tools=[aws_docs], callback_handler=None)

print(researcher("How big can a single S3 object be?"))

A single Amazon S3 object can be a maximum of **5 terabytes (TB)** in size, though the largest
object that can be uploaded in a single PUT request is **5 GB** ...

Keep that 5 TB in mind, because we'll ask the ADK side the same question.

For general web search I used Exa's hosted MCP server because it didn't need an API key. Its free tier ran out partway through a single course build, so on Strands, web search stays off unless you add an Exa or Tavily key. For now, you don't need one, because list_docs_pages finds the pages on the docs site itself.

To read a page, web_fetch has two modes. The default, agentic, hands the page to a second agent and returns that agent's summary, and markdown returns the page itself. We use markdown mode because, for every fact, the researcher copies the sentence on the page that backs it up, and our code checks that sentence is really there.

from strands.vended_tools.web_fetch import make_web_fetch

fetch = make_web_fetch(mode="markdown", max_content_chars=50_000)

It needs the web-fetch extra, so pip install "strands-agents[web-fetch]".

ADK

ADK ships search as google_search, and it only works on Gemini. On any other model, like Claude, the agent still builds, but the first call fails with ValueError: Google search tool is not supported for model ....

MCP servers come in through McpToolset. This is the same AWS docs server, connected next to google_search:

import asyncio

from google.adk.agents import Agent
from google.adk.runners import InMemoryRunner
from google.adk.tools import google_search
from google.adk.tools.mcp_tool import McpToolset, StreamableHTTPConnectionParams
from google.genai import types

aws_docs = McpToolset(connection_params=StreamableHTTPConnectionParams(url="https://knowledge-mcp.global.api.aws"))
researcher = Agent(
    name="researcher",
    model="gemini-3.8-flash",
    instruction="Answer from the AWS documentation, in one sentence.",
    tools=[google_search, aws_docs],
    generate_content_config=types.GenerateContentConfig(
        tool_config=types.ToolConfig(include_server_side_tool_invocations=True)
    ),
)


async def main() -> None:
    runner = InMemoryRunner(agent=researcher, app_name="docs")
    session = await runner.session_service.create_session(app_name="docs", user_id="me")
    message = types.Content(role="user", parts=[types.Part(text="How big can a single S3 object be?")])
    async for event in runner.run_async(user_id="me", session_id=session.id, new_message=message):
        if event.is_final_response() and event.content and event.content.parts:
            print("".join(part.text or "" for part in event.content.parts))
    await aws_docs.close()


asyncio.run(main())

According to AWS documentation, a single Amazon S3 object can be up to 50 TB in size ...

The Strands agent said 5 TB and the ADK agent said 50 TB, and AWS's own page says "each object can be up to 50 TB in size". Both agents had the same AWS docs server. Claude searched once, got a result that mentioned both 5 TB and 50 TB, and answered 5 TB. Gemini kept going, with more searches, and in both runs I traced it read the S3 FAQ page itself before answering. That's why nothing a model says goes into a course unless it comes with a quote from the live page.

If you use McpToolset yourself, run all your agent calls on one event loop. The toolset keeps its MCP session on the loop it first ran on. My first version called asyncio.run for every agent call, and asyncio.run closes its loop when it's done, so every call after the first found its session on a closed loop.

ADK has its own page loader, load_web_page, but it kept losing the pages we needed. Terraform's docs redirect a lot, and it won't follow a redirect, so it came back with Failed to fetch url: ... instead of the page. When a page did load, it threw away every line of 3 words or fewer, which took out 84 of the 181 lines in one tutorial's code examples, $ terraform init and required_providers { included. So on the ADK side, the agents find the pages, and our own code fetches them in full.

The ADK example also had a generate_content_config line we haven't explained yet. It's there because of the very first real call.

What happened on the first real call?

The researcher's first job is to find the official docs and the current version, and hand them back as a typed plan through output_schema, which is how ADK asks a model for a typed answer. The first real call came back with this:

400 INVALID_ARGUMENT: Please enable tool_config.include_server_side_tool_invocations
to use Built-in tools with Function calling.

Hmm, function calling. That's weird because I hadn't given it a function.

ADK had. When an agent has both output_schema and tools, ADK quietly adds a function tool called set_model_response, and the model hands back its typed answer by calling it. So my agent was sending Google Search, a built in tool, together with a function tool, and the Gemini API wants you to say that's on purpose.

I tried GoogleSearchTool(bypass_multi_tools_limit=True) first and got the same 400, because ADK adds set_model_response after that setting is checked.

The fix was what the error asked for, turning on include_server_side_tool_invocations:

import asyncio

from google.adk.agents import Agent
from google.adk.runners import InMemoryRunner
from google.adk.tools import google_search
from google.genai import types
from pydantic import BaseModel


class Plan(BaseModel):
    official_domains: list[str]
    current_version: str | None = None


planner = Agent(
    name="plan",
    model="gemini-3.8-flash",
    instruction="Find the official docs domain and the current release of the subject. Use google_search.",
    tools=[google_search],
    output_schema=Plan,
    output_key="plan",
    generate_content_config=types.GenerateContentConfig(
        tool_config=types.ToolConfig(include_server_side_tool_invocations=True)
    ),
)


async def main() -> None:
    runner = InMemoryRunner(agent=planner, app_name="plan")
    session = await runner.session_service.create_session(app_name="plan", user_id="me")
    message = types.Content(role="user", parts=[types.Part(text="Terraform")])
    async for _ in runner.run_async(user_id="me", session_id=session.id, new_message=message):
        pass
    done = await runner.session_service.get_session(app_name="plan", user_id="me", session_id=session.id)
    print(done.state["plan"])


if __name__ == "__main__":
    asyncio.run(main())

{'official_domains': ['developer.hashicorp.com', 'terraform.io'], 'current_version': '1.16.4'}

Without the generate_content_config line, the 400 comes back. None of my 235 tests could have caught this, because they ran against a fake model, and a fake model accepts any request.

Which one found the current version?

The plan step asks for the subject's current version. On 27 September Terraform's was 1.16.4, per HashiCorp's own release API. Here's every plan step I ran while the Strands side searched through Exa:

Answer	Strands (Claude + Exa)	ADK (Gemini + Google Search)
1.16.4	1	8
1.16.3	5	0
1.16.2	3	0
"1.9.x (1.9 series, latest stable as of mid-2024)"	1	0

Twice Claude quoted the Terraform install page as saying "Version: 1.16.3" when the live page said 1.16.4. Exa's copy of the page was a few days behind, and Claude was quoting the copy. ADK's search is Google's index, and Strands' search is whatever server you plug in, so checking how fresh it is falls to you.

So now the version gets checked like every other fact. The plan has to name an official page that states it, with the exact words, and our code fetches that page and checks the words are there. Once Strands had web_fetch to read the live page, it got 1.16.4 too.

There's one bug left here. Every course's README says which version it was written against. If the feedback agent sends research back for more facts, a version our code couldn't find on the live page, like Claude's 1.16.3, can still end up in that README.

How do we get typed answers back?

Every agent in the system hands back a typed object, like a plan, a list of facts or a lesson. Both frameworks take a Pydantic model and give you something shaped like it, and they differ in where you pick it up.

Here's how they both do it:

Strands	ADK
`structured_output_model=` on the call or the agent	`output_schema=` on the agent
The object comes back on `result.structured_output`	A `dict` lands in session state under `output_key`
Works next to any tools	Adds a function tool, so built in tools need `include_server_side_tool_invocations`

Strands

We pass the model on the call, and get the object back on the result:

from pydantic import BaseModel
from strands import Agent


class Question(BaseModel):
    stem: str
    options: list[str]
    answer: int  # index into options
    explanation: str


agent = Agent(system_prompt="You write quiz questions about Terraform.", callback_handler=None)
result = agent("One question on what terraform init does.", structured_output_model=Question)

question = result.structured_output
print(type(question).__name__)
print(question.stem)
print(question.options[question.answer])

Question
What is the primary purpose of the `terraform init` command?
It initializes a working directory by downloading providers, modules, and setting up the backend

result.structured_output is a real Question. Strands gives the model a tool shaped like our schema and makes it call it, and if the model never does, you get StructuredOutputException. So the Strands side catches that and asks again, with the reason.

ADK

We put the schema on the agent, and the answer lands in session state:

import asyncio

from google.adk.agents import Agent
from google.adk.runners import InMemoryRunner
from google.genai import types
from pydantic import BaseModel


class Question(BaseModel):
    stem: str
    options: list[str]
    answer: int  # index into options
    explanation: str


agent = Agent(
    name="quiz",
    model="gemini-3.8-flash",
    instruction="You write quiz questions about Terraform.",
    output_schema=Question,
    output_key="question",
)


async def main() -> None:
    runner = InMemoryRunner(agent=agent, app_name="quiz")
    session = await runner.session_service.create_session(app_name="quiz", user_id="me")
    message = types.Content(role="user", parts=[types.Part(text="One question on what terraform init does.")])
    async for _ in runner.run_async(user_id="me", session_id=session.id, new_message=message):
        pass
    done = await runner.session_service.get_session(app_name="quiz", user_id="me", session_id=session.id)
    raw = done.state["question"]
    print(type(raw).__name__)
    question = Question.model_validate(raw)
    print(question.stem)
    print(question.options[question.answer])


asyncio.run(main())

dict
Which of the following actions is performed by running the 'terraform init' command?
Initializes the backend configuration, downloads provider plugins, and installs referenced modules

It comes back as a plain dict, so turning it into a Question is our job. And if the agent ends its turn without writing an answer, state stays empty. That's what stopped the first TypeScript build, with nothing was written to session state under 'answer', so the ADK side also asks again with the reason.

Where does the feedback loop live?

As we saw in Part 1, both frameworks can loop inside a graph. Where they differ is what a node can be, and what stops the loop.

Here's how they both do it:

Strands	ADK
`GraphBuilder`, conditional edges	`Workflow`, routed edges
`set_max_node_executions()` caps the loop	Your router stops the loop
A node has to be an agent or a `MultiAgentBase`	A node can be any function
Hitting the cap ends the graph as `FAILED`	Nothing to hit

The first difference matters here, because our stages aren't agents. The research stage, for example, is Python code that calls several agents and checks every quote in between.

Strands

In Strands, a graph node has to be an agent or a MultiAgentBase, and you only find that out when the graph runs. add_node accepts a plain function, so does build(), and then the graph reaches that node and stops with ValueError: Node 'stage' of type '<class 'function'>' is not supported.

So each stage gets wrapped in a small MultiAgentBase subclass. Here's the loop with stand-in stages that just print, so it runs without a model or any keys:

import asyncio

from strands.multiagent import GraphBuilder
from strands.multiagent.base import MultiAgentBase, MultiAgentResult, Status

board = {"reviews": 0, "verdict": None}


class Stage(MultiAgentBase):
    def __init__(self, name, work):
        super().__init__()
        self.name = name
        self.work = work

    async def invoke_async(self, task, invocation_state=None, **kwargs):
        # the graph runs nodes on its event loop, and real stages block, so off it they go
        await asyncio.to_thread(self.work)
        # results stays empty: Strands walks into it to build the next node's input,
        # so it has to be real results or nothing
        return MultiAgentResult(status=Status.COMPLETED, results={})


def teach():
    print("teaching")


def review():
    board["reviews"] += 1
    board["verdict"] = "revise" if board["reviews"] < 3 else "approved"
    print(f"review {board['reviews']}: {board['verdict']}")


builder = GraphBuilder()
builder.add_node(Stage("teaching", teach), "teaching")
builder.add_node(Stage("feedback", review), "feedback")
builder.add_edge("teaching", "feedback")
builder.add_edge("feedback", "teaching", condition=lambda state: board["verdict"] == "revise")
builder.set_entry_point("teaching")
builder.set_max_node_executions(10)
builder.reset_on_revisit(True)
graph = builder.build()

if __name__ == "__main__":
    result = graph("build the course")
    print(result.status)

teaching
review 1: revise
teaching
review 2: revise
teaching
review 3: approved
Status.COMPLETED

In Stage, results={} has to stay empty. If you put the stage's output there, Strands tries to read it as the next node's input and crashes with AttributeError: 'str' object has no attribute 'results'. So the stages share their work through board, a plain dict, and the graph only decides what runs next.

If the reviewer never approves, the same file stops after 10 node executions and ends Status.FAILED. It doesn't raise, so check result.status.

Without set_max_node_executions, build() only logs a warning, "Graph without execution limits may run indefinitely if cycles exist".

ADK

In ADK, a node can be a plain function, no subclass needed:

import asyncio

from google.adk import Event, Workflow
from google.adk.runners import InMemoryRunner
from google.genai import types

board = {"reviews": 0, "verdict": None}
MAX_REVIEWS = 4


async def teaching() -> None:
    # nodes run on the workflow's event loop, and real stages block, so off it they go
    await asyncio.to_thread(print, "teaching")


def feedback() -> None:
    board["reviews"] += 1
    board["verdict"] = "revise" if board["reviews"] < 3 else "approved"
    print(f"review {board['reviews']}: {board['verdict']}")


def route() -> Event:
    if board["verdict"] == "approved" or board["reviews"] >= MAX_REVIEWS:
        return Event(route="stop")
    return Event(route="revise")


def stop() -> None:
    return None


root_agent = Workflow(name="revision_loop", edges=[
    ("START", teaching, feedback, route),
    (route, {"revise": teaching, "stop": stop}),
])


async def main() -> None:
    runner = InMemoryRunner(agent=root_agent, app_name="loop")
    session = await runner.session_service.create_session(app_name="loop", user_id="me")
    message = types.Content(role="user", parts=[types.Part(text="build the course")])
    async for _ in runner.run_async(user_id="me", session_id=session.id, new_message=message):
        pass


if __name__ == "__main__":
    asyncio.run(main())

It prints the same thing, without the status line. In ADK, route is the only thing that ends this loop. Without the board["reviews"] >= MAX_REVIEWS check, a reviewer that never approves keeps it looping, because nothing in ADK limits how many times a node runs.

What the real system does with it

The learning system's graphs are those two shapes, with each review as a node of its own:

After each stage, a review decides what happens next. The run moves on, the stage runs again with the reviewer's reasons, or the run goes back to research if that's where the problem started. A stage gets 3 tries, and after that the run moves on and the course lists what's still wrong.

Going back to research used to start it again from nothing. On the Strands side, a reviewer found a few gaps, the rerun threw away 306 facts that had already been checked, and that one trip cost $11.81. Now research that comes back keeps everything it already checked and only looks for what's missing.

How do we keep the bill down?

A course takes hundreds of model calls, from picking pages to writing and reviewing every lesson, and more when a review sends something back. The Strands side runs on Claude Sonnet 4.6, which costs 4 times what Gemini 3.8 Flash does per token.

Part 1 covered the limits each framework has built in. Strands takes limits= on a call, and ADK takes RunConfig(max_llm_calls=...) on a run. But those only count within one call or one run, and a course is hundreds of them, so I wanted one number for the whole course, in dollars.

I added a spending cap to every course, and a meter that counts what every agent has spent. Before each model call, the cap checks the meter and refuses the call if we've exceeded the cap. Both frameworks give us a hook in exactly the right place, and they attach differently.

Here's how they both do it:

Strands	ADK
`add_hook` on an agent you already have	`before_model_callback` on the agent
Usage is on `agent.event_loop_metrics`	Usage is on each `llm_response`
Raise, and it comes out as is, or wrapped once a tool has run	Raise, and it comes back as an error event first

Strands

Strands already counts each agent's tokens in agent.event_loop_metrics, so the hook just reads them:

from strands import Agent
from strands.hooks import BeforeModelCallEvent

PRICE = (3.00, 15.00)  # USD per million tokens, in and out, for Claude Sonnet 4.6
CAP = 0.001


class BudgetExceeded(RuntimeError):
    pass


def spent(agent: Agent) -> float:
    usage = agent.event_loop_metrics.accumulated_usage
    return (usage["inputTokens"] * PRICE[0] + usage["outputTokens"] * PRICE[1]) / 1_000_000


def enforce_cap(event: BeforeModelCallEvent) -> None:
    if spent(event.agent) >= CAP:
        raise BudgetExceeded(f"spent ${spent(event.agent):.4f}, the cap is ${CAP}")


agent = Agent(system_prompt="Answer in one short paragraph.", callback_handler=None)
agent.add_hook(enforce_cap)

print(agent("What does terraform init do?"))
print(f"spent ${spent(agent):.4f}")
try:
    print(agent("And terraform plan?"))
except BudgetExceeded as e:
    print("stopped:", e)

`terraform init` initializes a Terraform working directory by downloading ...
spent $0.0018
stopped: spent $0.0018, the cap is $0.001

BeforeModelCallEvent also has a cancel field, which looks like the obvious tool. Cancelling hands the loop a fake assistant message, "model call denied by hook", so a plain call returns that as if the model had said it, and a structured call fails blaming a model that never got the call. Raising says what actually happened.

There's one more catch, and it only shows up in a real run. On the first model call of an invocation, your exception comes out exactly as you raised it, which is all the example above ever does. On any call after a tool has run, Strands wraps it in its own EventLoopException. My research agents use tools, so the first time the cap fired for real, except BudgetExceeded didn't match and the run died with a traceback. The fix is to look through the exception's causes.

ADK

Here we pass ADK the callbacks when we build the agent, and count the tokens as each response comes back:

import asyncio

from google.adk.agents import Agent
from google.adk.runners import InMemoryRunner
from google.genai import types

PRICE = (0.75, 3.75)  # USD per million tokens, in and out, for Gemini 3.8 Flash
CAP = 0.0001
tokens = {"in": 0, "out": 0}


class BudgetExceeded(RuntimeError):
    pass


def spent() -> float:
    return (tokens["in"] * PRICE[0] + tokens["out"] * PRICE[1]) / 1_000_000


def enforce_cap(callback_context, llm_request):
    if spent() >= CAP:
        raise BudgetExceeded(f"spent ${spent():.6f}, the cap is ${CAP}")


def count(callback_context, llm_response):
    usage = llm_response.usage_metadata
    if usage:
        tokens["in"] += usage.prompt_token_count or 0
        # thinking tokens are billed as output and reported separately
        tokens["out"] += (usage.candidates_token_count or 0) + (usage.thoughts_token_count or 0)


agent = Agent(
    name="tutor",
    model="gemini-3.8-flash",
    instruction="Answer in one short paragraph.",
    before_model_callback=enforce_cap,
    after_model_callback=count,
)


async def ask(runner, session_id, text):
    message = types.Content(role="user", parts=[types.Part(text=text)])
    async for event in runner.run_async(user_id="me", session_id=session_id, new_message=message):
        if event.error_code:
            print("error event:", event.error_code)
        elif event.is_final_response():
            print(event.content.parts[0].text)


async def main():
    runner = InMemoryRunner(agent=agent, app_name="tutor")
    session = await runner.session_service.create_session(app_name="tutor", user_id="me")
    await ask(runner, session.id, "What does terraform init do?")
    print(f"spent ${spent():.6f}")
    try:
        await ask(runner, session.id, "And terraform plan?")
    except BudgetExceeded as e:
        print("stopped:", e)


asyncio.run(main())

The `terraform init` command initializes a working directory containing Terraform ...
spent $0.001424
error event: BudgetExceeded
stopped: spent $0.001424, the cap is $0.0001

Two things are different here. Gemini 3.8 Flash thinks, and thinking is billed as output but reported separately as thoughts_token_count, so a meter that only adds candidates_token_count undercounts every call. And the exception comes back twice, first as an error event with error_code set to its class name, then raised again when the stream ends. So every loop over ADK's events checks for an error event and stops straight away.

The bigger difference is that ADK can put the cap on every agent at once. In both frameworks we can add enforce_cap to an agent after it's built, with add_hook on Strands, or by setting the agent's before_model_callback on ADK. But ADK also has plugins, which we register once for the whole app. A plugin's before_model_callback runs before every model call, for every agent. Strands has nothing like that, so each Strands agent needs its own hook.

What did it cost?

Here's what each course cost, from the meters, at list prices:

Course	Built with	Cost
Terraform	ADK, Gemini 3.8 Flash	$3.35 for the lessons (research and planning ran before the meter saved anything)
Terraform	Strands, Claude Sonnet 4.6	$18.36: research $7.06, planning $0.87, 23 lessons $10.43
TypeScript	ADK, Gemini 3.8 Flash, through `learn`	$2.94, everything included

Across the 2 days of building, with the lost runs, the restarts and the bugs, Claude on my Bedrock account came to about $59, going by AWS's Cost Explorer.

Two bugs let it get that high. The Strands meter remembered each agent's usage by id(agent), and Python hands a new object the id of one that's been thrown away, so fresh agents inherited dead ones' totals and the meter undercounted. One run reported -834,706 input tokens for a single role. And the cap lived in the process, so every restart got a fresh budget. Both are fixed now. Each agent's count lives in its own hooks, and spend is saved to disk the moment it's counted, so a restarted course carries on from what it already spent.

A cap that just stops leaves you with half a course, though. So now, halfway to the cap, the course switches to a cheaper model to finish, and tells you:

  $5.00 spent of your $10.00 cap, so the rest of this course is written by a cheaper model
  (global.anthropic.claude-haiku-4-5-20251001-v1:0).

On Strands that's Claude Haiku 4.5, at $1 in and $5 out per million tokens against Sonnet 4.6's $3 and $15. On ADK it's Gemini 3.5 Flash-Lite, at $0.30 and $2.50. Before trusting either, I asked each one to find Terraform's current version through the system's own Brain, with its tools and typed answers. Both found 1.16.4 on the live install page.

Swapping the model mid-course is easy on both sides, because each call builds a fresh agent anyway. Strands takes a different BedrockModel, and ADK takes a different model name. The cap, the switch point and the cheaper model are all settings in one file.

Bedrock's US-only inference profile costs 10% more than the global one, which I only found by asking AWS's price list. And Gemini 3.8 Flash's price doubles on 1 January 2027, to $1.50 in and $7.50 out.

Newer, cheaper Claude models came out while I was writing this. Sonnet 5.5 came to Bedrock on 28 September at $2 in and $10 out, and Haiku 5.5 on 7 October, from $0.10 in and $0.50 out.

Is the course any good?

Here's what each side produced from the same request, "I'd like to deeply understand Terraform and build projects along the way."

	Strands (Claude)	ADK (Gemini)
Facts checked against a live page	546, from 39 sources	162, from 19 sources
Lessons	23 in 5 units	12 in 4 units
Exercises that pass every check	12 of 23	12 of 12
Lessons that ship with a known problem	23	1
Cost	$18.36	$3.35, plus research and planning

The TypeScript course from earlier was built through learn. It has 12 lessons, all 12 exercises pass every check, and nothing is left on its known problems list, for $2.94.

I planned to judge the courses by their review scores. Then I did the lessons myself, and found more than the reviewers had.

The exercises

The first finished ADK course had 5 exercises out of 12 that failed their own checks, and 4 of them hit the same error, Unknown condition value ... This means you have executed a run block with command = plan. That one was my fault.

I'd told the teacher to write the tests with command = plan, and at plan time things like IDs don't exist yet. The tests run against a mock provider (i.e. a fake AWS), so command = apply doesn't create anything real or need an AWS account. It also makes up values like IDs, so the tests have something to check.

After that fix, I rebuilt the ADK course and worked through lesson 1 myself. I left one of its 3 TODOs alone, filled in the other two, and learn check still said "Passed". The system's rule was that an untouched exercise (i.e. every TODO left alone) has to fail its tests before the course ships. If the tests check just one TODO, that rule passes, so the TODO I left alone was never tested.

So now the system also checks each TODO on its own. It fills in every TODO with the answer except one, and the tests have to fail. It does that for each TODO in turn, so you can't skip any of them and still pass. 3 of the 12 lessons in that course had a TODO you could skip.

Even with every TODO checked on its own, a wrong answer can still pass. In unit 3, lesson 3 of the same course, a TODO asks you to output the ID of the EC2 instance the lesson builds. If you output its ARN instead, learn check still says "Passed". The mock provider makes up both values as random 8-character strings, so the test can't tell them apart.

TypeScript had a sneakier version. Node runs TypeScript by stripping the types out without checking them, so an exercise about types could pass with no types at all. So TypeScript exercises also go through the real compiler in strict mode, which is the TS7006 you saw earlier.

The quizzes

Here's where the right answer sat:

Right answer is option	1	2	3	4
ADK (Gemini)	30	5	1	0
Strands (Claude)	14	40	15	0

Gemini put the right answer first 83% of the time, and Claude put it second 58% of the time, so a learner who always picked 1 would pass most of ADK's quizzes. The course writer shuffles the options now, and after shuffling, the ADK course's 36 right answers are spread 10, 7, 9 and 10 across the 4 positions.

The reviewers

Gemini's reviewer signed off on all 12 of ADK's final lessons. Claude's reviewer, with the same prompt, didn't sign off on any of the Strands lessons it reviewed. So I swapped them. Claude reviewed ADK's 12 lessons and found 29 problems, at least one in every lesson, and most were real. One was a worked example that says terraform fmt scans the .terraform folder (that's where providers are cached), and another was a lesson about terraform state mv whose exercise never has you run it. Gemini reviewed Strands' 23 and passed 13 of them.

So Claude's course looks worse on paper because its reviewer is harder to please, and Gemini's looks clean partly because its reviewer isn't.

What the courses missed

Remember item 2 on my list, a course that covers what people actually do at work? Where I work, we publish Terraform modules, and every team that uses one pins a version, with ?ref= on a Git source or version on a registry module, and bumps it when there's a new release. We do that all the time.

Neither course taught it as something you do. The ADK course mentions it in one lesson's worked example and quiz, and the Strands course mentions it once in passing. Both researchers built their plans from how the docs are organised, and the docs don't put "publish a module and pin it downstream" on the front page.

That's a gap in the research, so that's where the fix went. The researcher now writes a list of what someone who uses the subject at work does routinely, and names the objective that teaches each item. A plain check flags any item with no objective, and the research reviewer reads each pairing and says when an objective only teaches part of an item.

Then I ran the same Terraform research twice more. The check did its job, and in both runs the reviewer caught items that were only half taught, and the research added objectives for them. Module pinning still didn't make it in, because neither run put it on the list in the first place. Of the 6 research runs that wrote this list, 2 put module pinning on it. The check can only hold the objectives to the list, and the list is still the model's idea of what people do at work.

What I think fixes it is asking. The intake already asks where you're starting from. If it also asked how your team uses the subject, I'd have told it we pin shared modules, and the research would have had to cover that.

Both courses say their exercises run against LocalStack, an AWS emulator (i.e. it pretends to be AWS on your own machine), but neither shows you how to set LocalStack up. In the ADK course, that just means the LocalStack steps fail. The Strands course could cost you money, though, because its last 6 lessons set Terraform up to talk to real AWS. Two of those lessons tell you to run terraform apply, so if you have AWS credentials on your machine, you'd create real VPCs and EC2 instances in your account.

learn check is safe in every lesson, because its tests run against the mock provider and never touch AWS. The Strands course's README warns about this at the top now.

What I think fixes it is running AWS exercises against an emulator by default, and letting you switch to your own AWS account if you want the real thing.

So which one?

Here's the whole comparison in one place:

	Strands	ADK
Web search	Bring an MCP server	`google_search`, Gemini only
Reading a page	`web_fetch`, markdown or agentic	`load_web_page`, which I couldn't use
Typed answers	`structured_output_model=` on the call or the agent, the object on the result	`output_schema` on the agent, a `dict` in session state, plus a flag next to built in tools
The feedback loop	`GraphBuilder`, capped by `set_max_node_executions`	`Workflow`, stopped by your router
A stage that isn't an agent	Subclass `MultiAgentBase`	Any function
A spending cap	`add_hook` on each agent	A callback on each agent, or one plugin for every agent
Errors from your hooks	Wrapped in `EventLoopException` once a tool has run	An error event first, raised at the end

Both frameworks ran the same stages through the same kind of graph, with the same typed answers.

How good the courses came out was down to the models and to my checks. Claude found 3 times as many facts and wrote twice as many lessons, at 4 times Gemini's price per token, and its reviewer was much harder to please. Gemini was cheaper, and its reviewer passed its own side's lessons. Most of the problems I found doing the lessons came from a check I hadn't written yet, or one I'd written wrong.

So which one would I pick? For myself, Strands. But ADK did some things better, and if those matter more for what you're building, it's the better pick.

Strands was easier to start with. Once AWS credentials and access to the model in Bedrock are set up, an agent is one Python file and three lines of code, and as it runs it prints each tool it calls. It also had set_max_node_executions, which stops the graph once it has run a set number of nodes. ADK has nothing like it, and it was my backstop in case the feedback loop ever went around forever.

ADK is lighter to set up, because a Gemini API key is enough to run it on your machine. adk web gives you a chat in the browser with a trace of every call next to it. Any plain function can be a stage in the workflow, where Strands needs it wrapped in a class. And its google_search tool found the current Terraform version 8 times out of 8, though it only works with Gemini.

And whichever framework you pick, get a second model to review the first.

Both Terraform courses and the TypeScript course are in the repo, and each lesson lists its known problems at the top. If you work through one and find something they missed, open an issue and tell me.

Can we build an AI agent that'll kill Udemy?

Chigozirim Eke — Tue, 06 Oct 2026 05:44:59 +0000

Lol, not a chance! But I'm glad that caught your attention.

I do want to build a replacement for it for myself, though.

Don't get me wrong, I love Udemy. I upskilled on Udemy, and it saw me through quite a few certifications. I still remember preparing for my very first cert with Stéphane Maarek.

But I'm in a different phase of life now, and long form video courses just aren't doing it for me anymore. I still use them when they're short and straight to the point. Maybe that's the ADHD speaking. Raise your hands if you can relate 🙋‍♀️

So I started thinking about what I'd want instead. I want to give a system some documentation and have it turn that material into a learn by doing curriculum.

What makes it different from the million other course generators out there?

Well, duh... it's ✨ agentic ✨.

And since I've been wanting to properly understand AWS Strands Agents and Google's Agent Development Kit (ADK), why not use this as an excuse to put both through their paces?

AWS and Google have done a fantastic job building their respective frameworks. The good thing is that you get a lot of capability out of the box. The bad thing, at least for me, is that there is a lot to cover.

So grab your coffee, blanket, or emotional support animal. Not sure why I put emotional support animal, but hey, why not.

In Part 1, we'll walk through Strands and ADK side by side, starting with a basic Hello World and gradually building up from there.

For Part 2, we'll take what we've learned and actually build the learning system.

So yes, we're technically building a low budget “Udemy replacement.”

Before we write code: what are we actually comparing?

Rather than starting with a feature checklist, I want to answer a few basic questions about how each framework works:

How do I define an agent?
How does the agent interact with the model?
How are tools defined and connected?
Where does state live?
What control points do I have during execution?
How do multiple agents work together?
What does the framework provide for evaluation and observability?
And finally, how can I deploy what I've built?

We'll use those questions throughout the comparison.

And because I have a Computer Science degree and apparently cannot start a technical project any other way...

Note: The examples in this series were written and tested with:

AWS Strands Agents: 1.55.1

Google ADK: 2.9.0

Python: 3.13

Pin them if you install both into one environment. Unpinned, pip install strands-agents google-adk quietly gives you google-adk 1.14.1, a whole major version back, because ADK caps OpenTelemetry lower than Strands does and pip backtracks through ADK to satisfy both. Pinning the OpenTelemetry pair too keeps that from coming back when either framework moves:
strands-agents==1.55.1
google-adk==2.9.0
opentelemetry-api==1.42.1
opentelemetry-sdk==1.42.1
Both frameworks are actively developed. strands-agents 1.58.0 and google-adk 2.11.0 were out by the time this went up, and APIs may change after the versions above. Everything here was checked against the installed packages on 27 September 2026, and the code in this post was assembled and run.

Every example below is in a companion repo: github.com/coolchigi/strands-vs-adk. The code is in part1-hello-world/, with one numbered folder per section, from 01-hello-world to 08-deployment, and both frameworks side by side in each. It ships a smoke_test.py that parses every file, resolves every import, and builds every agent it can without calling a model, so you can confirm your setup works before spending a token.

Hello World

Here's how they both do it:

Strands	ADK
`pip install strands-agents`	`pip install google-adk`
One Python file	A folder with `agent.py` and `__init__.py`
`agent = Agent()`	`root_agent = Agent(...)`
`python hello-world.py`	`adk run course_agent`
Bedrock with Claude Sonnet 4.6 by default	Model named on the agent

Strands

The smallest Strands agent is this, and you run it with python hello-world.py:

# hello-world.py
from strands import Agent

agent = Agent()

agent("Hello, world!")

Strands works with several model providers. By default it uses Claude Sonnet 4.6 on Amazon Bedrock, so you need AWS credentials and access to that model in Bedrock to run this. The Strands Python quickstart walks through setting that up. To use a different model, pass its id to Agent as model="...".

Agent is the main thing you work with in Strands. When we call it, it sends our message to the model, and once we add tools, it also runs them and passes their results back to the model.

ADK

ADK expects a folder.

course_agent/
├── __init__.py     # from . import agent
├── agent.py        # defines root_agent
└── .env            # GOOGLE_API_KEY="..."

ADK finds the agent by importing agent.py and reading a variable called root_agent:

# course_agent/agent.py
from google.adk.agents import Agent

root_agent = Agent(
    model="gemini-flash-latest",
    name="root_agent",
    description="Says hello.",
    instruction="You are a helpful assistant.",
)

Agent and LlmAgent are the same class, so you can use either name.

Then we run it:

adk run course_agent

Or run adk web to chat with the same agent in your browser, with a trace of each call next to the chat.

Putting the two side by side

A Strands agent is a Python object that we create and call in our own script. An ADK agent is a folder, and ADK's commands find it and run it for us. The ADK Python quickstart walks through the full setup.

Once they're running, both work the same way:

Your application
        │
        ▼
      Agent
        │
        ▼
      Model
        │
        ▼
    Response

That's Hello World on both frameworks. Next, we'll give the agent a tool, or "hands", as they're often called.

Giving our agent a tool

Our learning system will eventually need to do things like retrieve documentation, create exercises, and keep track of what we've covered. For now, let's start with something much smaller.

We'll give the agent a function that returns a course topic:

def get_course_topic() -> str:
    """Get the topic for the course."""
    return "Building AI agents"

Each framework has its own way of turning that function into a tool the agent can use.

Here's how they both do it:

Strands	ADK
`@tool` on the function	Nothing on the function
`tools=[get_course_topic]`	`tools=[get_course_topic]`
Reads name, docstring, type hints, defaults	Reads name, docstring, type hints, defaults
Becomes a `DecoratedFunctionTool`	Wrapped in a `FunctionTool`

Strands

In Strands, we turn a simple Python function into a tool for the agent using the @tool decorator. The decorator uses the function's name, docstring, type hints and defaults to describe the tool. The name and docstring tell the model what the tool does, and the type hints tell it what to pass in. Then we pass it to the agent in tools:

# hello-world.py
from strands import Agent, tool

@tool
def get_course_topic() -> str:
    """Get the topic for the course."""
    return "Building AI agents"

agent = Agent(tools=[get_course_topic])

agent("What topic should I teach?")

ADK

In ADK, there's no decorator. We pass the plain function into tools, and ADK reads the same things (the name, docstring, type hints and defaults), then wraps it in a FunctionTool for us:

# course_agent/agent.py
from google.adk.agents import Agent

def get_course_topic() -> str:
    """Get the topic for the course."""
    return "Building AI agents"

root_agent = Agent(
    name="course_agent",
    model="gemini-flash-latest",
    description="Helps create learning materials.",
    instruction="You help create learning materials.",
    tools=[get_course_topic],
)

We run it the same way as before, from the 02-tools folder, this time with a question:

adk run course_agent "What topic should I teach?"

Putting the two side by side

Both sides use the same plain Python function. The only difference is that Strands wants @tool on it, and ADK takes it as it is.

We've given the agent the function, but we never told it to call it. So when we ask "What topic should I teach?", who decides whether it gets called?

The answer is ... the agent loop.

What runs the agent loop?

Here's how they both do it:

Strands	ADK
`agent("What topic should I teach?")` Agent → Model → Tool → Model → Response	`runner.run_async(...)` Runner → Agent → Model → Tool → Event → Runner → ...
`Agent` runs the loop itself.	A separate `Runner` runs the loop and hands each step back to your code as an event.

Strands

We call the agent ourselves. It's the last line of hello-world.py:

# hello-world.py
agent("What topic should I teach?")

From there, Agent runs the loop on its own. It goes back and forth between the model and your tools until the model gives a final answer.

The Strands agent loop documentation describes this as a repeating cycle:

User input
    │
    ▼
  Model
    │
    ├── No tool call ───────► Final response
    │
    └── Tool call
          │
          ▼
        Tool
          │
          ▼
      Tool result
          │
          ▼
        Model
          │
          └──────────────► ...

The model decides whether it needs a tool. If it does, Strands executes the tool and adds the result to the conversation before calling the model again. If the model produces a final response instead, the loop ends.

Strands also gives us controls around that loop. We can set limits for turns and tokens, cancel an invocation, and attach hooks around different points in the agent lifecycle.

ADK

In ADK, nothing in our code calls root_agent.

In Hello World, we typed adk run course_agent and the CLI called it for us. It built a Runner, created a session, wrapped what we typed in a types.Content, and passed it to runner.run_async().

So to see the loop, we write those four steps ourselves.

The agent.py from the tools section doesn't change. The new code goes in main.py, next to the course_agent folder, and imports the agent from it:

course_agent/
├── __init__.py
├── agent.py        # root_agent lives here, unchanged
└── .env
main.py             # everything below goes here

First, a session service, which keeps track of the conversation, and the Runner that runs the agent loop:

# main.py
import asyncio

from course_agent.agent import root_agent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types

session_service = InMemorySessionService()

runner = Runner(
    app_name="course_agent",
    agent=root_agent,
    session_service=session_service,
)

Creating a session and running the agent both have to be awaited, and await only works inside an async function, so the rest of the code goes in one:

# main.py, continued
async def main():
    session = await session_service.create_session(
        app_name="course_agent",
        user_id="user",
    )

We create the session first because run_async needs its session_id. The Runner won't create one for us unless we build it with auto_create_session=True.

Then the message itself. ADK expects a types.Content, which is one message with a role saying who sent it, and a list of types.Parts holding what it says. Ours comes from the user, with one part holding our question:

# main.py, still inside main()
    message = types.Content(
        role="user",
        parts=[types.Part(text="What topic should I teach?")],
    )

And now the loop itself. The last line, asyncio.run(main()), is what starts it all:

# main.py, the rest of the file
    async for event in runner.run_async(
        user_id="user",
        session_id=session.id,
        new_message=message,
    ):
        if event.is_final_response():
            if event.content:
                print(event.content.parts[0].text)
            else:
                print(f"no content: {event.error_message}")


asyncio.run(main())

Before running it, export your key in the same terminal. python main.py doesn't read the .env file. Only the ADK commands that load your agent, like adk run and adk web, do that. Run it from 03-agent-loop, the folder that holds both main.py and course_agent:

cd part1-hello-world/03-agent-loop
export GOOGLE_API_KEY="..."
python main.py

On Windows PowerShell, set the key with $env:GOOGLE_API_KEY="...".

The important part of main.py is the async for.

run_async() hands us a stream of events while the agent runs. According to the ADK event loop documentation, the Runner starts the agent, and each time the agent produces an event, the Runner saves it and passes it on before the agent carries on.

Simplified, the event loop looks like this:

Application
    │
    ▼
  Runner
    │
    ▼
  Agent
    │
    ▼
  Model
    │
    ├── Tool call
    │      │
    │      ▼
    │     Tool
    │      │
    │      ▼
    │   Tool result
    │
    ▼
  Event
    │
    ▼
  Runner
    │
    ├── Process event
    ├── Save state
    └── Forward event
           │
           ▼
        Agent continues

An Event can represent things such as user input, an agent response, a tool call or result, or a state change. The Runner passes these events to the configured SessionService, which can apply state changes and add the event to the session history before the event is passed back to the application. See the ADK event loop documentation for the full flow.

The else in our loop is worth keeping. is_final_response() returns True for error events as well as answers, and an error event has no content. Without the else, a bad API key ends the run with AttributeError: 'NoneType' object has no attribute 'parts', which says nothing about the key. With it, you get no content: 400 INVALID_ARGUMENT ... API key not valid, and the run ends on Google's own error instead.

Putting the two side by side

So we already have our first significant difference.

With Strands, the Agent owns the loop. We invoke the agent, and it manages the repeated model and tool calls for us.

With ADK, the Agent is one piece of a bigger setup. The Runner runs it, and every step comes back as an Event that the Runner saves and passes on to our code.

That shows up in the setup. Strands took one line. ADK took a session service, a runner, a session, and a message before anything reached the model.

That extra piece, the Runner, is going to matter when we look at where conversation state actually lives.

Where does state live?

Here's how they both do it:

Strands	ADK
`agent.messages` `agent.state` Invocation state	`session.events` `session.state` `SessionService`
Conversation history Messages exchanged with the model	Conversation history Events recorded in the session
Agent state A key value store the agent, tools, and our code can read and write	Session state A dictionary attached to the session
Invocation state Temporary data available during one invocation	State scopes Session, user, app, and temporary state

Strands

Strands keeps the conversation history on the Agent:

from strands import Agent

agent = Agent()

agent("My name is Chigo.")

print(agent.messages)

The messages contain the conversation between the user and assistant, including tool calls and tool results.

Strands also has agent state:

agent.state.set("course_topic", "Building AI agents")

print(agent.state.get("course_topic"))

agent.state.delete("course_topic")

agent.state is a key value store. We go through get, set and delete, because it isn't a real dictionary and agent.state["course_topic"] raises TypeError. Our code and our tools can both read and write it, and none of it reaches the model as part of the conversation.

The keys are strings we choose. The values have to be JSON serializable: strings, numbers, booleans, lists, dicts or None. Store a function as a value and Strands raises ValueError.

Strands also has invocation state. It lives for one call to the agent and disappears afterwards. Our tools, and the hooks we'll meet in the next section, can read it while that call runs, and the model never sees it.

The Strands state documentation covers these different types of state.

If we need state to survive beyond the current process, Strands provides session management. Sessions can persist agent state and conversation history using a configured session manager and storage backend. The Strands session management documentation covers the available options.

So for Strands, we can think about it like this:

Agent
├── Conversation ──┐
├── State ─────────┤
│                  ▼
│               Session
│                  │
│                  ▼
│               Storage
│
└── Invocation state   (gone when the call ends)

ADK

ADK keeps conversation history and state inside a Session.

A session represents a conversation between a user and an agent. It contains the events generated during that interaction and the state associated with the session.

The application works with a SessionService to create and retrieve sessions. We can put starting state in while we create one:

session = await session_service.create_session(
    app_name="course_agent",
    user_id="user",
    state={"course_topic": "Building AI agents"},
)

Reading it back is what we'd expect:

print(session.state["course_topic"])
# Building AI agents

So what about changing it once the session exists?

The obvious move is to assign straight into session.state. ADK's documentation has a warning section telling us not to. Modifying session.state on a session we pulled out of the SessionService bypasses the event history and won't reliably save. The ADK state documentation covers what else goes wrong.

For anything an agent produces, we give the agent an output_key:

researcher = Agent(
    name="researcher",
    model="gemini-flash-latest",
    instruction="Find information from official documentation and other reliable sources.",
    output_key="research",
)

Whatever researcher returns now lands in session.state["research"], and the Runner saves it through an event, the same way it saves everything else.

output_key is how one agent's output becomes the next agent's input. We'll lean on that in Part 2.

Inside a tool, or a callback (those come up in the next section), the rule is the same. We write to tool_context.state or callback_context.state, and ADK records the change on the event as its state_delta, so it gets saved like everything else.

ADK also supports different state scopes, and the scope is the prefix on the key. user:topic is shared across every session for that user. app:topic is shared across the whole application. temp:topic lasts for the current invocation. No prefix means the key belongs to this session.

Whether any of that survives a restart depends on the SessionService we picked. InMemorySessionService holds everything in memory, so session, user: and app: state die with the process. DatabaseSessionService writes to our own database (install it with pip install "google-adk[db]") and VertexAiSessionService to the managed one, and under either of those the same state comes back. temp: is never written to storage at all.

The ADK Sessions documentation explains how sessions, events, and state work together.

So the ADK picture looks more like:

SessionService
      │
      ▼
   Session
   ├── Events
   └── State
       ├── session
       ├── user
       ├── app
       └── temporary

When ADK processes an event, the Runner works with the SessionService to persist the event and apply any state changes before continuing execution.

Putting the two side by side

The difference shows up when we write state:

Strands exposes conversation history and agent state through the Agent, with session management available when we need persistence across invocations. ADK puts conversation history and session state behind a Session and SessionService.

What control points do we have?

Here's how they both do it:

Strands	ADK
`agent.add_hook(...)`	callbacks passed to `Agent(...)`
`BeforeInvocationEvent`	`before_agent_callback=...`
`AfterInvocationEvent`	`after_agent_callback=...`
`BeforeModelCallEvent`	`before_model_callback=...`
`AfterModelCallEvent`	`after_model_callback=...`
`BeforeToolCallEvent`	`before_tool_callback=...`
`AfterToolCallEvent`	`after_tool_callback=...`
`limits={"turns": 5}`	`RunConfig(max_llm_calls=200)`

Both frameworks let us run our own code at different points while an agent is running.

Strands

Strands uses hooks.

We create the agent and register a hook with add_hook():

from strands import Agent
from strands.hooks import BeforeModelCallEvent

def log_model_call(event: BeforeModelCallEvent) -> None:
    print("Calling the model")

agent = Agent()
agent.add_hook(log_model_call)

The hook receives a BeforeModelCallEvent, with details about the model call that's about to happen.

Strands provides hooks for different stages of an invocation, including model calls, tool calls, and the overall invocation. There are multi-agent events too. BeforeNodeCallEvent and AfterNodeCallEvent fire around each node in a graph, which we'll build in the multi-agent section.

For example, we can inspect a tool call before it runs:

from strands.hooks import BeforeToolCallEvent

def inspect_tool_call(event: BeforeToolCallEvent) -> None:
    print(f"Calling: {event.tool_use['name']}")

agent.add_hook(inspect_tool_call)

The hook can also affect what happens next. Depending on the event, we can modify a request, prevent a tool call, retry a tool call, or modify a result.

Strands also lets us put limits on an invocation:

agent(
    "Create a lesson about photosynthesis.",
    limits={"turns": 5, "total_tokens": 50_000},
)

turns caps how many times the agent loop goes round, where one turn is a model call plus any tools that run after it. output_tokens and total_tokens cap token spend across the whole invocation.

The Strands hooks documentation covers the available hook events and how to register them. The agent loop documentation covers invocation limits.

ADK

ADK uses callbacks.

With ADK, we pass callbacks when defining the agent:

def before_model_callback(callback_context, llm_request):
    print("Calling the model")

root_agent = Agent(
    name="course_agent",
    model="gemini-flash-latest",
    instruction="You help create learning materials.",
    before_model_callback=before_model_callback,
)

There are callbacks for different stages of execution, eight slots on the agent, and each one takes a function we write like the one above:

before_agent_callback    after_agent_callback
before_model_callback    after_model_callback    on_model_error_callback
before_tool_callback     after_tool_callback     on_tool_error_callback

The callback functions receive context about the current execution. Depending on the callback, we can inspect or modify what is about to happen, or change the result before execution continues.

The ADK callbacks documentation covers the different callback types and what each one can do.

Putting the two side by side

With Strands, we register hooks on the agent:

agent.add_hook(log_model_call)

With ADK, we provide callbacks when defining the agent:

Agent(
    ...,
    before_model_callback=log_model_call,
)

Strands takes hooks at construction too, with Agent(hooks=[...]).

The two frameworks also put execution limits in different places. Strands takes them on the call:

agent(
    "...",
    limits={"turns": 5},
)

ADK puts them on the run instead, in a RunConfig we hand to the runner:

from google.adk.agents import RunConfig

config = RunConfig(max_llm_calls=200)

async for event in runner.run_async(..., run_config=config):
    ...

Go past the cap and ADK raises LlmCallsLimitExceededError.

ADK sets max_llm_calls to 500 whether we ask for it or not. Strands leaves limits unset, so nothing is capped until we say so. The ADK RunConfig documentation covers the other run level settings, including streaming mode and artifact saving.

So if we want to add code around a model or tool call, both frameworks give us a place to do it. Strands calls them hooks and ADK calls them callbacks. On Strands we can add a hook to an agent we already have, and on ADK the callbacks go in when we build the agent.

What if one agent isn't enough?

We've seen how both frameworks let us control what happens while an agent is running.

But what happens when one agent isn't enough?

For our learning system, we need a few different agents working together:

Researcher: Finds information from official documentation, blogs, and other reliable sources.
Curriculum Builder: Takes the research and turns it into a structured curriculum.
Teacher: Takes the curriculum and creates the actual lessons.
Feedback Agent: Reviews the work produced by the other agents and identifies what needs revision.

The basic flow looks like this:

Researcher
     │
     ▼
Curriculum Builder
     │
     ▼
Teacher
     │
     ▼
Feedback

But the Feedback Agent isn't only checking the lessons.

It needs to check the research, curriculum, and lessons, and send each one back for another pass when something's wrong.

So how do Strands and ADK let us put agents together?

Here's how they both do it:

Strands	ADK
An agent can be passed directly into another agent's `tools` list.	An agent can have other agents as `sub_agents`.
`tools=[researcher]`	`sub_agents=[researcher]`
The parent agent can decide when to call another agent.	A parent agent can delegate work to child agents.
Graphs can connect agents into a defined workflow.	Graph based workflows can connect agents and other executable nodes into a defined workflow.

Strands

The simplest way to have agents work together in Strands is to give one agent another agent as a tool.

from strands import Agent

researcher = Agent(
    name="researcher",
    system_prompt="Find information from official documentation, blogs, and other reliable sources."
)

curriculum_builder = Agent(
    name="curriculum_builder",
    system_prompt="Turn the research into a structured curriculum.",
    tools=[researcher],
)

Now curriculum_builder can decide when it needs the researcher.

Passing an Agent into tools converts it into a tool that takes one string and hands back the agent's answer. Anything more structured going in has to be squeezed through that one string.

We could continue this pattern by giving the Teacher access to the Curriculum Builder, and the Feedback Agent access to the Teacher:

teacher = Agent(
    name="teacher",
    system_prompt="Create lessons from the curriculum.",
    tools=[curriculum_builder],
)

feedback = Agent(
    name="feedback",
    system_prompt="Review the research, curriculum, and lessons.",
    tools=[teacher],
)

But there's a problem with representing our workflow this way.

The relationship is controlled by the agent making the call. If teacher needs the curriculum, it can call curriculum_builder. If feedback needs the lesson, it can call teacher.

But we already know the order we want. We also know that Feedback needs to review different parts of the system and potentially trigger a revision.

For that, Strands provides Graphs.

A Graph lets us define the nodes and edges between agents ourselves. Agents become nodes, and the edges set the order they run in. An edge can also point back to an earlier node, which is how we get the feedback loop our system needs.

Drop the tools wiring before doing this. The graph replaces it. Leave both in and every node can run the whole chain below it again through its tools: Feedback calls Teacher, which calls Curriculum Builder, which calls Researcher. And when a model asks for the same agent twice at once, Strands refuses the second call while the first is still running:

tool_name=<researcher>, tool_use_id=<...> | agent is already processing a request

That comes back to the calling model as a tool error, so it asks again. My run took ten minutes instead of ninety seconds and never finished. So the four agents below are the same as before, minus tools=[...]:

researcher = Agent(name="researcher", system_prompt="...")
curriculum_builder = Agent(name="curriculum_builder", system_prompt="...")
teacher = Agent(name="teacher", system_prompt="...")
feedback = Agent(name="feedback", system_prompt="...")

Then the graph itself:

from strands.multiagent import GraphBuilder

builder = GraphBuilder()

builder.add_node(researcher, "researcher")
builder.add_node(curriculum_builder, "curriculum")
builder.add_node(teacher, "teacher")
builder.add_node(feedback, "feedback")

builder.add_edge("researcher", "curriculum")
builder.add_edge("curriculum", "teacher")
builder.add_edge("teacher", "feedback")

Edges also take a condition, and that's how the graph gets its cycle:

def lesson_needs_work(state) -> bool:
    feedback = state.results["feedback"].result
    return "revise the lesson" in str(feedback).lower()

builder.add_edge("feedback", "teacher", condition=lesson_needs_work)

The condition gets the graph state. state.results is a dict keyed by node id, and each entry's .result holds what that node returned. So the Feedback Agent's own output decides whether the edge back to teacher is taken.

Then we bound it and build it:

builder.set_entry_point("researcher")
builder.set_max_node_executions(10)
builder.reset_on_revisit(True)

graph = builder.build()

Order matters here. build() takes a snapshot of the edges, so an edge added afterwards never makes it into the graph and the loop silently isn't there.

set_max_node_executions() caps the total number of node runs, which is what stops a feedback loop that never converges. Hit the cap and the graph ends with a FAILED status, so check the result.

set_execution_timeout() is the other way to put a boundary around it, with a time limit for the whole graph.

reset_on_revisit(True) clears a node's previous state when the graph comes back to it, so a second pass starts clean.

So our eventual workflow could look more like:

Researcher
     │
     ▼
Curriculum Builder
     │
     ▼
Teacher
     │
     ▼
Feedback
  │     │     │
  ▼     ▼     ▼
Research Curriculum Lessons
  │     │     │
  └─────┴─────┘
        │
        ▼
     Revision

The important part is that the Graph describes the relationships between the agents instead of leaving the entire workflow to one agent's decisions.

ADK

ADK gives us a parent and child relationship between agents.

An agent can contain other agents as sub_agents:

from google.adk.agents import Agent

researcher = Agent(
    name="researcher",
    model="gemini-flash-latest",
    instruction="Find information from official documentation, blogs, and other reliable sources."
)

curriculum_builder = Agent(
    name="curriculum_builder",
    model="gemini-flash-latest",
    instruction="Turn the research into a structured curriculum.",
    sub_agents=[researcher],
)

Here, curriculum_builder is the parent and researcher is its child.

Whether the parent hands work to the child is the model's decision, the same as the Strands version. ADK's source says the description field is what the model reads to decide whether to delegate. So sub_agents sets up the hierarchy, and the model still picks the path through it.

ADK also supports collaborative workflows, where a coordinator agent works with a set of sub agents.

But our learning system has a defined sequence. In ADK 2.x that's a graph based workflow.

Same warning as the Strands side: drop the sub_agents wiring first. Leave sub_agents=[researcher] on curriculum_builder and the model can hand control to researcher in the middle of the run, so a node runs twice or the workflow stops early. So, four fresh agents:

researcher = Agent(name="researcher", model="gemini-flash-latest", instruction="...")
curriculum_builder = Agent(name="curriculum_builder", model="gemini-flash-latest", instruction="...")
teacher = Agent(name="teacher", model="gemini-flash-latest", instruction="...", output_key="lesson")
feedback = Agent(name="feedback", model="gemini-flash-latest", instruction="...")

A graph is a list of edges, and each node can be an agent, a plain function, a tool, or another workflow:

from google.adk import Workflow

root_agent = Workflow(
    name="learning_system",
    edges=[
        ("START", researcher, curriculum_builder, teacher, feedback),
    ],
)

Each edge lists nodes that run one after another, and one node's output becomes the next node's input.

That covers the sequence. Now the feedback loop.

ADK's graphs loop too. The routes page calls it a back-edge: a router node returns a route, and one of its routes points at an earlier node.

from google.adk import Event, Workflow

rounds = {"n": 0}

def review(node_input: str, lesson: str) -> Event:
    rounds["n"] += 1
    if "revise the lesson" in node_input.lower() and rounds["n"] < 3:
        return Event(route="revise", output=f"Your lesson: {lesson}\nFeedback: {node_input}")
    return Event(route="done", output=node_input)

def finish(node_input: str) -> str:
    return node_input

root_agent = Workflow(
    name="learning_system",
    edges=[
        ("START", researcher, curriculum_builder, teacher, feedback, review),
        (review, {"revise": teacher, "done": finish}),
    ],
)

review is a plain Python function, and ADK is happy to make it a node. It reads what feedback said and picks the route. revise goes back to teacher, which is the cycle. Pass output on each route, because the event is what carries the text on. Leave it off and the next node gets None.

On the revise route, teacher only gets what review hands it, because an agent in a workflow only sees what the node before it passes along. ADK calls this single_turn mode. If review only sends the feedback, teacher never sees the lesson it's meant to fix, so it writes a brand new one.

To check, I had the feedback agent send back any lesson that didn't end with a quiz question. I ran it 3 times, and none of the rewritten lessons were about photosynthesis. One was about budgeting and the other 2 were about Python variables.

So we save the lesson and send it back with the feedback. output_key="lesson" on teacher puts its lesson in session state, and the lesson parameter on review reads it back, because ADK fills a function node's parameters from state. The revise route then hands teacher both.

Strands doesn't need any of this. When a graph comes back to a node, Strands hands it the original task along with the output of each node that points to it, which here means the plan and the feedback.

ADK checks one thing about that cycle when you build it: it has to pass through a routed edge. Wire feedback straight back to teacher and construction fails with "Unconditional cycle detected". What it doesn't do is stop a routed cycle. The same page says so plainly: "A graph cycle is not bounded automatically." The rounds counter is the only thing ending that loop, and it's ours.

ADK also has LoopAgent, one of its template workflows, which runs its sub agents over and over until one escalates or max_iterations runs out. Build one on 2.9.0 and it raises a DeprecationWarning:

LoopAgent is deprecated in favor of Workflow and will be removed in a
future version. Workflow cannot yet be used as an LlmAgent sub-agent.

Python hides that warning unless the code runs as a script, under pytest, or with -W default, so plenty of people will never see it. The graph is the way forward either way.

So both frameworks declare the loop inside the graph. The difference is who stops it. Strands bounds it for you with set_max_node_executions(). ADK leaves the exit to your router.

If the Feedback Agent finds a problem, we still need to decide which agent gets another pass. On both frameworks, we write that routing into the graph ourselves, so it doesn't depend on one agent choosing to call another.

We'll come back to this in Part 2, because it decides how the Feedback Agent gets wired.

Putting the two side by side

At this point, the two frameworks give us several ways to build the same system.

	Strands	ADK
Define a specialist agent	`Agent(...)`	`Agent(...)`
Give one agent another agent to call	`tools=[researcher]`	`sub_agents=[researcher]`
Define a fixed workflow	Graph	Graph based workflow
Pass output between workflow steps	Graph edges	Edges, each node's output is the next one's input
Represent a feedback loop	Cyclic Graph, capped by `set_max_node_executions()`	Routed back-edge, stopped by our router
Let a coordinator choose which specialist to call	Agents as Tools	Collaborative workflow

With Strands, we can start with agents as tools and move to a Graph when we need the workflow to be explicit. Strands' Graph pattern supports sequential execution, parallel branches, conditional paths, and feedback loops.

ADK's graph workflows cover the sequence and the loop. The loop just doesn't stop itself.

For our learning system, we'll use the four agents we just defined, with the Feedback Agent checking each stage's work and sending it back when something's wrong.

That gives us the structure we'll use when we build the actual system in Part 2.

What does the framework provide for evaluation and observability?

We've now got agents, tools, state, hooks and callbacks, and ways to connect multiple agents.

Okay, but how do I know if it's actually working? There are two things I want us to look at:

Evaluation: Did the agent do what I expected?
Observability: What actually happened while it was running?

Here's how they both do it:

Strands	ADK
Evaluators we import and run in code	Datasets we write to a file
`ToolCalled`, `Equals`, `Contains`	`tool_trajectory_avg_score`, `response_match_score`
`TrajectoryEvaluator` and other LLM judges	`final_response_match_v2`, `hallucinations_v1`, `safety_v1`
Metrics on the result object	OpenTelemetry to Cloud Trace
Bring your own dashboard	BigQuery Agent Analytics through the Agents CLI

Evaluating a Strands agent

Strands has something for both.

The Strands Evals SDK covers several types of evaluation, including checking the final output, tool usage, trajectories, and using an LLM as a judge. It ships separately:

pip install strands-agents-evals

Watch the name. You install strands-agents-evals and you import strands_evals. Guessing the install name from the import gets you strands-evals, which is a squatted 4KB stub on PyPI by an unrelated author.

Let's start with something simple. Say we want to make sure our course agent actually uses the get_course_topic tool when it runs.

A deterministic evaluator can check that directly:

from strands_evals.evaluators import ToolCalled

evaluator = ToolCalled(tool_name="get_course_topic")

The evaluator simply checks the agent's trajectory and determines whether that tool was called. The deterministic evaluators documentation also includes evaluators such as Equals, Contains, StartsWith, and StateEquals.

But some things are harder to check with an exact rule.

For example, we might want to know whether the Researcher used the right tools in the right order and avoided unnecessary calls. We could write deterministic checks for specific sequences, but that becomes harder when there are multiple valid ways for an agent to complete a task.

Strands provides evaluators that can use a model to assess an agent's behaviour against a rubric. For example, the TrajectoryEvaluator lets us define what a good tool usage trajectory should look like:

from strands_evals.evaluators import TrajectoryEvaluator

evaluator = TrajectoryEvaluator(
    rubric="""
    Evaluate the tool usage trajectory:
      1. Were the right tools chosen for the task?
      2. Were the tools used in a logical order?
      3. Were unnecessary tools avoided?

    Score 1.0 if the tools were used correctly and efficiently.
    Score 0.5 if the right tools were used but the sequence was suboptimal.
    Score 0.0 if the wrong tools were used or there were major inefficiencies.
    """,
    include_inputs=True,
)

The evaluator can then look at the agent's trajectory and score it against the rubric. This is useful when we're testing behaviour that is difficult to reduce to a single expected string.

The Trajectory Evaluator documentation goes further into evaluating the sequence of actions and tool calls made during an execution.

We can also evaluate the actual response. Strands provides LLM based evaluators for things like helpfulness and correctness, where another model acts as the judge and scores the agent's response against the evaluation criteria.

That gives us two useful approaches:

Deterministic evaluator
    ↓
"Did the agent call get_course_topic?"
    ↓
Exact check


LLM based evaluator
    ↓
"Did the agent use the right tools in a sensible way?"
    ↓
Judgement against a rubric

For our learning system, we could use deterministic checks for things we know should always happen, such as making sure the Researcher calls the documentation search tool. We could use an LLM based evaluator for things that require judgement, such as whether the research was actually useful.

Evaluating an ADK agent

ADK comes at this from the dataset end. We describe the cases we want to test in a file, pick the metrics, and let ADK run the agent against them.

There are two file formats. A .test.json file holds a single agent interaction. An .evalset.json file holds multiple, longer sessions.

Evaluation isn't in the base install. Both routes below need the eval extra, and the CLI tells you so if you forget:

pip install "google-adk[eval]"

Then we can run a dataset from the command line:

adk eval course_agent tests/researcher.evalset.json

Or from pytest, which keeps evaluation in the test suite next to everything else. The test is async, so it also needs pip install pytest pytest-asyncio:

import pytest

from google.adk.evaluation.agent_evaluator import AgentEvaluator


@pytest.mark.asyncio
async def test_researcher():
    await AgentEvaluator.evaluate(
        agent_module="course_agent",
        eval_dataset_file_path_or_dir="tests/researcher.evalset.json",
    )

ADK ships its own metrics. tool_trajectory_avg_score does an exact match on the tool call trajectory. response_match_score scores ROUGE-1 similarity against a reference response. For the judgement calls there's final_response_match_v2, rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1, hallucinations_v1, and safety_v1.

So we end up with the same split Strands gave us. Exact checks for things with a right answer, LLM judged metrics for things that need an opinion.

The ADK evaluation documentation covers the datasets, the metrics, and the three ways to run them: the CLI, the adk web UI, and pytest.

A detour, because this is about to get confusing

Some of what I show you from here on isn't ADK.

Google also ships the Agents CLI, which its own docs describe as a way to "Build, evaluate, and deploy ADK agents with a single unified CLI." It's a separate tool that sits on top of ADK. It scaffolds a project, wires up telemetry, runs evaluations, and handles deployment.

I'm flagging it because it changes what we're comparing. The Strands Evals SDK is a library we import into our code. The Agents CLI is a project toolchain we run from a terminal. They overlap, and the metric names are different on each side.

An Agents CLI project comes with a default evaluation dataset at tests/eval/datasets/basic-dataset.json. We add our own cases and run them with:

agents-cli eval run \
  --dataset tests/eval/datasets/custom-dataset.json \
  --metrics final_response_quality,grounding

The command runs the agent against the dataset, collects the resulting traces and grades them against the metrics we selected. The eval fix loop in the Agents CLI documentation recommends running the evaluation, inspecting what failed, making a change, and running it again.

For our learning system, a test case could represent something we expect one of our agents to accomplish.

For example, we could give the Researcher a prompt like:

{
  "prompt": "Research AWS S3 Lifecycle Management for a certification lesson."
}

We could then evaluate whether the Researcher produced useful, grounded research and used the tools we expect.

We could do something similar with the Teacher:

{ "prompt": "Create a lesson on S3 Lifecycle Management from the provided curriculum." }

Here, we aren't looking for one exact lesson. There are many ways the Teacher could explain the topic correctly.

Instead, we care about whether the lesson meets the requirements we gave it. Does it accurately explain S3 Lifecycle Management? Does it stay grounded in the research? Does it cover the concepts required by the curriculum? Does the hands on exercise actually give the learner a chance to apply what they just learned?

Those are judgement calls, so a model has to make them.

The Agents CLI metrics include general_quality, instruction_following, tool_use_quality, final_response_quality, grounding, and hallucination, and agents-cli eval metric list prints the rest. The evaluation system also supports custom metrics, including rubric based evaluation using a judge model.

For example, we could define a rubric for the Teacher that asks the evaluator to judge whether the lesson accurately covers the concepts required by the curriculum, stays grounded in the research, and provides a meaningful hands on exercise.

We can also evaluate the agent's execution trace rather than looking only at its final response. ADK's trajectory evaluation documentation covers evaluating the sequence of tools an agent used. The Agents CLI's tool_use_quality metric evaluates tool selection, parameter accuracy, and step sequence correctness.

That gives us a few different ways to test our system.

Agent	What we could evaluate	Example
Researcher	Tool usage, source quality and research relevance	Did it search the right documentation and gather useful information for the topic?
Curriculum Builder	Instruction following, coverage and curriculum quality	Did it cover the required concepts and turn them into a useful learn by doing progression?
Teacher	Accuracy, grounding and instructional quality	Does the lesson accurately teach the required concepts and give the learner a meaningful exercise?
Feedback Agent	Feedback quality and correctness	Did it identify real issues without inventing problems?

The evaluation method depends on what we're trying to measure. A deterministic check works well for something concrete, such as whether the Researcher called a required tool. For a subjective check, like whether a Teacher's lesson is accurate and useful, an LLM based evaluator can judge the result against a set of criteria.

Evaluation vs Feedback

There is one distinction worth calling out: the Feedback Agent is not the evaluation system.

The Feedback Agent is part of our application that reviews the work produced by the other agents and identifies what should be revised.

Evaluation is part of our development workflow. It tells us whether the agents themselves are behaving the way we designed them to.

Learning system

Researcher
    ↓
Curriculum Builder
    ↓
Teacher
    ↓
Feedback Agent
    ↓
Revision
    └────────→ relevant agent


Development workflow

Agent
    ↓
Evaluation
    ↓
Identify failure
    ↓
Improve agent
    ↓
Evaluate again

The learning system's loop improves the content, and the development loop improves the agents.

Observing a Strands agent

Strands builds observability in, with metrics, logs and traces. The Strands observability documentation explains how these pieces fit together.

We can access execution metrics directly from the result returned by an agent:

from strands import Agent
agent = Agent()
result = agent("Create a lesson about photosynthesis.")

print(f"Total tokens: {result.metrics.accumulated_usage['totalTokens']}")
print(f"Execution time: {sum(result.metrics.cycle_durations):.2f} seconds")
print(f"Tools used: {list(result.metrics.tool_metrics.keys())}")

So if our Teacher suddenly becomes much slower, we can look at the execution time and see which tools were used during that invocation. The Strands metrics documentation also exposes information about token usage, tool calls, execution time, errors, and agent loop cycles.

Strands also collects local execution traces. We can inspect them through the result:

result = agent("Create a lesson about photosynthesis.")

print(result.metrics.get_summary())

The summary gives us a view of the execution hierarchy, including the agent's cycles, model calls, and tool calls.

For deeper tracing, Strands uses OpenTelemetry. A trace can show the model invocations and tool executions that happened during an agent run, including information such as model inputs and outputs, token usage, tool parameters, and execution time. The Strands traces documentation covers how those traces are structured.

Observing an ADK agent

ADK also uses OpenTelemetry, with Google Cloud's observability tools sitting around it.

Suppose our Teacher fails an evaluation because the lesson isn't grounded in the research we provided. How do we understand what happened while that lesson was being created? Which tools did our agent use? Did a tool call fail?

To answer those questions, we need to see the execution itself.

Following an agent's execution

The Agents CLI uses OpenTelemetry to collect information about an ADK agent's execution. That information can be sent to Cloud Trace, a Google Cloud service for recording and inspecting the operations that happen during a request.

A trace represents one execution. Within that trace, individual operations are recorded as spans. For an agent, those spans can include things such as model calls and tool executions.

You can think of it as moving from:

"What did the agent return?"

"What happened while the agent was producing that result?"

The Cloud Trace documentation walks through how the Agents CLI connects agent executions to Cloud Trace.

Let's try it.

We've been running our agents locally, so we'll start there. The Agents CLI works inside a project it creates for us, following its getting started guide:

agents-cli create my-agent
cd my-agent
agents-cli install

From inside it, we can run the agent and send the trace to Google Cloud at the same time:

agents-cli run "Create a lesson about S3 Lifecycle Management" \
  --otel-to-cloud

In case the name didn't give it away, the --otel-to-cloud flag sends the trace from our local run to Google Cloud.

Once the run finishes, we can open GCP Console → Trace → Trace Explorer and find the trace for that request.

The View Traces section of the Agents CLI documentation shows where these traces appear in the Google Cloud Console.

There, we can inspect the operations that made up the run.

Instead of only seeing the lesson that came back from the Teacher, we can see the execution that produced it.

Local execution and deployed agents

Currently we are sending the agent's trace to GCP so we can inspect it there, but the agent is still running locally. When we eventually deploy our agent, the execution itself happens in Google Cloud. Cloud Trace is enabled by default for deployed agents.

Hopefully this diagram helps you make sense of where we're at:

Local agent
  │
  │ run agent
  ↓ Our machine
  │
  │ send trace
  ↓
  Google Cloud
  │
  ↓ Trace Explorer

Our development workflow would look something like:

Run locally
↓
Inspect the trace
↓
Find something interesting
↓
Change the agent
↓ Run it again

When an execution fails, we can go from a failed evaluation case to the execution that produced it and inspect what the agent actually did. We're still missing something. A trace in itself can show us the operations that took place during the run, but how do we see the input and output associated with those calls?

Looking at prompts and responses

Prompt response logging is separate from Cloud Trace. The Agents CLI can record prompt and response information in Google Cloud Storage as JSONL and in a BigQuery completions table.

The prompt response logging documentation shows how this works.

Now we have the execution from Cloud Trace, and the prompts and responses that went with it. That's enough to investigate one run in detail.

But our agents are going to run many times. The Researcher might run many times while gathering material, the Teacher could generate hundreds of lessons, and our evaluations could run across all of them.

We could inspect each trace individually, but eventually we'd want to ask questions across all of those executions.

Which tools are being called most often? How many errors are we seeing? Are there patterns across our runs?

Those are SQL questions.

Looking across many executions

BigQuery is Google's data warehouse. It gives us a place to store structured data and query it using SQL.

The Agents CLI provides BigQuery Agent Analytics for ADK based agents. It's an opt in plugin that records detailed agent events in an agent_events table, giving us a way to analyze our agent runs across many executions.

The BigQuery Agent Analytics documentation explains how to enable it and configure the analytics plugin. When creating an ADK based project, we can include the plugin with the --bq-analytics flag:

agents-cli create my-agent \
  -d cloud_run \
  --bq-analytics

That flag adds the BigQuery Agent Analytics plugin to the generated project. The configuration lives in app/agent.py, where the plugin is added to the ADK application.

The plugin needs a dependency the base install does not carry, so the import fails with
No module named 'google.api_core' until we add the extra:

pip install "google-adk[bigquery-analytics]"

The generated configuration looks roughly like this:

import os

from google.adk.apps import App
from google.adk.plugins.bigquery_agent_analytics_plugin import (
    BigQueryAgentAnalyticsPlugin,
    BigQueryLoggerConfig,
)

bq_config = BigQueryLoggerConfig(
    enabled=True,
    table_id="agent_events",
)

bq_analytics_plugin = BigQueryAgentAnalyticsPlugin(
    project_id=os.environ.get("GOOGLE_CLOUD_PROJECT"),
    dataset_id=os.environ.get("BQ_ANALYTICS_DATASET_ID"),
    table_id=bq_config.table_id,
    config=bq_config,
    location=os.environ.get("GOOGLE_CLOUD_LOCATION", "US"),
)

app = App(
    name="my-agent",
    root_agent=root_agent,
    plugins=[bq_analytics_plugin],
)

The important part here is the BigQueryAgentAnalyticsPlugin. We configure it with the Google Cloud project and dataset where the events should be stored, then add the plugin to our ADK application.

There are more configuration options available, including filtering which event types are recorded and controlling how larger pieces of content are handled. We don't need those for our example, so we'll keep the configuration simple.

Once events are being recorded, we can query them with SQL.

For example, we could look at tool calls and errors across our runs:

SELECT
  timestamp,
  event_type,
  JSON_VALUE(content, '$.tool') AS tool_name,
  JSON_QUERY(content, '$.args') AS tool_args,
  status,
  error_message
FROM `YOUR_PROJECT_ID.YOUR_AGENT_NAME_telemetry.agent_events`
WHERE event_type IN ('TOOL_STARTING', 'TOOL_ERROR')
ORDER BY timestamp DESC;

The docs' version of this query filters on TOOL_COMPLETED and uses JSON_VALUE, and its tool_args column comes back empty every time. The plugin only writes the arguments on TOOL_STARTING and TOOL_ERROR rows, and they're a JSON object, which JSON_VALUE returns as NULL. JSON_QUERY keeps them.

Now we're asking questions across our executions instead of opening traces one at a time.

The example queries in the documentation show other ways to analyze the recorded events, including token usage.

For our project, this gives us another way to investigate a failed evaluation. We can take the failed case, find the execution behind it, inspect what happened, and then look across our other runs to see whether the same problem is happening elsewhere.

Putting the two side by side

At this point, we've looked at evaluation and observability separately. Here's how the pieces line up:

	Strands	ADK
Evaluation	Evals SDK with deterministic and LLM based evaluators	`.test.json` and `.evalset.json` datasets, run with `adk eval` or pytest
Evaluating tool use	`ToolCalled`, trajectory evaluators	`tool_trajectory_avg_score`, plus `tool_use_quality` in the Agents CLI
Observability	Metrics, logs, local traces and OpenTelemetry	OpenTelemetry, with Cloud Trace and BigQuery wired up by the Agents CLI
Investigating one execution	Execution metrics and traces	Cloud Trace and prompt response logging (Agents CLI)
Investigating many executions	Metrics and external observability integrations	BigQuery Agent Analytics (Agents CLI)

And finally, how can I deploy what I've built?

We've been running everything locally so far.

How do we deploy?

Strands hands us a Python application and leaves the rest to us. ADK has a command for it, two in fact. adk deploy ships in google-adk itself with four targets: agent_engine, cloud_run, docker and gke. Google's Agents CLI is a separate tool that wraps a whole project around that, and it's the one I'll use below.

Here's how they both do it:

Strands	ADK
We package the application	`agents-cli deploy`
We write the Dockerfile	The CLI builds the container
We provision the infrastructure	`-d cloud_run`, `agent_runtime` or `gke`
Lambda, Fargate, App Runner, EKS, EC2, AgentCore	Agent Runtime, Cloud Run, GKE

Strands: bring your own runtime

With Strands, our agent is part of a Python application. Once we're ready to deploy it, we need to package that application and decide where it should run.

One option is Docker. The Strands Docker deployment guide walks through containerizing a Strands agent so we can run the same application locally or on a cloud service.

For example, we could have a Dockerfile like this:

FROM public.ecr.aws/docker/library/python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

requirements.txt needs strands-agents, fastapi and uvicorn. The CMD starts a web server, because the app below is a FastAPI app with nothing to do if you just run it with python.

The Dockerfile packages our Python application and its dependencies into an image. We can then run that image wherever we choose.

AWS gives us several options such as Lambda, Fargate, App Runner, EKS, EC2, and Bedrock AgentCore.

If we choose the Fargate deployment for example, we would create a containerized FastAPI application. We'd use AWS CDK to create the Fargate infrastructure, build the Docker image, and deploy the service.

The application might look something like this:

from fastapi import FastAPI
from pydantic import BaseModel
from strands import Agent

app = FastAPI()


class Question(BaseModel):
    question: str


@app.post("/ask")
def ask(body: Question):
    agent = Agent(system_prompt="You are a helpful certification teacher.")
    result = agent(body.question)
    return {"response": str(result)}

FastAPI gives our application an HTTP endpoint. The Strands agent is still the same agent we've been working with.

Two things in there are on purpose. The agent is built inside the handler, one per request. An Agent holds its own conversation, so a single shared one would mix every caller's messages together, and Strands refuses a second call while the first is still running. And str(result) is the text of the answer. result.message is the whole message, role and content blocks included.

From there, the deployment process depends on the AWS service we choose.

ADK: deploy with the Agents CLI

With ADK, we have the Agents CLI from the evaluation section.

The CLI can create a project with a deployment target already configured. For example, if we want to deploy our agent to Cloud Run:

agents-cli create my-agent \
  -d cloud_run

The -d cloud_run option sets Cloud Run as the deployment target. The Agents CLI deployment documentation covers the available deployment targets, which currently include Agent Runtime, Cloud Run, and GKE.

Before deploying, we tell Google Cloud which project to use with Google's command line tool, gcloud. It comes with the Google Cloud SDK, which you install separately:

gcloud config set project YOUR_DEV_PROJECT_ID

Then we deploy:

agents-cli deploy

agents-cli deploy knows where to deploy because we chose Cloud Run with -d cloud_run when we created the project.

For Cloud Run, the CLI builds a container from our source and deploys it as a Cloud Run service. We don't have to write a Dockerfile, the way we did for Strands.

We can list what's deployed with:

agents-cli deploy --list

The Agents CLI command reference also covers --no-wait, which starts a deployment and returns straight away, and --status for checking on one started that way.

An agent also needs cloud resources around it, like service accounts, permissions and storage for telemetry. For a basic deployment, agents-cli deploy on its own is enough. When we want to set those resources up ourselves, the Agents CLI has a separate command, agents-cli infra, and the deployment documentation explains how the two fit together.

So with ADK, the CLI takes care of more of the steps between our local project and the deployed application.

Putting the two side by side

	Strands	ADK
What are we deploying?	Our Python application containing the Strands agent	Our ADK project
How do we deploy?	Choose a deployment option and configure it for our application	`agents-cli deploy`
Containerization	We package the application when the chosen deployment uses containers	The CLI handles the container build for supported targets such as Cloud Run
Infrastructure	We configure the AWS resources needed by our chosen deployment	`agents-cli infra` can provision infrastructure when we need more control
Deployment targets	Lambda, Fargate, App Runner, EKS, EC2, AgentCore, and others	Agent Runtime, Cloud Run, GKE

We started with:

How do I define an agent?

And ended up with:

How do I evaluate it, see what it is doing, and actually run it somewhere?

Here's what we have learned so far:

Question	Strands	ADK
Define an agent	`Agent(...)`	`Agent(...)`
Connect tools	`tools=[...]`	`tools=[...]`
Run the agent	`Agent` runs the loop itself	A separate `Runner` runs the loop
Store state	Agent state, messages, sessions	Sessions, events, state scopes
Add execution control	Hooks, `limits=` on the invocation	Callbacks, `RunConfig` on the run
Connect agents	Agents as tools, Graphs	Sub agents, collaborative workflows, Graphs
Define workflows	Graphs	Graph based workflows
Evaluate behaviour	Strands Evals SDK	ADK evaluation tooling
Observe execution	OpenTelemetry, metrics, logs	OpenTelemetry, Cloud Trace, Google Cloud tooling
Deploy	Your own infrastructure or AgentCore	Google Cloud deployment targets

For our learning system, we now have enough pieces to make some decisions.

We know what our agents look like.

We know how they'll communicate.

We know where their state can live.

We know how to control execution.

We know how to connect them into a workflow with a feedback loop.

And we know how we'll eventually test, observe, and deploy the thing.

Now we can actually build it.

That's Part 2.

RAG Powered Apps with Amazon Bedrock, Part 2: Automating the RAG Pipeline with Terraform

Chigozirim Eke — Fri, 07 Aug 2026 00:47:33 +0000

Before you start: This picks up where Part 1 left off. In Part 1, we walked through setting up a Bedrock Knowledge Base manually through the AWS Console.

Introduction & Motivation

I started this project with a simple goal: build a reusable Terraform setup that could deploy the infrastructure behind a "Chat with PDF" application end to end.

When Amazon Bedrock launched, I immediately started experimenting. Like many early GenAI workflows, my first few prototypes were built directly through the AWS Console. The console experience is great for quickly understanding how the pieces fit together, but once I started iterating, I wanted something more repeatable.

I wanted to be able to spin environments up, tear them down, and quickly test different architecture decisions. For example, I wanted to compare the cost differences between using S3 Vectors and OpenSearch as the vector store without manually rebuilding the infrastructure every time.

At the time, I couldn't find a Terraform module that covered the complete RAG pipeline I was looking for: S3, Bedrock Knowledge Bases, OpenSearch Serverless, Lambda, IAM, and all the glue in between.

So I built my own. Partly because I needed it, and partly because understanding how these services connect together is the best way to learn.

What Are We Building?

A couple of terraform modules to automate everything we clicked through manually in Part 1. One terraform apply brings up the full stack:

S3 Bucket: your document store. Encrypted at rest, versioning on, zero public access.
OpenSearch Serverless: the vector database. Stores the embeddings Bedrock generates during ingestion.
Bedrock Knowledge Base: orchestrates the chunking, embedding, and storage of documents, and retrieval at query time.
Ingestion Lambda: triggered automatically when you upload a file to S3. Starts a Bedrock ingestion job so documents are chunked, embedded, and indexed without ClickOps.
Query Lambda: accepts a natural language question, calls RetrieveAndGenerate, and returns an answer with source citations.

Full source code + ReadMe: Bedrock Project. If you run into issues or want to extend the module, feel free to open an issue.

Architecture

Project Structure

rag-bedrock-project/
├── main.tf
├── variables.tf
├── outputs.tf
├── backend.tf
├── terraform.tfvars.example
├── bootstrap/
└── modules/
    ├── storage/
    ├── opensearch/
    ├── bedrock/
    └── lambda/

Each module owns one piece of the infrastructure and exposes only what other modules need through outputs. The root main.tf connects everything by passing outputs from one module as inputs to another.

Multiple modules might feel like overkill, but I learned early in my development journey that clear boundaries make systems easier to reason about. Each module owns one responsibility, which makes debugging much easier when something breaks.

The Lambda module does not need to know how OpenSearch is configured. It only receives the IDs and values it needs through variables.

Implementation

Step 1: Bootstrap Remote State First

Before running terraform apply on anything, you need somewhere to store your Terraform state.

If you're new to Terraform, state is how Terraform keeps track of the infrastructure it manages. Every resource it creates is recorded in a terraform.tfstatefile so future runs know what already exists.

Keeping state locally ties it to your machine. This project uses S3 remote state so Terraform has a durable, shared source of truth. With Terraform 1.10, native S3 state locking removes the need for a separate DynamoDB table.

The bootstrap/ directory provisions the resources needed for Terraform's remote backend and deployment permissions. Run the Terraform commands below once before deploying the main stack.

All commands use aws-vault, which stores AWS credentials in your OS keychain and injects temporary credentials at runtime. The --no-session flag skips STS session tokens, which some IAM operations reject. Ignore this if you're not using aws-vault

aws-vault exec YOUR_PROFILE --no-session -- \
  terraform -chdir=bootstrap init

aws-vault exec YOUR_PROFILE --no-session -- \
  terraform -chdir=bootstrap apply \
  -var="project_name=my-rag" -var="environment=dev"

Bootstrap creates two things: the S3 state bucket and a scoped deployer IAM policy.

The initial bootstrap requires AdministratorAccess because the deployer policy does not exist yet. Once bootstrap completes, attach the generated policy to your IAM user and remove AdministratorAccess. From that point forward, deployments run with least-privilege permissions.

The bootstrap step outputs the S3 backend configuration and deployer policy ARN. Copy the backend configuration into backend.tf, then initialize Terraform again to migrate state.

aws-vault exec YOUR_PROFILE --no-session -- terraform init

Then swap to the scoped policy (this can also be done through the AWS Console):

# Attach the deployer policy
aws-vault exec YOUR_PROFILE --no-session -- aws iam attach-user-policy \
  --user-name YOUR_IAM_USER \
  --policy-arn YOUR_DEPLOYER_POLICY_ARN

# Drop AdministratorAccess
aws-vault exec YOUR_PROFILE --no-session -- aws iam detach-user-policy \
  --user-name YOUR_IAM_USER \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Step 2: Configure Your Variables

cp terraform.tfvars.example terraform.tfvars

The module only requires a few inputs. The setting worth paying attention to is embedding_dimensions because it controls the size of the vectors stored in OpenSearch.

For this development environment, we use 512 dimensions. It reduces storage costs while keeping retrieval quality acceptable. We'll revisit this choice when looking at the Bedrock Knowledge Base configuration.

embedding_dimensions = 512  # 256 or 512, half the storage cost vs 1024

💡 First time using Bedrock?

Bedrock subscribes your account to a foundation model automatically the first time you invoke it, but only if the identity making that first call has the AWS Marketplace permissions (aws-marketplace:ViewSubscriptions and aws-marketplace:Subscribe) to finish the subscription. There is a setup window of up to 15 minutes where your calls may even succeed while the subscription is still being finalized in the background.

Here is the trap I fell into. My first invocation came from the scoped Lambda role, which does not carry those Marketplace permissions. Calls worked for a few minutes, then every request started failing with an access error once the subscription failed to complete. The fix is a one-time step: subscribe the model once using an identity that does have Marketplace permissions, such as your admin user, either by accepting the model agreement in the Bedrock console or by invoking the model once. The subscription is account-wide, so you do it once rather than per region. After that, the Lambda role needs only bedrock:InvokeModel.

One extra step for Anthropic models like Claude: before that first invocation you also have to submit a one-time First Time Use form describing your use case. You do it once per account, or once at the organization's management account, from the model catalog in the Bedrock console or with the PutUseCaseForModelAccess API. Access is granted as soon as the form goes through.

Step 3: Lambda The Query Handler and the Trigger

Lambda Module

Lambda connects the application workflow together. There are two functions with both running Node.js 20 on ARM64. ARM64 (Graviton) is cheaper to run than x86 for the same memory allocation, and since these functions are mostly waiting on AWS services rather than doing heavy computation, there was no reason to pay the x86 premium.

The ingestion handler (ingest.mjs) handles the document ingestion flow. It is triggered by S3 uploads and starts a Bedrock ingestion job so new documents are processed and indexed:

// lambda/src/ingest.mjs

export async function handler(event) {
  const command = new StartIngestionJobCommand({
    knowledgeBaseId: process.env.KNOWLEDGE_BASE_ID,
    dataSourceId: process.env.DATA_SOURCE_ID,
  });

  try {
    const response = await client.send(command);
    return { statusCode: 202, body: JSON.stringify({ jobId: response.ingestionJob.ingestionJobId }) };
  } catch (error) {
    if (error.name === "ConflictException") {
      // An ingestion job is already running. That's fine, the new file will be picked up.
      return { statusCode: 202, body: JSON.stringify({ message: "Ingestion already in progress" }) };
    }
    throw error;
  }
}

One subtle issue: Bedrock only allows one ingestion job at a time. If several files are uploaded together, multiple S3 events can trigger Lambda invocations, and subsequent requests receive a ConflictException.

We treat this as expected behavior and return 202, because the existing ingestion job will pick up the newly uploaded files.

The query handler (query.mjs) handles retrieval and generation. It accepts a natural language question and uses Bedrock Knowledge Bases to retrieve relevant context and generate an answer with citations:

// lambda/src/query.mjs

export async function handler(event, context) {
  // Parses event.body.query (API Gateway proxy format)
  const query = parseQuery(event);

  const command = new RetrieveAndGenerateCommand({
    input: { text: query.trim() },
    retrieveAndGenerateConfiguration: {
      type: "KNOWLEDGE_BASE",
      knowledgeBaseConfiguration: {
        knowledgeBaseId: KNOWLEDGE_BASE_ID,
        modelArn: MODEL_ARN,
      },
    },
  });

  // Retries once on throttle/5xx if there's > 15 seconds left in the timeout budget
  let response;
  try {
    response = await client.send(command);
  } catch (error) {
    if (isRetryable(error) && hasTimeForRetry(context)) {
      response = await client.send(command);
    } else {
      return buildResponse(503, { error: "Service temporarily unavailable" });
    }
  }

  return buildResponse(200, {
    answer: response.output?.text || "",
    citations,
  });
}

For observability, logging follows a strict rule: structured JSON, no PII. We log request IDs, durations, and error types, but never store the actual user query or generated response.

function log(level, requestId, message) {
  console.log(JSON.stringify({
    level,
    timestamp: new Date().toISOString(),
    requestId: requestId || undefined,
    message,
  }));
}

Security: Lambda dependencies are locked in package-lock.json. No floating version ranges means deployments are reproducible and less likely to unexpectedly pull in a compromised dependency.

Step 4: Deploy

The Bedrock and OpenSearch modules have a circular dependency. The Bedrock Knowledge Base requires the AOSS collection endpoint, while the OpenSearch data access policy requires the Bedrock KB role ARN. On a fresh deployment, Terraform cannot resolve both resources at the same time.

The solution is to break the cycle by hand: create the collection first, tell the rest of the stack where it lives, then apply everything else.

# Create just the AOSS collection
aws-vault exec YOUR_PROFILE --no-session -- terraform apply -target=module.opensearch

Once it exists, grab its endpoint and paste it into terraform.tfvars so Bedrock knows where to point:

opensearch_collection_endpoint = "https://your-collection-id.us-east-1.aoss.amazonaws.com"

Now deploy the rest. This first full run also pulls in the OpenSearch Terraform provider, so it needs an init in front of it:

aws-vault exec YOUR_PROFILE --no-session -- terraform init
aws-vault exec YOUR_PROFILE --no-session -- terraform apply

The first AOSS deployment takes a while (~10 minutes). After that the endpoint is captured in your state and tfvars, so day-to-day changes are just terraform apply. The README has the exact commands if you want to follow along line by line.

One thing worth knowing if you're using aws-vault: some IAM operations reject the session tokens it generates by default. If you encounter an InvalidClientTokenId error during deployment, retry with --no-session.

The README contains the complete deployment reference.

Step 5: Test It

Upload, check status, and invoke

Upload a document. The ingestion Lambda triggers automatically through the S3 event:

aws-vault exec YOUR_PROFILE --no-session -- \
  aws s3 cp ./my-document.pdf \
  s3://$(terraform output -raw document_bucket_name)/

Before querying, verify that Bedrock has finished processing the document:

aws-vault exec YOUR_PROFILE --no-session -- \
  aws bedrock-agent list-ingestion-jobs \
  --knowledge-base-id $(terraform output -raw knowledge_base_id) \
  --data-source-id $(terraform output -raw data_source_id) \
  --region us-east-1

Once the ingestion job is COMPLETE, invoke the query Lambda:

aws-vault exec YOUR_PROFILE --no-session -- \
  aws lambda invoke \
  --function-name $(terraform output -raw query_function_name) \
  --payload '{"body":"{\"query\":\"What is the document about?\"}","requestContext":{"requestId":"test-1"},"headers":{}}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/response.json &amp;&amp; cat /tmp/response.json

The payload uses the API Gateway format expected by the Lambda handler, which is why the query is nested inside body.

A successful response looks like:

{
  "statusCode": 200,
  "headers": { "Content-Type": "application/json" },
  "body": "{\"answer\":\"The document covers...\",\"citations\":[...]}"
}

If you get an empty answer, check the ingestion status first. The Knowledge Base cannot retrieve documents until the ingestion job completes.

Step 6: Cleanup

Before running terraform destroy, there is one thing to update. The Bedrock data source uses a data_deletion_policy that defaults to DELETE.

During teardown, Bedrock attempts to remove vectors from OpenSearch as part of deleting the data source. If the AOSS collection is destroyed in the same operation, Bedrock can no longer reach it and the deletion can get stuck.

Set the policy to RETAIN first, apply the change, then destroy:

# modules/bedrock/main.tf

resource "aws_bedrockagent_data_source" "s3" {
  name                 = "${var.config.environment}-${var.config.project_name}-s3-source"
  knowledge_base_id    = aws_bedrockagent_knowledge_base.main.id
  data_deletion_policy = "RETAIN"

  # ... rest of config
}

Apply the change:

aws-vault exec YOUR_PROFILE --no-session -- terraform apply

Full cleanup: tear down the stack

aws-vault exec YOUR_PROFILE --no-session -- terraform destroy

To also remove the bootstrap resources, first empty the versioned Terraform state bucket, then destroy the bootstrap stack:

# Remove all object versions from the state bucket
aws-vault exec YOUR_PROFILE --no-session -- \
  aws s3api delete-objects \
  --bucket YOUR_STATE_BUCKET_NAME \
  --delete "$(aws-vault exec YOUR_PROFILE --no-session -- \
    aws s3api list-object-versions \
    --bucket YOUR_STATE_BUCKET_NAME \
    --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' \
    --output json)"

# Destroy bootstrap
aws-vault exec YOUR_PROFILE --no-session -- \
  terraform -chdir=bootstrap destroy

Gotchas and Cost

AOSS is the biggest cost driver in this stack. Even the minimum capacity configuration can add hundreds of dollars per month for a continuously running collection. If you are experimenting, destroy the stack when you are done.

Common issues:

AOSS collection won't create

You likely hit the policies-first timing issue. Make sure encryption, network, and access policies are included in depends_on.

KB ingestion fails

Verify that the KB role has s3:GetObject access to the bucket and aoss:APIAccessAll access to the collection. Also confirm the AOSS collection is in an ACTIVE state before starting ingestion.

Lambda returns 400 (`query is required`)

The query Lambda expects event.body.query. Make sure the payload wraps the query inside a body field as shown in Step 5.

Lambda returns 503

The function collapses any Bedrock failure into a generic 503, so the real reason is in CloudWatch, not the response body. Two things cause it in practice.

The boring one is throttling. The function retries once automatically, and sustained traffic may need backoff or a limit increase.

The one that actually cost me time is IAM. This project generates with Claude Sonnet 4.5, which you reach through a cross-region inference profile rather than a plain model ID. An inference profile is really a router: it forwards your request to the underlying model in whichever US region has capacity. So IAM checks you twice, once on the profile and again on the foundation model it lands on. Grant bedrock:InvokeModel on only the profile and the call fails with an AccessDeniedException on GetInferenceProfile, which the Lambda dutifully turns into a 503. The role needs InvokeModel and GetInferenceProfile on the profile, plus InvokeModel on the foundation model in each region the profile can route to (us-east-1, us-east-2, us-west-2). The Terraform in the repo already wires this up, but it is worth knowing why all those grants are there.

Empty citations array

The document format may not be supported well by Bedrock chunking. Plain text, PDF, Markdown, and HTML generally work best.

`terraform destroy` gets stuck

The default data_deletion_policy = "DELETE" can cause teardown issues when the AOSS collection is destroyed at the same time. Set it to RETAIN, apply, and then destroy.

What's Next?

The Knowledge Base is live and queryable from the CLI. In Part 3, we're putting an API Gateway in front of the query Lambda and wiring up a React frontend. The query Lambda's response shape already works with API Gateway's proxy integration, and the only thing left is CORS headers and the API Gateway resource itself.

Disclaimer

This is strictly for educational purposes. You will be charged for the resources created when you follow along. Remember to clean up after use.