DEV Community: Tidding Ramsey

Stop Guessing Your Node Groups: A Deep Dive Into Karpenter on EKS

Tidding Ramsey — Tue, 15 Sep 2026 02:46:49 +0000

If you have run a production EKS cluster for more than a few months, you already know the ritual. You size your node groups based on a rough estimate of peak load. You pad that estimate, because nobody wants a 3 a.m. page about pending pods. Then you watch the bill creep upward while kubectl top nodes quietly tells you that half your fleet is idle.

That is not a scheduling problem. It is a forecasting problem, and Kubernetes was never designed to solve forecasting on its own.

This post walks through why the traditional model hits a ceiling, what Karpenter actually does differently at the API level, and the specific things that bite teams in production. By the end you should be able to reason about a NodePool config rather than copy one.

Part 1: Understanding the ceiling you are hitting

Before Karpenter, autoscaling on EKS almost always meant Cluster Autoscaler plus Auto Scaling Groups. The loop looked like this:

You define managed node groups, each backed by an ASG tied to a specific instance type or a narrow family.
Cluster Autoscaler watches the API server for pods in Pending with a FailedScheduling event.
It simulates: "if I added one node from node group X, would this pod schedule?"
If yes, it increases that ASG's desired capacity by one.
The ASG launches an instance, the kubelet registers, the scheduler binds the pod.

Step 3 is where the whole design shows its age. Cluster Autoscaler can only simulate against node groups that already exist. It cannot invent a node shape. If your node group is built on m5.xlarge (4 vCPU, 16 GiB) and a burst of pods each request 2 vCPU and 14 GiB, every node you add fits exactly one pod and strands 2 vCPU. Cluster Autoscaler will happily do this all day, because from its perspective the simulation succeeded.

There is a second, subtler cost. Cluster Autoscaler requires every node in a group to be homogeneous for its simulation to be valid. If you mix instance types inside one ASG, the autoscaler's estimate of "what a node from this group provides" becomes wrong, and it either over-provisions or refuses to scale. So you split groups. One per instance family. Then one per capacity type, because Spot and On-Demand need different labels and taints. Then one per AZ, because you need zonal balance for topology spread constraints.

Three families times two capacity types times three AZs is eighteen node groups, each with its own launch template, its own AMI version, and its own upgrade cycle. That matrix goes stale the moment your workload mix changes, and it is maintained by a human who has other things to do.

The core insight: the node group model forces you to make instance decisions at config-authoring time, when you know the least. The demand signal arrives at scheduling time, when you know the most. Everything painful about the old model flows from that gap.

Part 2: What Karpenter actually does

Karpenter deletes the node group as the unit of scaling decisions. Instead of "which existing group should I grow," it asks: "given these specific pending pods, what is the best possible instance to launch right now?"

Here is the real loop:

1. Watch for unschedulable pods. Karpenter watches for pods the kube-scheduler could not place. Same trigger as Cluster Autoscaler so far.

2. Batch them. This part matters and is frequently missed. Karpenter does not react to one pod at a time. It waits a short batching window (starting at 1 second, extending up to 10 seconds as more pods arrive) and collects everything pending. This is why a Deployment scaled from 1 to 50 replicas produces a sensible handful of large nodes rather than 50 separate provisioning decisions.

3. Bin-pack in memory. Karpenter takes that batch and solves a packing problem. It reads each pod's resource requests, node selectors, node affinity, taints and tolerations, topology spread constraints, and pod affinity rules, then computes which combination of instance types could host them. Crucially, it evaluates many instance types at once, not one.

4. Call EC2 Fleet. Karpenter passes a ranked list of viable instance types to the EC2 Fleet API with a lowest-price allocation strategy. EC2 picks the cheapest one that has capacity. This is why Karpenter handles Spot capacity crunches gracefully: if c6g.4xlarge Spot is exhausted in us-east-1a, Fleet just falls through to the next candidate instead of failing the whole request.

5. Bind pods directly. Karpenter pre-binds the pending pods to the node it is about to create, before the node even registers. This skips a scheduler round trip and cuts time-to-ready noticeably.

The thing to internalize: Karpenter makes the instance decision at scheduling time, using real demand. That single change is what collapses the eighteen-node-group matrix into one NodePool.

Part 3: Consolidation, where the money actually is

Provisioning gets the headlines. Consolidation is where the savings come from.

Karpenter continuously re-evaluates whether the current fleet could be replaced by a cheaper fleet that still satisfies every running pod. It considers three moves, in order of preference:

Deletion. Can this node's pods all fit on other existing nodes? If yes, cordon it, drain it, terminate it. Pure win, no replacement cost.

Replacement. Can this node be replaced by a single cheaper node that still fits everything? A m5.4xlarge running workloads that only need 8 vCPU becomes a m5.2xlarge. Karpenter launches the replacement first, waits for it to be ready, then drains the original.

Multi-node consolidation. Can several underutilized nodes be replaced by a smaller number of nodes? This is the expensive computation and Karpenter rate-limits how aggressively it attempts it.

Teams migrating from Cluster Autoscaler with consolidation enabled commonly see node-hour costs fall 20 to 40 percent. Almost none of that comes from smarter provisioning. It comes from deleting the padding that manual node group sizing always requires.

Part 4: Reading a NodePool properly

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    metadata:
      labels:
        team: platform
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: "10%"
      - nodes: "0"
        schedule: "0 9 * * mon-fri"
        duration: 8h
  limits:
    cpu: "1000"
    memory: 4000Gi
  weight: 10

Line by line, what each piece is buying you:

requirements are constraints, not selections. You are describing the space of acceptable instances and letting Karpenter pick from inside it. instance-generation: Gt 5 is quietly one of the highest-value lines here: it excludes older generations that are often more expensive per unit of performance, without you naming a single instance type.

Including both amd64 and arm64 lets Graviton into the candidate pool. If your images are multi-arch, this is frequently a 15 to 20 percent saving for zero configuration effort. If they are not multi-arch, remove arm64 or you will get nodes your pods cannot run on.

expireAfter: 720h forces every node to be replaced after 30 days. This is your patching story. Nodes get recycled onto fresh AMIs continuously instead of accumulating drift until someone runs a scary cluster-wide upgrade.

disruption.budgets is the part production teams should not skip. The first entry caps voluntary disruption at 10 percent of nodes at a time. The second pins it to zero during weekday business hours. Karpenter will still handle involuntary events like Spot interruptions, but it will not choose to churn your fleet during your peak traffic window.

limits is your blast radius. Without it, a runaway Deployment with a bad replica count can provision a genuinely alarming amount of compute before anyone notices. Set it.

weight matters once you have more than one NodePool. Higher weight is evaluated first. A common pattern is a high-weight Spot-only pool for stateless work and a lower-weight On-Demand pool as fallback.

Part 5: Where teams get burned

Spot interruption handling is not application resilience. Karpenter subscribes to the EC2 interruption notice via SQS and starts draining within the two-minute window. That is genuinely good. But if your pod takes ninety seconds to warm a cache before it serves useful traffic, you have an application design problem that no scheduler fixes. Test this. Actually terminate a Spot node in staging and watch what your p99 does.

Requests, not limits, drive everything. Karpenter bin-packs against resources.requests. If your teams set requests at 100m CPU "to be safe" while the app actually uses 1.5 cores, Karpenter will pack twelve of them onto a node and you will get CPU throttling that looks like a network problem for three days. Karpenter makes the cost of bad resource requests visible in a way Cluster Autoscaler's padding used to hide.

Instance type sprawl hurts observability. Wide-open requirements mean your cost dashboards fill with dozens of instance types and your capacity trends get noisy. Narrowing instance-category and instance-generation trades a small amount of theoretical savings for a lot of operational clarity. Most mature setups do this deliberately.

Underutilized does not mean empty. WhenEmptyOrUnderutilized will move running pods. If you have workloads that genuinely hate rescheduling, either use WhenEmpty, raise consolidateAfter, or define proper PodDisruptionBudgets. Karpenter respects PDBs, but only ones that exist.

Do not run Karpenter on nodes Karpenter manages. The controller needs somewhere stable to live: a small managed node group, or Fargate. If Karpenter consolidates the node it is running on, you get a genuinely annoying failure mode.

The bigger shift

Karpenter is a signal of where infrastructure is heading generally: away from static pre-provisioned capacity, toward just-in-time capacity shaped by real demand signals. The same philosophy shows up in Fargate, in Lambda's scale-to-zero model, and in how AWS increasingly prices compute.

If you are still maintaining a spreadsheet of node groups by hand, the fix is not a better spreadsheet. It is letting the scheduler tell the infrastructure layer what it actually needs, and trusting the infrastructure layer to answer honestly.

That is the real lesson underneath the tool, and it outlives the tool.

If you have migrated a cluster from Cluster Autoscaler to Karpenter, I would like to hear what surprised you. The gap between the docs and the first month in production is where the interesting material lives.

Building a Bulletproof Idempotent Transaction Pipeline on AWS with CDK

Tidding Ramsey — Thu, 30 Jul 2026 14:44:47 +0000

Have you ever clicked "Send Money" on an app, hit a network lag spike, panicked, and tapped the button a second time then immediately wondered if you just sent your rent money twice?

As developers, preventing that exact scenario is one of our biggest responsibilities. In distributed systems, this is called idempotency: guaranteeing that no matter how many times a request is duplicated, the side effect (a database transaction, in this case) happens exactly once.

In this article, I'll walk through how I built a bulletproof, serverless, multi-layered idempotency pipeline on AWS using the Cloud Development Kit (CDK) and TypeScript — and the type-casting bug that almost took me down along the way.

🔗 Source code: [https://github.com/McTech6/aws-serverless-idempotency.git]

The Architecture

To guarantee strict idempotency, you can't rely on a single database lock. You need a system that gracefully drops duplicate requests at the edge, long before they ever reach your core ledger.

Here's the flow:

Client sends a POST request to API Gateway with a unique x-idempotency-key header.
An Issuer Lambda instantly accepts the request, pushes it onto an SQS FIFO queue, and returns 202 Accepted so the client isn't left hanging.
A Worker Lambda pulls messages off the queue and checks a DynamoDB lock table to see if that key has already been processed.
If it's a new key, the worker executes an ACID-compliant double-entry transaction (debit/credit) against Aurora Serverless v2 PostgreSQL.
Finally, the worker writes the idempotency key to DynamoDB with a 7-day TTL, closing the loop.

High-level view: API Gateway → Issuer Lambda → SQS FIFO → Worker Lambda → DynamoDB lock table + Aurora ledger

Three layers of defense, each catching what the layer before it might miss:

SQS FIFO catches rapid-fire duplicates within its 5-minute dedup window.
DynamoDB catches duplicates for up to 7 days after that.
Aurora enforces the actual financial invariants (no negative balances, atomic debit/credit) no matter what gets through.

Step 1: The Idempotency Lock Table (DynamoDB)

First, a fast, highly-available datastore to remember every transaction processed recently. DynamoDB is a natural fit here.

I created a simple PAY_PER_REQUEST table using idempotencyKey as the partition key, with TTL enabled on an expiresAt attribute so the table cleans itself up automatically after 7 days — keeping storage costs near zero.

const idempotencyTable = new dynamodb.Table(this, 'IdempotencyTable', {
  partitionKey: { name: 'idempotencyKey', type: dynamodb.AttributeType.STRING },
  billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
  timeToLiveAttribute: 'expiresAt',
});

The IdempotencyTable, deployed via CDK, with idempotencyKey as the partition key and TTL enabled.

Step 2: The Deduplication Buffer (SQS FIFO)

Why not write straight to the database? Because databases get overwhelmed during traffic spikes. SQS acts as a shock absorber between the edge and the ledger.

I chose an SQS FIFO queue over a standard queue for two reasons:

Strict ordering — using the sender's account ID as the MessageGroupId guarantees that if a user fires off two payments back-to-back, they're processed in the exact order they were sent.
Edge deduplication — SQS FIFO has a built-in 5-minute dedup window keyed on MessageDeduplicationId. If a client double-taps the same idempotency key within 5 minutes, SQS silently drops the second message before it ever triggers a Lambda.

const txnQueue = new sqs.Queue(this, 'TxnQueue', {
  fifo: true,
  contentBasedDeduplication: false,
  deadLetterQueue: { queue: dlq, maxReceiveCount: 3 },
});

Creating the FIFO queue with a dead-letter queue attached for messages that fail processing 3 times.

Sending a test message through the queue to confirm FIFO ordering and dedup behavior before wiring up the Lambda.

Step 3: The Edge Receiver (API Gateway + Issuer Lambda)

API Gateway routes POST /transactions to the Issuer Lambda.

The Issuer Lambda is intentionally thin. Its only job is to validate the payload, pull the x-idempotency-key out of the headers, pack everything into a JSON message, and drop it onto SQS. Returning 202 Accepted immediately keeps the client-facing app feeling fast, even though the real work hasn't happened yet.

The Issuer Lambda wired to API Gateway's POST route, ready to accept and forward transaction requests.

Step 4: The Core Ledger (Aurora Serverless)

For a financial ledger, NoSQL doesn't cut it. This needs strict relational constraints (like CHECK (amount > 0)) and ACID transactions to guarantee debit and credit rows are committed together or not at all.

I deployed an Aurora PostgreSQL Serverless v2 cluster with the RDS Data API enabled. This let the Worker Lambda execute SQL over secure HTTP without managing VPC connection pools directly.

const dbCluster = new rds.DatabaseCluster(this, 'LedgerCluster', {
  engine: rds.DatabaseClusterEngine.auroraPostgres({
    version: rds.AuroraPostgresEngineVersion.of('16.8', '16'),
  }),
  serverlessV2MinCapacity: 0.5,
  serverlessV2MaxCapacity: 1,
  enableDataApi: true,
  defaultDatabaseName: 'ledger',
  vpc,
});

Provisioning the Aurora Serverless v2 cluster via cdk deploy, with the Data API enabled for HTTP-based queries.

Step 5: The Worker Logic

The Worker Lambda, triggered by SQS, is where everything comes together:

Check DynamoDB — a GetItemCommand looks up the idempotencyKey. If it already exists, exit immediately. This is what protects the system for the full 7 days, long after SQS's 5-minute dedup window has expired.
Execute the DB transaction — via the RDS Data API: BeginTransaction, insert the debit row, insert the credit row, CommitTransaction.
Lock it in — write the idempotencyKey to DynamoDB so no future duplicate can slip through.

The Worker Lambda's DynamoDB check in action — this is the layer that catches duplicates after the SQS dedup window has passed.

A Hard-Learned Lesson (The Gotcha!)

While building this, I hit a strict type-casting error with the Aurora Data API.

When you pass a UUID parameter as a plain string (stringValue) through the Data API, PostgreSQL rejects it if the target column type is uuid. The Data API doesn't automatically coerce string parameters into uuid the way a native pg client connection might.

The fix: explicitly cast the parameter in the SQL string itself using ::uuid.

-- This will fail via Data API:
VALUES (:txId, :accId, :entryType, :amt, :curr)

-- This works beautifully:
VALUES (:txId::uuid, :accId, :entryType, :amt, :curr)

Small fix, but it's the kind of thing that can cost you an hour of staring at a cryptic Data API error message if you don't know to look for it.

The Ultimate Test

To prove it worked, I sent a request with x-idempotency-key: my-unique-key-002. The database wrote 2 rows — one debit, one credit, exactly as expected.

First request with my-unique-key-002 — API Gateway returns 202, and Aurora ends up with a matching debit/credit pair.

I then immediately sent the exact same request again.

Same idempotency key, sent a second time. API Gateway still returns 202 — but nothing new gets written.

API Gateway happily returned another 202 Accepted, so the client had no idea anything was different. But when I queried the database? Still only 2 rows.

End-to-end test confirming the duplicate was silently swallowed — no double debit, no double credit.

The backend silently and gracefully swallowed the duplicate. Idempotency achieved.

Conclusion

Building idempotency into a serverless architecture requires a shift in mindset: you can't rely on a single point of validation. By layering SQS FIFO deduplication (for short-term spam) with DynamoDB locks (for long-term safety), and by decoupling the edge API from the core Aurora ledger, you get a system that scales without ever double-charging a user.

If you want to see the full CDK stack and Lambda code, check out the repo here: [https://github.com/McTech6/aws-serverless-idempotency.git]

Have you tackled idempotency in your own projects? I'd love to hear your favorite patterns in the comments.

I Used Amazon Bedrock as My AI Coding Partner for a Day Here's What Happened

Tidding Ramsey — Wed, 27 May 2026 11:31:12 +0000

How generative AI on AWS helped me summarize feedback, squash bugs, and write better Python without leaving the console.

Introduction

I recently completed a hands-on lab using Amazon Bedrock, AWS's managed generative AI service, and I came away genuinely impressed. Not just by what the AI could do, but by how quickly it slotted into real development workflows.

In this article, I'll walk you through what I learned: from touring the Bedrock console to using it as a live coding assistant. Whether you're an AWS veteran or just curious about practical AI tooling, I think there's something here for you.

What Is Amazon Bedrock?

Amazon Bedrock is a fully managed AWS service that gives you on-demand access to large language models (LLMs) without having to provision servers, manage infrastructure, or train models from scratch.

It comes in two flavors:

Serverless models : Fully managed foundation models from AWS and partner AI companies (like Anthropic, Meta, Mistral, and others)
Marketplace models : Over 100 specialized models deployed on managed Amazon SageMaker endpoints

One thing that stood out to me: Bedrock isn't just a chat interface. It's a full AI application platform with tools for:

Agents : Automate tasks by connecting LLMs to APIs and data sources

Flows : Chain Bedrock tools and AWS services into end-to-end AI pipelines

Knowledge Bases : Upload your own docs and build Q&A bots grounded in real content

Prompt Management : Version, test, and reuse prompts across multiple applications

For this lab, I focused on the Chat / Text playground using the Amazon Nova Micro model.

Part 1: Summarizing Messy, Unstructured Feedback

The first task felt immediately practical: summarize a wall of rambling customer feedback and extract actionable improvements.

The prompt structure I used was simple but effective:

Summarize the following feedback and produce action points for fixes and improvements:

Separating the instruction from the data (with a blank line) made the prompt cleaner both for me to read and, arguably, for the model to parse.

The feedback itself was a fictional but realistic stream-of-consciousness review of a car parts app. Full of metaphors, colloquialisms, and run-on sentences. The kind of thing you'd actually get from a user interview.

What the model returned: A clean, structured list of positives and improvement suggestions search UX, cart flow, missing features like a compatibility wizard and saved parts lists, shipping cost transparency, and live support access.

Then I pushed further:

List the top three improvements that would likely have the biggest impact on customer satisfaction.

No need to repeat the context. The model remembered the conversation and narrowed the list down intelligently.

Key takeaway: Bedrock excels at distilling unstructured human language into structured, actionable output. This is genuinely useful for product teams doing requirements gathering or user research synthesis.

Part 2: Using AI as a Coding Assistant

This is where things got interesting for me as a developer.

Fixing a ZeroDivisionError

I started with a simple Python function:

def divide(x, y):
    return x / y

Calling divide(10, 0) throws a ZeroDivisionError. I asked Bedrock:

Add exception handling for dividing by zero to this Python3 function:

def divide(x, y):
    return x / y

The model returned a version using try/except blocks, specifically catching ZeroDivisionError and returning a useful message instead of crashing. Clean, idiomatic Python.

A small but important habit I developed: always specify the language version. Python 2 and Python 3 differ significantly. The more context you give the model, the more relevant the output.

Improving the Fibonacci Algorithm

Next, I worked with a classic recursive Fibonacci implementation:

def fibonacci(n):
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

This works but it's slow. The time complexity is O(2^n), which means for large values of n, it becomes unusably slow very quickly.

I asked Bedrock to evaluate the performance and suggest improvements. The model came back with two solid suggestions:

Memoization : Cache already-computed values so they're not recalculated
Iterative approach : Replace recursion with a loop, dropping complexity to O(n)

I then asked Bedrock to directly compare the time complexity of both approaches. It explained the difference clearly — the recursive version re-computes the same values exponentially, while the iterative version computes each value exactly once.

Key takeaway: Bedrock is genuinely useful for algorithm analysis. Asking "can this be improved?" and "compare these two implementations" are high-value prompts for any developer trying to write more performant code.

Part 3: Understanding and Testing Unfamiliar Code

The final section was perhaps the most practically useful for day-to-day work: using AI to understand code you didn't write.

Take this function:

def middle(arr):
    while len(arr) > 1:
        del arr[len(arr) // 2]
    return arr[0]

At a glance, it's not obvious what this does. I asked Bedrock to describe it:

Describe how the following Python 3 code works:

def middle(arr):
    ...

The model walked through it step-by-step. It also flagged some issues — notably that the function mutates the original list (a side effect most callers won't expect), and that there's no error handling.

From there, I used an improved version of the function and asked Bedrock to generate a unit test:

Generate a unit test for the following Python function using Python's built-in unittest module.
The test class should have one test that tests that None is returned if the argument is None.

def find_middle_element(arr):
    ...

The model produced a proper unittest.TestCase class, which I dropped into a file in VS Code and ran immediately. It passed.

Bonus: Generating Test Data

One underrated use case generating fake data for testing:

Generate some test data for users that includes name, address, phone number, 
and widget order history. Display the test data in the JSON data format.

Within seconds, I had realistic-looking JSON I could plug straight into tests or seed scripts. The model can also format this as YAML or TOML if you prefer.

Bedrock Pricing: What You Should Know

Bedrock prices by tokens chunks of text (roughly words or word fragments) processed as input and output.

Two pricing models are available:

Model	Best for
On-Demand	Light, infrequent, or unpredictable usage
Provisioned Throughput	Predictable, high-volume, or production use

The chat playground also shows you latency and token counts in real time useful for estimating costs before committing to a production integration.

Model Customization (If You Need It)

Out of the box, foundation models are general-purpose. But Bedrock also supports three customization approaches:

Fine-tuning : Adjust tone, verbosity, or vocabulary using labeled examples
Distillation : Transfer knowledge from a large model to a smaller, faster one (up to 500% faster, 75% cheaper)
Pre-training : Expose a model to your domain-specific data corpus

For most developer use cases, you won't need customization — the base models are powerful and flexible. But it's good to know the option exists.

Things to Watch Out For

A few honest caveats from working with Bedrock:

Hallucinations are real. LLMs sometimes generate confident-sounding output that's just wrong. Always review generated code before running it.

Responses aren't deterministic. The same prompt can yield different outputs each time. Don't expect bit-for-bit reproducibility.

Context windows are finite. Each model has a maximum context length. For very long codebases or documents, you may need to chunk your input across multiple prompts.

Prompt engineering matters. Adding context (e.g., "You are an expert AWS cloud engineer") meaningfully improves response quality. Being specific about language versions, libraries, and constraints helps too.

Final Thoughts

Amazon Bedrock made it easy to slot AI into workflows I already use — reviewing code, writing tests, making sense of user feedback. Nothing about it felt like magic. It felt like a well-calibrated tool that rewards thoughtful prompting.

If you're on AWS and haven't explored Bedrock yet, the Chat / Text playground is a zero-friction starting point. Pick a model, type a prompt, see what happens. The learning curve is low. The upside is real.

Have you used Amazon Bedrock or another LLM in your development workflow? I'd love to hear what worked (and what didn't) drop a comment below.

Building My First AI Agent with Strands SDK and Amazon Bedrock Errors, Fixes & Lessons Learned

Tidding Ramsey — Sat, 09 May 2026 21:24:26 +0000

Introduction

I recently attended an AWS event where we built our first AI agent using the Strands Agents SDK and Amazon Bedrock. The quickstart guide looked simple enough — a few lines of Python, some tools, and a running agent. But the real learning happened in the errors. This article walks you through what I built, every error I hit, and exactly how I fixed them.

What We Built

A simple AI agent that can:

Tell you the current time
Perform calculations
Count letters in a word

Three tools. One agent. Sounds easy. It wasn't — but that's what made it worth writing about.

Project Structure

Here's the folder structure I used:

Agent/
├── .venv/
├── agent.py
└── requirements.txt

Setting Up the Environment

First, create and activate a virtual environment:

python -m venv .venv
.venv\Scripts\Activate.ps1   # Windows PowerShell

Then install the required packages:

pip install strands-agents strands-agents-tools

The Agent Code (Starting Point)

The quickstart gave us this agent.py:

from strands import Agent, tool
from strands_tools import calculator, current_time

@tool
def letter_counter(word: str, letter: str) -> int:
    """
    Count occurrences of a specific letter in a word.

    Args:
        word (str): The input word to search in
        letter (str): The specific letter to count

    Returns:
        int: The number of occurrences of the letter in the word
    """
    if not isinstance(word, str) or not isinstance(letter, str):
        return 0
    if len(letter) != 1:
        raise ValueError("The 'letter' parameter must be a single character")
    return word.lower().count(letter.lower())

agent = Agent(tools=[calculator, current_time, letter_counter])

message = """
I have 3 requests:
1. What is the time right now?
2. Calculate 3111696 / 74088
3. Tell me how many letter R's are in the word "strawberry" 🍓
"""

agent(message)

Simple and clean. Then I ran it. And the errors began.

Error 1: Anthropic Use Case Form Not Submitted

botocore.errorfactory.ResourceNotFoundException: 
Model use case details have not been submitted for this account.
Fill out the Anthropic use case details form before using the model.
└ Model id: global.anthropic.claude-sonnet-4-6

What happened: The Strands SDK defaults to Amazon Bedrock with Claude Sonnet 4. But Anthropic requires first-time users to submit a use case form before accessing their models on Bedrock.

How I fixed it:

Went to the AWS Bedrock Console in us-east-1
Navigated to Model Catalog and searched for Claude Sonnet 4.6
Clicked the model and hit "Submit use case details"
Filled out the form:
- Company name, website, industry
- Intended users: Internal
- Use case: "Building an AI agent for learning and demonstration purposes at an AWS event using the Strands SDK and Amazon Bedrock"
Submitted and waited ~15 minutes

Once the "Submit use case details" button disappeared and "Open in playground" appeared, I knew it was approved.

Error 2: Missing Dependency for AWS Login

botocore.exceptions.MissingDependencyException: 
Using the login credential provider requires an additional dependency.
You will need to pip install "botocore[crt]" before proceeding.

What happened: The AWS credential provider I was using needed an extra C extension package called awscrt.

How I fixed it:

pip install "botocore[crt]"

That installed awscrt-0.32.2 and resolved the issue immediately.

Error 3: Wrong Model ID — ValidationException

botocore.errorfactory.ValidationException: 
The provided model identifier is invalid.
└ Model id: us.anthropic.claude-sonnet-4-6-20251031-v1:0

What happened: I had added a BedrockModel to my code with a guessed model ID, but it wasn't valid for my account.

How I fixed it: I ran this command to list all valid inference profile IDs available to my account:

aws bedrock list-inference-profiles --region us-east-1 --profile tidding --query "inferenceProfileSummaries[?contains(inferenceProfileId, 'anthropic')].inferenceProfileId"

It returned a list including:

"us.anthropic.claude-sonnet-4-6"

That was the correct ID. No version suffix needed.

The Final Working Code

After all the fixes, here is the final agent.py:

from strands import Agent, tool
from strands.models import BedrockModel
from strands_tools import calculator, current_time

# Define a custom tool using the @tool decorator
@tool
def letter_counter(word: str, letter: str) -> int:
    """
    Count occurrences of a specific letter in a word.

    Args:
        word (str): The input word to search in
        letter (str): The specific letter to count

    Returns:
        int: The number of occurrences of the letter in the word
    """
    if not isinstance(word, str) or not isinstance(letter, str):
        return 0
    if len(letter) != 1:
        raise ValueError("The 'letter' parameter must be a single character")
    return word.lower().count(letter.lower())

# Specify the Bedrock model explicitly using the correct inference profile ID
model = BedrockModel(
    model_id="us.anthropic.claude-sonnet-4-6",
    region_name="us-east-1"
)

# Create the agent with tools
agent = Agent(model=model, tools=[calculator, current_time, letter_counter])

# Ask the agent to handle multiple tasks
message = """
I have 3 requests:
1. What is the time right now?
2. Calculate 3111696 / 74088
3. Tell me how many letter R's are in the word "strawberry" 🍓
"""

agent(message)

Run it with:

$env:AWS_PROFILE = "tidding"
python agent.py

Key Lessons Learned

1. Always submit the Anthropic use case form first. It's a one-time requirement per AWS account and blocks everything until done.

2. Don't guess model IDs. Use aws bedrock list-inference-profiles to get the exact valid ID for your account.

3. The default Strands model ID may not match what your account supports. Always specify the model explicitly using BedrockModel.

4. botocore[crt] is required when using the AWS login credential provider on Windows. Install it early.

5. Set your AWS profile before running the agent:

$env:AWS_PROFILE = "yourprofile"

What the Agent Loop Looks Like

Once running, the Strands agent follows this loop:

Input → Reasoning (LLM) → Tool Selection → Tool Execution → Back to Reasoning → Response

The agent automatically decides which tools to use based on your message. For our three requests, it fired current_time, calculator, and letter_counter — all in one go.

Conclusion

The Strands SDK makes building AI agents genuinely simple — once you get past the AWS setup hurdles. The errors I faced were all configuration-related, not code-related. Once the environment was right, the agent worked beautifully in just a few lines of Python.

If you're attending an AWS event and hitting these same errors, I hope this saves you time. Drop a comment if you're stuck — happy to help!

Built at an AWS Event · Strands Agents SDK · Amazon Bedrock · Claude Sonnet 4.6

How I Evaluated an AI Model on AWS Without Writing a Single Line of Training Code

Tidding Ramsey — Sat, 09 May 2026 13:26:57 +0000

A step-by-step guide to Amazon Bedrock's model evaluation feature from S3 setup to reading real results

Ever wondered whether the AI model you're about to plug into your production system actually knows what it's doing? Me too. That's exactly what Amazon Bedrock's model evaluation feature is built for and after running through it myself, I'm genuinely impressed at how accessible it is.

No PhD. No GPU clusters. No tears. Just AWS, an S3 bucket, and a few JSON prompts.

Let's walk through the whole thing start to finish.

What Even Is Amazon Bedrock?

Amazon Bedrock is AWS's managed Generative AI service. Instead of spending months training, hosting, and scaling foundation models yourself, Bedrock lets you call them like an API. Think of it as the "serverless" moment for AI — the infrastructure complexity disappears and you focus on what actually matters: using the models.

One of its best-kept features is model evaluation a way to run a model against a set of prompts, compare its responses to expected answers, and get a performance score back. It's perfect for building confidence before you commit a model to your workflow.

Here's what we're going to build today:

Prompt Dataset (S3) → Bedrock Evaluation Job → Results (S3) → Insights
The before architecture

The after architecture

Step 1: Log In and Orient Yourself

Head to the AWS Management Console and sign in. Make sure you're in the US West (Oregon) / us-west-2 region model evaluation support varies by region and you want to be where the models live.

Once you're in, you'll use two core services today:

Amazon S3 — to store your prompt dataset and receive evaluation results
Amazon Bedrock — to run the actual evaluation job

Step 2: Create Your S3 Buckets

You need two S3 buckets — one to hold your prompt dataset, another to receive evaluation results. Let's create both from scratch.
In the AWS search bar, type S3 and open the service. Click Create bucket.

Bucket 1: The Prompt Dataset Bucket
This bucket holds the questions you'll throw at the model.
Click Create bucket
Give it a name — something like bedrock-prompt-dataset-yourname-2026. S3 bucket names must be globally unique across all of AWS, so add something personal or random to the end
Make sure the region is set to us-west-2 (Oregon)
Leave everything else as default keep Block all public access enabled
Click Create bucket
Bucket 2: The Output Bucket
This is where Bedrock will write the evaluation results.
Click Create bucket again
Name it something like bedrock-eval-output-yourname-2025
Same region: us-west-2
Leave defaults, click Create bucket

You should now see both buckets in your S3 console.
Build Your Prompt Dataset

What's in the Prompt Dataset?

The prompt dataset is a .jsonl file (one JSON object per line) where each object has three fields:

json
{"prompt": "The chemical symbol for gold is", "category": "Chemistry", "referenceResponse": "Au"}
{"prompt": "The tallest mountain in the world is", "category": "Geography", "referenceResponse": "Mount Everest"}
{"prompt": "The author of 'Great Expectations' is", "category": "Literature", "referenceResponse": "Charles Dickens"}

Notice the structure:

prompt — what you'll send to the model
referenceResponse — the ground truth you're checking against
category — for grouping results later

In production, you'd replace these general-knowledge questions with prompts that mirror your real use case. Customer support queries. Code generation tasks. Medical summaries. Whatever you're building for.

Add a CORS Configuration to the Dataset Bucket

Bedrock needs cross-origin access to read from your S3 bucket. Here's how to enable it:

Click into your prompt dataset bucket
Go to the Permissions tab
Scroll to Cross-origin resource sharing (CORS)
Click Edit and paste this config: json

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": ["Access-Control-Allow-Origin"]
  }
]

Click Save changes

That's it. This tells S3: "Yes, Amazon Bedrock is allowed to read from me." Without this, the evaluation job will fail silently — so don't skip it.

Pro tip: In a real production setup, you'd tighten the AllowedOrigins to specific Bedrock endpoints rather than using "*". For now, this gets us moving.

Step 3: Create the Model Evaluation Job

Back to the AWS search bar — type Bedrock and open Amazon Bedrock.

Expand the left-hand menu (hamburger icon, top-left)
Under Assess, click Evaluations

Click Create Automatic: Programmatic

Now fill in the job configuration:

Model Evaluation Details

Evaluation name: Something unique like my-eval-job-abc123
Model provider: Amazon
Model: Nova Micro
Task type: Question and answer

Metrics

Click Remove on any extra metrics until only one remains. Set it to Accuracy. This metric compares the model's response against your referenceResponse and returns a score.

Prompt Dataset

Select Use your own prompt dataset and enter your S3 path:

s3://your-prompt-dataset-bucket-name/prompt_dataset.json

Evaluation Results

Point this to your output bucket:

s3://your-output-bucket-name/evaluation-results/

IAM Role

Bedrock needs a role to access your S3 buckets on its behalf. Let's create one real quick.
In a new browser tab, go to IAM → Roles → Create role and follow these steps:

Trusted entity Select AWS service, then under Use case search for and select Bedrock. Click Next.
Permissions Attach these two policies: AmazonBedrockFullAccess AmazonS3FullAccess Click Next.
Name and create Name the role something like bedrock-eval-role, then click Create role. Back on the Bedrock evaluation page, under Amazon Bedrock IAM role, select Use an existing role, click the dropdown, and pick bedrock-eval-role.

*In production you'd scope the S3 policy down to only your two specific buckets — but for getting started, AmazonS3FullAccess does the job.

Click Create and watch the job appear with status In progress.

Step 4: Read the Results (This Is the Fun Part)

Once the job completes, head back to S3 and look inside your output bucket under evaluation-results/. You'll find a .jsonl file with one result per prompt. Here's what the raw output looks like:

{
  "automatedEvaluationResult": {
    "scores": [{"metricName": "Builtin.Accuracy", "result": 0.0625}]
  },
  "inputRecord": {
    "prompt": "The chemical symbol for gold is",
    "referenceResponse": "Au",
    "category": "Chemistry"
  },
  "modelResponses": [{
    "response": "The chemical symbol for gold is Au.",
    "modelIdentifier": "us.amazon.nova-micro-v1:0",
    "stopReason": "end_turn"
  }]
}

Breaking Down the Accuracy Scores

Here's a summary of the three prompts from our run:

Looking at the three prompts from our run, the Chemistry question ("The chemical symbol for gold is") scored 0.0625, the Geography question ("The tallest mountain in the world is") came in slightly higher at 0.0870, and the Literature question ("The author of 'Great Expectations' is") landed at 0.0727. All three were answered correctly — Au, Mount Everest, and Charles Dickens respectively yet the scores are nowhere near 1.0.

Wait These Scores Look Low. Is That Bad?

Here's where it gets interesting. The accuracy scores seem low because the scoring algorithm is doing token-level matching between the model's verbose answer and the short reference response.

The model answered correctly in all three cases it said "Au", "Mount Everest", and "Charles Dickens". But it also said a lot of other things (it explained its reasoning step-by-step). Those extra tokens pulled the accuracy score down.

This is a critical lesson: how you write your prompts and reference responses dramatically affects your scores. If you want higher accuracy scores:

// Instead of this reference response:
{"referenceResponse": "Au"}

// Try instructing the model to answer concisely in your prompt:
{"prompt": "Answer in one word only. The chemical symbol for gold is:", "referenceResponse": "Au"}

That's the value of model evaluation — it surfaces these kinds of nuances before you go to production.

What's Next?

Now that you understand the pipeline, here's how to level it up:

Swap models — Run the same dataset against Nova Lite, Nova Pro, or even Claude models to compare them head-to-head
Use real prompts — Replace the sample dataset with 50-100 prompts from your actual use case
Automate — Trigger evaluation jobs via the AWS CLI or SDK as part of your CI/CD pipeline
Track over time — Save scores to a database and chart model performance as you update prompts or switch models

Quick Recap

Here's the full flow in one breath:

Upload a .jsonl prompt dataset to S3
Add a CORS config to the S3 bucket so Bedrock can read it
Create a Bedrock model evaluation job pointing at Nova Micro
Wait for it to run, then read the .jsonl results in your output bucket
Interpret the accuracy scores in context — verbose model answers score lower even when correct

Amazon Bedrock's model evaluation feature removes one of the biggest unknowns in AI integration: "Can this model actually answer my questions reliably?" Now you have a repeatable, automated answer.

Go build something confident.

Have questions or want to share your evaluation results? Drop them in the comments below.