DEV Community: Gunnar Grosch

Visualizing AWS Lambda Durable Function Workflows with durable-viz

Gunnar Grosch — Fri, 27 Mar 2026 15:20:58 +0000

If you're new to durable functions, start with the durable functions post for the core concepts. The short version: your handler re-runs from the beginning on every resume, but completed steps return cached results instantly instead of re-executing. The SDK handles this checkpointing and replay transparently.

Durable functions encourage you to write sequential code. But the execution flow isn't always sequential. You have parallel branches that fan out and converge. Conditionals that route to callbacks or skip to the end. Invocations that call other Lambda functions. The more primitives you use, the harder it gets to see the full picture just by reading the handler.

I hit this when building the purchasing coordinator. Five specialist agents dispatched in parallel, a conditional approval callback, plan and synthesis steps on either side. The code reads top to bottom, but the workflow branches and converges in ways that aren't obvious from the source. Two primitives in particular: context.invoke() calls another Lambda function with automatic checkpointing (unlike the AWS SDK's lambda.invoke(), the result is cached so the target function isn't called again on replay), and waitForCallback suspends the workflow until an external signal arrives. This is how the purchasing coordinator pauses for human approval.

So I built durable-viz: a static analysis tool that turns durable function handlers into flowcharts. No deployment, no execution, no AWS credentials. Point it at a source file and it extracts the workflow structure from the code.

It supports TypeScript, Python, and Java. You can run it as a CLI (Mermaid output, browser, or JSON), or as a VS Code extension with a live diagram panel next to your code.

What It Does

Run durable-viz against any file containing a durable function handler:

npx durable-viz handler.ts --open

It parses the file, extracts the durable primitives (step, parallel, invoke, waitForCallback, conditionals), builds a directed graph, and renders it as a Mermaid flowchart. The --open flag generates an interactive HTML page with zoom, pan, and PNG export.

Here's what the purchasing coordinator from the multi-agent post looks like:

npx durable-viz src/handlers/coordinator.ts

The four-phase flow is immediately visible: plan, five specialists fanning out from a parallel node, synthesize, and the conditional approval callback gated by a diamond. The "no" branch skips straight to End. Each node shape encodes the primitive type (the --open browser view and VS Code extension add color coding). These primitives (step, parallel, invoke, waitForCallback) are the SDK methods that automatically checkpoint their results. On replay, completed primitives return cached results without re-executing. That's what makes them "durable."

The tool didn't execute the code or read a deployment. It parsed the TypeScript AST, found withDurableExecution(), walked the handler body, and extracted every durable primitive with its name and structure. The five specialist branches came from resolving the SPECIALISTS registry object at module scope.

CLI

The default output is Mermaid flowchart syntax printed to stdout. Paste it into GitHub Markdown, Notion, Confluence, or any Mermaid-compatible renderer.

npx durable-viz handler.ts

You can change the graph direction from top-down to left-right:

npx durable-viz handler.ts --direction LR

The --open flag generates a self-contained HTML page and opens it in your browser. Dark theme, scroll-to-zoom, click-drag panning, and fit-to-view. You can save the diagram as a high-resolution PNG for documentation, pull requests, or presentations.

npx durable-viz handler.ts --open

For custom tooling, --json outputs the raw workflow graph (nodes, edges, source line numbers):

npx durable-viz handler.ts --json

VS Code Extension

The extension renders the diagram in a side panel next to your code. Install from the VS Code Marketplace, then open a durable function handler and run Durable Viz: Open Lambda Durable Function Workflow from the command palette.

Feature	Description
Click-to-navigate	Click any node to jump to that line in the source file
Auto-refresh	Diagram updates on file save
Save PNG	Export the diagram as a high-resolution transparent PNG
Source view	View the raw Mermaid syntax or JSON graph

The extension supports zoom, pan, direction toggle, and fit-to-view. See the Marketplace listing for the full feature list.

Multi-Language Support

The tool supports TypeScript/JavaScript, Python, and Java. Each language has its own parser, but the graph model, edge builder, and renderers are shared.

TypeScript / JavaScript

Uses ts-morph for full AST parsing. This is the most capable parser with two features the others don't have:

Function-reference following. If your handler calls a helper function that accepts DurableContext, the parser resolves the call and inlines the helper's durable primitives at the call site. Only works for functions defined in the same file.
Registry key resolution. For context.parallel() calls that use .map() over a module-scope registry object, the parser enumerates the registry keys to show all possible parallel branches. This is how the purchasing coordinator's five specialists appear in the diagram even though the code dispatches them dynamically.

Python

npx durable-viz examples/order_processor.py --direction LR

Finds @durable_execution decorated handlers and extracts context.<method>() calls. Uses indentation to determine block boundaries.

Java (preview)

npx durable-viz Handler.java --open

Finds classes extending DurableHandler and extracts ctx.<method>() calls from the handleRequest method. Some primitives (parallel, waitForCallback, waitForCondition) are still in development in the Java durable execution SDK.

Both the Python and Java parsers use regex rather than full AST parsing. This keeps the tool as a single Node.js package without requiring Python or Java parser dependencies. The trade-off: standard single-line call patterns work well, but method calls split across many lines or unusual argument formatting may not be detected. For most idiomatic durable function code, it works without issues.

Supported Primitives

Primitive	TypeScript	Python	Java (preview)
Step	`context.step()`	`context.step()`	`ctx.step()`
Invoke	`context.invoke()`	`context.invoke()`	`ctx.invoke()`
Parallel	`context.parallel()`	`context.parallel()`	in development
Map	`context.map()`	`context.map()`	in development
Wait	`context.wait()`	`context.wait()`	`ctx.wait()`
Wait for Callback	`context.waitForCallback()`	`context.wait_for_callback()`	in development
Create Callback	`context.createCallback()`	`context.create_callback()`	`ctx.createCallback()`
Wait for Condition	`context.waitForCondition()`	`context.wait_for_condition()`	in development
Child Context	`context.runInChildContext()`	`context.run_in_child_context()`	`ctx.runInChildContext()`

TypeScript also detects context.promise.all(), context.promise.any(), context.promise.race(), and context.promise.allSettled().

Visual encoding

Each primitive type has a distinct shape and color:

Node	Shape	Color
Start / End	Stadium	Blue
Step	Rectangle	Green
Invoke	Trapezoid	Amber
Parallel / Map	Hexagon	Purple
Wait / Callback	Circle	Red
Condition	Diamond	Indigo
Child Context	Subroutine	Teal

How It Works

The tool performs static analysis on your source file. It never imports, executes, or deploys your code.

The architecture is a three-stage pipeline:

[source file] → Parser → WorkflowGraph → Renderer → [output]
                  │                          │
          TypeScript / Python / Java    Mermaid / JSON

The parser interface

Adding a new language means implementing two methods:

export interface Parser {
  extensions: string[]
  parseFile(filePath: string, options?: ParseOptions): WorkflowGraph
}

extensions declares which file types the parser handles. parseFile takes a file path and returns a WorkflowGraph with nodes, edges, and source line numbers. The dispatcher selects the right parser by file extension.

The graph model

The parser produces a WorkflowGraph: an ordered list of nodes with branches (for parallel blocks) and metadata (for conditionals). Here's a simplified view of what each node looks like:

interface WorkflowNode {
  id: string
  kind: 'start' | 'end' | 'step' | 'invoke' | 'parallel' | 'map'
      | 'wait' | 'waitForCallback' | 'condition' | /* ... */  // maps to the primitives table above
  label: string
  branches?: WorkflowBranch[]   // for parallel/map nodes
  thenCount?: number            // for conditions: nodes in the then-branch
  thenReturns?: boolean         // for conditions: does then-branch return?
  sourceLine?: number           // 1-based line number for click-to-navigate
}

The edge builder constructs edges from the node list, handling sequential flow, parallel fan-out/fan-in, and conditional routing.

For conditionals, the parser tracks whether the if block ends with a return. If it does, the "yes" branch connects to End instead of falling through. The "no" branch skips the conditional block and continues to the next node.

Registry key resolution in practice

Here's a pattern from the purchasing coordinator. Don't worry about the specifics of the durable function code. The key thing is that the parallel branches are built dynamically at runtime from a registry object:

const SPECIALISTS: Record<string, { functionName: string; display: string }> = {
  'price-research': { functionName: process.env.PRICE_RESEARCH_FUNCTION!, display: 'Price Research' },
  'financing': { functionName: process.env.FINANCING_FUNCTION!, display: 'Financing' },
  // ... 3 more
}

const results = await context.parallel('specialists',
  plan.specialists.map((spec) => ({
    name: spec.name,
    func: async (ctx) => {
      const result = await ctx.invoke(spec.name, specialist.functionName, { prompt: spec.prompt })
      return { name: spec.name, response: result.response }
    },
  }))
)

The .map() call means the parallel branches are determined at runtime. The parser can't execute the code, but it can look at the SPECIALISTS object and enumerate its keys. It finds five keys, creates five invoke branches, and labels them with the key names. This is how the diagram shows all five specialists even though the code builds the branch list dynamically.

If the parser can't resolve the registry (the object is imported from another file, or the pattern doesn't match), it falls back to showing a single representative branch.

If you point the tool at a file that isn't a durable function handler, or a file with syntax errors, it exits with a clear error message. The VS Code extension shows the error in the webview panel instead of a diagram.

Design Decisions

Static analysis over runtime tracing

The main design choice: parse the code, don't execute it. Runtime tracing would give you the actual execution path for a specific input, but it requires deployment, credentials, and a real invocation. Static analysis gives you all possible paths from the source alone. You see every parallel branch, every conditional route, every callback. The trade-off is that dynamic branches (like the specialist .map()) require heuristics to resolve.

For documentation and code review, seeing all possible paths is usually more useful than seeing one specific execution. For debugging a specific run, the durable execution history API (get-durable-execution) is the right tool.

Language-agnostic graph model

The parsers are language-specific. The graph model, edge builder, and renderers are not. Adding a new language means writing a parser that produces WorkflowGraph nodes. Everything downstream is shared. The TypeScript parser uses ts-morph for full AST analysis. The Python and Java parsers use regex, which handles standard patterns well but can miss unusual formatting. The regex approach was a deliberate trade-off: full AST parsing for Python would require a Python parser dependency, and Java would need a Java parser. Regex keeps the tool as a single Node.js package.

Visual encoding for primitive types

Each primitive type gets a unique shape and color combination so you can identify the primitive at a glance without reading labels. Steps are green rectangles (the most common node). Invocations are amber trapezoids (they call out to external functions). Parallel blocks are purple hexagons (they branch). Callbacks are red circles (they suspend execution). Conditionals are indigo diamonds (standard flowchart convention). The color palette is optimized for dark backgrounds since most developer tools use dark themes.

Try It Out

You'll need:

Node.js 20+

Run against the examples

Clone the repo and run against the included examples:

git clone https://github.com/gunnargrosch/durable-viz.git
cd durable-viz

# TypeScript order workflow
npx durable-viz examples/order-workflow.ts --open

# Python order processor
npx durable-viz examples/order_processor.py --open

# Java order processor
npx durable-viz examples/OrderProcessor.java --open

Run against the purchasing coordinator

If you have the multi-agent purchasing demo cloned:

cd durable-multi-agent-purchasing
npx durable-viz src/handlers/coordinator.ts --open

Run against your own handler

For your own durable function handlers, npx downloads and runs the tool directly. No cloning needed:

npx durable-viz path/to/your-handler.ts --open

Install the VS Code extension

Search "Durable Viz" in the Extensions panel, or run:

ext install gunnargrosch.durable-viz

Open a durable function handler, open the command palette, and run Durable Viz: Open Lambda Durable Function Workflow.

Additional Resources

durable-viz on GitHub
durable-viz on npm
VS Code Marketplace
Multi-Agent Systems on AWS Lambda with Durable Functions: The purchasing coordinator used as the primary example
AWS Lambda Durable Functions: Building Long-Running Workflows in Code: Durable execution primitives and the support triage demo
AWS Lambda Durable Functions documentation

Run npx durable-viz against your handler and share the diagram. I'd love to see what your workflows look like!

Multi-Agent Systems on AWS Lambda with Durable Functions

Gunnar Grosch — Wed, 25 Mar 2026 13:27:42 +0000

In my previous post on multi-agent systems, I built a purchasing coordinator where a coordinator agent routes requests to specialist agents based on RISEN prompt contracts. RISEN structures system prompts into five components: Role, Instructions, Steps, Expectation, and Narrowing. In a multi-agent system, the Steps section encodes the routing logic and the Narrowing section prevents agents from doing each other's work. A laptop triggers Price Research and Delivery. A used car triggers all five specialists. The routing logic lives in the prompts, not in code. It works, but it runs in a single process. No fault isolation, no independent scaling, no durability. If the process crashes halfway through specialist consultations, you start over.

The durable functions post solved the durability problem for a support ticket triage workflow: checkpoint each step, suspend for human review, resume where you left off. But that was a single-agent workflow.

This post combines the two. The same purchasing coordinator, deployed to AWS Lambda, with each specialist as its own Lambda function. The coordinator is a durable function that checkpoints every specialist call. If it's interrupted after consulting three of five specialists, it resumes from the fourth. When a high-value purchase needs human approval, the function suspends, compute charges stop, and it picks up exactly where it left off when the approver responds.

The two SDKs have distinct roles. The Strands Agents SDK handles the AI reasoning: the planning agent decides which specialists to call, and the synthesis agent produces the recommendation. The durable execution SDK handles the infrastructure: checkpointing, parallel dispatch, suspension, and replay. The coordinator uses both. The specialists only use Strands.

The complete source code is on GitHub: github.com/gunnargrosch/durable-multi-agent-purchasing.

What Changes from the In-Process Demo

The multi-agent post used tool() callbacks for specialist dispatch: each specialist was defined as a Strands tool, and the coordinator agent called them as functions within the same process. That's the simplest possible architecture, and it's fine for development. Here's what changes when you deploy:

	In-Process (previous post)	Lambda + Durable (this post)
Specialist invocation	`tool()` callback, in-process	`context.invoke()`, separate Lambda function
Execution model	Sequential (one specialist at a time)	Parallel via `context.parallel()`
Fault tolerance	None. Process crash = restart	Checkpointed. Resume from last completed step
Scaling	Single process	Each specialist scales independently
Human-in-the-loop	Not supported	`waitForCallback()` with zero-cost suspension
IAM isolation	Shared process permissions	Per-specialist least-privilege policies
Routing visibility	Real-time hook as each tool is called	Checkpointed plan step with routing summary
Infrastructure	`npm start`	SAM template, 6 Lambda functions

The RISEN prompts carry over with only minor adjustments. The coordinator's prompt splits into two phases (plan and synthesis) instead of a single invocation, because the durable function needs to checkpoint the plan before dispatching specialists. The specialist prompts are unchanged.

How It Works

If you haven't read the durable functions post, here's the key mental model: durable functions use checkpoint and replay. Your handler re-executes from the top on every resume, but completed steps return their cached results instantly without re-executing. New work picks up from where it left off. The SDK manages this transparently. You write sequential code and the infrastructure handles the rest.

[CoordinatorFunction] — Lambda Durable Function (Sonnet, 1536MB)
  ├→ context.step('plan')            → Planning agent selects specialists
  ├→ context.parallel('specialists') → Runs selected specialists concurrently:
  │     ├─ context.invoke(PriceResearchFunction)  (Haiku)   ← always
  │     ├─ context.invoke(FinancingFunction)       (Haiku)   ← if value > $5K
  │     ├─ context.invoke(DeliveryFunction)        (Haiku)   ← if physical product
  │     ├─ context.invoke(RiskAssessmentFunction)  (Sonnet)  ← if value > $10K / used
  │     └─ context.invoke(ContractReviewFunction)  (Haiku)   ← if subscription/lease
  ├→ context.step('synthesize')      → Synthesis agent combines findings
  └→ context.waitForCallback()       → Human approval (when requireApproval=true)

Three things to notice:

context.invoke() is a new primitive not covered in the durable functions post. It's the SDK's built-in method for calling other Lambda functions. It checkpoints the result automatically and suspends the coordinator while waiting, so you don't pay for compute during specialist execution. Despite the SDK's API reference describing it as invoking "another durable function," context.invoke() works with any Lambda function. The specialists here are standard functions without DurableConfig.
context.step() wraps the planning and synthesis phases with retry strategies, just like the Bedrock calls in the support triage demo. Each step checkpoints its result. On replay, it returns the cached result without re-executing.
context.parallel() wraps the specialist invocations so they run concurrently, each independently checkpointed. If specialist 3 of 5 fails, the other four results are preserved.

The SAM Template

Here are the key parts of the coordinator's definition in the SAM template. The template defines 6 functions total: the coordinator plus 5 specialists that follow the same pattern.

CoordinatorFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: coordinator.handler
    MemorySize: 1536
    Timeout: 300
    Environment:
      Variables:
        COORDINATOR_MODEL_ID: !Sub global.${CoordinatorModelId}
        PRICE_RESEARCH_FUNCTION: !Ref PriceResearchFunction
        # ... remaining specialist function references
    Policies:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
      - Statement:
          - Effect: Allow
            Action:
              - bedrock:InvokeModel
              - bedrock:InvokeModelWithResponseStream
            Resource: # Bedrock model + inference profile ARNs
          - Effect: Allow
            Action: lambda:InvokeFunction
            Resource:
              - !GetAtt PriceResearchFunction.Arn
              - !GetAtt FinancingFunction.Arn
              - !GetAtt DeliveryFunction.Arn
              - !GetAtt RiskAssessmentFunction.Arn
              - !GetAtt ContractReviewFunction.Arn
    DurableConfig:
      ExecutionTimeout: 86400
      RetentionPeriodInDays: 7
    AutoPublishAlias: live

A few things to note:

No API Gateway, no function URLs, no public endpoints. The multi-agent post previewed two deployment options: Lambda with HTTP endpoints, or AgentCore containers. This implementation goes a simpler route. context.invoke() calls specialists directly via the Lambda API. The coordinator's IAM policy grants lambda:InvokeFunction on each specialist ARN. Specialists are unreachable from outside the coordinator's execution role.
Shared handler, different prompts. All five specialists use the same specialist.ts handler. The PROMPT_NAME environment variable selects which RISEN prompt to load. This keeps the template repetitive but predictable: each specialist block differs only in its name, PROMPT_NAME, and model ID.
Per-specialist model selection. Risk Assessment gets AdvancedSpecialistModelId (Sonnet) for stronger reasoning. The other four get SpecialistModelId (Haiku). Same pattern as the in-process demo, now enforced at the infrastructure level.
DurableConfig with 24-hour execution timeout. The used car scenario with human approval can sit overnight. Each individual invocation is bounded by Timeout: 300 (the coordinator's per-replay limit). ExecutionTimeout: 86400 is the outer wall-clock limit across all replays and suspensions.
InvokeModelWithResponseStream is included because the Strands SDK's BedrockModel may use streaming internally for token generation. Without it, the coordinator would get AccessDeniedException on agent invocations.
ESM build. The coordinator uses ESM (Format: esm) with esbuild. The full template includes a Banner that injects a createRequire shim because some dependencies expect CommonJS require. This is a known pattern for ESM Lambda functions with mixed dependencies. See the repo's template.yaml for the complete Metadata block.

The Coordinator Handler

The coordinator is the only durable function. Here's the handler skeleton showing the four durable phases. The full source includes types, the specialist registry, retry strategies, and routing summary logic.

export const handler = withDurableExecution(async (
  event: CoordinatorEvent,
  context: DurableContext
): Promise<Record<string, unknown>> => {

  // Phase 1: Plan — agent decides which specialists to consult
  const plan = await context.step<AnalysisPlan>('plan', async () => {
    let capturedPlan: AnalysisPlan | null = null
    const planTool = tool({
      name: 'create_analysis_plan',
      inputSchema: z.object({
        specialists: z.array(z.object({
          name: z.enum(['price-research', 'financing', 'delivery',
                        'risk-assessment', 'contract-review']),
          prompt: z.string(),
        })),
      }),
      callback: async (input) => { capturedPlan = input; return 'Plan created.' },
    })
    const agent = new Agent({ model, systemPrompt: planPrompt, tools: [planTool], printer: false })
    await agent.invoke(event.request)
    if (!capturedPlan) throw new Error('Planning agent did not call create_analysis_plan')
    return capturedPlan
  }, { retryStrategy: bedrockRetry })

  // Phase 2: Consult specialists in parallel via context.invoke()
  const results = await context.parallel<{ name: string; response: string }>('specialists',
    plan.specialists.map((spec) => ({
      name: spec.name,
      func: async (ctx) => {
        try {
          const specialist = SPECIALISTS[spec.name]
          if (!specialist) {
            throw new Error(`Unknown specialist: ${spec.name}`)
          }
          const result = await ctx.invoke<{ prompt: string }, { response: string }>(
            spec.name, specialist.functionName, { prompt: spec.prompt }
          )
          return { name: spec.name, response: result.response }
        } catch (err) {
          const display = SPECIALISTS[spec.name]?.display ?? spec.name
          return { name: spec.name, response: `[${display} unavailable: ${err instanceof Error ? err.message : 'unknown error'}]` }
        }
      },
    }))
  )

  // Phase 3: Synthesize specialist findings into a recommendation
  const response = await context.step('synthesize', async () => {
    // Build findings from checkpointed parallel results, invoke synthesis agent
    // (see full source for details)
  }, { retryStrategy: bedrockRetry })

  // Phase 4: Human approval (optional)
  if (event.requireApproval) {
    const approval = await context.waitForCallback<ApprovalPayload>(
      'approval',
      async (callbackId) => { context.logger.info('approval_callback_created', { callbackId }) },
      { timeout: { hours: 24 }, serdes: defaultSerdes }
    )
    if (!approval.approved) return { status: 'rejected', recommendation: response }
  }

  return { status: event.requireApproval ? 'approved' : 'completed', recommendation: response }
})

Let's walk through what's happening in each phase.

Phase 1: Plan

The planning phase is the biggest change from the in-process demo. In the previous post, the coordinator was a single agent that both decided which specialists to call and synthesized their findings. Here, planning and synthesis are separate agents wrapped in separate context.step() calls.

Why split them? Checkpointing. In the in-process demo, if the coordinator crashes after calling three specialists, you lose the routing decision and all three results. With durable functions, the plan is checkpointed as a single unit. If the function replays after the plan step, it returns the cached plan instantly without calling Bedrock again.

The planning agent uses tool() with a Zod schema to produce structured output. The create_analysis_plan tool captures the plan into a closure variable, and the step returns it as its checkpointed result. If the agent doesn't call the tool, the step throws an error. The retry strategy will attempt it three times, but this is a non-transient failure: if the model didn't call the tool on the first attempt, retries won't help. After retries are exhausted, the execution fails.

Phase 2: Parallel specialists

This is where context.invoke() replaces tool() callbacks. Each specialist invocation is a branch inside context.parallel():

const result = await ctx.invoke<{ prompt: string }, { response: string }>(
  spec.name,             // step name for checkpoint history
  specialist.functionName, // Lambda function name from environment
  { prompt: spec.prompt }  // payload
)

context.invoke() calls the specialist Lambda function directly, checkpoints the result, and suspends the coordinator while waiting. You don't pay for coordinator compute while specialists are executing. On replay, completed invocations return their cached results without re-invoking the specialist.

Each branch has a try/catch for graceful degradation. If the Delivery specialist times out, the coordinator still gets results from Price Research, Financing, Risk Assessment, and Contract Review. The synthesis agent notes the gap and advises the buyer to investigate delivery independently. This is a meaningful improvement over the in-process demo, where a failed specialist tool call could derail the entire coordinator.

Phase 3: Synthesize

The synthesis agent receives all specialist findings and produces a structured recommendation. Like the plan step, the entire synthesis is one checkpointed unit. The synthesis prompt's Narrowing section prevents it from overriding specialist findings or filling in gaps from failed specialists.

Phase 4: Human approval

When requireApproval is true (the used car scenario sets this), the function suspends at waitForCallback. Compute charges stop. The approver reviews the recommendation and sends a callback via the Lambda API or the demo's interactive prompt. The function resumes and returns the final status.

Note the serdes: defaultSerdes option on the callback. As covered in the durable functions post's gotchas, waitForCallback defaults to passthrough serialization (not JSON.parse). Without defaultSerdes, approval.approved would be undefined at runtime even though TypeScript thinks it's a boolean.

The Specialist Handler

All five specialists share one handler. The PROMPT_NAME environment variable selects the behavior:

const promptName = process.env.PROMPT_NAME!
const systemPrompt = loadPrompt(promptName)
const model = createModel(SPECIALIST_MODEL)

export const handler = async (
  event: SpecialistEvent,
  context: Context
): Promise<{ response: string }> => {
  const logger = makeLogger({ handler: promptName, requestId: context.awsRequestId })
  const tools = loadSpecialistTools(promptName, logger)
  const agent = new Agent({ model, systemPrompt, tools, printer: false })
  const result = await agent.invoke(event.prompt)
  return { response: result.toString() }
}

Specialists are standard Lambda functions, not durable. They don't need checkpointing because each one completes in a single invocation (a Bedrock call and response processing). The coordinator's context.invoke() handles the durability: if a specialist invocation times out, the coordinator can retry from the checkpoint without re-running specialists that already succeeded.

loadSpecialistTools returns specialist-specific tools (see src/lib/specialist-tools.ts). Most specialists are pure reasoning (no tools). The Price Research specialist has a save_price_snapshot tool that logs structured price data. In production, that tool could write to DynamoDB or call a pricing API. The coordinator never sees these tools. They're scoped to each specialist's domain.

The Prompt Split

The in-process demo had one coordinator prompt with routing in the Steps section. The durable version splits this into two prompts:

coordinator-plan handles routing. Here's the Steps and Expectation sections:

# Steps
1. Read the purchase request and identify what is being purchased, its likely category
   (vehicle, electronics, real estate, software, etc.), the approximate value range,
   and any special circumstances mentioned or implied (used/secondhand, financing needed,
   physical delivery, contract or subscription, high value).
2. Select specialists using these routing rules:
   - Always include price-research to compare options and assess market value.
   - Include financing if the estimated value exceeds $5,000, or if financing or a loan
     is mentioned or implied.
   - Include delivery if the item is a tangible physical product that must be physically
     received — electronics, appliances, vehicles, furniture, machinery. Vehicles always
     need delivery planning (transport, pickup, or test drive logistics). Do not include
     for purely digital purchases (software, SaaS subscriptions, downloadable content).
   - Include contract-review if the purchase involves a subscription, lease, warranty
     agreement, service contract, purchase agreement, or any multi-year financial
     commitment. Vehicle and real estate purchases always involve contracts.
   - Include risk-assessment if the estimated value exceeds $10,000, the item is used or
     secondhand, or the category carries known risk (vehicles, real estate, machinery).
     Do not include for new electronics, appliances, or standard retail under $10,000.
3. For each selected specialist, write a focused prompt describing what to analyze about
   this specific purchase. Include relevant details from the request (item, price,
   condition, location, urgency). The specialist's own system prompt defines its role —
   do not repeat the role description in your prompt.
4. Call create_analysis_plan exactly once with the selected specialists and their prompts.

# Expectation
A single call to create_analysis_plan containing:
- An array of specialists, each with a name (from the allowed set) and a prompt string.
- Only specialists whose routing criteria are met.
- Prompts that are specific to this purchase, not generic templates.

These are the same routing rules from the original coordinator prompt in the multi-agent post. The difference: instead of calling specialist tools directly (Steps 2-6 in the original said "invoke the research_prices tool," "invoke the evaluate_financing tool"), the plan agent calls create_analysis_plan once with all selected specialists. The actual dispatch happens via context.invoke() in Phase 2.

coordinator-synthesis handles the final recommendation. It receives specialist findings and produces the buyer-facing output. Its Narrowing prevents it from contradicting specialists or filling in missing analysis.

This split means the coordinator makes two agent invocations (plan + synthesize) instead of one. Each agent invocation may involve multiple Bedrock round-trips internally as the Strands SDK handles reasoning. The trade-off is worth it: the plan is checkpointed before any specialist is called, and the synthesis is checkpointed after all specialists complete. On replay, both return cached results instantly.

Testing with the Local Runner

The durable functions post showed LocalDurableTestRunner for the support triage workflow. The multi-agent demo adds a new pattern: registerFunction for mocking context.invoke() targets.

// Register mock handlers for each specialist Lambda function
runner = new LocalDurableTestRunner({ handlerFunction: handler });
runner
  .registerFunction("PriceResearchFunction", specialistHandler)
  .registerFunction("FinancingFunction", specialistHandler)
  .registerFunction("DeliveryFunction", specialistHandler)
  .registerFunction("RiskAssessmentFunction", specialistHandler)
  .registerFunction("ContractReviewFunction", specialistHandler);

// Run the full workflow and verify the result
const result = runner.run({ payload: laptopPayload });
const output = await result;
expect(output.getStatus()).toBe("SUCCEEDED");
expect(output.getResult().routing.called).toContain("Price Research");

registerFunction maps function names to local handlers. When the coordinator calls context.invoke("PriceResearchFunction", payload), the test runner routes it to the registered mock instead of calling Lambda. This lets you test the full checkpoint/replay lifecycle without deploying or calling Bedrock. The full test suite also tests callback suspension and resumption using runner.getOperation() and sendCallbackSuccess(), the same pattern from the durable functions post.

The test suite covers seven scenarios: standard flow, all-5-specialists flow, approval, rejection, specialist failure with graceful degradation, callback failure, and planning agent failure.

Try the Demo

The repo includes an interactive demo with three purchase scenarios:

Scenario	Value	Specialists	Approval
Laptop	$1,500	Price Research + Delivery	No
Used car	$18,000	All 5 specialists	Yes (`waitForCallback`)
SaaS subscription	$200/mo	Price Research + Contract Review	Yes (`waitForCallback`)

You'll need:

Node.js 24+ and npm
For cloud mode: AWS SAM CLI 1.153.1+ and Bedrock access to Claude Sonnet 4.6 and Claude Haiku 4.5

Local mode (no AWS credentials needed)

git clone https://github.com/gunnargrosch/durable-multi-agent-purchasing.git
cd durable-multi-agent-purchasing
npm install
npm run demo:local -- --ticket=used-car

Local mode uses mocked Bedrock responses. The used car scenario exercises all four durable primitives: step (plan and synthesis), invoke (specialist calls), parallel (concurrent dispatch), and waitForCallback (human approval where you play the purchase approver).

Cloud mode (real Bedrock responses)

sam build
sam deploy --guided
npm run demo:cloud -- --ticket=used-car --region=us-east-1

Cloud mode invokes the deployed coordinator with real Bedrock calls. You'll see actual AI-generated specialist analyses and a synthesized recommendation. The demo polls execution history and prompts you when the approval callback is created.

The demo uses direct aws lambda invoke with --invocation-type Event for simplicity. In production, the coordinator would typically sit behind an upstream service: an API Gateway endpoint receiving purchase requests, an EventBridge rule triggered by order events, or an SQS queue processing a backlog. The coordinator itself doesn't care how it's invoked. It receives the event payload and the durable SDK handles the rest.

Inspecting execution history

After a cloud run, you can inspect the execution in the Lambda console under the Durable executions tab, or via the CLI:

aws lambda list-durable-executions-by-function \
  --function-name durable-multi-agent-purchasing-CoordinatorFunction

aws lambda get-durable-execution \
  --durable-execution-arn "<arn-from-list>"

The execution history shows each step's status and timing: when the plan completed, how long each specialist took, and whether the callback is pending or resolved.

Design Decisions

context.invoke() over HTTP

The multi-agent post previewed two deployment approaches: Lambda with API Gateway/Function URLs (HTTP), or AgentCore containers. This demo takes a third path: context.invoke() for direct Lambda-to-Lambda invocation.

The result is simpler than either preview option. No API Gateway resources, no function URLs, no SigV4 signing, no HTTP client configuration. The coordinator calls specialists via the Lambda API, and the durable SDK handles checkpointing and retry. Specialists are unreachable from outside the coordinator's execution role, which is better isolation than an HTTP endpoint with IAM auth.

The trade-off: specialists are only callable from within a durable function. If you later need specialists accessible from other services (a REST API, a Step Functions state machine, another team's coordinator), you'd need to add API Gateway or function URLs at that point. For this use case, where one coordinator owns all specialist dispatch, direct invocation is the right call.

Splitting plan from synthesis

The in-process coordinator was a single agent invocation: read the request, call specialist tools, synthesize findings. With durable functions, that becomes two separate agents wrapped in separate context.step() calls.

This introduces an extra Bedrock call, but the checkpointing benefits are significant. The plan is preserved before any specialist runs. If the function replays after three of five specialists complete, the plan doesn't need to be regenerated. The synthesis is preserved after all specialists complete. If the function replays during the approval callback, the recommendation doesn't need to be regenerated.

The alternative would be wrapping the entire coordinator in a single step. That would mean one Bedrock conversation (cheaper) but no intermediate checkpoints. A failure during synthesis would replay from the beginning, including all specialist invocations. With the split, a synthesis failure only retries the synthesis.

Planning agent with tool capture

The planning agent uses tool() with a Zod schema to produce structured output. This is the same pattern from the in-process demo, but used differently. In the previous post, tools were the dispatch mechanism (calling tools = calling specialists). Here, the tool is purely for structured output capture. The plan step returns the captured plan as its checkpointed result, and context.invoke() handles the actual specialist dispatch.

Why not just have the planning agent return JSON directly? Tool calling with a schema gives you validation at the SDK level. If the agent returns a plan with an invalid specialist name, Zod catches it before the plan is checkpointed. Without the tool, you'd parse and validate the JSON yourself, and an invalid plan could be checkpointed and break on every subsequent replay.

Graceful degradation in parallel

Each specialist branch in context.parallel() has a try/catch that returns an error message instead of throwing:

catch (err) {
  return { name: spec.name, response: `[unavailable: ${err instanceof Error ? err.message : 'unknown error'}]` }
}

This means a failed specialist doesn't fail the entire parallel block. The synthesis agent receives partial results and notes the gap. Without the catch, a failed specialist would fail its branch. Whether that fails the entire parallel block depends on the completion config, but either way, synthesis would never run with partial results. With the catch, every branch succeeds (some with error messages), the parallel block always completes, and the synthesis agent can work with whatever it has.

Cost

The main cost is Bedrock token usage, not Lambda compute.

Scenario	Specialists	Agent invocations	Estimated cost
Laptop (2 specialists)	2	~4 (plan + 2 specialists + synthesize)	~$0.02-0.05
Used car (5 specialists)	5	~7 (plan + 5 specialists + synthesize)	~$0.05-0.15

The coordinator uses Sonnet (~$3/$15 per million input/output tokens). Most specialists use Haiku (~$1/$5 per million input/output tokens). Risk Assessment uses Sonnet. Lambda compute for the active execution periods (plan, specialist dispatch, synthesis, replay) totals ~$0.0001. During the waitForCallback suspension, compute charges are zero.

Things to Watch For

Checkpoint size. Each step result is serialized and stored as a checkpoint. The durable functions post covered the 256KB limit per checkpoint. With 5 specialists returning Bedrock responses, the parallel result could get large if the model is verbose. Monitor response sizes. If you hit the limit, truncate or summarize specialist responses before returning them from the branch, or store full responses in S3 and return a reference.

Debugging failures. If the plan step fails after exhausting retries, or a specialist consistently times out, the execution moves to FAILED status. Use get-durable-execution to see which step failed and the error message. The coordinator uses context.logger (replay-aware), so CloudWatch Logs show each phase's progress without duplicate lines from replays. Specialist failures are easier to debug since each specialist has its own log group (sam logs --name PriceResearchFunction --tail).

Replay safety. Durable functions re-execute your handler from the top on every resume. Completed context.step() and context.invoke() calls return cached results, but any code outside those primitives runs again on every replay. If you add a side effect (writing to DynamoDB, sending a notification, calling an external API), wrap it in a context.step() so it executes exactly once. The coordinator's comment on the save-recommendation step shows this pattern. Without the step wrapper, you'd send duplicate notifications every time the function replays.

Cold starts. The used car scenario invokes 5 specialist Lambda functions in parallel. If all 5 are cold, that's 5 concurrent cold starts: arm64 Node.js 24, the Strands SDK, and the Bedrock client. This can add several seconds to the first specialist round. The in-process demo had no cold start penalty for specialists since everything ran in one process. In practice, the cold start overhead is small relative to the Bedrock inference time that follows. For workflows that already include multi-second model calls and human approval, a few extra seconds on the first invocation is rarely the bottleneck.

Wrapping Up

This post showed how to take a multi-agent system from a single-process demo to a deployed, fault-tolerant system on Lambda with durable functions. The RISEN prompts carry over with minimal changes. The architectural shift is from in-process tool calls to checkpointed Lambda-to-Lambda invocations with independent scaling, failure isolation, and human-in-the-loop approval.

Additional Resources

Demo repository for this post
Building Multi-Agent Systems with RISEN Prompts and Strands Agents: The in-process multi-agent demo this builds on
AWS Lambda Durable Functions: Building Long-Running Workflows in Code: Durable execution primitives and the support triage demo
AWS Lambda Durable Functions documentation
Durable Execution SDK for JavaScript (GitHub)
Strands Agents SDK (TypeScript)

What multi-agent workflow would you deploy with durable functions? Let me know in the comments!

AWS Lambda Durable Functions: Building Long-Running Workflows in Code

Gunnar Grosch — Tue, 17 Mar 2026 22:06:16 +0000

If you've built anything non-trivial on AWS Lambda, you've hit the wall. The function runs for 15 minutes and it's stateless. Any multi-step workflow requires stitching together Step Functions, SQS queues, DynamoDB tables for state, and a whole lot of glue. It works, but it's a lot of infrastructure for what should be straightforward sequential logic.

AWS Lambda Durable Functions, launched at re:Invent 2025, change that. You write sequential code in a single Lambda function. The SDK handles checkpointing, failure recovery, and suspension. Your function can run for up to a year, and you only pay for active compute time. During waits (human approvals, timers, external callbacks), the function suspends and compute charges stop.

In this post, I'll walk through what problem durable functions solve, how the checkpoint/replay model works, and then dig into a complete AI-powered support ticket workflow in TypeScript that demonstrates every primitive in action.

What Problem This Solves

Here's a scenario most teams deal with: a support ticket arrives, someone needs to triage it, figure out who should handle it, wait for them to respond, and then close the loop with the customer. Before durable functions, you had a few options:

Step Functions: Define an ASL state machine with states for each step, configure IAM for each integration, manage the state machine as a separate resource. Great for cross-service orchestration, but heavyweight for application logic that naturally reads as sequential code.

SQS + multiple Lambda functions: Break the workflow into separate functions connected by queues. Now you're managing message formats, dead-letter queues, idempotency, and correlating state across function boundaries.

Polling loop with DynamoDB: One function writes state to DynamoDB, another polls for changes. Works, but you're paying for polling compute and managing your own state machine.

All three approaches take what should be straightforward sequential logic and spread it across multiple services, IAM policies, and configuration files.

With durable functions, that same workflow looks like this:

const handler = async (event: TicketEvent, context: DurableContext) => {
  const analysis = await context.step("analyze", async () => analyzeTicket(event));
  const response = await context.waitForCallback("agent-review",
    async (callbackId) => notifyAgent(callbackId, analysis)
  );
  if (analysis.needsEscalation) {
    await context.waitForCallback("specialist-review",
      async (callbackId) => notifySpecialist(callbackId, analysis)
    );
  }
  await context.parallel("close-ticket", [
    { name: "reply", func: async (ctx) => ctx.step("send-reply", async () => sendReply(response)) },
    { name: "survey", func: async (ctx) => ctx.step("send-survey", async () => sendSurvey(event)) },
  ]);
  return { status: "resolved", ticketId: event.ticketId };
};

One function. Sequential code. The SDK handles checkpointing each step, suspending during the human review waits, and resuming when the callbacks arrive.

How Checkpoint/Replay Works

This is the part that makes everything else make sense. Durable functions use a checkpoint and replay model. Here's how it works:

First invocation: Your handler runs from the beginning. Each context.step() executes your code and checkpoints the result.
Suspension: When context.wait() (fixed-duration pause) or context.waitForCallback() (external signal) is called, the function terminates. Compute charges stop.
Resumption: When the wait completes or a callback arrives, Lambda invokes your handler again from the beginning. But this time, completed steps return their cached results instantly without re-executing. Execution picks up from the first non-checkpointed operation.

First invocation:
  analyze     ->  [executes Bedrock call, checkpoints result]
  agent-review ->  [creates callback, function suspends]

Second invocation (agent responds):
  analyze     ->  [returns cached result, skips Bedrock call]
  agent-review ->  [returns callback result]
  close-ticket ->  [sends reply + survey in parallel]

There's one critical rule that falls out of this: code outside steps re-executes on every replay and must be deterministic. If you use Date.now(), Math.random(), or crypto.randomUUID() outside a step, you'll get different values on each replay. Wrap non-deterministic operations in steps.

// Wrong: different value on each replay
const id = crypto.randomUUID();

// Right: checkpointed, same value on every replay
const id = await context.step("gen-id", async () => crypto.randomUUID());

What You'll Build

A support ticket triage workflow where AI handles the first pass and humans make the final call. This is the pattern that makes durable functions click: the AI analysis takes seconds, but the human reviews take hours or days. Without durable functions, you'd need to persist state somewhere and wire up resumption logic. With them, you just write await context.waitForCallback() and the function suspends until the human responds.

Primitive	What It Does	Where You'll See It
`step()`	Execute and checkpoint an atomic operation	AI ticket analysis with Bedrock
`waitForCallback()`	Suspend until an external system responds	Agent review, specialist escalation
`parallel()`	Run multiple branches concurrently	Customer reply + satisfaction survey
Retry strategies	Automatic retry with exponential backoff	Bedrock API calls
`context.logger`	Replay-aware structured logging (suppresses duplicate output during replay)	Throughout

The complete source code is on GitHub: github.com/gunnargrosch/durable-support-triage. Clone the repo and run npm run demo to try the full workflow locally with mocked Bedrock responses, or deploy to AWS and run it with real Bedrock.

Getting Started

You'll need:

An AWS account with credentials configured
AWS SAM CLI 1.153.1 or later (minimum version with DurableConfig support)
Node.js 24 or later
Access to Amazon Bedrock with Claude Haiku 4.5 enabled in your region

Clone and install

git clone https://github.com/gunnargrosch/durable-support-triage.git
cd durable-support-triage
npm install

The SAM Template

Here's the template.yaml:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: AI-powered support ticket triage with durable functions

Parameters:
  BedrockModelId:
    Type: String
    Default: anthropic.claude-haiku-4-5-20251001-v1:0
    Description: Bedrock foundation model ID (uses global inference profile prefix automatically)

Globals:
  Function:
    Timeout: 120
    MemorySize: 256
    Runtime: nodejs24.x

Resources:
  SupportTriageFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: durable-support-triage
      Handler: index.handler
      CodeUri: src/
      DurableConfig:
        ExecutionTimeout: 604800
        RetentionPeriodInDays: 14
      Policies:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
        - Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action:
                - bedrock:InvokeModel
              Resource:
                - !Sub "arn:aws:bedrock:*::foundation-model/${BedrockModelId}"
                - !Sub "arn:aws:bedrock:${AWS::Region}:${AWS::AccountId}:inference-profile/global.${BedrockModelId}"
      AutoPublishAlias: live
      Environment:
        Variables:
          BEDROCK_MODEL_ID: !Sub "global.${BedrockModelId}"
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: false
        Target: es2022
        EntryPoints:
          - index.ts

Outputs:
  FunctionArn:
    Value: !GetAtt SupportTriageFunction.Arn
  AliasArn:
    Value: !Ref SupportTriageFunctionAliaslive

A few things to note about this template:

Globals.Timeout: 120 is the standard Lambda invocation timeout. It applies to each individual invocation (each replay round), not the overall workflow. Two minutes is plenty for each replay round. ExecutionTimeout in DurableConfig is the total wall-clock time for the entire durable execution.
DurableConfig is the only new property compared to a standard Lambda function. ExecutionTimeout is in seconds (604,800 = 7 days). The individual callback timeouts handle the per-step boundaries, but the execution timeout is your outer safety net. A ticket that needs specialist review might sit over a weekend, so 7 days gives headroom. RetentionPeriodInDays controls how long execution history is kept (1 to 90 days).
AutoPublishAlias: live automatically creates a Lambda version and alias on each deploy. This is important for two reasons: durable functions require a qualified ARN (with version or alias) for invocation, and Lambda pins each execution to the version that started it. If you deploy new code while an execution is suspended, replay still uses the original version. This prevents inconsistencies from code changes mid-workflow.
AWSLambdaBasicDurableExecutionRolePolicy is an AWS managed policy that grants the checkpoint and state permissions your function needs (lambda:CheckpointDurableExecutions, lambda:GetDurableExecutionState) plus the standard CloudWatch Logs permissions.
Bedrock IAM uses a cross-region inference profile (global. prefix on the model ID) so that Bedrock routes requests to whichever region has capacity. The policy needs two resource ARNs: the foundation model (wildcard region, no account ID) and the inference profile (your region and account). The BedrockModelId parameter defaults to Claude Haiku 4.5 but you can override it at deploy time with --parameter-overrides BedrockModelId=<model-id>. Check Bedrock model availability for what's enabled in your region.

The RISEN Prompt

The AI triage uses Amazon Bedrock with Claude Haiku 4.5 to analyze incoming tickets. I'm using the RISEN framework for the system prompt. RISEN structures prompts into five components: Role, Instructions, Steps, Expectation, and Narrowing. Each component serves a specific purpose, and together they produce consistent, structured output that your code can reliably parse.

Here's the system prompt for the triage agent:

const TRIAGE_SYSTEM_PROMPT = `
# Role
You are a senior technical support analyst with 10 years of experience
triaging customer support tickets for a SaaS platform. You specialize in
categorizing issues by severity, identifying root causes, and drafting
professional responses.

# Instructions
Analyze the incoming support ticket and produce a structured triage
assessment with category, priority, sentiment, a suggested response,
and an escalation recommendation.

# Steps
1. Read the ticket subject and body to identify the core issue.
2. Categorize the issue (billing, technical, account, feature-request, other).
3. Assess priority based on business impact and urgency (critical, high, medium, low).
4. Evaluate customer sentiment (frustrated, neutral, positive).
5. Draft a suggested response that acknowledges the issue and outlines next steps.
6. Determine whether the ticket needs specialist escalation.

# Expectation
Return a JSON object with this exact structure:
{
  "category": "billing" | "technical" | "account" | "feature-request" | "other",
  "priority": "critical" | "high" | "medium" | "low",
  "sentiment": "frustrated" | "neutral" | "positive",
  "suggestedResponse": "string",
  "needsEscalation": boolean,
  "escalationReason": "string or null",
  "summary": "One-sentence summary of the issue"
}

# Narrowing
- Return only raw JSON. Do not wrap it in markdown code fences, backticks,
  or any other formatting. No explanation, no preamble, no commentary.
- Do not fabricate account details or order numbers not present in the ticket.
- Do not promise refunds, credits, or policy exceptions in the suggested response.
- needsEscalation MUST be false unless one of these exact conditions is met:
  1. The ticket describes confirmed or suspected data loss.
  2. The ticket describes a security breach, unauthorized access, or credential compromise.
  3. The ticket involves a legal or compliance issue.
  4. The customer tier is "enterprise".
  For all other tickets (billing issues, bugs, feature requests, general questions),
  needsEscalation MUST be false regardless of priority or sentiment.
- Keep the suggested response under 200 words.
`;

The Narrowing section does the heavy lifting for reliability. The explicit numbered escalation conditions prevent the model from over-escalating standard tickets (without these, Haiku flagged a routine CSV bug for specialist review). The constraint against promising refunds keeps the AI from making commitments that only a human should make. The pipe syntax in the Expectation section ("billing" | "technical" | ...) is instructional for the model, not literal JSON. It tells the model which values are valid without requiring a separate schema document. Note that "return only raw JSON" doesn't guarantee it: some models still wrap output in markdown code fences despite the instruction. The handler strips them defensively before parsing.

The Handler

Here's the handler from src/index.ts, trimmed to show the durable execution primitives. The full source (input validation, response parsing, integration stubs) is in the repo. Types are defined in src/types.ts. The TRIAGE_SYSTEM_PROMPT shown in the RISEN section above is defined at module scope.

import {
  withDurableExecution, DurableContext,
  createRetryStrategy, JitterStrategy, defaultSerdes,
} from "@aws/durable-execution-sdk-js";
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
import type { TicketEvent, TriageResult, AgentReview, SpecialistReview, TicketResolution } from "./types";

const bedrock = new BedrockRuntimeClient();

async function closeTicket(
  context: DurableContext,
  name: string,
  email: string,
  ticketId: string,
  response: string
): Promise<void> {
  await context.parallel(name, [
    {
      name: "send-reply",
      func: async (ctx) =>
        ctx.step("reply", async () => {
          await sendCustomerReply(email, ticketId, response);
        }),
    },
    {
      name: "send-survey",
      func: async (ctx) =>
        ctx.step("survey", async () => {
          await sendSatisfactionSurvey(email, ticketId);
        }),
    },
  ]);
}

export const handler = withDurableExecution(
  async (event: TicketEvent, context: DurableContext): Promise<TicketResolution> => {
    validateEvent(event);

    context.logger.info("Ticket received", {
      ticketId: event.ticketId,
      customerTier: event.customerTier,
    });

    // Step 1: AI analyzes the ticket using Bedrock
    const analysis = await context.step(
      "analyze-ticket",
      async () => {
        const response = await bedrock.send(new InvokeModelCommand({
          modelId: process.env.BEDROCK_MODEL_ID,
          contentType: "application/json",
          accept: "application/json",
          body: JSON.stringify({
            anthropic_version: "bedrock-2023-05-31",
            max_tokens: 1024,
            system: TRIAGE_SYSTEM_PROMPT,
            messages: [
              {
                role: "user",
                content: `Ticket ID: ${event.ticketId}\nCustomer Tier: ${event.customerTier}\nSubject: ${event.subject}\n\n${event.body}`,
              },
            ],
          }),
        }));
        return parseBedrockResponse(response.body as Uint8Array);
      },
      {
        retryStrategy: createRetryStrategy({
          maxAttempts: 3,
          initialDelay: { seconds: 2 },
          maxDelay: { seconds: 30 },
          backoffRate: 2.0,
          jitter: JitterStrategy.FULL,
        }),
      }
    );

    // Step 2: Support agent reviews AI suggestion
    const agentReview = await context.waitForCallback<AgentReview>(
      "agent-review",
      async (callbackId) => {
        await notifyAgent({ callbackId, ticketId: event.ticketId, analysis, customerTier: event.customerTier });
      },
      { timeout: { hours: 8 }, serdes: defaultSerdes }
    );

    const finalResponse = agentReview.editedResponse || analysis.suggestedResponse;

    if (!agentReview.approved) {
      return { status: "rejected", ticketId: event.ticketId, category: analysis.category, priority: analysis.priority, finalResponse: "" };
    }

    // Step 3: If escalation needed, wait for specialist
    if (analysis.needsEscalation) {
      const specialistResponse = await context.waitForCallback<SpecialistReview>(
        "specialist-review",
        async (callbackId) => {
          await notifySpecialist({ callbackId, ticketId: event.ticketId, analysis, agentNotes: agentReview.agentNotes });
        },
        { timeout: { days: 3 }, serdes: defaultSerdes }
      );

      const resolvedResponse = specialistResponse.response || finalResponse;
      await closeTicket(context, "close-escalated-ticket", event.contactEmail, event.ticketId, resolvedResponse);
      return { status: "escalated", ticketId: event.ticketId, category: analysis.category, priority: analysis.priority, finalResponse: resolvedResponse };
    }

    // Step 4: Send reply and survey in parallel
    await closeTicket(context, "close-ticket", event.contactEmail, event.ticketId, finalResponse);
    return { status: "resolved", ticketId: event.ticketId, category: analysis.category, priority: analysis.priority, finalResponse };
  }
);

Let's walk through what's happening:

withDurableExecution wraps your async function and returns a standard Lambda handler. The runtime calls it like any other handler; the SDK intercepts the execution to manage checkpoints. The BedrockRuntimeClient is instantiated at module scope, which is standard Lambda practice for connection reuse across warm-start invocations. Each replay is a new Lambda invocation, but it may reuse a warm container just like any regular invocation.

validateEvent runs before the first context.step(). This is intentional: if the payload is malformed, the execution fails immediately instead of after the Bedrock step has already been checkpointed. Durable executions that fail after partial checkpointing are harder to reason about than ones that fail fast. Validation is deterministic and cheap, so re-running it on every replay is harmless.

context.step("analyze-ticket", ...) calls Amazon Bedrock with the RISEN prompt and checkpoints the result. The retry strategy handles transient Bedrock API errors (throttling, temporary unavailability) with exponential backoff. parseBedrockResponse handles the response parsing separately: it strips markdown code fences (some models wrap JSON output despite the prompt instruction), validates the response structure, and gives clear error messages on parse failures. If the function replays later, this step returns the cached analysis without calling Bedrock again. That matters for both cost and consistency: you don't want the AI to produce a different triage on replay.

context.waitForCallback("agent-review", ...) is where the function suspends. The SDK creates a callback ID and passes it to your submitter function (which sends it to Slack, email, or your ticketing UI). The submitter runs exactly once, on the invocation that creates the callback. On replay, the SDK skips the submitter entirely and returns the callback result directly. This is important: even though the submitter isn't wrapped in a context.step(), it won't re-execute on replay. The SDK then terminates the Lambda function. Compute charges stop. The agent might respond in minutes or hours. When they do, an external system calls SendDurableExecutionCallbackSuccess with their review. Lambda resumes the function from where it left off. The serdes: defaultSerdes option is required for typed callbacks. Without it, the SDK uses passthrough serialization (not JSON.parse) and agentReview.approved would be undefined at runtime even though TypeScript thinks it's a boolean. This isn't documented yet: step() defaults to JSON serdes, but waitForCallback defaults to passthrough. The SDK exports defaultSerdes for this purpose. If the callback times out (8 hours for the agent, 3 days for the specialist), the SDK throws a CallbackTimeoutError. In production, wrap the callback in a try/catch to handle the timeout (re-queue the ticket, notify a manager, or auto-escalate).

If the agent rejects the AI suggestion (approved: false), the workflow returns early with a rejected status without sending a customer reply. Your ticketing system handles the next step (re-queue, reassign, or manual response).

The escalation path adds a second waitForCallback. If the AI flagged the ticket for escalation (security concern, data loss, enterprise customer), the function suspends again waiting for a specialist. The specialist sends back a SpecialistReview with a response and notes. This callback has a 3-day timeout because specialist reviews can take time. Without durable functions, you'd need a separate state machine or database to track which tickets are waiting for specialists.

context.logger replaces console.log. During replay, completed steps don't re-execute, but code outside steps does. context.logger suppresses duplicate log output during replay so your CloudWatch Logs stay clean. With console.log, you'd see the same log lines repeated on every replay invocation.

closeTicket extracts the parallel close-out into a helper. Both the escalation and standard paths send a customer reply and satisfaction survey concurrently using context.parallel. Each branch gets its own child context with isolated state tracking. The helper takes a dynamic context name so the escalated and standard close-out steps are distinguishable in the execution history.

Testing Locally

The testing SDK (@aws/durable-execution-sdk-js-testing) lets you run durable functions locally without deploying. Here's the key pattern from src/index.test.ts, showing how to drive a callback-based workflow in a test:

import { LocalDurableTestRunner } from "@aws/durable-execution-sdk-js-testing";

// jest.mock replaces BedrockRuntimeClient so analyze-ticket returns controlled results
// (see full mock setup in the repo)
import { handler } from "./index";

describe("Support Triage", () => {
  let runner: LocalDurableTestRunner;

  beforeAll(async () => {
    await LocalDurableTestRunner.setupTestEnvironment({ skipTime: true });
  });

  afterAll(async () => {
    await LocalDurableTestRunner.teardownTestEnvironment();
  });

  beforeEach(() => {
    runner = new LocalDurableTestRunner({ handlerFunction: handler });
  });

  afterEach(async () => {
    await runner.reset();
  });

  it("should resolve a standard ticket after agent review", async () => {
    const result = runner.run({
      payload: {
        ticketId: "TKT-001",
        customerId: "CUST-123",
        customerTier: "pro",
        subject: "Cannot export CSV reports",
        body: "When I click the export button, nothing happens.",
        contactEmail: "customer@example.com",
      },
    });

    // The handler suspends at waitForCallback("agent-review").
    // getOperation blocks until the callback is created.
    const agentCallback = await runner.getOperation("agent-review");
    const agentDetails = await agentCallback.waitForData();
    await agentDetails.sendCallbackSuccess(JSON.stringify({
      approved: true,
      editedResponse: "We have identified the CSV export issue and a fix is rolling out today.",
      agentNotes: "Known bug, fix in deploy pipeline",
    }));

    const output = await result;
    expect(output.getStatus()).toBe("SUCCEEDED");
    expect(output.getResult()).toMatchObject({ status: "resolved", ticketId: "TKT-001" });
  });

  // Additional tests in the repo: escalation flow, agent rejection, callback failure
});

setupTestEnvironment and teardownTestEnvironment are static methods that start and stop the local checkpoint server. They run once per test file in beforeAll/afterAll. The runner instance is created per test in beforeEach and reset in afterEach. The skipTime: true option fast-forwards any context.wait() calls so tests run instantly.

The interesting part is the callback interaction: runner.getOperation("agent-review") blocks until the handler reaches waitForCallback("agent-review") and creates the callback. Then sendCallbackSuccess simulates the external system responding. This lets you test the full suspend/resume lifecycle without deploying. The repo includes tests for all three paths: standard resolution, escalation with specialist review, and agent rejection. There's also a test that uses sendCallbackFailure to verify error handling when an external system reports a failure.

Run the tests with npm test, or use the run-durable CLI to run the handler directly with a payload:

npx run-durable --skip-time --verbose --event '{"ticketId":"TKT-001","subject":"Test ticket"}' src/index.ts

Try the Demo

The repo includes an interactive demo that runs the entire workflow in your terminal, showing each stage: AI analysis, human review prompts, checkpoint history, and final resolution.

Run npm run demo to open the interactive menu, or skip it with a direct command:

Local mode (no AWS credentials needed)

npm run demo:local -- --ticket=standard

Local mode uses mocked Bedrock responses so you can see the full workflow without calling AWS. Here's what the first few steps look like:

  ──────────────────────────────────────────────
  Ticket TKT-001
  Customer:  CUST-123 (pro)
  Subject:   Cannot export CSV reports
  ──────────────────────────────────────────────

  ▶ step: analyze-ticket
    ✓ Checkpointed analyze-ticket (1204ms)

  ──────────────────────────────────────────────
  AI Triage Result
  Category:    technical
  Priority:    high
  Sentiment:   frustrated
  Escalation:  No
  ──────────────────────────────────────────────

  ⏸ waitForCallback: agent-review
    Function suspended. Compute charges stopped.

The demo pauses at each callback and prompts you to respond as the agent or specialist. It walks through three ticket scenarios:

Standard ticket (pro tier, CSV export bug): AI analyzes, agent approves with edits, reply sent.
Enterprise escalation (security concern): AI flags for escalation, agent approves, specialist reviews, reply sent.
Agent rejection (feature request): AI suggests a response, agent rejects, ticket returned for manual handling.

At each human review step, the demo pauses and lets you play the role of the support agent or specialist. You approve, reject, or edit the AI's suggestion. The demo shows the execution history after each step so you can see the checkpoint/replay model in action.

Cloud mode (real Bedrock responses)

npm run demo:cloud -- --ticket=standard --region=us-east-2

Cloud mode invokes the deployed Lambda function with real Bedrock calls. You'll see actual AI-generated triage analysis and can watch the durable execution checkpoints in the Lambda console. Add --profile=<name> if you're not using the default AWS profile.

Deploying and Invoking

Build and deploy:

sam build
sam deploy --guided

Once deployed, invoke the function asynchronously with a qualified ARN. The AutoPublishAlias in the template created a live alias:

aws lambda invoke \
  --function-name durable-support-triage:live \
  --invocation-type Event \
  --durable-execution-name "ticket-TKT-001" \
  --cli-binary-format raw-in-base64-out \
  --payload '{
    "ticketId": "TKT-001",
    "customerId": "CUST-123",
    "customerTier": "pro",
    "subject": "Cannot export CSV reports",
    "body": "When I click the export button, nothing happens.",
    "contactEmail": "customer@example.com"
  }' \
  response.json

A few things to note:

--invocation-type Event is required for long-running workflows. Synchronous invocation (RequestResponse) times out after 15 minutes.
--durable-execution-name provides built-in idempotency. If you invoke the function twice with the same execution name, the second invocation returns the existing execution instead of creating a duplicate. Using the ticket ID as the execution name is a natural fit. Note: execution names must be alphanumeric, hyphens, or underscores. If your ticket IDs contain dots, slashes, or other special characters, sanitize them first.
:live on the function name is the alias qualifier. Without it, you'll get InvalidParameterValueException: Durable execution requires qualified function identifier.

Monitoring execution progress

Check execution status in the Lambda console under the Durable executions tab. You'll see each step's status and timing, including when the function suspended waiting for the agent callback.

You can also check programmatically:

aws lambda get-durable-execution \
  --durable-execution-arn "arn:aws:lambda:us-east-2:123456789012:function:durable-support-triage:live/durable-execution/ticket-TKT-001/<run-id>"

Replace <run-id> with the run ID returned in the initial invoke response. You can also find it in the Lambda console under the Durable executions tab.

Completing the callbacks

When the support agent finishes their review, send the callback from your ticketing system:

aws lambda send-durable-execution-callback-success \
  --callback-id "your-callback-id-here" \
  --cli-binary-format raw-in-base64-out \
  --result '{
    "approved": true,
    "editedResponse": "Hi, thanks for reporting this. We have identified the issue and a fix is rolling out today.",
    "agentNotes": "Known bug in CSV export module"
  }'

The --result value is a JSON string. The CLI handles serialization, so you pass the JSON object directly (unlike the test SDK, where you explicitly call JSON.stringify()).

The function resumes, checks for escalation, and continues. If a specialist callback is pending, the same pattern applies: the specialist's system calls send-durable-execution-callback-success when their review is complete. There's also send-durable-execution-callback-failure for when an external system needs to report an error (e.g., agent rejects the ticket, or an integration fails).

You can also react to execution state changes via EventBridge. Lambda emits events to the default event bus when executions start, succeed, fail, or time out. Create an EventBridge rule with this event pattern:

{
  "source": ["aws.lambda"],
  "detail-type": ["Durable Execution Status Change"]
}

Gotchas and Hard-Won Lessons

Determinism is not optional

This is the rule that trips people up. Code outside steps re-executes on every replay. If it produces different results each time, your workflow breaks in subtle ways.

// This generates a different ID on each replay
const requestId = crypto.randomUUID();
await context.step("use-id", async () => saveToDb(requestId));

// This generates one ID, checkpoints it, and returns the same value on replay
const requestId = await context.step("gen-id", async () => crypto.randomUUID());
await context.step("use-id", async () => saveToDb(requestId));

The same applies to Date.now(), Math.random(), API calls, and database queries. If it can return different values, wrap it in a step.

The ESLint plugin (@aws/durable-execution-sdk-js-eslint-plugin) catches common violations. Set it up early.

Closure mutations are lost on replay

Variables you modify inside a step are not preserved across replays:

let total = 0;
await context.step("calculate", async () => {
  total = 42; // This mutation is lost on replay
});
console.log(total); // Always 0 on replay

// Instead, return values from steps
total = await context.step("calculate", async () => 42);
console.log(total); // Always 42

This is because the step function doesn't re-execute on replay. It returns the cached result. But the closure variable was modified by the function body, which never ran. Return values from steps instead of modifying outer variables.

The qualified ARN requirement is easy to miss

Durable functions require a qualified function identifier: a version number, alias, or $LATEST. Using an unqualified ARN silently fails or throws InvalidParameterValueException.

The AutoPublishAlias SAM property solves this. It creates a new version and updates the alias on every deploy. If you're using EventBridge Scheduler or other services to invoke your function, make sure they target the alias ARN, not the unqualified ARN.

More steps means slower replay

Every time your function resumes, the SDK replays from the beginning, returning cached results for completed steps. The more steps you have, the more replay overhead per resumption.

This is a trade-off. More granular steps give you better debuggability and more precise retry boundaries. Fewer steps give you faster replay. In practice, group related operations into a single step unless you need separate retry behavior or checkpoint boundaries.

context.wait() is the simplest superpower

The "How Checkpoint/Replay Works" section mentions context.wait() for fixed-duration pauses, but the handler only uses waitForCallback(). In practice, context.wait() is one of the most useful primitives. Need a 24-hour cooling-off period before sending a follow-up? One line:

await context.wait("cooling-off", { hours: 24 });

The function suspends, compute charges stop, and Lambda resumes it 24 hours later. No cron jobs, no EventBridge Scheduler, no polling.

You can't enable durable execution on existing functions

DurableConfig can only be set when creating a function. You can't toggle it on an existing function. If you need to migrate, you'll need to create a new function. Plan for this.

Changing DurableConfig in CloudFormation also requires resource replacement, not an in-place update.

Checkpoint payloads have a 256KB limit

Each step result is serialized and stored as a checkpoint. If a step returns an object larger than 256KB, you'll get a CheckpointUnrecoverableExecutionError.

The workaround: store large data in S3 or DynamoDB and return a reference from the step.

const dataRef = await context.step("store-large-data", async () => {
  const key = `tickets/${ticketId}/attachments.json`;
  await s3.putObject({ Bucket: bucket, Key: key, Body: JSON.stringify(largeData) });
  return { bucket, key };
});

Know your failure modes

If a step throws an unrecoverable error (after all retries are exhausted), the execution moves to a FAILED state. You can inspect the error in the Lambda console or via the get-durable-execution API. For cases where you want to fail immediately without retries, throw an UnrecoverableInvocationError:

import { UnrecoverableInvocationError } from "@aws/durable-execution-sdk-js";
throw new UnrecoverableInvocationError("Customer account not found");

Failed executions can't be resumed. You'd need to start a new execution. Design your workflows so that steps are idempotent in case you need to re-run from scratch.

Use Lambda versions for deploy safety

If you update your function code while an execution is suspended, replay uses the version that started the execution. This prevents inconsistencies from code changes mid-workflow. AutoPublishAlias handles this, but it's worth understanding why: if your new code changes a step's return shape or removes a step, replay on the old version still works because Lambda pins executions to their starting version.

Version pinning protects your function code, but it doesn't protect external schemas. If an execution suspends on Monday waiting for a callback, and on Wednesday your ticketing system starts sending a different JSON structure in the callback payload, the Monday execution will fail when it resumes. Keep callback payloads backwards compatible for as long as executions can be in flight.

Plan your observability early

For production workflows that can run for days, go beyond basic CloudWatch Logs. Set up CloudWatch alarms on stuck executions (no state change within expected timeframes), use the EventBridge integration to track execution lifecycle events, and consider CloudWatch Logs Insights queries for filtering by execution name across replays.

When to Use What

Durable functions and Step Functions are not competing. They solve different problems.

	Durable Functions	Step Functions
Workflow definition	Sequential code in your language	Amazon States Language (JSON/YAML)
Best for	Application logic, tightly coupled workflows	Cross-service orchestration
Service integrations	Via SDK in your code	220+ native integrations
Debugging	CloudWatch Logs, execution history	Visual console, step-by-step
Infrastructure	Single Lambda function	State machine + Lambda functions
Scaling	Lambda concurrency limits	Distributed Map for large-scale parallel processing
Mental model	Write code	Design state machines

On cost: durable functions use standard Lambda pricing for active compute time. During waits, compute charges stop. Step Functions charges per state transition, which adds up for high-volume workflows. See Lambda pricing and Step Functions pricing for current details.

Use durable functions when your workflow is application logic that reads naturally as sequential code. Support ticket triage, approval workflows, AI agent loops, saga patterns.

Use Step Functions when you're orchestrating across multiple AWS services, need visual debugging, or need the 220+ native integrations. ETL pipelines, media processing, infrastructure provisioning.

Use both together when you have a high-level orchestration (Step Functions) that delegates to individual workflows (durable functions) for complex application logic.

What's Next

This post covered the fundamentals: what durable functions are, how checkpoint/replay works, and how to build and test a complete AI-powered workflow with human-in-the-loop callbacks. In the next post, I'll use durable functions to build a multi-agent orchestration workflow where multiple AI agents collaborate on complex tasks with checkpointed reasoning.

Additional Resources

What workflows are you thinking about building with durable functions? Let me know in the comments!

Chaos Engineering for AWS Lambda: failure-lambda 1.0

Gunnar Grosch — Thu, 12 Mar 2026 16:44:15 +0000

I wrote the first version of failure-lambda back in 2019. The idea was simple: inject faults into AWS Lambda functions so you can test how your system behaves when things go wrong. Latency spikes, exceptions, blocked network calls. The kind of failures that happen in production whether you're ready for them or not.

That version worked. People used it. But the codebase was showing its age. JavaScript with no types. AWS SDK v2. A flat configuration format that only allowed one failure mode at a time. And it only worked with Node.js.

failure-lambda 1.0 is a ground-up rewrite. TypeScript, AWS SDK v3, a feature flag configuration model, two new failure modes (timeout and corruption), and a Lambda Layer that brings fault injection to any managed runtime with zero code changes.

Why Chaos Engineering for Lambda?

If you're building on Lambda, you're building on a managed service. You don't manage servers, but you still manage dependencies. Your function calls DynamoDB, S3, third-party APIs, other microservices. Any of those can be slow, unreliable, or unavailable.

The question isn't whether failures will happen. It's whether your system handles them gracefully when they do. Does your function retry correctly? Does your circuit breaker trip? Does your API return a useful error message instead of a 500?

Failure injection lets you answer those questions before your users do. Enable latency injection and watch your downstream timeouts. Block a dependency with the denylist and see if your fallback logic works. Return a 503 and check that your retry policy backs off properly.

The important part: you control when and how these failures happen. Start with one mode at a low percentage in a test environment. Increase gradually. Build confidence that your system does what you think it does.

What's New

The short version: everything. TypeScript with full type definitions. AWS SDK v3. Two new failure modes.

The old format was flat: one failure mode, one rate, one toggle. The new format treats each mode as an independent feature flag. You can enable latency injection at 50% and DNS denylist at 100% simultaneously:

{
  "latency": {"enabled": true, "percentage": 50, "min_latency": 200, "max_latency": 500},
  "denylist": {"enabled": true, "deny_list": ["s3.*.amazonaws.com"]}
}

Seven failure modes:

Mode	Effect
`latency`	Adds random delay between configured bounds
`timeout`	Sleeps until the function is about to time out
`exception`	Throws an exception with a configurable message
`statuscode`	Returns a specific HTTP status code
`diskspace`	Fills `/tmp` to consume available disk space
`denylist`	Blocks network calls to matching hostnames
`corruption`	Mangles the response body after the handler returns

Beyond the modes:

Event-based targeting. Match conditions restrict injection to specific requests. Only corrupt GET requests to the prod stage. Only add latency to requests hitting /api. Conditions support exact match, exists, startsWith, and regex operators.
```
{
  "latency": {
    "enabled": true,
    "min_latency": 200,
    "max_latency": 500,
    "match": [{"path": "requestContext.http.path", "operator": "startsWith", "value": "/api"}]
  }
}
```
AppConfig Feature Flags. Native support for the AWS.AppConfig.FeatureFlags profile type. AppConfig gives you deployment strategies and automatic rollback, useful when you don't want an accidental "enable all failures at 100%" to take down your environment.
Middy middleware. If you use Middy, import failure-lambda/middy instead of wrapping your handler.
CLI. npx failure-lambda gives you an interactive CLI for managing failure configuration. Check status, enable modes, disable everything. Supports both SSM and AppConfig backends and saves connection profiles so you don't retype region and parameter names every time.

The Lambda Layer

This is the biggest addition. The npm package requires you to import failure-lambda and wrap your handler. That's fine for Node.js, but it doesn't help if your Lambda functions are written in Python, Java, .NET, or Ruby.

The Lambda Layer solves this. Add the layer to your function, set two environment variables, and fault injection works without touching your code. No imports, no wrapper, no middleware.

AWS_LAMBDA_EXEC_WRAPPER=/opt/failure-lambda-wrapper
FAILURE_INJECTION_PARAM=failureLambdaConfig

Under the hood, it's a lightweight Rust proxy that sits between your handler and the Lambda Runtime API. Single static binary, no runtime dependencies, negligible cold start impact. On each invocation, the proxy reads your failure configuration from SSM or AppConfig and decides whether to inject faults before or after forwarding the request to your handler. Your code never knows it's there.

┌─────────────────────────────────────────────────┐
│ Lambda Execution Environment                    │
│                                                 │
│  Lambda Runtime API                             │
│       │                                         │
│       ▼                                         │
│  failure-lambda proxy (Rust)                    │
│       │                                         │
│       ├── Read config from SSM / AppConfig      │
│       ├── Inject fault? ──yes──▶ Return early   │
│       │                         (statuscode,    │
│       │                          exception,     │
│       │                          latency)       │
│       no                                        │
│       │                                         │
│       ▼                                         │
│  Your handler (any runtime)                     │
│       │                                         │
│       ├── Inject fault? ──yes──▶ Modify response│
│       │                         (corruption)    │
│       no                                        │
│       │                                         │
│       ▼                                         │
│  Response returned                              │
└─────────────────────────────────────────────────┘

The layer works with all managed Lambda runtimes that support AWS_LAMBDA_EXEC_WRAPPER: Node.js, Python, Java, .NET, and Ruby. Both x86_64 and arm64 architectures. Custom runtimes can use the layer if they support AWS_LAMBDA_EXEC_WRAPPER: the runtime bootstrap must check for the variable and invoke the specified executable before starting its own runtime loop. Download the zip from the GitHub release, publish it to your account, and you're ready to go.

Getting Started: npm Package

For Node.js, the npm package gives you the most control.

You'll need:

Node.js 20 or later
An AWS account with permissions to create SSM parameters and Lambda functions
ssm:GetParameter granted to the function's execution role

npm install failure-lambda

Wrap your handler:

import failureLambda from "failure-lambda";

export const handler = failureLambda(async (event, context) => {
  return { statusCode: 200, body: "OK" };
});

Create an SSM parameter with your failure configuration:

aws ssm put-parameter \
  --name failureLambdaConfig \
  --type String \
  --value '{"latency": {"enabled": false, "min_latency": 100, "max_latency": 400}}' \
  --region eu-west-1

Set the FAILURE_INJECTION_PARAM environment variable on your Lambda function to the parameter name, grant ssm:GetParameter, and deploy.

When you're ready to inject a fault:

npx failure-lambda enable latency --param failureLambdaConfig --region eu-west-1

Getting Started: Lambda Layer

For any runtime, the layer path requires no code changes.

You'll need:

An AWS account with permissions to publish Lambda layers and create Lambda functions
AWS CLI configured
ssm:GetParameter granted to the function's execution role

Download failure-lambda-layer-x86_64.zip or failure-lambda-layer-aarch64.zip from the latest release.

Publish the layer:

aws lambda publish-layer-version \
  --layer-name failure-lambda \
  --zip-file fileb://failure-lambda-layer-x86_64.zip \
  --compatible-architectures x86_64 \
  --region eu-west-1

Add the layer ARN to your function, set AWS_LAMBDA_EXEC_WRAPPER=/opt/failure-lambda-wrapper and FAILURE_INJECTION_PARAM=failureLambdaConfig, create the SSM parameter, and grant ssm:GetParameter.

That's it. Your Python handler, your Java handler, your .NET handler: they all get the same fault injection capabilities without a single line of code changed.

Try It Out: Injecting Faults Step by Step

Let's walk through a concrete example. We'll deploy a simple function with the layer, verify it works normally, inject latency, inject a status code error, and then turn everything off. The whole thing uses a standard Node.js handler with zero failure-lambda code in it.

Here's the handler. It simulates a quick database lookup and returns the response time:

export const handler = async (event) => {
  const start = Date.now();
  await new Promise((resolve) => setTimeout(resolve, 5));
  return {
    statusCode: 200,
    body: JSON.stringify({
      message: "Order processed",
      duration_ms: Date.now() - start,
    }),
  };
};

The SAM template adds the layer and sets the two required environment variables. Nothing else:

DemoFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: index.handler
    Runtime: nodejs22.x
    CodeUri: src/
    Timeout: 10
    Layers:
      - !Ref FailureLambdaLayerArn
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/failure-lambda-wrapper
        FAILURE_INJECTION_PARAM: !Ref FailureConfig
    Policies:
      - SSMParameterReadPolicy:
          ParameterName: !Ref FailureConfig

The full template including Parameters definitions is in the examples directory. The snippet above shows only the function resource for clarity.

After deploying with sam build && sam deploy --guided, we hit the endpoint a few times to see steady state:

The examples below use curl -s -o - -w ' HTTP %{http_code} | %{time_total}s\n' to append status code and total request time to each response. Standard curl won't show this without the -w flag.

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":5}   HTTP 200 | 0.21s

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":5}   HTTP 200 | 0.19s

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":6}   HTTP 200 | 0.19s

5ms handler duration, ~190ms end-to-end on warm invocations. That's our baseline. Now let's see what happens when conditions aren't ideal.

Injecting latency

A third-party API starts responding slowly. A DynamoDB table is throttling. A downstream microservice is under load. These are real scenarios, and you want to know how your system behaves before they happen in production.

Update the SSM parameter to add 500-1000ms of random latency on every invocation:

aws ssm put-parameter --name <your-param-name> --type String --overwrite \
  --value '{"latency": {"enabled": true, "percentage": 100, "min_latency": 500, "max_latency": 1000}}' \
  --region eu-west-1

Configuration is cached for 60 seconds. If you update the SSM parameter and immediately hit the endpoint, you'll see the old behavior. Wait for the cache to refresh before testing.

Wait about 60 seconds for the configuration cache to refresh, then hit the endpoint again:

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":11}   HTTP 200 | 0.99s

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":5}    HTTP 200 | 1.08s

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":5}    HTTP 200 | 1.14s

Responses jumped from ~190ms to over a second. The handler itself still runs in 5ms: the latency is injected by the proxy before the handler executes. This simulates what happens when a dependency responds slowly. Does your API Gateway timeout kick in at the right threshold? Do callers retry or give up? Does a slow function cause a queue to back up?

Injecting a status code error

Latency is one thing. Complete failure is another. A downstream service returning 5xx errors, an expired API key, a misconfigured endpoint: all of these surface as error responses. Replace the config with a 503 Service Unavailable:

aws ssm put-parameter --name <your-param-name> --type String --overwrite \
  --value '{"statuscode": {"enabled": true, "percentage": 100, "status_code": 503}}' \
  --region eu-west-1

After the cache refreshes (up to 60 seconds):

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Injected status code 503"}   HTTP 503 | 0.31s

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Injected status code 503"}   HTTP 503 | 0.21s

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Injected status code 503"}   HTTP 503 | 0.18s

The handler never runs. The proxy short-circuits the invocation and returns a 503 directly. This represents a function that's failing for any reason: a permissions change, a missing environment variable, an unhandled exception. Does your frontend show a useful error message or a blank page? Does your Step Functions workflow retry or fail the entire execution? Does your monitoring alert you?

What the logs show

The proxy writes structured JSON logs to CloudWatch. You can see exactly what it's doing on each invocation:

{"source":"failure-lambda","action":"config","config_source":"ssm","enabled_flags":"[]"}
{"source":"failure-lambda","action":"config","config_source":"ssm","enabled_flags":"[\"latency\"]"}
{"source":"failure-lambda","mode":"latency","action":"inject","latency_ms":670,"min_latency":500.0,"max_latency":1000.0}
{"source":"failure-lambda","mode":"latency","action":"inject","latency_ms":859,"min_latency":500.0,"max_latency":1000.0}
{"source":"failure-lambda","mode":"latency","action":"inject","latency_ms":933,"min_latency":500.0,"max_latency":1000.0}
{"source":"failure-lambda","action":"config","config_source":"ssm","enabled_flags":"[\"statuscode\"]"}
{"source":"failure-lambda","mode":"statuscode","action":"inject","status_code":503}
{"source":"failure-lambda","mode":"statuscode","action":"inject","status_code":503}
{"source":"failure-lambda","mode":"statuscode","action":"inject","status_code":503}

Every config fetch and every injection is logged with the mode, action, and parameters. You can query these in CloudWatch Logs Insights:

fields @timestamp, mode, action
| filter source = "failure-lambda"
| sort @timestamp desc

Turning it off

The CLI can disable everything in one command:

npx failure-lambda disable --all --param <your-param-name> --region eu-west-1

Or set the parameter back to an empty config manually:

aws ssm put-parameter --name <your-param-name> --type String --overwrite \
  --value '{}' --region eu-west-1

After the cache refreshes, everything is back to normal:

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":5}   HTTP 200 | 0.33s

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":5}   HTTP 200 | 0.23s

$ curl https://ij58ovr5s5.execute-api.eu-west-1.amazonaws.com/
{"message":"Order processed","duration_ms":5}   HTTP 200 | 0.19s

No redeployment. No code changes. The proxy saw the empty config, disabled injection, and passed everything through.

Cleaning up

To remove everything deployed in this walkthrough:

sam delete
aws ssm delete-parameter --name <your-param-name> --region eu-west-1
aws lambda delete-layer-version --layer-name failure-lambda --version-number <version> --region eu-west-1

What You Learn

The walkthrough above uses a single function. A real application with multiple functions, queues, and dependencies will surface more. But even one function reveals things about your system:

Timeout behavior (latency, timeout). Are your Lambda timeouts, API Gateway timeouts, and client timeouts configured consistently? Latency injection exposes mismatches fast. A function with a 10-second timeout behind an API Gateway with a 3-second timeout will fail in ways that look intermittent.
Retry and backoff (statuscode, exception). When a function returns a 503, do callers retry with exponential backoff or hammer the endpoint? Do SQS redrive policies work as configured? Injecting errors at a percentage less than 100% lets you see if partial failures are handled differently than complete outages.
Error propagation (statuscode, exception). Does a failure in one function produce a clear error message at the API boundary, or does it cascade into a generic 500? Injecting status codes and exceptions at different points in a call chain shows you exactly where error context gets lost.
Alerting and observability (any mode). Do your CloudWatch alarms fire? Do they fire quickly enough? Injecting faults and watching your dashboards is the most direct way to validate your monitoring. If you don't get paged during a controlled experiment, you won't get paged during an incident.
Fallback behavior (denylist). If you've built fallback logic for when a dependency is unavailable, does it actually work? The denylist mode blocks specific hostnames, so you can test what happens when S3 or DynamoDB is unreachable without affecting other dependencies.
Capacity and scaling (latency). What happens when latency increases and concurrent executions climb? Do you hit reserved concurrency limits? Does a slow function cause upstream queues to grow? These are the kinds of cascading effects that are hard to predict and easy to test.

The point of chaos engineering isn't to cause outages. It's to discover how your system responds to conditions that will eventually occur, in a controlled way, before your users encounter them.

Troubleshooting

Injection isn't happening

The most common cause is a missing ssm:GetParameter permission on the function's execution role. Check CloudWatch Logs for a permission denied error from the proxy. The second most common cause is the configuration cache: changes take up to 60 seconds to take effect. If you've just updated the SSM parameter, wait for the next cache refresh before concluding injection isn't working.

Architecture mismatch

If you publish the x86_64 layer and attach it to an arm64 function (or vice versa), the proxy binary won't execute. Download the correct zip for your function's architecture: failure-lambda-layer-x86_64.zip or failure-lambda-layer-aarch64.zip.

AWS_LAMBDA_EXEC_WRAPPER has no effect

The AWS_LAMBDA_EXEC_WRAPPER mechanism is built into managed Lambda runtimes. Custom runtimes need to explicitly support it by checking for the variable and invoking the wrapper before starting their own runtime loop. If you're using a custom runtime that doesn't implement this, the layer won't intercept anything.

Design Decisions

A few things I learned building this.

Feature flags over a single toggle

The original version had one failure mode active at a time. That's not how production fails. In the real world, you might have a slow dependency and a flaky DNS resolution at the same time. The feature flag model lets you compose failures. Each mode is independent with its own percentage, so you can build realistic failure scenarios.

Why Rust for the layer

The proxy sits in the critical path of every Lambda invocation. It needs to be fast with minimal memory overhead. Rust was the natural choice: predictable performance, no garbage collector pauses, and the single binary keeps the layer small.

Caching with a purpose

Every invocation used to call SSM to get the configuration. That's unnecessary latency and API costs. The library now caches SSM responses for 60 seconds by default. For AppConfig, the Lambda extension already handles caching, so the library disables its own cache entirely to avoid staleness.

Validation that fails closed

If your configuration JSON is malformed or has invalid values, the library logs a clear error and disables injection. It doesn't crash your function and it doesn't silently inject with bad parameters. Regex patterns in denylist rules are checked for nested quantifiers to prevent ReDoS.

What's Next

failure-lambda 1.0 brings TypeScript, a feature flag configuration model, seven failure modes, and a Lambda Layer that works across all managed runtimes without touching your code. This release covers the core use cases I've seen in practice. There are things I'd like to explore next: more granular targeting with Lambda function aliases, integration with AWS Fault Injection Service, and better observability into what's being injected across a fleet of functions. If you have ideas, open an issue.

Additional Resources

What failure scenarios have you tested in your serverless applications? I'm curious what surprises people find with timeout mismatches and retry behavior. Let me know in the comments.

Can It Run DOOM? Playing DOOM in Claude Code with doom-mcp

Gunnar Grosch — Wed, 11 Mar 2026 02:04:03 +0000

DOOM runs on everything. Calculators, ATMs, pregnancy tests, an oscilloscope you bought at a garage sale. It was only a matter of time before AI coding assistants joined the list. Now they have.

doom-mcp is an MCP server that embeds the real DOOM engine and exposes it to Claude Code, Kiro, Cursor, or any MCP client through three tools. The AI sees the game, fights demons, navigates levels, and reports what's happening. You can direct it or let it play on its own.

> Let's play DOOM!

  Starting E1M1 on medium difficulty...

  HP:100 Armor:0 | Pistol (key:2) | Ammo: 50b 0s 0r 0c | Kills:0 | (-416,256) 0deg | E1M1
  No enemies in sight. Exploring the corridor ahead.

> Go forward and find some enemies

  Sprinting down the hallway... I can see an opening ahead.

  HP:100 Armor:0 | Pistol (key:2) | Ammo: 50b 0s 0r 0c | Kills:0 | (0,256) 0deg | E1M1
  ENEMIES IN SIGHT (aim for angle~0 then fire): Imp (HP:60) to your left (turn_left ~9) nearby

  There's an Imp to the left! Turning to face it...

> Shoot it!

  Aimed and firing!

  HP:100 Armor:0 | Pistol (key:2) | Ammo: 47b 0s 0r 0c | Kills:0 | (0,256) 30deg | E1M1
  ENEMIES IN SIGHT (aim for angle~0 then fire): Imp (HP:15) AHEAD close

  Hit! The Imp is wounded (HP:15). One more burst should finish it.

  HP:95 Armor:0 | Pistol (key:2) | Ammo: 44b 0s 0r 0c | Kills:1 | (0,256) 30deg | E1M1
  ITEMS: Stimpack (+10 HP) slightly to your right (turn_right ~4) point-blank (~3 ticks fwd+run to reach)
  1 kill! Want to see a screenshot?

Getting Started

Register with Claude Code

claude mcp add doom --scope user -- npx -y doom-mcp

For Kiro, Cursor, Windsurf, or any other MCP client, add to .mcp.json:

{
  "mcpServers": {
    "doom": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "doom-mcp"]
    }
  }
}

Play

Open a new session and say: "Let's play DOOM"

The AI will ask which mode you want, start on E1M1, and begin.

Play Modes

Two ways to play:

User-directed: You give commands ("go forward", "open that door", "shoot the imp"). The AI executes one action at a time and describes what happens. Good for a text-adventure feel where you call the shots and the AI handles the execution.

Autonomous: The AI makes all decisions: movement, combat, exploration. You watch and intervene if you want. It's genuinely entertaining to watch it work through a level, spot an Imp, and decide whether to charge or take cover.

WAD Files

doom-mcp ships with Freedoom out of the box. Freedoom is a free and open-source replacement IWAD (DOOM's game data format) with its own levels and enemy designs. If you want the original id Software levels, enemies, and atmosphere, the shareware DOOM1.WAD is free to download legally. Set the path in your MCP config:

{
  "mcpServers": {
    "doom": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "doom-mcp"],
      "env": {
        "DOOM_WAD_PATH": "/path/to/DOOM1.WAD"
      }
    }
  }
}

If you own DOOM or DOOM 2, those WADs work the same way.

A Few Things I Learned Building This

FFI over subprocess

The obvious approach is to run DOOM as a child process and communicate via pipes. The problem is timing and synchronization: you're fighting the engine's internal clock, process startup overhead, and serialization on every frame.

Instead, doom-mcp embeds doomgeneric (a portable C implementation of the DOOM engine) directly via Rust FFI (Foreign Function Interface). Rust was the right choice here: it has excellent FFI support for C code, compiles to a single native binary, and gives memory safety without a garbage collector that could interrupt the game loop. No subprocess spawning, no pipes. Each tool call advances the engine by calling doomgeneric_Tick() directly and reading the frame buffer in-memory.

Virtual time

DOOM normally ties its game clock to wall time. That's fine for a real-time player, but it's wrong for an AI that might take 500ms to decide its next move. Without intervention, the engine would skip ticks during the AI's thinking time and produce non-deterministic behavior.

The solution is to decouple the engine's clock from wall time entirely. Each doomgeneric_Tick() call advances exactly one game tic (1/35th of a second) regardless of how much real time has passed. Gameplay is fully deterministic: the same sequence of actions always produces the same result.

Line-of-sight, not wallhack

Enemy detection could just iterate the object list and report everything on the map. That would be cheating in a way that makes the game too easy and less interesting.

Instead, the server performs a proper line-of-sight check for each enemy, the same check the DOOM engine uses internally (P_CheckSight()). The AI only sees enemies it could see if it were a human looking at the screen. When an enemy moves behind a wall or around a corner, it drops from the AI's view immediately. It still needs to explore to find things.

What the AI gets per action

Each doom_action call returns structured game state alongside a small PNG:

HP, armor, ammo by type, current weapon, kill count
Visible enemies with human-readable direction and distance (Imp (HP:60) to your left (turn_left ~9) nearby)
Nearby items within pickup range, with CLOSING/RECEDING indicators
Nearby doors and switches (NEARBY: Door AHEAD (use to activate))
A 200x125 thumbnail PNG using a 216-color palette for inference

The structured data gives the AI something it can reason about without having to interpret pixel-level vision. The image fills in the spatial context. Together they let the AI make reasonable decisions: "Imp to my left, nearby, HP 60 — turn left and fire."

Tools Reference

doom_start

Starts or restarts a game. Safe to call at any time.

Parameter	Type	Default	Description
`skill`	int 1-5	3	1=baby, 2=easy, 3=medium, 4=hard, 5=nightmare
`episode`	int 1-4	1	Episode number
`map`	int 1-9	1	Map number

doom_action

Advances the game by executing actions for a number of ticks.

Parameter	Type	Required	Description
`actions`	string	yes	Comma-separated: `forward`, `backward`, `turn_left`, `turn_right`, `strafe_left`, `strafe_right`, `fire`, `use`, `run`, `1`-`7`
`ticks`	int 1-105	no	Ticks to advance. Default 7. 7 ticks ≈ 0.2s, 35 ticks ≈ 1s

Weapon keys: 1=fists, 2=pistol, 3=shotgun, 4=chaingun, 5=rocket launcher, 6=plasma, 7=BFG.

doom_screenshot

Saves a full-resolution 320x200 screenshot to the system temp directory and opens it in the default image viewer. Does not advance the game. Note: the viewer launch will fail silently on headless systems or SSH sessions.

How Well Does It Actually Play?

Realistically: well enough to be fun. On E1M1 at medium difficulty, it gets 5-10 kills in a typical 50-action session. It can navigate corridors, spot enemies, aim, and fire. It struggles with enemies behind partial cover and complex door sequences.

It improves significantly when you direct it. "There's an Imp to your left" turns a wandering AI into a focused combatant. The user-directed mode is where most of the entertainment is. Two AI agents in deathmatch is the obvious next experiment, and the architecture could extend to other doomgeneric-compatible titles: Heretic, Hexen, DOOM II.

The token cost is real: each action call is roughly 1,500-2,500 total tokens (input and output combined: game state text plus the PNG). A 50-action session is 75,000-125,000 tokens, which works out to roughly $0.50-2.00 depending on your model. Worth it.

Additional Resources

doom-mcp on GitHub: Source, docs, and examples
doom-mcp on npm: Package page
doomgeneric by ozkl: The portable DOOM engine this is built on
Freedoom: The open-source IWAD that ships with the package
DOOM1.WAD shareware download: The original shareware episode, free and legal

The real question was never whether it could run DOOM. It's what you do with it now that it can. Let me know in the comments how far you get.

Circuit Breakers on AWS Lambda: Why In-Memory State Silently Fails

Gunnar Grosch — Mon, 09 Mar 2026 21:53:44 +0000

You added a circuit breaker to your Lambda function. It compiles, your tests pass, and it works correctly in local testing. But it's silently useless. The problem isn't the implementation. It's an assumption every in-memory circuit breaker makes that doesn't hold on Lambda.

What Circuit Breakers Do

The circuit breaker pattern comes from Michael Nygard's Release It! and is named after the electrical component. Think about the services your Lambda functions actually call: a payment processor, a third-party enrichment API, a database under load, another service in your own fleet. Anything external your function depends on is a downstream service, and any of them can start responding slowly or fail outright. Slow is often worse than down. A dependency that takes 10 seconds to time out costs you 10 seconds of held concurrency per call, not a fast failure you can handle gracefully.

That concurrency cost is the Lambda-specific reason to care. When a downstream call hangs, your function holds a concurrency unit. At 100 concurrent executions and a 10-second timeout, one flaky dependency can saturate your function in seconds, throttling every other request, including the ones with nothing to do with the sick service. The cascade happens fast: payment API slows down → order function saturates concurrency → order requests fail → the service calling orders backs up → users see errors across your entire checkout flow.

When a downstream service starts failing, a circuit breaker stops calling it entirely, returns a fallback response immediately, and probes for recovery. It also gives the downstream service breathing room: instead of a flood of timeouts hammering something that's already struggling, it gets near-silence while the circuit is open. The naming follows the electrical analogy: a closed circuit is complete and current flows; an open circuit is broken and nothing gets through.

Three states:

CLOSED: Normal operation. Calls go through. Failures are counted.
OPEN: Circuit tripped. Calls fail fast without reaching the downstream service. A timeout runs.
HALF-OPEN: One trial call allowed. If it succeeds, the circuit closes. If it fails, it reopens with a longer timeout.

The problem is how Lambda runs code.

Why In-Memory State Fails on Lambda

Lambda's concurrency model is built around isolated execution environments. From the AWS documentation: "For each concurrent request, Lambda provisions a separate instance of your execution environment." Two simultaneous invocations of the same function run in two separate environments with completely independent memory spaces. There is no shared memory between them.

Consider what this means for a circuit breaker with a failure threshold of 5. Your function is receiving 50 concurrent requests. A downstream service starts failing:

Execution environment 1 takes a request. The call fails. Its local failure count: 1/5.
Execution environment 2 takes a request. The call fails. Its local failure count: 1/5.
...
Execution environment 50 takes a request. The call fails. Its local failure count: 1/5.

50 failures have hit the downstream service. No circuit has opened. Each environment has counted 1 failure and needs 4 more before it does anything. Meanwhile, all 50 environments continue sending requests to a service that is already failing. In the worst case, where traffic distributes evenly across environments, you need up to 250 total failures before any single execution environment opens its circuit.

And that's assuming the same 50 execution environments handle all the traffic. Lambda scales by adding new execution environments as load increases. Each new environment starts with a failure count of zero. As long as traffic grows and new environments spin up, the fleet will always have environments that haven't seen enough failures to open. The circuit can never effectively protect you across the fleet.

This isn't a hypothetical. The most widely-used Node.js circuit breaker libraries (opossum, cockatiel) store state in process memory. They work correctly in a single-process server where all traffic goes through one circuit. They don't work for Lambda's distributed execution model. opossum does provide state export and import hooks (toJSON()) specifically documented for serverless environments, but these don't solve the cross-environment isolation problem: each environment still starts from whatever state you restore, not a live shared view of current circuit state.

Provisioned Concurrency reduces but doesn't eliminate this problem. PC keeps a fixed number of execution environments initialized and warm, so they accumulate local failure counts across more requests than standard on-demand environments. But they're still isolated from each other, and scaling events still add fresh environments that start at zero. In-memory state is less useless with PC, but it's still wrong at any meaningful concurrency level.

Lambda also periodically terminates execution environments for runtime maintenance and updates, even for continuously invoked functions. An environment accumulating failure counts can be replaced with a fresh one starting at zero at any time, adding another layer of unreliability to in-memory state.

Lambda Managed Instances (launched at re:Invent 2025) are an exception: they support multiple concurrent invocations per environment, so in-memory state accumulates across requests within the same environment. The argument above applies to standard Lambda functions, which remain the default.

Shared State Across Execution Environments

The fix is to store circuit state in a shared external store. When execution environment 1 records a failure, execution environment 2 sees it. When any environment opens the circuit, every environment stops calling the downstream service. Yes, this adds a network call to every invocation. The Performance and Cost sections have the numbers. For most workloads the overhead is small, and CachedProvider can reduce it further. ElastiCache (Valkey) is the fastest option (sub-millisecond reads) and is the right choice if your functions are already in a VPC. DynamoDB is the right default for most Lambda workloads: no VPC required, single-digit millisecond latency, and it supports atomic operations and conditional writes for concurrent safety. Adding a VPC solely for circuit breaker state adds deployment complexity and a modest cold start overhead, which isn't worth it unless you're already VPC-attached.

circuitbreaker-lambda is an open-source library I built that takes the DynamoDB path. It stores circuit state in DynamoDB and shares it across all execution environments running the same function.

Two paths to choose from:

	npm package	Lambda Layer
Runtimes	Node.js 20+	Any managed runtime
Integration	Import library	HTTP calls to local sidecar
Cold start overhead	~50ms	~350ms

Both paths share the same DynamoDB state schema, so a Node.js function using the npm package and a Python function using the Layer can share circuit state for the same downstream service.

Getting Started: npm Package

Install

npm install circuitbreaker-lambda

Requires Node.js 20+.

Create a DynamoDB table

aws dynamodb create-table \
  --table-name circuitbreaker-table \
  --attribute-definitions AttributeName=id,AttributeType=S \
  --key-schema AttributeName=id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

Set the environment variable

Add CIRCUITBREAKER_TABLE as a Lambda environment variable. In a SAM template:

Environment:
  Variables:
    CIRCUITBREAKER_TABLE: circuitbreaker-table

Grant the function access to DynamoDB

The function needs GetItem and UpdateItem on the table. In a SAM template:

Policies:
  - Statement:
      - Effect: Allow
        Action:
          - dynamodb:GetItem
          - dynamodb:UpdateItem
        Resource: !GetAtt CircuitBreakerTable.Arn

Use it in your handler

import { CircuitBreaker } from 'circuitbreaker-lambda'

// callDownstreamService is any async function that calls your downstream service.
// It should throw on failure; the circuit breaker catches the throw and counts it.
// Initialized outside the handler so the same instance
// is reused across warm invocations of this execution environment
const breaker = new CircuitBreaker(callDownstreamService, {
  failureThreshold: 5,
  successThreshold: 2, // successes (across any environment) required to close from HALF-OPEN
  timeout: 10000, // ms to wait in OPEN state before allowing a trial call (HALF-OPEN)
  fallback: async () => ({ data: 'cached response' }),
})

export const handler = async () => {
  try {
    const result = await breaker.fire()
    return { statusCode: 200, body: JSON.stringify(result) }
  } catch (err) {
    // fire() throws if the circuit is OPEN and no fallback is configured,
    // or if the downstream call fails and propagates the error.
    return { statusCode: 503, body: JSON.stringify({ error: 'Service unavailable' }) }
  }
}

fire() calls callDownstreamService. If the call succeeds, it records a success in DynamoDB. If it fails, it records a failure. When the failure count hits the threshold, it opens the circuit and subsequent calls return the fallback immediately (or throw if no fallback is configured). Every execution environment handling that function reads the same DynamoDB state. successThreshold counts successes across any execution environment via shared DynamoDB state, following the same last-writer-wins behavior as failure counts. Real fallbacks return something useful under degradation: cached data, a default empty state, or a simplified response. The { data: 'cached response' } placeholder in the example is where that goes.

If you're using Middy middleware, there's an integration that wraps your handler directly:

import middy from '@middy/core'
import { circuitBreakerMiddleware } from 'circuitbreaker-lambda/middy'

export const handler = middy(myHandler)
  .use(circuitBreakerMiddleware({
    failureThreshold: 5,
    fallback: async () => ({ statusCode: 503, body: 'Service unavailable' }),
  }))

The middleware wraps the entire handler rather than a specific downstream function. When the circuit is OPEN, the middleware short-circuits the handler before it runs and returns the fallback response. Without a fallback configured, it throws so your error handler can respond. If your handler calls multiple downstream services, use the npm package directly with a distinct circuit ID for each service.

Circuit IDs and Shared State

The circuit ID is what links circuit state to a specific downstream service. By default it uses AWS_LAMBDA_FUNCTION_NAME. Two execution environments running the same function share one circuit because they have the same function name and read from the same DynamoDB item.

If one function calls multiple downstream services, give each a distinct circuit ID:

const paymentBreaker = new CircuitBreaker(callPaymentService, {
  circuitId: 'payment-service',
})
const inventoryBreaker = new CircuitBreaker(callInventoryService, {
  circuitId: 'inventory-service',
})

If multiple functions protect the same downstream service and you want them to share a circuit, give them the same ID. A circuit open in one function will be seen by all functions using that ID. For the Lambda Layer, use the same circuit ID string in the HTTP path across all functions.

Getting Started: Lambda Layer

If your functions use a runtime other than Node.js, or if you want a single circuit breaker deployment that works across runtimes, the Lambda Layer is the other path. It ships a Rust extension that runs as a local sidecar on port 4243. Your handler makes HTTP calls to it instead of importing a library.

Add the layer to your SAM template

Download the layer zip from the GitHub releases page. The Rust extension is architecture-specific: download the x86_64 build for standard Lambda functions or the arm64 build for Graviton. Reference it as a AWS::Serverless::LayerVersion resource and attach it to your function. The examples/layer/template.yaml in the repo shows the full setup with both architectures. The key function configuration:

LayerNodeFunction:
  Type: AWS::Serverless::Function
  Properties:
    Layers: [!Ref CircuitBreakerLayer]
    Environment:
      Variables:
        CIRCUITBREAKER_TABLE: !Ref CircuitBreakerTable

Node.js handler

const CIRCUIT_ID = process.env.AWS_LAMBDA_FUNCTION_NAME
const CB_URL = 'http://127.0.0.1:4243'

// Check circuit state before calling downstream
const check = await fetch(`${CB_URL}/circuit/${CIRCUIT_ID}`)
const { allowed, state } = await check.json()

if (!allowed) {
  return { statusCode: 503, body: JSON.stringify({ error: 'Circuit OPEN', state }) }
}

// Call downstream and report result
try {
  const result = await callDownstream()
  await fetch(`${CB_URL}/circuit/${CIRCUIT_ID}/success`, { method: 'POST' })
  return { statusCode: 200, body: JSON.stringify(result) }
} catch (err) {
  await fetch(`${CB_URL}/circuit/${CIRCUIT_ID}/failure`, { method: 'POST' })
  throw err  // Lambda returns a non-200; event sources like SQS will retry
}

Python handler

import json
import os
import urllib.request

circuit_id = os.environ.get('AWS_LAMBDA_FUNCTION_NAME', 'default')
cb_url = 'http://127.0.0.1:4243'

def handler(event, context):
    try:
        with urllib.request.urlopen(f'{cb_url}/circuit/{circuit_id}', timeout=5) as resp:
            check = json.loads(resp.read())
    except Exception:
        # Sidecar unavailable — fail open and allow the downstream call
        check = {'allowed': True}

    if not check['allowed']:
        return {'statusCode': 503, 'body': json.dumps({'error': 'Circuit OPEN'})}

    try:
        result = call_downstream()
        urllib.request.urlopen(
            urllib.request.Request(f'{cb_url}/circuit/{circuit_id}/success', method='POST'),
            timeout=5
        )
        return {'statusCode': 200, 'body': json.dumps(result)}
    except Exception as e:
        urllib.request.urlopen(
            urllib.request.Request(f'{cb_url}/circuit/{circuit_id}/failure', method='POST'),
            timeout=5
        )
        # For event-driven triggers like SQS, raise here instead so Lambda retries.
        return {'statusCode': 500, 'body': json.dumps({'error': str(e)})}

Both examples make two HTTP calls to the sidecar per invocation: one to check state before the downstream call, one to report the result. These are loopback calls to 127.0.0.1, not network calls, so the round-trip is sub-millisecond. The Rust sidecar also runs the CachedProvider logic internally, so it rarely reaches DynamoDB on warm invocations. The warm latency numbers in the Performance section reflect this.

The Layer approach requires more boilerplate per handler, but it works in any managed runtime and keeps the state management and DynamoDB logic out of your application code. The handler wires up local HTTP calls to the sidecar. That part does live in your code. But the actual circuit state tracking, DynamoDB reads and writes, failure counting, and backoff logic are all inside the Rust extension.

Design Decisions

Fail-open

The term comes from physical security, not circuit states: a fail-open lock releases when power fails, defaulting to permissive. Here it means the same thing. If the DynamoDB state provider is unavailable, requests pass through rather than failing. This is a deliberate trade-off. The alternative is failing closed: a transient DynamoDB error takes down your service even if the downstream service it's protecting is completely healthy. Your circuit breaker becomes a single point of failure.

Fail-open accepts that brief periods of unprotected calls are better than self-inflicted downtime. State provider errors are logged as structured JSON so you can monitor and alert on them, but they don't block requests. The counter-argument: if DynamoDB is unavailable during an active downstream incident, fail-open leaves traffic unprotected. For most workloads this is the right call: a simultaneous DynamoDB outage and downstream failure is an unlikely combination, and failing closed (blocking all traffic because the circuit breaker can't read state) makes things worse. If your downstream is fragile enough that this scenario is a real concern, a lower-level fallback or degraded mode is a better answer than fail-closed.

For the Lambda Layer path, there are two sidecar failure modes to distinguish. If the extension fails during the INIT phase, Lambda restarts the execution environment entirely. The handler never runs, and Lambda retries automatically. If the extension crashes after initialization during an invocation, fetch calls to http://127.0.0.1:4243 throw connection refused errors. For this second case, wrap the sidecar calls in a try/catch and fail open: allow the downstream call to proceed. The same principle applies as with DynamoDB unavailability.

Warm invocation caching

Every fire() call reads circuit state from DynamoDB. For a function handling high throughput, that's a DynamoDB read on every invocation. You can reduce this with the CachedProvider, which caches state in memory for warm execution environments:

const breaker = new CircuitBreaker(callDownstream, {
  cacheTtlMs: 200, // use cached state for 200ms on warm invocations
})

On a warm invocation, the cache is checked first. If the state is fresh, no DynamoDB call is made. The cache is write-through: when state is saved to DynamoDB, the cache is also updated. Keep the TTL short. A long cache window can delay the CLOSED to OPEN transition: an execution environment that cached a CLOSED state won't see a newly-opened circuit until the cache expires. 200ms is a reasonable starting point: it caps the detection lag while cutting DynamoDB reads significantly for high-throughput functions. Increase the TTL to reduce costs further at the cost of slower circuit detection. Decrease it for faster propagation at higher DynamoDB cost.

What the DynamoDB item looks like

When debugging a stuck circuit, this is what you're looking for in the table:

{
  "id": "my-function-name",
  "circuitState": "OPEN",
  "failureCount": 5,
  "successCount": 0,
  "nextAttempt": 1741234567890,
  "lastFailureTime": 1741234557890,
  "consecutiveOpens": 1
}

circuitState is CLOSED, OPEN, or HALF-OPEN. nextAttempt is a Unix timestamp in milliseconds. The circuit won't probe until after that time. consecutiveOpens tracks how many consecutive HALF-OPEN→OPEN transitions have occurred, which drives the exponential backoff on the timeout.

The library uses last-writer-wins writes rather than atomic increments. Under extreme concurrent failures (many execution environments failing at the exact same millisecond) some failure counts can be lost: if 10 environments all read failureCount: 4 and each write 5, the count advances by 1 instead of 10. In practice this means the circuit may take slightly longer to open than the threshold suggests under burst concurrency. It will still open. For the CLOSED→OPEN transition itself, multiple environments writing OPEN simultaneously all succeed, which is fine: you want the circuit open. Atomic counter increments via DynamoDB's ADD operation could prevent lost failure counts, but a state transition updates multiple fields simultaneously: state, failure count, and timestamp. Last-writer-wins on the full item keeps the write logic simple at the cost of occasional lost counts under extreme concurrency. If your function handles high burst concurrency, set failureThreshold lower than you would in a single-process application. Lost counts mean the effective threshold is higher than the configured value, so a lower setting brings the actual behavior closer to the intended one.

Exponential backoff on repeated failures

When a circuit transitions from HALF-OPEN back to OPEN (a recovery probe failed), the timeout before the next probe doubles. This prevents a repeatedly-failing service from being probed too aggressively. The backoff resets when the circuit closes successfully. The maxTimeout option caps how long the backoff can grow.

HALF-OPEN probe behavior

With shared DynamoDB state, when the circuit transitions to HALF-OPEN, every warm execution environment that reads the updated state may attempt a trial call. Unlike a single-process circuit breaker where exactly one probe goes out, a fleet of 50 environments can send up to 50 simultaneous probes to a recovering downstream service. CachedProvider staggers probes across the TTL window as environments pick up the state change at different times, but doesn't eliminate the burst. A single-leader approach (using a DynamoDB conditional write to claim the probe slot) would be more precise, and it's tracked as a future improvement in the repo. The current behavior favors simplicity: the probe burst is proportional to the number of warm environments, which is typically small for functions with reasonable traffic patterns, and distributed leader election adds significant complexity for a probe that's designed to be retried on failure anyway.

Custom state backends

The StateProvider interface is pluggable. If you need Redis, a relational database, or anything else, implement two methods:

class RedisProvider implements StateProvider {
  async getState(circuitId: string): Promise<CircuitBreakerState | undefined> { ... }
  async saveState(circuitId: string, state: CircuitBreakerState): Promise<void> { ... }
}

const breaker = new CircuitBreaker(fn, { stateProvider: new RedisProvider() })

DynamoDB is the right default for most Lambda workloads. Valkey or Redis makes sense if you're already VPC-attached and running ElastiCache for caching: reusing existing infrastructure avoids the extra DynamoDB dependency. For most teams, running a cache cluster solely for circuit state isn't worth the VPC overhead and operational cost.

Performance

Using the npm package and Lambda Layer, here are measured results from a test run with 50 warm invocations per configuration and a shared DynamoDB table in the same region. All functions were configured at 512MB memory. The "downstream" in all cases was an HTTP call through an API Gateway endpoint backed by DynamoDB, which could be toggled healthy or unhealthy. The HTTP round-trip through API Gateway accounts for the ~590ms baseline. Raw DynamoDB read latency is single-digit milliseconds. Cold start times scale inversely with memory allocation. Lambda allocates CPU proportionally to memory, so at 128MB (where CPU is highly constrained) you would expect larger overhead, particularly for the Layer which initializes a Rust extension sidecar alongside the function runtime.

Cold start (forced by updating a function environment variable):

Configuration	Cold start	vs. baseline
Baseline (no circuit breaker)	1300ms	—
npm package (Node.js)	1353ms	+4%
Lambda Layer (Node.js)	1679ms	+29%
Lambda Layer (Python)	1541ms	+18%

The Layer cold start penalty comes from initializing the Rust extension sidecar alongside the function runtime. It's a one-time cost per execution environment. Since August 2025, AWS bills for the Lambda INIT phase on managed runtimes with ZIP deployment packages, so the Layer's +29% cold start overhead (379ms) is now both a latency and a cost consideration for functions with frequent cold starts.

Warm invocations (50 calls each):

Configuration	Median	p99
Baseline (no circuit breaker)	590ms	620ms
npm package (Node.js)	592ms	621ms
Lambda Layer (Node.js)	589ms	797ms
Lambda Layer (Python)	585ms	639ms

The npm package p99 (621ms) is essentially identical to baseline (620ms). The Lambda Layer Node.js p99 (797ms) is higher because the Rust extension sidecar occasionally adds latency on the first few invocations after a warm start. The median is fine but the tail is longer. The Layer configurations showing slightly below baseline median are within measurement noise, not a genuine speedup. With CachedProvider, DynamoDB reads are eliminated for subsequent invocations within the TTL window, which brings tail latency down for high-throughput functions.

Cost

Each fire() call reads circuit state from DynamoDB (one GetItem) and writes on state changes. At on-demand pricing, a GetItem on a small item costs $0.125 per million read request units. At one million Lambda invocations per day (around 11 RPS), that's roughly $0.125/day for the reads. State writes only happen on failures and state transitions, so they're a rounding error for a healthy function. During an active failure scenario writes increase. At $1.25 per million write request units, a function failing on every invocation could see $1.25/day in write costs before the circuit opens and stops the calls. In practice the circuit opens quickly (after the first threshold of failures per environment), so write volume drops sharply once OPEN. With CachedProvider at 200ms TTL and warm execution environments, reads drop by an order of magnitude on high-throughput functions.

Testing

The package includes a MemoryProvider for unit testing. Pass it as the stateProvider option to skip DynamoDB entirely in tests:

import { CircuitBreaker, MemoryProvider } from 'circuitbreaker-lambda'

const breaker = new CircuitBreaker(callDownstream, {
  stateProvider: new MemoryProvider(),
  failureThreshold: 3,
})

MemoryProvider uses an in-memory Map and is not safe for production. It's for tests and local development only.

When NOT to Use a Circuit Breaker

Not every Lambda function needs one:

Very low concurrency. If your function runs at a single execution environment (low traffic, no bursting), in-memory circuit breakers work: there's only one environment, so state is effectively shared. The overhead of distributed state isn't worth it for something handling a few requests per minute.
Calls to AWS services. The AWS SDK handles retries, timeouts, and transient failures with exponential backoff. Wrapping a DynamoDB GetItem or an S3 PutObject in a circuit breaker adds complexity without much benefit. AWS manages the resilience layer for you.
When fail-fast isn't better than retry. Circuit breakers are for cascading failure protection. If your function's caller expects a synchronous result and there's no meaningful fallback response, letting the error propagate and retry may be simpler than managing circuit state.

Try It Yourself

The repo includes two deployable SAM examples with a toggleable downstream service so you can watch the full circuit lifecycle (healthy calls, failures accumulating, circuit opening, recovery) against real AWS infrastructure:

examples/sam/: npm package example. Single Node.js function at /. Toggle the downstream at /toggle, check circuit state at /status.
examples/layer/: Layer example. Node.js (/node) and Python (/python) functions side by side, sharing the same Layer and DynamoDB table.
examples/minimal-npm/ and examples/minimal-layer/: Stripped-down versions if you just want the bare minimum code without the toggle/status test infrastructure.

Additional Resources

circuitbreaker-lambda on GitHub: Source, docs, and examples
circuitbreaker-lambda on npm: Package page
Using the circuit-breaker pattern with AWS Lambda extensions and Amazon DynamoDB: AWS Compute Blog post covering the same Lambda extension + DynamoDB architecture
AWS Prescriptive Guidance: Circuit Breaker: AWS' recommended approach using DynamoDB
AWS Lambda execution environment documentation: The concurrency and isolation model this post is based on
Circuit Breaker pattern (Martin Fowler): The canonical pattern reference

The silent failure mode of in-memory circuit breakers on Lambda isn't obvious until you're debugging a production incident. If you're running a circuit breaker today, check whether it's sharing state across execution environments. If it's not, it's not protecting you. The fix is a DynamoDB table and three lines of configuration. The alternative is finding out during the next downstream outage. Let me know in the comments how you're handling downstream resilience on Lambda.

Building Multi-Agent Systems with RISEN Prompts and Strands Agents

Gunnar Grosch — Thu, 05 Mar 2026 20:40:04 +0000

The RISEN post introduced system prompts as behavioral contracts. One reader comment cut to the core of what comes next: "What happens when you have multiple agents that each need their own contract?"

The answer isn't complicated, but it's specific. The Expectation section of one agent defines the input format for the next. Narrowing prevents agents from doing each other's work. Steps encode the routing logic: which specialists to call, when, and why. The contract between agents lives in the prompts, not in orchestration code.

This post builds a working multi-agent system that demonstrates these contracts in practice. Here's what it looks like. Three different purchase requests, three completely different agent journeys:

"I need a 15 inch laptop for work"
  → Calling Price Research Agent
  → Calling Delivery & Logistics Agent
Agents called:  Price Research, Delivery & Logistics
Agents skipped: Financing, Risk Assessment, Contract Review

"I want to buy a VW Golf, probably a used one"
  → Calling Price Research Agent
  → Calling Financing Agent
  → Calling Risk Assessment Agent
  → Calling Delivery & Logistics Agent
Agents called:  Price Research, Financing, Risk Assessment, Delivery & Logistics
Agents skipped: Contract Review

"Looking for office space to rent, two-year lease, around 20 people"
  → Calling Price Research Agent
  → Calling Financing Agent
  → Calling Contract Review Agent
  → Calling Risk Assessment Agent
Agents called:  Price Research, Financing, Contract Review, Risk Assessment
Agents skipped: Delivery & Logistics

Same coordinator, same five specialists available. The coordinator reads the request, decides which ones are needed, and only calls those. The rest of this post explains how.

Agents, Not Workflows

The instinct when building a multi-agent system is to reach for a workflow engine. Define the steps, wire the handoffs, control the flow. This is the opposite of what makes agents useful. A workflow engine decides what happens next. An agent decides what happens next. When you hardcode the routing, you lose the ability for the agent to reason about what's actually needed. A laptop doesn't need risk assessment. A used car does. That's a judgment call, not a branching condition. With hardcoded routing, every new category of purchase means a code change. With the routing in the prompt, the coordinator handles it on its own.

The DEV415 session on A2A and MCP at re:Invent 2025 makes this point in a production context: you design agent behaviors, not control flow. The SwiftShip demo shows it running on AWS with Lambda functions and Agent-to-Agent communication.

The approach in this post is simpler: agents as tools. Each specialist is wrapped as a tool() function that the coordinator can invoke. No HTTP endpoints, no message queues, no infrastructure beyond a single TypeScript process. The pattern is the same as the production architecture. The implementation is small enough to clone and run in two minutes.

When to Split Into Multiple Agents

Not every task needs multiple agents. Here are signals that splitting makes sense:

Different expertise domains. If two sections of your prompt have completely different Narrowing constraints ("only assess security, never comment on performance"), those are different agents.
Different model requirements. If one part of the task needs strong reasoning and another just needs fast summarization, using the same model for both is wasteful.
Conditional execution. If some parts of the work only apply to some inputs, a single agent either does unnecessary work or has complex conditional logic in its Steps. Multiple specialists with a routing coordinator handles this cleanly.
Independent failure isolation. If one specialist fails or times out, the others can still complete. A single agent either succeeds or fails entirely.

If your task fits in one prompt with consistent Narrowing and no conditional branches, keep it as one agent. Adding coordination overhead for its own sake makes the system slower and harder to debug.

RISEN as Coordination Contracts

In a single-agent system, RISEN structures the output. In a multi-agent system, RISEN structures the coordination. Each component does double duty:

Expectation defines the handoff format. What one agent returns is what the next agent reads. The Price Research Agent's Expectation section says "return the typical price range with budget, mid-range, and premium tiers." That's what the coordinator gets back and synthesizes with findings from other specialists.

Narrowing defines ownership boundaries. The Financing Agent's Narrowing says "do not assess market pricing, delivery logistics, contract terms, or product risk." The Risk Assessment Agent's Narrowing says the same about financing, delivery, and contracts. No agent steps on another agent's job. Without this, agents drift into each other's domains and produce redundant or contradictory advice.

Steps encode routing logic. The coordinator's Steps section IS the routing decision. It's not code. It's plain English in the system prompt:

# Steps
1. Read the purchase request and identify: what is being purchased, the
   likely category, the approximate value range, and any special
   circumstances.
2. Always invoke the research_prices tool.
3. If the estimated value exceeds $5,000, or if financing is mentioned,
   invoke the evaluate_financing tool.
4. If the item is a tangible physical product, invoke the plan_delivery
   tool. Physical products always require delivery or collection planning.
5. If the purchase involves a subscription, lease, or multi-year
   commitment, invoke the review_contract tool.
6. If the estimated value exceeds $10,000, the item is used, or the
   category carries known risk, invoke the assess_risk tool.
7. Synthesize all specialist reports into a structured recommendation.

The model reads these instructions, assesses the purchase request against them, and calls the appropriate tools. A $1,000 laptop triggers Steps 2 and 4 (price research and delivery). A used car triggers Steps 2, 3, 4, and 6 (price research, financing, delivery, and risk). An office lease triggers Steps 2, 3, 5, and 6 (price research, financing, contract review, and risk). Delivery is skipped because office space is not a physical product. The routing is conditional and emerges from the prompt.

The Demo: Smart Purchasing Coordinator

The demo has one coordinator agent and five specialist agents. Each specialist is wrapped as a tool() function and passed to the coordinator:

Specialist	When called	Narrowing constraint
Price Research	Always	Only pricing. No risk, financing, or delivery.
Financing	Value > $5K or financing mentioned	Only financing. No pricing or contracts.
Delivery & Logistics	Physical product	Only logistics. No pricing or risk.
Risk Assessment	High value, used goods, or risky category	Only risk. No pricing or delivery.
Contract Review	Subscription, lease, or multi-year commitment	Only contract terms. No pricing or risk.

Agents as tools

The Strands Agents TypeScript SDK doesn't have a built-in agent.asTool() method. Instead, you wrap each specialist using the tool() function. The callback creates a fresh agent, invokes it, and returns its output:

import { tool } from '@strands-agents/sdk'
import { z } from 'zod'
import { invokeSpecialist } from './create-specialist-agent.js'
import { riskAssessmentPrompt } from '../prompts/risk-assessment.js'
import { ADVANCED_MODEL } from '../models.js'

export const assessRisk = tool({
  name: 'assess_risk',
  description: 'Identifies purchase risks, recommends due diligence steps, '
    + 'and estimates realistic total cost of ownership.',
  inputSchema: z.object({
    item: z.string().describe('What is being purchased'),
    condition: z.enum(['new', 'used', 'refurbished', 'unknown']),
    estimatedValue: z.string(),
    riskContext: z.string().optional(),
  }),
  callback: async (input) =>
    invokeSpecialist(riskAssessmentPrompt, input, { modelId: ADVANCED_MODEL }),
})

The invokeSpecialist helper creates the agent, invokes it with a 60-second timeout, and returns the string output. The shared helper keeps each tool wrapper to a few lines:

export async function invokeSpecialist(
  systemPrompt: string,
  input: Record<string, unknown>,
  options?: SpecialistOptions
): Promise<string> {
  const agent = new Agent({
    model: createModel(options?.modelId ?? SPECIALIST_MODEL),
    systemPrompt,
    tools: options?.tools,
    printer: false,
  })

  let timer: ReturnType<typeof setTimeout>

  // Timeout produces a rejection, not partial output.
  // If a specialist times out, the coordinator gets an error, not a half-answer.
  const result = await Promise.race([
    agent.invoke(`Purchase request details:\n${JSON.stringify(input, null, 2)}`),
    new Promise<never>((_, reject) => {
      timer = setTimeout(() => reject(new Error('Specialist timed out')), 60_000)
    }),
  ]).finally(() => clearTimeout(timer))

  return result.toString()
}

Notice two things: the Risk Assessment Agent uses a different model (ADVANCED_MODEL, which defaults to Sonnet 4.6) because risk analysis requires stronger reasoning than standard price research (which runs on Haiku 4.5). And options.tools lets specialists have their own sub-tools. The Price Research Agent has a save_price_snapshot tool that writes structured price data to a local JSON file. The coordinator never sees this tool. It's scoped to the specialist.

The coordinator wiring

The coordinator itself is straightforward. It gets the RISEN prompt, all five specialist tools, and a hook for routing visibility:

const hook = new RoutingHook()
const agent = new Agent({
  model: createModel(COORDINATOR_MODEL),
  systemPrompt: coordinatorPrompt,
  tools: allTools,
  hooks: [hook],
  printer: false,
})

const result = await agent.invoke(request)
printRecommendation(result.toString())
printSummary(hook.getCalledTools())

Hooks for routing visibility

The RoutingHook uses the SDK's BeforeToolCallEvent to print each specialist as the coordinator decides to call it. Readers see the routing decisions happen in real time before the specialist output appears:

export class RoutingHook implements HookProvider {
  private readonly calledTools: string[] = []

  getCalledTools(): string[] {
    return [...this.calledTools]
  }

  registerCallbacks(registry: HookRegistry): void {
    registry.addCallback(BeforeToolCallEvent, (event) => {
      const toolName = event.toolUse.name
      const displayName = allToolNames[toolName]
      if (displayName) {
        this.calledTools.push(toolName)
        console.log(`  → Calling ${displayName} Agent`)
      }
    })
  }
}

The hook also tracks which tools were called, which powers the summary at the end showing called vs. skipped agents. This is observability for multi-agent systems without any infrastructure: one hook provider, attached to the coordinator, watching the decisions flow.

Running It

You saw the routing output at the top: laptop triggers two specialists, used car triggers four, office lease triggers a different four (contract review instead of delivery). Here's what the full output looks like for the used car:

════════════════════════════════════════════════════════════
PURCHASE REQUEST
════════════════════════════════════════════════════════════
"I want to buy a VW Golf, probably a used one"

Coordinator is analyzing your request...

  → Calling Price Research Agent
  → Calling Financing Agent
  → Calling Risk Assessment Agent
  → Calling Delivery & Logistics Agent

════════════════════════════════════════════════════════════
PURCHASING RECOMMENDATION
════════════════════════════════════════════════════════════
[Coordinator synthesis: pricing tiers for used Golfs, financing options
 with monthly payment estimates, risk assessment covering DSG transmission
 and hidden maintenance costs, delivery logistics for vehicle collection]

────────────────────────────────────────────────────────────
Agents called:  Price Research, Financing, Risk Assessment, Delivery & Logistics
Agents skipped: Contract Review
────────────────────────────────────────────────────────────

The coordinator calls four specialists, waits for all of them, then synthesizes their findings into a single recommendation. Each specialist stays in its lane: the Risk Assessment Agent talks about DSG transmission issues and hidden maintenance costs, the Financing Agent talks about loan terms and monthly payments, and neither comments on the other's domain. That separation comes from the Narrowing section of each specialist's RISEN prompt.

The two can also produce tension. Risk might flag a $3,000 first-year repair budget while Financing offers attractive loan terms. The coordinator doesn't resolve that tension. Its Expectation section says to surface findings from each specialist clearly attributed. Presenting both sides is the right call. Choosing one would mean overriding a specialist's domain, which is exactly what Narrowing is supposed to prevent.

Evaluating the Routing

How do you know the coordinator is making the right calls? The same pattern from the eval post applies here: define expected behavior, run it, check the results.

The routing eval replaces the real specialist tools with stubs that return immediately. The coordinator still runs against the real LLM, so this tests the RISEN Steps routing logic without paying for five specialist invocations per case:

const stubTools = Object.keys(allToolNames).map((name) =>
  tool({
    name,
    description: `Stub for ${name}`,
    inputSchema: z.object({ item: z.string() }).passthrough(),
    callback: async () => `[stub response from ${name}]`,
  })
)

Four test cases with expected routing:

ROUTING EVAL
Coordinator: global.anthropic.claude-sonnet-4-6
Test cases: 4 (specialist tools stubbed)

Budget laptop
  PASS called Price Research
  PASS called Delivery & Logistics
  PASS skipped Financing
  PASS skipped Risk Assessment
  PASS skipped Contract Review

Used car with financing
  PASS called Price Research
  PASS called Financing
  PASS called Risk Assessment
  PASS called Delivery & Logistics
  PASS skipped Contract Review

Office lease
  PASS called Price Research
  PASS called Financing
  PASS called Contract Review
  PASS called Risk Assessment
  PASS skipped Delivery & Logistics

SaaS subscription
  PASS called Price Research
  PASS called Contract Review
  PASS skipped Delivery & Logistics

RESULT: 4/4 routing cases passed

The SaaS case checks fewer assertions than the others. Financing and Risk Assessment are omitted from that test because the coordinator's decision on them is borderline for a 50-seat enterprise tool: the annual cost might exceed the financing and risk thresholds depending on how the coordinator estimates the value. The eval only asserts on routing decisions that are unambiguously right or wrong.

The calibration loop caught two prompt issues. The coordinator was calling the Delivery Agent for SaaS subscriptions (a purely digital product). Adding a Narrowing constraint ("Do not invoke DeliveryAgent for purely digital purchases") fixed that. It was also inconsistently calling Delivery for laptops because Step 4 said "requires shipping" rather than asserting that physical products always need delivery planning. Making the Step explicit ("Physical products always require delivery or collection planning, even if the buyer has not mentioned it") stabilized it.

What Changes on AWS (Preview of the Next Post)

This demo runs in a single process. Everything happens in-memory: the coordinator calls tool functions, those functions create specialist agents, the specialists return strings. That's fine for development and for understanding the pattern. But it doesn't scale, it has no fault isolation, and there's no way to monitor or manage the agents independently.

The core pattern stays identical: each specialist is a callable endpoint, the coordinator's tools make HTTP calls instead of function calls, and the routing logic in the RISEN Steps doesn't change at all. Two paths to get there:

Option 1: Lambda with Function URLs or API Gateway. Each specialist becomes a Lambda function exposed over HTTP. You can use Lambda Function URLs (simpler, direct IAM auth) or API Gateway (more control, useful if corporate policy restricts Function URLs). Either way, the coordinator's tool callbacks switch from local invokeSpecialist calls to HTTP requests, and IAM auth restricts invocation to the coordinator's execution role only. The SwiftShip demo uses this pattern with Function URLs: a triage agent calling payment, warehouse, and order agents over HTTP.

Option 2: Amazon Bedrock AgentCore Runtime. AgentCore is a serverless runtime purpose-built for AI agents. Each agent deploys as a containerized Express service inside AgentCore, which handles session isolation per user, automatic scaling, and built-in observability. It supports the Strands TypeScript SDK and the A2A protocol. The deployment model is more involved than Lambda (Docker and ECR are required). Choose it when you need per-user session state, want A2A protocol support without building your own routing layer, or need a runtime that scales agent sessions independently rather than per-request.

In both cases, the RISEN prompts carry over unchanged. The next post walks through a full deployment of this purchasing coordinator demo.

Design Decisions

Per-agent model selection

Not all agents need the same model. The coordinator and Risk Assessment Agent use Sonnet 4.6 for stronger reasoning. Standard specialists (Price Research, Financing, Delivery, Contract Review) use Haiku 4.5, which is faster and cheaper. The model IDs are centralized in models.ts with environment variable overrides:

export const COORDINATOR_MODEL = process.env.COORDINATOR_MODEL_ID
  ?? 'global.anthropic.claude-sonnet-4-6'

export const SPECIALIST_MODEL = process.env.SPECIALIST_MODEL_ID
  ?? 'global.anthropic.claude-haiku-4-5-20251001-v1:0'

export const ADVANCED_MODEL = process.env.ADVANCED_MODEL_ID
  ?? 'global.anthropic.claude-sonnet-4-6'

Sub-agent tools

The Price Research Agent has its own tool (save_price_snapshot) that writes structured price data to a local JSON file. The coordinator never sees this tool. It's scoped to the specialist.

In a production system, that tool could be anything: a call to a live pricing API, a search against a product catalog, a query to a DynamoDB table, a vector search against a knowledge base. The point is that specialist agents aren't just prompt wrappers. Each one can have its own tool surface, scoped to its domain, invisible to the coordinator. The coordinator stays focused on routing. The specialist handles whatever retrieval or action its domain requires.

Stub-based routing eval

Testing routing decisions is cheaper than testing specialist output quality. By stubbing the specialist callbacks, the routing eval runs four coordinator LLM calls instead of twenty. The stubs return immediately, so the eval completes in under a minute. This is practical for the iterative calibration loop.

What's missing: conversation

The coordinator is one-shot. It reads the request, calls specialists, synthesizes, done. A real purchasing advisor would ask follow-up questions: "What year range are you considering?" or "Do you have a trade-in?" After delivering a recommendation, a conversational coordinator could handle "Tell me more about the financing options" by calling just the Financing Agent again with the new context.

The Strands SDK supports multi-turn conversation through the agent's message history. Making this demo conversational would mean wrapping the coordinator invocation in a loop that reads user input and feeds it back to the same agent instance. The RISEN prompts wouldn't change. The coordinator's Steps already describe when to call each specialist, and those decisions would apply on follow-up turns too. The main addition would be a new Step telling the coordinator to ask clarifying questions when the request is ambiguous before routing to specialists.

This is a natural extension but adds enough complexity (input loop, conversation state, deciding when to re-route vs. answer directly) that it's better as a separate iteration than a first demo.

Additional Resources

multi-agent-risen-demo: Demo repo with all code from this post
Writing System Prompts That Actually Work: The RISEN Framework for AI Agents: The RISEN framework post this builds on
Evaluating Agent Output Quality: Lightweight Evals Without a Framework: The eval patterns used for routing evaluation
DEV415: Building Scalable Self-Orchestrating AI Workflows with A2A and MCP: re:Invent 2025 session on production multi-agent architecture
SwiftShip demo: Production A2A demo from the DEV415 session
Strands Agents SDK (TypeScript): Agent framework used in the demo

The contracts between your agents are just prompts. Change the Steps, change the routing. Add a Narrowing constraint, prevent an overlap. No code changes required. That's the payoff of RISEN in a multi-agent context: the coordination logic is readable, editable, and testable without touching the orchestration code.

What purchase would you try first? Run npm start with something unexpected and see which specialists the coordinator calls. Let me know in the comments!

Evaluating Agent Output Quality: Lightweight Evals Without a Framework

Gunnar Grosch — Tue, 03 Mar 2026 16:20:38 +0000

In Writing System Prompts That Actually Work, I ended with this advice: "run it against a few representative inputs and check the output against your Expectation section." That's a good starting point. But if you've iterated on a few prompts, you've probably noticed the problem: eyeballing doesn't scale. You change the Steps section, re-run your test input, skim the output, and think "yeah, that looks better." Then two iterations later you realize you broke something that was working before. You're doing regression testing by memory.

If you haven't read the RISEN post: RISEN is a framework for writing agent system prompts with five components: Role, Instructions, Steps, Expectation, and Narrowing. The key idea is that each component doubles as an eval lever. Expectation defines what the output should look like, so it tells you what structural checks to write. Narrowing defines what the agent should avoid, so it tells you what scope violations to flag.

This post covers practical evaluation patterns for agent output. No heavyweight eval framework required. Three tiers: structural checks you can run in pure code, an LLM-as-judge pattern for content quality, and a calibration loop for tuning both. I'll walk through each tier with a working demo that evaluates a RISEN-structured code review agent that reviews Lambda functions for security vulnerabilities, performance issues, and AWS best practice violations.

Why Eval Before Picking a Framework

Evaluation frameworks exist for a reason. The Strands Agents Python SDK includes an evals package with output evaluators, trajectory evaluators, and benchmark runners. If you're building a production agent in Python with dozens of test cases and CI integration, use it.

But most of the time you're not there yet. You're still iterating on your system prompt, changing a sentence in Narrowing to see if it stops the agent from going off-scope. For that stage, you need something lighter.

Here's the model I use:

Tier	What it checks	Cost	Speed
Structural	Format compliance, section presence, vocabulary	Free	Instant
LLM Judge	Content quality, finding detection, reasoning	~$0.01/check	Seconds
Human	Calibration, edge cases, subjective quality	Your time	Minutes

If you want to run it first and read later:

git clone https://github.com/gunnargrosch/agent-evals-demo.git
cd agent-evals-demo
npm install
npm run eval

Two principles guide the approach:

Binary pass/fail over scales. A finding is caught or it isn't. A section is present or it isn't. Likert scales (1-5 ratings) sound more nuanced, but they're harder to calibrate and harder to act on. If you need to score "how good" a finding is, you don't yet know what "good" means for your use case. Define it first, then check for it.

RISEN components are eval levers. Each component of your system prompt maps directly to something you can check:

Expectation defines structural checks: are the required sections present? Is the summary table formatted correctly?
Narrowing defines scope checks: did the agent stay in bounds? Did it avoid things you told it to avoid?
Steps define content checks: did the agent follow the workflow? Did it catch the issues each step should surface?

The Test Cases

To evaluate a code review agent, you need code to review. Not random code, but code with known issues so you can check whether the agent found them.

I built three test cases:

Test case	Purpose	Expected findings
Basic Vulnerabilities	Well-known issues. Calibration baseline.	5 (SSN exposure, no validation, client in handler, wildcard CORS, no error handling)
Subtle Issues	Harder problems. Tests depth.	4 (NoSQL injection, scan vs query, missing idempotency, no batch write)
False Positive Bait	Correct code that looks suspicious. Tests precision.	0 (should find nothing wrong)

The first is the same Lambda function from the RISEN demo: a user lookup function with SSN exposure, missing input validation, a DynamoDB client instantiated inside the handler, wildcard CORS, and no error handling. It has obvious problems that any decent review should catch. If your agent misses SSN exposure on this function, something is fundamentally broken.

The second is harder. It processes SQS messages and writes to DynamoDB with a FilterExpression built by string concatenation. That's NoSQL injection, but it's subtler than SQL injection and many reviewers miss it. The code also uses Scan instead of Query and doesn't handle SQS redelivery (missing idempotency).

The third is the interesting one. It has test constants that look like hardcoded secrets (TEST_API_KEY = 'test-ak-00000...'), a structured error handler that could look like error swallowing, and environment variable configuration that could be mistaken for hardcoded values. A good reviewer should find nothing wrong here. A trigger-happy one will flag false positives.

Each test case is a TypeScript object with the code, expected findings, and things that should not be flagged:

interface TestCase {
  name: string
  description: string
  input: string
  expectedFindings: Finding[]
  expectedAbsent: string[]
}

interface Finding {
  id: string
  description: string
  severity: 'critical' | 'high' | 'medium' | 'low'
  keywords: string[]
}

The keywords array gives the LLM judge contextual hints about what vocabulary to look for. Keywords are passed directly to the judge as part of the user message. The judge reads them and uses them to decide whether the review demonstrated real understanding of the problem. For the NoSQL injection finding: ['injection', 'FilterExpression', 'concatenat', 'ExpressionAttributeValues', 'parameteriz']. The review doesn't need all of them, but it needs to demonstrate understanding of the actual problem, not just mention a related keyword in passing.

The stems ('concatenat', 'parameteriz') are intentional. The judge reads them semantically, so 'concatenat' cues the judge to look for "concatenation", "concatenated", and similar. Calibrate keyword specificity carefully: too generic and the judge gives credit for tangentially related mentions; too specific and you miss a review that describes the problem correctly but uses different phrasing.

Tier 1: Structural Checks

Structural checks are pure code. No LLM calls, no cost, instant results. They verify that the agent's output matches the format defined in your Expectation and respects the boundaries in your Narrowing.

For the code review prompt, the Expectation section says:

Return a structured review with a summary table listing each finding with its severity, a detailed section for each finding with severity level, description, problematic code, and corrected code, and a summary count of findings by severity at the end.

That translates directly to checks:

function runStructuralChecks(output: string): StructuralCheckResult[] {
  const checks: StructuralCheckResult[] = []

  // Derived from Expectation: "summary table listing each finding"
  const hasTable = /\|.*severity.*\|/i.test(output)
    || /\|.*finding.*\|/i.test(output)
  checks.push({
    name: 'Summary table present',
    passed: hasTable,
    detail: hasTable
      ? 'Found summary table with findings'
      : 'No summary table found.',
  })

  // Derived from Expectation: "corrected code"
  // Assumes paired backtick fences — malformed output with odd backtick count
  // produces a non-integer, which Math.floor rounds down silently.
  const codeBlockCount = (output.match(/```
{% endraw %}
/g) || []).length / 2
  checks.push({
    name: 'Code blocks with fixes',
    passed: codeBlockCount >= 1,
    detail: {% raw %}`Found ${Math.floor(codeBlockCount)} code block(s)`{% endraw %},
  })

  // Derived from Narrowing: "Do not suggest rewriting the entire function"
  const outputLines = output.split('\n').length
  const longestCodeBlock = getLongestCodeBlock(output)
  const isFullRewrite = longestCodeBlock > 40 && longestCodeBlock > outputLines * 0.4
  checks.push({
    name: 'No full rewrite',
    passed: !isFullRewrite,
    detail: isFullRewrite
      ? {% raw %}`Longest code block is ${longestCodeBlock} lines.`{% endraw %}
      : 'Code blocks are targeted fixes.',
  })

  return checks
}
{% raw %}

The full demo includes six checks: summary table, severity vocabulary (at least two severity levels used), code blocks, no full rewrite, summary count at end, and scope compliance (no style/readability suggestions). All derived from two RISEN components.

Here's what the output looks like:


text
Structural Checks (6/6 passed)

  PASS Summary table present
  PASS Severity vocabulary
  PASS Code blocks with fixes
  PASS No full rewrite
  PASS Summary count at end
  PASS Stays within scope

These are pattern-matching checks, not semantic ones. An agent that rephrases a style suggestion to dodge the regex will pass. That's fine: structural checks catch gross format violations cheaply. The judge handles subtlety.

Structural checks catch the most common prompt problems: the agent ignored your format instructions, or it drifted out of scope. They're the first thing to run because they're free and fast. If structural checks fail, there's no point running the judge.

Tier 2: LLM-as-Judge

Structural checks tell you the output looks right. They can't tell you the content is right. For that, you need a judge: a second agent that reads the review output and assesses whether specific findings were caught.

Different model, mandatory reasoning

Two design decisions matter here. First, the judge uses a different model than the review agent. The review agent runs on Claude Sonnet 4.5. The judge runs on Claude Haiku 4.5. Using the same model to judge its own output creates self-enhancement bias: the model that produced a vague or incomplete finding will tend to accept that same vague finding as "caught." A different model gives you a more honest assessment, and Haiku is fast and cheap enough to run per-finding.

Second, the judge must write its reasoning before its verdict. This is the same principle behind chain-of-thought prompting: forcing the model to explain its logic before committing to an answer produces better answers. The judge prompt's Expectation section enforces this:


text
# Expectation
Return exactly one JSON object with this structure:
{
  "findingId": "<the finding ID provided>",
  "reasoning": "<2-3 sentences explaining why you believe the finding was
                 or was not caught>",
  "caught": <true or false>
}

Reasoning comes before the verdict in the JSON. The model writes the reasoning first, then decides.

The judge prompt

The judge gets a RISEN-structured prompt. It's worth showing in full. This is a RISEN prompt evaluating another RISEN prompt's output, and the structure makes that relationship explicit:


text
# Role
You are a code review evaluator. You assess whether a code review
correctly identified a specific security or performance issue. You are
precise and literal: a finding is caught only if the review clearly
describes the problem and its impact.

# Instructions
You will receive a code review output and a specific finding to check
for. Determine whether the review caught the finding. Return your
assessment as JSON.

# Steps
1. Read the finding description and keywords carefully.
2. Search the review output for mentions of the issue.
3. Assess whether the review identified the core problem (not just
   mentioned a related keyword in passing).
4. Write your reasoning first, then your verdict.

# Expectation
Return exactly one JSON object with this structure:
{
  "findingId": "<the finding ID provided>",
  "reasoning": "<2-3 sentences explaining why you believe the finding
                 was or was not caught>",
  "caught": <true or false>
}

Return ONLY the JSON object. No markdown fences, no extra text.

# Narrowing
- A finding is "caught" only if the review identifies the specific
  problem described, not just a vaguely related concern.
- If the review mentions the general area but misses the specific
  vulnerability (e.g., mentions DynamoDB but not the injection vector),
  that is NOT caught.
- Do not give credit for partial matches. The review must demonstrate
  understanding of the actual issue.

The Narrowing is where the precision comes from. Without it, the judge tends to give credit for proximity. The review mentions DynamoDB? Close enough to "NoSQL injection"? No. The review has to demonstrate understanding of the actual vulnerability.

Running the judge

The judge evaluates each expected finding independently:


typescript
async function judgeFindings(
  reviewOutput: string,
  findings: Finding[]
): Promise<JudgeVerdict[]> {
  // Each finding gets its own agent instance and runs in parallel.
  // Promise.allSettled means one timeout or error doesn't block the rest.
  const results = await Promise.allSettled(
    findings.map(async (finding) => {
      const agent = createJudgeAgent(judgePrompt)
      const prompt = `
Review output to evaluate:
---
${reviewOutput}
---

Finding to check:
- ID: ${finding.id}
- Description: ${finding.description}
- Severity: ${finding.severity}
- Keywords that indicate detection: ${finding.keywords.join(', ')}

Did the review catch this finding?`

      const result = await agent.invoke(prompt)
      return parseJudgeResponse(result.toString(), finding.id)
    })
  )

  return results.map((result, i) =>
    result.status === 'fulfilled'
      ? result.value
      : { findingId: findings[i].id, reasoning: 'Judge failed', caught: false }
  )
}

For the false positive test case, a separate judge prompt checks whether the review incorrectly flagged correct code as problematic.

The judge returns a JSON object with reasoning first and caught second. The caught boolean is what drives the PASS/FAIL in the terminal output; the reasoning string is what gets printed below it. You defined that schema in the Expectation section of the judge prompt.

Here's what the judge output looks like on the Subtle Issues test case:


text
Judge Verdicts (2/4 caught)

  PASS nosql-injection
    The review explicitly identifies the NoSQL injection vulnerability,
    clearly describing the core problem: user input is concatenated
    directly into the FilterExpression without parameterization.

  PASS scan-instead-of-query
    The review explicitly identifies this issue, clearly describing the
    core problem: the code uses ScanCommand which reads the entire table
    before filtering.

  FAIL missing-idempotency
    The review does not identify the missing idempotency issue. While
    the review addresses error handling, it does not discuss the specific
    problem of duplicate orders when SQS messages are reprocessed.

  FAIL no-batch-write
    The review does not identify the inefficiency of sequential
    PutCommand operations in a loop instead of BatchWriteCommand.

The baseline prompt catches 2 out of 4 findings on the harder test case. It finds the NoSQL injection and the scan problem, but misses idempotency and batch writes. 50% on the subtle test case isn't a failure: it's the starting point. That's the data you need to improve the prompt. If your results look different across runs, that's expected. The calibration section covers non-determinism and what to do about it.

Tier 3: The Calibration Loop

The first time you run evals, you'll disagree with some of the judge's verdicts. That's expected and useful. The calibration loop is how you turn those disagreements into a better rubric.

The process:

Run the eval.
Read every judge verdict, especially the reasoning.
For each disagreement, decide which of three things happened:
- The judge is wrong. Tighten the judge prompt's Narrowing or add a keyword to the finding definition.
- The agent is wrong. The agent should have caught this. Tighten the review prompt's Steps.
- The test case is wrong. The expected finding is unreasonable, or the code doesn't actually have the issue you thought it did. Fix the test case.

After my first run, I found two calibration issues. The "summary count at end" structural check was too strict: it only looked in the last 800 characters and required specific phrasing. I widened the search window and added more patterns. The false positive check for "hardcoded secret" was catching cases where the review mentioned test constants neutrally rather than flagging them as issues. I tightened the false positive judge prompt to distinguish between "flagged as a finding" and "mentioned in passing."

Two iterations were enough to get a stable rubric for three test cases. If you have more test cases or a more complex agent, you might need three or four rounds.

After the summary, the eval also prints a Suggestions section that maps each failure back to the RISEN component to edit: missed findings point to Steps, structural failures point to Expectation, false positives point to Narrowing. It doesn't tell you what to change, but it tells you where to look.

A note on non-determinism

LLMs don't produce the same output twice. Your eval might pass on one run and fail on the next for the same prompt and same input. This is normal, and it matters for how you interpret results.

For structural checks, non-determinism is rarely an issue: the agent either returns a table or it doesn't. For the judge, a borderline verdict (the review hinted at a finding but didn't nail it) may flip between runs. If a finding fails consistently, it's a real signal. If it flips, that's a signal too: the finding is on the boundary of what the current prompt reliably catches, and the judge rubric may need tightening.

For CI, this means not treating every eval failure as a blocker. Run structural checks in CI: they're stable. Use judge results as monitoring: track pass rates over multiple runs and alert on sustained regressions rather than single failures. The --ci flag exits non-zero on any failure; use it in CI only once your rubric is stable enough that flakiness is rare.

A simple strategy for borderline findings: run the eval three times and consider the finding caught if it passes at least two of three runs. The demo doesn't do this automatically. You'd wire it up in your CI script, but it filters out most random variance without being too permissive. The README notes which findings are intermittent in the baseline prompt; those are good candidates for this approach before you tighten the prompt further.

Comparing Across Prompt Iterations

The real payoff comes when you change your prompt and want to know if the change helped. The compare script runs both prompts against all test cases and shows what changed.

The v2 prompt adds explicit steps for NoSQL injection detection and idempotency checking:


text
# Steps
...
2. Check for security issues: injection (including NoSQL injection via
   string concatenation in DynamoDB expressions), overly permissive IAM
   assumptions, hardcoded secrets, missing input validation.
3. Check for data handling: look for string concatenation in
   FilterExpression, KeyConditionExpression, or ProjectionExpression.
   These must use ExpressionAttributeValues with placeholders.
4. Check for idempotency: if processing messages from SQS, SNS, or
   EventBridge, verify the handler is idempotent.
...

Here's the comparison output:


text
COMPARISON: BASELINE vs VARIANT

Basic Vulnerabilities
  Structural: 6/6 -> 6/6
  Findings:   5/5 -> 5/5

Subtle Issues
  Structural: 6/6 -> 6/6
  Findings:   2/4 -> 3/4 (+1)
    + now catches: missing-idempotency

False Positive Bait
  Structural: 5/6 -> 6/6 (+1)
  False pos:  0 -> 1 (-1)

The v2 prompt catches the missing idempotency issue that the baseline missed. That's the targeted improvement. But it also introduced a false positive on the clean code: it incorrectly flagged "wildcard CORS" on a function that uses environment-based CORS configuration.

This is the trade-off you're always navigating with prompt changes. Adding specificity to Steps improves recall (catches more real issues) but can hurt precision (flags more non-issues). The eval gives you data to make that trade-off deliberately instead of guessing.

The Same Pattern, Different Domain

Code review is a useful test bed, but the interesting question is whether the pattern holds for something more subjective. The demo includes a second domain: content review. An agent that reviews blog post drafts for completeness, structure, and technical accuracy.

The test cases are blog post drafts instead of Lambda functions. The expected findings are things like "missing prerequisites section" and "unexplained command flags" instead of "SSN exposure" and "NoSQL injection." The structural checks are completely different: section checklist present, finding categories used, readiness assessment at end. No severity tables, no code blocks with fixes.

But the pieces that change are exactly three files:

Test cases: blog post drafts with known issues, using the same TestCase interface.
Structural checks: regex/string checks derived from the content review prompt's Expectation section.
Agent prompt: a RISEN-structured prompt for technical editing instead of security review.

The judge, report formatter, and TestCase/Finding types stay the same. Run it with:


bash
npm run eval:content


text
TEST CASE: Incomplete Tutorial

Structural Checks (5/5 passed)

  PASS Section checklist present
  PASS Finding categories
  PASS Actionable suggestions
  PASS Readiness assessment
  PASS Stays within scope

Judge Verdicts (4/5 caught)

  PASS missing-prerequisites
  PASS unexplained-command-flags
  PASS missing-import
  PASS no-troubleshooting
  FAIL missing-conclusion

Different domain, same eval pattern. If you're building a summarization agent, a customer support agent, or anything else, the approach is the same: define test cases with known expected findings, write structural checks from your Expectation section, and let the judge handle content quality.

Running the Demo

The demo repo has everything you need to run this yourself.

You'll need:

An AWS account with Amazon Bedrock access to Claude Sonnet 4.5 and Claude Haiku 4.5
Node.js 20+
AWS credentials configured (AWS_PROFILE or default)

Getting started:


bash
git clone https://github.com/gunnargrosch/agent-evals-demo.git
cd agent-evals-demo
npm install
npm test

npm test runs the unit tests for the structural checks. No LLM calls, no AWS credentials needed. A good first step to verify the setup.

Commands:

Command	What it does	Time
`npm run eval:structural`	Code review structural checks only. No judge, no cost.	~30s
`npm run eval`	Code review full eval: structural + LLM judge.	~2min
`npm run eval:content`	Content review eval: structural + LLM judge.	~1min
`npm run compare`	Baseline vs v2 code review prompt, side by side.	~4min

You can also run the full eval with the v2 prompt:


bash
npm run eval -- --v2

To test your own code review prompt, write it in a text file and pass it with --prompt:


bash
npm run eval -- --prompt ./my-code-review-prompt.txt

If you want to eval a completely different kind of agent without writing any TypeScript, --test-case accepts a JSON file of test cases and --skip-structural skips the built-in code review checks:


bash
npm run eval -- --prompt ./my-prompt.txt --test-case ./my-cases.json --skip-structural

The compare script accepts --baseline and --variant for comparing any two prompt files:


bash
npm run compare -- --baseline ./my-prompt-v1.txt --variant ./my-prompt-v2.txt

Pass --ci to any eval script to exit non-zero on failures, useful for pipeline integration:


bash
npm run eval -- --ci

The eval uses Sonnet 4.5 for the review agent and Haiku 4.5 for the judge. A full npm run eval across all three test cases costs roughly $0.10-0.15 at current Bedrock pricing. npm run compare runs six agent calls plus the judge, so roughly double.

When to Graduate to a Framework

This approach works well for iterating on a single agent's system prompt with a handful of test cases. When you hit these signals, it's time to look at a framework:

More than 10 test cases. You'll want parallel execution, caching, and proper test runners.
CI integration. The demo has a --ci flag that exits non-zero on failures, so you can hook it into a pipeline. But once you need test result history, trend tracking, or gating deploys across multiple agents, a framework handles that better.
Multiple agents coordinating. Trajectory evaluation (did the agents take the right steps?) matters as much as output evaluation.
Team collaboration. Others need to run and extend evals without understanding your bespoke scripts.

The Strands Agents SDK includes an evals package (Python) with OutputEvaluator and TrajectoryEvaluator classes that handle these scenarios. Note that this is the Python SDK. The TypeScript SDK doesn't include an evals package yet, so graduating means either switching to Python or building on top of what this demo started. The lightweight approach in this post is for the earlier stage: when you're still figuring out what "good" looks like for your agent's output.

Additional Resources

agent-evals-demo: Demo repo with all code from this post
Writing System Prompts That Actually Work: The RISEN Framework for AI Agents: The RISEN framework post this builds on
risen-prompt-demo: Demo repo for the RISEN post
Strands Agents SDK (TypeScript): Agent framework used in the demo
Strands Agents evals (Python): Full eval framework for when you graduate
Your AI Product Needs Evals: Hamel Husain's guide to building evals

If you try it, the calibration loop is where the interesting disagreements show up. What does your current approach to agent eval look like? Let me know in the comments!

Writing System Prompts That Actually Work: The RISEN Framework for AI Agents

Gunnar Grosch — Sun, 01 Mar 2026 17:32:04 +0000

You've probably written a system prompt that looks like this:

You are a helpful assistant. Help the user with their request.

It works. The model responds. But the output is unpredictable. Ask it to review code and you get a mix of style comments and security findings with no consistent structure. Ask it to diagnose an incident and it gives you a wall of text that buries the actionable steps. Ask it to design an architecture and it picks services without explaining trade-offs.

If you're building agents that need to produce consistent, structured output, whether that's a single-agent workflow or a multi-agent system, the problem isn't the model. It's the prompt. A vague system prompt gives the model no framework for structuring its reasoning, so it improvises every time. Sometimes the improvisation is great. Sometimes it misses the point entirely. You can't build reliable agents on "sometimes."

The RISEN Framework

RISEN is a structured approach to writing system prompts. Each letter represents a component:

Component	What it does
Role	Who the agent is. Expertise, experience, specialization.
Instructions	What you want it to do. The core task.
Steps	How to get there. The ordered workflow.
Expectation	What the output should look like. Format, structure, sections.
Narrowing	What to exclude. Constraints, boundaries, scope limits.

You'll see the E defined as "End Goal" in some formulations. Expectation is a deliberate choice here: for agents, what matters is the structural contract for the output, not a vague goal statement. "Produce a useful architecture" is an end goal. "Return sections for Requirements Summary, Service Selection with trade-off tables, SAM Template, and Cost Estimate" is an expectation.

Most people only write the I part. "Review the code." "Diagnose the issue." "Design an architecture." That's an instruction with no context about who's doing the work, what process to follow, what format to use, or what to leave out.

RISEN fills in the rest. The result isn't just a prompt. It's a behavioral contract. The agent knows what role it's playing, what steps to follow, what structure to produce, and what boundaries to respect.

Why This Matters for Agents

System prompts matter more for agents than for simple chat. In a chat application, a vague system prompt means the user gets a mediocre answer and can follow up. In an agentic workflow, a vague system prompt means the agent takes actions based on an ambiguous understanding of its role. It might use the wrong tools, skip steps, or produce output that downstream agents can't parse.

In multi-agent systems (whether you're using protocols like A2A and MCP, or frameworks like Strands Agents), each agent's system prompt is its behavioral contract with the rest of the system. A warehouse management agent in a logistics pipeline needs to know exactly what decisions it owns, what format to return, and what to escalate. "You are a warehouse assistant" doesn't cut it.

The SwiftShip demo from re:Invent session DEV415 is a good example. It's a logistics platform with four agents (Triage, Order, Payment, Warehouse) that coordinate to resolve delivery exceptions. Every agent has a RISEN-structured system prompt. The Triage Agent's Steps section is a full decision tree: classify the exception, determine the resolution strategy, invoke the right specialist agents in the right order (Payment before Warehouse before Order for replacements), and produce a resolution summary. The Narrowing section prevents it from handling general customer inquiries and enforces that it never processes refunds without confirming the exception type. That's not a prompt. That's an orchestration contract.

This is also where Narrowing earns its place. Without explicit constraints, agents over-deliver. An incident response agent might suggest rewriting the application code when all you need right now is "switch DynamoDB to on-demand capacity." Narrowing keeps the agent focused on what's useful for the current context.

The Difference in Practice

I put together a demo repo with three scenarios that show the difference between basic and RISEN system prompts. Each scenario sends the same user prompt to the same model twice: once with a one-sentence system prompt, once with a RISEN-structured prompt. Same model, same input, different guidance. The demo uses Strands Agents with Amazon Bedrock.

Scenario 1: Incident response

The user prompt is a DynamoDB throttling alert: 4,850 WCU consumed against 1,000 provisioned, 2,347 throttled requests, deployed a new Lambda version 45 minutes ago.

Basic prompt:

You are an incident response assistant. Help diagnose and resolve
AWS issues.

RISEN prompt (abbreviated, full version in the demo repo):

# Role
You are an AWS site reliability engineer on an on-call rotation
with 10 years of experience operating production serverless workloads.

# Instructions
Perform a structured diagnosis. Identify the most likely root cause,
provide immediate mitigation steps, and recommend longer-term fixes.

# Steps
1. Parse the alert details: service, metric, threshold, duration.
2. List the top 3 most likely root causes in order of probability.
3. For each, describe evidence that would confirm or rule it out.
4. Provide immediate mitigation steps executable in under 5 minutes.
5. Recommend longer-term fixes with estimated effort.

# Expectation
Sections: Alert Summary, Probable Root Causes (ranked), Diagnostic
Steps, Immediate Mitigation, Long-Term Fixes. Include specific
metric names, CLI commands, and thresholds.

# Narrowing
- Operator has CLI access but cannot deploy code changes during
  the incident.
- Focus on mitigation first. Restoring service is the priority.
- Do not suggest "contact AWS Support" as a first step.
- All commands should use AWS CLI v2 syntax.

The basic prompt gives a solid response. It correctly identifies the new Lambda deployment as the likely cause, provides useful CLI commands, and suggests scaling up DynamoDB. But it's organized as a narrative with emoji headers and ends by asking the operator what to do:

## 🚨 Immediate Issue
Your write capacity is being consumed at **485% of provisioned capacity**...

## 🔍 Root Cause Hypothesis
Given the timeline, the new Lambda deployment is the likely culprit.

...

**What would you like to do first? Scale the table or rollback the Lambda?**

That question is the wrong instinct for an incident response agent. At 2 AM, you don't want a conversation. You want a ranked action plan.

The RISEN prompt produces exactly that. Root causes are ranked with confidence percentages:

## 1. **Lambda Write Amplification (90% confidence)**
## 2. **Hot Partition Key Issue (70% confidence)**
## 3. **SQS Message Backlog Processing (60% confidence)**

Each mitigation option includes cost and impact:

## Option A: Increase DynamoDB Write Capacity (60 seconds)
aws dynamodb update-table \
  --table-name order-events-prod \
  --provisioned-throughput ReadCapacityUnits=1000,WriteCapacityUnits=5000
Impact: Eliminates throttling immediately. Table update takes 30-60 seconds.
Cost: ~$0.35/hour additional ($2,336/month vs $467/month baseline)

And long-term fixes come with effort estimates ("Effort: 15 minutes", "Effort: 4 hours") so you can prioritize. The Narrowing constraint about not deploying code during an incident kept the response focused on what an on-call engineer can actually do without waking up the development team.

Scenario 2: Architecture decision

This scenario adds a twist: both agents get the same AWS MCP server tools for searching AWS documentation, checking service limits, and validating recommendations. Same tools, same model, same user prompt. The only difference is the system prompt.

The user prompt describes requirements for a real-time order notification system: 50,000 orders per day, multiple notification channels, customer preferences, 30-second delivery SLA, under $500/month.

Basic prompt:

You are an AWS solutions architect. Help design cloud architectures.

RISEN prompt (abbreviated):

# Role
You are a principal AWS solutions architect specializing in
event-driven serverless architectures.

# Instructions
Design an AWS architecture. Evaluate service options, justify
choices with trade-offs, and provide a SAM template snippet.
Use the AWS documentation tools to validate your recommendations.

# Steps
1. Restate requirements as functional and non-functional.
2. Identify the core architectural pattern.
3. For each component, list 2-3 service options with trade-offs.
   Use the documentation tools to verify current service limits
   and pricing.
4. Select and justify the recommended option.
5. Describe the data flow end to end.
6. Provide a SAM template snippet.
7. Call out operational considerations.

# Expectation
Sections: Requirements Summary, Architecture Pattern, Service
Selection (with trade-off tables), Data Flow, SAM Template,
Operational Considerations. Include a monthly cost estimate.

# Narrowing
- Prefer serverless over instance-based.
- Use managed services only.
- SAM templates should be valid YAML, not pseudocode.
- Cost estimates using current us-east-1 pricing.
- Use the documentation tools only to verify specific facts
  (pricing, limits, quotas). Do not use them to generate
  the architecture itself.

Both agents used the MCP tools. But look at what they did with them.

The basic prompt queried the documentation and jumped straight to a recommendation:

## Recommended Architecture: Event-Driven Real-Time Order Notification System
...
### Core Components
#### 1. Event Ingestion Layer
- **Amazon EventBridge**: Central event bus for order events
...
Would you like me to:
1. Generate CDK/CloudFormation templates for this architecture?
2. Create the Lambda function code with full error handling?

No alternatives evaluated. No trade-offs. And it ends by asking what to do next.

The RISEN prompt used the same tools to verify facts, then produced trade-off tables for every component:

### 3.1 Event Ingestion Layer

| Service           | Pros                         | Cons                       | Verdict      |
|-------------------|------------------------------|----------------------------|--------------|
| EventBridge       | Native filtering, $1/M events| Limited transformation     | SELECTED     |
| Kinesis Streams   | Replay, high throughput      | $11/month min, overkill    |              |
| SQS               | Simple, cheap                | No native fanout           |              |

Decision: EventBridge - 8.33 events/sec peak << 10,000/sec limit

The Steps guided the agent to evaluate before deciding. The Narrowing constraint "Use the documentation tools only to verify specific facts" kept the tool usage focused: the agent looked up pricing and limits, not architectures. The result was a full architecture document with a SAM template, a cost breakdown ($253.51/month against the $500 budget), and operational considerations including scaling limits and monitoring.

Scenario 3: Code review

The user prompt is a Lambda function with several issues: SDK client instantiated inside the handler, no input validation, sensitive data (SSN) returned in the API response, wildcard CORS headers, and no error handling.

Basic prompt:

You are a code review assistant. Review code for issues and suggest improvements.

RISEN prompt (abbreviated):

# Role
You are a senior AWS security engineer specializing in serverless
application security.

# Instructions
Review the provided code for security vulnerabilities, performance
issues, and AWS best practice violations. Prioritize findings by
severity and provide fix recommendations with corrected code.

# Steps
1. Identify the AWS services and patterns in use.
2. Check for security issues: injection, overly permissive IAM,
   hardcoded secrets, missing input validation.
3. Check for performance issues: cold start impact, unnecessary
   SDK client instantiation.
4. Check for reliability issues: missing error handling, no retries.
5. For each finding, provide severity, the problematic code,
   and a corrected snippet.

# Expectation
Structured review organized by severity. Each finding includes:
severity level, description, problematic code, corrected code.
End with a summary count.

# Narrowing
- Focus on production impact. Ignore style preferences.
- Do not suggest rewriting the entire function or switching runtimes.
- Limit the review to security, performance, and reliability.

Both prompts catch the SSN exposure. But look at how the output differs.

The basic prompt opens with emoji-coded sections and mixes severity with style:

## Critical Issues 🔴
### 1. **Security Vulnerability - Sensitive Data Exposure**
...
## Medium Priority Issues 🟠
### 7. **Type Safety**
- `event: any` loses type safety

It also generates a full rewrite of the function (which the reviewer didn't ask for) and ends with a brief summary.

The RISEN prompt produces a consistent structure: every finding follows the same format (Severity, Description, Problematic Code, Corrected Code) and ends with a summary table:

| Severity     | Count | Issues                                     |
|--------------|-------|--------------------------------------------|
| **Critical** | 2     | NoSQL injection, PII exposure (SSN)        |
| **High**     | 3     | Missing auth, permissive CORS, no errors   |
| **Medium**   | 3     | Cold start, input validation, null checks  |
| **Low**      | 2     | TypeScript any type, missing headers       |
| **Total**    | **10**|                                            |

The Narrowing constraint "do not suggest rewriting the entire function" kept the RISEN response focused on targeted fixes. The basic prompt had no such guardrail and generated a complete replacement.

Try It Yourself

You'll need:

Node.js 20+
AWS credentials configured for Amazon Bedrock access
Python 3.10+ and uvx (for the architecture scenario's AWS MCP server integration)

git clone https://github.com/gunnargrosch/risen-prompt-demo.git
cd risen-prompt-demo
npm install

Run a scenario:

npm run code-review
npm run incident
npm run architecture

Each scenario runs the basic prompt first, then the RISEN prompt, so you can see the difference in your terminal.

Building Your Own RISEN Prompts

A few things I've noticed that make RISEN prompts more effective:

Role is more than a job title

"You are a code reviewer" gives the model a vague persona. "You are a senior AWS security engineer specializing in serverless application security" tells it what lens to apply. The more specific the role, the more the model draws on relevant knowledge. Include years of experience, domain expertise, and the specific technology stack.

Get the step granularity right

Too few steps and the model skips reasoning. Too many and it gets rigid. Three to seven steps tends to work. Each step should represent a distinct phase, not a sub-task. If you find yourself writing "2a, 2b, 2c," that's one step with internal detail, not three steps.

Make Narrowing specific

The most common mistake is forgetting Narrowing entirely. The second most common is making it too vague. "Keep it focused" isn't a constraint. "Do not suggest services in preview or limited availability" is. Write constraints that you could objectively check against the output.

Don't skip Expectation for agents

For single-use prompts, Expectation is nice to have. For agents whose output feeds into other agents or structured workflows, it's required. Specify sections, ordering, format (bullet points, tables, code blocks). If you skip one section, don't skip this one.

Your first RISEN prompt won't be your last. Run it against a few representative inputs and check the output against your Expectation section. If the structure is right but the content is off, adjust the Role or Steps. If the agent keeps going out of scope, tighten the Narrowing. If the output format is inconsistent, make the Expectation more specific. The framework gives you five independent levers to tune.

Other Frameworks Worth Knowing

RISEN isn't the only structured approach. Anthropic and OpenAI both publish recommended prompt structures for their models that cover similar ground: role, instructions, output format, examples, and constraints. If your agent uses tools extensively, RISE-M extends RISEN with a sixth component, Methods, which covers when and how to use each tool. The architecture scenario above is a lightweight version of this: the Steps and Narrowing sections include tool usage guidance ("verify current service limits" in Steps, "only to verify specific facts" in Narrowing). If your tool-specific constraints keep growing, a dedicated Methods section may be cleaner.

The frameworks overlap. The value isn't in picking the "right" one. It's in moving from an unstructured one-liner to any systematic approach that covers role, task, process, format, and constraints.

Start Here

If you want to try RISEN on your next agent, here's a blank template you can copy and fill in:

# Role
You are a [job title/expertise] specializing in [domain]. You have
[years of experience] with [specific technologies/tools].

# Instructions
[Core task in 1-2 sentences. What should the agent accomplish?]

# Steps
1. [First thing the agent should do]
2. [Second thing]
3. [Continue until the workflow is complete]

# Expectation
[Output format: sections, tables, code blocks, bullet points.
Specify the structure the response should follow.]

# Narrowing
- [What to exclude or ignore]
- [Scope boundaries]
- [Constraints on format, length, or approach]

Fill in Role first (it shapes everything else), then Instructions, then Steps. Expectation and Narrowing come last because they depend on knowing what the agent is doing and how.

Additional Resources

RISEN Prompt Demo Repository
SwiftShip Multi-Agent Demo (RISEN in a production-style multi-agent system)
Strands Agents SDK
Anthropic System Prompt Guide
OpenAI GPT-4.1 Prompting Guide
The Prompt Report (Zhou et al.)
Anthropic: Building Effective Agents

Have you applied a structured framework to your agent system prompts? What changed in the output when you did? I'd like to hear about it in the comments.

Streaming Bedrock Responses Through API Gateway and Lambda

Gunnar Grosch — Wed, 25 Feb 2026 09:05:13 +0000

If you're building applications that call Amazon Bedrock through API Gateway and Lambda, your users are probably staring at a spinner. The model generates tokens progressively, but the standard Lambda integration buffers the entire response before sending anything back. For a typical LLM response, that's 8-10 seconds of nothing, then everything at once.

API Gateway response streaming fixes this. Tokens flow from Bedrock through Lambda and API Gateway to the client as they're generated. The first token arrives in ~500ms. The total generation time stays the same. The difference is entirely in when the user starts seeing output. Beyond latency, streaming also lifts two constraints that matter for larger workloads: the 10 MB response payload limit and the 29-second default integration timeout. Streaming responses can run for up to 15 minutes and exceed 10 MB.

I put together a demo that runs both approaches side by side so you can see the difference for yourself. Two Lambda functions, same model, same prompt, same API Gateway. One streams, one buffers. The streaming panel fills up token by token while the buffered panel sits there waiting.

Metric	Streaming	Buffered
Time to first byte	~500ms	~8-10s
Total time	~8-10s	~8-10s
User experience	Progressive, real-time	Waiting, then all at once

How It Works

Streaming requires changes in two places: the API Gateway configuration and the Lambda handler. Neither is complicated on its own. The part that trips people up is getting them to work together.

The API Gateway side

In the OpenAPI spec, the streaming endpoint needs two things that the standard endpoint doesn't:

A different Lambda invocation URI path: /response-streaming-invocations instead of /invocations
A responseTransferMode: STREAM property on the integration

Here's what that looks like:

/streaming:
  post:
    x-amazon-apigateway-integration:
      type: AWS_PROXY
      httpMethod: POST
      uri:
        Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2021-11-15/functions/${StreamingFunction.Arn}/response-streaming-invocations"
      responseTransferMode: STREAM
      passthroughBehavior: when_no_match

Compare that to the standard endpoint:

/non-streaming:
  post:
    x-amazon-apigateway-integration:
      type: AWS_PROXY
      httpMethod: POST
      uri:
        Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${NonStreamingFunction.Arn}/invocations"
      passthroughBehavior: when_no_match

The URI path version changes from 2015-03-31 to 2021-11-15, and response-streaming-invocations replaces invocations. Under the hood, this tells API Gateway to use Lambda's InvokeWithResponseStreaming API instead of the standard Invoke. Miss either of those details and your streaming endpoint silently falls back to buffered behavior. No error, just a longer wait.

The Lambda side

The streaming handler uses awslambda.streamifyResponse() to wrap the handler function. This gives you a writable HttpResponseStream instead of requiring you to return a response object:

import { BedrockRuntimeClient, ConverseStreamCommand } from '@aws-sdk/client-bedrock-runtime';
import type { APIGatewayProxyEvent } from 'aws-lambda';

const client = new BedrockRuntimeClient({ region: process.env.AWS_REGION });

const streamingHandler = async (
  event: APIGatewayProxyEvent,
  responseStream: NodeJS.WritableStream
): Promise<void> => {
  const httpResponseStream = awslambda.HttpResponseStream.from(responseStream, {
    statusCode: 200,
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Connection': 'keep-alive',
      'Access-Control-Allow-Origin': '*',
    },
  });

  const body = JSON.parse(event.body || '{}');
  const command = new ConverseStreamCommand({
    modelId: process.env.BEDROCK_MODEL_ID,
    messages: [{ role: 'user', content: [{ text: body.prompt }] }],
    inferenceConfig: { maxTokens: 2048 },
  });

  const response = await client.send(command);

  for await (const chunk of response.stream!) {
    if (chunk.contentBlockDelta?.delta?.text) {
      const token = chunk.contentBlockDelta.delta.text;
      httpResponseStream.write(`data: ${JSON.stringify({ token })}\n\n`);
    }
  }

  httpResponseStream.write('data: [DONE]\n\n');
  httpResponseStream.end();
};

export const handler = awslambda.streamifyResponse(streamingHandler);

Each token gets written as a Server-Sent Event the moment Bedrock generates it. The data: [DONE] sentinel tells the client the stream is complete.

The HttpResponseStream.from() call is doing something important behind the scenes: it writes the response metadata (status code, headers) as a JSON object followed by an 8-null-byte delimiter that API Gateway uses to separate metadata from the response body. If you're not using HttpResponseStream.from(), you're responsible for writing that delimiter yourself.

The buffered handler does the same Bedrock call but accumulates everything in memory:

const response = await client.send(command);
let fullResponse = '';

for await (const chunk of response.stream!) {
  if (chunk.contentBlockDelta?.delta?.text) {
    fullResponse += chunk.contentBlockDelta.delta.text;
  }
}

return {
  statusCode: 200,
  headers: corsHeaders,
  body: JSON.stringify({ response: fullResponse }),
};

Same model, same prompt, same token generation speed. The only difference is when the client sees the output.

The `awslambda` global

One thing worth noting: awslambda is a global object injected by the Lambda runtime. It's not in any npm package, and @types/aws-lambda doesn't include it either. In TypeScript, you need a type declaration for it:

declare const awslambda: {
  HttpResponseStream: {
    from: (
      responseStream: NodeJS.WritableStream,
      metadata: { statusCode: number; headers: Record<string, string> }
    ) => NodeJS.WritableStream;
  };
  streamifyResponse: (
    handler: (event: APIGatewayProxyEvent, responseStream: NodeJS.WritableStream) => Promise<void>
  ) => (event: APIGatewayProxyEvent) => Promise<void>;
};

This is the kind of detail that's easy to miss. Without this declaration, TypeScript will reject your handler at compile time.

Deploy and Try It

You'll need:

An AWS account with Bedrock model access enabled
AWS CLI configured with credentials
AWS SAM CLI
Node.js 20+

Clone and deploy:

git clone https://github.com/gunnargrosch/apigw-lambda-streaming.git
cd apigw-lambda-streaming
cd functions && npm install && cd ..
sam build
sam deploy --guided

SAM outputs the API Gateway base URL when deployment completes. Copy it.

The interactive demo

Open demo.html in a browser and paste your API Gateway URL. Click Run Comparison (or press Cmd+Enter on macOS, Ctrl+Enter on Windows/Linux). Both endpoints fire simultaneously. The streaming panel fills up token by token while the buffered panel shows a spinner.

The results panel at the bottom shows Time to First Byte for both approaches and the speedup factor. For longer responses (a few paragraphs or more), streaming TTFB is typically 10-20x faster than buffered. Shorter responses show a smaller gap since the buffered endpoint finishes sooner.

Testing with curl

# Streaming: tokens appear progressively as SSE events
curl -N -X POST https://<api-id>.execute-api.<region>.amazonaws.com/demo/streaming \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Write a short story about serverless computing"}'

# Buffered: waits for the complete response
curl -X POST https://<api-id>.execute-api.<region>.amazonaws.com/demo/non-streaming \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Write a short story about serverless computing"}'

The -N flag on the streaming curl disables output buffering so you see tokens as they arrive.

The Infrastructure

The SAM template defines two Lambda functions sharing an execution role with bedrock:InvokeModelWithResponseStream and bedrock:InvokeModel permissions. Both functions use Node.js 20.x, 256 MB memory, and a 120-second timeout. The Bedrock model ID is configurable via a SAM parameter:

Parameters:
  BedrockModelId:
    Type: String
    Default: us.anthropic.claude-sonnet-4-5-20250929-v1:0
    Description: Bedrock model ID

Override it during deployment to use a different model:

sam deploy --parameter-overrides BedrockModelId=us.anthropic.claude-haiku-4-5-20251001-v1:0

API Gateway is configured with an inline OpenAPI spec via AWS::Include, which keeps the streaming-specific integration properties in a separate openapi.yaml file rather than buried in the SAM template.

Things to Know

A few operational details worth being aware of before you ship this to production:

Idle timeouts: Regional and Private API endpoints have a 5-minute idle timeout on streaming responses. Edge-optimized endpoints have a 30-second idle timeout. For LLM token streams this is rarely an issue since tokens arrive continuously, but if you're calling a slower model or one that pauses during longer reasoning chains, keep this in mind.
Bandwidth throttling: The first 10 MB of a streaming response has no bandwidth restrictions. After that, data is throttled to 2 MB/s. Not an issue for LLM token streams, but worth knowing if you're streaming larger payloads.
Pricing: Each 10 MB of streamed response data (rounded up to the nearest 10 MB) is billed as a single API request. For typical LLM responses, this means one request per call, same as buffered.
Not supported with streaming: VTL response transformation, integration response caching, and content encoding. If you rely on any of these, you'll need to handle them differently.
Observability: API Gateway adds three new access log variables for streaming: $context.integration.responseTransferMode (BUFFERED or STREAMED), $context.integration.timeToAllHeaders, and $context.integration.timeToFirstContent. Useful for monitoring TTFB at the API Gateway level.

When to Use This

Response streaming makes the biggest difference for LLM applications where users are waiting for generated text: chatbots, content generation, code assistants, summarization tools. The total time doesn't change, but the perceived latency drops significantly.

This demo focuses on Bedrock, but response streaming works with any Lambda or HTTP proxy integration. A few other use cases where it helps:

Large file downloads: Streaming lets responses exceed the standard 10 MB payload limit, so you can serve large datasets, reports, or media files directly through API Gateway without routing through S3 pre-signed URLs.
Long-running operations with progress updates: An endpoint that runs a multi-step workflow can stream progress events back to the client as each step completes, instead of forcing the client to poll a separate status endpoint.
Web and mobile TTFB optimization: Any API response that takes more than a second or two to fully compute benefits from streaming partial results early. Server-side rendering, search results, or aggregation queries can send the first chunk while the backend continues processing.

A few situations where streaming matters less:

Batch processing: No user is watching. Buffer the response and process it when it's complete.
Short responses: If the backend returns in under a second, streaming adds complexity without a noticeable UX improvement.
Structured output you need to parse as a whole: If your application needs the complete JSON response before it can do anything useful, streaming partial data doesn't help.

Clean Up

sam delete

Additional Resources

Have you added response streaming to your Bedrock applications? I'd like to hear about your experience in the comments.

Building AI Agents in TypeScript with the Strands Agents SDK

Gunnar Grosch — Mon, 23 Feb 2026 17:59:40 +0000

Strands Agents has been Python-only since launch. If your application code is TypeScript, that meant either switching languages for the agent layer or building your own tool-use loop from scratch. Neither is great.

The SDK now has TypeScript support, currently in preview. Breaking changes are possible and not all Python SDK features are available yet. But the core agent loop, tool use, and streaming all work, and the patterns feel natural if you're already writing TypeScript. I put together a demo repo with five standalone examples that progressively build from a minimal agent to multi-turn conversation. Each one is a single file you can run directly. No build step, no boilerplate.

The Setup

You'll need Node.js 20+ and AWS credentials configured for Amazon Bedrock access. The TypeScript SDK supports Bedrock, OpenAI, Gemini, and custom providers. These examples use Bedrock.

git clone https://github.com/gunnargrosch/strands-agents-ts-demo.git
cd strands-agents-ts-demo
npm install

All examples use tsx for direct TypeScript execution. No compilation needed.

If you'd rather follow along in your own project:

npm init -y && npm pkg set type=module
npm install @strands-agents/sdk zod
npm install --save-dev @types/node typescript tsx

Example 1: The Simplest Agent

How little code does it take to get an agent running?

import { Agent, BedrockModel } from '@strands-agents/sdk'

const agent = new Agent({
  model: new BedrockModel(),
  systemPrompt: 'You are a helpful assistant that likes to be edgy in a funny way.',
  printer: false,
})

const prompt = process.argv[2] || 'Which city is best, Gothenburg or Stockholm?'

console.log(`Prompt: ${prompt}\n`)
const result = await agent.invoke(prompt)
console.log(result.toString())

That's it. BedrockModel() with no arguments defaults to Claude Sonnet 4.5. printer: false disables the SDK's built-in console output, which clutters your terminal when you want to control the formatting yourself.

npm run simple

Or pass your own prompt:

npm run simple "What's the meaning of life?"

Example 2: Adding a Tool

A bare agent is just a chat wrapper. Tools are where it gets useful:

import { Agent, BedrockModel, tool } from '@strands-agents/sdk'
import { z } from 'zod'

const calculator = tool({
  name: 'calculate',
  description: 'Perform basic math operations',
  inputSchema: z.object({
    operation: z.enum(['add', 'subtract', 'multiply', 'divide']),
    a: z.number(),
    b: z.number(),
  }),
  callback: ({ operation, a, b }) => {
    switch (operation) {
      case 'add': return a + b
      case 'subtract': return a - b
      case 'multiply': return a * b
      case 'divide':
        if (b === 0) throw new Error('Division by zero')
        return a / b
    }
  },
})

const agent = new Agent({
  model: new BedrockModel(),
  systemPrompt: 'You are a helpful math assistant. Always explain how you calculated the result.',
  tools: [calculator],
  printer: false,
})

The agent decides when to call the tool based on the prompt. Ask it "What is 42 times 17?" and it invokes the calculator, then explains the result.

The Zod part matters more than it looks. If the model sends parameters that fail validation, the SDK catches the error and sends it back to the model as a tool result, giving it a chance to correct the input on the next loop iteration. If you've built custom agent loops before, you know how much code goes into handling bad tool inputs and retrying. Here that's just the agent loop doing its job.

npm run calculator

Example 3: Streaming and Model Configuration

The first two examples wait for the full response before printing anything. That's fine for short answers, but for anything longer you want tokens streaming as they arrive:

import { Agent, BedrockModel } from '@strands-agents/sdk'

const model = new BedrockModel({
  // modelId: 'us.anthropic.claude-sonnet-4-5-20250929-v1:0',
  // region: 'us-east-1',
  temperature: 0.7,
  maxTokens: 1024,
})

const agent = new Agent({
  model,
  systemPrompt: 'You are a helpful assistant. Keep responses concise.',
  printer: false,
})

const prompt = process.argv[2] || 'Explain how a CPU works in five sentences.'

console.log(`Prompt: ${prompt}\n`)
for await (const event of agent.stream(prompt)) {
  if (event.type === 'modelContentBlockDeltaEvent' && event.delta.type === 'textDelta') {
    process.stdout.write(event.delta.text)
  }
}
process.stdout.write('\n')

agent.stream() returns an async iterable of every event in the agent loop: text deltas, tool call events, tool results, metadata. We filter for textDelta because we only want the model's text on stdout, not the tool orchestration noise.

The commented-out lines show how to pin a specific model and region. The us. prefix means cross-region inference, routing requests across US regions. Worth considering for agent workloads: multi-step loops that make several model calls in sequence benefit from the distributed capacity.

npm run streaming

Example 4: Multi-Tool Orchestration

This is where agents start to feel like agents rather than fancy API wrappers. The researcher example has three tools: search, compare, and summarize. The agent decides which to call and in what order:

const search = tool({
  name: 'search',
  description: 'Search the knowledge base for information about a topic',
  inputSchema: z.object({
    query: z.string().describe('The search query'),
  }),
  callback: ({ query }) => {
    const key = query.toLowerCase()
    for (const [topic, info] of Object.entries(knowledgeBase)) {
      if (key.includes(topic)) return { found: true, topic, info }
    }
    return { found: false, topic: key, info: 'No results found.' }
  },
})

The compare tool structures side-by-side comparisons, and summarize creates formatted output. Ask the agent to compare TypeScript and Rust and here's what actually happens:

Prompt → Agent
  ├─ Agent calls search("TypeScript") → gets results
  ├─ Agent calls search("Rust") → gets results
  ├─ Agent calls compare(typescript_data, rust_data) → structured comparison
  ├─ Agent calls summarize(comparison) → formatted output
  └─ Agent returns final response with the summary

This is the part worth paying attention to. The agent isn't following a hardcoded sequence. It's making decisions at each step based on what the previous tool returned. It could search both topics first, then compare. Or it could search one, realize it needs more context, and search again before moving on. The model drives the orchestration.

npm run researcher

The knowledge base here is simulated, but the pattern is real. Replace it with API calls, database queries, or file operations and you have a working research agent.

Example 5: Multi-Turn Conversation

All the previous examples are single-shot: one prompt, one response. Real applications usually need back-and-forth. Call agent.invoke() multiple times on the same instance and it maintains the conversation history using a sliding window manager:

const rl = createInterface({ input: process.stdin, output: process.stdout })

console.log('Multi-turn chat. Conversation context is preserved across messages.')
console.log('Type "exit" to quit.\n')

while (true) {
  const input = await ask('You: ')
  if (input.trim().toLowerCase() === 'exit') break

  const result = await agent.invoke(input)
  console.log(`\nAgent: ${result.toString()}\n`)
}

Each call adds to the message history. The agent remembers what you discussed and can reference earlier context. This is the foundation for chatbots, interactive assistants, or any agent that needs ongoing interaction.

npm run chat

What's Not Covered

The TypeScript SDK is still catching up to the Python SDK. A few things that aren't available yet: structured output, multi-agent patterns (graph, swarm, workflow orchestration), callback handlers for streaming (TypeScript uses async iterators exclusively), module-based tool loading, and the observability/telemetry stack. The documentation tracks what's available in each language.

What is here covers the core well: model invocation, tool use with type safety, streaming, and conversation management. For most agent use cases, that's the foundation you build on.

These five examples all run locally against Bedrock. The next interesting question is what happens when you deploy an agent as a service: behind an API, on Lambda, handling concurrent requests with proper error handling and observability. That's where the patterns get more nuanced, and it's what I plan to cover next.

Additional Resources

If you try any of these examples or build something with the SDK, I'd like to hear about it in the comments.

Building the AWS Serverless Power for Kiro

Gunnar Grosch — Sun, 22 Feb 2026 14:46:04 +0000

Kiro can scaffold a Lambda function and wire up an API Gateway. But ask it to choose between Step Functions and Durable Functions for your workflow, or to configure a Kafka event source mapping with the right VPC setup and authentication, and you'll hit a wall. General knowledge gets you a working template. Practical experience gets you one that holds up in production.

Powers, introduced at re:Invent 2025, are Kiro's answer to this. A Power bundles an MCP server connection, best practices, and workflow guidance into a package that loads dynamically when relevant. Without framework context, agents guess at patterns and configurations. Powers give the agent instant access to specialized knowledge, but only when it's actually needed, avoiding the context bloat you get from loading everything upfront.

I'd already encoded this serverless expertise as a Claude Code plugin. The AWS Serverless Kiro Power brings that same knowledge to Kiro. This post walks through what it covers, how it's built, and what I learned along the way.

What the Power Covers

The scope is the full serverless development lifecycle on AWS, backed by 25 tools from the AWS Serverless MCP Server and ten steering guides that provide decision-making context. Project initialization, building, local testing, deployment, web application hosting, event source mappings, security, and observability. Rather than listing every capability, here's a concrete example of the difference the Power makes.

Kafka ESM setup: without the Power vs. with it

Tell Kiro "set up a Lambda function to process messages from my MSK cluster." Without the Power, you get a Lambda function with an MSK event source mapping. The basics are there, but the batch size is the default, there's no error handling for partial batch failures, the IAM policy is too broad, and the VPC configuration doesn't account for your cluster being in private subnets. It works until it doesn't.

With the Power, the esm_guidance tool asks about your cluster configuration before generating anything. The steering guides inform the agent to configure BisectBatchOnFunctionError, set up a DLQ for failed messages, and choose a batch size based on your message throughput. secure_esm_msk_policy generates a least-privilege IAM policy scoped to your specific cluster ARN instead of using wildcards. The VPC configuration uses private subnets with security group rules for your broker ports. esm_optimize tunes the parallelization factor if you need higher throughput. The result handles the edge cases that only show up after you've run the setup in production for a while.

That pattern repeats across the Power. Project setup through sam_init uses the right template for your use case instead of the default. deploy_webapp handles the full CloudFront and S3 setup for Lambda Web Adapter deployments. The observability guide knows which CloudWatch metrics to alarm on and what thresholds make sense. The troubleshooting guide maps symptoms to root causes.

How It's Built

aws-serverless-kiro-power/
├── POWER.md                           # Tool docs, best practices, troubleshooting
├── mcp.json                           # MCP server connection config
└── steering/                          # On-demand workflow guidance
    ├── getting-started.md             # Prerequisites and first-use walkthrough
    ├── sam-project-setup.md           # SAM initialization and workflow
    ├── cdk-project-setup.md           # CDK constructs, testing, pipelines
    ├── web-app-deployment.md          # Full-stack deployment patterns
    ├── event-sources.md               # Event source mapping configuration
    ├── event-driven-architecture.md   # EventBridge, Pipes, schema registry
    ├── orchestration-and-workflows.md # Step Functions, Durable Functions
    ├── observability.md               # Logging, tracing, metrics, dashboards
    ├── optimization.md                # Performance and cost tuning
    └── troubleshooting.md             # Symptom-based diagnosis

The split between POWER.md and the steering files is deliberate. POWER.md is always loaded. Its frontmatter defines the activation keywords that tell Kiro when to load the Power:

---
name: "aws-serverless"
displayName: "AWS Serverless"
description: "Build and deploy serverless applications with AWS Lambda, SAM, API Gateway, EventBridge, Step Functions, and event-driven architectures"
keywords: ["serverless", "lambda", "sam", "cdk", "api gateway", "aws", "deployment", "cloudformation", "event-driven", "microservices", "backend", "web app", "dynamodb", "kinesis", "sqs", "kafka", "deploy", "cloudwatch", "cold start", "rest api", "s3", "eventbridge", "function url", "step functions", "durable functions", "state machine"]
author: "Gunnar Grosch"
---

Below the frontmatter, POWER.md contains tool documentation with parameter tables and quick-reference best practices. The steering files load on demand when the agent hits a specific workflow. This matters because agent context windows have limits. Loading all ten guides upfront would eat context budget on knowledge the agent might not need for the current task. On-demand loading means the Kafka ESM guide loads when you ask about Kafka, not when you're deploying a web app.

mcp.json connects to the AWS Serverless MCP Server with --allow-write and --allow-sensitive-data-access flags enabled. You can remove either flag if you want the agent to advise without modifying your AWS account.

Design Decisions

A few things I learned building this:

Keep steering files focused on decisions, not templates

Early versions had full YAML templates, CI/CD pipelines, and code examples in the steering files. The problem is that the MCP tools already generate these. Having them in the steering files just duplicates content and wastes context. The final versions focus on decision-making guidance: when to use which deployment type, how to choose batch sizes, what metrics to alarm on.

Document every tool with actual parameters

I validated every tool's parameters against the actual MCP server schemas. This matters because if POWER.md says a parameter is called function_identifier but the tool actually expects resource_name, the agent will fail silently or hallucinate parameters. All 25 tools have correct required/optional parameter listings.

Follow the official Power structure

The official Kiro Powers (Stripe, Neon, Datadog, and others) follow a consistent structure in POWER.md: frontmatter with keywords, overview, available steering files, full MCP tool documentation, usage examples, Do/Don't best practices, Error/Cause/Solution troubleshooting, and configuration. Following this pattern means Kiro knows exactly where to find what it needs.

The expertise is portable, the packaging isn't

I built the Claude Code plugin first. The ten steering guides transferred to the Kiro Power almost entirely. The packaging didn't: Claude Code uses SKILL.md as the entry point, Kiro uses POWER.md with different frontmatter conventions. Claude Code uses .mcp.json, Kiro uses mcp.json. Claude Code's plugin includes a PostToolUse hook that auto-validates SAM templates after edits, which doesn't package into a Power. You could set up that hook in Kiro separately, but it's not something the Power installs for you. Distribution is different too: marketplace installation versus direct GitHub import.

But the real investment is the expertise layer: the decision trees, the operational knowledge, the judgment calls that come from experience. Writing that once and adapting the packaging was significantly less work than building from scratch. This is the same idea behind the Agent Skills standard: encode expertise in a format that multiple tools can consume. If you structure knowledge as clear decision guidance in markdown files, the per-tool adaptation is mostly mechanical.

Getting Started

Prerequisites

You'll need:

AWS CLI configured with credentials
AWS SAM CLI installed
Docker Desktop (for local testing)
Python 3.10+ with uv package manager (this runs the MCP server locally, regardless of what language your application uses)

Installation

Open the Powers panel in Kiro
Click "Add Custom Power" and select "Import power from GitHub"
Enter: https://github.com/gunnargrosch/aws-serverless-kiro-power
Press "Enter" to confirm

Try It Out

The Power activates on keywords like serverless, lambda, sam, deploy, dynamodb, kinesis, sqs, and others. Just describe what you want to build:

> I need a Python API on Lambda with DynamoDB behind it
> the iterator age on my kinesis stream keeps climbing, help me figure out why
> set up an EventBridge bus for order events with routing to three downstream services
> my cold starts are killing me, what can I do

Kiro uses the MCP tools to scaffold, build, test, and deploy, following the patterns in the steering files.

Making It Better

Both the Claude Code plugin and the Kiro Power are open source. The underlying expertise is shared, so a fix in one improves both. Some areas where contributions would be most useful:

Steering guides for additional event source patterns (DocumentDB, MQ, S3 event notifications)
Real-world troubleshooting scenarios that the current guides don't cover
Corrections where the guidance doesn't match your production experience

Open an issue or a PR on whichever repo you're using.

Additional Resources

What Powers have you built or are you thinking about building? Let me know in the comments.