DEV Community: Varun Sharma

Streaming Bedrock Tokens with AWS Lambda Function URLs: Real Architecture, Real Tradeoffs

Varun Sharma — Sat, 03 Oct 2026 04:50:32 +0000

When I first wired up Amazon Bedrock's ConverseStream to a web frontend, I assumed the tricky part would be parsing the EventStream chunks in JavaScript.

It wasn't. The real headache was everything sitting between Bedrock and the browser.

If you deploy a standard Lambda function behind an API Gateway HTTP API, the problem becomes obvious on the first test: nothing reaches the screen until the entire response finishes. For a quick 20-word greeting, nobody notices. But for a multi-paragraph Claude 3.7 or Nova response that takes 15 seconds to generate, your user is left staring at a dead loading spinner.

Bedrock streams tokens natively, but getting those chunks to a client browser over serverless AWS infrastructure without buffering them in transit turns out to have a few sharp edges.

When to Use API Gateway vs. Function URLs

Most tutorials reflexively point you to API Gateway. And to be fair, AWS quietly addressed the biggest historical complaint in late 2025: REST APIs finally support native response streaming (via STREAM transfer mode), and you can raise integration timeouts up to 15 minutes.

So before you bypass API Gateway, look at your existing stack:

If your workload already runs behind an API Gateway REST API—and you rely on its usage plans, request validation, or Cognito authorizers—you probably don't need to reinvent the wheel. Switch your method integration to Transfer Mode: STREAM, configure your client to handle chunked responses, and keep your existing API management in place.

However, there are two practical reasons you might want Lambda Function URLs instead:

First, API Gateway HTTP APIs (the cheaper, lighter variant most teams prefer for greenfield serverless apps) still do not support response streaming. They buffer the entire payload before returning a single byte.

Second, the economics shift at scale. REST APIs charge $3.50 per million calls plus an extra $0.09 per gigabyte of streamed data (after the first 10 GB). If you're building a chat application streaming millions of long completions a month, paying API Gateway's data transfer and request markup on high-volume token streams starts looking like unnecessary overhead.

Lambda Function URLs with InvokeMode: RESPONSE_STREAM give you native HTTP chunked transfer encoding (Transfer-Encoding: chunked) directly from the Lambda microVM to the client. You skip the API Gateway layer and its per-request fees completely.

The trade-off is access control. A Function URL is a naked public HTTPS endpoint unless you lock it down. And doing that behind CloudFront without breaking streaming takes some deliberate design.

flowchart TD
    Client[Web Browser: fetch ReadableStream] -->|SSE Stream| CF[Amazon CloudFront CDN]
    CF -->|Chunked HTTP with X-Origin-Verify| FURL[Lambda Function URL: RESPONSE_STREAM]
    FURL -->|EventStream| Bedrock[Amazon Bedrock ConverseStream]

The CloudFront & Auth Dilemma

AWS documentation frequently suggests locking down Function URLs with AWS_IAM auth and putting CloudFront Origin Access Control (OAC) in front.

Here’s why that falls apart for interactive web apps:

CloudFront OAC cannot sign browser-initiated POST request bodies with SigV4 without a custom edge signing proxy or Lambda@Edge worker. Browser fetch requests cannot generate AWS SigV4 signatures without baking IAM access keys into client-side JavaScript—an obvious security disaster.

If you flip the Function URL to AuthType: NONE so browsers can POST prompts directly, your Lambda endpoint is exposed to anyone on the internet who discovers the URL.

The battle-tested serverless pattern uses a layered compromise:

Origin Restriction: Set the Function URL to AuthType: NONE, but place CloudFront in front of it. Have CloudFront inject a secret custom header (X-Origin-Verify: <secret>) when forwarding requests to the origin. In the Lambda handler, verify this header using constant-time string comparison (crypto.timingSafeEqual in Node, secrets.compare_digest in Python) before touching Bedrock. Any direct hit to the Function URL gets bounced with a 403.
User Authentication: Remember that X-Origin-Verify only proves the request came through CloudFront; it is not user authentication. In a production application, client requests must carry user session tokens (Cognito JWTs, session cookies, or API keys). You can validate these at the CloudFront edge via Lambda@Edge / CloudFront Functions, or inside the Lambda handler.
Edge Buffering Policies: CloudFront's default behavior will buffer chunked streams. You must explicitly attach two AWS-managed policies to the cache behavior:
- CachePolicyId: 4135ea2d-6df8-44a3-9df3-44ca84e08fad (CachingDisabled): Tells CloudFront edge locations to flush SSE chunks immediately instead of aggregating them.
- OriginRequestPolicyId: b689b0a8-53d0-40ab-baf2-68738e2966ac (AllViewerExceptHostHeader): Rewrites the viewer's Host header to match the Function URL domain. Without this policy, Lambda rejects the request with a 403 because the Host header does not match the origin.
WAF Protection: Attach AWS WAF to CloudFront to enforce IP rate limits, geo-restrictions, and bot control before requests reach Lambda.

The Node.js 22 Gotcha: Bedrock Event Ordering

In Node.js 22, response streaming is supported via awslambda.streamifyResponse(). Wrapping responseStream with awslambda.HttpResponseStream.from(responseStream, headers) lets you emit HTTP status codes and CORS headers before you write SSE (text/event-stream) chunks.

Here is a subtle gotcha that cost me an afternoon:

If you follow naive SDK samples, you'll probably look for token usage inside the chunk.messageStop event. But in Bedrock's actual ConverseStream protocol, chunk.messageStop only carries the stopReason. Token usage metrics (inputTokens, outputTokens) arrive in a subsequent metadata chunk right before the connection closes.

If you attempt to read chunk.metadata?.usage inside the messageStop branch, it evaluates to undefined every single time.

To accurately report token counts, you must collect stopReason and usage across the loop and send your terminal completion event once both have arrived:

// lambda/index.mjs
import crypto from "node:crypto";
import {
  BedrockRuntimeClient,
  ConverseStreamCommand,
} from "@aws-sdk/client-bedrock-runtime";

const bedrock = new BedrockRuntimeClient({
  region: process.env.AWS_REGION || "us-east-1",
});

// Hardcode allowed models — never trust client-supplied model IDs
const ALLOWED_MODELS = new Set([
  "amazon.nova-pro-v1:0",
  "amazon.nova-lite-v1:0",
  "us.anthropic.claude-3-7-sonnet-20250219-v1:0",
  "us.anthropic.claude-3-5-haiku-20241022-v1:0",
]);

const DEFAULT_MODEL_ID = process.env.BEDROCK_MODEL_ID || "amazon.nova-pro-v1:0";
const ALLOWED_ORIGIN = process.env.ALLOWED_ORIGIN || "https://yourdomain.com";
const EXPECTED_ORIGIN_VERIFY = process.env.ORIGIN_VERIFY_SECRET;
const EXPECTED_API_KEY = process.env.APP_API_KEY;

const MAX_TOTAL_PROMPT_CHARS = 4000;
const MAX_SYSTEM_CHARS = 1000;
const MAX_BODY_BYTES = 50 * 1024; // 50 KB

function safeCompare(a, b) {
  if (typeof a !== "string" || typeof b !== "string") return false;
  const bufA = Buffer.from(a);
  const bufB = Buffer.from(b);
  if (bufA.length !== bufB.length) return false;
  return crypto.timingSafeEqual(bufA, bufB);
}

export const handler = awslambda.streamifyResponse(
  async (event, responseStream, context) => {
    const headers = {
      "Content-Type": "text/event-stream; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "Connection": "keep-alive",
      "X-Accel-Buffering": "no",
      "Access-Control-Allow-Origin": ALLOWED_ORIGIN,
      "Access-Control-Allow-Methods": "POST, OPTIONS",
      "Access-Control-Allow-Headers": "Content-Type, Authorization, x-api-key, x-origin-verify",
    };

    if (event.requestContext?.http?.method === "OPTIONS") {
      const res = awslambda.HttpResponseStream.from(responseStream, { statusCode: 204, headers });
      res.end();
      return;
    }

    // 1. Verify CloudFront origin secret using constant-time comparison
    if (EXPECTED_ORIGIN_VERIFY) {
      const originHeader = event.headers?.["x-origin-verify"] || event.headers?.["X-Origin-Verify"];
      if (!safeCompare(originHeader, EXPECTED_ORIGIN_VERIFY)) {
        const res = awslambda.HttpResponseStream.from(responseStream, {
          statusCode: 403,
          headers: { "Content-Type": "application/json", "Access-Control-Allow-Origin": ALLOWED_ORIGIN },
        });
        res.write(JSON.stringify({ error: "Forbidden: Direct Function URL access blocked." }));
        res.end();
        return;
      }
    }

    // 2. Optional application-layer API key check (useful for service-to-service calls)
    if (EXPECTED_API_KEY) {
      const requestApiKey = event.headers?.["x-api-key"] || event.headers?.["X-Api-Key"];
      if (!safeCompare(requestApiKey, EXPECTED_API_KEY)) {
        const res = awslambda.HttpResponseStream.from(responseStream, {
          statusCode: 401,
          headers: { "Content-Type": "application/json", "Access-Control-Allow-Origin": ALLOWED_ORIGIN },
        });
        res.write(JSON.stringify({ error: "Unauthorized." }));
        res.end();
        return;
      }
    }

    // 3. Payload size check
    const rawBody = event.body || "";
    if (Buffer.byteLength(rawBody, "utf8") > MAX_BODY_BYTES) {
      const res = awslambda.HttpResponseStream.from(responseStream, {
        statusCode: 413,
        headers: { "Content-Type": "application/json", "Access-Control-Allow-Origin": ALLOWED_ORIGIN },
      });
      res.write(JSON.stringify({ error: "Payload exceeds 50 KB limit." }));
      res.end();
      return;
    }

    const stream = awslambda.HttpResponseStream.from(responseStream, {
      statusCode: 200,
      headers,
    });

    const sendSSE = (data, eventType = "message") => {
      const payload = typeof data === "string" ? data : JSON.stringify(data);
      stream.write(`event: ${eventType}\ndata: ${payload}\n\n`);
    };

    try {
      const body = event.body ? JSON.parse(event.body) : {};
      const prompt = String(body.prompt || "Explain distributed consensus in two sentences.");
      const systemPrompt = String(body.system || "");
      const conversationHistory = Array.isArray(body.messages) ? body.messages : [];
      const modelId = String(body.modelId || DEFAULT_MODEL_ID);

      if (systemPrompt.length > MAX_SYSTEM_CHARS) {
        sendSSE({ error: true, message: `System prompt exceeds ${MAX_SYSTEM_CHARS} char limit.` }, "error");
        stream.end();
        return;
      }

      // Check total characters across prompt string or conversation history
      let totalInputChars = 0;
      if (conversationHistory.length > 0) {
        for (const msg of conversationHistory) {
          if (Array.isArray(msg.content)) {
            for (const part of msg.content) {
              if (part.text) totalInputChars += String(part.text).length;
            }
          }
        }
      } else {
        totalInputChars = prompt.length;
      }

      if (totalInputChars > MAX_TOTAL_PROMPT_CHARS) {
        sendSSE({ error: true, message: `Input text (${totalInputChars} chars) exceeds ${MAX_TOTAL_PROMPT_CHARS} limit.` }, "error");
        stream.end();
        return;
      }

      if (!ALLOWED_MODELS.has(modelId)) {
        sendSSE({ error: true, message: `Model '${modelId}' is not allowed.` }, "error");
        stream.end();
        return;
      }

      sendSSE({ status: "connected", modelId }, "init");

      const messages = conversationHistory.length > 0
        ? conversationHistory
        : [{ role: "user", content: [{ text: prompt }] }];

      const command = new ConverseStreamCommand({
        modelId,
        messages,
        system: systemPrompt ? [{ text: systemPrompt }] : undefined,
        inferenceConfig: { maxTokens: 2048, temperature: 0.7 },
      });

      const bedrockResponse = await bedrock.send(command);

      let stopReason = null;
      let tokenUsage = null;

      for await (const chunk of bedrockResponse.stream) {
        if (chunk.contentBlockDelta?.delta?.text) {
          sendSSE({ text: chunk.contentBlockDelta.delta.text }, "delta");
        }
        if (chunk.messageStop) {
          stopReason = chunk.messageStop.stopReason;
        }
        if (chunk.metadata?.usage) {
          tokenUsage = chunk.metadata.usage;
        }
      }

      // Send completion event once both stopReason and usage have arrived
      sendSSE({ stopReason, usage: tokenUsage }, "done");
    } catch (err) {
      console.error("Bedrock stream error:", err);
      sendSSE({ error: true, message: err.message }, "error");
    } finally {
      stream.end();
    }
  }
);

The Python Concurrency Trap: Event Loop Starvation

If you build this in Python with FastAPI and run it on Lambda via AWS Lambda Web Adapter (AWS_LWA_INVOKE_MODE=response_stream), there is a deceptive concurrency trap waiting for you.

The reason this catches so many developers off guard is that FastAPI looks and feels fully asynchronous. You declare async def chat_stream(...), you return a StreamingResponse, and you assume FastAPI handles concurrent users effortlessly.

The illusion breaks because Boto3's converse_stream isn't an asyncio library. Under the hood, botocore's EventStream wraps a synchronous blocking network socket. If you write:

# WARNING: This destroys concurrency across all users
for event in response.get("stream"):  # BLOCKS the main event loop thread!
    yield f"data: {event}\n\n"

When you write for event in stream:, Python performs synchronous socket reads directly on the main thread running the asyncio event loop. Every time the worker waits 30–80ms for Bedrock to push the next token chunk, the entire event loop halts. Under concurrent load, all incoming requests freeze waiting on each other's network packets.

The clean solution is a classic producer-consumer pattern:

Offload the blocking Boto3 socket iterator to a background daemon thread, push events into an asyncio.Queue using loop.call_soon_threadsafe(), and let your FastAPI route await queue.get().

And make sure to handle client disconnects: if a user closes their tab mid-generation, your generator loop exits, but the background thread will keep happily burning Bedrock tokens until the model finishes unless you signal a threading.Event to abort it:

# lambda/python_adapter/main.py
import os
import json
import asyncio
import secrets
import threading
from typing import AsyncGenerator, Optional
import boto3
from fastapi import FastAPI, HTTPException, Request, Header
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field

app = FastAPI()
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
_STREAM_END = object()

ALLOWED_MODELS = {
    "amazon.nova-pro-v1:0",
    "amazon.nova-lite-v1:0",
    "us.anthropic.claude-3-7-sonnet-20250219-v1:0",
    "us.anthropic.claude-3-5-haiku-20241022-v1:0",
}

EXPECTED_ORIGIN_VERIFY = os.getenv("ORIGIN_VERIFY_SECRET")

def safe_compare(val: Optional[str], expected: Optional[str]) -> bool:
    if not val or not expected:
        return False
    return secrets.compare_digest(val.strip(), expected.strip())

class ChatRequest(BaseModel):
    prompt: Optional[str] = Field(None, max_length=4000)
    system: Optional[str] = Field("You are a concise, accurate AI assistant.", max_length=1000)
    modelId: Optional[str] = Field("amazon.nova-pro-v1:0")

async def stream_bedrock_events(request: ChatRequest, http_req: Request) -> AsyncGenerator[str, None]:
    if request.modelId not in ALLOWED_MODELS:
        yield f"event: error\ndata: {json.dumps({'error': f'Model {request.modelId} not allowed.'})}\n\n"
        return

    queue = asyncio.Queue()
    loop = asyncio.get_running_loop()
    stop_event = threading.Event()

    def worker():
        try:
            response = bedrock.converse_stream(
                modelId=request.modelId,
                messages=[{"role": "user", "content": [{"text": request.prompt or "Hello"}]}],
            )
            for event in response.get("stream"):
                if stop_event.is_set():
                    break
                loop.call_soon_threadsafe(queue.put_nowait, event)
            loop.call_soon_threadsafe(queue.put_nowait, _STREAM_END)
        except Exception as e:
            loop.call_soon_threadsafe(queue.put_nowait, e)

    thread = threading.Thread(target=worker, daemon=True)
    thread.start()

    yield f"event: init\ndata: {json.dumps({'status': 'connected', 'modelId': request.modelId})}\n\n"

    stop_reason = None
    token_usage = None

    try:
        while True:
            # Check client disconnection to avoid paying for unread tokens
            if await http_req.is_disconnected():
                stop_event.set()
                break

            try:
                item = await asyncio.wait_for(queue.get(), timeout=0.5)
            except asyncio.TimeoutError:
                continue

            if item is _STREAM_END:
                break
            if isinstance(item, Exception):
                yield f"event: error\ndata: {json.dumps({'error': str(item)})}\n\n"
                break

            if "contentBlockDelta" in item:
                text = item["contentBlockDelta"]["delta"]["text"]
                yield f"event: delta\ndata: {json.dumps({'text': text})}\n\n"
            elif "messageStop" in item:
                stop_reason = item["messageStop"].get("stopReason")
            elif "metadata" in item:
                token_usage = item["metadata"].get("usage")

        if not stop_event.is_set():
            yield f"event: done\ndata: {json.dumps({'stopReason': stop_reason, 'usage': token_usage})}\n\n"
    finally:
        stop_event.set()

@app.post("/stream")
async def chat_endpoint(
    req: ChatRequest,
    http_req: Request,
    x_origin_verify: Optional[str] = Header(None, alias="x-origin-verify"),
):
    if EXPECTED_ORIGIN_VERIFY and not safe_compare(x_origin_verify, EXPECTED_ORIGIN_VERIFY):
        raise HTTPException(status_code=403, detail="Forbidden: Direct Function URL access blocked.")

    return StreamingResponse(
        stream_bedrock_events(req, http_req),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

The Practical Threat Model

When you put an LLM behind an open HTTP endpoint, the threat model is less about data breaches and more about financial denial-of-service. An attacker doesn't need to crack your encryption; they just need to flood your endpoint with requests asking for 4,000 output tokens each.

Here are the guardrails you need in place before deploying:

Scope your IAM execution role: Avoid Resource: "*". In fact, avoid foundation-model/* as well. Pin your policy strictly to the foundation model and inference profile ARNs you intend to support:

  - Effect: Allow
    Action:
      - bedrock:InvokeModelWithResponseStream
      - bedrock:ConverseStream
    Resource:
      - !Sub "arn:${AWS::Partition}:bedrock:${AWS::Region}::foundation-model/amazon.nova-pro-v1:0"
      - !Sub "arn:${AWS::Partition}:bedrock:${AWS::Region}::foundation-model/amazon.nova-lite-v1:0"
      - !Sub "arn:${AWS::Partition}:bedrock:${AWS::Region}::foundation-model/anthropic.claude-3-5-haiku-20241022-v1:0"
      - !Sub "arn:${AWS::Partition}:bedrock:${AWS::Region}::foundation-model/anthropic.claude-3-7-sonnet-20250219-v1:0"
      - !Sub "arn:${AWS::Partition}:bedrock:${AWS::Region}:${AWS::AccountId}:inference-profile/*"
      - !Sub "arn:${AWS::Partition}:bedrock:*:*:inference-profile/us.anthropic.claude-3-7-sonnet-20250219-v1:0"
      - !Sub "arn:${AWS::Partition}:bedrock:*:*:inference-profile/us.anthropic.claude-3-5-haiku-20241022-v1:0"

Enforce an in-code allowlist: Never pass req.body.modelId straight to the SDK. Validate every request against an explicit set of approved models.
Cap input text & payloads: Tally character counts across prompts, conversation histories (4,000 chars), and system prompts (1,000 chars), and reject raw bodies over 50 KB before calling Bedrock.
Halt on client disconnect: As shown in the Python implementation, abort the stream worker if the client disconnects so you aren't paying for tokens no viewer will ever see.
WAF rate limiting: Attach AWS WAF to CloudFront with an IP-based rate limit (e.g. 100 requests per 5 minutes) to absorb automated scraping.

Bandwidth & Throughput Limits

Lambda response streaming delivers an initial 6 MB unthrottled burst, after which subsequent throughput is capped at 2 MB/s up to a maximum 200 MB response. For text-based LLM token streaming, that ceiling is orders of magnitude higher than the generation speed of current foundation models.

Wrap-Up

There is no single "right" serverless streaming pattern on AWS:

If your application already lives behind API Gateway REST APIs and depends on its usage plans, API keys, or Cognito integration, flip the integration transfer mode to STREAM and keep your existing infrastructure.

But if you want a direct, low-latency streaming pipeline for a web or mobile client without paying API Gateway invocation and data processing markup, Lambda Function URLs with CloudFront origin verification deliver a clean, fast, serverless streaming architecture.

All of the code—including working SAM templates, CDK v2 stack, Node and Python handlers, and local test scripts—is on GitHub:

GitHub Repository: sharma-the-karma/serverless-bedrock-token-streaming

Building an AWS DevOps Agent That Diagnoses but Never Deploys

Varun Sharma — Sat, 03 Oct 2026 02:22:29 +0000

Most CI/CD pipeline failures take longer to diagnose than to fix. An IAM role is missing an action after an update. A Terraform state lock didn't clear after a cancelled job. A container layer cache busted on an architecture mismatch. The fix itself is usually a two-line patch, but finding it means waiting for a massive raw terminal log to render in GitHub Actions and hunting for the error line.

I wanted to automate that triage step without adding complex machinery. When a GitHub Actions run fails, a lightweight workflow pulls the failure log, runs it through Amazon Bedrock using Claude, and comments on the pull request with a diagnosis and suggested diff. The implementation is plain Python, GitHub Actions, AWS OIDC federation, and Bedrock. The code is on GitHub at github.com/sharma-the-karma/aws-devops-agent.

How it works

The triage job only runs when a pipeline fails, so there is no persistent infrastructure or long-lived AWS keys in repository secrets.

flowchart LR
    A[GitHub Actions CI Workflow] -->|Fails| B[Triage Trigger]
    B --> C[Log Distiller Engine]
    C -->|High-Signal Context| D[AWS Bedrock Converse API]
    D -->|Claude| E[Diagnostic Report]
    E --> F[PR Comment with Diff]

    subgraph AWS Cloud
        D
        G[IAM OIDC Role] -.->|Federates| B
    end

The sequence is straightforward:

A CI job fails on a pull request, triggering the triage workflow via the workflow_run event.
The runner requests a temporary token from GitHub's OIDC provider and assumes an IAM role in AWS.
A Python script downloads the failed step's logs, strips ANSI color codes, and extracts the lines surrounding the failure signature.
The distilled snippet is sent to Bedrock's Converse API.
The diagnosis is posted back to the pull request, editing the previous comment if one already exists.

What I learned

Raw logs wreck the model's reasoning

Feeding the entire raw console log into an LLM does not work well. A typical CI run prints thousands of lines of package downloads, container build steps, and environment setup before hitting the failure. In testing, sending that full output bloated the prompt to tens of thousands of tokens. The model frequently latched onto benign compiler warnings earlier in the log rather than the actual fatal error at the bottom, while adding 15 to 20 seconds of unnecessary latency.

The fix was a small distillation utility (src/log_distiller.py). It strips ANSI formatting and scans for common failure markers like AccessDeniedException, Python tracebacks, Terraform errors, and non-zero exit codes. When it finds a match, it extracts a rolling window of 15 lines before and 35 lines after:

import re
from typing import Dict, Any

ANSI_ESCAPE_PATTERN = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")

ERROR_INDICATORS = [
    re.compile(r"error[:\s]", re.IGNORECASE),
    re.compile(r"traceback \(most recent call last\):", re.IGNORECASE),
    re.compile(r"failed with exit code \d+", re.IGNORECASE),
    re.compile(r"accessdeniedException", re.IGNORECASE),
    re.compile(r"terraform error[:\s]", re.IGNORECASE),
]

def extract_failure_context(raw_log: str, window_before: int = 15, window_after: int = 35) -> Dict[str, Any]:
    clean_lines = ANSI_ESCAPE_PATTERN.sub("", raw_log).splitlines()
    selected_indices = set()

    for idx, line in enumerate(clean_lines):
        for pattern in ERROR_INDICATORS:
            if pattern.search(line):
                start = max(0, idx - window_before)
                end = min(len(clean_lines), idx + window_after + 1)
                selected_indices.update(range(start, end))
                break

    sorted_indices = sorted(selected_indices)
    extracted = [f"L{i+1:04d}: {clean_lines[i]}" for i in sorted_indices]

    return {
        "distilled_log": "\n".join(extracted),
        "total_lines_analyzed": len(clean_lines),
        "lines_extracted": len(extracted)
    }

This reduces the payload from tens of thousands of tokens down to roughly 500 to 800 tokens of high-signal context. Response times drop to a few seconds, and the model focuses on the actual failure rather than ambient build noise.

Don't give the agent permanent keys

Avoid creating static AWS access keys (AKIA...) for CI/CD runners. If a pull request modifies workflow files or runs untrusted scripts, stored secrets can be exposed.

Instead, configure an IAM OIDC Identity Provider in AWS for GitHub Actions. The runner exchanges a short-lived GitHub token for temporary STS credentials.

data "aws_iam_policy_document" "github_oidc_trust" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:sharma-the-karma/aws-devops-agent:ref:refs/heads/main"]
    }
  }
}

Because workflow_run triggers execute in the context of the default branch, the sub claim can be pinned to ref:refs/heads/main rather than using a wildcard. The role's policy grants bedrock:InvokeModel and nothing else. The agent cannot read S3, modify security groups, or alter infrastructure.

Use the Converse API

Amazon Bedrock provides the Converse API, which standardizes system prompts, messages, and inference parameters across foundation models.

import boto3

SYSTEM_PROMPT = """You are an AWS DevOps and Infrastructure Architect.
Inspect failing CI/CD logs, diagnostic traces, and infrastructure errors.
Provide a concise, practical triage assessment with actionable solutions.

Structure your response into:
1. Root Cause Breakdown (what failed and its category)
2. Immediate Fix / Patch (exact code snippet or command)
3. Preventive Action & Blast Radius (how to prevent it and potential side-effects)
"""

client = boto3.client("bedrock-runtime", region_name="us-east-1")

response = client.converse(
    modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=[
        {
            "role": "user",
            "content": [{"text": f"CI/CD Failure Trace:\n{distilled_log}"}]
        }
    ],
    system=[{"text": SYSTEM_PROMPT}],
    inferenceConfig={
        "maxTokens": 2048,
        "temperature": 0.2,
        "topP": 0.9,
    }
)

diagnostic_report = response["output"]["message"]["content"][0]["text"]

I used anthropic.claude-3-5-sonnet-20241022-v2:0 here, but for standard CI failure triage where you just need quick stack trace interpretation, anthropic.claude-3-5-haiku-20241022-v1:0 is faster, cheaper, and handles compiler errors just as effectively. A temperature of 0.2 keeps the diagnostic output consistent across runs.

Update the PR comment instead of adding a new one

When developers push multiple commits while troubleshooting a broken branch, bot comments can easily flood the pull request.

To prevent this, the agent tags each comment with a hidden HTML marker () and inspects existing comments before posting. If an earlier report exists, it edits the comment in place:

COMMENT_TAG = "<!-- aws-devops-agent-triage -->"

def post_or_update_pr_triage_comment(repo, pr_number: int, analysis_markdown: str):
    body = f"{COMMENT_TAG}\n### AWS DevOps Agent Triage Report\n\n{analysis_markdown}"
    pull_request = repo.get_pull(pr_number)

    for comment in pull_request.get_issue_comments():
        if COMMENT_TAG in comment.body:
            comment.edit(body)
            return "Updated existing triage comment."

    pull_request.create_issue_comment(body)
    return "Created new triage comment."

Note that the workflow's GITHUB_TOKEN requires pull-requests: write permissions for this step.

Never let the agent apply infrastructure changes

Letting an agent automatically commit code or run terraform apply is dangerous.

A simple example illustrates why: an ECS service update fails because a subnet is misconfigured. An agent can correctly identify the subnet mismatch and propose the right configuration. However, changing subnet attributes on certain AWS resources triggers a replacement, destroying the existing resource before creating the new one. An automated agent executing that change on its own can easily cause unintended downtime or data loss.

The agent's role is to diagnose the failure, explain the blast radius, and provide a diff. The engineer reviews the diff and decides when and how to deploy it.

What a report looks like

Here is a representative example of what the agent posts to a pull request when an ECS service update fails on an IAM permission:

AWS DevOps Agent Triage Report

1. Root Cause Breakdown
The deployment failed with AccessDeniedException while calling ecs:UpdateServicePrimaryTaskSet. The execution role assumed by the CI runner lacks this action in its attached policy. This is an infrastructure / IAM permissions issue.

2. Immediate Fix / Patch
Add the missing action to your CI deployer policy in infra/iam_deployer.tf:
 statement {
   actions = [
     "ecs:UpdateService",
     "ecs:DescribeServices",
+    "ecs:UpdateServicePrimaryTaskSet"
   ]
   resources = [aws_ecs_service.api_service.arn]
 }
3. Preventive Action & Blast Radius
Review deployment policies whenever you introduce CodeDeploy or blue/green task sets on ECS services. Applying this change is non-destructive and requires no resource recreation.

Try it yourself

Clone the repo:

git clone https://github.com/sharma-the-karma/aws-devops-agent.git
cd aws-devops-agent

Deploy the IAM role:

cd infra
terraform init
terraform apply -var="github_org=sharma-the-karma" -var="github_repo=aws-devops-agent"

Add the role ARN from the output as AWS_ROLE_ARN in your repository secrets (Settings > Secrets and variables > Actions).

To test locally against the sample log without calling anything for real:

cd src
pip install -r requirements.txt
python triage_agent.py --log-file sample_failure.log --dry-run

Next steps for this setup include hooking into CloudWatch alarm webhooks so the same triage flow works for runtime ECS task crashes and Lambda errors, rather than just CI pipeline failures.

The complete code, Terraform configuration, and sample logs are available on GitHub: github.com/sharma-the-karma/aws-devops-agent.

Productionizing Agentic GenAI on AWS: What Actually Breaks When You Scale MCP

Varun Sharma — Fri, 18 Sep 2026 01:32:43 +0000

Every MCP demo looks the same. Someone spins up a local server, points a client at it, watches a tool call succeed once, and calls the architecture done. Then the business asks when it ships, and suddenly you're dealing with non-deterministic compute, IAM roles that need to be scoped per tool instead of per service, and a token bill that can move faster than anyone's watching it.

This is what that transition looks like in practice, and what tends to get skipped when implementing MCP in production.

Stop reaching for EKS by default

Containerizing the MCP server and running it on ECS or EKS is the default move, because that's what "production" means on most platform teams. It works, but you inherit Kubernetes' entire operational surface: autoscaling tuned for a workload that doesn't scale in any predictable pattern, IAM role assumption per agent session, all of it. And this is for something that's mostly request/response with an occasional long-running tool call in the middle.

Two AWS-native paths are worth trying first.

Amazon Bedrock Agents is the original managed agent service, and a reasonable fit if your use case matches its orchestration and action-group model.

Amazon Bedrock AgentCore is the newer suite and the more relevant one here, though it's worth being precise about what's actually inside it since the naming gets muddled in most blog posts written about it. AgentCore Runtime is the serverless compute layer that hosts your agent or MCP server code so you're not managing containers. AgentCore Gateway solves a different problem: it's a managed front door that turns existing APIs, Lambda functions, and OpenAPI or Smithy specs into MCP-compatible tools, and can front other agents through passthrough targets, aggregating and securing access to whatever's already running rather than hosting anything new itself. If the goal is to stop managing containers, that's Runtime. If the goal is exposing ten internal APIs as one governed MCP endpoint without hand-writing a server for each, that's Gateway. It's an easy pair to mix up on a first read.

Whichever one fits, put it in Terraform, mainly because "which IAM role got attached to which agent in which environment" is a question you don't want to answer by clicking through the console six months from now, after whoever set it up has moved teams.

Watching for cost, not just errors

A 500, a stack trace, a page in the middle of the night: that's usually how you find out something broke. Agents don't give you that. One that gets stuck retrying, or decides to re-check its own output a few times, or loops back through the same tool call, won't throw an error for any of it. It just spends money, and you find out when the invoice lands, roughly a month too late to do anything about it.

A few things are worth the setup cost. Tagging at the session or tenant level, not just the service level, matters because "cost per API call" tells you nothing when the real unit of spend is "cost per agent decision" — you want to answer which customer, which run, and which day, without stitching together six log tables to get there.

Rate limiting deserves more attention than teams usually give it. Put real limits on the gateway in front of the MCP server, not aspirational ones. A burst limit of 20 and a sustained rate of 10 requests per second looks overly cautious right up until an agent stuck in a loop hits it four seconds into a session, at which point it's the only thing standing between you and a much worse morning.

And trace the decision path, not just the request path. X-Ray and CloudWatch will hand you latency numbers and error rates, but the thing worth seeing is why the agent called tool B right after tool A. Without that, "the agent got expensive" stays a mystery instead of turning into a fix.

Assume it will eventually do something you didn't plan for

The moment an agent can write instead of just read, being wrong stops meaning bad output and starts meaning modified infrastructure. That's the point where a bug goes from embarrassing to expensive.

Scope IAM roles per tool, not per agent. A tool built to query one table shouldn't ride on a role that also happens to reach S3 or security groups, no matter how tempting it is to reuse one role across five tools to save setup time. Authenticate before any of this fires; Cognito in front of the MCP endpoint is unglamorous, and it's still the right call. And put a guardrail layer, Bedrock Guardrails or something equivalent, between the model and anything sensitive. Prompt injection aimed at getting an agent to leak data or push an unauthorized change is already one of the more common ways these systems get abused.

What this looks like in Terraform

This covers only the identity and routing skeleton. Deployment, stage, method, and usage-plan-to-API-key resources are left out on purpose, to keep the snippet focused on the pieces that matter for the argument above.

# Execution role for the agent — trust policy scoped to Bedrock only
resource "aws_iam_role" "mcp_agent_role" {
  name = "mcp-agent-execution-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "bedrock.amazonaws.com" }
    }]
  })
}

# Per-tool policy — this agent can read from one bucket, nothing else
resource "aws_iam_role_policy" "mcp_tool_s3_read" {
  name = "mcp-tool-s3-read-policy"
  role = aws_iam_role.mcp_agent_role.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:GetObject", "s3:ListBucket"]
      Resource = [
        "arn:aws:s3:::mcp-agent-data-source",
        "arn:aws:s3:::mcp-agent-data-source/*"
      ]
    }]
  })
}

# Front door for the MCP client (e.g. an IDE integration)
resource "aws_api_gateway_rest_api" "mcp_api" {
  name        = "mcp-gateway"
  description = "API Gateway for MCP agent execution"
}

# Cognito-backed authorizer — no unauthenticated calls reach the agent
resource "aws_api_gateway_authorizer" "cognito" {
  name            = "mcp-cognito-authorizer"
  rest_api_id     = aws_api_gateway_rest_api.mcp_api.id
  type            = "COGNITO_USER_POOLS"
  provider_arns   = [aws_cognito_user_pool.mcp_pool.arn]
  identity_source = "method.request.header.Authorization"
}

# Rate limiting — this is what stops a retry loop from becoming a bill
resource "aws_api_gateway_usage_plan" "mcp_usage_plan" {
  name        = "mcp-standard-tier"
  description = "Throttling and daily quota for MCP agent invocations"

  api_stages {
    api_id = aws_api_gateway_rest_api.mcp_api.id
    stage  = aws_api_gateway_stage.prod.stage_name
  }

  quota_settings {
    limit  = 1000
    offset = 0
    period = "DAY"
  }

  throttle_settings {
    burst_limit = 20
    rate_limit  = 10
  }
}

One thing worth flagging before you copy this: if you need to track spend per tenant, this snippet alone won't get you there. A usage plan without an aws_api_gateway_usage_plan_key tied to an actual API key enforces its limits at the stage level, not per client. Add an API key per tenant and attach it to the plan if that attribution is a real requirement and not just something that'd be nice to have.

It's the same stuff you'd already do for any service and infrastructure: scope access tightly, put limits in front of it, trace what it's doing, make sure it's authenticated. The only reason it feels like a bigger deal here is that when it goes wrong, nothing pages you. You just get a bill, or a change you didn't approve, weeks after the fact. Put the guardrails in before that happens, not after you've already found out the hard way.

The AWS EBS Snapshot Cleanup Pitfalls Nobody Warns You About

Varun Sharma — Wed, 09 Sep 2026 20:30:53 +0000

If you run AWS at any real scale, you eventually accumulate thousands of
stale EBS snapshots — leftovers from decommissioned volumes, forgotten AMIs,
one-off backups nobody ever cleaned up. Cloud Custodian
(c7n) is a popular way to find and delete them on a schedule. This post
walks through four pitfalls we hit while building a "safe" cleanup
pipeline, and the general lessons underneath them — useful for anyone
running Custodian, or any policy-as-code cleanup tool, against a large
fleet.

The setup

The pattern is simple and common:

Mark snapshots older than N days for deletion (tag them, don't touch them yet).
Some days later, copy each marked snapshot as a safety net, then delete the original.
After a retention window, age off the safety copies too.

Custodian supports all of this natively with a mark-for-op / marked-for-op
tag pair and copy / delete actions. Stripped to its essentials, the
three stages look like this:

# Stage 1 -- mark eligible snapshots, touch nothing else
policies:
  - name: snapshot-mark-eligible
    resource: aws.ebs-snapshot
    filters:
      - type: age
        days: 90
        op: greater-than
      - "tag:cleanup-exclude": absent
    actions:
      - type: mark-for-op
        tag: cleanup_marked
        op: delete
        days: 0

# Stage 2 -- copy-then-delete, gated by the marked-for-op tag
policies:
  - name: snapshot-copy-then-delete
    resource: aws.ebs-snapshot
    filters:
      - type: marked-for-op
        tag: cleanup_marked
        op: delete
      - "tag:cleanup-exclude": absent
    actions:
      - type: copy
        target_region: us-east-1
        target_key: alias/aws/ebs
        encrypted: true
        tags:
          safety-copy: "true"
      - type: delete

# Stage 3 -- age off the safety copies after a retention window
policies:
  - name: snapshot-copy-age-off
    resource: aws.ebs-snapshot
    filters:
      - "tag:safety-copy": present
      - type: age
        days: 14
        op: greater-than
    actions:
      - type: delete

Run each stage with c7n-org, scoped to every account in your fleet:

# Always dry-run first -- writes matched resources to output/ without acting
c7n-org run -c accounts.yml -u policies/snapshot-copy-then-delete.yml \
  -r us-east-1 -s output --dryrun --cache-period 0

# Scope to a single account while validating a change
c7n-org run -c accounts.yml -u policies/snapshot-copy-then-delete.yml \
  -r us-east-1 -a my-account --dryrun --cache-period 0

# Execute for real once the dry-run output looks right
c7n-org run -c accounts.yml -u policies/snapshot-copy-then-delete.yml \
  -r us-east-1 --cache-period 0

On paper, it's a clean, auditable, reversible three-stage workflow. In
practice, four assumptions quietly broke it.

Worth being precise about two different things that both look like
"pipeline" here, because they fail differently:

The three stages (mark → copy+delete → age-off) really are a pipeline in the classic sense — each one's output (a tag, with a date on it) is the next one's input, separated by real elapsed time. Pitfalls here look like "the wrong things get picked up at the wrong stage."
How a single stage actually executes is not a pipeline at all — it's a fan-out worker: loop over accounts independently (no shared state, each one succeeds or safely retries on its own), and within an account, loop over batches of snapshots capped by a concurrency limit. Pitfalls here look like "something got throttled" or "a batch half-finished."

Keep those two mental models separate and the rest of this post is easier
to place: pitfalls #1, #2, and #4 are stage-pipeline pitfalls; pitfall #3
is a fan-out-worker pitfall.

What actually runs this

The mental model above — an isolated worker per account — makes "just run
it in Lambda" a tempting read. In practice, a bare Lambda struggles with
this specific workload, for three reasons:

The 15-minute execution wall. Snapshot copies — especially cross-account/region ones, or anything re-encrypting with a new KMS key — routinely take longer than 15 minutes for volumes in the 100+ GB range. A Lambda running the copy-then-poll-then-delete sequence in one invocation will hard-timeout mid-batch.
Wasted polling cost. wait_for_copies below is a time.sleep() loop. Inside Lambda you're billed for vCPU/memory for the whole time the execution context sits there waiting on an AWS storage backend to finish work Lambda has no part in.
Alignment with Stages 1 and 3. Those two already run via c7n-org across the fleet, which means a persistent workspace, established cross-account IAM role chaining, and a CLI toolchain are already in place. A scheduled container (ECS Fargate or AWS Batch) or a CI/CD runner (GitHub Actions, GitLab CI) reuses all of that directly; a Lambda would need its own separate deployment and permissions story.

If you do want this serverless, the fix isn't a bigger Lambda timeout —
there isn't one — it's decomposing the work: AWS Step Functions
orchestrating short InitiateCopy → native Wait state → CheckStatus
→ DeleteOriginal Lambdas, so no single invocation is ever blocked on a
copy that runs for hours. Absent that decomposition, a scheduled
container or CI job is the simpler, better-fitting choice — and it's what
the rest of this post assumes.

One more note before the pitfalls: the fixes below replace Stage 2 —
copy-then-delete is the unsafe part, so that's the piece that moves from a
declarative Custodian policy to a script you control directly via boto3.
Stages 1 (mark) and 3 (age-off) are simple tag filters with no ordering or
concurrency hazard, so they're left running as Custodian policies exactly
as shown above; only the middle stage changes shape.

Pitfall #1: "copy first, then delete" doesn't mean what you think

A policy like this looks safe:

actions:
  - type: copy
    target_region: us-east-1
    target_key: alias/aws/ebs
    encrypted: true
  - type: delete

Actions run in order — copy, then delete — against the same matched
resource set, in a single policy pass. The instinct is to read that as
"the copy will exist before the original is removed." It doesn't say that.

CopySnapshot is an asynchronous AWS API call. It returns immediately
with a new snapshot ID in pending state; the actual data copy can take
anywhere from seconds to hours depending on size. Custodian's copy action,
for same-region copies, does not wait for that to finish — it fires the API
call and moves straight to tagging the result, and the policy then moves
straight to the delete action. There is no dependency enforced between
"the copy exists" and "the copy is actually usable."

The failure mode: if a copy later fails (hits a transient error, a KMS
permission hiccup, a service-side throttle) after the original has already
been deleted, you're left with a snapshot in error state and no way to
recover the data it was supposed to be safety-netting. This isn't a
theoretical edge case — AWS's own CopySnapshot API reference confirms it
directly: a copy can return successfully and still fail asynchronously
afterward, moving to error state with no dependency on anything that
happened before it. It doesn't take a high failure rate to matter here: on
data you can't regenerate, even a rare miss is a real loss, not a rounding
error — and early on, running this by hand, that's exactly what forced the
move away from manual batches: babysitting every run closely enough to
catch a late copy failure before delete ran stopped being sustainable well
before the fleet did.

The fix: don't let delete run in the same breath as copy. Split the
two into separate steps, and gate the delete on the copy reaching a
terminal, successful state:

Trigger the copy (via the cloud API directly, or via your policy engine).
Poll the copy's status until it's completed or error (with a timeout).
Only delete the original for snapshots whose copy reached completed.
Anything that errored or timed out gets left alone and retried next run.

This also has a nice side effect: because you're now calling the copy API
yourself instead of relying on the policy engine's internal bookkeeping, you
get an exact mapping between "this original" and "this copy" — instead of
trying to reconstruct it after the fact from a Description field, which
(as pitfall #2 below shows) is not reliable.

def copy_batch(ec2, region, source_ids, target_key, encrypted=True):
    """Copy each source snapshot directly via boto3 so the source-id ->
    copy-id mapping is exact, not guessed from a Description field."""
    mapping = {}
    for source_id in source_ids:
        # Build kwargs conditionally: botocore rejects KmsKeyId=None outright
        # (it's a string-typed field), so only include it when target_key is
        # actually set. The API also requires Encrypted=True whenever
        # KmsKeyId is supplied -- enforce that here so a caller can't
        # silently pass encrypted=False alongside a target_key.
        copy_kwargs = {
            "SourceSnapshotId": source_id,
            "SourceRegion": region,
            "Encrypted": True if target_key else encrypted,
        }
        if target_key:
            copy_kwargs["KmsKeyId"] = target_key

        copy_id = ec2.copy_snapshot(**copy_kwargs)["SnapshotId"]
        mapping[source_id] = copy_id
    return mapping


def wait_for_copies(ec2, copy_ids, poll_seconds=15, timeout_seconds=900):
    """Poll until every copy reaches a terminal state, or give up at the timeout."""
    states, pending, deadline = {}, set(copy_ids), time.time() + timeout_seconds
    while pending and time.time() < deadline:
        resp = ec2.describe_snapshots(SnapshotIds=list(pending))
        for snap in resp["Snapshots"]:
            if snap["State"] in ("completed", "error"):
                states[snap["SnapshotId"]] = snap["State"]
                pending.discard(snap["SnapshotId"])
        if pending:
            time.sleep(poll_seconds)
    for copy_id in pending:
        states[copy_id] = "timeout"  # never reached a terminal state in time
    return states


def ready_for_delete(mapping, states):
    """Only originals whose copy is confirmed 'completed' are safe to delete."""
    return [src for src, copy_id in mapping.items() if states.get(copy_id) == "completed"]

Pitfall #2: a fixed-schedule cleanup system is only as good as its tag detection

The mark stage typically excludes anything under active lifecycle
management — snapshots tagged by AWS Backup, or by a snapshot lifecycle
manager, since deleting those would fight a system that's already managing
retention on its own schedule.

The failure mode: the naive way to detect "is this managed by a
lifecycle tool" is to check for specific custom tags that tool's policies
are configured to set. The problem: lifecycle managers often also stamp
a fixed, undocumented prefix on every resource they touch, regardless of
what custom tags an individual policy configures — and if your detection
logic only checks the policy-specific custom tags (built by enumerating
currently active policies), you get a silent false negative the moment:

a policy is deleted or rotated, but its old snapshots remain, or
a policy's custom tag configuration doesn't match what your detection script expects.

We found this the hard way: cross-referencing a "snapshots managed by X"
report against a known-managed sample turned up a false negative on every
snapshot we checked — each one came back "clear to delete" when it
shouldn't have. The fix: stop trying to enumerate policies and
instead just check for the lifecycle tool's own fixed, unconditional tag
prefix directly — something like aws:dlm:* for AWS Data Lifecycle
Manager — as a filter, independent of any specific policy's configuration.
That prefix gets set on every snapshot the tool manages, full stop, with no
way to configure it off. (The same idea applies if AWS Backup is in the mix
too — check for its own unconditional tag prefix rather than its
policy-specific tags.)

The lesson generalizes: when you're trying to detect "is this resource
managed by some other system," prefer checking for that system's own
unconditional markers over reverse-engineering its current configuration.
Configuration drifts and gets deleted; the tool's own fingerprint on the
resource doesn't.

Pitfall #3: the fan-out worker has a concurrency ceiling, and it's not documented up front

The failure mode: even once the copy-before-delete ordering is fixed,
there's a second limit that only shows up at scale — AWS caps the number
of in-flight snapshot copies per destination region, per account, at
20. Fire off more than that concurrently and every excess request fails
immediately with a ResourceLimitExceeded error — no queueing, no
automatic backoff, just a hard rejection.

Worth being precise about this one, since it's tempting to look for a
quota-console fix: AWS's own Service Quotas listing marks "Concurrent
snapshot copies per destination Region" as not adjustable. This isn't
like a soft default you can raise with a self-service request — AWS
raised the account-wide default itself from 5 to 20 in 2020, and some
accounts have reportedly gotten a further increase through a support
ticket, but neither path is something your pipeline can assume or trigger
on its own. Batching and backoff aren't a workaround for a ceiling you
just haven't asked AWS to raise yet — they're required regardless of what
ceiling you end up with.

This is easy to miss in testing because small batches never get close to
the cap. It shows up the first time your fan-out worker reaches a large
account: the first ~20 copies succeed, and everything after that starts
failing in a burst — a reminder that "one independent unit of work per
account" doesn't mean "no cross-account-shaped limits at all." The limit
is per account/region, but it still governs how aggressively any single
account's unit of work can run.

The fix is external batching, since the copy engine itself typically
exposes no concurrency knob:

Split the eligible set into batches comfortably under the cap (e.g. 15, not 20 — leave headroom for anything else in the account that might also be copying snapshots concurrently, like an unrelated backup job).
Sleep between batches long enough for in-flight copies to actually drain before the next batch fires — a copy that's still pending still counts against the limit.
On ResourceLimitExceeded, back off and retry the same batch rather than aborting the whole run. Because the next attempt re-queries live eligible state instead of working off a stale list, this retry is safe: anything already copied-and-deleted in a partially-successful batch simply won't reappear as "still eligible."
Expect large accounts to need much more conservative batch/sleep settings than small ones. A batch size and sleep interval that clears a 100-snapshot account instantly can still throttle hard against an 8,000-snapshot one — tune per account, not globally.
Once you're safely under the concurrency cap, the sleep interval often stops being the bottleneck. Each batch's completion time is bounded by its slowest copy, and a same-region copy that also re-encrypts the data can take many minutes regardless of how short you make the sleep. At that point the only lever left for total wall-clock time is running independent scopes — separate regions, separate accounts — concurrently, since each has its own concurrency ceiling and doesn't compete with the others.

This limit compounds with pitfall #1 above: a batch that partially fails with
ResourceLimitExceeded mid-copy is exactly the kind of situation where
"copy, then immediately delete" silently produces originals with no
completed backup. Handling the concurrency ceiling and verifying copy
completion are two separate fixes, and you need both.

Putting pitfalls #1–#3 together, the shape of the fix looks roughly like
this (pseudocode — the real thing is provider-SDK-specific, but the
control flow is what matters):

def process_account(account, batch_size=15, sleep_between=90):
    # Pitfall #2: filter out anything carrying another lifecycle tool's
    # own unconditional tag prefix (e.g. "aws:dlm:*") before it ever
    # reaches the eligible set below -- not just this policy's custom tags.
    eligible_snapshot_ids = get_eligible_snapshots(account, exclude_prefixes=["aws:dlm:"])

    for batch in chunk(eligible_snapshot_ids, batch_size):
        copy_ids = {}
        for snapshot_id in batch:
            copy_ids[snapshot_id] = start_copy_with_retry(
                snapshot_id, max_retries=3, backoff_seconds=300,
                retry_on="ResourceLimitExceeded",
            )

        states = wait_for_completion(copy_ids.values(), timeout_seconds=900)

        ready_to_delete = [
            src for src, copy_id in copy_ids.items()
            if states[copy_id] == "completed"
        ]
        not_ready = [
            (src, copy_id, states[copy_id]) for src, copy_id in copy_ids.items()
            if states[copy_id] != "completed"
        ]

        delete_originals(ready_to_delete)
        for src, copy_id, state in not_ready:
            log(f"{src}: copy {copy_id} ended in '{state}', leaving original in "
                f"place -- will retry next run")

        sleep(sleep_between)  # let in-flight copies elsewhere drain before the next batch

Three properties worth calling out: eligibility is filtered by fingerprint,
not policy config (pitfall #2), retry targets a single batch, not the
whole account (pitfall #3), and "not ready" never becomes "delete
anyway" (pitfall #1) — it becomes a log line and a no-op, deferring to the
next run's fresh eligibility check instead of forcing a decision on stale
information.

Pitfall #4: "skip AMI-linked snapshots" has a state your filter probably isn't checking

The delete stage should obviously never touch a snapshot that's still
backing an AMI — deleting it would silently break anyone's ability to
launch from that image. Custodian ships exactly this as a built-in filter
on aws.ebs-snapshot — type: unused excludes snapshots still
referenced by an AMI — and it's tempting to treat that as a solved
problem once it's in your policy.

The failure mode: AMIs aren't just "exists" or "doesn't exist" — they
also have a disabled state: still registered, still referencing its
snapshots, but not launchable. This isn't a hypothetical gap either — it's
a filed bug report against Custodian's own unused filter: it doesn't
catch snapshots that are only referenced by a disabled AMI, as opposed to
an active one. A filter that only checks for active/available AMIs will
happily let a snapshot through if the only AMI referencing it has been
disabled rather than fully deregistered. The snapshot matches every filter
in your policy, gets copied, and then the actual DeleteSnapshot call
fails at the AWS API level:

An error occurred (InvalidSnapshot.InUse) when calling the DeleteSnapshot
operation: The snapshot snap-xxxxxxxxxxxxxxxxx is currently in use by
ami-xxxxxxxxxxxxxxxxx

The unhelpful part: this exact message fires whether the blocking AMI is
active or disabled. AWS doesn't distinguish the two in the error text, so
there's no signal here that tells you "check disabled images too" — you
have to already know that before you go looking, which is precisely the
gap a filter that only checks active AMIs shares with a human reading
this error for the first time.

The compounding problem is what happens next if you don't catch this:
the delete fails, but the copy that ran right before it already succeeded.
On the next scheduled run, the probe step re-queries live eligibility,
finds the same original still sitting there (never deleted), and processes
it again — a fresh copy, another failed delete. Run that on a schedule for
days without noticing, and you get a steadily growing pile of duplicate
orphaned copies for the exact same handful of snapshots, none of which were
ever going to delete successfully in the first place. It's a quiet, compounding
waste: each occurrence looks like normal batch activity in the logs (a copy
started, a delete attempted) unless you're specifically watching for the
same source ID recurring across runs.

The fix has two parts:

Treat "in use by an AMI" as a state you verify independently of whatever built-in filter your policy engine provides, since a single boolean option may not cover every AMI state the underlying API cares about. Checking directly for any AMI (active or disabled) that references the snapshot as a block device mapping is the more defensive version.
Add a check for recurring delete failures on the same source ID across runs, not just a check for failures in the current run. A snapshot that fails to delete once might be transient; a snapshot that fails to delete on every run for a week is a signal your filter has a gap, and it should stop being re-copied until that gap is understood.

Concretely, that second point needs somewhere to keep count: log each
failed delete as a structured line (source ID, reason, run ID) and put a
metric filter on it — a CloudWatch Logs metric filter incrementing a
per-source-ID counter works, or just a failure_count column in whatever
state store your process_account loop already reads from. Either way,
the check before copying is simple: if a source ID's count crosses a
small threshold (three consecutive runs is enough to rule out a one-off
throttle), skip it and alert instead of copying it again.

The lesson generalizes from pitfall #2: when a cloud resource has a
lifecycle with more than two states, don't assume a filter option that
mentions the resource type has actually covered all of them. Read the
error message the API gives you when something slips through — it's often
more precise about the real constraint than the filter's documentation is.

A few smaller gotchas

Tag collisions between the three stages

If your "copy" step preserves the source snapshot's tags on the copy (a
reasonable default — you want the copy identifiable), watch out for reusing
the same tag key across the mark → copy+delete → age-off stages. A copy
of a snapshot that was already marked-for-deletion inherits that mark —
including its already-past due date. If the age-off stage checks the same
tag key, it can fire immediately instead of after the intended retention
window, because the inherited mark is already expired.

The fix is boring but effective: give each stage its own, never-reused tag
key, even if the semantics feel similar. It costs nothing and eliminates an
entire class of "why did this get deleted early" surprises.

Marked-for-op state outlives the run that created it

A fan-out worker scoped "by account" assumes each account's unit of work is
self-contained per run. But if the delete stage's filter is simply "anything
already tagged marked-for-op, past its date," it doesn't check which run
wrote that tag — only that the tag exists.

This showed up when an account got carved out of one rollout round and
rescheduled into a later one, because its copy+delete step kept hitting the
concurrency ceiling from pitfall #3 before the batching fix existed. By the
time it came back around in the later round, that round's mark stage
matched zero new snapshots there — everything eligible had already been
marked earlier. Running the later round's delete stage against that account
picked up the entire earlier backlog anyway, since the tag has no concept
of "which round wrote me."

Neither stage was wrong — mark and delete each did exactly what their
filters said. The mismatch is between two assumptions that feel like they
should line up but don't: "this account is in scope for this round" is an
account-level idea, while "only this round's marks should act" is a
tag-level idea the tag itself has no way to express.

The fix here is a checklist item, not a code change: before pointing a
delete stage at an account, ask what marked-for-op backlog could already be
sitting there from an earlier, unrelated pass, and decide on purpose
whether picking it up is correct — often it is, since it's still real work
that needs finishing. If it isn't, isolate that account into its own run
instead of folding it into a batch scoped for something else.

Copies aren't the only place to put the safety net

Not every retention window needs a live, same-region copy sitting around
for the full period. If the safety window is long and access is expected
to be rare, the EBS Snapshot Archive tier
is worth considering as an alternative to Stage 3's copy — it's built for
exactly this "keep it, but don't expect to touch it" case and can cut
storage cost significantly versus a standard live copy.

Takeaways

Know which model you're actually running. A multi-stage tag-driven workflow (mark → act → age-off) and a fan-out worker across independent units of work (accounts, batches) are both easy to call "the pipeline" — but they fail differently, and conflating them hides where a given pitfall actually lives.
"Sequential actions" in a policy engine is not the same as "this action waits for that one to finish." Verify intermediate state — like a copy actually reaching completed — before acting on it irreversibly.
Detect by fingerprint, not by configuration. When checking whether a resource is managed by another system, prefer that system's fixed, always-on markers over trying to reconstruct its current policy config.
Batch around concurrency limits, and make batches idempotent. Re-query the live eligible set before each batch rather than working off a stale precomputed list, so retries are safe and already-processed resources don't reappear.
Tag-driven state doesn't know which run wrote it. Account-level scoping and tag-level filtering are two different notions of "in scope." Before pointing a later run at an account, check what backlog an earlier, unrelated run may have already left tagged there.
A built-in filter option isn't proof a resource's whole lifecycle is covered. Custodian's unused filter sounds complete but only checks active AMIs; a disabled one still blocks the delete — this is a filed issue against the filter itself, not a hypothetical edge case. Watch for the same source ID failing repeatedly across runs — that's the filter telling you it has a gap, not just a transient error to retry.

None of this required exotic tooling — just a healthy suspicion of "looks
safe on paper" and a willingness to verify state with the cloud provider's
own APIs before taking an irreversible action.

Killing Cloud Waste: Automating FinOps, Anomaly Detection, and Budget Enforcement

Varun Sharma — Wed, 02 Sep 2026 22:26:05 +0000

Cloud infrastructure makes over-provisioning effortless. A single configuration change can deploy an unneeded high-memory cluster, and without guardrails, waste accumulates quickly: unattached storage volumes, oversized worker nodes, continuous non-production environments, and stale test resources. Periodic spreadsheet audits cannot keep up with high-velocity engineering teams where provisioning outpaces cost governance. Across large enterprise multi-cloud footprints in AWS and Azure, the solution is embedding FinOps directly into continuous integration and cloud governance pipelines rather than auditing spend retroactively.

1. Enforce tagging at deployment, not after the fact

You cannot optimize or allocate spend that lacks clear attribution. Every cloud asset requires consistent metadata designating its owning team, environment, and cost center — enforced strictly at provisioning time rather than reconciled after the billing cycle closes.

Using AWS Service Control Policies (SCPs) or Azure Policy, platform teams can prevent resource creation whenever mandatory tags are missing, regardless of local role privileges.

Example: AWS Service Control Policy requiring a CostCenter tag

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnforceRequiredCostCenterTag",
      "Effect": "Deny",
      "Action": [
        "ec2:RunInstances"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:volume/*"
      ],
      "Condition": {
        "Null": {
          "aws:RequestTag/CostCenter": "true"
        }
      }
    }
  ]
}

Note on scoping: Notice the scoping in the Resource block. Enforcing launch-time tags on "*" is a classic AWS trap. The RunInstances API creates multiple resources under the hood, and enforcing tags globally will break deployments because sub-resources like Elastic Network Interfaces (ENIs) often cannot be tagged in the initial request. Scoping explicitly to instances and volumes ensures the policy works as intended.

2. Hard budget limits for non-prod

Budget alerts frequently turn into inbox noise. Hard automated boundaries do not. In development, sandbox, and staging accounts, enforce spending caps programmatically instead of relying on email notifications.

When a sandbox environment breaches its monthly financial threshold, a budget notification triggers an event-driven automation (such as an AWS Lambda function) to attach a restrictive IAM permissions boundary across developer and CI/CD provisioning roles. This halts new infrastructure provisioning until existing waste is cleaned up or the billing cycle resets.

Example: IAM permissions boundary applied upon budget breach

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictProvisioningOnBudgetExceeded",
      "Effect": "Deny",
      "Action": [
        "ec2:RunInstances",
        "rds:CreateDBInstance",
        "eks:CreateCluster",
        "redshift:CreateCluster"
      ],
      "Resource": "*"
    }
  ]
}

This ensures teams actively decommission unused infrastructure rather than letting orphaned resources run indefinitely.

3. Shift-left cost guardrails in CI/CD

Catching provisioning anomalies after resources hit the cloud still incurs unnecessary cost and cleanup overhead. Shift-left FinOps moves cost visibility directly into the pull request workflow.

By running static infrastructure-as-code analysis (such as Infracost) during continuous integration, teams can automatically post monthly expenditure diffs onto pull requests. Pipelines can automatically fail if an infrastructure change introduces a delta exceeding defined budgets without platform team sign-off.

Example: GitHub Actions pull request cost check

name: FinOps CI Guardrail
on: [pull_request]
jobs:
  cost-estimate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: infracost/actions/setup@v3
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}
      - name: Calculate Terraform Cost Delta
        run: |
          infracost diff --path=terraform/ \
                         --format=json \
                         --out-file=infracost.json      
      - name: Enforce Spending Threshold
        run: |
          diff=$(jq '.diffTotalMonthlyCost | tonumber' infracost.json)
          if (( $(echo "$diff > 500" | bc -l) )); then
            echo "Cost increase (\$${diff}/mo) exceeds policy limit (\$500/mo). Architecture approval required."
            exit 1
          fi

Pipeline lessons learned: Note two crucial fixes here:

The cost comparison needs infracost diff (not breakdown, which only produces a point-in-time snapshot with no diffTotalMonthlyCost field to parse).

The failure message needs \$ to escape the dollar sign correctly in bash — $$ expands to the shell's process ID, not a literal $.

4. Automated off-hours scheduling for non-prod

Non-production environments run 168 hours a week, but development teams typically only use them for 40 to 50 hours. Leaving sandbox and staging compute active 24/7 wastes over 60% of its monthly cost.

Automate downtime using EventBridge Scheduler rules invoking a targeted Lambda function or Systems Manager Automation runbook:

Schedules: Trigger a stop payload at 7:00 PM local time Monday through Friday, and start compute back up at 7:00 AM. Keep resources stopped across weekends.
Target filters: Match on Environment == "non-prod" and ScheduleOptOut != "true", allowing long-running load tests to opt out temporarily with an expiration tag.
Resource scope: Target Auto Scaling Groups (scale min/desired capacity to 0), standalone RDS instances, and Karpenter/EKS nodepools.

5. Automated Cost Anomaly Alerting

Catching a cost misconfiguration on a monthly invoice is too late, but waiting for daily Cost and Usage Report (CUR) batches can still allow waste to accumulate.

A responsive setup utilizes native tools like AWS Cost Anomaly Detection. When an anomaly is detected, it triggers an EventBridge rule that invokes a Lambda function. Instead of dispatching vague threshold alerts, the function formats a concise Slack or Teams notification identifying the specific account or service spiking in spend, projecting the impact, and providing direct links to internal remediation runbooks.

6. Automated cleanup of orphaned resources

Unattached EBS volumes, unassociated Elastic IPs, and stale snapshots accumulate quietly over time. Rather than maintaining custom cleanup scripts, declarative engines like Cloud Custodian manage this through version-controlled rules.

A reliable safety pattern ensures no storage volume is deleted without a compliant snapshot taken first, followed by a mandatory grace period.

Example: Cloud Custodian policy for orphaned EBS volumes

policies:
  - name: ebs-orphaned-volume-backup
    resource: ebs
    description: Finds unattached EBS volumes, generates a backup snapshot, and marks for deletion after a grace period.
    filters:
      - State: available
      - "tag:custodian_cleanup": absent
    actions:
      - type: snapshot
        copy-tags:
          - CostCenter
          - Environment
      - type: tag
        key: custodian_cleanup
        value: "backed-up"
      - type: mark-for-op
        op: delete
        days: 7
        tag: custodian_cleanup_delete

  - name: ebs-orphaned-volume-delete
    resource: ebs
    description: Deletes unattached volumes that carry a verified backup tag and have completed the 7-day grace period.
    filters:
      - State: available
      - "tag:custodian_cleanup": "backed-up"
      - type: marked-for-op
        op: delete
        tag: custodian_cleanup_delete
    actions:
      - type: delete

The first policy acts as an immutable safety check: nothing enters the deletion queue without an associated backup tag. The second policy only evaluates assets that have survived the grace period with that tag present. Run these via periodic cron jobs, or deploy Custodian in Lambda mode to execute automatically on CloudWatch/EventBridge schedules.

Why this matters organizationally

Cost management is usually treated as an accounting problem. Once tagging, hard budget enforcement, pipeline checks, and automated hygiene are codified into the deployment platform, FinOps becomes an engineering standard. Spend visibility integrates directly into day-to-day developer workflows, preventing runaway cloud bills without manual intervention.

Join the conversation

How does your team handle cloud waste and runaway non-production costs? Drop a comment below — let's discuss whether you rely more on strict preventative guardrails (like SCPs) or automated remediation scripts.