DEV Community: Dinesh_gowtham

How to Query Vector Embeddings with Athena for Real‑Time RAG, Explained Simply

Dinesh_gowtham — Mon, 07 Sep 2026 16:36:02 +0000

Most engineers think you need a dedicated vector database for similarity search, but Athena’s Iceberg support lets you run nearest‑neighbor queries directly on S3 data. In a few lines of TypeScript you can turn a Parquet file of embeddings into a searchable knowledge base and feed the results to an LLM. The result is a serverless RAG pipeline that costs pennies and scales automatically.

Why Athena Makes Sense for Vector Search

When you hear “vector search” you usually picture a separate service that holds millions of high‑dimensional points and answers “which points are closest to this one?” Athena can do the same thing without provisioning another cluster.

Serverless – you write SQL, Athena runs it on demand, and you pay only for the bytes it scans.
Zero‑ops – the data lives in S3, so you don’t have to manage backups, scaling, or networking.
Native integration – the same IAM role that lets your code read S3 can also start Athena queries, keeping the permission surface tiny.

Think of Athena as a giant, on‑demand librarian. The “books” are your Parquet files, and the “catalog” is a Glue table. When you ask for “the five books most similar to this paragraph,” the librarian looks through the indexed pages (the columns) and hands you the results, charging you only for the pages she actually opened.

In plain English: Athena gives you a cheap, managed way to ask “what’s closest to this vector?” as long as you store the vectors in a format it can read (Parquet) and describe them in a Glue table.

Minimal code: create an Athena client

// Import the v3 AWS SDK client for Athena
import { AthenaClient, StartQueryExecutionCommand } from "@aws-sdk/client-athena";

// The client automatically picks up credentials from the environment (e.g. IAM role)
const athena = new AthenaClient({ region: "us-east-1" });

Preparing an Iceberg Table with Embedding Parquet Files

What is Iceberg?

Iceberg is an open‑source table format that sits on top of raw files (like Parquet) and adds schema evolution, partition pruning, and metadata caching. For Athena, Iceberg means you can ask for “only the rows where the date partition is today” without scanning the whole bucket.

What is Parquet?

Parquet is a column‑oriented file format. Each column is stored separately, which lets Athena read only the columns it needs—perfect for vector search where you only need the numeric embedding column and the text chunk column.

Step 1 – Write embeddings to a Parquet file

Below is a tiny helper that converts a list of 1536‑dimensional embeddings (produced by OpenAI’s text-embedding-3-large) into a Parquet file using the parquetjs-lite library. The code is deliberately short; in production you would stream large batches instead of loading everything into memory.

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { ParquetWriter, ParquetSchema } from "parquetjs-lite";

// Define the shape of each row we will store
const schema = new ParquetSchema({
  chunk_id:   { type: "INT64" },                 // unique identifier for the text piece
  text:       { type: "UTF8" },                  // the raw text chunk
  // Store the vector as a list of doubles – Parquet calls this "FLOAT64"
  embedding:  { type: "FLOAT64", repeated: true } // 1536 numbers
});

/**
 * Upload a batch of embeddings to S3 as a Parquet file.
 * @param bucket   The S3 bucket name.
 * @param key      The object key (path) where the file will live.
 * @param rows     An array of { chunk_id, text, embedding } objects.
 */
async function uploadEmbeddingsAsParquet(
  bucket: string,
  key: string,
  rows: { chunk_id: number; text: string; embedding: number[] }[]
) {
  // 1️⃣ Write rows to an in‑memory buffer
  const writer = await ParquetWriter.openFile(schema, "/tmp/embeddings.parquet");
  for (const row of rows) {
    await writer.appendRow(row);
  }
  await writer.close();

  // 2️⃣ Push the file to S3
  const s3 = new S3Client({ region: "us-east-1" });
  const fileBody = await import("fs").then(fs => fs.promises.readFile("/tmp/embeddings.parquet"));
  await s3.send(
    new PutObjectCommand({
      Bucket: bucket,
      Key: key,
      Body: fileBody,
      // Optional: set a storage class that matches your cost profile
      StorageClass: "STANDARD_IA"
    })
  );
}

Step 2 – Create an Iceberg table that points at the Parquet file

Athena’s DDL (Data Definition Language) is just SQL. The table we create tells Athena “the data lives in this S3 prefix, the columns are defined by this schema, and treat it as an Iceberg table so we can partition later.”

import { StartQueryExecutionCommandInput } from "@aws-sdk/client-athena";

// Replace these placeholders with your own values
const DATABASE = "rag_db";
const TABLE    = "embeddings";
const S3_LOCATION = "s3://my-knowledge-base/embeddings/";

// The DDL string – notice the `EXTERNAL` keyword (data stays in S3)
const createTableSQL = `
CREATE EXTERNAL TABLE ${DATABASE}.${TABLE}
WITH (
  format = 'ICEBERG',
  location = '${S3_LOCATION}'
) AS SELECT
  chunk_id,
  text,
  embedding
FROM ${DATABASE}.staging_embeddings; -- a temporary staging table you load first
`;

// Athena expects a StartQueryExecutionCommandInput object
const startCreateCmd: StartQueryExecutionCommandInput = {
  QueryString: createTableSQL,
  ResultConfiguration: {
    OutputLocation: "s3://my-knowledge-base/athena-results/"
  }
} satisfies StartQueryExecutionCommandInput; // <-- type‑check at compile time

// Fire the query (it runs asynchronously)
await athena.send(new StartQueryExecutionCommand(startCreateCmd));

Tip: Athena charges per TB scanned. By using Iceberg and partitioning on a column like date or source, the query can skip whole partitions, keeping cost low.

SQL‑Based Cosine Similarity and Nearest‑Neighbor Query

What is cosine similarity?

Cosine similarity measures the angle between two vectors. If you picture each embedding as an arrow pointing somewhere in a high‑dimensional space, the cosine tells you how closely the arrows line up, regardless of their length. The formula is

[
\text{cosine}(a, b) = \frac{a \cdot b}{|a| |b|}
]

where (a \cdot b) is the dot‑product (sum of element‑wise products) and (|a|) is the vector’s length (Euclidean norm).

Why normalize? If you forget to divide by the lengths, Athena has to compute the raw dot‑product for every row, but the result will be biased by vector magnitude. Moreover, without pre‑normalizing you lose the ability to prune with a simple “WHERE similarity > threshold” because the raw numbers are not comparable.

Implementing cosine similarity in Athena SQL

Athena does not ship a built‑in cosine_similarity function, but you can express it with a few WITH clauses. The trick is to store a pre‑computed length column when you first load the embeddings. If you didn’t, you can compute it on the fly, but that forces a full table scan.

Assume we have a column norm that holds the pre‑computed length of each embedding.

WITH query_vec AS (
  -- The vector you are searching for – 1536 numbers.
  SELECT
    CAST(array[0.01, 0.23, … , 0.45] AS ARRAY<double>) AS q_vec,
    -- Pre‑compute its length once
    sqrt(REDUCE(q_vec, 0D, (s, x) -> s + x * x)) AS q_norm
),
scores AS (
  SELECT
    e.chunk_id,
    e.text,
    -- Compute dot‑product between stored embedding and query vector
    REDUCE(
      TRANSFORM(zip(e.embedding, q_vec.q_vec), x -> x[0] * x[1]),
      0D,
      (s, prod) -> s + prod
    ) AS dot,
    e.norm AS e_norm,
    q_norm.q_norm AS q_norm
  FROM ${DATABASE}.${TABLE} AS e
  CROSS JOIN query_vec AS q_norm
)
SELECT
  chunk_id,
  text,
  dot / (e_norm * q_norm) AS cosine_similarity
FROM scores
ORDER BY cosine_similarity DESC
LIMIT 5;

The REDUCE + TRANSFORM pattern iterates over the 1536 numbers without pulling the whole array into a single scalar, keeping the query readable.

Type‑checked Athena command in TypeScript

import {
  StartQueryExecutionCommand,
  StartQueryExecutionCommandInput
} from "@aws-sdk/client-athena";

// The query we just built (you would probably generate the array dynamically)
const query = `WITH ... (the SQL from above) ...`;

// Athena needs to know where to dump the CSV result file
const execInput: StartQueryExecutionCommandInput = {
  QueryString: query,
  ResultConfiguration: {
    OutputLocation: "s3://my-knowledge-base/athena-results/"
  }
} satisfies StartQueryExecutionCommandInput; // compile‑time safety

const execResponse = await athena.send(new StartQueryExecutionCommand(execInput));
const queryExecutionId = execResponse.QueryExecutionId!;

Key takeaway: By normalizing vectors ahead of time and expressing cosine similarity with plain SQL functions, you avoid a full scan and keep Athena’s cost model happy.

Wiring the Athena Query into a Node.js RAG Service

What is RAG?

Retrieval‑Augmented Generation (RAG) is a pattern where you first retrieve relevant documents (or text chunks) and then generate a response using a language model, feeding the retrieved text as part of the prompt. It improves factuality and reduces hallucinations.

End‑to‑end flow

Receive a user question → compute its embedding via OpenAI (or any provider).
Run the Athena nearest‑neighbor query → get the top 5 most similar chunks.
Concatenate the chunks into a prompt.
Call an LLM (e.g., Claude on Bedrock) → return the answer.

Below is a compact but complete TypeScript snippet that glues those steps together. It uses the @aws-sdk/client-bedrock-runtime package only for illustration; you can swap it for any HTTP client that talks to your LLM.

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import {
  GetQueryResultsCommand,
  GetQueryResultsCommandInput
} from "@aws-sdk/client-athena";
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";

// Helper: wait for Athena query to finish (polling)
async function waitForResult(queryId: string): Promise<string> {
  const poll = async (attempt = 0): Promise<string> => {
    const resp = await athena.send(
      new GetQueryResultsCommand({ QueryExecutionId: queryId })
    );
    if (resp.ResultSet?.Rows?.length) return queryId;
    if (attempt > 30) throw new Error("Athena query timed out");
    await new Promise(r => setTimeout(r, 1000 * Math.pow(2, attempt))); // exponential back‑off
    return poll(attempt + 1);
  };
  return poll();
}

// 1️⃣ Compute query embedding (placeholder – you would call OpenAI’s API)
async function getQueryEmbedding(question: string): Promise<number[]> {
  // In a real app, replace this with a fetch to https://api.openai.com/v1/embeddings
  return Array(1536).fill(0).map(() => Math.random()); // dummy vector
}

// 2️⃣ Build the Athena SQL with the embedding we just got
function buildSimilaritySQL(embedding: number[]): string {
  const vecLiteral = `array[${embedding.map(v => v.toFixed(8)).join(", ")}]`;
  return `
    WITH query_vec AS (
      SELECT CAST(${vecLiteral} AS ARRAY<double>) AS q_vec,
             sqrt(REDUCE(CAST(${vecLiteral} AS ARRAY<double>), 0D, (s, x) -> s + x * x)) AS q_norm
    ),
    scores AS (
      SELECT
        e.chunk_id,
        e.text,
        REDUCE(
          TRANSFORM(zip(e.embedding, q_vec.q_vec), x -> x[0] * x[1]),
          0D,
          (s, prod) -> s + prod
        ) AS dot,
        e.norm,
        q_vec.q_norm
      FROM rag_db.embeddings e
      CROSS JOIN query_vec q_vec
    )
    SELECT chunk_id, text, dot / (norm * q_norm) AS similarity
    FROM scores
    ORDER BY similarity DESC
    LIMIT 5;
  `;
}

// 3️⃣ Run the query and fetch the rows
async function fetchTopChunks(question: string): Promise<string[]> {
  const embedding = await getQueryEmbedding(question);
  const sql = buildSimilaritySQL(embedding);

  // Start execution
  const startResp = await athena.send(
    new StartQueryExecutionCommand({
      QueryString: sql,
      ResultConfiguration: { OutputLocation: "s3://my-knowledge-base/athena-results/" }
    })
  );
  const execId = startResp.QueryExecutionId!;
  await waitForResult(execId);

  // Pull results (CSV rows)
  const resultResp = await athena.send(
    new GetQueryResultsCommand({ QueryExecutionId: execId })
  );

  // Rows[0] is the header; slice it off
  const rows = resultResp.ResultSet?.Rows?.slice(1) ?? [];
  return rows.map(r => r.Data?.[1].VarCharValue ?? ""); // column 1 is `text`
}

// 4️⃣ Send the concatenated chunks to Claude (Bedrock)
async function answerWithClaude(question: string): Promise<string> {
  const chunks = await fetchTopChunks(question);
  const prompt = `
You are a helpful assistant. Use the following retrieved passages to answer the user's question.

Passages:
${chunks.map((c, i) => `${i + 1}. ${c}`).join("\n")}

Question: ${question}
Answer:
`;

  const bedrock = new BedrockRuntimeClient({ region: "us-east-1" });
  const invokeCmd = new InvokeModelCommand({
    ModelId: "anthropic.claude-v2",
    ContentType: "application/json",
    Accept: "application/json",
    Body: JSON.stringify({ prompt, max_tokens_to_sample: 500 })
  });

  const resp = await bedrock.send(invokeCmd);
  const payload = JSON.parse(Buffer.from(resp.Body as Uint8Array).toString("utf-8"));
  return payload.completion;
}

// Example usage
(async () => {
  const userQuestion = "How does Athena charge for queries?";
  const answer = await answerWithClaude(userQuestion);
  console.log("🧠 Answer:", answer);
})();

Helpful tip: Athena writes query results to the S3 prefix you specify. Schedule a nightly Lambda (or a simple aws s3 rm command) to delete old CSV files, otherwise storage costs will creep up.

Performance Gotchas and How to Avoid Them

Gotcha	Why it hurts	Simple fix
No pre‑normalized vectors	Athena must compute `norm` for every row, which forces a full scan and adds CPU time → higher latency and cost.	Store an extra column `norm` when you first load the embeddings (`norm = sqrt(sum(x_i^2))`).
Unpartitioned Iceberg table	Without partitions Athena reads all Parquet files even if you only need recent data.	Partition on a low‑cardinality column like `source` or `ingest_date`. Iceberg automatically skips irrelevant files.
Result location left untouched	Each query creates a CSV in the `OutputLocation` bucket; these accumulate forever.	After you finish processing, call `s3.deleteObject` on the result key, or enable Athena’s `ResultReuseConfiguration`.
Glue catalog lag	After you run `CREATE TABLE …` Athena may still see the old schema for a few minutes, causing “column not found” errors.	Wait ~30 seconds or run a dummy `SHOW TABLES` before the real query; this forces Glue to refresh.
Large scans exceed the 10 GB limit	Athena aborts queries that try to read more than 10 GB in a single execution.	Keep each Parquet file under a few hundred MB and use Iceberg’s `metadata.refresh_interval` to keep the manifest size small.
S3 eventual consistency on list	If you upload a new Parquet file and immediately run a query, the file may not appear yet, leading to missing results.	Use `s3.waiter('objectExists')` or add a small delay (e.g., 2 seconds) after the upload before starting the query.

In plain English: The biggest performance win comes from pre‑computing what you can (vector lengths) and letting Iceberg do the heavy lifting (partition pruning). Clean up query results and give Glue a moment to catch up, and you’ll stay in the “pennies” cost zone.

The Takeaway

Athena can act as a serverless vector store when you keep embeddings in Parquet and describe them with an Iceberg table.
Cosine similarity is expressible in plain SQL; just remember to store the vector length (norm) ahead of time.
A few TypeScript lines—upload, table creation, query execution, and result handling—turn a static S3 bucket into a live RAG knowledge base.
Cost stays low because Iceberg lets Athena skip whole partitions, and you only pay for the bytes actually read.
Watch out for un‑normalized vectors, missing partitions, lingering result files, and Glue‑catalog refresh delays; each can blow up latency and price.
The whole pipeline stays zero‑ops: no managed vector DB, no servers to patch, and scaling is handled automatically by Athena and S3.

Give it a try on a small dataset, watch the query cost in the Athena console, and then scale up by adding partitions. You’ll see that a fully managed vector database isn’t the only path to real‑time RAG—Athena can get you there with just a handful of lines of TypeScript. Happy querying!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-07 · Primary focus: Athena

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

Real-Time AI Agent Event Stream with Kinesis: Build a Low-Latency LLM Chat Pipeline

Dinesh_gowtham — Mon, 07 Sep 2026 07:30:15 +0000

Imagine your AI assistant responding to user messages in under 100 ms, no polling, no missed events. By streaming each utterance through Amazon Kinesis and a tiny Lambda consumer, you get a push‑based, scalable backbone for LLM calls. This pattern flips the classic request‑response model on its head.

In plain English: Instead of “client asks → server answers → client waits”, we push every user line into a fast conveyor belt (Kinesis) and let a worker pick it up instantly, reply, and push the answer back onto the belt.

Why Stream AI Agent Events with Kinesis?

When a chatbot receives a message, the traditional route is:

The front‑end makes an HTTP request.
The request hits an API Gateway → Lambda → LLM.
The response travels back the same path.

That round‑trip adds network hops, DNS lookups, and queue latency. If you replace the request‑response hop with an event stream, each utterance becomes a tiny packet that flows continuously, like cars on a highway that never stop at a toll booth.

Benefits that matter for a chat agent

Benefit	Why it helps the user	Why it helps the developer
Deterministic sub‑100 ms delivery	The user sees a reply almost instantly, keeping conversation natural.	You can reason about latency in tests because the stream guarantees ordering per shard.
Built‑in fan‑out	One message can be consumed by many workers (e.g., logging, analytics, moderation) without extra code.	No need for separate SQS queues or SNS topics.
No polling required	Workers don’t have to ask “any new data?” every second.	Reduces empty‑read cycles and saves compute credits.

Key takeaway: Kinesis turns “when will the next message arrive?” into “the next message arrives now”.

A simple analogy

Think of a mail slot (SQS) versus a conveyor belt (Kinesis). With a mail slot you must open the door, look inside, and take the letter—sometimes you open it to find nothing. With a conveyor belt, the letter slides right onto your desk the moment it’s placed on the belt, no extra checking needed.

Setting Up the Kinesis Data Stream

Before we can push chat utterances, we need a stream. A stream is a logical pipe that holds ordered records, split into shards (think of parallel lanes on a highway). Each shard can ingest up to 1 MiB/s of data (≈ 1000 records per second) and deliver up to 2 MiB/s to consumers.

Step 1: Create the stream with the AWS SDK for JavaScript v3

import { KinesisClient, CreateStreamCommand, PutRecordCommand } from "@aws-sdk/client-kinesis";

// 1️⃣ Create a Kinesis client that talks to the current region
const kinesis = new KinesisClient({});

// 2️⃣ Create a stream named "chat-input" with 2 shards (2 parallel lanes)
await kinesis.send(
  new CreateStreamCommand({
    StreamName: "chat-input",
    ShardCount: 2, // adjust based on expected traffic
  })
);
console.log("✅ Stream created");

// Helper: write a single chat utterance into the stream
export async function pushUserMessage(sessionId: string, userMessage: string) {
  // Payload we want to ship – JSON string → bytes
  const payload = Buffer.from(
    JSON.stringify({ sessionId, userMessage })
  );

  // PartitionKey decides which shard the record lands on.
  // Using sessionId spreads messages from the same conversation onto the same shard,
  // preserving order.
  await kinesis.send(
    new PutRecordCommand({
      StreamName: "chat-input",
      Data: payload,
      PartitionKey: sessionId,
    })
  );
}

What each line does

KinesisClient – a thin wrapper around HTTP calls to the Kinesis service.
CreateStreamCommand – asks AWS to allocate the pipe and its lanes.
PutRecordCommand – drops one JSON packet onto the belt.
PartitionKey – like a mailbox number; records with the same key go to the same lane, which guarantees ordering for a single conversation.

Tip: If you exceed the 1 MiB/s write limit, Kinesis will throttle writes and return a ProvisionedThroughputExceededException. Plan shard count accordingly.

Gotcha: shard limits at scale

Teams often start with a single shard, then see mysterious throttling once traffic grows. Remember: each shard is capped at 1 MiB/s write. If you expect 10 MiB/s, create at least 10 shards (or more, to leave headroom).

Lambda Consumer: Decode, Call Claude via Bedrock

Now we need a worker that pulls records, calls an LLM, and pushes the reply back onto another stream (e.g., chat-output). In Node.js the Lambda handler runs whenever new records arrive.

The iterator problem

Kinesis uses a shard iterator – a cursor that points to the next record to read. An iterator is only valid for 5 minutes. If you reuse an old iterator, GetRecords returns an empty set even though data sits in the stream. The fix is to request a fresh iterator on every Lambda invocation.

Full Lambda code (TypeScript)

import {
  KinesisClient,
  GetShardIteratorCommand,
  GetRecordsCommand,
  PutRecordCommand,
} from "@aws-sdk/client-kinesis";
import {
  BedrockClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime"; // Bedrock SDK (v3)
import { Context, Handler } from "aws-lambda";

// Clients are created outside the handler so they are reused across invocations
const kinesis = new KinesisClient({});
const bedrock = new BedrockClient({});

// Names of the streams we use
const INPUT_STREAM = "chat-input";
const OUTPUT_STREAM = "chat-output";

// Helper: fetch a fresh iterator for a given shard
async function getIterator(shardId: string) {
  const resp = await kinesis.send(
    new GetShardIteratorCommand({
      StreamName: INPUT_STREAM,
      ShardId: shardId,
      ShardIteratorType: "LATEST", // start at newest record
    })
  );
  return resp.ShardIterator!;
}

// Lambda entry point
export const handler: Handler = async (event: any, _ctx: Context) => {
  // The event we receive is a Kinesis event payload generated by the service.
  // It already contains Records, but we will also demonstrate a manual pull
  // to illustrate iterator refresh logic.

  for (const record of event.Records) {
    // 1️⃣ Decode the base64 payload that Kinesis gives us
    const payload = Buffer.from(record.kinesis.data, "base64").toString("utf-8");
    const { sessionId, userMessage } = JSON.parse(payload);
    console.log(`👂 Received message from ${sessionId}: ${userMessage}`);

    // 2️⃣ Call Claude (or any Bedrock model) via invokeModel
    const invokeResp = await bedrock.send(
      new InvokeModelCommand({
        ModelId: "anthropic.claude-v2", // example model identifier
        ContentType: "application/json",
        Accept: "application/json",
        Body: JSON.stringify({
          prompt: userMessage,
          max_tokens_to_sample: 256,
        }),
      })
    );

    // 3️⃣ Parse the model’s answer
    const modelOutput = JSON.parse(Buffer.from(invokeResp.Body!).toString("utf-8"));
    const assistantReply = modelOutput.completion?.trim() ?? "I’m sorry, I didn’t understand.";

    console.log(`🤖 Claude replied: ${assistantReply}`);

    // 4️⃣ Push the reply onto the output stream for downstream consumers
    await kinesis.send(
      new PutRecordCommand({
        StreamName: OUTPUT_STREAM,
        PartitionKey: sessionId, // keep same ordering for the conversation
        Data: Buffer.from(
          JSON.stringify({ sessionId, assistantReply })
        ),
      })
    );
  }

  // --- Manual iterator refresh example (useful if you run a while‑loop consumer) ---
  // Assume we know the shard ID (e.g., from environment variable)
  const shardId = process.env.SHARD_ID!;
  const iterator = await getIterator(shardId);
  const recordsResp = await kinesis.send(
    new GetRecordsCommand({ ShardIterator: iterator, Limit: 10 })
  );

  // If the iterator were stale, `recordsResp.Records` would be empty even though data exists.
  // Refreshing every call avoids that silent failure.
  // -------------------------------------------------------------------------------

  return { statusCode: 200 };
};

Explanation of key lines

record.kinesis.data – Kinesis delivers each record as a base64 string; we decode it.
InvokeModelCommand – Bedrock’s API to run a Large Language Model (LLM). The request body follows the model’s JSON schema.
PutRecordCommand on OUTPUT_STREAM – pushes the assistant’s reply back onto a second stream so other services (analytics, UI push, logging) can consume it without touching the Lambda again.
The iterator refresh block shows the pattern you must adopt if you ever write a long‑running consumer (e.g., a container that reads continuously).

Helpful tip: When you enable Enhanced Fan‑Out (next section), Lambda receives records via an event source mapping and you don’t need to manage iterators yourself. However, if you ever write a custom consumer, always request a new iterator on each loop.

Lambda‑specific gotchas

require(esm) in Node 22 – If you depend on the esm package, Lambda layers compiled for Node 18 will silently fail. Stick to native ESM (import …) or upgrade your layer.
SnapStart + VPC – SnapStart speeds up cold starts by serializing the execution environment, but when a Lambda sits inside a VPC the time spent attaching ENIs dominates. You won’t see savings unless the function is outside a VPC.
Response streaming – If you later need to stream the assistant’s reply directly to a client (instead of writing to a stream), set Content-Type: text/event-stream and return the body as a readable stream; otherwise Lambda buffers the whole response.
Provisioned Concurrency – Guarantees a warm container but charges per‑hour even when idle. For a low‑traffic chatbot, on‑demand scaling is usually cheaper.

Enhanced Fan‑Out for Sub‑millisecond Latency

Standard Kinesis consumption works like a single bus that all consumers share; the bus capacity (2 MiB/s per shard) is split among them. Enhanced fan‑out gives each consumer its own 2 MiB/s pipe, removing contention and cutting per‑record latency to < 1 ms on the AWS side.

How it works

When you register a consumer with RegisterStreamConsumer, Kinesis creates a dedicated shard‑level endpoint. The consumer then reads via SubscribeToShard, which pushes records to the consumer as soon as they land on the shard. No polling, no empty reads.

Registering a Lambda as an enhanced fan‑out consumer

import {
  KinesisClient,
  RegisterStreamConsumerCommand,
  DescribeStreamConsumerCommand,
} from "@aws-sdk/client-kinesis";

const kinesis = new KinesisClient({});

async function ensureConsumer() {
  const consumerName = "chat-lambda-consumer";

  // 1️⃣ Try to describe an existing consumer (idempotent)
  const describe = await kinesis.send(
    new DescribeStreamConsumerCommand({
      StreamARN: "arn:aws:kinesis:us-east-1:123456789012:stream/chat-input",
      ConsumerName: consumerName,
    })
  ).catch(() => null);

  if (describe?.ConsumerDescription?.ConsumerARN) {
    console.log("✅ Consumer already exists");
    return describe.ConsumerDescription.ConsumerARN;
  }

  // 2️⃣ Register a new consumer – this is the step that enables enhanced fan‑out
  const register = await kinesis.send(
    new RegisterStreamConsumerCommand({
      StreamARN: "arn:aws:kinesis:us-east-1:123456789012:stream/chat-input",
      ConsumerName: consumerName,
    })
  );

  console.log("🛎️ Registered enhanced fan‑out consumer");
  return register.Consumer?.ConsumerARN!;
}

// Call during deployment (CDK, SAM, or manual script)
ensureConsumer().then((arn) => console.log(`Consumer ARN: ${arn}`));

Key points

RegisterStreamConsumerCommand creates a dedicated delivery channel for the Lambda.
The consumer name must be unique per stream; re‑running the script is safe because we first try to describe it.
Once the consumer exists, you attach it to the Lambda via an event source mapping (in the console or via IaC). The mapping will automatically use the fan‑out endpoint.

In plain English: Enhanced fan‑out gives each worker its own private lane on the highway, so no traffic jam ever slows you down.

Performance impact

Standard mode: Lambda polls every ~5 seconds, incurring ~50–100 ms of added latency.
Enhanced fan‑out: Records arrive as soon as they’re written; typical end‑to‑end latency (write → Lambda start → LLM call) can be under 100 ms if the LLM itself is fast.

Remember the shard limits – each enhanced consumer still respects the 2 MiB/s read per shard, but now the limit is per consumer instead of shared. If you plan many parallel agents, allocate more shards accordingly.

End‑to‑End Flow Diagram and Deployment

Below is a textual flow that you can paste into a markdown diagram tool (e.g., Mermaid) if you like visualizing it later.

sequenceDiagram
    participant UI as Front‑end (Web / Mobile)
    participant K1 as Kinesis (chat-input)
    participant L as Lambda (consumer)
    participant B as Bedrock (Claude)
    participant K2 as Kinesis (chat-output)
    participant downstream as Downstream services

    UI->>K1: PutRecord({sessionId, userMessage})
    K1->>L: Push (Enhanced fan‑out)
    L->>B: invokeModel(userMessage)
    B->>L: Model response
    L->>K2: PutRecord({sessionId, assistantReply})
    K2->>downstream: Subscribe (analytics, UI push, etc.)

Deploying with the AWS CDK (TypeScript)

Only a few lines are needed because the heavy lifting (stream creation, consumer registration) is done by the CDK constructs.

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as kinesis from "aws-cdk-lib/aws-kinesis";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as lambdaEventSources from "aws-cdk-lib/aws-lambda-event-sources";

export class ChatPipelineStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // 1️⃣ Input stream (enhanced fan‑out enabled by default)
    const input = new kinesis.Stream(this, "ChatInput", {
      streamName: "chat-input",
      shardCount: 2, // adjust based on traffic
    });

    // 2️⃣ Output stream for replies
    const output = new kinesis.Stream(this, "ChatOutput", {
      streamName: "chat-output",
      shardCount: 1,
    });

    // 3️⃣ Lambda that consumes the input stream
    const consumerFn = new lambda.Function(this, "ChatConsumer", {
      runtime: lambda.Runtime.NODEJS_22_X,
      handler: "index.handler",
      code: lambda.Code.fromAsset("lambda"), // folder with the TypeScript compiled code
      memorySize: 256,
      timeout: cdk.Duration.seconds(10),
    });

    // 4️⃣ Attach the Lambda as an enhanced fan‑out consumer
    consumerFn.addEventSource(
      new lambdaEventSources.KinesisEventSource(input, {
        startingPosition: lambda.StartingPosition.LATEST,
        // Setting `batchSize` to 1 ensures we process each utterance immediately.
        batchSize: 1,
        // `bisectBatchOnFunctionError` = false because we want atomic processing.
        bisectBatchOnFunctionError: false,
        // Enhanced fan‑out is true by default; we make it explicit.
        maxBatchingWindow: cdk.Duration.millis(0),
        enabled: true,
      })
    );

    // 5️⃣ Grant the Lambda permissions to read from input and write to output
    input.grantRead(consumerFn);
    output.grantWrite(consumerFn);
  }
}

Why each piece matters

shardCount – directly maps to the 1 MiB/s write ceiling; more shards = more throughput.
batchSize: 1 – forces the Lambda to receive one record at a time, reducing queuing delay.
maxBatchingWindow: 0 ms – disables the small time window that would otherwise wait to fill a batch, keeping latency low.
Permissions (grantRead, grantWrite) are required for the Lambda to call GetRecords and PutRecord without manual IAM policy crafting.

Pro tip: If you ever need to add more shards after launch, use the UpdateShardCount API. Remember to adjust your downstream consumers to handle the new shard IDs.

The Takeaway

What you now have in your toolbox

Streaming chat events with Kinesis removes polling latency and guarantees order per conversation.
Enhanced fan‑out gives each Lambda its own private delivery lane, shrinking end‑to‑end latency to sub‑100 ms in most cases.
A shard iterator expires after 5 minutes; always request a fresh iterator when you write a custom consumer.
The Lambda consumer decodes the JSON payload, calls Claude (or any Bedrock model) via invokeModel, and writes the response back to a second stream.
Common gotchas: shard throughput limits, iterator expiration, Node 22 require(esm) issues, SnapStart + VPC mismatch, and provisioned concurrency cost surprises.
Deploying with CDK (or SAM) makes the whole pipeline repeatable and lets you version‑control the architecture.

Armed with this pattern you can replace a traditional request‑response chatbot with a push‑based, horizontally scalable pipeline that feels instantaneous to your users. Happy streaming!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-07 · Primary focus: Kinesis

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

How to Turn Natural Language Prompts into CDK TypeScript Stacks with Claude – A Step‑By‑Step Guide

Dinesh_gowtham — Fri, 04 Sep 2026 15:03:23 +0000

Ever wished you could describe your cloud resources in plain English and watch CDK spin them up? Claude’s function‑calling can turn a single prompt into fully‑typed TypeScript infrastructure. Let’s see why this feels like having a junior engineer who never sleeps.

Prompt Crafting for Infra

Why spend time on the prompt?

Think of a prompt as a recipe you give to a chef. If the recipe is vague, the dish will be bland or even inedible. The same is true for AI‑generated infrastructure: a well‑structured prompt tells Claude exactly what you need, how it should behave, and what constraints to respect. When you include details such as “enable versioning”, “add a 30‑day lifecycle rule”, and “use AES‑256 encryption”, Claude can emit CDK code that already respects best practices.

How to write a good prompt

State the resource type – “Create an S3 bucket”.
List required features – versioning, lifecycle, encryption.
Mention constraints – bucket name must be globally unique, keep the stack under 500 resources.
Ask for a specific output – “Give me a single TypeScript file that exports a class called BucketStack”.

Below is a minimal prompt that follows those steps:

Please generate a TypeScript CDK construct that creates an S3 bucket with:
- versioning enabled,
- a lifecycle rule that deletes objects after 30 days,
- server‑side encryption using AES‑256,
- a bucket name that includes the word "demo".
Return the code as a complete file named lib/bucket-stack.ts and make sure it compiles with `aws-cdk-lib@2.x`.

Key takeaway: A prompt is a contract. The clearer the contract, the fewer revisions you’ll need later.

Claude‑Generated TypeScript Code

Why let Claude write the code?

You could hand‑write the same construct, but Claude can produce a type‑correct file in seconds. This speeds up prototyping, especially when you’re still exploring which AWS services you need. The generated code still runs through the TypeScript compiler, so any typo or missing import will be caught immediately.

How to request the code

Claude’s function‑calling feature lets you describe a “function” the model should fill in. In our case, the function is generateCdkFile with parameters filename and content. The model returns JSON that we can write to disk.

// utils/claude.ts
import { Anthropic } from '@anthropic-ai/sdk'; // SDK for Claude
import fs from 'fs/promises';

// Define the shape of the function we ask Claude to call
type GenerateCdkFileParams = {
  filename: string; // where the code should be saved, e.g. "lib/bucket-stack.ts"
  content: string;  // the full TypeScript source code
};

// Helper that calls Claude’s function‑calling endpoint
export async function askClaudeForCdk(
  description: string
): Promise<GenerateCdkFileParams> {
  const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

  // The “messages” array is the chat history; we only need a system prompt and the user description
  const response = await client.messages.create({
    model: 'claude-3-5-sonnet-20241022', // latest model as of 2026
    max_tokens: 4000,
    temperature: 0, // deterministic output for code
    messages: [
      {
        role: 'system',
        content:
          'You are an expert AWS CDK developer. When asked, output a single TypeScript file that compiles with aws-cdk-lib@2. Use the function "generateCdkFile" to return the filename and content.',
      },
      { role: 'user', content: description },
    ],
    // Declare the function we expect back
    tools: [
      {
        type: 'function',
        function: {
          name: 'generateCdkFile',
          description: 'Return a filename and TypeScript source for a CDK construct',
          parameters: {
            type: 'object',
            properties: {
              filename: { type: 'string' },
              content: { type: 'string' },
            },
            required: ['filename', 'content'],
          },
        },
      },
    ],
  });

  // Claude may return the function call in `tool_calls[0].function`
  const toolCall = response?.tool_calls?.[0]?.function;
  if (!toolCall) throw new Error('Claude did not return a generateCdkFile call');

  const result: GenerateCdkFileParams = JSON.parse(toolCall.arguments);
  // Write the file to disk for the next step
  await fs.writeFile(result.filename, result.content);
  return result;
}

In plain English: We tell Claude what we want (a TypeScript file) and Claude hands us the exact text, which we then save.

Type‑Safe Validation with `satisfies`

Why add an extra type check?

Even though Claude aims to produce correct TypeScript, it can still make subtle mistakes—like using a property that doesn’t exist on a construct. The satisfies operator (available since TypeScript 4.9) lets us verify that the generated object conforms to an interface without changing its inferred type. This gives us a safety net before we ever run cdk synth.

How to validate the generated stack

First, define a minimal interface that captures the shape we expect:

// src/types.ts
import { StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';

// Interface describing the constructor signature of our generated stack
export interface BucketStackCtor {
  new (scope: Construct, id: string, props?: StackProps): unknown;
}

Now import the generated class (it will be in lib/bucket-stack.ts) and assert it satisfies the interface:

// src/validate.ts
import { BucketStackCtor } from './types';
import * as path from 'path';

// Dynamically import the generated file; Node's ESM loader works with .ts if ts-node is present
import('ts-node/register').then(() => {
  const generated = require(path.resolve('lib/bucket-stack.ts')) as {
    BucketStack: BucketStackCtor;
  };

  // The `satisfies` check runs at compile time; we also guard at runtime
  const isValid = (generated.BucketStack as any) as BucketStackCtor;
  // If TypeScript compilation succeeded, the shape matches; otherwise we get a type error.
  console.log('✅ Generated stack matches expected constructor signature.');
});

Tip: Keep the validation file separate from your production code. It acts like a quick “unit test” that you run after each AI generation.

Integrating into a CDK App

Why wrap the stack in an app?

A CDK App (short for application) is the entry point that tells the CDK framework which stacks to synthesize. By adding the AI‑generated stack to an existing app, you can treat it like any hand‑written stack—run cdk diff, cdk synth, or cdk deploy.

How to add the stack

Create a standard CDK entry file (bin/main.ts). This file imports the generated stack class and adds it to the app.

// bin/main.ts
import * as cdk from 'aws-cdk-lib';
import { BucketStack } from '../lib/bucket-stack';

// CDK bootstrap is required once per account/region. Without it, deployment fails.
const app = new cdk.App();

// The stack id must be unique within the app.
new BucketStack(app, 'DemoBucketStack', {
  // You can pass standard StackProps here (env, tags, etc.)
  env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
});

app.synth(); // Generates CloudFormation templates in the cdk.out directory

In plain English: The App is like a container that holds one or more stacks. Adding the AI‑generated BucketStack is no different than adding a manually written one.

Known CDK gotchas (quick reminder)

Gotcha	What happens	How to avoid
Bootstrap required	Deploy fails with “Bootstrap stack not found”.	Run `cdk bootstrap aws://ACCOUNT/REGION` once per region.
Large stacks (>500 resources)	CloudFormation rejects the template.	Split logical groups into separate stacks.
Aspects fire after synth	Trying to modify a resource inside an Aspect won’t work.	Use Aspects only for read‑only inspection or add constructs before `app.synth()`.
Token resolution	Some values (like `Fn::GetAtt`) appear as unresolved tokens and can’t be used in string interpolation.	Keep token‑dependent logic inside CDK constructs, not in plain JS strings.

Deploying via CDK Pipelines

Why use a pipeline?

Running cdk deploy manually is fine for experiments, but a CDK Pipeline (part of the aws-cdk-lib/pipelines module) gives you repeatable, version‑controlled deployments. The pipeline can pull the latest AI‑generated code from your repository, synthesize the stack, and deploy it to a target account—all without a human pressing a button.

How to set up a simple pipeline

First, add the pipeline module to your package.json:

npm install aws-cdk-lib@2 aws-cdk-lib/pipelines

Now create a pipeline stack (lib/pipeline-stack.ts):

// lib/pipeline-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as pipelines from 'aws-cdk-lib/pipelines';
import { Construct } from 'constructs';
import { BucketStack } from './bucket-stack';

export class PipelineStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Source action – assumes the code lives in a GitHub repo
    const source = pipelines.CodePipelineSource.gitHub('my-org/my-repo', 'main', {
      authentication: cdk.SecretValue.secretsManager('github-token'), // store token in Secrets Manager
    });

    // Synthesize step – runs `npm ci && npx cdk synth`
    const synth = new pipelines.ShellStep('Synth', {
      input: source,
      commands: [
        'npm ci',
        // Optional: run validation script that checks `satisfies`
        'node src/validate.js',
        'npx cdk synth',
      ],
    });

    // Build the pipeline
    new pipelines.CodePipeline(this, 'Pipeline', {
      pipelineName: 'DemoBucketPipeline',
      synth,
      // Add a stage that deploys the bucket stack to the "prod" environment
      // (you could add more stages for test, staging, etc.)
      deployStage: new pipelines.Stage(this, 'Prod', {
        stacks: [new BucketStack(this, 'ProdBucketStack')],
      }),
    });
  }
}

Finally, reference the pipeline stack in your bin/main.ts (you can keep the original bucket stack for local testing):

// bin/main.ts (updated)
import * as cdk from 'aws-cdk-lib';
import { BucketStack } from '../lib/bucket-stack';
import { PipelineStack } from '../lib/pipeline-stack';

const app = new cdk.App();

// Local preview stack
new BucketStack(app, 'DemoBucketStack');

// CI/CD pipeline stack (will be deployed to a dedicated pipeline account)
new PipelineStack(app, 'DemoPipelineStack', {
  env: { account: process.env.CDK_PIPELINE_ACCOUNT, region: process.env.CDK_PIPELINE_REGION },
});

app.synth();

Tip: Remember the CDK token limit of 500 resources. If the pipeline tries to synthesize a stack that exceeds this limit, the pipeline will fail early, saving you time and money.

Dealing with Claude’s token limit

Claude truncates responses that exceed its token budget. When you ask for a very large stack, the model may return only the first part of the file. To avoid this:

Ask for chunks – “Give me the first 200 lines, then the next 200 lines.”
Use the streaming API – it sends tokens as they are generated, letting you concatenate the pieces on the client side.
Keep prompts focused – generate one logical unit (e.g., an S3 bucket) at a time, then compose them in your CDK app.

In plain English: Think of Claude like a short‑hand writer; if the page gets too crowded, it stops. Break the job into smaller pages, or let Claude write continuously and stitch the pages together.

The Takeaway

You now have a repeatable workflow that turns a natural‑language description into a type‑safe CDK stack, validates it, and deploys it through an automated pipeline.

Prompt matters more than code. A clear recipe yields correct infrastructure.
Claude’s function‑calling returns raw TypeScript that you can write straight to disk.
satisfies gives compile‑time confidence that the AI‑generated class matches the shape you expect.
CDK apps treat generated stacks like any other; just remember to bootstrap the target account/region.
Pipelines make deployment reproducible and catch errors (including token‑limit truncation) before they reach production.
Chunked or streaming responses keep Claude from cutting off code when you need larger constructs.

Give it a try: describe a DynamoDB table, an SNS topic, or even a full VPC in plain English, and watch Claude hand you ready‑to‑deploy TypeScript. Happy building!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-04 · Primary focus: CDK

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

How AI Agents Work: Building the Plan‑Act‑Observe Loop with AWS Step Functions

Dinesh_gowtham — Thu, 03 Sep 2026 15:11:10 +0000

AI agents are often portrayed as needing custom servers and endless glue code. In reality, the entire plan‑act‑observe cycle can be modeled as a deterministic state machine that runs fully managed. This post shows you how to wire Claude’s responses, a tool‑call API, and feedback handling together using Step Functions and API Gateway.

What Is the Plan‑Act‑Observe Loop?

Why it matters – An AI agent is just a computer program that decides what to do, does it, then looks at the result and decides the next move. This three‑step rhythm—plan, act, observe—is the engine behind everything from chat assistants that run code to robots that navigate a warehouse. If we can make the loop explicit, we gain the ability to retry, log, and scale each step without hand‑crafted orchestration.

Key terms

Plan – a text prompt that tells a language model (LLM) what goal to pursue and which tool it might need.
Act – an external operation, such as calling a micro‑service, running code, or hitting a database.
Observe – feeding the result of the act back into the LLM so it can adjust its next plan.

Think of the loop like a choose‑your‑own‑adventure book. Each page (state) tells the reader (the model) what options exist, the reader makes a choice (act), then opens the next page (observe) based on the outcome.

In plain English – The loop is just “ask → do → check” repeated until the goal is reached.

Mapping the Loop to Step Functions States

Why Step Functions? – AWS Step Functions is a fully managed service that lets you describe a workflow as a state machine. Each state can call a Lambda, invoke a Bedrock model, or make an HTTP request. The service already gives you retries, timeout handling, and visual execution history, which means you don’t have to write that plumbing yourself.

How the mapping looks

Loop stage	Step Functions state type	What it does
Plan	`Task` (Bedrock InvokeModel)	Sends a prompt to Claude and receives a plan JSON
Act	`Task` (API Gateway HTTP integration)	Calls the external tool (e.g., a code‑execution service)
Observe	`Pass` + `Choice`	Sends the tool’s output back to Claude, decides whether to loop again

The whole cycle can be expressed in Amazon States Language (ASL), a JSON dialect that Step Functions reads. Below is a minimal, runnable ASL snippet that shows the three states plus a retry policy for the LLM call.

{
  "Comment": "Plan‑Act‑Observe loop using Claude + external tool",
  "StartAt": "GeneratePlan",
  "States": {
    "GeneratePlan": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:bedrock:invokeModel",
      "Parameters": {
        "ModelId": "anthropic.claude-v2",
        "Body.$": "$.prompt"
      },
      "Retry": [
        {
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 2,
          "BackoffRate": 2,
          "MaxAttempts": 3
        }
      ],
      "Next": "CallTool"
    },
    "CallTool": {
      "Type": "Task",
      "Resource": "arn:aws:states:::apigateway:invoke",
      "Parameters": {
        "ApiEndpoint": "https://abcde.execute-api.us-east-1.amazonaws.com/prod/run",
        "Method": "POST",
        "Headers": {
          "Content-Type": "application/json"
        },
        "Body.$": "$.plan.tool_input"
      },
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.toolError",
          "Next": "FailLoop"
        }
      ],
      "Next": "ObserveResult"
    },
    "ObserveResult": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:bedrock:invokeModel",
      "Parameters": {
        "ModelId": "anthropic.claude-v2",
        "Body.$": "States.Format('Observe: {}', $.toolResult)"
      },
      "Next": "ShouldContinue"
    },
    "ShouldContinue": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.observation.done",
          "BooleanEquals": false,
          "Next": "GeneratePlan"
        }
      ],
      "Default": "Success"
    },
    "Success": {
      "Type": "Succeed"
    },
    "FailLoop": {
      "Type": "Fail",
      "Cause": "Tool invocation failed"
    }
  }
}

Tip – The Retry block protects the LLM call from transient network hiccups, but without a proper timeout you could loop forever on a permanently failing tool.

CDK Boilerplate for the State Machine

Below is a TypeScript CDK construct that creates the same workflow. The comments walk a beginner through each line.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as sfn from 'aws-cdk-lib/aws-stepfunctions';
import * as tasks from 'aws-cdk-lib/aws-stepfunctions-tasks';
import { BedrockClient } from '@aws-sdk/client-bedrock';
import { SfnClient } from '@aws-sdk/client-sfn';

// A tiny helper to keep the ASL JSON readable
const loopDefinition: sfn.StateMachineFragment = new sfn.Pass(this, 'Start')
  // First state: ask Claude for a plan
  .next(new tasks.CallAwsService(this, 'GeneratePlan', {
    service: 'bedrock',
    action: 'invokeModel',
    parameters: {
      ModelId: 'anthropic.claude-v2',
      Body: sfn.JsonPath.stringAt('$.prompt')
    },
    // retry policy matches the ASL example
    retryOnServiceExceptions: true,
    maxAttempts: 3,
    backoffRate: 2,
    interval: cdk.Duration.seconds(2)
  }))
  // Second state: call the external tool through API Gateway
  .next(new tasks.CallApiGatewayRestApi(this, 'CallTool', {
    api: myApi,                     // defined elsewhere, see next section
    method: 'POST',
    stageName: 'prod',
    requestBody: sfn.TaskInput.fromJsonPathAt('$.plan.tool_input')
  })
    .addCatch(new sfn.Fail(this, 'FailLoop', {
      cause: 'Tool invocation failed'
    })))
  // Third state: feed tool output back to Claude
  .next(new tasks.CallAwsService(this, 'ObserveResult', {
    service: 'bedrock',
    action: 'invokeModel',
    parameters: {
      ModelId: 'anthropic.claude-v2',
      Body: sfn.JsonPath.stringAt('$.toolResult')
    }
  }))
  // Decision point: are we done?
  .next(new sfn.Choice(this, 'ShouldContinue')
    .when(sfn.Condition.booleanEquals('$.observation.done', false), loopDefinition) // loop
    .otherwise(new sfn.Succeed(this, 'Success')));

export class AgentStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create the state machine from the fragment above
    new sfn.StateMachine(this, 'PlanActObserveSM', {
      definition: loopDefinition,
      timeout: cdk.Duration.minutes(5) // prevents endless execution
    });
  }
}

Key takeaway – By treating each loop step as a separate state, you inherit Step Functions’ built‑in observability and error handling without writing any custom glue code.

Implementing Claude Calls with the Bedrock SDK Integration

Why use Bedrock directly? – Bedrock is AWS’s managed LLM gateway. It abstracts the vendor‑specific API (Claude, Titan, etc.) behind a single, IAM‑controlled endpoint. Using the @aws-sdk/client-bedrock package lets you call Claude without managing keys or HTTP signatures yourself.

How the integration works – The InvokeModelCommand expects a JSON payload where prompt is a string. The response contains bytes that you must decode into UTF‑8 text. Below is a minimal Lambda function that the Step Functions Task state can invoke.

import { InvokeModelCommand, BedrockClient } from '@aws-sdk/client-bedrock';
import { APIGatewayProxyHandler } from 'aws-lambda';

// Re‑use a single client across invocations (cold‑start friendly)
const bedrock = new BedrockClient({ region: 'us-east-1' });

export const handler: APIGatewayProxyHandler = async (event) => {
  // Pull the prompt from the incoming event – Step Functions puts it under "input"
  const prompt = event.body?.prompt ?? 'You are an AI assistant.';

  // Build the Bedrock request
  const command = new InvokeModelCommand({
    modelId: 'anthropic.claude-v2', // Claude model identifier
    body: Buffer.from(JSON.stringify({ prompt })), // Bedrock expects a Uint8Array
    contentType: 'application/json',
    accept: 'application/json'
  });

  try {
    const response = await bedrock.send(command);
    // Decode the model's bytes back into a string
    const result = Buffer.from(response.body as Uint8Array).toString('utf-8');
    return {
      statusCode: 200,
      body: JSON.stringify({ result })
    };
  } catch (err) {
    console.error('Claude invocation failed', err);
    return {
      statusCode: 500,
      body: JSON.stringify({ error: 'Invocation error' })
    };
  }
};

In plain English – This Lambda just shovels a text prompt to Claude and returns whatever Claude says, handling network errors in a single place.

Gotchas to watch

Payload size – Both the input (prompt) and the output (result) must stay under 256 KB because Step Functions limits each state’s input and output to that size. If you need larger data, store it in S3 and pass a reference.
Standard vs. Express – The Standard workflow keeps a full execution history (up to 25 000 events). A long‑running agent may hit that ceiling, so watch the history size in CloudWatch.

Calling an External Tool via API Gateway Integration

Why expose the tool through API Gateway? – API Gateway gives you a HTTP endpoint that can sit in front of any compute target (Lambda, Fargate, ECS, or even an external SaaS). It also provides request validation, throttling, and CORS handling out of the box.

How to wire it – The Step Functions Task state can use the built‑in apigateway:invoke integration, which removes the need for a separate Lambda wrapper. The following CDK snippet creates a mock “code‑execution” service and gives it a public REST endpoint.

import * as apigw from 'aws-cdk-lib/aws-apigateway';
import * as lambda from 'aws-cdk-lib/aws-lambda';

// Simple Lambda that pretends to run code and returns a result
const runner = new lambda.Function(this, 'CodeRunner', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromInline(`
    exports.handler = async (event) => {
      const { code } = JSON.parse(event.body);
      // In a real system you would sandbox this!
      const result = eval(code); // ⚠️ unsafe, only for demo
      return { statusCode: 200, body: JSON.stringify({ result }) };
    };
  `)
});

// API Gateway front‑end
const api = new apigw.RestApi(this, 'ToolApi', {
  deployOptions: { stageName: 'prod' },
  // CORS misconfiguration is a common support ticket – enable it here
  defaultCorsPreflightOptions: { allowOrigins: apigw.Cors.ALL_ORIGINS }
});

const run = api.root.addResource('run');
run.addMethod('POST', new apigw.LambdaIntegration(runner), {
  // 29‑second integration timeout cannot be increased – keep Lambda fast
  integration: { timeout: cdk.Duration.seconds(29) }
});

Tip – Remember that API Gateway’s integration timeout is 29 seconds. If the external tool might run longer, break the work into smaller chunks or use an asynchronous pattern (e.g., SQS + Lambda).

Gotchas to watch

JWT authorizers – The REST API (v1) does not support JWT authorizers natively. If you need token‑based auth, you must use a Lambda authorizer or switch to HTTP API (v2).
Throttling – Limits are applied per AWS account, not per API. A busy unrelated API can starve this one, so consider request‑level throttling or a dedicated account for critical agents.

Observability, Retries, and Common Pitfalls

Why observability matters – An AI agent may loop many times before finishing. Without clear logs, you can’t tell whether it’s stuck, failing, or making progress. Step Functions already emits execution history, but you can enrich it with CloudWatch Logs, X‑Ray tracing, and custom metrics.

How to add logging – In CDK you can attach a LogGroup to the state machine and set a loggingConfiguration.

import * as logs from 'aws-cdk-lib/aws-logs';

// Create a dedicated log group with a 30‑day retention policy
const smLogGroup = new logs.LogGroup(this, 'AgentLogs', {
  retention: logs.RetentionDays.THIRTY,
  removalPolicy: cdk.RemovalPolicy.DESTROY
});

new sfn.StateMachine(this, 'PlanActObserveSM', {
  definition: loopDefinition,
  timeout: cdk.Duration.minutes(5),
  logs: {
    destination: smLogGroup,
    level: sfn.LogLevel.ALL, // captures each state transition
    includeExecutionData: true // puts input/output into the logs
  }
});

In plain English – This configuration writes every state entry and exit to CloudWatch, so you can later replay the exact sequence that led to a failure.

The silent‑retry gotcha

Step Functions will automatically retry a state if you configure a Retry block. If you forget to set a TimeoutSeconds or a Catch on the tool‑call state, a permanently failing external service will cause the workflow to bounce forever, consuming resources and inflating your bill.

Safe pattern

Timeout – Set a hard limit on each task (e.g., 30 seconds for the tool call).
Catch – Capture any error and move to a Fail state that records the problem.
Back‑off – Use an exponential back‑off strategy (IntervalSeconds, BackoffRate) to avoid hammering a flaky service.

new tasks.CallApiGatewayRestApi(this, 'CallTool', {
  // …parameters omitted for brevity
  timeout: cdk.Duration.seconds(20), // ensures we don’t wait forever
})
.addCatch(new sfn.Fail(this, 'ToolFailed', {
  cause: 'Tool did not respond in time',
  error: 'Timeout'
}), {
  errors: ['States.TaskFailed'],
  resultPath: '$.toolError'
});

Other known limits

Service	Limitation	Practical impact
Step Functions	256 KB input/output per state	Large model responses must be stored elsewhere
Step Functions	25 000 events per execution (Standard)	Very long loops can truncate history
Step Functions	Distributed Map → 10 000 concurrent child executions	Parallel tool calls need throttling
API Gateway	29 s integration timeout	Long‑running tool must be asynchronous
API Gateway	CORS misconfiguration → “Missing Access‑Control‑Allow‑Origin” errors	Always set `defaultCorsPreflightOptions` for public APIs

Tip – Keep the loop short (few iterations) or store intermediate data in DynamoDB/S3; otherwise you’ll bump into the event‑history or payload limits.

The Takeaway

Key points to remember

A Plan‑Act‑Observe loop can be expressed as a deterministic state machine; Step Functions gives you that machine for free.
Use Bedrock’s InvokeModel task to talk to Claude; keep payloads under 256 KB or offload to S3.
Expose external tools via API Gateway; respect the 29‑second timeout and configure CORS early.
Add explicit Retry, Timeout, and Catch blocks; otherwise the workflow may retry forever on a broken tool.
Leverage CloudWatch logging and the built‑in execution history to debug loops; watch the 25 000‑event limit for very long agents.
Remember service‑specific limits (Step Functions input size, API Gateway throttling) when you scale from a prototype to production.

Next Steps

Deploy the example – Copy the CDK stack into a new project, run cdk deploy, and trigger the state machine from the console.
Persist large data – Replace direct prompt/result passing with S3 object references once you exceed the 256 KB limit.
Add async tool calls – Use an SQS queue + Lambda pattern for tools that need more than 29 seconds.
Instrument with X‑Ray – Enable X‑Ray tracing on the state machine to get a visual trace of each iteration.

By treating AI agents as orchestrated state machines, you trade a heap of custom glue code for a platform that already handles retries, observability, and scaling. The result is a cleaner, more reliable agent that you can iterate on without worrying about underlying infrastructure. Happy building!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-03 · Primary focus: StepFunctions

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

How IAM Roles Anywhere Lets Your AI Agents Run Securely On‑Prem and In ECS, Explained Simply

Dinesh_gowtham — Wed, 02 Sep 2026 15:12:16 +0000

AI agents need credentials, but stuffing API keys into containers is a ticking time bomb. IAM Roles Anywhere flips the script, giving you short‑lived, auditable permissions without ever touching a secret file. Learn the exact steps to wire it up for your Node.js/TypeScript AI workloads.

Why IAM Roles Anywhere Matters for AI Agents

When an AI model talks to other AWS services (S3, Bedrock, DynamoDB…) it must prove “who it is”. The most common pattern is to drop an access key‑ID and secret‑access‑key into a secret store, mount that file into the container, and hope nobody steals it.

In plain English: Hard‑coded keys are like leaving your house keys under the doormat. Anyone who finds the container can walk straight in.

IAM Roles Anywhere (RA) replaces those static keys with short‑lived credentials that are created on demand. The flow is:

You create a trust anchor – a public key that tells AWS “I trust certificates signed by this CA”.
Your AI agent gets an X.509 certificate (the private half stays on the machine).
The agent sends the certificate to AWS STS (Security Token Service) and asks for a role.
STS returns temporary credentials (access key, secret key, session token) that are valid for a few minutes.

Because the credentials expire quickly and are logged, you gain zero‑trust behavior: the moment a certificate is revoked or expires, the agent can no longer call AWS.

Analogy

Think of IAM Roles Anywhere like a single‑use backstage pass at a concert. The pass is printed just before you need it, works for a short window, and the security staff can instantly see who used it and when. If the pass is lost, it’s already worthless after its time window.

Key takeaway: RA gives AI agents a dynamic, auditable identity instead of a permanent password.

Creating a Trust Anchor and Issuing a Certificate

The “why”

Before any agent can ask AWS for credentials, AWS must know which certificate authorities (CAs) it trusts. This is the trust anchor. If the anchor is missing or malformed, every request fails with AccessDeniedException – often with no obvious clue that the PEM order is wrong.

Step‑by‑step

Generate a private key and a self‑signed CA certificate (or use an internal PKI).
Upload the CA certificate to IAM as a trust anchor.
Issue a leaf certificate for the AI agent, signed by the CA.

Below is a minimal Node.js script that uses the IAM SDK to create the trust anchor. (You could also do it from the console; the code shows the API shape.)

// trust-anchor.ts
import {
  IAMClient,
  CreateOpenIDConnectProviderCommand,
  CreateOpenIDConnectProviderCommandInput,
} from "@aws-sdk/client-iam";
import { readFileSync } from "node:fs";

// 1️⃣ Load the CA certificate in PEM format.
//    The file must contain the full chain: leaf → intermediate → root, each block separated by a newline.
const caPem = readFileSync("./ca-cert.pem", "utf-8");

// 2️⃣ Build the request object.
//    IAM expects the certificate string exactly as‑is; any extra whitespace or wrong order breaks the call.
const input: CreateOpenIDConnectProviderCommandInput = {
  // The URL is a placeholder; IAM uses it only as an identifier.
  Url: "https://my-company-iam-roles-anywhere",
  // The PEM string must be a single entry – no extra line breaks at start/end.
  ThumbprintList: [], // Not needed for RA; keep empty.
  // Provide the full certificate chain in the correct order.
  CertificateBody: caPem,
};

async function registerTrustAnchor() {
  const iam = new IAMClient({});
  try {
    const cmd = new CreateOpenIDConnectProviderCommand(input);
    const response = await iam.send(cmd);
    console.log("Trust anchor created, ARN:", response.OpenIdConnectProviderArn);
  } catch (err) {
    console.error("Failed to create trust anchor:", err);
  }
}

registerTrustAnchor();

Important PEM tip

The certificate file must list the leaf certificate first, then any intermediates, and finally the root. If you accidentally put the root at the top, IAM silently rejects the request with AccessDeniedException even though the ARN you pass looks correct.

Tip: Open the PEM in a text editor and verify the order: -----BEGIN CERTIFICATE----- (leaf) → -----BEGIN CERTIFICATE----- (intermediate) → -----BEGIN CERTIFICATE----- (root).

Issuing a leaf certificate

You can use OpenSSL or your corporate PKI. Here’s a one‑liner that creates a 30‑day certificate for the agent:

openssl req -new -newkey rsa:2048 -nodes -keyout agent.key.pem \
  -subj "/CN=ai-agent.example.com/O=MyCompany" \
  -out agent.csr.pem

openssl x509 -req -in agent.csr.pem -CA ca-cert.pem -CAkey ca-key.pem \
  -CAcreateserial -days 30 -sha256 -out agent-cert.pem

Keep agent.key.pem private on the host; agent-cert.pem will be mounted read‑only into the container.

Configuring an ECS Task to Assume the Role

The “why”

An ECS task runs inside a container. To let that container call AWS, you attach an IAM role to the task definition. With RA, the task’s execution role stays static (it only needs sts:AssumeRoleWithWebIdentity permission), while the application role is fetched at runtime using the certificate.

If you mistakenly assign the permission to the task role instead of the execution role, the container will fail to retrieve credentials, and the error will appear only in CloudWatch logs.

Minimal task definition (JSON)

{
  "family": "ai-agent-task",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "1024",
  "memory": "2048",
  "executionRoleArn": "arn:aws:iam::123456789012:role/EcsExecutionRole",   // <-- needs sts:AssumeRoleWithWebIdentity
  "taskRoleArn": "arn:aws:iam::123456789012:role/AIApplicationRole",      // <-- optional, not used for RA
  "containerDefinitions": [
    {
      "name": "ai-agent",
      "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/ai-agent:latest",
      "essential": true,
      "environment": [
        { "name": "AWS_REGION", "value": "us-east-1" },
        { "name": "CERT_PATH", "value": "/certs/agent-cert.pem" },
        { "name": "KEY_PATH", "value": "/certs/agent.key.pem" }
      ],
      "mountPoints": [
        {
          "sourceVolume": "certs",
          "containerPath": "/certs",
          "readOnly": true
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/ai-agent",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ],
  "volumes": [
    {
      "name": "certs",
      "host": {
        "sourcePath": "/opt/ai-agent/certs"
      }
    }
  ]
}

What each part does

executionRoleArn – the role that ECS itself uses to pull images and to call STS on behalf of the container. It must contain the permission sts:AssumeRoleWithWebIdentity.
taskRoleArn – not required for RA; you could still use it for other AWS calls that don’t need the dynamic certificate.
mountPoints – maps the host directory that holds agent-cert.pem and agent.key.pem into the container at /certs.

In plain English: The container gets a static role that only lets it ask AWS for a temporary role, and the certificate is the secret that proves who it is.

Node.js Code: Using the IAM SDK with TypeScript’s `satisfies`

The “why”

Now that the container has its certificate, it needs to ask STS for temporary credentials. The call we make is AssumeRoleWithWebIdentity, where the web identity token is the PEM‑encoded X.509 certificate. The response contains three strings (AccessKeyId, SecretAccessKey, SessionToken) that we feed to any other AWS SDK client, such as the Bedrock client for Claude.

Using TypeScript’s satisfies operator lets us declare the shape of the request object once and get compile‑time safety without locking us into a specific type alias.

Full example (`agent.ts`)

// agent.ts
import {
  IAMClient,
  AssumeRoleWithWebIdentityCommand,
  AssumeRoleWithWebIdentityCommandInput,
} from "@aws-sdk/client-iam";
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
import { readFileSync } from "node:fs";
import { fromTemporaryCredentials } from "@aws-sdk/credential-providers";

// ---------------------------------------------------------------------------
// 1️⃣ Load the PEM‑encoded certificate (the "web identity token").
//    The file must contain the leaf certificate *only*.
// ---------------------------------------------------------------------------
const certPath = process.env.CERT_PATH ?? "/certs/agent-cert.pem";
const certPem = readFileSync(certPath, "utf-8");

// ---------------------------------------------------------------------------
// 2️⃣ Build the AssumeRole request.
//    We use `satisfies` so the object is checked against the SDK’s input type
//    but the inferred type stays as a plain literal (useful for later spreads).
// ---------------------------------------------------------------------------
const assumeInput = {
  RoleArn: "arn:aws:iam::123456789012:role/AIApplicationRole", // the role we want temporary creds for
  RoleSessionName: "ai-agent-session",                       // an identifier that appears in CloudTrail
  WebIdentityToken: certPem,                                 // our X.509 cert becomes the token
  DurationSeconds: 900,                                      // 15 minutes – short enough for security
} satisfies AssumeRoleWithWebIdentityCommandInput;

// ---------------------------------------------------------------------------
// 3️⃣ Call STS via IAM client to get temporary credentials.
// ---------------------------------------------------------------------------
async function getTemporaryCredentials() {
  const iam = new IAMClient({}); // region comes from AWS_REGION env var
  const cmd = new AssumeRoleWithWebIdentityCommand(assumeInput);
  try {
    const resp = await iam.send(cmd);
    // The response contains AccessKeyId, SecretAccessKey, SessionToken.
    if (!resp.Credentials) {
      throw new Error("No credentials returned from AssumeRole");
    }
    return {
      accessKeyId: resp.Credentials.AccessKeyId!,
      secretAccessKey: resp.Credentials.SecretAccessKey!,
      sessionToken: resp.Credentials.SessionToken!,
    };
  } catch (err) {
    // Specific handling for expired or malformed certificates
    if ((err as any).name === "ExpiredTokenException") {
      console.error("⚠️ Certificate has expired – rotate it and redeploy.");
    } else {
      console.error("Failed to assume role:", err);
    }
    throw err; // re‑throw so the caller knows we failed
  }
}

// ---------------------------------------------------------------------------
// 4️⃣ Use the temporary credentials to call Claude on Bedrock.
// ---------------------------------------------------------------------------
async function invokeClaude(prompt: string) {
  const tempCreds = await getTemporaryCredentials();

  // Build a credential provider that returns the temporary creds.
  const credProvider = fromTemporaryCredentials({
    // The provider expects a function that returns a promise of credentials.
    masterCredentials: {
      accessKeyId: tempCreds.accessKeyId,
      secretAccessKey: tempCreds.secretAccessKey,
      sessionToken: tempCreds.sessionToken,
    },
  });

  const bedrock = new BedrockRuntimeClient({
    region: process.env.AWS_REGION,
    credentials: credProvider,
  });

  const invokeCmd = new InvokeModelCommand({
    modelId: "anthropic.claude-v2", // Example model
    contentType: "application/json",
    accept: "application/json",
    body: JSON.stringify({
      prompt,
      max_tokens_to_sample: 200,
      temperature: 0.7,
    }),
  });

  const response = await bedrock.send(invokeCmd);
  const result = JSON.parse(Buffer.from(response.body as Uint8Array).toString());
  console.log("Claude response:", result);
}

// ---------------------------------------------------------------------------
// 5️⃣ Run a simple test when the container starts.
// ---------------------------------------------------------------------------
(async () => {
  try {
    await invokeClaude("Explain why a short‑lived credential is safer than an API key.");
  } catch (e) {
    console.error("Agent failed:", e);
    process.exit(1);
  }
})();

What each block does

Load certificate – reads the PEM file that proves the container’s identity.
assumeInput – declares the request shape; satisfies guarantees we didn’t miss a required field.
getTemporaryCredentials – contacts STS, handles ExpiredTokenException (common when the 30‑day cert rolls over).
invokeClaude – builds a Bedrock client using the temporary credentials and sends a prompt.

Key takeaway: The whole credential dance lives inside the container; no secret file ever leaves the host, and the temporary keys disappear after 15 minutes.

Testing, Rotating, and Auditing the Credential Flow

The “why”

Even a perfect setup can go sideways if you don’t verify it regularly. Testing proves the certificate chain works, rotation prevents surprise expirations, and audit logs let you answer “who called Bedrock at 3 am?”.

Step‑by‑step checklist

Local dry‑run – Run node agent.ts on your laptop with a copy of the certificate. The script should print the Claude response or a clear error.
ECS integration test – Deploy the task definition with aws ecs run-task. Tail the logs (aws logs tail /ecs/ai-agent) and watch for the success line.
Certificate rotation – Schedule a CI job that generates a new leaf cert 5 days before the old one expires, places it in the host directory, and restarts the service.
CloudTrail audit – Enable CloudTrail for AssumeRoleWithWebIdentity. In the console, filter by eventName=AssumeRoleWithWebIdentity and verify the userIdentity.sessionContext.sessionIssuer.arn matches your AIApplicationRole.

Gotcha: PEM order again

When you replace the certificate, double‑check the file still contains only the leaf cert. Including the intermediate or root in the same file confuses the token parser and results in an AccessDeniedException with no hint in the container logs.

Tip: Keep two separate files: agent-cert.pem (leaf only) and ca-chain.pem (full chain) – the latter is only needed when you register the trust anchor.

Automating rotation with a simple script

#!/usr/bin/env bash
# rotate-cert.sh – run on the host that stores /opt/ai-agent/certs
set -euo pipefail

# 1️⃣ Generate a new leaf cert (30‑day validity)
openssl req -new -newkey rsa:2048 -nodes -keyout agent.key.pem \
  -subj "/CN=ai-agent.example.com/O=MyCompany" -out agent.csr.pem

openssl x509 -req -in agent.csr.pem -CA ca-cert.pem -CAkey ca-key.pem \
  -CAcreateserial -days 30 -sha256 -out new-agent-cert.pem

# 2️⃣ Swap the files atomically
mv new-agent-cert.pem /opt/ai-agent/certs/agent-cert.pem
mv agent.key.pem   /opt/ai-agent/certs/agent.key.pem

# 3️⃣ Restart the ECS service (or task) so the new cert is picked up
aws ecs update-service --cluster my-cluster --service ai-agent-service --force-new-deployment
echo "✅ Certificate rotated and service redeployed."

Running this script as a daily cron ensures you never see a surprise ExpiredTokenException.

The Takeaway

What you should remember after reading this guide

IAM Roles Anywhere lets an AI agent prove its identity with a short‑lived X.509 certificate instead of a static API key.
The trust anchor (CA certificate) must be uploaded to IAM with the correct PEM order; otherwise AWS silently denies the request.
In ECS, give the execution role permission sts:AssumeRoleWithWebIdentity; the task role is not used for RA.
The TypeScript agent loads the certificate, calls AssumeRoleWithWebIdentity, and uses the returned temporary credentials to talk to Bedrock (or any other AWS service).
Handle ExpiredTokenException and rotate certificates before they expire; automate the rotation to avoid downtime.
CloudTrail logs every AssumeRole call, giving you a full audit trail of which AI agent performed which action.

By following the steps above, you can run powerful AI workloads on‑premises or in ECS without ever storing long‑lived secrets, keeping your system both secure and observable. Happy building!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-02 · Primary focus: IAM

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

How to Build an AI Coding Assistant with Amazon Bedrock Claude Function Calling in Node.js

Dinesh_gowtham — Wed, 02 Sep 2026 07:17:17 +0000

Ever wondered how to turn Claude on Bedrock into a real‑time pair programmer that can edit your code for you? In a few minutes you’ll see a complete, type‑safe example that calls Claude, parses its structured tool output, and stitches the result into a live workflow. No magic black box – just plain Node.js, TypeScript, and AWS services you already trust.

Why Claude Function Calling Matters for Coding Assistants

When you ask a language model to “fix this bug”, the model usually replies with a block of text that looks like code. Your application then has to trust that the text is correct, extract it, and apply it. If the model makes a tiny syntax mistake, the whole process crashes.

Function calling solves that problem. Instead of returning raw text, Claude can emit a structured JSON payload that describes the exact edit it wants to make (file name, line range, replacement text, etc.). Your code can validate that payload against a TypeScript type, guaranteeing that the assistant never hands you malformed data.

In plain English: Function calling turns “Claude says some code” into “Claude tells us exactly what to change, in a format we can verify”.

Analogy – the mechanic’s checklist

Imagine you bring a car to a mechanic. A casual conversation (“I think the engine is noisy”) leaves you guessing. A proper checklist (“Replace spark plug #3, tighten belt A”) gives you a precise, actionable list. Function calling is the checklist for code changes.

Setting Up Bedrock Access and IAM Permissions

Before any code runs, you need two things:

Bedrock permission – the ability to call the InvokeModel operation on the Claude model you choose.
IAM permission – an AWS Identity and Access Management (IAM) role that your Node.js process assumes, with policies that let it talk to Bedrock and start a Step Functions state machine.

1. Create an IAM role for the assistant

import {
  IAMClient,
  CreateRoleCommand,
  PutRolePolicyCommand,
} from "@aws-sdk/client-iam";

const iam = new IAMClient({ region: "us-east-1" });

async function createAssistantRole() {
  // 1️⃣ Create a role that can be assumed by Lambda or an EC2 instance
  const createRole = new CreateRoleCommand({
    RoleName: "BedrockCodingAssistantRole",
    AssumeRolePolicyDocument: JSON.stringify({
      Version: "2012-10-17",
      Statement: [
        {
          Effect: "Allow",
          Principal: { Service: "lambda.amazonaws.com" },
          Action: "sts:AssumeRole",
        },
      ],
    }),
  });
  const role = await iam.send(createRole);

  // 2️⃣ Attach the minimum permissions we need
  const policy = new PutRolePolicyCommand({
    RoleName: role.Role.RoleName!,
    PolicyName: "BedrockAndStepFunctionsPolicy",
    PolicyDocument: JSON.stringify({
      Version: "2012-10-17",
      Statement: [
        // Bedrock InvokeModel permission
        {
          Effect: "Allow",
          Action: "bedrock:InvokeModel",
          Resource: "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-v2",
        },
        // Step Functions start execution permission
        {
          Effect: "Allow",
          Action: "states:StartExecution",
          Resource: "arn:aws:states:us-east-1:123456789012:stateMachine:LintAndFormatSM",
        },
      ],
    }),
  });
  await iam.send(policy);
  console.log(`Created role ${role.Role.RoleName}`);
}
createAssistantRole().catch(console.error);

What the code does

CreateRoleCommand builds an IAM role that Lambda (or any AWS compute service you choose) can assume.
PutRolePolicyCommand adds an inline policy granting exactly the two actions we need: bedrock:InvokeModel for Claude and states:StartExecution for Step Functions.

Tip: IAM evaluation is first‑deny, then‑allow. If any other policy (for example a Service Control Policy at the organization level) denies bedrock:InvokeModel, your role will still be blocked.

2. Attach the role to your runtime

If you run the code from a local machine, configure the AWS CLI with a profile that can sts:AssumeRole into the role you just created. For a Lambda function, set the role ARN in the console or CloudFormation.

Gotchas you’ll hit

Area	Gotcha	Why it matters
IAM	Explicit deny in a separate policy overrides our allow	Your code will get `AccessDeniedException` even though the inline policy looks correct
IAM	Default `AssumeRole` session duration is 1 hour	Long‑running lint‑and‑format jobs that exceed an hour will silently stop because the token expires
Bedrock	Token limit is per minute, not per request	Burst traffic can trigger `ThrottlingException` even if each request is tiny

Writing the Node.js TypeScript Wrapper with `satisfies`

The heart of the assistant is a tiny function that:

Sends a prompt to Claude asking it to produce a tool call (the JSON edit description).
Receives the response, which is wrapped inside a string field called toolResult.
Parses that string as JSON and verifies it matches a TypeScript interface using the satisfies operator (available from TS 4.9).

Define the shape of Claude’s tool response

// The exact data Claude will return when we ask it to edit code
interface CodeEditToolResult {
  filePath: string;           // Relative path of the file to edit
  startLine: number;          // 1‑based line number where the edit begins
  endLine: number;            // 1‑based line number where the edit ends
  replacement: string;        // The new code that should replace the range
}

// Helper type to assert that the parsed JSON conforms to the interface
type ValidEdit = {
  // Using the `satisfies` keyword ensures the object matches CodeEditToolResult
  // at compile time but does not widen the type at runtime.
  edit: unknown;
} & { edit: CodeEditToolResult } satisfies { edit: CodeEditToolResult };

The wrapper that talks to Bedrock

import {
  BedrockRuntimeClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime";

const bedrock = new BedrockRuntimeClient({ region: "us-east-1" });

/**
 * Sends a code snippet to Claude and asks for a structured edit.
 *
 * @param snippet The raw code the developer wants fixed.
 * @returns A validated CodeEditToolResult object.
 */
export async function requestCodeEdit(snippet: string): Promise<CodeEditToolResult> {
  // 1️⃣ Build the prompt that tells Claude we expect a tool call
  const prompt = `
You are a pair‑programmer. Given the following JavaScript snippet, return a JSON object
that describes the exact edit needed. Use the following schema:
{
  "filePath": string,
  "startLine": number,
  "endLine": number,
  "replacement": string
}
Wrap the JSON in a tool call named "applyEdit". Do NOT add extra text.
---
${snippet}
`;

  // 2️⃣ Call Claude via Bedrock. Important: request JSON output.
  const command = new InvokeModelCommand({
    // Model identifier for Claude v2 (replace with the version you have access to)
    modelId: "anthropic.claude-v2",
    // The body must be a JSON string, per Bedrock API contract
    body: JSON.stringify({
      prompt,
      temperature: 0.0,          // deterministic output for tooling
      maxTokens: 1024,
    }),
    // Crucial: tell Bedrock we want JSON back
    contentType: "application/json",
    accept: "application/json",   // <‑‑ the gotcha: missing this returns plain text
  });

  const response = await bedrock.send(command);
  // The response payload is a Uint8Array; convert to string
  const payload = Buffer.from(response.body).toString("utf-8");
  const parsed = JSON.parse(payload) as { result: string; toolResult?: string };

  // 3️⃣ The toolResult field contains the JSON string we asked for
  if (!parsed.toolResult) {
    throw new Error("Claude did not return a toolResult. Check your prompt or headers.");
  }

  // 4️⃣ Parse the toolResult string into an object
  const rawResult = JSON.parse(parsed.toolResult) as unknown;

  // 5️⃣ Type‑safety check using `satisfies`
  const validated: ValidEdit = { edit: rawResult } satisfies { edit: CodeEditToolResult };
  return validated.edit;
}

Why each piece matters

The accept: "application/json" header tells Bedrock to put the tool output inside toolResult. Forgetting it gives you plain text, which breaks the JSON parsing step.
The satisfies keyword lets the TypeScript compiler verify that rawResult matches CodeEditToolResult without casting (as). If Claude ever adds an extra field or misspells a key, the compiler will warn you during development.

Key takeaway: Using satisfies gives you compile‑time confidence while still allowing the runtime to accept any JSON that matches the shape.

Orchestrating Multi‑Step Tool Calls with Step Functions

A single Claude call can suggest an edit, but a good coding assistant also runs linters and formatters to make sure the change follows project standards. AWS Step Functions let you glue together several micro‑services into a reliable, retry‑aware workflow.

What is Step Functions?

Step Functions is a serverless state machine service. Think of it as a flowchart where each box is a Lambda (or container) that does one piece of work, and arrows dictate the order and error handling.

Define the state machine (JSON Amazon States Language)

{
  "Comment": "Lint, format, and return edited code",
  "StartAt": "RunLint",
  "States": {
    "RunLint": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:LintFunction",
      "ResultPath": "$.lintResult",
      "Next": "RunFormatter"
    },
    "RunFormatter": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:FormatterFunction",
      "ResultPath": "$.formatResult",
      "End": true
    }
  }
}

Explanation

RunLint invokes a Lambda that runs ESLint (or any linter) on the edited file.
RunFormatter runs Prettier (or another formatter) on the lint‑clean code.
The state machine returns the final, cleaned code to the caller.

Plain English: Step Functions is the conductor that makes sure each instrument (lint, format) plays in the right order and recovers from mistakes.

Deploy the state machine with the SDK

import {
  SFNClient,
  CreateStateMachineCommand,
} from "@aws-sdk/client-sfn";

const sfn = new SFNClient({ region: "us-east-1" });

async function createLintAndFormatSM() {
  const definition = JSON.stringify({
    Comment: "Lint, format, and return edited code",
    StartAt: "RunLint",
    States: {
      RunLint: {
        Type: "Task",
        Resource: "arn:aws:lambda:us-east-1:123456789012:function:LintFunction",
        ResultPath: "$.lintResult",
        Next: "RunFormatter",
      },
      RunFormatter: {
        Type: "Task",
        Resource: "arn:aws:lambda:us-east-1:123456789012:function:FormatterFunction",
        ResultPath: "$.formatResult",
        End: true,
      },
    },
  });

  const command = new CreateStateMachineCommand({
    name: "LintAndFormatSM",
    roleArn: "arn:aws:iam::123456789012:role/StepFunctionsExecutionRole",
    definition,
  });

  const result = await sfn.send(command);
  console.log(`Created state machine ${result.stateMachineArn}`);
}
createLintAndFormatSM().catch(console.error);

Gotcha reminder – the IAM role used by Step Functions must have lambda:InvokeFunction permission for both Lambda ARNs, otherwise the execution will stop with an AccessDenied error.

Putting It All Together: A Live Coding Assistant Demo

Now we combine everything:

Receive a developer’s code snippet (e.g., via an HTTP endpoint).
Call requestCodeEdit to get a structured edit from Claude.
Start the Step Functions state machine, passing the edited code.
Return the final, linted, formatted code to the caller.

Minimal Express server (TypeScript)

import express from "express";
import { requestCodeEdit } from "./claudeWrapper";
import {
  SFNClient,
  StartExecutionCommand,
} from "@aws-sdk/client-sfn";

const app = express();
app.use(express.json());

const sfn = new SFNClient({ region: "us-east-1" });
const STATE_MACHINE_ARN = "arn:aws:states:us-east-1:123456789012:stateMachine:LintAndFormatSM";

app.post("/edit", async (req, res) => {
  const { code, filePath } = req.body;
  if (typeof code !== "string" || typeof filePath !== "string") {
    return res.status(400).json({ error: "Invalid payload" });
  }

  try {
    // 1️⃣ Ask Claude for the edit description
    const edit = await requestCodeEdit(code);

    // 2️⃣ Build the input for Step Functions – include the replacement code
    const sfInput = {
      filePath: edit.filePath,
      startLine: edit.startLine,
      endLine: edit.endLine,
      replacement: edit.replacement,
    };

    // 3️⃣ Start the state machine; it will run lint → format
    const startCmd = new StartExecutionCommand({
      stateMachineArn: STATE_MACHINE_ARN,
      input: JSON.stringify(sfInput),
    });
    const execution = await sfn.send(startCmd);

    // 4️⃣ Poll for completion (simple example – in production use callbacks or SNS)
    const poll = async () => {
      const { describeExecution } = await import("@aws-sdk/client-sfn");
      const describeCmd = new describeExecution.Command({
        executionArn: execution.executionArn,
      });
      const status = await sfn.send(describeCmd);
      if (status.status === "RUNNING") {
        await new Promise((r) => setTimeout(r, 500));
        return poll();
      }
      if (status.status === "FAILED") {
        throw new Error(`Step Functions failed: ${status.error}`);
      }
      return JSON.parse(status.output!);
    };

    const finalResult = await poll();

    // 5️⃣ Return the polished code to the developer
    res.json({ editedCode: finalResult.formatResult });
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: (err as Error).message });
  }
});

app.listen(3000, () => console.log("Assistant listening on http://localhost:3000"));

Explanation of the flow

The HTTP /edit endpoint receives raw code and a file path.
requestCodeEdit talks to Claude, ensuring the accept header is set so the JSON lands in toolResult.
The returned CodeEditToolResult is passed as the input payload for the Step Functions state machine.
The state machine runs the LintFunction and FormatterFunction micro‑services (you can implement them as simple Lambdas that call ESLint/Prettier).
Once the state machine finishes, the server replies with the cleaned, edited code.

Tip: For production workloads replace the naive polling loop with an SNS topic or EventBridge rule that notifies your service when the execution finishes.

Handling rate limits and streaming responses

Bedrock limits tokens per minute, not per request. If you expect many developers to hit the endpoint at once, add a simple token bucket middleware:

import rateLimit from "express-rate-limit";

const limiter = rateLimit({
  windowMs: 60_000, // 1 minute
  max: 30, // max 30 requests per minute per IP
  standardHeaders: true,
  legacyHeaders: false,
});

app.use(limiter);

If you need streaming (e.g., streaming Claude’s partial thoughts), Bedrock returns Server‑Sent Events (SSE). Node.js does not parse SSE automatically, so you must read the response line‑by‑line and split on \n\n. For the purpose of a coding assistant, a single synchronous response is usually sufficient, but be aware that the streaming feature exists and has its own parsing complexity.

The Takeaway

What you now have: a fully type‑safe, multi‑step AI coding assistant built with Amazon Bedrock, IAM, and Step Functions, all driven from a modest Node.js/TypeScript codebase.

Function calling turns raw text into verified JSON, eliminating fragile string parsing.
IAM policies must be explicit: allow bedrock:InvokeModel and states:StartExecution, and remember that any explicit deny wins.
accept: application/json is non‑negotiable; without it Claude’s response lands in a plain‑text field, breaking the tool‑result workflow.
The satisfies keyword gives compile‑time safety without runtime casts, catching schema mismatches early.
Step Functions orchestrates linting and formatting, providing retries, error handling, and a clear separation of concerns.
Rate limits are per‑minute tokens, so guard your endpoint with a bucket or queue to avoid sudden throttling.

With these building blocks, you can extend the assistant to handle multiple files, integrate unit‑test generation, or even hook into your CI pipeline. The core pattern—prompt → structured tool output → validated JSON → orchestrated micro‑services—remains the same, and it scales nicely as your AI‑augmented development workflow grows. Happy coding!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-02 · Primary focus: Bedrock

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

How to Securely Store and Rotate LLM API Keys with AWS Secrets Manager in Node.js

Dinesh_gowtham — Tue, 01 Sep 2026 15:31:47 +0000

You’ve probably copied your OpenAI or Claude API key into source code and hoped no one ever sees it. In practice, that mistake leads to accidental leaks and painful rotation cycles. This guide shows a fail‑safe way to keep your LLM credentials hidden, auto‑rotate them, and fetch them efficiently at runtime.

Why Secrets Manager Beats Env‑Vars for LLM Keys

The problem with environment variables

An environment variable (env‑var) is a piece of data that a process reads from its operating system at start‑up. It feels convenient because you can process.env.API_KEY anywhere in your code. The downside is that the value lives in plain text on the host, can be printed in logs, and is often checked into version control by accident.

Think of an env‑var like writing a house key on a sticky note and taping it to the front door. Anyone who can see the door can copy the key.

What Secrets Manager gives you

AWS Secrets Manager (a managed service that stores, rotates, and audits secrets) acts like a digital safe deposit box. Your key never leaves the safe in clear text; the service returns it only after your code authenticates with AWS.

Encryption at rest – AWS encrypts the secret with a customer‑managed KMS key.
Fine‑grained access – IAM policies let you decide which Lambda or EC2 instance can read the secret.
Automatic rotation – You can attach a Lambda that fetches a fresh key from the LLM provider and writes it back.

In plain English: Secrets Manager keeps the secret hidden until the moment you really need it, and it can change the secret for you without you touching the code.

Quick comparison (code)

// ❌ Bad: hard‑coded env‑var (leaks easily)
const apiKey = process.env.CLAUDE_API_KEY;

// ✅ Good: fetch from Secrets Manager (encrypted, auditable)
import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";

const client = new SecretsManagerClient({ region: "us-east-1" });
async function getKeyFromSM(name: string) {
  const cmd = new GetSecretValueCommand({ SecretId: name });
  const resp = await client.send(cmd);
  return resp.SecretString ?? "";
}

The second snippet never stores the key in the source tree, and AWS logs every access for you to review.

Setting Up Automatic Rotation for an LLM API Key

Why rotate?

LLM providers (OpenAI, Anthropic, etc.) treat API keys like passwords. If a key is exposed, anyone can consume your quota, incur costs, and potentially damage your reputation. Rotating the key regularly limits the window an attacker has.

The rotation workflow

Create a secret – Store the initial LLM key as a JSON payload, e.g. {"key":"sk-abc123"}.
Enable rotation – Point the secret to a Lambda function that knows how to ask the LLM provider for a new key.
Schedule – AWS runs the Lambda on the interval you choose (e.g., every 30 days).
Versioning – Each rotation creates a new version of the secret. The label AWSCURRENT always points to the newest version, while AWSPREVIOUS holds the former one.

During the short moment when the Lambda writes the new version, AWSCURRENT flips to the fresh value before your application has finished using the old one. If your code assumes the secret is still in the old format, you can see JSON parse errors.

Implementing the rotation Lambda (minimal)

// rotate-claude-key.ts
import {
  SecretsManagerClient,
  GetSecretValueCommand,
  PutSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";
import fetch from "node-fetch"; // Node 22 includes fetch, but keep for clarity

const sm = new SecretsManagerClient({ region: "us-east-1" });

export const handler = async (event: any) => {
  // 1️⃣ Read the current secret (it contains the old key)
  const getCmd = new GetSecretValueCommand({ SecretId: event.SecretId });
  const current = await sm.send(getCmd);
  const oldPayload = JSON.parse(current.SecretString ?? "{}");

  // 2️⃣ Ask Claude for a new key (mocked request)
  const resp = await fetch("https://api.anthropic.com/v1/keys/rotate", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${oldPayload.key}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ reason: "scheduled rotation" }),
  });
  const newData = await resp.json(); // expects { key: "sk-new..." }

  // 3️⃣ Write the new key back to Secrets Manager
  const putCmd = new PutSecretValueCommand({
    SecretId: event.SecretId,
    SecretString: JSON.stringify({ key: newData.key }),
    // Mark this version as the new current one
    VersionStages: ["AWSCURRENT"],
  });
  await sm.send(putCmd);
};

Important notes

Cost awareness – Secrets Manager charges $0.40 per secret each month. Ten micro‑services that each keep its own key would cost $4 / month.
Lambda latency – The rotation Lambda should finish quickly; otherwise the secret stays in an inconsistent state.
Cross‑account access – If your Lambda lives in a different AWS account, you need both a resource policy on the secret and an IAM policy granting secretsmanager:GetSecretValue.

Tip: Give the rotation Lambda only the permissions it needs (GetSecretValue, PutSecretValue) and nothing more. The principle of least privilege reduces accidental exposure.

Type‑Safe Retrieval in Node.js Using the `satisfies` Operator

Why type safety matters

When you fetch a secret you receive a string. Turning that string into a usable object is a classic place where runtime errors happen (e.g., a missing field or a malformed JSON). TypeScript’s satisfies operator lets you describe the shape you expect and have the compiler verify it, without forcing a cast.

Defining the secret shape

// secret-types.ts
export interface ClaudeSecret {
  key: string; // the raw Claude API key, starts with "sk-"
}

Retrieval function with `satisfies`

// getClaudeKey.ts
import {
  SecretsManagerClient,
  GetSecretValueCommand,
  ResourceNotFoundException,
} from "@aws-sdk/client-secrets-manager";
import { ClaudeSecret } from "./secret-types";

const client = new SecretsManagerClient({ region: "us-east-1" });

/**
 * Fetches the Claude API key from Secrets Manager.
 * Throws if the secret is missing or does not match the expected shape.
 */
export async function fetchClaudeKey(secretName: string): Promise<string> {
  try {
    const cmd = new GetSecretValueCommand({ SecretId: secretName });
    const resp = await client.send(cmd);
    const raw = resp.SecretString ?? "{}";

    // Parse JSON and assert shape with `satisfies`
    const parsed = JSON.parse(raw) as unknown;
    if ((parsed as ClaudeSecret).key === undefined) {
      throw new Error("Secret does not contain a 'key' field");
    }
    const secret = parsed satisfies ClaudeSecret; // compile‑time check only

    return secret.key;
  } catch (err) {
    // Gracefully handle the moment a rotation deletes the old version
    if (err instanceof ResourceNotFoundException) {
      console.warn("Secret not found – possibly during rotation. Retrying later.");
      throw err; // caller can decide to retry
    }
    // Re‑throw any other unexpected errors
    throw err;
  }
}

Key points

ResourceNotFoundException is thrown when the secret version you asked for no longer exists (common during rotation).
The satisfies keyword ensures secret has a key property of type string. It does not coerce the value; if the JSON is malformed you still get a runtime error, which we catch early.

In plain English: satisfies is like a checklist you hand to the compiler: “Make sure the object looks like this, but don’t change it for me.”

Caching Secrets Locally Without Stale Data

Why cache?

Calling GetSecretValue on every Lambda invocation costs money (the API call itself isn’t free) and adds latency. A simple in‑memory cache can serve the same secret for many requests, but you must avoid serving an outdated key after rotation.

Cache design

Singleton – One instance per Node process.
TTL (time‑to‑live) – Refresh after a short interval (e.g., 5 minutes).
Version check – Compare the VersionId returned by Secrets Manager; if it changes, replace the cached value immediately.

Implementation

// secretCache.ts
import {
  SecretsManagerClient,
  GetSecretValueCommand,
  GetSecretValueResponse,
} from "@aws-sdk/client-secrets-manager";
import { ClaudeSecret } from "./secret-types";

type CacheEntry = {
  secret: ClaudeSecret;
  versionId: string; // unique identifier for the secret version
  expiresAt: number; // epoch ms when we consider the entry stale
};

class SecretCache {
  private client = new SecretsManagerClient({ region: "us-east-1" });
  private cache: Map<string, CacheEntry> = new Map();
  private readonly ttlMs = 5 * 60 * 1000; // 5 minutes

  /** Returns the secret, refreshing it if needed */
  async get(secretName: string): Promise<ClaudeSecret> {
    const now = Date.now();
    const entry = this.cache.get(secretName);

    // If we have a fresh entry, return it immediately
    if (entry && entry.expiresAt > now) {
      return entry.secret;
    }

    // Otherwise fetch a fresh version from Secrets Manager
    const fresh = await this.fetchFromSM(secretName);
    this.cache.set(secretName, {
      secret: fresh.secret,
      versionId: fresh.versionId,
      expiresAt: now + this.ttlMs,
    });
    return fresh.secret;
  }

  /** Low‑level fetch that also returns the version id */
  private async fetchFromSM(name: string): Promise<{ secret: ClaudeSecret; versionId: string }> {
    const cmd = new GetSecretValueCommand({ SecretId: name });
    const resp: GetSecretValueResponse = await this.client.send(cmd);

    const raw = resp.SecretString ?? "{}";
    const parsed = JSON.parse(raw) as unknown;
    const secret = parsed satisfies ClaudeSecret; // compile‑time shape check

    // `VersionId` is guaranteed by the SDK when the secret exists
    const versionId = resp.VersionId ?? "";
    return { secret, versionId };
  }
}

// Export a singleton instance
export const secretCache = new SecretCache();

How it avoids stale data

The cache expires after 5 minutes, guaranteeing at most that many minutes of lag.
If a rotation happens and the version id changes, the next call will fetch the new version because the old entry will be older than ttlMs.

Tip: Adjust the TTL based on your traffic pattern and cost tolerance. For low‑traffic Lambdas, a 1‑minute TTL often balances freshness and price.

Putting It All Together: A Minimal AI Agent That Calls Claude Securely

The goal

We will build a tiny function askClaude that:

Retrieves (and caches) the Claude API key from Secrets Manager.
Calls Claude’s /v1/complete endpoint with a user prompt.
Returns the generated text.

All error handling respects rotation windows and secret‑not‑found cases.

Full example

// aiAgent.ts
import { secretCache } from "./secretCache";
import fetch from "node-fetch"; // Node 22 includes global fetch; import for clarity

const CLAUDE_SECRET_NAME = "prod/claude/api-key";

/**
 * Sends a prompt to Claude and returns the model's response.
 * The function hides all credential handling behind the cache.
 */
export async function askClaude(prompt: string): Promise<string> {
  // 1️⃣ Get a fresh (or cached) API key
  const secret = await secretCache.get(CLAUDE_SECRET_NAME);
  const apiKey = secret.key;

  // 2️⃣ Build the request payload
  const body = {
    model: "claude-3-sonnet-20240229",
    prompt,
    max_tokens_to_sample: 256,
  };

  // 3️⃣ Call Claude's HTTP API
  const resp = await fetch("https://api.anthropic.com/v1/complete", {
    method: "POST",
    headers: {
      "x-api-key": apiKey,
      "content-type": "application/json",
    },
    body: JSON.stringify(body),
  });

  // 4️⃣ Handle HTTP errors (including auth failures during rotation)
  if (!resp.ok) {
    if (resp.status === 401) {
      // Unauthorized – likely because the key rotated while this Lambda was warm
      console.warn("Claude returned 401 – key may be stale. Flushing cache and retrying.");
      // Force a cache refresh on the next call
      secretCache["cache"].delete(CLAUDE_SECRET_NAME);
      throw new Error("Authentication failed – retry later");
    }
    const errBody = await resp.text();
    throw new Error(`Claude API error ${resp.status}: ${errBody}`);
  }

  const result = await resp.json();
  return result.completion?.trim() ?? "";
}

/* Example usage (e.g., inside an AWS Lambda handler) */
export const handler = async (event: any) => {
  const userPrompt = event.body?.prompt ?? "Tell me a joke.";
  try {
    const answer = await askClaude(userPrompt);
    return { statusCode: 200, body: JSON.stringify({ answer }) };
  } catch (e) {
    console.error("Failed to get Claude response:", e);
    return { statusCode: 500, body: "Internal server error" };
  }
};

What happens under the hood

The first call to askClaude loads the key from Secrets Manager and stores it in the in‑memory cache.
Subsequent calls reuse the cached key, saving a network round‑trip and a Secrets Manager bill.
If Claude rejects the key (common right after rotation), we clear the cached entry so the next request forces a fresh fetch.

Key takeaway: By separating credential retrieval, caching, and API interaction, you keep each piece simple and resilient to rotation quirks.

The Takeaway

Secrets Manager > env‑vars – it encrypts, audits, and can rotate keys without code changes.
Automatic rotation writes a new version, flips the AWSCURRENT label, and may cause a brief mismatch; handle that with retry logic.
satisfies operator gives compile‑time confidence that the secret’s JSON matches the shape you expect.
Cache wisely – a singleton with a short TTL avoids extra charges while still picking up new versions quickly.
Graceful failure – detect 401 responses from the LLM provider, flush the cache, and let the caller retry.

By following these steps you can store LLM API keys safely, rotate them automatically, and use them in a Node.js (or TypeScript) service without exposing secrets or paying surprise bills. Happy coding!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-01 · Primary focus: SecretsManager

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

Tracing AI Agent Workflows with AWS X‑Ray: A Step‑By‑Step Guide for Node.js

Dinesh_gowtham — Mon, 31 Aug 2026 18:05:24 +0000

When AI agents call external services, the invisible hand that stitches the calls together is often a mystery. X‑Ray lets you visualize every plan‑act‑observe step, turning guesswork into concrete data you can debug and improve.

In plain English: X‑Ray is like a GPS for your code, showing you every road your request travels.

Why Observability Matters for AI Agents

AI agents are tiny decision‑makers that repeatedly plan (ask a model what to do), act (call an API or write to a database), and observe (read the result and decide the next move).

If one of those steps stalls, you only see the symptom— a timeout or a missing record— not the root cause.

Observability means you have visibility (knowing what happened), tracing (following the path of a request), and metrics (measuring performance). Without it, you’re shooting in the dark.

Key takeaway: Adding observability to an agent gives you the same debugging confidence you have when you look at a microservice’s logs.

Analogy

Think of an AI agent as a courier delivering a package through several post offices. If the package never arrives, you want to know at which post office it got lost. X‑Ray is the tracking number that tells you exactly where the package was at each step.

Setting Up X‑Ray in a Node.js Project

Before you can start drawing maps, you need the right tools. We’ll use the @aws-sdk/client-xray package, which talks to the X‑Ray service, and the aws-xray-sdk-core package, which adds tracing hooks inside your Node.js code.

# Create a new project folder
mkdir ai-agent-xray && cd ai-agent-xray

# Initialise a Node.js project (choose defaults)
npm init -y

# Install TypeScript and the AWS SDKs we need
npm install typescript ts-node @types/node \
  @aws-sdk/client-xray @aws-sdk/client-dynamodb \
  @aws-sdk/client-bedrock-runtime aws-xray-sdk-core

# Initialise a basic tsconfig.json
npx tsc --init

What each package does

@aws-sdk/client-xray – low‑level client that lets you create sampling rules or retrieve trace data.
aws-xray-sdk-core – higher‑level library that automatically creates segments (chunks of a trace) and propagates context across async calls.

Tip: Run npm install inside a Lambda layer if you plan to deploy to Lambda; this keeps cold‑start size small.

Minimal X‑Ray bootstrap (src/xray.ts)

// src/xray.ts
import * as AWSXRay from 'aws-xray-sdk-core';

// Enable automatic tracing of HTTP/HTTPS calls
AWSXRay.captureHTTPsGlobal(require('http'));
AWSXRay.captureHTTPsGlobal(require('https'));

// Export the X‑Ray SDK instance for reuse
export const xray = AWSXRay;

The two captureHTTPsGlobal lines tell X‑Ray to watch every outgoing HTTP request, which is exactly what the Bedrock client uses under the hood.

In plain English: After this file runs, any fetch or https.request you make will be automatically noted in a trace.

Instrumenting the Agent Loop with the X‑Ray SDK

Now we write a tiny planner‑act‑observe loop. The loop:

Plans – asks Claude (via Bedrock) for the next action.
Acts – writes the plan result to DynamoDB.
Observes – reads the DynamoDB entry to decide whether to continue.

Each step gets its own subsegment (a child of the main request segment) so we can see timings separately.

// src/agent.ts
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
import { DynamoDBClient, PutItemCommand, GetItemCommand } from "@aws-sdk/client-dynamodb";
import { xray } from "./xray";                     // our X‑Ray bootstrap
import { v4 as uuidv4 } from "uuid";               // simple ID generator

// ---------------------------------------------------------------------
// Helper to create a Bedrock client (simulated for this guide)
// ---------------------------------------------------------------------
const bedrock = new BedrockRuntimeClient({ region: "us-east-1" });
const dynamo = new DynamoDBClient({ region: "us-east-1" });

/**
 * One iteration of the agent loop.
 * @param sessionId Unique identifier for this run.
 */
export async function runIteration(sessionId: string) {
  // The outermost segment represents the whole iteration.
  return xray.captureAsyncFunc("AgentIteration", async (subsegment) => {
    try {
      // ---------- PLAN ----------
      const plan = await xray.captureAsyncFunc("Plan", async (planSeg) => {
        const prompt = "You are a helpful assistant. What should I do next?";
        const command = new InvokeModelCommand({
          modelId: "anthropic.claude-v2",   // placeholder
          contentType: "application/json",
          body: JSON.stringify({ prompt })
        });
        const response = await bedrock.send(command);
        const text = Buffer.from(response.body).toString("utf-8");
        planSeg?.addAnnotation("model", "claude");
        planSeg?.addMetadata("prompt", prompt);
        return text;
      });

      // ---------- ACT ----------
      await xray.captureAsyncFunc("Act", async (actSeg) => {
        const putCmd = new PutItemCommand({
          TableName: "AgentResults",
          Item: {
            SessionId: { S: sessionId },
            Timestamp: { N: Date.now().toString() },
            Result: { S: plan }
          }
        });
        await dynamo.send(putCmd);
        actSeg?.addAnnotation("dynamoTable", "AgentResults");
      });

      // ---------- OBSERVE ----------
      const observation = await xray.captureAsyncFunc("Observe", async (obsSeg) => {
        const getCmd = new GetItemCommand({
          TableName: "AgentResults",
          Key: { SessionId: { S: sessionId } }
        });
        const data = await dynamo.send(getCmd);
        const result = data.Item?.Result?.S ?? "";
        obsSeg?.addMetadata("storedResult", result);
        return result;
      });

      // Return the observation so the caller can decide the next step
      subsegment?.addMetadata("finalObservation", observation);
      return observation;
    } catch (err) {
      // Mark the whole iteration as failed
      subsegment?.addError(err as Error);
      throw err;
    }
  });
}

Explanation of key lines

xray.captureAsyncFunc("AgentIteration", ...) – creates the top‑level segment for the whole loop iteration.
Inside each step we call captureAsyncFunc again, which creates a subsegment.
addAnnotation stores short key‑value pairs that you can filter on in the console.
addMetadata stores richer data (like the full prompt) that you can view when you click a trace.

Tip: Keep subsegment names short and consistent; they become the labels you’ll see in the trace viewer.

Gotcha: Async boundaries

X‑Ray automatically follows the Node.js async context, but only when you use the SDK’s wrappers (captureAsyncFunc, capturePromise). If you fire a raw setTimeout or a library that spawns its own event loop, the trace can break. In such cases you need to manually pass the segment header (AWS_XRAY_TRACE_ID) through the call.

Configuring Sampling Rules for Cost‑Effective Tracing

By default X‑Ray samples 5 % of requests, discarding the rest. That sounds fine until your agent runs 100 times per second; you’ll still see a few traces, but the fast loop may never appear because the sampling decision happens before the loop starts.

Why sampling matters

Each sampled trace costs a small amount. If you let the default rule run on a high‑traffic agent, you either pay a lot for a flood of traces, or you miss the data you actually need.

Creating a custom rule that keeps every iteration for the first 10 seconds, then samples 10 %

// src/sampling.ts
import {
  XRayClient,
  PutSamplingRulesCommand,
  SamplingRuleRecord,
} from "@aws-sdk/client-xray";

/**
 * Deploys a sampling rule that:
 *   • Records 100 % of traces for the "AgentIteration" segment name
 *   • Falls back to 10 % after the first 10 seconds of each minute
 */
export async function installCustomRule() {
  const client = new XRayClient({ region: "us-east-1" });

  const rule: SamplingRuleRecord = {
    RuleName: "AgentIterationHighResolution",
    Priority: 100,                    // lower numbers = higher priority
    FixedRate: 0.1,                   // 10 % default
    ReservoirSize: 5,                 // keep first 5 requests per second
    ServiceName: "*",                 // apply to any service
    ServiceType: "*",
    Host: "*",
    HTTPMethod: "*",
    URLPath: "*",
    Version: 1,
    // The "RuleARN" is not needed when creating; AWS adds it.
  };

  // The rule above captures 10 % of all traffic. To guarantee every
  // AgentIteration is kept for the first 10 seconds, we add a second rule:
  const highResRule: SamplingRuleRecord = {
    RuleName: "AgentIterationAlways",
    Priority: 50,
    FixedRate: 1.0,                   // 100 % sampling
    ReservoirSize: 0,
    ServiceName: "*",
    ServiceType: "*",
    Host: "*",
    HTTPMethod: "*",
    URLPath: "*",
    Version: 1,
    // We’ll use a custom attribute filter in the SDK later.
  };

  // Send both rules to X‑Ray
  await client.send(
    new PutSamplingRulesCommand({
      SamplingRuleRecords: [rule, highResRule],
    })
  );

  console.log("Custom sampling rules installed");
}

How it works

Reservoir – a bucket that lets a few requests through even when the fixed‑rate is low.
Priority – X‑Ray evaluates rules from low to high priority; the first match wins. By giving the “always” rule a higher priority (lower number), we guarantee that any trace that we label AgentIteration will be kept.
Cost control – after the first 10 seconds (when the high‑resolution rule expires), the fallback rule samples only 10 % of the rest.

In plain English: Think of sampling like a security guard who lets a certain number of guests into a museum. You can tell the guard to let all guests in during opening hour, then only a few later on.

Adding a custom attribute so the “always” rule can match only our agent

// src/agent.ts (add near the top of runIteration)
import { Segment } from "aws-xray-sdk-core";

// When we start the outer segment, attach a custom attribute
return xray.captureAsyncFunc("AgentIteration", async (subsegment) => {
  // Mark this segment so our sampling rule can recognise it
  (subsegment as Segment).addAnnotation("component", "ai-agent");
  // ... rest of the code stays the same

Now the “always” rule can be narrowed to annotation.component = "ai-agent" using the X‑Ray console’s rule editor, keeping costs predictable.

Viewing Traces in CloudWatch Application Signals

X‑Ray stores trace data in its own service, but CloudWatch Application Signals provides a unified view that mixes metrics, logs, and traces. To see our agent traces:

Open the CloudWatch console.
Choose Application Signals → Traces.
In the filter bar, type component = "ai-agent" to isolate our agent traces.
Click a trace to expand the timeline. You’ll see three subsegments: Plan, Act, Observe, each with its own latency bar.

Example screenshot description (no actual image)

The timeline shows a 120 ms “Plan” bar, a 45 ms “Act” bar, and a 30 ms “Observe” bar. Hovering over “Plan” reveals the prompt text we stored as metadata.

Key takeaway: CloudWatch Application Signals lets you jump from a high‑level latency chart straight into the detailed X‑Ray trace without leaving the console.

Exporting trace data for offline analysis

If you want to run statistical analysis (e.g., average plan time across 10 k iterations), you can export traces to S3 via the X‑Ray daemon or use the GetTraceSummaries API:

// src/export.ts
import {
  XRayClient,
  GetTraceSummariesCommand,
} from "@aws-sdk/client-xray";

export async function listRecentTraces() {
  const client = new XRayClient({ region: "us-east-1" });
  const now = Date.now();
  const oneHourAgo = now - 60 * 60 * 1000;

  const cmd = new GetTraceSummariesCommand({
    StartTime: new Date(oneHourAgo),
    EndTime: new Date(now),
    FilterExpression: "annotation.component = \"ai-agent\"",
    Sampling: false, // only return sampled traces (we ensured sampling above)
  });

  const result = await client.send(cmd);
  console.log(`Found ${result.TraceSummaries?.length ?? 0} agent traces`);
}

Running node -r ts-node/register src/export.ts will print the number of traces you collected in the last hour.

The Takeaway

In plain English: Adding X‑Ray to an AI agent is like giving the agent a diary that records every thought, action, and reaction, and then letting you read that diary in a tidy, searchable UI.

Observability lets you pinpoint latency spikes or failures inside the fast, repeatable loops of an AI agent.
The X‑Ray SDK (aws-xray-sdk-core) creates segments and subsegments that map directly to planner‑act‑observe steps.
Default sampling drops most traces; a custom rule that forces 100 % sampling for the agent component keeps critical data while a lower‑rate fallback controls cost.
Propagating context across async boundaries (e.g., SQS, SNS) requires you to pass the trace header manually; otherwise the trace will break.
CloudWatch Application Signals aggregates traces with logs and metrics, giving a single pane where you can filter by the component = "ai-agent" annotation.
Exporting traces via the X‑Ray API or daemon lets you run offline analytics, useful for capacity planning or SLA reporting.

Now you have a concrete, low‑overhead way to watch your AI agent’s brain at work, spot inefficiencies, and keep your AWS bill in check. Happy tracing!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-08-31 · Primary focus: XRay

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

CloudWatch EMF Explained Simply: How to Emit Zero‑Overhead Custom Metrics from Your AI Node.js Service

Dinesh_gowtham — Mon, 31 Aug 2026 08:50:20 +0000

Most engineers reach for the CloudWatch Metrics API and end up writing extra SDK calls that add latency and cost. Embedded Metrics Format (EMF) lets you ship rich, query‑able metrics by just writing specially‑formatted JSON logs. Learn how to turn your AI inference code into a self‑monitoring powerhouse with no extra overhead.

Why EMF Matters for AI Services

The problem we’re solving

An LLM (large language model) inference endpoint can handle hundreds of requests per second. Each request has a latency, a token count, and sometimes an error. If you log only the raw request/response, you must later run expensive log‑scans to calculate averages, percentiles, or error rates. Adding a second “metrics” call (e.g., PutMetricData) creates extra network hops, adds milliseconds to every inference, and inflates your AWS bill.

The EMF answer

Embedded Metrics Format (EMF) is a way to embed metric data directly inside a CloudWatch Logs event. Think of a log line as a post‑it note that not only tells you what happened but also carries a tiny scoreboard of numbers. CloudWatch reads the scoreboard, extracts the numbers, and stores them as regular CloudWatch metrics—without any separate API call.

In plain English: Write one JSON log line, get both a log entry and a metric for free.

Why AI teams should care

Zero added latency – no extra HTTP request, just a console.log.
Cost‑effective – you pay only for the log storage you already have; you avoid per‑metric‑data‑point charges from PutMetricData.
Richness – a single log line can contain many dimensions (e.g., model name, request ID) that become dimensions on the metric automatically.
Observability consistency – the same source that you debug with logs is also the source of your dashboards and alarms.

Embedding Metrics in CloudWatch Logs

The format at a glance

An EMF log entry is a JSON object that contains a top‑level @aws key. Inside @aws you must provide:

Timestamp – epoch milliseconds.
CloudWatchMetrics – an array describing each metric you’re publishing.
Namespace – a logical bucket (e.g., MyAIService) that groups related metrics.

All other top‑level keys become dimensions (labels) or measurements (numeric values) for the metric.

{
  "@aws": {
    "Timestamp": 1725067200000,
    "CloudWatchMetrics": [
      {
        "Namespace": "MyAIService",
        "MetricName": "InferenceLatency",
        "Dimensions": [["ModelName", "Endpoint"]]
      }
    ]
  },
  "ModelName": "gpt‑4‑mini",
  "Endpoint": "text‑completion",
  "InferenceLatency": 124,
  "TokenCount": 57,
  "Success": true
}

InferenceLatency, TokenCount, and Success are the measurements.
ModelName and Endpoint are the dimensions that let you break down the metric later.

Gotcha #1 – missing `@aws` fields

If you forget Timestamp or Namespace, CloudWatch silently drops the EMF data. The log still appears in CloudWatch Logs, but no metric shows up.

Tip: Validate your JSON with a small unit test before shipping it to production.

Gotcha #2 – subscription filter

CloudWatch only parses EMF if the log group has a subscription filter that forwards logs to the CloudWatch Metrics processing pipeline. Without it, the EMF line stays a plain log.

aws logs put-subscription-filter \
  --log-group-name /aws/ecs/my-ai-service \
  --filter-name EMFProcessor \
  --filter-pattern "" \
  --destination-arn arn:aws:logs:us-east-1:123456789012:destination:CloudWatchMetrics

In plain English: Think of the subscription filter as a “mailroom clerk” that reads each envelope (log) and extracts the scorecard (metric). If the clerk isn’t hired, the scorecard never leaves the envelope.

Cost considerations for the surrounding ecosystem

CloudWatch Logs Insights – each query scans the entire log volume at $0.005 per GB. At high traffic, frequent ad‑hoc queries can become pricey.
Log retention – by default logs never expire. Set a retention policy (e.g., 30 days) to avoid runaway storage costs.
Metric resolution – 1‑second granularity costs roughly three times more than 1‑minute. For most AI latency monitoring, 1‑minute is sufficient.
Alarms on missing data – during a deployment, a new version may temporarily stop emitting a metric. Alarms configured to treat “missing data = breaching” will fire unnecessarily. Set “missing data = ignore” or use a “breaching” alarm only after a warm‑up period.

Node.js 22: Using `diagnostics_channel` for Automatic EMF Emission

Why `diagnostics_channel`?

Node.js 22 introduced the diagnostics_channel module as a low‑overhead way for libraries to publish structured data about runtime events. It works like a radio station: the service (your inference code) broadcasts a message, and any listener (the EMF formatter) can pick it up without altering the main execution path.

Setting up a channel

// diagnosticsChannel.ts
import { createChannel } from 'node:diagnostics_channel';

// Create a channel named "ai-inference". The name is arbitrary but must be consistent.
export const inferenceChannel = createChannel('ai-inference');

Whenever an inference finishes, we publish a payload to this channel. The payload will later be turned into an EMF log line.

Listener that writes EMF JSON

// emfLogger.ts
import { inferenceChannel } from './diagnosticsChannel.js';
import { CloudWatchLogsClient, PutLogEventsCommand } from '@aws-sdk/client-cloudwatch-logs';
import { format } from 'util';

// Create a CloudWatch Logs client – we only need it once.
const cwLogs = new CloudWatchLogsClient({ region: 'us-east-1' });

// Helper to get the current timestamp in milliseconds.
const now = () => Date.now();

inferenceChannel.subscribe((event) => {
  // Build the EMF payload.
  const emf = {
    '@aws': {
      Timestamp: now(),
      CloudWatchMetrics: [
        {
          Namespace: 'MyAIService',
          MetricName: 'InferenceLatency',
          Dimensions: [['ModelName', 'Endpoint']],
        },
      ],
    },
    ModelName: event.model,
    Endpoint: event.endpoint,
    InferenceLatency: event.latencyMs,
    TokenCount: event.tokenCount,
    Success: event.success,
  };

  // Serialize to JSON – this is the single log line CloudWatch will parse.
  const message = JSON.stringify(emf);

  // In a real container you would write to stdout; CloudWatch Agent picks it up.
  // For demonstration we also push directly via SDK (optional, not required for EMF).
  console.log(message);
});

Explanation of each line

createChannel – creates a named broadcast pipe.
subscribe – registers a callback that runs as soon as the channel receives data, with virtually zero overhead.
@aws block – follows the EMF spec described earlier.
console.log(message) – writes the JSON to stdout; the AWS CloudWatch Agent (or the container’s logging driver) ships it to CloudWatch Logs.

Key takeaway: By using diagnostics_channel, you separate metric emission from business logic. The inference code just “fires an event”; the EMF formatter does the rest without adding latency.

Putting It All Together: A Real‑World AI Inference Service Example

Service skeleton

// server.ts
import http from 'node:http';
import { inferenceChannel } from './diagnosticsChannel.js';
import { startXRay } from './xraySetup.js';

// Start X‑Ray tracing (see next section for details)
await startXRay();

// Simple HTTP server that pretends to run an LLM inference.
const server = http.createServer(async (req, res) => {
  const start = Date.now();
  const requestId = crypto.randomUUID(); // unique ID for tracing

  // Simulate token counting and latency
  const tokenCount = Math.floor(Math.random() * 200) + 1;
  const latencyMs = Math.random() * 300 + 50; // 50‑350 ms

  // Randomly inject an error 5% of the time
  const success = Math.random() > 0.05;

  // Emit the EMF event via diagnostics_channel
  inferenceChannel.publish({
    model: 'gpt‑4‑mini',
    endpoint: 'text‑completion',
    latencyMs: Math.round(latencyMs),
    tokenCount,
    success,
  });

  // Respond to the caller
  res.writeHead(success ? 200 : 500, { 'Content-Type': 'application/json' });
  res.end(
    JSON.stringify({
      requestId,
      success,
      latencyMs: Math.round(latencyMs),
      tokenCount,
    })
  );
});

server.listen(8080, () => {
  console.log('AI inference service listening on :8080');
});

What this code shows

The business path (the HTTP handler) does not contain any CloudWatch SDK calls.
All metric data travels through diagnostics_channel, keeping the request path lean.
Errors are captured as a boolean (Success) that can be aggregated into an error‑rate metric.
The service also starts an X‑Ray segment (next section) so each request is traceable end‑to‑end.

Wiring the EMF logger

Remember the emfLogger.ts file from the previous section? Import it once so the subscription is active.

// index.ts
import './emfLogger.js'; // side‑effect import registers the listener
import './server.js';

When the container starts, the listener is registered, and every call to inferenceChannel.publish results in a single JSON log line.

Gotcha #3 – X‑Ray cold‑start overhead

If you enable X‑Ray on a Lambda or a container without adjusting the daemon, the X‑Ray SDK can add 50‑100 ms to cold starts. In our container we start the daemon once at boot time, and we configure the SDK to use the sampling rule that records only 5 % of requests in production.

// xraySetup.ts
import { XRayClient, PutSamplingRulesCommand } from '@aws-sdk/client-xray';
import { config } from 'node:process';

export async function startXRay() {
  // Turn off the default “all‑requests” sampler.
  const xray = new XRayClient({ region: 'us-east-1' });
  const cmd = new PutSamplingRulesCommand({
    SamplingRuleRecords: [
      {
        SamplingRule: {
          RuleName: 'AIInferenceLowSample',
          Priority: 1,
          FixedRate: 0.05, // 5 % of requests
          ReservoirSize: 5,
          ServiceName: '*',
          ServiceType: '*',
          Host: '*',
          HTTPMethod: '*',
          URLPath: '*',
          Version: 1,
          // Optional: limit to 1‑second intervals to avoid bursts.
        },
        // No specific tags needed here.
      },
    ],
  });
  await xray.send(cmd);
  console.log('X‑Ray sampler configured: 5 % of requests will be traced.');
}

Tip: Run a quick benchmark (ab -n 1000 -c 50 http://localhost:8080) before and after adding X‑Ray to see the cold‑start impact.

End‑to‑end flow recap

Client makes an HTTP request.
Server starts an X‑Ray segment (tracing) and notes the start time.
Inference logic runs (simulated here).
At the end, the code publishes an event to diagnostics_channel.
The EMF listener builds a JSON line that includes metrics and dimensions.
The container’s logging driver ships the line to CloudWatch Logs.
CloudWatch’s EMF processor extracts the metric and stores it under MyAIService.
You can now build dashboards, alarms, or use CloudWatch Application Signals for anomaly detection—all without a separate metrics SDK call.

Monitoring with CloudWatch Application Signals

What are Application Signals?

Application Signals is a newer CloudWatch feature that aggregates latency, error, and request‑count data automatically from multiple sources (including EMF) and runs machine‑learning‑based anomaly detection. It shows you “normal” ranges and flags outliers.

Connecting EMF data

Because our EMF payload already includes InferenceLatency (a latency metric) and Success (an error flag), Application Signals can ingest them directly. You only need to enable the feature on the log group.

aws logs put-subscription-filter \
  --log-group-name /aws/ecs/my-ai-service \
  --filter-name ApplicationSignals \
  --filter-pattern "" \
  --destination-arn arn:aws:logs:us-east-1:123456789012:destination:CloudWatchApplicationSignals

After a few minutes, the CloudWatch console will show a new Application Signals view for the MyAIService namespace, with automatic charts for:

p50 / p95 latency – derived from InferenceLatency.
Error rate – calculated from Success = false.
Token count distribution – useful for capacity planning.

In plain English: Application Signals works like a health‑monitoring smartwatch that reads your EMF “pulse” and alerts you when something feels off.

Gotcha #4 – Missing data during deployments

When you spin up a new container version, the first few seconds may not emit any EMF lines (e.g., if warm‑up logic runs before the first request). Application Signals treats those gaps as “missing data” and can mistakenly flag a spike. Configure the “missing data treatment” in the signal’s alarm settings to “ignore” or “missing = good”.

Cross‑account observability

If your AI service runs in a dev account but you want metrics in a central monitoring account, you need to add an observability access policy to the log group:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountRead",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::987654321098:root" },
      "Action": ["logs:PutLogEvents", "logs:CreateLogStream"],
      "Resource": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/ecs/my-ai-service:*"
    }
  ]
}

Now the central account can subscribe to the same log group and see the EMF metrics without duplicating data pipelines.

The Takeaway

Key points to remember

EMF turns a single JSON log line into a full CloudWatch metric, removing the need for a separate PutMetricData call.
The @aws block, correct namespace, and a subscription filter are mandatory; otherwise CloudWatch discards the metric silently.
diagnostics_channel (Node.js 22) provides a lightweight, zero‑overhead way to broadcast inference results to an EMF formatter.
A small listener converts those events into JSON and writes them to stdout; the CloudWatch Agent does the rest.
Application Signals can consume your EMF metrics automatically, giving you anomaly detection and out‑of‑the‑box dashboards.
Watch out for log‑group retention, Logs Insights query costs, metric resolution pricing, and X‑Ray cold‑start overhead.

Summary bullets

Emit once, read twice – one log line becomes both a log entry and a metric.
Configure the pipeline – @aws fields + subscription filter = metric materialization.
Use diagnostics_channel to keep metric emission out of the request path.
Tie in X‑Ray sparingly; set a low sampling rate to avoid cold‑start penalties.
Leverage Application Signals for automatic latency/error charts and ML‑driven alerts.
Guard against hidden costs – set log retention, choose 1‑minute resolution, and adjust missing‑data alarm behavior.

Now you can instrument your AI inference service with zero added latency, keep your CloudWatch bill in check, and gain instant visibility into latency, token usage, and reliability—all from a single, well‑structured log line. Happy monitoring!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-08-31 · Primary focus: CloudWatch

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

Claude Function Calling with Lambda Function URLs: Building a Secure, Zero‑Config AI Endpoint

Dinesh_gowtham — Fri, 28 Aug 2026 21:30:07 +0000

When you need a fast LLM endpoint, the first instinct is to spin up API Gateway—but Lambda Function URLs let you skip that extra hop entirely. Pair them with Claude’s function‑calling feature and you get a self‑contained AI service that’s ready in minutes. Yet most engineers still cling to the old, noisy stack.

Why Lambda Function URLs Beat API Gateway for LLM Endpoints

The problem with the “classic” stack

Imagine you’re sending a postcard. Using API Gateway is like handing the postcard to a postal clerk who stamps it, checks the address, and then hands it to the carrier. It works, but it adds a tiny delay and another place where something can go wrong.

Lambda Function URLs are the front‑door key: the postcard goes straight from your hand to the mailbox (the Lambda). There’s no middle‑man to configure, no extra cost for a separate service, and the latency is a few milliseconds lower.

Why that matters for large language models (LLMs)

Low latency – LLM calls already take 150‑300 ms; shaving even 20 ms off the surrounding HTTP hop improves the end‑user feel.
Zero‑config – No separate stage, stage variables, or deployment packages to keep in sync.
Built‑in HTTPS – Function URLs automatically expose a TLS‑secured endpoint, so you don’t have to attach a custom domain just to get encryption.

In plain English: A Lambda Function URL is the quickest, simplest way to expose a Lambda over HTTPS. If you only need one function (your Claude proxy) you can skip API Gateway entirely.

Key takeaway: For a single‑function AI service, Function URLs give you fewer moving parts, lower cost, and a clearer mental model.

Setting Up Claude Function Calling in a Node.js 22 Lambda

The “why” of function calling

Claude’s function‑calling feature lets the model suggest a structured action (a tool call) instead of returning free‑form text. Think of it like a customer asking a clerk for a receipt; the clerk hands back a neatly formatted paper rather than a scribbled note. This structure makes it safe to execute code—your Lambda can trust the JSON shape and act on it.

Minimal Lambda skeleton

Below is a complete Lambda handler written for Node.js 22 (the current LTS version). It:

Parses the incoming HTTP POST body (the payment request).
Sends a fetch request to Claude’s /v1/chat/completions endpoint, describing a single tool called validatePayment.
Reads Claude’s JSON‑encoded tool call, validates it, and returns a clean response.
Persists both the incoming request and Claude’s raw reply to an S3 bucket for audit.

// file: src/index.ts
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

// The S3 bucket where we store audit logs.
// Replace with your bucket name (must already exist).
const AUDIT_BUCKET = process.env.AUDIT_BUCKET ?? "my-ai-audit-bucket";

// Create a single S3 client – it reuses HTTP connections automatically.
const s3 = new S3Client({});

/**
 * Lambda entry point. The runtime passes an `event` object that contains
 * the raw HTTP request when the function is invoked via a Function URL.
 */
export const handler = async (event: any) => {
  try {
    // ---------- 1️⃣ Parse the incoming payment request ----------
    // `event.body` is a JSON string because Function URLs forward the raw body.
    const requestPayload = JSON.parse(event.body);
    // Example shape we expect:
    // { "orderId": "12345", "amountCents": 1999, "currency": "USD", "cardToken": "tok_abc" }

    // ---------- 2️⃣ Call Claude with a tool definition ----------
    const claudeResponse = await callClaude(requestPayload);

    // ---------- 3️⃣ Extract the tool call (validatePayment) ----------
    const toolResult = extractToolResult(claudeResponse);

    // ---------- 4️⃣ Persist request + response for audit ----------
    await persistAudit(event.body, JSON.stringify(claudeResponse));

    // ---------- 5️⃣ Return the tool result to the caller ----------
    return {
      statusCode: 200,
      headers: {
        "Content-Type": "application/json",
        // Simple CORS header – adjust the origin as needed.
        "Access-Control-Allow-Origin": "*",
      },
      body: JSON.stringify(toolResult),
    };
  } catch (err: any) {
    // ---------- Error handling ----------
    const status = err.statusCode ?? 500;
    return {
      statusCode: status,
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ error: err.message ?? "Internal error" }),
    };
  }
};

/**
 * Calls Claude’s chat completion endpoint with a single tool called
 * `validatePayment`. The tool tells Claude how to format a call that we
 * can safely execute.
 */
async function callClaude(paymentPayload: any) {
  // Claude API key should be stored in Secrets Manager / Parameter Store.
  const CLAUDE_API_KEY = process.env.CLAUDE_API_KEY!;
  const CLAUDE_URL = "https://api.anthropic.com/v1/chat/completions";

  // The tool definition – this tells Claude the JSON schema it must return.
  const tools = [
    {
      name: "validatePayment",
      description: "Check a credit‑card payment request for validity",
      input_schema: {
        type: "object",
        properties: {
          orderId: { type: "string" },
          amountCents: { type: "integer" },
          currency: { type: "string" },
          cardToken: { type: "string" },
        },
        required: ["orderId", "amountCents", "currency", "cardToken"],
      },
    },
  ];

  // Build the request payload for Claude.
  const body = {
    model: "claude-3-5-sonnet-20240620",
    max_tokens: 1024,
    messages: [
      { role: "user", content: "Validate this payment request." },
      { role: "assistant", tool_calls: [] }, // placeholder for tool call
    ],
    tools, // pass our tool definition
    // We also give Claude the raw payment data so it can decide whether to call the tool.
    // (Claude can also ask follow‑up questions; we keep it simple here.)
    // In a real app you might embed the payload in a system message.
    // For demo purposes we send it as part of the user message.
    // Example:
    // "Here is the payload: { ... }"
  };

  // Add the actual payload to the user content.
  (body.messages[0] as any).content = `Payment payload: ${JSON.stringify(
    paymentPayload,
  )}`;

  const response = await fetch(CLAUDE_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-key": CLAUDE_API_KEY,
      // Claude expects an explicit version header.
      "anthropic-version": "2023-06-01",
    },
    body: JSON.stringify(body),
  });

  if (!response.ok) {
    // Propagate HTTP errors as JavaScript exceptions.
    const errText = await response.text();
    const error = new Error(`Claude API error ${response.status}: ${errText}`);
    (error as any).statusCode = response.status;
    throw error;
  }

  // Claude returns a JSON structure that includes `tool_calls` when it wants to invoke a function.
  return await response.json();
}

/**
 * Pulls the JSON payload out of Claude’s tool call response.
 * Throws a ValidationError if the schema is not respected.
 */
function extractToolResult(claudeResponse: any) {
  // The path to tool calls can differ by model version; here we use the standard shape.
  const toolCalls = claudeResponse?.choices?.[0]?.message?.tool_calls;
  if (!toolCalls || toolCalls.length === 0) {
    throw new Error("Claude did not return a tool call");
  }

  const call = toolCalls[0];
  if (call.name !== "validatePayment") {
    throw new Error(`Unexpected tool name: ${call.name}`);
  }

  // `function_arguments` is a JSON string – parse it.
  let args: any;
  try {
    args = JSON.parse(call.function_arguments);
  } catch {
    throw new Error("Failed to parse Claude's tool arguments");
  }

  // Very light validation – in production you would use a JSON schema validator.
  const required = ["orderId", "amountCents", "currency", "cardToken"];
  for (const key of required) {
    if (!(key in args)) {
      throw new Error(`Missing required field ${key}`);
    }
  }

  // Return the clean, validated object.
  return { validated: true, details: args };
}

/**
 * Writes both the raw inbound request and Claude’s raw reply to S3.
 * Using a timestamped key makes it easy to browse audit logs later.
 */
async function persistAudit(rawRequest: string, rawResponse: string) {
  const timestamp = new Date().toISOString();
  const requestKey = `audit/${timestamp}_request.json`;
  const responseKey = `audit/${timestamp}_claude.json`;

  // Put the request object.
  await s3.send(
    new PutObjectCommand({
      Bucket: AUDIT_BUCKET,
      Key: requestKey,
      Body: rawRequest,
      ContentType: "application/json",
    }),
  );

  // Put the Claude response object.
  await s3.send(
    new PutObjectCommand({
      Bucket: AUDIT_BUCKET,
      Key: responseKey,
      Body: rawResponse,
      ContentType: "application/json",
    }),
  );
}

Explanation of the most important lines

Line	What it does
`event.body`	The HTTP payload sent by the caller (the payment request).
`fetch(CLAUDE_URL, …)`	Calls Claude over HTTPS using the built‑in `fetch` API (no extra library needed).
`tools`	Describes a tool – a function that Claude can ask you to run. The schema tells Claude exactly what keys and types to send back.
`tool_calls`	The place in Claude’s response where the model tells you “I want to run `validatePayment` with these arguments”.
`PutObjectCommand`	An AWS SDK command that stores an object (a file) in an S3 bucket.
`process.env.CLAUDE_API_KEY`	Pulls the secret API key from the Lambda’s environment variables – never hard‑code secrets.

Tip: When you first test locally, set process.env.CLAUDE_API_KEY in a .env file and use the dotenv package. In production you should store the key in AWS Secrets Manager and grant the Lambda read access via an IAM role.

Gotcha #1 – Payload size limit

Lambda Function URLs default to a 6 KB request body limit. Claude’s tool‑call JSON can easily exceed that when you include rich data (e.g., a whole order object). If you forget to raise MaximumPayloadSize (via the console or aws lambda update-function-url-config), the runtime silently truncates the body and you’ll see mysterious “invalid JSON” errors.

aws lambda update-function-url-config \
  --function-name MyClaudeProxy \
  --auth-type NONE \
  --max-payload-size 64KB

In plain English: Think of the default limit as a tiny mailbox slot; you need to ask AWS for a bigger slot before you start dropping larger letters in.

Securely Exposing the Endpoint with IAM Auth and CORS

Why security matters even for a “private” AI helper

An LLM can be instructed to generate code or manipulate data. If anyone on the internet can hit your endpoint, they could flood Claude with malicious prompts, rack up usage charges, or even exfiltrate data from your S3 bucket. IAM authentication gives you a strong, AWS‑native gatekeeper without adding a separate auth layer.

Enabling IAM auth on the Function URL

When you create the Function URL, set --auth-type AWS_IAM. The runtime will then require a signed request (SigV4). A typical front‑end can obtain temporary credentials from Amazon Cognito or from an EC2 instance role.

aws lambda create-function-url-config \
  --function-name MyClaudeProxy \
  --auth-type AWS_IAM \
  --cors '{"AllowOrigins":["https://myapp.example.com"],"AllowMethods":["POST"],"AllowHeaders":["Authorization","Content-Type"]}'

CORS – Cross‑Origin Resource Sharing, a browser security feature. The JSON above tells browsers that only https://myapp.example.com may call the endpoint, and only POST with the listed headers are allowed.

Minimal IAM policy for a caller

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "lambda:InvokeFunctionUrl",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:MyClaudeProxy"
    }
  ]
}

Attach this policy to the role that your front‑end (or another Lambda) assumes.

Key takeaway: IAM auth + a tight CORS rule gives you a “door with a lock and a peephole” – only callers with proper AWS credentials can open it, and browsers can’t be tricked into sending data from another site.

Known gotcha in the Node 22 runtime

Node 22 introduced native ES modules (.mjs) by default. If you still use require('esm') to force CommonJS, the Lambda layer that bundles @aws-sdk/client-lambda silently fails to load, producing a cryptic “Cannot find module” error at cold start. The fix is to switch the file extension to .js and add "type": "module" in package.json, or keep everything CommonJS by naming the file .cjs.

Persisting Calls and Auditing with S3 Object Lambda

Why you need an audit trail

Financial operations (like payment validation) are often subject to compliance rules. Keeping a tamper‑evident log of both the caller’s request and Claude’s exact reply helps you answer “who did what, when”.

Using S3 Object Lambda for on‑the‑fly redaction

Sometimes you must redact sensitive fields (e.g., cardToken) before the log is stored long‑term. S3 Object Lambda lets you attach a small Lambda that transforms the object as it’s being written.

aws s3control create-access-point-for-object-lambda \
  --name audit-redact-ap \
  --region us-east-1 \
  --configuration '{
    "SupportingAccessPoint": "arn:aws:s3:us-east-1:123456789012:accesspoint/my-audit-bucket-ap",
    "TransformationConfigurations": [{
      "Actions": ["GetObject", "PutObject"],
      "ContentTransformation": {
        "AwsLambda": {
          "FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:RedactCardToken"
        }
      }
    }]
  }'

Your RedactCardToken Lambda receives the raw object, removes cardToken, and returns the sanitized version to S3.

Analogy: Think of an S3 Object Lambda as a security guard at a mailroom who opens every envelope, removes any classified documents, and then reseals it before it goes into storage.

Simple redaction Lambda (Node 22)

// file: redact.ts
export const handler = async (event: any) => {
  const original = JSON.parse(event.getObjectContext.inputS3Url);
  // Delete the sensitive field
  delete original.cardToken;
  return {
    statusCode: 200,
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(original),
  };
};

Tip: Because the redaction Lambda runs for every put, keep it tiny (no external SDKs) to avoid extra latency.

Testing, Debugging, and the One Gotcha You’ll Hit

Local testing with SAM CLI

The Serverless Application Model (SAM) CLI can invoke a Function URL locally, letting you see the full request/response cycle without deploying.

sam local invoke MyClaudeProxy \
  -e events/payment-request.json \
  --env-vars env.json

events/payment-request.json might contain:

{
  "body": "{\"orderId\":\"A100\",\"amountCents\":2500,\"currency\":\"USD\",\"cardToken\":\"tok_123\"}"
}

Debugging payload truncation

If Claude returns a tool_calls array but the Lambda logs Claude did not return a tool call, the most common cause is the 6 KB limit mentioned earlier. Check CloudWatch logs for a line like:

2026-08-28T12:34:56.789Z    ERROR   JSON Parse error: Unexpected end of JSON input

That indicates the incoming body was cut off. Verify the Function URL’s MaximumPayloadSize setting.

Handling HTTP 4xx from Claude

Claude may respond with 400 Bad Request if the tool schema is malformed. Your callClaude helper already throws an error that bubbles up to the top‑level catch. To surface a friendlier message to the caller:

if (response.status >= 400 && response.status < 500) {
  const errMsg = await response.text();
  const err = new Error(`Invalid request: ${errMsg}`);
  (err as any).statusCode = response.status;
  throw err;
}

Key takeaway: Centralizing error handling lets you map Claude’s HTTP errors to your own API’s error model, keeping the front‑end experience consistent.

Gotcha #2 – Lambda response streaming

If you later decide to stream Claude’s response (useful for very large payloads), you must set Content-Type: application/octet-stream and add Transfer-Encoding: chunked headers. Omitting those causes the runtime to buffer the entire response, which defeats the purpose of streaming and can hit the 6 KB limit again.

The Takeaway

What you now have in your toolbox

Function URLs are the fast, zero‑config way to expose a single Lambda as an HTTPS endpoint. No API Gateway, no extra costs.
Claude’s function‑calling tool gives you a structured JSON contract, making it safe to execute model‑suggested actions.
IAM authentication + CORS provide a strong, AWS‑native security perimeter without writing custom auth code.
S3 Object Lambda lets you redact or transform audit logs on the fly, keeping compliance simple.
Payload‑size limit (6 KB) is the hidden roadblock most developers miss; raise MaximumPayloadSize early in the setup.
Error handling and streaming require explicit headers; otherwise the runtime falls back to buffering and you’ll hit hidden limits.

By stitching these pieces together, you can spin up a production‑grade, self‑contained AI endpoint in under ten minutes—perfect for payment validation, order verification, or any other task where you want the model’s reasoning plus the safety of a typed contract. Happy coding!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-08-28 · Primary focus: Lambda

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

How to Automate Pull‑Request Code Reviews with ChatGPT’s Function Calling and AWS Lambda (Node.js/TypeScript)

Dinesh_gowtham — Fri, 28 Aug 2026 14:31:52 +0000

AI code reviewers sound risky, but OpenAI’s function‑calling lets you turn a language model into a deterministic reviewer. In a few lines of TypeScript you can parse a PR diff, ask ChatGPT to generate a structured review, and post it back to GitHub—all inside a secure Lambda function. No custom prompt‑hacking required, just clean schema‑driven calls.

Why Function Calling Changes the Game

When you ask a language model to “review this code”, it replies with free‑form text. That text is great for conversation but terrible for automation because the format can drift over time. Function calling is a feature that forces the model to return data that matches a schema you define.

In plain English: Think of the model as a helpful intern. Instead of letting the intern write a loose essay, you give them a fill‑in‑the‑blank form. The intern still uses its knowledge, but the result is a predictable JSON object you can hand to other tools.

Why does this matter for code review?

Determinism – Your CI/CD pipeline can treat the output as a contract (summary string, issues array) rather than parsing prose.
Testability – You can write unit tests that compare the returned JSON to an expected shape.
Safety – If the model tries to “hallucinate” extra fields, the API will reject the call, preventing malformed data from reaching GitHub.

The only extra work is writing a function definition that describes the JSON shape. The model does the heavy lifting of filling it in, while you stay in control of the surrounding logic.

Setting Up the Lambda Function

A Lambda function is a tiny piece of code that runs in AWS without you managing servers. For a PR reviewer we need a function that:

Receives a GitHub webhook (HTTP POST) when a pull request is opened or updated.
Extracts the list of changed files and builds a diff string.
Calls OpenAI’s chat completion endpoint with a function definition.
Posts the JSON result back to GitHub as a comment.

Minimal package.json

{
  "name": "pr-review-lambda",
  "version": "1.0.0",
  "type": "module",
  "main": "dist/index.js",
  "scripts": {
    "build": "tsc",
    "start": "node dist/index.js"
  },
  "dependencies": {
    "openai": "^4.20.0",
    "@aws-sdk/client-lambda": "^3.540.0",
    "axios": "^1.7.2"
  },
  "devDependencies": {
    "typescript": "^5.5.2"
  }
}

Why type: "module"? Node 22 runs ES modules by default; this avoids the classic require(esm) pitfall that silently breaks Lambda layers.

Tip: When you add a new layer to a Lambda, double‑check that the layer also uses ES modules. Mixing CommonJS (require) and ESM (import) can cause cryptic runtime errors.

Basic handler skeleton

// src/index.ts
import { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';
import axios from 'axios';
import { OpenAI } from 'openai';

// Environment variables set in the Lambda configuration
const OPENAI_API_KEY = process.env.OPENAI_API_KEY!;
const GITHUB_COMMENT_URL = process.env.GITHUB_COMMENT_URL!; // e.g. https://api.github.com/repos/owner/repo/issues/123/comments

const openai = new OpenAI({ apiKey: OPENAI_API_KEY });

export const handler = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
  try {
    // 1️⃣ Parse GitHub webhook payload
    const payload = JSON.parse(event.body ?? '{}');
    const diff = await buildDiffFromPayload(payload);

    // 2️⃣ Ask ChatGPT for a structured review
    const review = await getReviewFromOpenAI(diff);

    // 3️⃣ Send the review back to GitHub
    await postReviewToGitHub(review);

    // 4️⃣ Lambda must return a proper HTTP response
    return {
      statusCode: 200,
      body: JSON.stringify({ message: 'Review posted' })
    };
  } catch (err) {
    console.error('Error in PR review Lambda', err);
    return {
      statusCode: 500,
      body: JSON.stringify({ error: (err as Error).message })
    };
  }
};

The handler function is the entry point AWS calls. It follows the same input → work → output pattern you already know from Express or Fastify.

Key takeaway: Keeping the Lambda thin (only orchestration) makes it easier to test each step in isolation.

Defining the Review Function Schema

OpenAI’s function calling expects a JSON schema that follows the OpenAPI style. The schema tells the model exactly which fields it may return and what types they have. For our reviewer we need:

{
  "name": "generate_review",
  "description": "Create a concise code‑review summary and list of issues found in the diff.",
  "parameters": {
    "type": "object",
    "properties": {
      "summary": {
        "type": "string",
        "description": "One‑sentence overview of the overall health of the PR."
      },
      "issues": {
        "type": "array",
        "description": "List of strings, each describing a single problem or suggestion.",
        "items": { "type": "string" }
      }
    },
    "required": ["summary", "issues"]
  }
}

Where the schema lives in code

// src/openaiSchema.ts
export const reviewFunction = {
  name: 'generate_review',
  description: 'Create a concise code‑review summary and list of issues found in the diff.',
  parameters: {
    type: 'object' as const,
    properties: {
      summary: {
        type: 'string' as const,
        description: 'One‑sentence overview of the overall health of the PR.'
      },
      issues: {
        type: 'array' as const,
        description: 'List of strings, each describing a single problem or suggestion.',
        items: { type: 'string' as const }
      }
    },
    required: ['summary', 'issues']
  }
};

Using as const tells TypeScript to treat the literal values as exact types, which later gives us type‑safe handling of the model’s response.

Analogy: The schema is like a blueprint for a LEGO set. The model has many bricks (knowledge) but can only build according to the instructions you provide.

The parsing gotcha

OpenAI sometimes packs the function call payload as a JSON string inside the function_call.arguments field, rather than as an object. If you try to access arguments.summary directly, you’ll get undefined and the Lambda crashes.

// src/openaiHelper.ts
import { ChatCompletionMessageParam } from 'openai/resources/chat/completions';

export async function getReviewFromOpenAI(diff: string) {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini-2024-07-18',
    messages: [{ role: 'user', content: diff }],
    functions: [reviewFunction],
    function_call: { name: 'generate_review' }
  });

  // The model may return arguments as a string; we must detect and parse safely.
  const fc = response.choices[0].message?.function_call;
  if (!fc) throw new Error('OpenAI did not return a function call');

  let args: any;
  try {
    // If `fc.arguments` is already an object, JSON.parse will throw – so we catch both cases.
    args = typeof fc.arguments === 'string' ? JSON.parse(fc.arguments) : fc.arguments;
  } catch (e) {
    console.error('Failed to parse function arguments', fc.arguments);
    throw new Error('Malformed function call payload');
  }

  // At this point `args` matches the schema { summary: string, issues: string[] }
  return args as { summary: string; issues: string[] };
}

Tip: Wrap the parsing in a try / catch block and log the raw payload. That way you can see exactly what OpenAI sent when the format changes.

Calling OpenAI from Node.js with Type‑Safe Types

Now that the schema is defined and the parsing logic is guarded, we can focus on the type‑safety that prevents runtime surprises.

Strongly typed OpenAI request

import { ChatCompletionCreateParams } from 'openai/resources/chat/completions';

// Build the request with explicit types so the compiler checks required fields.
const request: ChatCompletionCreateParams = {
  model: 'gpt-4o-mini-2024-07-18',
  messages: [{ role: 'user', content: diff }],
  functions: [reviewFunction],
  function_call: { name: 'generate_review' }
};

const response = await openai.chat.completions.create(request);

If you accidentally miss function_call, TypeScript will flag the error before you run the code.

Why typing matters for production pipelines

Early feedback – Your IDE instantly tells you when you forget a required property.
Clear contracts – Anyone reading the Lambda later knows exactly what shape the OpenAI request and response have.
Refactor safety – Renaming summary to overview in the schema automatically updates the type definition, so you don’t end up with mismatched keys.

In plain English: Think of TypeScript as a spell‑checker for your code’s grammar, catching mistakes before they become bugs in a live review.

Putting It All Together: Deploy and Test

1️⃣ Build the diff from the GitHub webhook

GitHub’s payload contains pull_request.changed_files and a files_url. We fetch each file’s patch and concatenate them.

// src/diffBuilder.ts
import axios from 'axios';

export async function buildDiffFromPayload(payload: any): Promise<string> {
  const pr = payload.pull_request;
  const filesUrl = pr.url + '/files';
  const token = process.env.GITHUB_TOKEN!; // needs repo scope

  const res = await axios.get(filesUrl, {
    headers: { Authorization: `token ${token}` }
  });

  // `res.data` is an array of file objects; we keep only the `patch` strings.
  const patches = res.data
    .filter((f: any) => !!f.patch) // ignore binary files
    .map((f: any) => `--- ${f.filename}\n${f.patch}`);

  // Join patches with a clear delimiter so the model knows where one file ends.
  return patches.join('\n\n');
}

If the webhook arrives for a “draft” PR, you might skip the review by returning an empty string early. This keeps the Lambda cheap.

2️⃣ Post the review back to GitHub

// src/githubPoster.ts
export async function postReviewToGitHub(review: { summary: string; issues: string[] }) {
  const commentBody = `**Automated Review**\n\n**Summary:** ${review.summary}\n\n**Issues:**\n${review.issues
    .map(issue => `- ${issue}`)
    .join('\n')}`;

  await axios.post(
    GITHUB_COMMENT_URL,
    { body: commentBody },
    { headers: { Authorization: `token ${process.env.GITHUB_TOKEN!}` } }
  );
}

GitHub expects a JSON payload { body: "string" }. By building the comment string in code we avoid markdown mistakes that would otherwise be hard to debug.

Key takeaway: Keeping the comment assembly in its own function makes it trivial to adjust formatting later (e.g., add emojis or check‑lists).

3️⃣ Deploy the Lambda

# 1️⃣ Compile TypeScript
npm run build

# 2️⃣ Zip the bundle (exclude devDependencies)
zip -r pr-review.zip dist node_modules

# 3️⃣ Create or update the function (example using AWS CLI)
aws lambda create-function \
  --function-name PRReview \
  --runtime nodejs22.x \
  --handler dist/index.handler \
  --zip-file fileb://pr-review.zip \
  --role arn:aws:iam::123456789012:role/lambda-execution \
  --environment Variables={OPENAI_API_KEY=...,GITHUB_TOKEN=...,GITHUB_COMMENT_URL=...}

Gotcha: If you enable SnapStart on a Lambda that sits in a VPC, the cold‑start time will still be dominated by the VPC attachment, so SnapStart gives no real benefit. For a PR reviewer you typically don’t need VPC access, so keep the function out of a VPC to stay fast and cheap.

4️⃣ Test locally before pushing

You can simulate a GitHub webhook with a JSON file:

aws lambda invoke \
  --function-name PRReview \
  --payload file://sample-payload.json \
  response.json

Inspect response.json. If you see a 500 error, check CloudWatch logs. The most common failure is the JSON‑string parsing issue described earlier.

Tip: Add a console.log(JSON.stringify(response, null, 2)) inside getReviewFromOpenAI just for local debugging. Remove or guard it in production to avoid leaking model data.

The Takeaway

Function calling turns a free‑form LLM output into a predictable JSON contract you can trust in CI pipelines.
TypeScript types enforce that contract at compile time, catching missing fields before they hit a live Lambda.
Parsing the function_call.arguments field safely avoids crashes when OpenAI returns a string instead of an object.
A thin Lambda orchestration (parse webhook → build diff → call OpenAI → post comment) keeps the codebase maintainable and cheap to run.
AWS-specific gotchas (SnapStart with VPC, ESM vs. CommonJS, provisioning costs) are easy to sidestep with the patterns shown above.

With these pieces in place you now have a repeatable, testable, and production‑ready automated code‑review step—without resorting to brittle prompt hacks or opaque black‑box behavior. Happy reviewing!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-08-28 · Primary focus: ChatGPTForEngineers

All code blocks are intended to be correct and runnable, but please verify them
against OpenAI's official docs before using in production.

Find an error? Drop a comment — corrections are always welcome.

Prompt Caching at the Edge: Using CloudFront Functions and Lambda to Speed Up Claude Calls

Dinesh_gowtham — Thu, 27 Aug 2026 21:06:37 +0000

LLM APIs like Claude feel snappy—until latency spikes hit your users. By caching prompt‑response pairs right at the edge, you can cut round‑trip time to milliseconds. This post shows you how to make that happen with CloudFront Functions and a Lambda origin.

Why Prompt Caching Matters for LLM‑Powered Apps

When a user types a question, your front‑end sends the text to an LLM (large language model) API, waits for the model to generate a reply, and then shows the answer. The user experience is dominated by two things:

Network latency – the time it takes for the request to travel from the user’s browser to the API endpoint and back.
Model compute time – how long the LLM needs to think.

Even if the model itself is fast, the network hop to the provider’s data center can add 100 ms – 300 ms, and sometimes more during traffic spikes. For a chat UI that refreshes every few seconds, those extra milliseconds feel like a noticeable lag.

Prompt caching means storing the exact prompt (the user’s message) together with the response (the model’s answer) in a fast lookup table. If the same prompt arrives again within a short window, you can return the cached answer instantly, without touching the LLM provider at all.

In plain English: Think of the cache as a “sticky note” on the receptionist’s desk. If someone asks the same question twice, the receptionist can hand them the note instead of calling the manager again.

Freshness vs. Speed

LLM responses are not immutable—new data, temperature settings, or model updates can change the answer. A short time‑to‑live (TTL) of a few minutes gives you a good trade‑off: most users repeat recent prompts, but you still get new answers after a reasonable window.

Setting Up a CloudFront Distribution with an Edge Key‑Value Store

The big picture

Edge KV store – a tiny key‑value database that lives on every CloudFront edge node.
CloudFront Function – a lightweight JavaScript snippet (max 2 MB) that runs on every request before it reaches the origin. It can read from the KV store but cannot make outbound HTTP calls.
Lambda origin – a regular AWS Lambda (not Lambda@Edge) that actually calls Claude’s API, stores the result in the KV store, and returns the JSON payload.

Create the KV store

# Install the AWS SDK for JavaScript (v3) if you need it locally
npm install @aws-sdk/client-cloudfront

// kv-setup.ts – run once with node
import {
  CreateKeyGroupCommand,
  CreateCachePolicyCommand,
  CreateOriginRequestPolicyCommand,
  CreateDistributionCommand,
  CloudFrontClient,
} from "@aws-sdk/client-cloudfront";

const client = new CloudFrontClient({ region: "us-east-1" });

async function createDistribution() {
  // The KV store is defined via a Cache Policy that enables “Cache‑Based Origin Request”.
  // In the console this appears as “Cache key and origin request settings”.
  const cachePolicy = await client.send(
    new CreateCachePolicyCommand({
      CachePolicyConfig: {
        Name: "ClaudePromptCachePolicy",
        // We want the request body (the prompt) to be part of the cache key.
        ParametersInCacheKeyAndForwardedToOrigin: {
          EnableAcceptEncodingBrotli: false,
          EnableAcceptEncodingGzip: false,
          // The request body is not automatically part of the key,
          // so we forward it to the origin (Lambda) where we will hash it.
          // The KV store itself is accessed via the Function, not the cache policy.
        },
        DefaultTTL: 0, // we control TTL manually in the KV store
        MaxTTL: 0,
        MinTTL: 0,
        // Important: Edge Functions can only read from KV if the cache behavior
        // points to a "function association" with “viewer request”.
        // The association is set later in the console or via the API.
        // No extra code needed here.
      },
    })
  );

  // Omitted: create the origin (Lambda ARN) and the distribution.
  // The key point for the beginner is that the distribution must have:
  //   - a Viewer Request Function (our edge function)
  //   - an Origin (our Lambda) for /chat
  //   - the default behavior points to the Lambda.
}
createDistribution().catch(console.error);

Key takeaway: CloudFront Functions can read/write a built‑in KV store, but they can’t talk to the internet. They’re perfect for a quick “look‑aside” cache; the heavy lifting stays in Lambda.

Gotchas to keep in mind

Item	Why it matters
2 MB code limit for Functions	Keep the script tiny; avoid large libraries.
No outbound network	Trying to fetch Claude from the function returns 502.
Cache invalidation delay (up to 60 s)	If you delete a KV entry, the edge may still serve it for a minute.
OAI (Origin Access Identity) is deprecated	If you still use it, CloudFront will log warnings but still work; plan to switch to Origin Access Control.

Writing the Lambda Origin that Calls Claude’s API

The Lambda does three things:

Parse the incoming JSON body to get the prompt.
Call Claude’s HTTP endpoint with fetch.
Store the prompt‑response pair in the CloudFront KV store (using the @aws-sdk/client-cloudfront CreateKeyValueStore‑style API).

Minimal Lambda code (Node 20)

// lambda-handler.ts
import { CloudFrontClient, PutKeyValueStoreCommand } from "@aws-sdk/client-cloudfront";

// Claude endpoint – replace with your actual URL and API key.
const CLAUDE_ENDPOINT = "https://api.anthropic.com/v1/complete";
const CLAUDE_API_KEY = process.env.CLAUDE_API_KEY!;

// TTL in seconds for the edge KV entry.
const KV_TTL = 300; // 5 minutes

export const handler = async (event: any) => {
  // -------------------------------------------------
  // 1️⃣ Extract the prompt from the incoming request.
  // -------------------------------------------------
  const body = JSON.parse(event.body);
  const prompt = body.prompt?.trim();
  if (!prompt) {
    return {
      statusCode: 400,
      body: JSON.stringify({ error: "Missing 'prompt' field" }),
    };
  }

  // -------------------------------------------------
  // 2️⃣ Call Claude. (fetch is available in Node 20)
  // -------------------------------------------------
  const claudeResponse = await fetch(CLAUDE_ENDPOINT, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-key": CLAUDE_API_KEY,
    },
    body: JSON.stringify({
      prompt,
      max_tokens: 256,
      temperature: 0.7,
    }),
  });

  if (!claudeResponse.ok) {
    const err = await claudeResponse.text();
    return {
      statusCode: claudeResponse.status,
      body: JSON.stringify({ error: err }),
    };
  }

  const claudeData = await claudeResponse.json();
  const answer = claudeData.completion;

  // -------------------------------------------------
  // 3️⃣ Store prompt → answer in the Edge KV store.
  // -------------------------------------------------
  // The KV store lives on CloudFront, not Lambda, but the SDK
  // lets us write to it from anywhere with the right permissions.
  const cf = new CloudFrontClient({});
  await cf.send(
    new PutKeyValueStoreCommand({
      // “ClaudePromptCache” is the name you gave the KV store
      // when you created the distribution.
      KeyValueStoreName: "ClaudePromptCache",
      Items: [
        {
          Key: prompt, // exact prompt string as the key
          Value: answer,
          // TTL tells the edge when the entry expires.
          // Edge will automatically drop it after 5 minutes.
          // The attribute name is `TTL`, expressed in seconds.
          TTL: KV_TTL,
        },
      ],
    })
  );

  // -------------------------------------------------
  // 4️⃣ Return the answer to the caller.
  // -------------------------------------------------
  return {
    statusCode: 200,
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ answer }),
  };
};

Why we use `PutKeyValueStoreCommand`

The CloudFront SDK treats the edge KV store like any other key‑value database: you give it a key (the prompt) and a value (the answer). The TTL ensures the entry disappears after the freshness window.

Gotchas in Lambda

Gotcha	Explanation
Node 22 `require(esm)` breakage	If you upgrade the runtime, any `require` of an ES module will silently fail. Stick to `import` syntax or pin the runtime to Node 20.
SnapStart + VPC = no benefit	SnapStart speeds up cold starts only when the function has no VPC attachment. If you need VPC (e.g., to reach a private Claude endpoint), the startup time stays the same.
Response streaming needs headers	If you ever switch to streaming Claude’s output, you must set `Transfer-Encoding: chunked` and a matching `Content-Type`. Otherwise CloudFront buffers the whole response.
Provisioned Concurrency cost	Enabling it makes the function always warm, but you’ll be billed even when idle. For a low‑traffic chat app, on‑demand is cheaper.
Lambda@Edge size limits	We are NOT using Lambda@Edge here; we use a regular regional Lambda, so the 1 MB response limit does not apply.

Storing and Invalidating Claude Responses in the Edge KV Store

Hashing the prompt for a stable key

Raw prompts can be long, and the KV store limits key length (max 256 bytes). A simple way to keep keys short is to hash the prompt with SHA‑256 and store the original prompt as metadata (optional).

import crypto from "crypto";

function hashPrompt(prompt: string): string {
  // Create a hex‑encoded SHA‑256 hash of the prompt.
  // This produces a fixed‑length, URL‑safe string.
  return crypto.createHash("sha256").update(prompt).digest("hex");
}

You would then use hashPrompt(prompt) as the KV key, and optionally store the original prompt in the value (e.g., as a JSON object {prompt, answer}) for debugging.

Invalidation strategy

Because the TTL is only five minutes, you rarely need manual invalidation. However, if you release a new model version or change the temperature, you may want to wipe the cache immediately.

// invalidate-cache.ts
import {
  CloudFrontClient,
  DeleteKeyValueStoreCommand,
} from "@aws-sdk/client-cloudfront";

const client = new CloudFrontClient({});

export async function clearCache() {
  await client.send(
    new DeleteKeyValueStoreCommand({
      KeyValueStoreName: "ClaudePromptCache",
    })
  );
  console.log("Edge KV store cleared");
}

Tip: Deleting the whole store is fast, but the edge may still serve stale data for up to 60 seconds due to propagation delay. If you need per‑key removal, you must overwrite the key with a very short TTL (e.g., 1 second) instead.

Gotchas for KV operations

Gotcha	Fix
KV write latency	Writes are asynchronous; the function returns to the caller before the edge fully propagates the entry. In practice the next request sees the entry within a few hundred milliseconds.
Maximum value size (10 KB)	Claude answers are usually small, but if you request large completions, truncate or compress before storing.
Permission errors	The Lambda’s execution role must have `cloudfront:UpdateKeyValueStore` permission. Add a policy like `{"Effect":"Allow","Action":"cloudfront:UpdateKeyValueStore","Resource":"*"}.`

Implementing a CloudFront Function to Serve Cached Responses

A CloudFront Function runs on the viewer request event. It can read from the KV store, and if it finds a hit, it can craft a response and stop further processing (i.e., skip the Lambda origin).

Minimal Function code (JavaScript, 2 MB limit)

// cf-function.js
/**
 * CloudFront Function – viewer request.
 * Looks for a cached answer to the incoming prompt.
 * If found, returns it immediately; otherwise lets the request
 * continue to the Lambda origin.
 */

function handler(event) {
  var request = event.request;
  var uri = request.uri; // e.g., "/chat"
  // We only care about POST /chat
  if (uri !== "/chat" || request.method !== "POST") {
    return request; // let everything else pass through
  }

  // -------------------------------------------------
  // 1️⃣ Extract the request body (the prompt JSON).
  // -------------------------------------------------
  // CloudFront Functions can only read the body if the
  // request is sent with the "application/json" content type.
  var body = request.body && request.body.text;
  if (!body) {
    // No body – forward to Lambda for proper error handling.
    return request;
  }

  var prompt;
  try {
    var parsed = JSON.parse(body);
    prompt = parsed.prompt?.trim();
    if (!prompt) throw "missing";
  } catch (e) {
    // Malformed JSON – forward to Lambda.
    return request;
  }

  // -------------------------------------------------
  // 2️⃣ Look up the prompt in the Edge KV store.
  // -------------------------------------------------
  // The KV store is called "ClaudePromptCache". The key is the
  // raw prompt string (or a hash if you used one in Lambda).
  var kvStore = "ClaudePromptCache";
  var cached = KV.get(kvStore, prompt);
  // KV.get returns null if the key does not exist or has expired.

  if (cached) {
    // -------------------------------------------------
    // 3️⃣ Build a synthetic response with the cached answer.
    // -------------------------------------------------
    var response = {
      statusCode: 200,
      statusDescription: "OK",
      headers: {
        "content-type": { value: "application/json" },
        "cache-control": { value: "max-age=0, private" },
      },
      body: JSON.stringify({ answer: cached }),
    };
    // Returning a response short‑circuits the rest of the
    // CloudFront pipeline – Lambda never runs.
    return response;
  }

  // -------------------------------------------------
  // 4️⃣ No cache hit → let the request travel to Lambda.
  // -------------------------------------------------
  return request;
}

How the function works, step by step

Check the URI – We only care about /chat POST requests.
Parse the JSON body – If we can’t read the prompt, we give up early.
KV lookup – KV.get(store, key) reads from the edge store. No network call, just a local memory read.
Cache hit – We craft a tiny HTTP response and return it. CloudFront sends it straight to the viewer.
Cache miss – The request proceeds to the origin (our Lambda), which will compute, store, and answer.

Analogy: Imagine a library with a “quick‑look” shelf at the front desk. If the book you need is already on that shelf, the librarian hands it to you right away. If not, you have to go to the back rooms (the Lambda) to fetch it.

Deploying the Function

CloudFront Functions are deployed with the @aws-sdk/client-cloudfront CreateFunctionCommand. Example (run locally):

npm install @aws-sdk/client-cloudfront

// deploy-function.ts
import {
  CloudFrontClient,
  CreateFunctionCommand,
  PublishFunctionCommand,
} from "@aws-sdk/client-cloudfront";
import * as fs from "fs";

const client = new CloudFrontClient({ region: "us-east-1" });

async function deploy() {
  const code = fs.readFileSync("cf-function.js", "utf8");
  const create = await client.send(
    new CreateFunctionCommand({
      Name: "ClaudePromptCacheFunction",
      FunctionConfig: {
        Comment: "Look up Claude prompt cache at the edge",
        Runtime: "cloudfront-js-1.0", // the only supported runtime
      },
      FunctionCode: Buffer.from(code),
    })
  );
  console.log("Created:", create.FunctionSummary?.ARN);

  // Publishing makes the function live.
  const publish = await client.send(
    new PublishFunctionCommand({
      Name: "ClaudePromptCacheFunction",
      IfMatch: create.ETag, // ensures we publish the right version
    })
  );
  console.log("Published version:", publish.FunctionSummary?.FunctionMetadata?.FunctionARN);
}
deploy().catch(console.error);

After publishing, attach the function to the Viewer Request event of the /chat cache behavior in the CloudFront console or via the API.

Gotchas for Functions

Gotcha	Fix
No outbound HTTP	Never try `fetch` inside the function; you’ll see a 502 error.
2 MB size limit	Keep the code tiny, avoid bundlers, use native `JSON.parse`.
Only simple headers	Functions can set headers but can’t manipulate cookies beyond simple key‑value pairs.
Propagation delay (up to 60 s)	After a new version is published, give the distribution a minute before testing.

The Takeaway

What you now have: a three‑piece pipeline that reduces the perceived latency of Claude calls from hundreds of milliseconds to a few milliseconds for repeat prompts.

Edge KV store gives you an ultra‑fast lookup table that lives on every CDN node.
CloudFront Functions can read that store on every request, but they cannot talk to the internet, so they must delegate the actual API call to a Lambda.
Lambda does the heavy work: calls Claude, writes the result back to the KV store with a short TTL, and returns the answer.
TTL of 5 minutes balances freshness with speed; you can clear the whole store when you roll out a new model version.
Common pitfalls (no outbound calls from Functions, code‑size limits, permission scopes, and propagation delays) are easy to avoid once you know them.

Now you can add prompt caching to any LLM‑backed feature—chatbots, code assistants, or summarization services—and give your users an experience that feels instant, even when the underlying model lives far away. Happy building!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-08-27 · Primary focus: CloudFront

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

DEV Community: Dinesh_gowtham

How to Query Vector Embeddings with Athena for Real‑Time RAG, Explained Simply

Why Athena Makes Sense for Vector Search

Minimal code: create an Athena client

Preparing an Iceberg Table with Embedding Parquet Files

What is Iceberg?

What is Parquet?

Step 1 – Write embeddings to a Parquet file

Step 2 – Create an Iceberg table that points at the Parquet file

SQL‑Based Cosine Similarity and Nearest‑Neighbor Query

What is cosine similarity?

Implementing cosine similarity in Athena SQL

Type‑checked Athena command in TypeScript

Wiring the Athena Query into a Node.js RAG Service

What is RAG?

End‑to‑end flow

Performance Gotchas and How to Avoid Them

The Takeaway

Real-Time AI Agent Event Stream with Kinesis: Build a Low-Latency LLM Chat Pipeline

Why Stream AI Agent Events with Kinesis?

Benefits that matter for a chat agent

A simple analogy

Setting Up the Kinesis Data Stream

Step 1: Create the stream with the AWS SDK for JavaScript v3

Gotcha: shard limits at scale

Lambda Consumer: Decode, Call Claude via Bedrock

The iterator problem

Full Lambda code (TypeScript)

Lambda‑specific gotchas

Enhanced Fan‑Out for Sub‑millisecond Latency

How it works

Registering a Lambda as an enhanced fan‑out consumer

Performance impact

End‑to‑End Flow Diagram and Deployment

Deploying with the AWS CDK (TypeScript)

The Takeaway

How to Turn Natural Language Prompts into CDK TypeScript Stacks with Claude – A Step‑By‑Step Guide

Prompt Crafting for Infra

Why spend time on the prompt?

How to write a good prompt

Claude‑Generated TypeScript Code

Why let Claude write the code?

How to request the code

Type‑Safe Validation with satisfies

Why add an extra type check?

How to validate the generated stack

Integrating into a CDK App

Why wrap the stack in an app?

How to add the stack

Known CDK gotchas (quick reminder)

Deploying via CDK Pipelines

Why use a pipeline?

How to set up a simple pipeline

Dealing with Claude’s token limit

The Takeaway

How AI Agents Work: Building the Plan‑Act‑Observe Loop with AWS Step Functions

What Is the Plan‑Act‑Observe Loop?

Mapping the Loop to Step Functions States

CDK Boilerplate for the State Machine

Implementing Claude Calls with the Bedrock SDK Integration

Calling an External Tool via API Gateway Integration

Observability, Retries, and Common Pitfalls

The silent‑retry gotcha

The Takeaway

Next Steps

How IAM Roles Anywhere Lets Your AI Agents Run Securely On‑Prem and In ECS, Explained Simply

Why IAM Roles Anywhere Matters for AI Agents

Analogy

Creating a Trust Anchor and Issuing a Certificate

The “why”

Step‑by‑step

Important PEM tip

Issuing a leaf certificate

Configuring an ECS Task to Assume the Role

The “why”

Minimal task definition (JSON)

What each part does

Node.js Code: Using the IAM SDK with TypeScript’s satisfies

The “why”

Full example (agent.ts)

Step 1 – Write embeddings to a Parquet file

Step 2 – Create an Iceberg table that points at the Parquet file

Step 1: Create the stream with the AWS SDK for JavaScript v3

Type‑Safe Validation with `satisfies`

Node.js Code: Using the IAM SDK with TypeScript’s `satisfies`

Full example (`agent.ts`)

Writing the Node.js TypeScript Wrapper with `satisfies`

Type‑Safe Retrieval in Node.js Using the `satisfies` Operator

Retrieval function with `satisfies`

Creating a custom rule that keeps every iteration for the first 10 seconds, then samples 10 %

Gotcha #1 – missing `@aws` fields