Prompt Caching with Claude in Node.js: How to Cut Latency and Save Tokens, Explained Simply

LLM calls feel slow because every request re‑generates the same answer. By treating a full prompt‑plus‑variables as a cacheable unit you can reuse Claude’s output for identical tasks. This post shows a production‑ready pattern that lives inside a Lambda function and uses Node.js’s diagnostics_channel for zero‑cost observability.

Why Prompt Caching Matters for Claude

When you ask Claude (or any large language model) a question, the model does a lot of heavy lifting: it runs billions of parameters through a transformer network, then streams back text. If the prompt—the exact string you send—doesn’t change, the model will produce the same output every time (assuming deterministic settings like temperature=0).

In plain English: If you ask the same question in the same way, you get the same answer. The work is repeated for no benefit.

Repeating that work costs two things:

1. Latency – the round‑trip time for the HTTP request plus the model’s compute time.
2. Tokens – the unit the provider bills you for. Even if you’re only reading the result, Claude still counts the prompt tokens and the generated tokens.

If you can remember the answer the first time, subsequent calls become a cheap memory lookup. That’s exactly what a cache does: store a result keyed by a signature of the request and return it instantly when the signature repeats.

The deterministic promise

Claude’s output is deterministic only when you lock the randomness parameters:

• temperature – a number (0‑1) that controls how “creative” the model is. 0 means deterministic.
• max_tokens – the maximum length of the generated completion.

If you change either of those, the same prompt could legitimately produce a different answer. Therefore they must be part of the cache key; forgetting them is the classic gotcha that leads to stale or low‑quality results.

Key takeaway: A cache is only safe when the whole request (prompt + all settings that affect output) is immutable.

Designing a Deterministic Prompt Signature

A signature is a short, fixed‑size identifier that uniquely represents the full request. The usual recipe:

1. Build the full prompt – concatenate the static system prompt, any variable pieces (e.g., a GitHub PR diff), and the model settings (temperature, max_tokens).
2. Hash the string – run a cryptographic hash function (SHA‑256) to get a 64‑character hex string.
3. Use the hash as the cache key – look it up in a map or LRU store.

Think of the hash like a library’s call number. You could store the entire book (the full prompt) on the shelf, but the call number lets you find it instantly.

Minimal example of building a signature

import { createHash } from "crypto";

/**
 * Turn a request into a reproducible cache key.
 *
 * @param prompt   The full text you will send to Claude.
 * @param temperature  Model randomness setting (0 = deterministic).
 * @param maxTokens    Upper bound on generated tokens.
 * @returns a 64‑character hex string.
 */
function makeSignature(
  prompt: string,
  temperature: number,
  maxTokens: number
): string {
  // Concatenate everything in a fixed order.
  const raw = `${prompt}|temp=${temperature}|max=${maxTokens}`;

  // SHA‑256 produces a 256‑bit (32‑byte) digest; we encode it as hex.
  return createHash("sha256").update(raw).digest("hex");
}

Tip: Keep the concatenation format stable (| separator) so that two logically identical requests never produce different hashes because of whitespace differences.

Implementing the Cache with diagnostics_channel and an In‑Memory Store

Why diagnostics_channel?

Node.js ships with diagnostics_channel, a lightweight publish/subscribe system that incurs no overhead when no listeners are attached. By emitting a custom event each time we hit or miss the cache, we can wire that event to a CloudWatch metric without adding any runtime cost in the happy‑path.

The in‑memory LRU cache

An LRU (Least Recently Used) cache automatically evicts the oldest entries when it reaches a size limit. This keeps memory bounded, which is crucial for a Lambda that may be reused many times.

We’ll use the tiny lru-cache package (which works in the Lambda runtime without native dependencies).

npm install lru-cache undici @aws-sdk/client-lambda

Putting it together

import { createHash } from "crypto";
import { request } from "undici"; // lightweight HTTP client
import LRU from "lru-cache";
import { channel } from "node:diagnostics_channel";

// ---------- 1. Cache setup ----------
const cache = new LRU<string, string>({
  max: 500, // store up to 500 entries (adjust for your memory budget)
  ttl: 1000 * 60 * 60, // 1 hour TTL – fresh enough for PR reviews
});

// Create a diagnostics channel named "claire-cache"
const cacheChannel = channel("claire-cache");

// Emit an object like { hit: true, key: <hash> } for each lookup.
function recordCacheEvent(hit: boolean, key: string) {
  if (cacheChannel.hasSubscribers) {
    cacheChannel.publish({ hit, key, timestamp: Date.now() });
  }
}

// ---------- 2. Signature helper ----------
function makeSignature(prompt: string, temperature: number, maxTokens: number): string {
  const raw = `${prompt}|temp=${temperature}|max=${maxTokens}`;
  return createHash("sha256").update(raw).digest("hex");
}

// ---------- 3. Claude call ----------
async function callClaude(
  prompt: string,
  temperature: number,
  maxTokens: number,
  apiKey: string
): Promise<string> {
  const response = await request("https://api.anthropic.com/v1/complete", {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "x-api-key": apiKey,
      // Claude expects an explicit version header
      "anthropic-version": "2023-06-01",
    },
    body: JSON.stringify({
      model: "claude-3-5-sonnet-20240620", // latest model as of 2026
      prompt,
      temperature,
      max_tokens_to_sample: maxTokens,
    }),
  });

  const data = await response.body.json();
  // The generated text is in `completion` according to Claude’s spec.
  return data.completion as string;
}

// ---------- 4. Main entry point (Lambda handler) ----------
export const handler = async (event: any) => {
  // Extract input – assume event contains a PR number.
  const prNumber: number = Number(event.prNumber);
  const apiKey = process.env.ANTHROPIC_API_KEY!; // keep secret in Lambda env

  // 4a. Build a deterministic prompt.
  const prompt = await buildPromptForPR(prNumber); // defined later
  const temperature = 0; // deterministic for caching
  const maxTokens = 1024;

  // 4b. Create cache key.
  const key = makeSignature(prompt, temperature, maxTokens);

  // 4c. Try cache first.
  let answer = cache.get(key);
  if (answer) {
    recordCacheEvent(true, key);
    console.log("Cache hit for PR", prNumber);
  } else {
    recordCacheEvent(false, key);
    console.log("Cache miss – calling Claude for PR", prNumber);
    answer = await callClaude(prompt, temperature, maxTokens, apiKey);
    cache.set(key, answer);
  }

  // 4d. Return the review summary.
  return {
    statusCode: 200,
    body: JSON.stringify({ prNumber, review: answer }),
  };
};

// ---------- 5. Helper: fetch PR diff and build prompt ----------
import { Octokit } from "@octokit/rest";

const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });

async function buildPromptForPR(prNumber: number): Promise<string> {
  // Pull the diff – keep it short for the example.
  const { data } = await octokit.pulls.get({
    owner: "my-org",
    repo: "my-repo",
    pull_number: prNumber,
    mediaType: {
      format: "diff",
    },
  });

  // Static system prompt that tells Claude what we want.
  const systemPrompt = `
You are an expert code reviewer. Summarize the changes in plain English,
highlight potential bugs, and suggest one concrete improvement.
Only output the summary, no extra formatting.`;

  // Combine system prompt and diff.
  return `${systemPrompt}\n\n--- Diff start ---\n${data}\n--- Diff end ---`;
}

What the code does, step by step

1. Cache creation – LRU holds up to 500 entries for an hour.
2. Diagnostics channel – claire-cache publishes a tiny JSON object each time we check the cache. CloudWatch can subscribe and turn those into custom metrics.
3. Signature generation – makeSignature hashes the prompt plus the deterministic settings.
4. Cache lookup – cache.get(key). On a hit we skip the network; on a miss we call Claude via undici.
5. Result storage – after a fresh Claude call we store the answer back into the LRU.
6. Prompt building – buildPromptForPR fetches a PR diff from GitHub and stitches it with a static instruction. The static instruction never changes, so the only variable part is the diff itself.

Analogy: Think of the cache as a coffee shop’s “favorite order” board. If a customer always orders the same latte with the same milk and syrup, the barista can write it down once and just hand it over the next time, saving the time it takes to grind beans and steam milk again.

Deploying the Pattern as a Lambda Function

Why Lambda?

AWS Lambda gives you automatic scaling, per‑invocation billing, and built‑in integration with CloudWatch. For a code‑review bot, you typically receive a webhook from GitHub, invoke the function, and return the summary.

Packaging for Node.js 22

Node.js 22 introduced native ES module support (type: "module" in package.json). However, the esm loader that some older Lambda layers rely on can break silently. To avoid that:

1. Set "type": "module" only if every file uses import/export.
2. Keep the handler file as a CommonJS module (module.exports.handler = …) because the Lambda runtime still expects that shape.
3. Do not bundle the lru-cache native code – it works out‑of‑the‑box.

Terraform snippet (optional, but helpful)

resource "aws_lambda_function" "pr_review" {
  function_name = "pr-review-cache"
  runtime       = "nodejs22.x"
  handler       = "dist/index.handler" # compiled output
  role          = aws_iam_role.lambda_exec.arn

  # Zip the compiled TypeScript output.
  filename = data.archive_file.lambda_zip.output_path

  environment {
    variables = {
      ANTHROPIC_API_KEY = var.anthropic_api_key
      GITHUB_TOKEN      = var.github_token
    }
  }

  # SnapStart is *not* useful when the function is attached to a VPC,
  # because the cold start time is dominated by ENI attachment, not code init.
}

Gotcha alert: If your function sits inside a VPC, enabling SnapStart gives you no latency benefit; the network interface creation dominates the start‑up time.

Adding the CloudWatch metric via diagnostics_channel

You can forward the cache events to CloudWatch using an async listener that runs once per container start.

import { CloudWatchClient, PutMetricDataCommand } from "@aws-sdk/client-cloudwatch";

const cw = new CloudWatchClient({});

if (cacheChannel.hasSubscribers === false) {
  // Subscribe only once per container.
  cacheChannel.subscribe((msg) => {
    const metric = {
      MetricName: msg.hit ? "CacheHit" : "CacheMiss",
      Dimensions: [{ Name: "FunctionName", Value: process.env.AWS_LAMBDA_FUNCTION_NAME! }],
      Unit: "Count",
      Value: 1,
    };
    const cmd = new PutMetricDataCommand({
      Namespace: "ClaudeCache",
      MetricData: [metric],
    });
    // Fire‑and‑forget – we don’t await to keep latency low.
    cw.send(cmd).catch((e) => console.error("Metric send error:", e));
  });
}

Because the listener is attached to the diagnostics channel, no extra code runs when no events are emitted. This is the “zero‑cost” observability promised earlier.

Tip: In the Lambda console you can now create a CloudWatch dashboard that shows CacheHit vs CacheMiss and instantly spot whether your cache is being effective.

Validating Cache Hits and Measuring Savings

How to test locally

1. Deploy the Lambda.
2. Invoke it twice with the same PR number (via the AWS CLI or Postman).
3. Observe the logs: the first run should print “Cache miss – calling Claude…”, the second “Cache hit for PR…”.
4. In CloudWatch, check the ClaudeCache namespace – you should see a CacheHit metric increment.

Real‑world numbers

Metric	Without cache	With cache (first request)	With cache (subsequent)
Average latency (ms)	~1,250	~1,250	~15
Tokens billed per request	1,200 (prompt + gen)	1,200	0 (response served from memory)
Cost per 1,000 calls	$0.12*	$0.12*	~$0.001*

*Claude pricing in 2026: $0.10 per 1M input tokens, $0.30 per 1M output tokens. The numbers are illustrative.

Automating verification

You can embed a tiny test harness into your CI pipeline that:

1. Calls the Lambda with a known PR number.
2. Records the response time and the CloudWatch metric.
3. Fails the build if the hit ratio falls below, say, 80 % for a month.

aws lambda invoke \
  --function-name pr-review-cache \
  --payload '{"prNumber": 42}' \
  out.txt && cat out.txt

Key takeaway: By watching the CacheHit metric you can quantify the exact latency and token savings, turning an abstract performance claim into a concrete number.

The Takeaway

• Deterministic prompts (fixed prompt + temperature = 0 + max‑tokens) are safe to cache.
• Hash the whole request with SHA‑256; the hash becomes a tiny, stable cache key.
• diagnostics_channel lets you emit hit/miss events at no runtime cost, enabling CloudWatch metrics without extra code paths.
• In‑memory LRU keeps memory bounded and automatically evicts stale entries, perfect for short‑lived Lambda containers.
• Include every setting that influences output (temperature, max_tokens) in the cache key—forgetting them causes silent quality regressions.
• Deploy with care: Node 22’s ES‑module quirks, VPC cold‑start behavior, and SnapStart limitations can hide performance problems if you’re not aware.

With these pieces in place, a Claude‑powered code‑review bot can go from “slow and pricey” to “instant and cheap”, all while giving you clear observability into how often the cache helps. Happy caching!

---

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-22 · Primary focus: NodeJSPerformance

All code blocks are intended to be correct and runnable, but please verify them
against the Node.js docs before using in production.

Find an error? Drop a comment — corrections are always welcome.