API Gateway HTTP API + Lambda Function URLs: Building a Prompt‑Caching LLM Proxy, Explained Simply

LLM‑powered features like code completion are often wrapped in heavyweight Lambda functions that add latency and cost. By letting API Gateway’s HTTP API route directly to a lightweight Function URL and adding a tiny in‑process cache, you get sub‑100 ms responses for repeated prompts. This post shows exactly how to wire it together with Node.js 22’s native fetch and diagnostics_channel.

Why Prompt Caching Matters for LLM Proxies

When a user asks an LLM (large language model) the same question twice, the model does the same heavy computation both times. That work costs money and adds time. A prompt cache stores the model’s answer for a given prompt so the second request can be answered instantly, without contacting the model again.

In plain English: Think of a cache like a sticky note you put on your desk. The first time you look up a phone number you write it down; the next time you need it, you just glance at the note instead of dialing the operator again.

Why keep the cache inside the Lambda instead of an external store?

• Speed – a Map object lives in the same process, so a lookup is a few nanoseconds.
• Cost – no extra DynamoDB reads or Redis traffic.
• Simplicity – no extra IAM policies, no VPC endpoints.

The trade‑off is that the cache disappears when the Lambda container is recycled, but for high‑frequency prompts that’s fine; the next cold start will rebuild the cache quickly.

Quick code sketch

// A very small in‑process cache.
// Keys are SHA‑256 hashes of the prompt string.
const cache = new Map<string, string>();

Key takeaway: Caching at the function level can shave tens of milliseconds off every repeat request, and you don’t need a separate service to make it happen.

---

Setting Up an HTTP API with a Lambda Function URL

API Gateway offers two flavors: REST API (v1) and HTTP API (v2). The latter is lighter, cheaper, and supports Function URLs – a direct HTTPS endpoint that invokes a Lambda without the extra integration layer.

Why use a Function URL instead of the classic “Lambda integration”?

• Less latency – the request goes straight to the Lambda, no extra JSON mapping.
• Simpler permissions – you only need to allow the API’s principal to call the URL.
• Lower cost – HTTP APIs are priced per million requests, while REST APIs add a per‑request surcharge.

Step‑by‑step creation with the AWS SDK

Below is a minimal script that creates an HTTP API, a route /prompt, and ties it to a Lambda Function URL. It also shows the binary‑support flag that prevents the 6 KB body‑drop bug.

import {
  ApiGatewayV2Client,
  CreateApiCommand,
  CreateIntegrationCommand,
  CreateRouteCommand,
  UpdateApiCommand,
} from "@aws-sdk/client-api-gateway";

// Replace with your region and Lambda ARN
const REGION = "us-east-1";
const LAMBDA_URL = "https://abcde12345.lambda-url.us-east-1.on.aws";

const client = new ApiGatewayV2Client({ region: REGION });

async function createHttpApi() {
  // 1️⃣ Create the HTTP API
  const apiResult = await client.send(
    new CreateApiCommand({
      Name: "PromptCacheAPI",
      ProtocolType: "HTTP",
      // Enable binary support so bodies >6 KB are not stripped
      // (the list of supported media types must include application/json)
      BinaryMediaTypes: ["application/json"],
    })
  );

  const apiId = apiResult.ApiId!;
  console.log(`API created: ${apiId}`);

  // 2️⃣ Create an integration that points to the Lambda Function URL
  const integrationResult = await client.send(
    new CreateIntegrationCommand({
      ApiId: apiId,
      IntegrationType: "HTTP_PROXY", // direct HTTP call
      IntegrationUri: LAMBDA_URL,
      PayloadFormatVersion: "2.0",
    })
  );

  const integrationId = integrationResult.IntegrationId!;
  console.log(`Integration created: ${integrationId}`);

  // 3️⃣ Create a route that listens on POST /prompt
  await client.send(
    new CreateRouteCommand({
      ApiId: apiId,
      RouteKey: "POST /prompt",
      Target: `integrations/${integrationId}`,
    })
  );

  // 4️⃣ (Optional) Enable CORS – the most common support ticket
  await client.send(
    new UpdateApiCommand({
      ApiId: apiId,
      CorsConfiguration: {
        AllowOrigins: ["*"],
        AllowMethods: ["POST"],
        AllowHeaders: ["Content-Type", "Authorization"],
      },
    })
  );

  console.log(`Route /prompt ready at https://${apiId}.execute-api.${REGION}.amazonaws.com/prompt`);
}

createHttpApi().catch((e) => console.error(e));

What the script does

1. CreateApiCommand – spins up an HTTP API and tells API Gateway to treat application/json as binary. This avoids the hidden bug where payloads larger than 6 KB are silently dropped before they reach the Lambda.
2. CreateIntegrationCommand – tells the API to forward the request to the Lambda Function URL (HTTP_PROXY mode means “just forward the request as‑is”).
3. CreateRouteCommand – maps the HTTP verb + path (POST /prompt) to the integration.
4. UpdateApiCommand – adds a permissive CORS (Cross‑Origin Resource Sharing) configuration, which is the #1 support question teams run into.

Tip: After creating the API, give the API’s execution role permission to invoke the Lambda Function URL (lambda:InvokeFunctionUrl) and set the URL’s auth type to NONE or AWS_IAM depending on your security model.

---

Implementing a Tiny In‑Process Cache in Node.js 22

Now that the front‑door is ready, let’s write the Lambda handler. Node.js 22 ships with the fetch API built‑in, so we don’t need any extra HTTP client library.

Why store the cache in a Map rather than a plain object?

• Deterministic iteration order – useful when you later want to add LRU (least‑recently‑used) eviction.
• Better key handling – Map lets you use any type as a key; we’ll use a string hash.

Full handler code

import { createHash } from "crypto";
import { fetch } from "node-fetch"; // native in Node 22, import for clarity
import { diagnosticChannel } from "diagnostics_channel";

/**
 * Simple SHA‑256 hash of the prompt. Using a cryptographic hash
 * guarantees a fixed‑length key regardless of prompt size.
 */
function hashPrompt(prompt) {
  return createHash("sha256").update(prompt, "utf8").digest("hex");
}

/**
 * In‑process cache: Map<hash, string>. The value is the complete
 * Server‑Sent Events (SSE) payload we received from Claude.
 */
const cache = new Map();

/**
 * Exponential backoff helper for retrying 429 (Too Many Requests) errors.
 */
async function retryWithBackoff(fn, attempts = 4, delay = 200) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (err.status !== 429) throw err; // only handle throttling
      const wait = delay * 2 ** i;
      console.warn(`Got 429, retrying in ${wait}ms`);
      await new Promise((r) => setTimeout(r, wait));
    }
  }
  throw new Error("Exceeded retry attempts");
}

/**
 * Lambda entry point – receives an APIGatewayProxyEvent (HTTP API payload)
 * and returns a ProxyResult (HTTP response).
 */
export async function handler(event) {
  // -------------------------------------------------
  // 1️⃣ Parse and validate incoming JSON
  // -------------------------------------------------
  let body;
  try {
    body = JSON.parse(event.body);
    if (typeof body.prompt !== "string") throw new Error("Missing prompt");
  } catch (e) {
    return { statusCode: 400, body: JSON.stringify({ error: "Invalid JSON" }) };
  }

  const prompt = body.prompt.trim();
  const key = hashPrompt(prompt);

  // -------------------------------------------------
  // 2️⃣ Check the in‑process cache
  // -------------------------------------------------
  if (cache.has(key)) {
    diagnosticChannel.publish("promptCache.hit", { key });
    return {
      statusCode: 200,
      headers: { "Content-Type": "text/event-stream" },
      body: cache.get(key),
    };
  }

  diagnosticChannel.publish("promptCache.miss", { key });

  // -------------------------------------------------
  // 3️⃣ Call Claude via Bedrock using native fetch
  // -------------------------------------------------
  const region = process.env.AWS_REGION || "us-east-1";
  const bedrockUrl = `https://bedrock-runtime.${region}.amazonaws.com/model/anthropic.claude-v2/invoke`;

  const requestBody = JSON.stringify({
    prompt,
    max_tokens_to_sample: 256,
    temperature: 0.7,
    stream: true, // ask Bedrock to stream SSE
  });

  // The retry wrapper only retries on 429.
  const response = await retryWithBackoff(() =>
    fetch(bedrockUrl, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        // The SDK would normally sign this request; for brevity we assume
        // the Lambda execution role has the needed Bedrock permissions.
      },
      body: requestBody,
    })
  );

  if (!response.ok) {
    const errText = await response.text();
    console.error(`Bedrock error ${response.status}: ${errText}`);
    return { statusCode: 502, body: JSON.stringify({ error: "Upstream failure" }) };
  }

  // -------------------------------------------------
  // 4️⃣ Pipe the streaming SSE back to the client
  // -------------------------------------------------
  // We accumulate the stream into a string so we can cache it.
  let ssePayload = "";
  const decoder = new TextDecoder();

  // The fetch response body is a Node.js ReadableStream.
  for await (const chunk of response.body) {
    const text = decoder.decode(chunk);
    ssePayload += text; // store for caching
    // Immediately forward to the API Gateway client.
    // API Gateway HTTP API expects the full response body at the end,
    // but we can still build it incrementally.
  }

  // Save to cache for future identical prompts.
  cache.set(key, ssePayload);
  diagnosticChannel.publish("promptCache.store", { key });

  // Return as Server‑Sent Events (SSE) content type.
  return {
    statusCode: 200,
    headers: { "Content-Type": "text/event-stream" },
    body: ssePayload,
  };
}

Explanation of key lines

Line	Reason
hashPrompt	Turns any length prompt into a short, fixed‑size key.
cache.has(key)	Quick lookup to decide if we can skip the external call.
retryWithBackoff	Handles the common Bedrock throttling response (429) by waiting longer each attempt.
fetch(..., { stream: true })	Asks Bedrock to stream its answer as SSE, which we then forward.
diagnosticChannel.publish	Emits lightweight events that other tools can listen to (covered later).
cache.set(key, ssePayload)	Stores the whole SSE payload for the next identical request.

In plain English: The handler first looks in its own notebook (the Map). If the answer is already written down, it reads it out immediately. If not, it asks Claude, writes the spoken answer into the notebook, and then hands the answer back to the caller.

Gotchas you might hit

• Binary payload loss – if you forget to set BinaryMediaTypes to include application/json, prompts larger than 6 KB will be truncated before they reach this code.
• CORS errors – make sure the API’s CORS config lists POST and Content-Type header, otherwise browsers block the request.
• Cold start after VPC – if your Lambda sits in a VPC, the container start time dominates; SnapStart gives no benefit in that case.

---

Streaming Claude Responses with Native fetch

Claude (the Anthropic model) can return a stream of tokens using Server‑Sent Events (SSE). Streaming lets the client start displaying text as soon as the first token arrives, which feels snappier than waiting for the whole response.

Why use native fetch instead of the AWS SDK’s InvokeModelCommand?

• No extra dependency – fetch is already part of Node 22.
• Full control over headers – we can set Accept: text/event-stream easily.
• Simpler streaming loop – the response body is an async iterable, perfect for for await … of.

Minimal example of consuming the SSE stream

async function streamClaude(prompt) {
  const url = `https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-v2/invoke`;
  const res = await fetch(url, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt, stream: true }),
  });

  // Guard against non‑200 responses early.
  if (!res.ok) throw new Error(`Bedrock returned ${res.status}`);

  // The body is an async iterator of Uint8Array chunks.
  const decoder = new TextDecoder();
  for await (const chunk of res.body) {
    const text = decoder.decode(chunk);
    // Each chunk may contain one or more SSE lines like:
    //   data: {"completion":"Hello"}
    console.log("Received chunk:", text);
    // Forward the raw text to the HTTP client (or accumulate).
  }
}

Key points

• res.body implements the AsyncIterable interface, meaning you can for await … of it directly.
• Each chunk is a Uint8Array; converting with TextDecoder gives the human‑readable SSE line.
• Because we set stream: true, Bedrock never buffers the whole answer; it pushes each token as soon as it’s generated.

Analogy: Think of a live sports broadcast. Instead of waiting for the entire game to finish before you hear the result, the commentator describes the play as it happens. Streaming works the same way for LLM output.

Handling 429 throttling with exponential backoff

Bedrock enforces a request‑per‑second quota per account. When you exceed it, the service returns HTTP 429. The helper retryWithBackoff from the previous section automatically retries with increasing delays (200 ms → 400 ms → 800 ms …). This pattern prevents your Lambda from failing outright while still respecting the service limits.

---

Observability with diagnostics_channel

diagnostics_channel is a lightweight Node.js feature that lets you emit custom events without pulling in a full‑blown tracing library. It works well inside Lambda because it adds virtually no latency and doesn’t need extra IAM permissions.

Why instrument the cache with diagnostics?

• Visibility – you can see cache hits vs. misses in CloudWatch Logs or a downstream collector.
• Low overhead – events are just objects passed to listeners; if no listener is attached, the call is a no‑op.
• Future‑proof – you can later attach an OpenTelemetry collector without touching the business logic.

Wiring up listeners

import { diagnosticChannel } from "diagnostics_channel";

// Listener that logs cache activity – you could replace this with a
// structured logger or a metric exporter.
diagnosticChannel.subscribe("promptCache.hit", (info) => {
  console.info(`[Cache] HIT for ${info.key}`);
});

diagnosticChannel.subscribe("promptCache.miss", (info) => {
  console.info(`[Cache] MISS for ${info.key}`);
});

diagnosticChannel.subscribe("promptCache.store", (info) => {
  console.info(`[Cache] STORED result for ${info.key}`);
});

Where to place the subscription?

Put the listener code in a separate module that the Lambda imports early (e.g., observability.js). Because the subscription runs at module load time, it executes once per container, not per invocation.

Tip: If you enable API Gateway access logs, you’ll see the same request IDs printed by both the API layer and your diagnostics events, making it easy to correlate end‑to‑end latency.

Seeing the data in CloudWatch

When the Lambda runs, the console.info statements appear in the function’s CloudWatch log stream. You can then create a Metric Filter that turns lines like [Cache] HIT* into a custom CloudWatch metric (PromptCacheHit). Plotting Hit vs. Miss over time shows you how effective the cache is.

---

The Takeaway

The core ideas that let you shrink LLM proxy latency.

• Function URLs + HTTP API give a direct, low‑overhead path to Lambda, avoiding the extra mapping layer that a classic REST API adds.
• Binary media support must be enabled for application/json payloads larger than 6 KB; otherwise the prompt never reaches the model.
• In‑process Map cache is enough to serve repeated prompts in under 100 ms, and it requires no extra AWS resources.
• Node.js 22’s native fetch makes streaming responses from Claude simple; each chunk can be piped straight to the client as Server‑Sent Events.
• Retry with exponential backoff gracefully handles Bedrock’s 429 throttling without exploding your Lambda’s error count.
• diagnostics_channel provides zero‑cost observability; you can turn cache activity into CloudWatch metrics or feed it into a tracing system later.

By combining these pieces, you can build a lean LLM proxy that feels fast, stays cheap, and remains easy to monitor. Happy coding!

---

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-10-12 · Primary focus: APIGateway

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.