Prompt Testing Pipelines with SQS: How to Version, Run, and Verify LLM Prompts Like Unit Tests

Developers treat prompts like magic strings, but that secrecy breeds hidden bugs. By wiring prompts into an SQS‑backed queue and running them through automated tests, you get repeatable, debuggable AI behavior. This post shows you how to turn prompt tweaking into a CI‑friendly workflow.

---

Why Prompt Testing Matters

When you ask an LLM (large language model) a question, you are sending a prompt – a piece of text that tells the model what you want. If the prompt changes even slightly, the answer can shift dramatically. In a codebase that ships features daily, an unnoticed prompt change can cause a UI glitch, a wrong calculation, or a compliance breach.

Treating prompts like any other source file gives you three safety nets:

1. Version control – you can see who changed what and why.
2. Automated regression checks – tests fail the moment a prompt produces an unexpected format.
3. Fast feedback – CI (continuous integration) tells you immediately whether a new prompt version is safe to merge.

In plain English: Think of a prompt as a recipe. If you forget to add “salt,” the dish tastes off. Unit tests are like a taste‑tester who samples every new batch before it reaches the restaurant.

The simplest test scenario

We will build a tiny test harness that:

• Pulls a JSON payload from an Amazon SQS queue (a managed message queue service).
• Calls Claude (Anthropic’s LLM) with the prompt inside the payload.
• Validates the response shape using Zod (a TypeScript schema validator).
• Exits with process.exit(0) for success or process.exit(1) for failure, which CodeBuild can interpret as a pass/fail result.

The same harness can be reused for dozens of prompts, each stored as a separate SQS message.

---

Storing Prompt Versions in SQS

Why a queue, not a plain file?

A queue gives you decoupling – the code that writes prompts does not need to know when the tests run. It also provides visibility timeout, a safety valve that prevents a message from being processed twice if a worker crashes.

Imagine a librarian (the queue) handing out a book (the prompt) to a reader (the Lambda). The book stays marked as “checked out” for a set period (visibility timeout). If the reader never returns it, the librarian puts it back on the shelf for the next person.

Setting up the queue

We will use a FIFO (first‑in‑first‑out) queue because ordering matters when you want to test prompts in a specific sequence.

// sqs-setup.ts
import { SQSClient, CreateQueueCommand } from "@aws-sdk/client-sqs";

// Create an SQS client that talks to the default region
const sqs = new SQSClient({});

// Parameters for a FIFO queue
const params = {
  QueueName: "prompt-test.fifo",          // ".fifo" suffix tells SQS this is FIFO
  Attributes: {
    FifoQueue: "true",                    // Enables FIFO behavior
    ContentBasedDeduplication: "false",   // We'll provide our own deduplication ID
    VisibilityTimeout: "30",              // Seconds a message stays hidden after being read
    ReceiveMessageWaitTimeSeconds: "20",  // Long polling (wait up to 20 s for a message)
  },
};

async function createQueue() {
  const command = new CreateQueueCommand(params);
  const response = await sqs.send(command);
  console.log("Queue URL:", response.QueueUrl);
}

createQueue().catch(console.error);

• SQSClient – the object that talks to the SQS API.
• CreateQueueCommand – tells SQS to make a new queue with the given attributes.

Tip: The default visibility timeout (30 s) must be longer than the longest LLM call you expect, otherwise the same message may be delivered a second time and your test will appear flaky.

Adding a prompt version

Each message contains a JSON body with three fields: id, prompt, and expectedSchema.

// enqueue-prompt.ts
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({});

const promptMessage = {
  id: "v1.2.0",                     // Human‑readable version tag
  prompt: "Summarize the following article in two sentences:\n{{article}}",
  // Zod schema expressed as a string for later reconstruction
  expectedSchema: `{ type: "object", properties: { summary: { type: "string" } }, required: ["summary"] }`,
};

async function sendPrompt() {
  const command = new SendMessageCommand({
    QueueUrl: "https://sqs.us-east-1.amazonaws.com/123456789012/prompt-test.fifo",
    MessageBody: JSON.stringify(promptMessage),
    MessageGroupId: "prompt-tests",               // Required for FIFO queues
    MessageDeduplicationId: promptMessage.id,    // Guarantees exactly‑once delivery
  });

  const response = await sqs.send(command);
  console.log("Message sent, ID:", response.MessageId);
}

sendPrompt().catch(console.error);

• MessageGroupId – groups related messages so they are processed in order.
• MessageDeduplicationId – prevents the same prompt version from being enqueued twice within a 5‑minute window.

Key takeaway: By putting each prompt version into a FIFO queue with a unique deduplication ID, you get a reliable source of truth that CI can pull from at any time.

---

Automating Prompt Execution with Lambda

Why Lambda?

AWS Lambda is a serverless compute service that runs code in response to events – in our case, an SQS message arrival. It scales automatically, incurs cost only while running, and integrates directly with SQS (no polling code required).

Lambda handler skeleton

// handler.ts
import {
  SQSHandler,
  SQSEvent,
  SQSRecord,
} from "aws-lambda";
import { Anthropic } from "@anthropic-ai/sdk";
import { z } from "zod";

// Initialise the Anthropic client once per container reuse
const client = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY, // Keep the key out of code, inject via env var
});

/**
 * Reads the prompt, calls Claude, validates the response.
 * Returns a boolean indicating success – the Lambda runtime will
 * exit with 0 (success) or throw to signal failure.
 */
export const handler: SQSHandler = async (event: SQSEvent) => {
  for (const record of event.Records) {
    const payload = JSON.parse(record.body);
    const { id, prompt, expectedSchema } = payload;

    // Reconstruct the Zod schema from the stored string
    const schema = z.object(JSON.parse(expectedSchema));

    // Replace placeholder with dummy data (in real life you may fetch fixtures)
    const filledPrompt = prompt.replace("{{article}}", "AI is changing the world.");

    // Call Claude – the LLM request may take several seconds
    const response = await client.completions.create({
      model: "claude-3-5-sonnet-20241022", // Example model name
      max_tokens: 256,
      prompt: filledPrompt,
    });

    // The LLM returns a JSON string in `completion`. Parse it.
    const parsed = JSON.parse(response.completion);

    // Validate using Zod – throws if shape does not match
    try {
      schema.parse(parsed);
      console.log(`✅ Prompt ${id} passed validation`);
    } catch (e) {
      console.error(`❌ Prompt ${id} failed validation`, e);
      // Throwing makes Lambda return an error, which CodeBuild interprets as test failure
      throw new Error(`Prompt ${id} validation error`);
    }
  }
};

• SQSHandler – a type that tells Lambda the event source is SQS.
• Anthropic SDK – official client library for Claude.

In plain English: Think of the Lambda as a kitchen robot that receives a recipe (the prompt), cooks the dish (calls the LLM), and then checks the plating against a picture (the Zod schema). If the plating is off, the robot raises an alarm.

Guarding against duplicate processing

If the Lambda times out before it finishes the LLM call, SQS will make the message visible again, causing a second Lambda invocation. To avoid flaky results:

• Set the Lambda timeout (Timeout setting) to greater than the SQS visibility timeout plus a safety margin (e.g., 45 s vs 30 s).
• Or, increase the SQS visibility timeout to exceed the worst‑case LLM latency (you can do this per‑message with ChangeMessageVisibility).

Tip: Adding a tiny deduplication token (the id we already have) to a DynamoDB table at the start of processing can give you an extra “once‑only” guarantee, but the visibility timeout adjustment is usually enough for CI runs.

---

Asserting LLM Outputs with Node:test

Why a test framework?

Node.js ships with a built‑in test runner (node:test) that works without extra dependencies. It provides assertions, a clear output format, and can be invoked directly from CodeBuild.

Minimal test file

// prompt.test.ts
import { test, describe, after } from "node:test";
import { strict as assert } from "assert";
import { spawn } from "child_process";

/**
 * Spawns the Lambda handler locally (using ts-node) with a single SQS message.
 * The child process exits with 0 on success, 1 on failure – we turn that into
 * a test assertion.
 */
function runPromptTest(messageFile: string): Promise<void> {
  return new Promise((resolve, reject) => {
    const child = spawn("ts-node", ["handler.ts"], {
      env: {
        ...process.env,
        // Simulate an SQS event by passing the JSON payload as STDIN
        // In a real Lambda the event comes from SQS automatically.
        SQS_EVENT: JSON.stringify({
          Records: [
            {
              body: require("fs").readFileSync(messageFile, "utf-8"),
              messageId: "test",
              receiptHandle: "test",
              attributes: {},
              messageAttributes: {},
              md5OfBody: "",
              eventSource: "aws:sqs",
              eventSourceARN: "arn:aws:sqs:us-east-1:123456789012:prompt-test.fifo",
              awsRegion: "us-east-1",
            },
          ],
        } as any),
      },
    });

    child.on("close", (code) => {
      if (code === 0) resolve();
      else reject(new Error(`Lambda exited with code ${code}`));
    });
  });
}

describe("Prompt regression suite", async () => {
  // Example: test the version we just enqueued
  await test("v1.2.0 returns a summary object", async () => {
    await runPromptTest("./sample-message.json");
  });
});

after(() => {
  console.log("All prompt tests finished");
});

• spawn – launches a new process that runs the handler as if Lambda were invoking it.
• SQS_EVENT – we fake the SQS event payload because node:test runs locally, not in AWS.

Running node --test prompt.test.ts will produce a clear pass/fail output that mirrors what CodeBuild expects.

Key takeaway: Using the native node:test runner means you don’t need a heavyweight testing library; the output integrates nicely with CI dashboards.

---

Integrating into CodeBuild Pipelines

Why CodeBuild?

AWS CodeBuild is a fully managed build service that compiles source, runs tests, and produces artifacts. It can be triggered by a Git push, a pull‑request, or a manual start, making it a natural place to execute our prompt test suite.

CodeBuild project definition (YAML)

# buildspec.yml
version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 20
    commands:
      - echo "Installing dependencies"
      - npm ci                         # Clean install from lockfile
      - echo "Installing AWS SDKs"
      - npm install @aws-sdk/client-sqs @aws-sdk/client-codebuild @anthropic-ai/sdk zod
  pre_build:
    commands:
      - echo "Fetching prompt messages from SQS"
      # Pull a single message for demo purposes; in production you may loop
      - node fetch-message.js > sample-message.json
  build:
    commands:
      - echo "Running prompt regression tests"
      - node --test prompt.test.ts     # Exits 0 on success, 1 on failure
  post_build:
    commands:
      - echo "Tests completed"
artifacts:
  files:
    - '**/*'
    - '!node_modules/**'   # Do not upload the whole node_modules folder

• install – sets up Node.js 20 (the LTS version at the time of writing) and installs the SDKs we need.
• pre_build – runs a tiny script (fetch-message.js) that calls receiveMessage on the SQS queue and writes the JSON to a file.
• build – executes the test runner; a non‑zero exit code marks the build as failed.

Handling environment secrets

CodeBuild environment variables are visible in logs unless you mark them as plain‑text / parameter store secrets. Store the Anthropic API key in AWS Secrets Manager and reference it in the build spec:

env:
  secrets-manager:
    ANTHROPIC_API_KEY: "my/anthropic/api-key"

Now the key never appears in plain text.

Tip: If your CodeBuild project runs inside a VPC, remember that it has no internet access by default. Attach a NAT Gateway or VPC endpoint for the Anthropic API, otherwise the LLM call will time out.

Dealing with batch failures

When you receive messages in batches (MaxNumberOfMessages up to 10), SQS treats the batch as all‑or‑nothing by default. If one message causes a Lambda error, the entire batch is returned to the queue, possibly causing a backlog. In a test pipeline you usually process a single message at a time, which avoids this complexity.

In plain English: Think of batch processing like a school bus: if one kid is sick, the whole bus goes back to the depot. Processing one kid per ride means the bus never has to turn around.

---

The Takeaway

Key takeaways

• Store each prompt version as a JSON message in an SQS FIFO queue; the queue becomes a single source of truth for CI.
• Use a Lambda function to read the message, call Claude via the Anthropic SDK, and validate the response with a Zod schema.
• Guard against duplicate processing by ensuring the Lambda timeout exceeds the SQS visibility timeout, or by extending the visibility timeout dynamically.
• Run the Lambda locally with node:test to get fast, expressive test results that CodeBuild can interpret as pass/fail.
• Wire the whole flow into a CodeBuild buildspec: install dependencies, fetch a message, run tests, and mask secrets securely.
• Remember the hidden gotchas: visibility timeout vs. Lambda timeout, FIFO TPS limits, long‑polling costs, and VPC networking for outbound LLM calls.

By treating prompts the same way you treat code—versioned, tested, and reviewed—you eliminate guesswork, catch regressions early, and give your team confidence that the AI part of your product behaves predictably. Happy testing!

---

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-09-23 · Primary focus: SQS

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.