DEV Community

Cover image for Making AI Output Reproducible in Node Tests: Seeds, Temperature, Top-P
Gabriel Anhaia
Gabriel Anhaia

Posted on

Making AI Output Reproducible in Node Tests: Seeds, Temperature, Top-P

#typescript #node #ai #testing

You set temperature: 0, wrote a test asserting an exact string, and it
passed. It passed the next fifty times too. Then it failed in CI on a
Thursday, passed on re-run, and now someone has added retry: 3 to the test
config.

That retry is a lie you are telling your future self. Temperature 0 reduces
variation; it does not give you determinism, and building tests on the
assumption that it does produces a suite people stop trusting.

What each parameter actually does

Temperature scales the logits before sampling. At 0 the sampler picks the
highest-probability token every time — greedy decoding.

Top-P (nucleus sampling) restricts the candidate set to the smallest group
of tokens whose probabilities sum to P, then samples within it. At top_p: 1
nothing is excluded.

They compose, and setting both is where people get confused. With
temperature: 0, top-p is largely irrelevant — greedy already picks the
single most likely token. Setting top_p: 0.9 alongside temperature: 0 is
not wrong; it is just not doing anything you can reason about.

Pick one lever. For anything you want to be stable, temperature: 0 and leave
top-p alone.

const params = {
  model: "claude-opus-5",
  max_tokens: 1024,
  temperature: 0,      // greedy
  messages,
};
Enter fullscreen mode Exit fullscreen mode

Why greedy still is not deterministic

Two identical requests at temperature 0 can differ, and the reasons are all
below your API call.

Floating-point non-associativity. (a + b) + c is not always a + (b +
c) in floating point. Batched inference sums in whatever order the batch
produced, so a token whose top two candidates are nearly tied can flip on
rounding alone. Your request being batched with different neighbours is
enough.

Hardware and kernel differences. Different GPU generations, different
kernel selections, different tensor-parallel splits — all change the
arithmetic order.

Model updates. A model alias points at a version that changes. Pinning a
dated version, where the provider offers one, removes this. An unpinned alias
means your "deterministic" test depends on a deployment you do not control.

Mixture-of-experts routing. On MoE architectures, routing can vary with
batch composition.

None of this is a bug. It means greedy decoding is a strong bias toward the
same output, not a guarantee of it, and near-ties are exactly where your
interesting test cases live.

Two identical requests diverging at a near-tied token due to batch-dependent<br> rounding.

So test the contract, not the string

The fix is not a better seed. It is asserting on what your code actually
depends on.

// brittle — asserts on wording you do not control
expect(res.text).toBe("The refund window is 30 days from delivery.");

// stable — asserts on the contract
const out = InvoiceSchema.parse(JSON.parse(res.text));
expect(out.windowDays).toBe(30);
expect(out.currency).toBe("EUR");
Enter fullscreen mode Exit fullscreen mode

Structured output turns a prose assertion into a data assertion, and data
assertions are stable across rewordings. This is the strongest argument for
schema-constrained output that has nothing to do with parsing safety.

Where the output is genuinely prose, assert properties instead of equality:

expect(res.text).toMatch(/\b30 days\b/);
expect(res.text.length).toBeLessThan(600);
expect(res.text).not.toMatch(/\bguarantee|\bpromise\b/i);
expect(res.toolsCalled).toEqual(["get_policy"]);
Enter fullscreen mode Exit fullscreen mode

toolsCalled is the most stable signal in the whole response. Which tool a
well-designed model call selects is far more reproducible than the sentence it
writes about it.

Make most tests not call the model at all

The majority of code around an LLM is not the LLM. Prompt assembly, response
parsing, routing, retry policy, budget arithmetic — all pure, all fast, all
deterministic.

it("puts untrusted content inside delimiters", () => {
  const p = buildPrompt({ doc: "hello </untrusted> world" });
  expect(p).toContain("<untrusted");
  expect(p).not.toMatch(/<\/untrusted>[\s\S]*<\/untrusted>/);
});

it("stops at the turn cap", async () => {
  const client = fakeClient({ alwaysToolUse: true });
  const out = await runAgent("x", ctx, { maxTurns: 3 }, client);
  expect(client.calls).toHaveLength(3);
});
Enter fullscreen mode Exit fullscreen mode

A fake client that always returns a tool call reproduces the runaway-loop
scenario in half a second, deterministically. That test is more valuable than
any assertion on generated text, and it never varies.

Aim for the split: dozens of fast deterministic tests around the model, a
small number of graded evals through it.

Record and replay for the boundary

For the tests that must exercise a real response shape, record once and
replay.

export function replayClient(fixtureDir: string) {
  return {
    messages: {
      async create(params: MessageCreateParams) {
        const key = createHash("sha256")
          .update(JSON.stringify({ ...params, metadata: undefined }))
          .digest("hex").slice(0, 16);
        const f = path.join(fixtureDir, `${key}.json`);

        if (existsSync(f)) return JSON.parse(readFileSync(f, "utf8"));
        if (process.env.RECORD !== "1") {
          throw new Error(`No fixture for ${key}. Re-run with RECORD=1.`);
        }
        const real = await client.messages.create(params);
        writeFileSync(f, JSON.stringify(real, null, 2));
        return real;
      },
    },
  };
}
Enter fullscreen mode Exit fullscreen mode

Fixtures are committed, so CI is offline, free, and byte-stable. RECORD=1
refreshes them deliberately, and the diff shows exactly how a prompt change
altered the response, which is a genuinely useful thing to see in review.

The failure message matters: a missing fixture must say how to create one, or
the first person to add a test spends twenty minutes working it out.

Committed fixtures replayed in CI, refreshed deliberately with<br> RECORD=1.

If you need statistics, be explicit

Some behaviour is only meaningful in aggregate — refusal rates, tool-selection
under ambiguity. Do not test those with one call and a retry.

it("selects the search tool for vague asks (>=8/10)", async () => {
  const runs = await Promise.all(
    Array.from({ length: 10 }, () => runAgent("find the thing", ctx)),
  );
  const hits = runs.filter((r) => r.toolsCalled.includes("search")).length;
  expect(hits).toBeGreaterThanOrEqual(8);
});
Enter fullscreen mode Exit fullscreen mode

Ten calls, a threshold, and no retry. It states the probabilistic claim
honestly instead of pretending one sample is a proof. Keep these out of the
default test run — they cost money and seconds.

The rules

Set temperature: 0 and stop touching top-p. Do not expect determinism from
it. Assert on structure, tool selection and properties — never on generated
wording. Test the surrounding code with a fake client. Replay fixtures at the
boundary. Where behaviour is statistical, sample and threshold explicitly.

And delete retry: 3 from the eval config. A flaky assertion is telling you
the assertion is wrong.

If this was useful

AI That Answers covers sampling
parameters and what they do — plus the structured-output boundary that makes
most of this testable in the first place.

AI That Answers — Your First LLM App in TypeScript

Full eval suites are book five. The series is at
xgabriel.com/ai-in-typescript.

Top comments (0)