Measuring Whether an LLM Gives the Same Answer Twice, in 30 Lines of JavaScript

Ask a model the same question twice and you may get two different lists. If you are testing what an assistant recommends, one run is an anecdote. You need a number for how much the answers repeat, and that number is cheap to compute.

We hit this while running a small study of which business directories one assistant names: 144 answers, 12 business types, 2 prompts, 3 runs each, once from memory and once with web search on. The tool below is the measurement we used on the lists, rewritten as a standalone script.

The measure: Jaccard overlap

Treat each answer as a set of names. The overlap of two answers is the size of what they share divided by the size of everything either one mentioned. Identical lists score 1. Lists with nothing in common score 0. It ignores order, which matters here, since a model reordering the same five names is not really disagreeing with itself.

For three runs of one prompt you get three pairs, so you average them.

The part that is easy to get wrong: names

"Apple Maps" and "Apple Business" point at the same listing. So do "BBB" and "Better Business Bureau". Compare raw strings and you measure how the model phrased things, not what it recommended. The fix is a small alias map, applied before anything is compared.

// How much do two answers to the same prompt agree? Jaccard overlap of the names they mention.
const ALIASES = new Map([
  ['better business bureau', 'bbb'],
  ['apple business', 'apple'],
  ['apple maps', 'apple'],
  ['bing places for business', 'bing places'],
]);

const canon = (name) => {
  const n = name.trim().toLowerCase();
  return ALIASES.get(n) ?? n;
};

const jaccard = (a, b) => {
  const A = new Set(a.map(canon));
  const B = new Set(b.map(canon));
  const both = [...A].filter((x) => B.has(x)).length;
  const either = new Set([...A, ...B]).size;
  return either === 0 ? 1 : both / either;
};

// Every pair of runs of one prompt. Three runs give three pairs.
const pairs = (runs) => runs.flatMap((r, i) => runs.slice(i + 1).map((s) => [r, s]));

const report = (runs) => {
  const scores = pairs(runs).map(([a, b]) => jaccard(a, b));
  const mean = scores.reduce((s, x) => s + x, 0) / scores.length;
  return { pairs: scores.length, mean: Number(mean.toFixed(2)), identical: scores.filter((x) => x === 1).length };
};

// Made-up runs, to show the mechanics. They are not data from our study.
const runs = [
  ['Yelp', 'Apple Maps', 'Bing Places', 'BBB', 'Angi'],
  ['Yelp', 'Apple Business', 'Bing Places for Business', 'Better Business Bureau', 'Angi'],
  ['Yelp', 'Apple Maps', 'Bing Places', 'Thumbtack', 'Nextdoor'],
];

console.log('with aliases   ', report(runs));
ALIASES.clear();
console.log('without aliases', report(runs));

Running it:

with aliases    { pairs: 3, mean: 0.62, identical: 1 }
without aliases { pairs: 3, mean: 0.26, identical: 0 }

Same answers, and the score more than doubles once the names are normalized. The first two runs are the same list in different words. Without the alias map, you would report a model far less consistent than it really is.

What we measured

On the real answers, the average overlap between two runs of the same prompt was 0.81 from memory and 0.63 with web search on. Exact matches were rare: 3 of 72 pairs from memory and 6 of 72 with search named exactly the same directories.

Our guess at the reason, which this data cannot prove: with search on, each run retrieves a different set of pages, and the answer inherits that. In those search runs the results held 652 site appearances from 234 different sites, and 78 of those sites appeared in only one run.

Three limits. It is one model on one day, so it says nothing about other engines. A mean hides per-prompt spread, so keep the per-prompt scores too. And an alias map is a judgement call: merging Apple Maps with Apple Business is defensible, merging two different review sites is not, so write yours down next to the result.

Using this

If you report anything like "the AI recommends X", run the prompt at least three times per setup and publish the overlap with it. A single screenshot cannot tell a stable recommendation from a lucky one.

The same reasoning applies to anyone checking how a site appears in AI answers. Check first that crawlers can read the page at all with an AI visibility checker, since retrieval comes before everything else. Then look at how citations feed AI search, and treat one good answer as one sample.