- Book: The TypeScript Library — the 5-book collection
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
The leaderboard sort your coding agent wrote came with a test file and a green run. The test checks three arrays, and all three pass.
// src/scores.ts
export function sortScores(xs: number[]): number[] {
return [...xs].sort();
}
// src/scores.test.ts
import { expect, it } from 'vitest';
import { sortScores } from './scores';
it('sorts scores', () => {
expect(sortScores([3, 1, 2])).toEqual([1, 2, 3]);
expect(sortScores([])).toEqual([]);
expect(sortScores([5, 5, 1])).toEqual([1, 5, 5]);
});
The function is wrong. Array.prototype.sort() with no comparator converts the elements to strings and compares them as strings (MDN), so [10, 9] stays [10, 9]. Every input in the test is a single digit, and with single digits string order and number order agree.
The agent didn't cheat. It wrote the code and the examples from the same idea of the problem, so the examples sit inside the part it already understood. A human does the same thing. The fix is to stop picking inputs yourself and describe what must be true for every input. Then a library generates the inputs.
That library in TypeScript is fast-check, and it plugs into Vitest.
Set up fast-check in Vitest
The Vitest integration is a separate package, @fast-check/vitest. Install it next to Vitest:
npm install --save-dev vitest @fast-check/vitest
At the time of writing, npm lists fast-check 4.10.2 and @fast-check/vitest 0.5.0, which declares Vitest ^4.1.0 || ^5.0.0 as its peer. The package gives you a test.prop that takes a list of arbitraries (input generators) and then a normal test body. Each generated value arrives as an argument.
import { expect } from 'vitest';
import { fc, test } from '@fast-check/vitest';
import { sortScores } from './scores';
test.prop([fc.array(fc.integer())])(
'output is ordered and keeps its length',
(xs) => {
const out = sortScores(xs);
expect(out).toHaveLength(xs.length);
for (let i = 1; i < out.length; i++) {
expect(out[i - 1]).toBeLessThanOrEqual(out[i]);
}
},
);
A property fails when its body throws, so plain expect calls work. Returning false also counts as a failure (fast-check docs). By default fast-check runs each property 100 times, with a seed taken from Date.now().
Three examples you picked, against a thousand inputs nobody picked.
Three properties worth asking for
You rarely know the exact output for a random input. You usually know something that must hold for every one of them. Three shapes cover most code an agent writes.
Invariant. Something about the output is always true. The property above is one: same length as the input, and each element no larger than the next, with no expected array anywhere in the test.
Round-trip. Two functions undo each other. Say the agent also wrote opaque pagination cursors:
// src/cursor.ts
export type Cursor = { id: number; after: string };
export function encodeCursor(c: Cursor): string {
const json = JSON.stringify(c);
return Buffer.from(json, 'utf8').toString('base64url');
}
export function decodeCursor(s: string): Cursor {
const json = Buffer.from(s, 'base64url').toString('utf8');
return JSON.parse(json) as Cursor;
}
The property: decode what you encoded and you get the input back.
const cursor = fc.record({
id: fc.nat(),
after: fc.string({ unit: 'grapheme' }),
});
test.prop([cursor])('decode undoes encode', (c) => {
expect(decodeCursor(encodeCursor(c))).toEqual(c);
});
Round-trips are the cheapest property to write and they catch encoding bugs that examples skip, because nobody types awkward strings into a hand-written test. The arbitrary does, once you widen it: fc.string() defaults to printable ASCII (string docs), and unit: 'grapheme' brings in the rest of Unicode.
Oracle. You have a slow, obvious implementation to compare against. The agent's leap-year check passes 2024, 2023 and 2020:
export function isLeapYear(y: number): boolean {
return y % 4 === 0;
}
JavaScript's own calendar can be the oracle. February 29 in a non-leap year rolls over to March 1:
const reallyLeap = (y: number) =>
new Date(Date.UTC(y, 1, 29)).getUTCDate() === 29;
test.prop(
[fc.integer({ min: 1970, max: 2400 })],
{ numRuns: 1000 },
)('agrees with the calendar', (y) => {
expect(isLeapYear(y)).toBe(reallyLeap(y));
});
The range starts at 1970 because Date.UTC maps years 0 to 99 onto 1900 to 1999 (MDN), and an oracle with a quirk of its own is a bad oracle. In that range only three years break the agent's rule: 2100, 2200 and 2300. If inputs were drawn uniformly, 100 runs would miss all three about half the time, since (428/431)^100 is roughly 0.5. At 1,000 runs the miss chance drops below 0.1%. fast-check doesn't draw uniformly, so treat those odds as a rough guide, but the point holds: a rare failure needs more runs, and numRuns is how you ask for them.
Shrinking: from a random mess to [2, 1000000000]
Run the invariant against the string-sorting sortScores and it fails. The first failing array fast-check finds is long and noisy. It doesn't stop there. It shrinks: it keeps trying smaller and simpler versions of the input, and keeps any that still fail.
The fast-check docs walk through a numeric sort checked against this same property, and show the report it produced:
Error: Property failed after 1 tests
{ seed: -1819918769, path: "0:...:3", endOnFailure: true }
Counterexample: [[2,1000000000]]
Shrunk 66 time(s)
Got error: AssertionError: expected 1000000000 to be less
than or equal to 2
Your seed, path and shrink count will differ. The shape of the answer usually won't: two numbers whose string order and number order disagree. Sixty-six shrink steps turned a random array into a test case you can read in a second. A failing property hands you the smallest input fast-check could find that still breaks the code.
Pin the seed, then let it go
A random failure you can't reproduce is worse than no test. The report gives you everything you need to replay it. The seed regenerates the same inputs. The path points at the failing case, so fast-check can jump straight to the shrunk counterexample. In a real report the path is a full string of numbers; the docs show it shortened as 0:...:3. And endOnFailure: true stops the run at the first failure instead of shrinking again.
While you debug, paste the seed into the test's options:
test.prop([fc.array(fc.integer())], { seed: -1819918769 })(
'output is ordered and keeps its length',
(xs) => {
// same body as before
},
);
The docs suggest the full { seed, path, endOnFailure } object to skip straight to the counterexample, and say to drop path and endOnFailure if you want the whole run back, shrinking included (read-test-reports).
Once the bug is fixed, take the seed out. A pinned seed runs the same inputs forever, which turns the property back into a fixed set of examples. Keep the counterexample as a plain regression test instead:
it('regression: sorts by number, not by string', () => {
// the string sort returned [1000000000, 2]
expect(sortScores([2, 1000000000])).toEqual([2, 1000000000]);
});
And the fix itself is one comparator: [...xs].sort((a, b) => a - b).
To raise the run count everywhere, use fc.configureGlobal in a file listed in Vitest's setupFiles. It replaces the whole global configuration, so the docs show merging with readConfigureGlobal:
// test/fc-setup.ts
import { fc } from '@fast-check/vitest';
fc.configureGlobal({
...fc.readConfigureGlobal(),
numRuns: Number(process.env.FC_RUNS ?? 100),
});
// vitest.config.ts
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: { setupFiles: ['./test/fc-setup.ts'] },
});
Keep 100 on your laptop for fast feedback and run FC_RUNS=1000 npx vitest run in CI, where a slower suite costs nothing but minutes. Every CI run gets a fresh seed, so the suite keeps exploring new inputs instead of replaying old ones.
Make the agent write the properties, in its own worktree
A property is a short statement about behaviour, so it is a reasonable thing to hand an agent. Ask for it in a separate step, so the properties come from the contract and not from the code it just wrote. A starting prompt to adapt:
Add fast-check properties for src/scores.ts and
src/cursor.ts with @fast-check/vitest. For each function
write at least one invariant, round-trip or oracle
property. Do not edit anything in src/ except test files.
If a property fails, stop and show me the counterexample.
The last line matters. Faced with a failing property, an agent can just as easily loosen the property as fix the code. You want the counterexample in front of you before either happens.
I build Hermes IDE, so weigh this part with that in mind. ⌘N (Ctrl+Shift+N on Windows and Linux) starts the task on its own hermes/ branch in its own worktree, with the prompt as the agent's first message (README). The agent can run 1,000 cases per property in that worktree while your own checkout stays as it was.
The ⌘N New task sheet: agent, project, a new hermes/ worktree, approval mode, model and effort, and a preview of the command Hermes will run.
Hermes saves every agent turn, and the Review Desk (⌘G, or Ctrl+Shift+G) shows the changes by turn or by file. If the agent wrote the function in one turn and the properties in the next, you read them apart: what it built, then what it claims about what it built. A line comment on the property turn goes back to the agent that wrote it. If you'd rather not add a tool, git log -p on the task branch shows the same split, one commit at a time, as long as the agent commits between steps.
The Review Desk in By turn view: two turns listed, turn 1's diff, the changed files and a commit message drafted from the turns.
Hermes is source-available under BSL 1.1, runs on macOS, Windows and Linux, and works with Claude Code, Codex and the other CLI agents with their own sign-ins. Download it here. The properties themselves need nothing but Vitest and fast-check.
If this was useful
Property tests lean on the type system more than example tests do: the arbitrary for a Cursor is the type, written as a generator, and a sharper type means fewer impossible inputs to filter out. That is the kind of TypeScript the collection is about. Start with the first two books if TypeScript is your main language. Pick the bridge book if you're coming from the JVM or PHP. Grab TypeScript in Production if you ship a TS library or monorepo at work.
- TypeScript Essentials — types, narrowing, modules, async, daily-driver tooling
- The TypeScript Type System — generics, mapped/conditional types, infer, template literals, branded types
- Kotlin and Java to TypeScript — JVM developers moving to TypeScript
- PHP to TypeScript — PHP 8+ developers moving to TypeScript
- TypeScript in Production — tsconfig, build tools, monorepos, library authoring, dual ESM/CJS, JSR
If you run more than one coding agent on a repo, Hermes IDE gives each task its own worktree and shows its work turn by turn.
All five books ship in ebook, paperback, and hardcover.
Top comments (1)
The [10, 9] sort bug is a clean catch, since single digit tests hide string ordering. But 1,000 random runs still miss the point if the generator never produces two digit numbers, so the arbitrary matters more than the run count.