How to Test a Randomized Web Experience Without Writing Flaky Tests

This post focuses on the testing techniques behind randomized interfaces rather than presenting an independent product review.

Randomness is useful in many web experiences:

• Card games
• Quiz question selection
• Recommendation prototypes
• Procedural illustrations
• Daily prompts
• Prize wheels
• Japanese 恋みくじ fortune draws

It is also an easy way to create unreliable tests.

Consider this small function:

typescript
function selectRandom(items: T[]): T {
const index = Math.floor(Math.random() * items.length);
return items[index];
}

The function looks too simple to cause problems. But how should we test it?

This test is obviously incorrect:

typescript
expect(selectRandom(["a", "b", "c"])).toBe("b");

Sometimes it passes. Most of the time it does not.

A more subtle mistake is repeating the function and expecting every possible result to appear:

typescript
const results = new Set(
Array.from(
{ length: 20 },
() => selectRandom(["a", "b", "c"])
)
);

expect(results).toEqual(new Set(["a", "b", "c"]));

This will usually pass, which makes it more dangerous. A random test that “usually passes” is still a flaky test.

The goal is not to remove randomness from the product. It is to remove uncontrolled randomness from the test environment.

Separate decision logic from the random source

The first improvement is to stop calling Math.random() directly inside the selection logic.

Instead, inject the source of randomness.

typescript
type RandomSource = () => number;

function selectRandom(
items: T[],
random: RandomSource = Math.random
): T {
if (items.length === 0) {
throw new Error("Cannot select from an empty collection");
}

const index = Math.floor(random() * items.length);
return items[index];
}

Production code can continue using the default:

typescript
const result = selectRandom(fortunes);

Tests can supply predictable values:

typescript
expect(
selectRandom(["a", "b", "c"], () => 0)
).toBe("a");

expect(
selectRandom(["a", "b", "c"], () => 0.5)
).toBe("b");

expect(
selectRandom(["a", "b", "c"], () => 0.999)
).toBe("c");

The selection algorithm is now deterministic during testing.

This pattern is a small form of dependency injection. Instead of hiding an external dependency inside the function, we pass it as an argument.

Randomness is a dependency just like time, network access, storage, and environment variables.

Test boundaries, not lucky outcomes

When a random number is expected to be in the range 0 <= value < 1, the most useful tests are near the boundaries.

For a three-item collection:

• 0 should select the first item.
• A value just below 1 / 3 should select the first item.
• Exactly 1 / 3 should select the second item.
• A value just below 2 / 3 should select the second item.
• Exactly 2 / 3 should select the third item.
• A value close to 1 should select the final item.

typescript
const values = ["first", "second", "third"];

expect(selectRandom(values, () => 0)).toBe("first");
expect(selectRandom(values, () => 0.332)).toBe("first");
expect(selectRandom(values, () => 1 / 3)).toBe("second");
expect(selectRandom(values, () => 0.665)).toBe("second");
expect(selectRandom(values, () => 2 / 3)).toBe("third");
expect(selectRandom(values, () => 0.999999)).toBe("third");

These tests verify the mapping between the random value and the collection index.

They do not attempt to prove that Math.random() itself is random. That is not the responsibility of this unit.

Decide how to handle invalid random sources

The injected function creates another question: what happens if it returns an invalid value?

typescript
selectRandom(values, () => 1);
selectRandom(values, () => -0.1);
selectRandom(values, () => Number.NaN);

The standard Math.random() function should not produce these values, but a custom source, mock, or future refactor might.

One option is to reject invalid values explicitly:

typescript
function validateRandomValue(value: number): void {
if (!Number.isFinite(value) || value < 0 || value >= 1) {
throw new RangeError(
"Random source must return a finite number from 0 up to, but not including, 1"
);
}
}

typescript
function selectRandom(
items: T[],
random: RandomSource = Math.random
): T {
if (items.length === 0) {
throw new Error("Cannot select from an empty collection");
}

const value = random();
validateRandomValue(value);

return items[Math.floor(value * items.length)];
}

Now invalid behavior fails immediately instead of producing an undefined result several steps later.

typescript
expect(() =>
selectRandom(["a"], () => 1)
).toThrow(RangeError);

Whether this validation belongs in production code depends on the application. But the behavior should be intentional and documented.

Weighted selection needs deterministic tests too

Some applications assign different weights to possible results.

typescript
type WeightedItem = {
value: T;
weight: number;
};

A weighted selector might look like this:

typescript
function selectWeighted(
items: WeightedItem[],
random: RandomSource = Math.random
): T {
if (items.length === 0) {
throw new Error("Cannot select from an empty collection");
}

const totalWeight = items.reduce(
(sum, item) => sum + item.weight,
0
);

if (totalWeight <= 0) {
throw new Error("Total weight must be positive");
}

let position = random() * totalWeight;

for (const item of items) {
position -= item.weight;

if (position < 0) {
  return item.value;
}

}

return items[items.length - 1].value;
}

Suppose the weights are:

typescript
const options = [
{ value: "gentle", weight: 1 },
{ value: "bright", weight: 2 },
{ value: "reflective", weight: 1 }
];

The total weight is four.

That means:

• gentle owns the interval from 0 to 1.
• bright owns the interval from 1 to 3.
• reflective owns the interval from 3 to 4.

Because the random function returns values between zero and one, we can test each interval deliberately:

typescript
expect(
selectWeighted(options, () => 0)
).toBe("gentle");

expect(
selectWeighted(options, () => 0.3)
).toBe("bright");

expect(
selectWeighted(options, () => 0.74)
).toBe("bright");

expect(
selectWeighted(options, () => 0.99)
).toBe("reflective");

Again, the test controls the input. It does not wait for the desired outcome to appear by chance.

Weighted selection should also reject invalid data:

typescript
function validateWeights(
items: WeightedItem[]
): void {
for (const item of items) {
if (!Number.isFinite(item.weight) || item.weight < 0) {
throw new Error("Weights must be finite and non-negative");
}
}
}

Useful edge cases include:

• An empty collection
• All weights set to zero
• A negative weight
• A NaN weight
• One item with all available weight
• Zero-weight items between valid candidates

Use seeded randomness for larger scenarios

Injecting a fixed value is excellent for unit tests. Integration tests sometimes need a sequence of different but repeatable values.

A seeded pseudo-random generator can provide that sequence.

typescript
function mulberry32(seed: number): RandomSource {
return function () {
let value = seed += 0x6d2b79f5;

value = Math.imul(value ^ (value >>> 15), value | 1);
value ^= value + Math.imul(
  value ^ (value >>> 7),
  value | 61
);

return (
  (value ^ (value >>> 14)) >>> 0
) / 4294967296;

};
}

The same seed produces the same sequence:

typescript
const randomA = mulberry32(12345);
const randomB = mulberry32(12345);

expect(randomA()).toBe(randomB());
expect(randomA()).toBe(randomB());
expect(randomA()).toBe(randomB());

This makes complex scenarios reproducible.

typescript
const random = mulberry32(2026);

const sequence = Array.from(
{ length: 10 },
() => selectRandom(fortunes, random)
);

If the test fails, the seed can be printed in the error message. The developer can then reproduce exactly the same sequence.

A seeded generator is useful for testing. It should not be mistaken for a cryptographically secure random source.

For security-sensitive tokens, lotteries involving real value, or authentication systems, use an appropriate cryptographic API instead.

Test invariants instead of exact sequences

Many randomized systems have rules that should remain true for every possible outcome.

For example:

• The selected result must belong to the eligible collection.
• A disabled result must never be selected.
• The function must return exactly one result.
• The original array must not be modified.
• A result with zero weight must never be returned.
• The selected locale must match the requested locale.

These are invariants.

typescript
const original = [...fortunes];
const selected = selectRandom(fortunes, () => 0.4);

expect(fortunes).toEqual(original);
expect(fortunes).toContain(selected);

For contextual selection:

typescript
const selected = drawFortune({
fortunes,
situation: "waiting_for_reply",
locale: "en",
random: () => 0.25
});

expect(selected.locale).toBe("en");
expect(
selected.situations.includes("waiting_for_reply") ||
selected.situations.includes("general")
).toBe(true);

Invariants are more stable than testing a specific message. Content can change while the behavioral contract remains valid.

Statistical tests are different from unit tests

Sometimes we do need to check whether a weighted distribution behaves approximately as expected.

For example, with weights 1, 2, and 1, the long-run proportions should approach:

• 25%
• 50%
• 25%

A simulation can detect a serious implementation mistake:

typescript
function simulate(
iterations: number,
random: RandomSource
) {
const counts = new Map();

for (let i = 0; i < iterations; i++) {
const result = selectWeighted(options, random);

counts.set(
  result,
  (counts.get(result) ?? 0) + 1
);

}

return counts;
}

typescript
const counts = simulate(
100_000,
mulberry32(7890)
);

const brightRatio =
(counts.get("bright") ?? 0) / 100_000;

expect(brightRatio).toBeGreaterThan(0.48);
expect(brightRatio).toBeLessThan(0.52);

There are two important details here.

First, the simulation uses a fixed seed, so it produces the same result in every test run.

Second, the tolerance is intentionally broader than an exact equality.

typescript
// Do not do this
expect(brightRatio).toBe(0.5);

A distribution test should not run as an unpredictable experiment inside every pull request. It should be deterministic and used to detect large deviations.

For systems where fairness has financial, legal, or competitive consequences, informal simulations are not enough. Those systems need specialist review and stronger statistical methods.

Test the UI as a state machine

A random experience is rarely only a selection function.

The interface may move through several stages:

typescript
type DrawState =
| "idle"
| "drawing"
| "revealed"
| "error";

The expected transitions can be described explicitly:

text
idle → drawing → revealed
idle → drawing → error
revealed → drawing → revealed

Tests should verify behavior at every stage.

typescript
test("disables the draw button during animation", async () => {
render( 0.4} />);

const button = screen.getByRole("button", {
name: /draw/i
});

await user.click(button);

expect(button).toBeDisabled();
expect(
screen.getByText(/drawing/i)
).toBeVisible();
});

After the result appears:

typescript
test("reveals exactly one result", async () => {
render( 0.4} />);

await user.click(
screen.getByRole("button", { name: /draw/i })
);

const results =
await screen.findAllByTestId("fortune-result");

expect(results).toHaveLength(1);
});

The random source remains injected all the way into the UI component.

This is important. Mocking only the lower-level function while allowing the component to call Math.random() elsewhere can reintroduce unpredictable behavior.

Control time as well as randomness

Fortune draws often include a short reveal animation.

Waiting for real timers slows the test suite and can cause timing failures.

typescript
function revealAfterDelay(
callback: () => void,
delay = 1000
) {
setTimeout(callback, delay);
}

With fake timers, the test controls time:

typescript
test("reveals the result after the animation", async () => {
vi.useFakeTimers();

render( 0.2} />);

await user.click(
screen.getByRole("button", { name: /draw/i })
);

expect(
screen.queryByTestId("fortune-result")
).not.toBeInTheDocument();

await vi.advanceTimersByTimeAsync(1000);

expect(
screen.getByTestId("fortune-result")
).toBeVisible();

vi.useRealTimers();
});

Randomness and time are two separate dependencies. Both should be controllable.

Test reduced-motion behavior

A polished animation should not become a requirement for completing the interaction.

When the visitor prefers reduced motion, the result can appear immediately or use a simpler transition.

typescript
function getRevealDelay(
prefersReducedMotion: boolean
): number {
return prefersReducedMotion ? 0 : 1000;
}

typescript
test("does not delay the result when reduced motion is preferred", () => {
expect(getRevealDelay(true)).toBe(0);
expect(getRevealDelay(false)).toBe(1000);
});

The test should verify that both paths reach the same final result.

Accessibility is not only a CSS concern. It can change application timing and state transitions.

Test focus after the reveal

When visual content changes, keyboard and screen-reader users need a clear indication of what happened.

The result heading can receive focus after the draw:

tsx
<h2
ref={resultHeadingRef}
tabIndex={-1}

{result.headline}

The test can verify that behavior:

typescript
test("moves focus to the revealed result", async () => {
render( 0.3} />);

await user.click(
screen.getByRole("button", { name: /draw/i })
);

const heading = await screen.findByRole(
"heading",
{ level: 2 }
);

expect(heading).toHaveFocus();
});

This test does not care which poetic sentence appears. It checks whether the interaction remains understandable.

Prevent multiple draws from one action

A fast double-click can expose race conditions.

Without protection, two timers may start and two results may be recorded.

typescript
test("accepts only one draw while a draw is in progress", async () => {
const random = vi.fn(() => 0.5);

render();

const button = screen.getByRole("button", {
name: /draw/i
});

await user.dblClick(button);

expect(random).toHaveBeenCalledTimes(1);
});

Disabling the button visually is not always enough. The event handler should also protect its state.

typescript
function beginDraw() {
if (state !== "idle" && state !== "revealed") {
return;
}

setState("drawing");
}

This is especially important when the draw creates analytics events, local history, or server requests.

Test analytics without testing the analytics provider

The product may record that a draw was completed.

The test should verify the application’s event contract, not make a real request to an analytics service.

typescript
type TrackEvent = (
name: string,
properties?: Record
) => void;

typescript
test("records one completion event", async () => {
const track = vi.fn();

render(
random={() => 0.6}
track={track}
/>
);

await user.click(
screen.getByRole("button", { name: /draw/i })
);

expect(track).toHaveBeenCalledTimes(1);
expect(track).toHaveBeenCalledWith(
"fortune_draw_completed",
expect.objectContaining({
locale: "en"
})
);
});

A test can also verify which information is not sent:

typescript
const properties = track.mock.calls[0][1];

expect(properties).not.toHaveProperty("name");
expect(properties).not.toHaveProperty("question");
expect(properties).not.toHaveProperty("relationshipDetails");

Testing privacy boundaries can be just as important as testing output.

Applying these tests to a real experience

I have been thinking about these testing patterns while working with Ichizenn’s 恋みくじ, a browser-based Japanese love-fortune experience.

The interesting part is that a result should feel unpredictable to the visitor while remaining completely reproducible in the test suite.

Those goals do not conflict.

Production can use a genuine runtime random source. Tests can inject a controlled one. Both environments exercise the same selection rules, state transitions, accessibility behavior, and error handling.

The test does not need to know whether the result says “wait patiently” or “take a small step forward.” It needs to know that the result was eligible, rendered once, announced correctly, and produced without exposing private context.

A practical testing checklist

For a randomized web interaction, I would test the following:

Selection

• Empty collections fail clearly.
• Boundary random values select the correct items.
• Every selected item belongs to the eligible collection.
• Disabled and zero-weight results are never returned.
• The input collection is not modified.

Weighting

• Interval boundaries map to the correct result.
• Negative and invalid weights are rejected.
• All-zero weights fail clearly.
• A seeded distribution test detects large implementation errors.

Interface state

• The draw button is disabled while drawing.
• Double-clicking does not create multiple draws.
• Success and error paths reach valid states.
• Timers are controlled in tests.
• A second draw does not leave the previous result active.

Accessibility

• Reduced-motion preferences are respected.
• Focus moves to the result.
• The result is available as real text.
• Status changes can be announced by assistive technology.
• The draw remains usable with a keyboard.

Privacy and analytics

• Events are emitted once.
• Events use the expected names and properties.
• Personal questions are not included.
• Test logs do not contain private user input.

Final thoughts

Testing a randomized interface does not require random tests.

The most reliable strategy is to make uncertainty controllable:

• Inject the random source.
• Use fixed values for unit tests.
• Use seeded sequences for larger scenarios.
• Test boundaries and invariants.
• Treat statistical checks separately.
• Control timers.
• Verify UI state and accessibility.
• Test what the application refuses to collect.

Visitors can still experience surprise.

Developers should not experience surprise when the test suite runs.

What other hidden dependencies—time, locale, network conditions, browser APIs, or feature flags—have caused flaky tests in your projects?

Comments

[helha]

One refinement I’m considering is replacing the bare random function with a small interface:

interface RandomGenerator {
  next(): number;
  seed?: number;
}

A function is sufficient for simple unit tests, but an object can make debugging larger scenarios easier. If a seeded integration test fails, the test report can include the seed and replay the exact sequence.

The application code would still depend on an abstraction rather than a specific generator:

function selectRandom<T>(
  items: T[],
  random: RandomGenerator
): T {
  return items[
    Math.floor(random.next() * items.length)
  ];
}

This could also prevent accidental mixing of several unrelated random sources inside the same interaction. Has anyone found this interface useful in production, or does it usually add more abstraction than value?

[keyoneok]

Disclosure: I’m involved in testing the project referenced in the article.

The distinction between deterministic behavior tests and statistical distribution tests is important. A unit test should verify that known input intervals map to the correct outcomes. It should not attempt to prove that Math.random() has a perfect distribution.

For weighted selection, I would keep one seeded simulation as a regression test with deliberately broad tolerances. Its purpose would be to detect major errors—such as applying a weight twice or making one branch unreachable—not to certify mathematical fairness.

If real money, prizes, or competitive outcomes were involved, I would treat that as a different problem requiring cryptographically appropriate randomness, threat modeling, and proper statistical review.

[doichizen]

Property-based testing seems like a natural next step for this approach.

Instead of writing examples for every possible collection, a test could generate collections with different lengths and verify invariants such as:

• The selected value always belongs to the input.
• The input collection is never mutated.
• Zero-weight entries are never selected.
• Empty collections always fail explicitly.
• The same seed produces the same sequence.

The most valuable feature would be shrinking. If a failure first appears with 200 generated fortune objects, the framework could reduce it to the smallest collection that still reproduces the problem.

Combined with seed logging, that would turn an apparently random failure into a small, repeatable test case—which is exactly what a reliable randomized system needs.