Make an order-dependent test reproducible before handing it to a coding agent

This tutorial was drafted with AI assistance. Its small JavaScript fixture was executed locally; it is not a production incident report.

A test that passes alone and fails after another test is often called flaky. That label describes the symptom, but it is a poor implementation brief. Before assigning a repair, make the order dependence explicit.

This example uses a shared cache. It gives a coding agent a failure that can be reproduced without waiting for CI to get unlucky, plus an acceptance condition that does not weaken the test.

1. Reduce the failure to two tests

One test warms a greeting service. The other expects a fresh fixture. When both use the same cache, the second test inherits state it did not create.

Save this as order-fixture.mjs and run node order-fixture.mjs:

import assert from "node:assert/strict";

const sharedCache = new Map();

function makeService(cache) {
  return {
    greet(name) {
      if (!cache.has(name)) cache.set(name, `Hello, ${name}`);
      return cache.get(name);
    },
    get size() { return cache.size; }
  };
}

function warm(service) {
  assert.equal(service.greet("Ada"), "Hello, Ada");
}

function cold(service) {
  assert.equal(service.size, 0, "fixture must start empty");
}

const shared = makeService(sharedCache);
warm(shared);
try {
  cold(shared);
} catch (error) {
  console.log(
    `BEFORE warm -> cold: FAIL (actual=${error.actual}, ` +
    `expected=${error.expected})`
  );
}

for (const order of [[warm, cold], [cold, warm]]) {
  for (let run = 0; run < 10; run++) {
    for (const test of order) test(makeService(new Map()));
  }
  console.log(
    `AFTER ${order.map(test => test.name).join(" -> ")}: ` +
    "10/10 order runs passed"
  );
}

The fixture's equivalent local run observed a cache size of one where the cold test expected zero. With a fresh cache for each test, both orders passed ten consecutive runs. The first failure is caught so the demonstration can continue; a real test suite should let the assertion fail normally.

This is a controlled example of state leakage. It does not demonstrate that every intermittent test failure involves a cache.

2. Rewrite the assignment around the evidence

The weak ticket is:

Fix the flaky greeting test. It usually passes when rerun.

That leaves the agent free to add retries, reorder the suite, clear production state, or change the expectation. None necessarily addresses the underlying fixture problem.

Use a brief like this instead:

Problem
The cold-fixture test fails after the warm-cache test.
The cache is shared between tests; expected size is 0,
but the failing order observes size 1.

Reproduction
Run the named tests in warm -> cold order.
Also run each test alone and in cold -> warm order.
Record exact commands and output.

Scope
Give each test an independently owned fixture.
Change test setup and test-only helpers.
Preserve production cache behavior and both assertions.

Acceptance
Both orders pass with fresh fixtures.
The cold test still requires an empty cache.
The warm test still verifies the returned greeting.
The relevant suite passes without retries or skipped tests.

Stop condition
If production singleton state prevents test isolation,
report the dependency and proposed design change first.

Delivery
Patch, reproduction before the fix, verification after it,
and any behavior the checks do not cover.

In a repository task, include the actual test names, paths, runner options, and failing output. Do not present this fixture's evidence as if it came from the repository.

3. Give the human runner a separate operating checklist

The task author prepares the failure evidence and scope. The teammate executing the task uses their own authorized workspace, tools, and coding agent. The reviewer decides whether the result satisfies the brief.

The runner's checklist can stay short:

• Confirm the selected repository and test environment.
• Have the agent reproduce the named order-dependent failure.
• Inspect fixture ownership in the patch.
• Check that production behavior and assertions remain intact.
• Return the exact test output and unresolved questions.

The Wagglet dual-prompt guide describes the related separation between detailed agent instructions and a concise human runbook. For AI task handoff for teams, the useful distinction is who supplies the context, who executes the bounded work, and who judges acceptance.

4. Review the mechanism, not just the green result

Fresh fixtures solve this example because neither test needs the other's state. Clearing one global cache between tests can still be unsafe if another test is using it concurrently. A production singleton may need a deliberate test seam rather than an improvised reset.

Two passing orders also do not prove that a larger suite has no hidden dependencies. They provide evidence about the failure you isolated. Run the relevant surrounding suite, and inspect whether timers, connections, or other resources survive teardown.

Keep the conclusion narrow: the named fixture no longer inherits another test's cache, the original assertions still hold, and the required checks passed. That is a repair another engineer can review.