git bisect run named the wrong commit 10 out of 10 times — so I gave every commit its own clean room

git bisect run is the right algorithm. What I did not expect is how confidently it lies when you leave it running unattended.

I generated a 255-commit Python history in which a dependency pin (pytz) moves from 2022.1 to 2023.3 at commit 150, together with the golden value of the failing test. Each commit's test is right under its own pin and wrong under the other. A real bug is planted at commit 173. Then I ran git bisect run on it ten times, in one reused virtualenv:

How	Named	Right
env built at the known-good ref	the pytz bump	0 / 5
env built at the failing ref	a changelog edit	0 / 5
fresh venv at every step	the planted commit	5 / 5

Ten of ten wrong, both flavours. The fixture is built to drift, so that is by construction — what it shows is the mechanism: bisect trusts one environment for every commit, and a moving pin makes some verdicts false, so the search walks into the wrong half and convicts an innocent commit with total confidence. (On a real regression with no drift — python-attrs/attrs 22.1.0 → 22.2.0, 135 commits — every method was right.)

The fix is not clever: give every suspect its own clean room. Faultline does that on Nebius Serverless AI Jobs — one terminating job per commit, on the stock python:3.11 image, installing that commit's own declared dependencies from scratch.

The honest column

On the drifting history, Faultline named the planted commit in 3 rounds, 19 jobs, 957 seconds, $0.0948. A fresh-venv git bisect run did it in 57 seconds on a laptop.

Faultline is slower. A probe's VM takes about 4 minutes 20 seconds to start for about 5 seconds of work, and Nebius throttles job creation (HTTP 429, undocumented), so a round costs minutes where a local step costs seconds. It is not a faster bisect; it is one you can leave running without knowing about the drift or writing the fresh-environment harness. On a suite that takes minutes to build per commit, parallel rounds should start to pay for their cold starts — I have not measured such a repository, so I don't claim it.

Three walls

1. A job cannot tell you what happened. The Jobs REST API has no exit code and no logs, so a probe cannot report a test result to a caller. My first workaround was a read-write bucket mounted into every probe, with the container writing a verdict file. Then I remembered what runs inside a probe: a stranger's conftest.py. Anything mounted into the job belongs to the stranger — including every other hunt's records. Now each probe gets no mount and no key, only one presigned PUT URL for its own verdict object:

export function presignPut(config: NebiusConfig, key: string, expiresSeconds: number, date?: Date): string {
  const url = new URL(objectUrl(config, key));
  return presignUrl({ /* SigV4, method PUT, one key, expires */ });
}

The worst a hostile test suite can do with it is rewrite its own verdict — which it already controls by choosing its exit code. A side effect: a uk-south1 probe writes into an eu-west1 bucket over plain HTTPS, so regions are interchangeable.

2. Two jobs per region, and the third is rejected, not queued. A new tenant's compute.instance.count is 2; a third job returns QuotaExceeded. So concurrency is a parameter of the backend, not a constant: 2 slots × 4 regions = 8 probes. Each round tests the two ends of the range plus as many suspects as there are free slots, and N probes cut the range into N+1 segments.

3. No daemon, no database. The control loop is a tick that every finishing probe pings. Two ticks can race, so the hunt record is replaced only with If-Match, and every dispatch is claimed first with If-None-Match: "*":

const header: Record<string, string> =
  "ifNoneMatch" in condition
    ? { "If-None-Match": condition.ifNoneMatch }
    : { "If-Match": condition.ifMatch };
const res = await s3Request(config, "PUT", key, body, {}, fetchImpl, header);
if (res.status === 412 || res.status === 409) return { ok: false, status: res.status };

I measured it on the real bucket before building on it: 20 races of 2–8 concurrent creators on one key, exactly one winner every time.

Where the model comes in — and where it doesn't

When test execution has named the culprit, one call goes to NVIDIA Nemotron 3 Super on Nebius Token Factory with the diff and the failing output, and it writes the root-cause paragraph. It is a reasoning model: with a small token budget it can return its reasoning and no answer. So the call asks for low reasoning effort with room to answer, reads content only, and treats an empty answer as an error rather than inventing one. The UI labels the paragraph as generated, next to a culprit labelled as proven by executed tests. The model explains; it never decides. (On the hero hunt it also got a detail wrong — it called UTC−6 "DST-adjusted" for Mexico City, which abolished DST in 2022 — and the project's DEMO.md says so.)

Limits

Python test suites and public GitHub repositories only. One live hunt at a time on the shared slots, with a daily cap. Slower than a careful local bisect on a fast suite.

You can watch a recorded real hunt (real job IDs, no cost) or run a live one, no login: https://faultline-edy-cus-projects.vercel.app. Code (MIT): https://github.com/edycutjong/faultline. Built for the Nebius Global AI Hackathon.