One Green Run Proves Nothing: Probe Agent Trajectory Variance

A single passing run tells you that one path through your agent exists. It does not tell you that the path is stable, and it says almost nothing about whether the bug you just "fixed" is gone. The cheaper your model calls get, the weaker the excuse for accepting n=1 as evidence.

The loop most teams run is depressingly short: read one trace, edit the prompt or a tool schema, rerun the failing task, watch it turn green, close the ticket. Nothing in that sequence separates a real fix from a lucky sample. A prompt tweak that shifts tool-choice probability from 0.4 to 0.6 will look like a triumph on the first rerun and like the original bug on the third.

Trajectory variance probing is the cheapest way to close that gap. Keep the input fixed, run the agent k times, normalize each run into a chain of decision signatures, then report where the chains first diverge. The output is not a pass rate. It is a fork point you can point at.

Canonicalize the decision, not the payload

You cannot compare raw traces. Timestamps drift, request IDs differ, retrieval returns different chunks, and absolute paths leak into arguments. Hash the decision the agent made and scrub everything volatile before you do it.

The harness below is a skeleton I have not run against a live endpoint. Treat it as a method you can execute, not as a measurement of any product.

# probe.py — canonical decision signatures for repeated runs
import hashlib, json

# fields that change on every run and carry no decision content
VOLATILE = {
    "ts", "timestamp", "request_id", "trace_id", "span_id",
    "elapsed_ms", "abs_path", "cache_hit", "retrieved_at",
}

def scrub(obj):
    if isinstance(obj, dict):
        return {k: scrub(v) for k, v in obj.items() if k not in VOLATILE}
    if isinstance(obj, list):
        return [scrub(v) for v in obj]
    return obj

def signature(step):
    payload = json.dumps(
        {"kind": step["kind"], "name": step["name"], "args": scrub(step.get("args", {}))},
        sort_keys=True, separators=(",", ":"),
    )
    return hashlib.sha256(payload.encode()).hexdigest()[:12]

def chain(trace):
    # only tool calls and final answers count; free-text reasoning is noise here
    return [signature(s) for s in trace if s["kind"] in ("tool", "final")]

One deliberate omission: tool results never enter the signature. A search that returns three different snippets is still the same decision, and hashing results turns every retrieval call into a false fork. Log results beside the signature, not inside it.

Find the fork, not the pass rate

With chains in hand, the interesting number is the length of the common prefix. Compute it against the modal (most frequent) chain, then aggregate where divergence happens.

from collections import Counter

def first_fork(a, b):
    for i, (x, y) in enumerate(zip(a, b)):
        if x != y:
            return i
    return None if len(a) == len(b) else min(len(a), len(b))

def probe_summary(chains, k):
    modal, hits = Counter(tuple(c) for c in chains).most_common(1)[0]
    forks = Counter()
    for c in chains:
        i = first_fork(list(modal), c)
        if i is not None:
            forks[i] += 1
    return {
        "k": k,
        "modal_share": hits / k,          # 1.0 means every run walked the same path
        "lengths": [len(c) for c in chains],
        "fork_histogram": dict(sorted(forks.items())),
    }

Read the histogram like a diagnostic, not a score. A fork at step 0 means the agent is unsure what the task is — usually a prompt or tool-description problem, and it will not be fixed by retries. A fork at step 4 after a shared prefix of three clean tool calls points at one specific observation being read two ways. Two runs that end early with identical prefixes but shorter tails are your loop-and-bail signature, and they belong in the same bucket as truncation, not in the same bucket as wrong answers.

Run it as a batch job rather than an interactive ritual:

# nightly against a sandbox with recorded tool responses; short k keeps the wall clock sane
python probe.py --task tasks/refund_flow.json --k 3 --out traces/2026-11-08/
# investigate a specific fork
python probe.py --task tasks/refund_flow.json --k 5 --only-fork 4

Making k affordable changes the decision

Variance probing is a statistical habit, and statistics are the first thing teams cut when each rerun costs money and each trace needs a home. That is the honest reason free model access and a free server option are relevant here: they move the probe from a quarterly audit to a nightly job.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The project offers free model access and a free server option; as of this writing the promotion is described by the operator as a large free token allowance plus a free server, and terms like that change, so confirm the current numbers on the project page before you build a cost model around them. What matters for the method is the shape of the economics: if a rerun is near-free, k=5 beats k=1 for the same reason a smoke test beats a coin flip. I run the probe loop conceptually like this — a small server holds the harness and the trace directory, the model endpoint supplies the repeated trajectories, and the artifact is a histogram, not a dashboard.

Situation	Probe k	Verdict
Prompt or tool-schema change about to merge	5	Worth it; fork histogram is your review artifact
Flaky failure reported by one user	5	Worth it; forks show whether it is env or reasoning
Latency work on a deterministic pipeline	0	Skip; there is no decision to sample
Task with real side effects (payments, emails, writes)	0 unless sandboxed	Do not probe in production
Long multi-hour runs	2–3	Cost dominates; probe the first 10 steps only

The last row matters more than it looks. You can truncate the probe to a prefix and still get a useful fork histogram, because most trajectory divergence happens early. That single trick makes the method survivable on a free tier with rate limits.

What this does not tell you

k=5 is not a confidence interval. It is a tripwire. If all five chains match, you have evidence of stability on that input, not proof, and a fork-free histogram on a task you already know is fragile usually means your signatures are too coarse — check whether you are scrubbing the field that actually varies. Temperature 0 is not determinism either; batching, retrieval, and tool state all reintroduce noise.

Do not use this on agents whose tools mutate shared state, and do not use it as a substitute for a real regression suite; a stable wrong path is still wrong, and the probe will happily confirm it five times. Teams without recorded inputs, or whose traces omit tool arguments entirely, should fix instrumentation first — the harness cannot compare what was never logged.

Fix the instrumentation, then fix k. Then, when a rerun goes green, you will know whether it went green once or went green every time, which is the only version of green worth shipping.

If you try the probe, start with one task you already consider flaky and let the fork histogram argue with your intuition.