He Turned a Use-After-Free Into a Hypothesis Queue. A Free Server Did the Voting.

He Turned a Use-After-Free Into a Hypothesis Queue. A Free Server Did the Voting.

The crash that would not reproduce

At 4 p.m. on a Tuesday, the monitoring channel lit up. A job server had crashed in production. The stack trace ended in EventLoop::dispatch. The same binary, rebuilt locally, survived the same test. Turn the test up to 64 threads and nothing happened. The fault existed somewhere between live traffic and a specific interleaving.

The team had one afternoon. A full defect investigation was out. They could not run a large test matrix on their laptops. The engineer on rotation used MonkeyCode's free model access and the free server option to split the problem into two cheap parts. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model would propose ranked hypotheses. The free server would run the test that each hypothesis pointed at.

Do not ask the model to fix it

The first rule was simple. The model was not allowed to produce a final patch. It was allowed to read an address sanitizer trace, a thread trace, and a two-point timeline. It had to return candidates. Each candidate had a location, an owner, a reason, a test command, and an optional patch path. It did not have to be right. It only had to be runnable.

This limits model errors. A wrong location costs one runner slot. A wrong reason may still lead to a useful test. A hallucinated patch is visible before it runs.

The runner

The free server option became a remote Linux worker. The engineer did not run the full CI suite. He ran one narrow binary per candidate. The script created a git worktree, applied an optional patch, built repro_test, and executed it with a timeout. The snippet is shortened for a one-day triage and assumes paths without spaces.

#!/usr/bin/env bash
set -euo pipefail

candidate_id=$1
repo_root=$2
patch_dir=$3
log_dir=$4

worktree=$repo_root/../wt-$candidate_id
git -C $repo_root worktree add $worktree HEAD >/dev/null 2>&1
trap 'git -C $repo_root worktree remove --force $worktree >/dev/null 2>&1 || true' EXIT

if [ -f $patch_dir/$candidate_id.patch ]; then
  if ! git -C $worktree apply $patch_dir/$candidate_id.patch; then
    echo APPLY_FAILED > $log_dir/$candidate_id.status
    exit 0
  fi
fi

cmake -S $worktree -B $worktree/build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo >/dev/null
cmake --build $worktree/build --target repro_test >/dev/null

set +e
timeout 180 $worktree/build/repro_test --threads=64 --iterations=5000 > $log_dir/$candidate_id.out 2>&1
status=$?
set -e

echo $status > $log_dir/$candidate_id.status

A small Python loop consumed the output. It did not decide whether a patch was correct. It only classified what the runner observed.

# runner.py
import json
import pathlib
import subprocess

repo_root = pathlib.Path('./job-server')
patch_dir = pathlib.Path('./patches')
log_dir = pathlib.Path('./logs')
log_dir.mkdir(exist_ok=True)

with open('hypotheses.json') as f:
    hypotheses = json.load(f)

def classify(status_text, out_text):
    status = status_text.strip()
    tail = out_text[-2000:]
    if status == 'APPLY_FAILED':
        return 'reject', tail or 'patch did not apply'
    if status == '124':
        return 'timeout', tail
    if status != '0':
        return 'crash', tail
    if 'ERROR: AddressSanitizer' in tail:
        return 'new_trace', tail
    return 'clean', tail

for h in hypotheses['candidates']:
    subprocess.run([
        'bash', 'triage_hypothesis.sh',
        h['id'], str(repo_root), str(patch_dir), str(log_dir)
    ], check=False)
    status = (log_dir / (h['id'] + '.status')).read_text()
    out = (log_dir / (h['id'] + '.out')).read_text(errors='replace')
    verdict, tail = classify(status, out)
    print(h['id'] + ' ' + verdict + ' ' + h['owner'])
    if verdict in ('crash', 'new_trace', 'timeout'):
        print(tail)

A table, not a judge

The runner did not prove correctness. It produced a signal with a narrow meaning.

status	meaning	action
clean	the test survived 5,000 iterations	keep the candidate, add a regression test
crash	the test died	read the tail, reject or refine
new sanitizer trace	the failure moved	useful if the trace changed
apply failed	patch context mismatch	rebase or drop
timeout	no result	reject or lower iterations

Every row in the table was a decision the team still had to make. The model did not make it for them.

What the one-day run found

The free model returned five candidates. Candidate h1 blamed a signal-handler-writable global. Its patch never applied. Candidate h2 blamed a Session that was freed by TimerQueue::expired while EventLoop::dispatch still held a pointer. The free server ran the test for h2. The baseline crashed around iteration 1,400. The patched build survived all 5,000 iterations at 64 threads.

That was not proof. It was a change of direction. The team stopped reading raw logs and started reviewing ownership around Session. They wrote a regression test that failed on the old code and passed on the reviewed patch.

Limitations

A clean run is a small signal. It does not prove a fix. The free server may differ from production in kernel version, compiler flags, libc, CPU count, or timing. A candidate that survives 5,000 iterations can still fail after two million. The model can invent an owner or a line number. It can also return a patch that seems harmless but changes behavior in a way the test does not cover.

No one should auto-apply model output. Here it was read, placed in a worktree, and run on one test. That still leaves room for a subtle wrong fix. Data confidentiality matters too. A stack trace can reveal internal paths or customer behavior. If the policy does not allow sending it, this workflow stops.

Who should not use this

A team with a fully reproducible bug should use a debugger, sanitizers, and a bisect first. A model is slower and less exact for that case. A team that cannot read C++ well should not accept code it cannot review. A team with flaky, expensive, or security-sensitive tests should not run model-generated candidates blindly.

The useful idea is the queue. The model fills it. A cheap runner empties it. The value came from making each hypothesis disposable, not from any single answer. Teams with free model access and a free Linux runner can try the same one-day triage pattern without making the model an owner of a single line.