Agent Patches Need a Sensor Queue, Not a Skip List

An agent patch should not merge on a single green bar that hid three different sensors behind one status. Property checks, fixture equality, and flaky integration tests do not fail the same way. Mixing them into one skip list lets the cheapest, noisiest sensor veto the only oracle that still means something.

Run tests as a priority queue after the agent writes. Hard properties first. Content-addressed inputs second. A statistical flake budget last. Do not promote a flake into a skip. Do not demote a property into a retry.

That ordering is the whole method. The rest of this article is a classifier, a decision table, and a small Python harness you can run locally. The harness is a proposed workflow, not a production study and not a claim about any live codebase.

Why one status bit is the wrong unit

CI collapses every check into pass or fail. Agent patches then learn the wrong lesson. They optimize the bar, not the behavior.

A property is a predicate over many inputs. If it fails once, the patch is wrong or the property is wrong. Retrying it is not information. A fixture check is a digest of a known input and a known output. If the digest moves, either the input was edited or the patch changed observable behavior. A flaky integration test is a noisy sensor. It can fail while the patch is fine, and it can pass while the patch is not.

Those three sensors need different policies. A skip list applies one policy to all of them. That is how a timeout in a browser test becomes a reason to ignore a broken idempotency invariant.

Classify every test before the agent runs

Do the classification in a manifest the agent cannot rewrite. Keep it next to the suite, in review, and hashed in CI. If the patch also edits the manifest, that edit is a separate review item. It is not part of the code change under test.

Use three classes only:

1. property — a predicate that must hold for every generated or enumerated input. Failure is blocking. No retry budget.
2. fixture — a pair of input bytes and expected bytes, compared by digest. Failure is blocking unless the digest change is reviewed as an intentional fixture update.
3. noisy — a test with at least one non-deterministic dependency (time, network, shared temp, scheduling). It never blocks on a single run. It blocks on a k-of-n rule.

If you cannot classify a test, it is noisy by default. That is conservative for merge, not generous. Unclassified tests do not get to be silent skips.

Decision table

Sensor	Single failure means	Allowed action	Merge rule
property	Predicate broken, or the property itself is false	Fix the patch or fix the property	Zero failures on the current seed list
fixture	Input changed, or output changed	Restore the input, or accept a new digest in review	Digests match the pinned set
noisy	Sensor error, or real regression	Re-run up to n times, require k passes	passes >= k and runs <= n

The table is the policy. The queue is the schedule. Properties are cheap compared with a wrong merge. Fixtures are cheap compared with a silent format change. Noisy tests are expensive and low-precision. Put them last so they cannot starve the oracles.

A sensor queue you can actually run

The example below is labeled as a proposal. It does not call a real test runner. It shows the control flow you want around one: classify, pin, then apply a binomial-style gate only to noisy rows.

# sensor_queue.py — proposed control flow, not a production runner
from __future__ import annotations

import hashlib
import json
import random
from dataclasses import dataclass
from pathlib import Path
from typing import Callable

@dataclass(frozen=True)
class Sensor:
    name: str
    kind: str  # property | fixture | noisy
    n: int = 1
    k: int = 1

@dataclass
class Report:
    name: str
    kind: str
    runs: int
    passes: int
    blocked: bool
    reason: str


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def run_property(fn: Callable[[int], bool], seeds: list[int]) -> tuple[int, int, str]:
    passes = 0
    for seed in seeds:
        if not fn(seed):
            return passes, len(seeds), f"property failed on seed={seed}"
        passes += 1
    return passes, len(seeds), "ok"


def run_fixture(path: Path, pinned: str) -> tuple[int, int, str]:
    digest = sha256_bytes(path.read_bytes())
    if digest != pinned:
        return 0, 1, f"fixture digest {digest[:12]} != pinned {pinned[:12]}"
    return 1, 1, "ok"


def run_noisy(fn: Callable[[], bool], n: int, k: int) -> tuple[int, int, str]:
    passes = 0
    for i in range(n):
        if fn():
            passes += 1
        if passes >= k:
            return passes, i + 1, "ok"
        remaining = n - (i + 1)
        if passes + remaining < k:
            return passes, i + 1, f"noisy gate failed: {passes}/{i + 1}, need {k}/{n}"
    return passes, n, f"noisy gate failed: {passes}/{n}, need {k}/{n}"


def queue_order(sensors: list[Sensor]) -> list[Sensor]:
    rank = {"property": 0, "fixture": 1, "noisy": 2}
    return sorted(sensors, key=lambda s: (rank[s.kind], s.name))


def evaluate(sensors: list[Sensor], fns: dict, fixtures: dict, seeds: list[int]) -> list[Report]:
    reports: list[Report] = []
    for sensor in queue_order(sensors):
        if sensor.kind == "property":
            passes, runs, reason = run_property(fns[sensor.name], seeds)
            blocked = reason != "ok"
        elif sensor.kind == "fixture":
            path, pinned = fixtures[sensor.name]
            passes, runs, reason = run_fixture(path, pinned)
            blocked = reason != "ok"
        elif sensor.kind == "noisy":
            passes, runs, reason = run_noisy(fns[sensor.name], sensor.n, sensor.k)
            blocked = reason != "ok"
        else:
            raise ValueError(f"unknown kind: {sensor.kind}")
        reports.append(Report(sensor.name, sensor.kind, runs, passes, blocked, reason))
        if blocked and sensor.kind in {"property", "fixture"}:
            break  # hard sensors stop the queue
    return reports

A property failure stops the queue. A fixture mismatch stops the queue. A noisy failure does not skip later properties, because properties already ran. That is the point of the order.

Numbered workflow for one patch

1. Freeze classification. Commit sensors.json with kind, and for noisy rows only, n and k. Do not let the agent edit this file in the same diff as production code.
2. Record fixture digests from the parent revision, not from the agent's working tree. Hash the input files the test reads, not the test source.
3. Choose property seeds before the agent starts. Store them as a list of integers. Reuse the same list on the patched tree. New seeds are a new experiment, not a silent retry.
4. Run property sensors against the patched tree. Any failure is a patch reject or a property bug. There is no third outcome.
5. Run fixture sensors. A digest change is either accidental input drift or a behavior change. Both need a human. Neither is a flake.
6. Run noisy sensors with early exit: stop at k passes, or stop when remaining runs cannot reach k.
7. Publish a per-sensor report. One line per sensor: kind, runs, passes, blocked, reason. Do not fold it back into a single boolean until a human has seen the kinds.

A minimal manifest looks like this:

{
  "seeds": [7, 19, 23, 41, 88],
  "sensors": [
    {"name": "idempotent_put", "kind": "property"},
    {"name": "reject_empty_key", "kind": "property"},
    {"name": "golden_invoice_v3.json", "kind": "fixture"},
    {"name": "checkout_browser", "kind": "noisy", "n": 7, "k": 6}
  ]
}

Pick k and n from the sensor's history, not from hope. If a browser test passed 90 of the last 100 runs on an unchanged tree, k/n = 6/7 is already stricter than a single run and still cheaper than treating it as a property. If you do not have history, start with k = n and cut n only after you measure noise. Do not start from k = 1.

What the agent is allowed to touch

The agent may edit application code. It may add a property if the property is reviewed like production code. It may not:

• delete or rename a property row to make the queue shorter
• rewrite a fixture file and the digest in the same patch without a separate review note
• move a property to noisy to gain a retry budget
• raise n or lower k on a noisy sensor in the same diff as the failing code

Those edits change the oracle. Treat oracle edits as interface changes. They need a smaller diff and a harder review than a helper rename.

If you run the coding agent on MonkeyCode's free model access and free server option, put this queue on that same checkout. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The value is colocation, not a model claim: the patch and the sensors see one tree, one seed list, and one set of fixture bytes. That is enough reason to keep the queue next to the agent. Use it only if you already need a machine for the agent run. It is not a substitute for a real CI worker that holds secrets or production data.

Properties that survive an agent patch

Write properties as predicates over values, not as traces of one happy path. The agent is good at cloning a happy path. It is weaker at preserving algebraic shape.

Examples that usually stay meaningful after a refactor:

• Idempotency: put(put(x)) == put(x) for every seed x.
• Round-trip: decode(encode(x)) == x for every seed x in the documented domain.
• Monotonicity: a counter, watermark, or cursor never moves backward.
• Authorization: a forbidden principal still gets a forbidden response after the patch.
• Error mapping: invalid input still yields a typed error, not an empty 200.

Keep the generator boring. Enumerate a list, or use a small integer seed to build a structure. If you need shrinking, shrink outside the merge gate. The gate should replay known seeds, not search a new space on every patch. Search is for the next property you add, after the current patch has already passed the pinned seeds.

Fixtures are inputs, not screenshots of the whole program

Pin the bytes the test reads. Do not pin the whole repository. A content digest of a 2 KB JSON invoice is a fixture. A digest of node_modules is a trap.

When the agent formats a file, the fixture must not move. When the agent changes serialization, the fixture must move, and a reviewer must say so. That distinction disappears if you hash the test file instead of the input file. Hash the input.

Store pinned digests beside the manifest:

golden_invoice_v3.json  sha256:9c3f1a0b...

CI recomputes the digest from the parent tree, then from the patched tree. Parent mismatch means your pin is stale. Patch mismatch means the agent changed bytes you said were golden.

Noisy tests get a budget, not a pardon

A flake is a sensor with a false-positive rate. Ignoring it sets that rate to 1 for merge purposes. You learn nothing.

Use a fixed budget. n is the maximum independent runs. k is the minimum passes. Independence matters. Re-running the same polluted worker five times is one run with extra logging. Rotate temp dirs, clocks, and ports between attempts, or do not count them as separate trials.

Stop early. If k = 6 and n = 7 and the first two runs fail, remaining runs cannot recover. Fail the sensor. Spend the machine time on the next patch, not on a sensor you already know is red under the rule.

Never convert a noisy fail into a skip during the same patch. If the sensor is too expensive, remove it from merge and keep it as nightly signal. That is an explicit policy change. It is visible. A skip list in the patch is not.

Limitations

This queue does not create an oracle. If you have no properties and no pinned inputs, you are left with noisy sensors and a k-of-n ritual. That ritual is still better than a skip, and it is still not proof.

The binomial-style gate assumes attempts are independent enough to matter. Shared caches, polluted containers, and leaked files violate that. The fixture digest assumes the interesting behavior is in the bytes you hashed. If the agent changes a side channel (metrics, logs, extra fields that tests ignore), the digest stays green.

Properties on tiny seed lists miss faults outside the list. That is acceptable for a merge gate only if you keep adding seeds when a bug escapes. It is not a replacement for fuzzing, staging, or a typed spec.

The method also assumes the agent cannot rewrite the classifier, the pins, and the application code in one unattended step. If your setup lets the model edit CI YAML, this queue is theater.

Who should not use this

Do not use this as a merge gate for safety-critical, cryptographic, or compliance-bound changes that need a real proof or a dedicated audit. A property with five seeds is not that proof.

Do not use it if every test in the repo is an end-to-end browser journey. Classify first. If almost everything is noisy, spend the week extracting two properties and one fixture, not tuning k and n.

Do not use a free shared server for tests that need production credentials, customer fixtures, or non-public model weights. Keep those sensors on a worker you control.

If the patch is a one-line comment change, the queue is overhead. Run the properties you already have and skip the essay.

Close

Agent output is cheap. Oracles are not. A skip list spends the oracle to save a few reruns. A sensor queue spends the reruns to keep the oracle. Start with classification, pin the inputs, and give noise a budget that cannot silence a property. That is enough structure to review the next patch as a hypothesis instead of as a green rectangle.