Can a System Grow Its Own Tests? Judges Born from Consequences

This is part four of the “Can You Build an Alternative to LLMs?” series. Part one separated real memory from a polished demo; part two tested self-correction; part three asked whether knowledge is merely a store. Code and experiment logs: AuraSDK.

A repository can have a green cargo test run while a bug is still alive in one of its functions.

That is not a paradox. A conventional test checks a handful of known agreements. If a mutation does not violate those particular agreements, it survives. For an autonomous system, then, the question is not only “can it run tests?” It is harder:

Can it construct a new judging test from the consequences of running code—one that separates a correct implementation from a plausible but incorrect one?

I did not ask a system to “invent a useful test” in free-form text. It gets a constrained process: observe execution, propose simple properties, try to break them on correct code, and turn only the survivors into Rust property tests. The final verdict is not delivered by a Python wrapper or an LLM. It is delivered by cargo test.

Why generating an assertion is not enough

The most dangerous test is not the one that finds nothing. It is the one that confidently fails on correct code.

Imagine a system sees several runs and infers: “whenever the result is non-empty, its length is always at least 3.” That may merely be an accidental bound of the examples it observed. Such a test can reject future correct implementations.

So a candidate property does not immediately become a test. It must pass this cycle:

live code executions
    → candidate property
    → attack it on correct code
    → reject it or scar the false generalization
    → test it against held-out mutations
    → Rust #[test] → cargo test

I call this an invented judge: not “a test written by a model,” but a constrained judge born from consequences and forced to survive refutation.

Five gates, not one attractive run

The first synthetic version showed that the mechanism could work: it killed 5 of 5 held-out mutations with maximum false-fire = 0.0.

That did not mean we had learned how to test Rust code. The experiment therefore passed through five gates:

1. a synthetic Python world;
2. a bridge to real cargo test;
3. seven ports of real Aura functions, compared with a static miner;
4. a property judge that is actually rendered into Rust;
5. a combined code-world gate over bodies of real Aura functions.

Two early results matter precisely because they were inconvenient.

First, an early “real Cargo gate” honestly ended as NOT_PROVEN, even though it already had kill = 1.0 and false-fire = 0.0. Its rule was trivial: it added no new power beyond the weak existing test. To fix this, the experiment added an abs_ratio mutation. It passed the weak Cargo test and evaded a simple boundary judge, but failed the arithmetic property matches_division.

Second, simply carrying a result from Python into Rust would have been self-deception. In the final version, each candidate property is rendered into a #[test] in tests/judges.rs; the same cargo test result is used for our method and the control groups.

What the final gate actually tested

The final script, scripts/run_invented_judge_cargo_property_gate.py, works with literal Rust bodies from six AuraSDK functions:

• filter_selected;
• disjoint;
• accuracy_per_mille;
• the count projection of prune_deaths;
• a bit version of presence_jaccard_rank_key;
• has_duplicates.

It introduced 12 mutations in the style of cargo-mutants. Four were killed by the base Cargo tests—a useful sanity check. Eight survived the weak suite, and those eight became the targets for new judges.

There is no Python oracle quietly supplying the correct answer in this gate. There is a Rust trace harness, a restricted vocabulary of property primitives, and a generated Rust test. The vocabulary is intentionally small: >=/<= bounds, scalar length, subset, empty output on empty input, symmetry, monotonicity, and permutation invariants.

That restriction matters. The system is not discovering arbitrary new mathematics. It is selecting and validating judges from a known, auditable language of properties.

A result that does not hide its own boundary

Under stable bounds, future inputs stayed in a regime already represented by the “lived” executions. Across three independent seeds, both approaches produced the same outcome:

Regime	Approach	Kill rate	False-fire	Net
Stable bounds	Invented judge	1.00	0.00	+1.00
Stable bounds	Static miner	1.00	0.00	+1.00

There is no reason to declare victory here. When past data truly covers the future, static generalization works too.

The interesting result appeared in the fragile-bounds regime. Here, future correct cases cross an accidental bound that the static miner had seen in earlier traces.

Regime	Approach	Kill rate	False-fire	Net
Fragile bounds	Invented judge	0.75	0.00	+0.75
Fragile bounds	Static miner	1.00	0.50	−0.50

At first glance, the static miner looks better: it killed every mutation. But it also falsely accused correct code in half of the held-out cases. Its net result is therefore negative.

Our judge missed one mutation. It did not do so because it failed to see it; its candidate bound turned out to be accidental. The process scarred that generalization and returned ABSTAIN instead of a false verdict. The key advantage here is not “kill more.” It is do not lie when the consequences are insufficient to justify generalization.

What “growing a test” means in a narrow, testable sense

This result does not say that a system now understands product requirements on its own. It establishes a narrower claim:

Within a specified language of properties, a system can use correct-code execution and mutation blind spots, discard some false generalizations, generate a Rust property test, and obtain an independent verdict through Cargo.

The formulation has four necessary parts:

• consequences, not only static code reading;
• counterexamples, not the first plausible hypothesis;
• refusal to judge when a property does not survive scrutiny;
• independent execution of the generated test in the project’s normal toolchain.

Without the last part, it is easy to accidentally measure the quality of your own evaluator rather than the quality of a test.

What this does not prove yet

The limits of the experiment should not be hidden in a footnote:

• the mutations were handwritten in the style of cargo-mutants, not generated by cargo-mutants itself;
• probe-input generators are still manually specified;
• the scale is six functions and eight mutations that survived the weak suite;
• the property vocabulary is predefined and restricted;
• this is a code-world gate, not evidence across an entire repository or arbitrary specifications.

The honest next step is either to integrate the loop into real development, or to replace manual mutations and probes with independent tools. It would be wrong to conclude that autonomous testing is solved.

Still, this is stronger than assertion autocomplete. The system does not merely write test text. It has to earn the right to write it: first by surviving refutation on correct code.

The final number

Stable regime: kill 1.00 at false-fire 0.00.
Fragile bounds: invented judge net +0.75; static miner net −0.50.

The important number is not only the kill rate. It says something else: when the future leaves the range of what has been lived, an honest ABSTAIN can be more valuable than a test that is confidently wrong.

The code, gate scenarios, and logs are available in AuraSDK. The next part of the series will test a different boundary: whether such loops can preserve a useful trace over time when the system is no longer allowed to look into the future.