Derivative turns software requirements into executable Python, but generated code is never allowed to certify itself. A separate validation pipelin...
For further actions, you may consider blocking this person and/or reporting abuse
Reporting the frozen blind result as 0/6 verified instead of tuning until it looked better is the part I'd trust most here. One thing to keep in mind when reading it: with 6 cases, 0/6 only rules out a true rate above roughly 46% (exact 95% bound), and 3/3 correct refusals is consistent with a true rate as low as about 29%. So the benchmark can't yet separate "cautious" from "refuses everything".
The number that would separate them is the false-accept rate: of the candidates the external oracle rejects, how many did the gate mark
verified? With zeroverifiedoutputs that rate is untestable by definition. Seeding the next unseen set with candidates that are known-wrong but pass the obvious tests (off-by-one at a boundary, wrong date arithmetic in your 90-day example, a swallowed exception) would test the gate directly, without needing the generator to improve first. Is the adversarial layer's test set derived from the same obligations the contract produced, or independently?V11 produced no verified artifacts, so its false-accept rate is undefined, but earlier blind runs did expose false positives: V9 recorded 4/5, while V10 recorded 0/4.
The adversarial layer combines structural checks with requirement-derived semantic checks and validator-owned probes. We already test deliberate code corruptions, but a broader set of independently seeded behavioral faults would be a useful next experiment.
The key is measuring what the gate actually catches, independently of how well the generator performs.
Four seeded faults is a good start; the thing that makes the number mean something is where they come from. If the corruptions are generated from the same requirements the validator's semantic checks were derived from, a high catch rate partly measures that shared reading of the spec. Mutants written (or at least chosen) by someone who never saw the obligations, plus the boundary and date-arithmetic kinds that pass the obvious tests, give the independent estimate.
Two practical details. Drop equivalent mutants before scoring, otherwise the denominator is padded with faults that cannot be caught. And report it as caught/seeded with an exact interval, the same way as the 0/6: with 9 blind cases in total, V9 at 4/5 against V10 at 0/4 is suggestive but a long way from a stable rate, so it would be worth keeping those cases fixed and adding new ones rather than regenerating the set.
Agreed. The independence of the mutations matters as much as the independence of the validator. Mutants derived from the same compiled obligations could reproduce the same blind spots.Excluding equivalent mutants and reporting caught/seeded with exact intervals makes sense.
One distinction I'd preserve: V9 and V10 remain frozen historical evidence. We can reuse those cases for regression, but only newly frozen, unseen cases should count as fresh blind results. I'd rather measure the validator against independently constructed faults before deciding what needs to change in the architecture.
Agreed on keeping V9 and V10 as regression only. One thing to fix before the new set is built: the number of fresh cases and the decision rule. With so few cases the interval is wide, so write down in advance how many independently constructed faults you will freeze and what caught/seeded result would change the architecture versus not, and only then run them. If the threshold is picked after seeing the result, the new set stops being blind in the same way V9 and V10 have stopped being blind.
Agreed on freezing the experimental rules before running the next blind set. That's consistent with how we've handled the previous benchmarks.
The long-term goal is to distinguish correct from incorrect software reliably, but we're getting there step by step. This is a difficult engineering problem, and each experiment exposes something we can investigate. I'm happy to define sample sizes and evaluation criteria upfront. I'd just avoid treating a statistical threshold as an automatic architectural decision. First establish what actually fails, then understand why, and only then decide what needs to change.
The separation between generation and acceptance is strong, but I think the scope of the acceptance evidence deserves the same scrutiny as the generated artifact. A candidate can satisfy every obligation the validator knows about while still violating an assumption that never made it into the contract. That makes “verification completeness” an interesting property to measure alongside verification success: not only did every declared obligation pass, but how do we know the important requirements survived the translation into those obligations? The frozen contract helps prevent goalpost movement, but a trace from original requirement → obligation → validation evidence could make missing coverage observable rather than leaving it as an implicit assumption.
"The generator is not allowed to say 'I passed'" is the whole lesson for me. I learned it the cheap way, on a Chrome extension that roasts pull requests. Early on, the same model wrote the roast and graded it, and it kept approving jokes about files that weren't even in the diff. Splitting out a separate checker (I call it Jev) helped a lot. Then someone here suggested something almost embarrassingly simple that I'm adding now: before any model judges anything, a plain check that every file it mentions actually exists in the diff. Does Derivative have a cheap non-model gate like that before validation, or does everything go through the obligation contract?
Sure, Derivative already has deterministic, non-model gates. They cover file and path integrity, syntax, imports, entrypoints, interface contracts, manifests, and provenance.these checks are distributed across different stages, alongside obligation-based and adversarial validation. The architecture is considerably broader than the article can fully describe.
The principle is the same as in your example"don't ask a model to judge something that can be checked deterministically."
The acceptance-evidence point is the one I would push on. A separate review agent I ran on a change a build agent had already turned green took 117 s and $0.80, re-ran the unit suites (50 tests), build, lint delta and prettier, and returned 0 important findings and 6 nits. Then the report marked e2e not run, live server and seeded DB needed. Honest about its scope, but the verdict alone read as covering more than it did.
good example of the distinction I'm trying to make with Derivative. A separate review agent can provide useful evidence, but a green verdict is only meaningful within the scope of what was actually tested. Re-running 50 tests doesn't establish the behavior of an untested E2E path. That's why Forge ties acceptance to explicit contracts, executed validation and recorded evidence, while keeping external oracle acceptance separate.
The evidence should define the verdict, not the other way around
The rule that the generator is never allowed to say 'I passed', with only validation evidence authorizing packaging, is the part I would keep. When I built something similar the payoff came later: storing each run's failure evidence and checking that today's validator still catches last week's failures. Do you feed Forge's failures back to improve the generator, or keep the two sides independent on purpose?
Freezing the obligation contract upfront solves the generator moving its own goalposts, but it shifts the hardest boundary upstream to the requirement compiler. If an unstated assumption or interface edge gets dropped during that initial translation into structured constraints, the sandbox runs cleanly and the evidence bundle looks solid, but the build is certifying against an incomplete specification. Reporting the 0/6 external verified rate alongside the infeasibility passes is the right way to benchmark this, because most agent harnesses quietly soften the rubric whenever the pass rate looks unflattering.
I agree that this is a boundary worth testing, but I wouldn't call it the hardest boundary yet.
the frozen obligation contract gives us a stable point to test that hypothesis: compare the original requirement, the compiled obligations, and the resulting evidence without letting later stages move the target. if repeated runs show that assumptions or interface edges are being lost there, then we have a concrete engineering problem to redesign around.
I prefer not to design around a theoretical limit before reaching it. Push the current architecture until the evidence shows where it actually breaks, then change it based on that evidence. That's also why results like 0/6 external verified stay visible, they show what the system has and hasn't demonstrated, without softening the rubric.
Fair, and I'd mostly agree with the stance — with one exception. "Push it until the evidence shows where it breaks" is right when reaching the boundary costs a build. The requirement compiler is the one stage where you don't have to wait, because it's a function you can probe directly: it's the only component in the pipeline that gets no adversarial test at all. The candidate gets a sandbox and a mutation; the translation gets a snapshot diff.
The probe is the same shape as the one you already run one level down. Delete exactly one coordinate from a requirement — a named acceptance criterion, an interface, a persistence clause — recompile, and assert the obligation set loses that clause and nothing else. Three outcomes per axis, and each is informative:
Two things I'd expect to fall out, because they did when I ran the analogous instrument this morning. I have a 37-check consistency gate where each check must reject at least one planted corruption, and the audit prints, per corruption, the list of checks that fired. Two readings from it that bear here. First, "audit clean" means every check has at least one case it rejects — a check that rejects nothing is decoration, and it is invisible in a green report, which is your 7/7 problem arriving from the other side. Second, the check that fires is not always the check the corruption is named after: one corruption labelled for check 8 was caught by check 8's dependency check instead. So the mapping "which obligation witnesses which requirement sentence" has to be recorded per case rather than inferred from names — you already freeze the contract, so the cheap addition is the other direction: which requirement sentence produced each obligation.
That also gives the operational distinction you already make for the sandbox a counterpart in the compiler. You keep
sandbox_unavailableout of the software outcomes because "the evaluation could not legitimately happen" is not a failed requirement — the same is true one stage earlier: "this sentence could not be compiled into obligations on axis X" must not be reported as X satisfied, and should not silently become an empty obligation set either. An empty set passing is the same string as a real pass.Then the 0/6 external verified rate gets a denominator: N of M requirement axes carried into obligations, with the misses named. Curious whether the compiler currently keeps that back-pointer at all, or whether the contract is the only record of what survived the translation — if it's the latter, the dropped-assumption question can only be answered by reading two documents side by side, which is exactly the review nobody does under deadline.
That's a useful experiment, and much of the traceability already exists. Requirement atoms retain source fragments, conditional obligations link back to their parent requirements, and coverage continues through plans, tests, and evidence.
What that doesn't yet prove is that every original requirement survives compilation. A differential test could measure that directly.
One detail matters: requirement IDs are positional, and some clauses legitimately affect multiple contract fields, so the comparison needs semantic matching rather than a raw set diff.
I'd measure those failures first, then change the architecture if the evidence calls for it.
I ran the differential test rather than describing it, because your compiler turned out to be runnable offline —
RequirementCompiler().compile()imports on stdlib alone, no model provider in the path — so requirement survival can be measured directly instead of argued about. Revision cited:855017712cbf.Corpus: the requirements the repo ships, taken by AST rather than by grep —
benchmarks/forge_extended_benchmark.json(30),forge_heldout_benchmark.json(15), theblind_v*case bundles, plus the 19 strings in tests andcore/forge/benchmark.py. 174 unique requirements, split into 2,908 source clauses by my own splitter, which is deliberately not yours — that independence is the point, since a diff between two things that segment identically measures nothing.Does every requirement survive compilation? Measured three ways:
source_fragmentequalityYour caveat is worth 78 percentage points of false alarms. A raw set diff on this compiler reports about four clauses in five as lost when they are not — and the reason is exactly the one you gave, that the atom text is a normalization of the clause rather than the clause. On the 0.4% residue I have to be careful: all 11 sit at 0.50–0.58 content-word overlap, i.e. straddling my threshold, so I can't call them losses. At 0.5 they vanish. The genuine silent-loss rate is ≤0.4% and is not yet distinguishable from my own threshold — that is the honest bound, and separating them needs hand-reading, not a tighter cutoff.
The positional-ID warning is confirmed exactly, and the alias rate is 100%. Prepending one clause ("Support a --verbose flag.") to 40 requirements: 168 IDs kept their number and changed meaning; 0 kept their meaning. Because
R{index:03d}is assigned by index over the clause list, one insertion at the front renames every clause after it. So a raw ID diff on a single well-formed mutation emits 168 false changes — it is not merely imprecise, it is monotone in the insertion position and therefore unusable for exactly the test you proposed. A differential test over IDs has to be run onsource_fragment, or on the semantic matching above; there is no third option in the current design.On plural impact — yours is the majority case, not the exception. With strict containment (an atom's text appears in a field, or the field's text appears in the atom), 87.9% of atoms reach two or more distinct contract-field groups, the modal count being 2 and a further 7% reaching 3. So the "one clause → several fields" allowance isn't an edge case needing a special rule: it is how the compiler works for nine atoms in ten, and a set diff that assumes one field per clause is wrong by construction.
Per-clause responsibility, raw vs semantic. Removing each source clause in turn and recompiling (2,908 perturbations): a raw exact-text diff calls 11.4 atoms "lost" per single-clause removal, essentially all of them collateral from resegmentation; semantic matching at the same threshold reduces that to 0.6 per removal. Same 18× reduction, same cause.
Two things I'd flag rather than claim. First, in 50% of the single-clause removals the semantic diff shows no loss at all — the remaining text re-splits so that some other atom still witnesses the removed clause's content. That is either benign redundancy or the compiler quietly re-absorbing a clause, and those 50% are the sample I would read by hand next; the aggregate cannot tell the two apart. Second, my splitter and yours disagree by construction, so the 2,908 is my segmentation and not a property of your compiler — which is what makes it usable as a differential, and also means the 0.4% is an upper bound on my split's losses, not on yours.
Boundary: no model in the loop, so this measures the deterministic compiler only — the planner and coder stages are untouched and their survival rate is not measured here. Corpus is the repo's own 174; one revision; containment thresholds 0.6 for semantics and exact-substring for fields, both stated because both move the numbers.
This was valuable enough that I ran an independent audit against the exact revision you tested.
The 174 unique requirements are confirmed. I couldn't reproduce your aggregate percentages without the original splitter and matching rules, since independent segmentation methods produced substantially different counts.
But the audit did uncover concrete semantic issues: conditional negation,
unlesshandling, exact-output whitespace, and a downstream validation case where a 30-day requirement was implemented as 90 days without being caught by the local validator.So the investigation produced real findings, even though the aggregate lexical coverage isn't sufficient to establish semantic preservation.
I'll approach these as reproducible regression cases first, without assuming they require an architectural redesign.
Thanks for taking the time to investigate this. If you can share your script and the 11 unmatched clauses, we can compare the results directly.
Thú vị cái cách tiếp cận "không tin tưởng đầu ra của chính mình" — nghe ngược đời nhưng thực tế rất hợp lý khi làm việc với LLM. Pattern này nhắc mình đến self-consistency checking hoặc critique loop mà một số agent framework đang thử áp dụng: sinh code → chạy test/static analysis → nếu fail thì feedback lại cho model sửa, lặp cho đến khi pass hoặc đạt ngưỡng tin cậy.
Câu hỏi nảy sinh: Derivative handle thế nào khi requirement mơ hồ hoặc conflict? Có cơ chế clarification loop quay lại hỏi user không, hay nó cố gắng infer tối đa rồi mới báo uncertain? Cái boundary giữa "auto-fix" vs "cần human-in-the-loop" thường là chỗ dễ bị lạm dụng nhất.
Cũng tò mò về phần executable Python — nó gen thẳng script standalone hay có scaffold project structure (tests, deps, config)? Nếu gen full project thì vấn đề dependency hallucination (import lib không tồn tại, version sai) xử lý ra sao? — found it via LabAgent, site: labagent .tech