Assumption Density Belongs Beside Pass Rate

A coding-agent score that reports only test pass rate is closer to a press release than a measurement. The missing column is an assumption ledger that records invented imports, environment keys, and unauthorized files. Until that ledger travels beside the score, readers cannot tell a solved task from a story the model completed for itself. The method below pins a dataset contract, defines three additive metrics, and records controls so the resulting numbers stay measurements.

Recent talk about agent workflows keeps circling one ordinary failure: the system fills silence with confident guesses. That pattern is not limited to cloud consoles or to chat transcripts from agent demos. It appears whenever a coding model is scored on a repository that never named real dependencies. A green suite can still hide a hallucinated SDK, a guessed secret name, or a helper file the prompt never authorized.

Treating those inventions as free context turns a benchmark into a short story contest with tests attached. Cheap generation makes the problem louder, because extra files cost almost nothing to emit and look like diligence. Technical debt then arrives as undeclared coupling rather than as slow human typing. A public number that ignores that coupling is not a ranking; it is advertising with a decimal point.

The dataset therefore has to be a frozen contract, not a prompt folder polished after every leaderboard shock. Each task should live at a git tag that pins the statement, the allowed interface, the hidden oracle, and a machine-readable allowlist. The model receives the statement and the interface during generation, and it never receives the oracle or the allowlist. After the run, a scorer compares the produced tree with that allowlist and writes a ledger that must be published with the numeric score.

The following task document is a proposed contract, not a result from any live suite, and it is offered only as a reproducible shape.

# tasks/payments-retry.v1.yaml  (proposed contract, unexecuted example)
id: payments-retry
rev: v1.4.0
prompt_sha256: 9f3c88a1b0d2e4f6a8c0
interface:
  module: billing.retry
  function: schedule_retry
  signature: "(payment_id: str, attempt: int) -> RetryPlan"
allow:
  imports: ["datetime", "billing.types"]
  env: []
  files_write: ["billing/retry.py"]
oracle:
  test_path: tests/oracle/test_payments_retry.py
  hidden: true
mutation_budget:
  rename_identifiers: true
  reword_prompt: false

The allowlist is the difference between a test and a rumor about how payment services usually look. When the model imports Redis because queues are common, the ledger records an invented dependency even if the oracle still passes. Pass rate answers whether something ran under hidden tests. The ledger answers whether the agent stayed inside the contract the dataset actually offered. Publishing one column without the other is how marketing numbers acquire false precision.

Metrics should be additive so a later reader can rebuild the headline from the parts without a keynote. The proposed trio is oracle pass rate, assumption violations per thousand generated lines, and interface drift against the pinned signature. Oracle pass rate is binary per task and a mean across the set, and it is not pass@k unless the protocol states both k and the sample rule. Assumption density is counted from AST imports, environment reads, and writes outside the allowed file list. Interface drift catches renamed parameters and extra required kwargs that make a patch useless to the caller the dataset described.

A compact scorer can make those three numbers boring enough to trust. The script below is a proposed harness fragment and is not presented as a measured leaderboard or as a vendor comparison.

# assumption_ledger.py  (proposed, unexecuted example)
from __future__ import annotations

import ast
import hashlib
import json
from pathlib import Path


def file_sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def invented_imports(source: str, allowed: set[str]) -> list[str]:
    tree = ast.parse(source)
    found: list[str] = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                root = alias.name.split(".", 1)[0]
                if root not in allowed:
                    found.append(alias.name)
        elif isinstance(node, ast.ImportFrom) and node.module:
            root = node.module.split(".", 1)[0]
            if root not in allowed:
                found.append(node.module)
    return found


def invented_env_reads(source: str, allowed_env: set[str]) -> list[str]:
    tree = ast.parse(source)
    hits: list[str] = []
    for node in ast.walk(tree):
        if (
            isinstance(node, ast.Subscript)
            and isinstance(node.value, ast.Attribute)
            and node.value.attr == "environ"
            and isinstance(node.slice, ast.Constant)
            and isinstance(node.slice.value, str)
            and node.slice.value not in allowed_env
        ):
            hits.append(node.slice.value)
    return hits


def score_patch(task: dict, patch_path: Path, oracle_ok: bool) -> dict:
    source = patch_path.read_text(encoding="utf-8")
    lines = max(source.count("\n"), 1)
    allowed_imports = set(task["allow"]["imports"])
    allowed_env = set(task["allow"]["env"])
    imports = invented_imports(source, allowed_imports)
    envs = invented_env_reads(source, allowed_env)
    violations = len(imports) + len(envs)
    return {
        "task_id": task["id"],
        "rev": task["rev"],
        "prompt_sha256": task["prompt_sha256"],
        "patch_sha256": file_sha256(patch_path),
        "oracle_pass": int(oracle_ok),
        "assumption_violations": violations,
        "assumption_per_kloc": 1000.0 * violations / lines,
        "invented_imports": imports,
        "invented_env": envs,
    }


if __name__ == "__main__":
    task = json.loads(Path("tasks/payments-retry.v1.json").read_text())
    report = score_patch(task, Path("billing/retry.py"), oracle_ok=True)
    print(json.dumps(report, indent=2))

Controls keep the same lucky patch from being scored as if it were a new discovery on a new day. Temperature is fixed and recorded, and if a vendor hides that knob the protocol records the hiding instead of pretending the value was zero. Each task starts from a detached tag and an empty conversation, because leftover files are undeclared prompt text. Seeds are stored when the API exposes them, and wall-clock duration is observational rather than a quality claim. The command trail below is a proposed ritual for one revision, not a statement about quotas, chips, or lasting availability.

# proposed control ritual for one pinned task revision
git fetch origin tag tasks/payments-retry/v1.4.0
git switch --detach tasks/payments-retry/v1.4.0
git clean -fdx
export EVAL_TEMPERATURE=0
export EVAL_NOTE="empty-conversation, hidden-oracle, allowlist-enforced"
python3 tools/run_agent.py \
  --task tasks/payments-retry.v1.yaml \
  --out "runs/$(date -u +%Y%m%dT%H%M%SZ)"
python3 tools/assumption_ledger.py \
  --run runs/latest \
  --task tasks/payments-retry.v1.yaml
python3 -m pytest tests/oracle/test_payments_retry.py -q

A useful analogy is a locksmith contest that scores only whether the door opened from the hallway. If contestants may mint any key while walking toward the lock, the fastest time says little about skill with the posted cylinder. Coding scores that omit the assumption ledger are that hallway: the door opens, and the invented key is silently added to the legend. Identifier mutation inside a stated budget is the other half of the cylinder, because a static prompt becomes a memorization target the moment a score is public.

The method needs an execution surface that does not already contain yesterday's failed patch, and that is the only place a product belongs in this protocol. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator-supplied facts used here are limited to free model access and a free server option, which are enough to run the contract, the scorer, and the hidden oracle in one throwaway workspace. Readers who already isolate evals on a clean machine can ignore that surface and still keep the ledger format.

Limitations are part of the protocol rather than an apology pasted under a round percentage. The AST walk does not catch dynamic imports, shell-outs, or configuration smuggled through comments, so a determined agent can still hide assumptions. Hidden oracles can be reverse-engineered if tasks are reused without mutation, which is why the contract carries a revision field and a mutation budget. The three metrics are not a substitute for human review of security-sensitive patches, and they say nothing about latency, dollar cost, or license risk. Sample sizes below a few dozen tasks will wobble until a single lucky invention moves the mean.

Teams that need one glossy percentage for a launch post should not use this approach, because the ledger almost always lengthens the story. Organizations without authority to pin a third-party repository at a tag should not pretend a floating main branch is a dataset. Educators who want students to explore freely can keep the oracle visible and drop the allowlist, provided the session is named as a tutorial. Comparative claims that omit temperature, task revision, and violation counts remain marketing even when the arithmetic is internally consistent.

The honest artifact is a JSON object that a stranger can replay without trusting a slide. It names the task revision, the prompt hash, the patch hash, the oracle bit, and the invented symbols. When those fields travel together, a later reader can reject the metric and still reproduce the run. When they do not travel together, the community is asked to applaud a door that opened for reasons nobody wrote down.