DEV Community

Cover image for When Code Bends to Pass Its Tests, the Tests Are the Finding
Konstantin Tarandevich
Konstantin Tarandevich

Posted on Edited on Originally published at site.tarandevich.org

When Code Bends to Pass Its Tests, the Tests Are the Finding

One function in my codebase began with an import inside its body. The comment above it gave the reason: a top-level import would move cited lines.

That comment is not about Python, and not about the function. It is about the tests. A test somewhere expected a certain line of this file to stay exactly where it was, and the code had learned to keep still.

It was not the only one. An independent code audit before the open-source release of SWCraft, run separately by two AI agents, Claude and Codex, counted 5,843 file.py:NNN citations in the architecture model's contract data, each checked by a test. Its summary was blunt: the code had been shaped to its own verification process, not to common practice.

SWCraft is an experimental AUTOSAR architecture-as-code environment I am building, introduced in the opening article of this series. The architecture model lives in version-controlled YAML; deterministic tools validate it and generate ARXML, diagrams and documentation; an LLM agent reads and changes the same model through MCP. It is developed with AI agents, and that turns out to matter here.

What the line numbers were for

The architecture model has a contract: every element kind, every field, its type and its constraints. A contract is only useful if the code honours it, so SWCraft keeps a coverage record. For every field the contract declares, the record says where the code actually reads it:

identifier  →  validator.py:7769
Enter fullscreen mode Exit fullscreen mode

Tests check the record. A declared field that no code reads is a gap; a citation that points at nothing is a lie. Both fail the build.

The intent was sound. The granularity was a line.

What it did to the code

A line citation breaks whenever anything above it changes. Insert one line near the top of a module several thousand lines long and every citation below it is wrong. The tests go red, and someone has to re-pin the numbers.

Re-pinning is tedious, so the code started to avoid it:

  • Imports moved inside functions. A new top-level import shifts every line below it. A local import shifts nothing that is cited.
  • New functions went to the end of the module, not next to the code they belonged with, so no cited line would move.
  • About 120 comments and docstring sentences existed only to explain line stability. The audit first counted 47; the cleanup found the rest.
  • At least 24 commits did nothing but re-pin numbers.

None of these is a bug. Every test passed. The code simply stopped looking like Python written by someone who reads PEP 8, and started looking like Python written to keep a check quiet.

What they actually proved

Less than they seemed to.

A line citation proves that a particular line of a particular file has particular text. It does not prove that the text is a read of the field, that the read is in the function doing the work, or that the function is ever called.

When the citations were finally rewritten, 5 of them turned out to have drifted to the wrong place. The old gates had passed all five. In the same pass, 91 recorded "override reasons", the explanations for why a field was exempt, were stale.

So the precision was partly an illusion. The citations were exact about a location and vague about the thing that mattered.

Cite the symbol, not the line

The fix was to cite what the code means, not where it sits:

identifier  →  validator.py::check_identifier_conformance
Enter fullscreen mode Exit fullscreen mode

The checker parses the named function with Python's ast module and requires that it really reads the declared field. A deleted function fails. A renamed function fails. A function that no longer reads the field fails. A new import at the top of the file does nothing at all.

After the rewrite: 6,595 symbol citations, 0 line citations.

129 local imports went back to the top of their modules. 26 stayed local, each now with a stated category instead of "it would move cited lines".

That had a measurable cost, and I measured it: command start-up became 8 to 13 ms slower, about 350 ms to 360 ms on a Mac, 2 to 4 %. I would pay it again.

The honest trade-off

A symbol is coarser than a line. "The field is read somewhere in this function" is a strong statement for a 20-line function and a weak one for a 1,173-line function, which SWCraft has.

The answer is not to go back to line numbers. It is to split the giant functions, which is the next part of the same cleanup. They are rare: 9 functions of about 6,500 are longer than 500 lines, and 99 % are under 200. A precise citation into a function nobody can read was never the real precision anyway.

The second lesson: a rule that produced 370 exceptions

Once the line numbers were gone, I wanted a rule that would stop them from coming back. The first version was simple: a test does not read source text.

Applied literally, it made 370 tests in 107 files into exceptions.

They were not the problem. They parse source with ast to check structure: that one package does not import another, that a forbidden call does not appear, that platform-specific code stays where it belongs. That is ordinary static analysis, and a project of this size needs it.

When exceptions pile up like that, the exceptions are not wrong. The rule is. That judgement was mine; the agents had applied the rule exactly as written.

Reading source was never the defect. The defect was what some tests asserted. So the rule was rewritten to judge the assertion:

  • Forbidden: pinning a line number, or requiring an exact phrase in documentation, docstrings, comments or source.
  • Allowed: checks of code structure, which is static analysis, and bans on tokens in user-facing files.

Under that rule, the tests that did break it were cleaned up:

What the tests did Count What happened
Only pinned a phrase 264 Deleted
Checked consistency between records 113 Moved to the internal suite
Pinned text where behaviour was the point 15 Rewritten to check behaviour

Each of the 15 rewritten tests was proven the same way: a fault was planted on purpose, and the test had to catch it. A rewritten test that catches nothing is just a shorter way to pass.

A ratchet, so it does not grow back

A rule in a document is a wish. Both lessons went into lint, which runs on every task and in CI before every merge:

  • Line citations: a zero check. Any file.py:NNN in the tracked code and documentation fails lint. There is no baseline and no allow-list. Dated records (plans and decision records) keep the citations of their date.
  • The test rule: a ratchet. A scanner reads every product test and flags one that requires an exact phrase of documentation, a docstring, a comment or source. The exceptions that remain are listed in a baseline that may only shrink: the update step lowers values and drops entries, and refuses any new one. After the cleanup the list went from 106 entries to 26, in 14 files.
  • The same ratchet holds the code's shape: function length, complexity, module size and the mypy error count per package may only go down. That is what keeps the giant functions from getting bigger while they wait to be split.

The ratchet has limits, and they are written into it. The zero check does not see a citation spelled another way, such as "line 12 of the validator". The scanner reads Python tests only; the TypeScript tests of the editor extension are not covered yet.

Nobody designed this

Nothing here is new to anyone who writes tests for a living. A test that pins a line number or an exact phrase is a change-detector test, a known anti-pattern, and "test behaviour, not implementation" is standard advice. I am not reporting a discovery. What interests me is how a well-known anti-pattern got into a project where every piece of work went through a written plan, review rounds and a CI gate before it merged, and how it grew to thousands of instances.

This was not code written without checks. It was code written with too many of the wrong ones. No one decided to freeze the code.

It accumulated. Every review round found something. The cheapest durable fix each time was the same: add a check that pins it. Pinned things stay fixed. Each step was locally reasonable.

AI agents are very good at exactly this. Ask an agent to make sure a defect never comes back and it will add a check, quickly and correctly, and the check will pass. Then ask it to add a feature, and it will keep the code still enough for every existing check to stay green: an import inside a function, a new function at the end of the file, a comment explaining why. It does not push back on the check, because the check is the specification it was given.

So the checks grew, and the code bent around them, one reasonable commit at a time. From inside, the test suite looked thorough.

There was gating all along. The citation gates were exactly the gate that froze the code. But every check measured the project against its own rules. None measured it against common practice: there was no check of function size, complexity or type errors until the audit.

It took an audit to see that thorough had turned into rigid, and the audit was done by agents too: Claude and Codex, separately. They were not smarter than the agents that wrote the tests. They were asked a different question: not whether the code passes its gates, but how it compares with common practice. The knowledge was never missing. Ask any agent how to write a test and it will say "test behaviour, not implementation". Knowing the rule and applying it while fixing one finding at a time are different things.

That changes what I think a review should ask. "Do the tests pass?" always ended in yes. The better question was: what did the code have to do to make them pass?

The rule I would keep

If code has to be bent to pass a check, that is a finding against the check, not a constraint on the code.

Cite what the code means, not where it sits. Judge a test by what it asserts, not by what it reads. And when an agent adds a check that pins something, ask what it will cost the next change, because the agent will not.


Originally published at site.tarandevich.org. More on SWCraft, an experimental architecture-as-code environment for AUTOSAR Classic, at swcraft.tarandevich.org.

Top comments (0)