DEV Community: Alex

ArchKeel After a 121-File Refactoring Experiment

Alex — Tue, 22 Sep 2026 04:36:24 +0000

Before the first coding agent touched DATAMIMIC CE, I froze the target architecture.

The current code disagreed with it in 613 places.

Six and a half hours later, the violation baseline was empty.

Then review found a public API regression and a gate script that could not fail.

This was not another test of whether ArchKeel can print architecture violations. I covered the idea behind the deterministic gate in the first article.

This experiment asked a different question:

Can a target architecture written first as a deterministic contract drive a multi-step refactoring by coding agents without silently moving the target?

I wrote that hypothesis before the target contract and before any refactoring step. The order was part of the experiment.

The protocol

The codebase was DATAMIMIC CE at a fixed commit. The checker was ArchKeel 0.5.1. One agent orchestrated the run, ten implementation-agent runs changed the code, and one separate agent advised on the design. Everything stayed local. No push and no CI.

Each accepted step had to pass four gates:

The target did not widen.
The known architecture debt only shrank.
Existing DSL descriptors kept their observable behaviour.
The relevant lint, type and test results stayed at least at the starting state.

The implementing agents could change code. They could not edit the architecture contract or add entries to the known-violation baseline.

A new violation meant the change was wrong. An unavoidable contract change needed a recorded amendment bound to the exact before and after versions.

The initial measurement found:

613 declared architecture violations
557 baseline entries

83 forbidden component edges
475 imports past component facades
28 violations in the inner model and runtime graphs
1 five-component cycle
24 string dispatches in tasks
1 eval outside runtime

The target described eight parts of DATAMIMIC and every dependency allowed between them.

The target was fixed. The implementation had to move.

The contract drove the work

The refactoring moved vocabulary to its owner, separated workers from tasks, removed runtime dependencies from model objects and exporters, moved source routing into runtime, replaced string dispatch with enums, introduced component facades and finally moved every cross-component import through them.

After each step ArchKeel checked two different things:

Did the known debt shrink?

Did somebody make the target easier to satisfy?

The baseline answered the first question. Comparing the new contract with the previous contract answered the second.

A shrinking violation count is weak evidence if the implementation can grant itself another dependency. The run therefore accepted four explicit amendments, but no new requires edge, relaxed rule or baseline entry.

The baseline moved like this:

613 -> 551 -> 513 -> 483 -> 444 -> 390 -> 353 -> 329 -> 182 -> 0

The run finished after 11 steps beyond the initial measurement, plus later review corrections.

121 product files changed
+2308 / -1788 lines, mostly moves
about 6.5 hours wall clock
0 declared violations
empty baseline
declared_rules: PASS

A full validation over roughly 450 files took around three seconds. The architecture gate was never the slow part.

Of the 907 pre-existing descriptors, 885 could be verified unchanged. The other 22 already varied between repeated executions of the same starting code, so I marked them unverified instead of treating unstable output as evidence.

Some architecture became genuinely better

The empty baseline was not only import rewriting.

The 23-module cross-package cycle between clients, contexts, data sources, exporters, storage, services and statements disappeared.

io, model and domain no longer take Context or SetupContext parameters. They started with 17, 4 and 3 of those parameters.

DataSourceRegistry went from roughly 1,000 lines to 382 lines of scalar loaders. Source routing moved to runtime, where the execution context belongs.

Authoring stopped building a fake SetupContext. The CLI, Python API and dry-run now share one run_descriptor path.

String dispatch in tasks went from 24 cases to zero. eval remained only inside runtime.

ArchKeel also rejected changes that came directly from the design plan. Three “move this verbatim” instructions would have introduced a forbidden construct or external dependency in the new owner. An inner contract rejected another planned move. The plan was not trusted more than the code.

Review found what the architecture result could not show

The overnight run reached the contract. It was still not merge-ready.

After a failed execution, the public Python API returned an empty captured result instead of the rows produced before the failure. The function signature still looked valid. The component dependencies were legal. No architecture rule could distinguish the broken behaviour.

A regression test could. It passed on the starting code, failed on the refactored code and passed again after the correction.

The second failure was in my gate script. It piped test output through tail without pipefail. A red test command could therefore leave the wrapper green.

ArchKeel itself failed closed. The shell around it did not.

The corrected gate now checks every command's own exit state, and a deliberately failing probe proves that the whole gate turns red.

This is the boundary I want to keep:

tests      -> behaviour
type check -> types
ArchKeel   -> architecture

ArchKeel should make the required behavioural checks part of the declared gate. It should not pretend to replace them.

I blamed the agents too quickly

The facade result looked like a clean Goodhart example.

At the start there were 475 imports bypassing component facades. At the end there were none. But three of the resulting facades were mostly re-export barrels:

model.api     114 names
domains.api    46 names
io.api         29 names
runtime.api     5 names

The import paths changed. Much of the coupling did not.

My first explanation was that the agents had found the cheapest way through the architecture gate.

The independent review corrected that explanation.

The implementation brief explicitly prescribed re-export-only facades. The design plan derived their export lists from the names existing consumers already imported. The agents implemented what I asked for. ArchKeel enforced what I specified.

The weak target was mine.

I had defined who may depend on whom:

runtime -> model

I had not defined what model should offer:

model provides:
  parse(...)
  element_definition(...)
  validate(...)

A component graph can enforce dependency direction. It cannot create a useful component API from arrows alone.

What the contract did not measure

Runtime ended in the correct place, but it grew by 28 percent. Context-typed parameters inside runtime rose from 105 to 126. SourceRouter became a 666-line class of static methods. The old registry had moved behind a better boundary. It had not become a better design.

Type quality moved only where a rule required it:

Any annotations       245 -> 247
broad except           56 -> 56
cast                   10 -> 10
type: ignore           27 -> 27

string dispatch
inside tasks           24 -> 0

The logical component graph also said nothing about the physical Python layout. At the end the package root still contained 22 packages and 8 modules. Model code remained spread over model, parsers, statements, constants and enums. Runtime remained spread over runtime, tasks, workers, contexts, services and product_storage.

The contract answered one question well:

dependency architecture
Who may depend on whom?

It did not yet answer two others:

physical architecture -> Where should the code live?

public compatibility -> Which old paths must keep working?

Those dimensions need separate rules and separate evidence. Combining them into one architecture score would hide the difference.

What changes in the next experiment

The next target needs operations before dependency arrows. A facade should declare the questions another component may ask, not re-export every internal name the current consumers happen to use.

It also needs budgets for properties that should shrink even when they cannot reach zero in one run:

re-exported names per facade
cross-component coupling width
single-consumer exports
component size
context-typed parameters
module cycle edges

ArchKeel already has the underlying measurements for some of these. It does not yet ratchet them.

The next physical refactoring should also run in the opposite order: move code into the target package layout first, keep required old paths as explicit compatibility shims, then introduce the real component APIs, then fix dependency direction. This run did the facade work first and touched many imports twice.

Several smaller gaps now have concrete reproductions: facade types need to follow re-exports and inspect fields, string-dispatch checks need to follow module-level constants, private attribute access through an untyped parameter needs to become visible, and facade lifecycle changes must stop requiring manual contract surgery.

The gate belongs in ArchKeel as well. A declared set of architecture, test and type checks should fail closed without depending on a shell script getting every exit code right.

The experiment supports the hypothesis I wrote before the run: a fixed deterministic contract can drive a multi-step refactoring without silently widening the target, while separate gates protect observed behaviour.

It does not support the stronger claim that a dependency contract produces a clean architecture.

I had not specified enough.

The next run will test whether a target that names operations, placement and budgets changes that result.

Tests green, architecture worse: a deterministic gate for coding agents

Alex — Tue, 15 Sep 2026 15:25:04 +0000

My coding agents kept the tests green. The architecture still got worse.

In the DATAMIMIC EE core the agents didn't break the build. They broke the structure. Utilities landed in whatever module was closest, not where they belonged. Code imported past the public interface of another component. And the one that hurt most: clients got imported in places that had no business touching them, above all in the communication between data sources and tasks. In a small task a reviewer catches that. In a large, nested task it hides in a diff that looks reasonable, with every test green.

The problem has two halves. The first is obvious: an architecture change that nobody declared. The second is easy to miss. A change can make the code harder to analyze, so the next report looks clean only because the analyzer sees less of the program. A gate that can't detect when its own visibility gets worse can't distinguish clean code from code it can no longer see.

Why instructions and an LLM judge weren't enough

We build the EE core spec driven. Architecture decisions live in ADRs, and we move them into the skills our agents load. On small tasks that works. On bigger tasks and longer sessions it doesn't hold. A model's blind spots shift with the seed and with how full the context is, and an AGENTS.md or a skill is a request to the agent, not a check on its output.

Asking a model to judge the pull request moves the same problem one level up. The answer changes when you rephrase the question, and an agent can argue with it. A gate that an agent can talk its way around isn't a gate.

Architecture tests aren't new either. ArchUnit, import-linter and dependency-cruiser check whether one snapshot of the code obeys a set of rules, and Archkeel does that too. What I needed on top was a comparison: the accepted state against the candidate, the change against what the agent said it would do, and a hard stop when the evidence itself got worse.

When the code gets blinder

Fixture A in the Archkeel repo is the smallest case. You can run it yourself.

 def run(key: str) -> int:
-    return first() + second()
+    handlers = {"first": first, "second": second}
+    return handlers[key]()

The tests pass. No new forbidden import, no cycle, no private access. A gate that only compares findings says nothing changed.

The architecture may still be fine. What changed is that a static analyzer can no longer prove it: two resolved calls became one unresolved call. For a deterministic gate, that's enough to reject an undeclared change.

expectation_fulfilled: FAIL
regression check failed in calls_unresolved: 0->1
regression check failed in unresolved_ratio: 0/2->1/1

The ratio is compared with integer cross-multiplication. No rounded percentages. No score.

What the gate checks

You describe a target architecture in a contract: components, the packages they own, the names each one makes public, and rules. Every ordered pair of components needs a decision, allowed or forbidden, each with a written reason. A pair nobody decided is an open decision, and validation stays red until it's gone.

Archkeel keeps its verdicts separate and never blends them into one number. Three of them carry the idea:

Verdict	Question
`observation_complete`	Did the scan see everything it claims to see?
`declared_rules`	Does the code obey the contract?
`expectation_fulfilled`	Did the change match what was declared, without regressions?

The third one is built for agents. Before an agent submits an implementation, it commits an expectation: what it intends to change in the architecture. Archkeel checks Git ancestry and the merge request history on the host to verify that the expectation was published before the first submission. An agent that writes the expectation afterwards gets rejected even when its code is clean. That's Fixture B.

Exit codes are boring on purpose. 0 passes, 1 rejects, 2 means Archkeel couldn't verify the input, always with a diagnostic that names the subject and a remedy. Unknown never becomes green.

The architect owns the target

My first onboarding got this wrong. init read today's imports and wrote them into the contract as the architecture. That report could only pass. It described the code, including every shortcut the agents had already taken.

In 0.3.0 init proposes components and their public interfaces, and it writes no dependency rule at all. Every pair is an open decision, listed by how many import sites use it. An import the code already has isn't a decision. If the target forbids it, the first report shows it red, and that's exactly what I want to see on day one.

Who decides? The architect. The packaged skill runs in two modes. In interview mode the agent reads the ADRs and architecture documents first, prepares every decision with a recommendation, confirms the overall picture once and then asks only about conflicts and gaps. When I choose against its recommendation, it asks why. In auto mode the agent decides alone: documents first, then documented principles, then its own judgment, labeled as such. Every rule records decided_by, and the report says it plainly: "156 of 159 rules decided by the agent, awaiting the architect."

We tried both on a field service app: Python 3.12, FastAPI, async SQLAlchemy on PostgreSQL with PostGIS, Redis and Taskiq for the background workers, OR-Tools for route planning. 13 components. The service's owner acted as the architect.

The first interview overwhelmed the architect. About 20 rounds of unranked questions over 156 component pairs. Some got answered in bulk, and three of those bulk answers contradicted the service's own architecture document. Nobody noticed until a second agent had decided the same 156 pairs blind, without seeing the architect's answers. It matched 140 of them, 89.7%. The three contradictions got fixed. Then the interviewing agent asked why the architect had gone against its recommendation on the remaining mismatches, and five more decisions changed.

That interview is why interview mode looks the way it does now. Recommendations with evidence, the overall picture confirmed once, questions only where there's a conflict or a gap.

The first report of the final target failed with 162 violations. 148 of them are one edge: the use cases import the persistence adapter directly, and both targets forbid that. The run also found five defects in Archkeel that no fixture had shown. One of them counted an import twice when a forbidden dependency and the interface boundary both caught it, and nearly doubled the headline. All five are fixed in 0.3.0. The anonymized contracts, both reports and the pair-by-pair comparison are in the repo under docs/evidence/internal-service/.

That agreement rate is one service, measured once. It's not a general accuracy of auto mode, and I won't sell it as one.

One verdict, two readers

The reviewer gets an HTML report with the decision first, then the evidence. The agent gets the same result as JSON, plus a packaged instruction file from uvx archkeel skill install claude (or codex). I don't want one truth for the machine and a friendlier story for the human.

For the drift itself there's a component flow view. Every card is a component from the contract, every line an import between components that Archkeel observed, drawn by the number of import sites. Teal lines conform to the contract. A dashed red line breaks a rule and carries the rule's id. A dotted amber line is a dependency the code uses that nobody has decided yet.

That's the small shop sample from Archkeel's fixtures, with violations planted on purpose.

It checks itself

Archkeel runs its own gate in CI. Its contract has 6 components, and I decided all 30 component pairs myself: 46 rules, none of them by an agent. It requires exactly one owner per module, no component cycles, and it forbids getattr, hasattr, cast, eval, exec, dynamic imports and type: ignore anywhere in the package. Those are my rules for Archkeel, not defaults you inherit. Every enforcing rule was proven by planting a violation and watching the check fail.

Determinism is measured, not assumed. A test runs the report three times on two clones in different paths, with different hash seeds, working directories, time zones and locales, and requires byte-identical output without normalization. That covers one machine and one Python build. Across operating systems and Python versions I haven't proven it yet.

What it can't do

It won't tell you that two competing implementations of the same idea exist, unless a rule or a regression exposes them. It checks private access through imports, so import pkg; pkg._member slips through. It proves publication order, not that nobody edited privately before publishing. Host evidence comes from GitLab merge requests today; there's no GitHub adapter yet.

About one call in five stays unresolved: 630 of 3,303 on Archkeel itself, 998 of 4,318 on the service. They're counted and reported, never guessed. Runtime behavior, data flow and performance aren't observed at all. They belong in the reason behind a decision, and Archkeel checks that a reason exists, not that it's true.

Why it's open source

Before I wrote a line of Archkeel, I tried what already exists on the EE core: static analyzers, code graph tools, and cleanup helpers with and without an LLM behind them. For what I needed they added overhead instead of focus. None of them gave me the comparison I was after: the accepted architecture against the candidate against the declared intent, with weaker evidence treated as a failure.

So Archkeel started as a guardrail for the agents working on the DATAMIMIC EE core. I extracted it because the problem isn't specific to our codebase. Components declare their public interface, and interface_boundary rejects any import from another component that reaches past it. That's the utility and client problem from the start of this post, turned into a red verdict with a file and a line.

It's MIT licensed. The code is on GitHub at rapiddweller/archkeel, the package on PyPI, and uvx archkeel --help is all it takes to start. Point it at a repository where agents write code and find a verdict that's wrong. Open an issue with it. Those are the cases I want.