ArchKeel After a 121-File Refactoring Experiment

Before the first coding agent touched DATAMIMIC CE, I froze the target architecture.

The current code disagreed with it in 613 places.

Six and a half hours later, the violation baseline was empty.

Then review found a public API regression and a gate script that could not fail.

This was not another test of whether ArchKeel can print architecture violations. I covered the idea behind the deterministic gate in the first article.

This experiment asked a different question:

Can a target architecture written first as a deterministic contract drive a multi-step refactoring by coding agents without silently moving the target?

I wrote that hypothesis before the target contract and before any refactoring step. The order was part of the experiment.

The protocol

The codebase was DATAMIMIC CE at a fixed commit. The checker was ArchKeel 0.5.1. One agent orchestrated the run, ten implementation-agent runs changed the code, and one separate agent advised on the design. Everything stayed local. No push and no CI.

Each accepted step had to pass four gates:

1. The target did not widen.
2. The known architecture debt only shrank.
3. Existing DSL descriptors kept their observable behaviour.
4. The relevant lint, type and test results stayed at least at the starting state.

The implementing agents could change code. They could not edit the architecture contract or add entries to the known-violation baseline.

A new violation meant the change was wrong. An unavoidable contract change needed a recorded amendment bound to the exact before and after versions.

The initial measurement found:

613 declared architecture violations
557 baseline entries

83 forbidden component edges
475 imports past component facades
28 violations in the inner model and runtime graphs
1 five-component cycle
24 string dispatches in tasks
1 eval outside runtime

The target described eight parts of DATAMIMIC and every dependency allowed between them.

The target was fixed. The implementation had to move.

The contract drove the work

The refactoring moved vocabulary to its owner, separated workers from tasks, removed runtime dependencies from model objects and exporters, moved source routing into runtime, replaced string dispatch with enums, introduced component facades and finally moved every cross-component import through them.

After each step ArchKeel checked two different things:

Did the known debt shrink?

Did somebody make the target easier to satisfy?

The baseline answered the first question. Comparing the new contract with the previous contract answered the second.

A shrinking violation count is weak evidence if the implementation can grant itself another dependency. The run therefore accepted four explicit amendments, but no new requires edge, relaxed rule or baseline entry.

The baseline moved like this:

613 -> 551 -> 513 -> 483 -> 444 -> 390 -> 353 -> 329 -> 182 -> 0

The run finished after 11 steps beyond the initial measurement, plus later review corrections.

121 product files changed
+2308 / -1788 lines, mostly moves
about 6.5 hours wall clock
0 declared violations
empty baseline
declared_rules: PASS

A full validation over roughly 450 files took around three seconds. The architecture gate was never the slow part.

Of the 907 pre-existing descriptors, 885 could be verified unchanged. The other 22 already varied between repeated executions of the same starting code, so I marked them unverified instead of treating unstable output as evidence.

Some architecture became genuinely better

The empty baseline was not only import rewriting.

The 23-module cross-package cycle between clients, contexts, data sources, exporters, storage, services and statements disappeared.

io, model and domain no longer take Context or SetupContext parameters. They started with 17, 4 and 3 of those parameters.

DataSourceRegistry went from roughly 1,000 lines to 382 lines of scalar loaders. Source routing moved to runtime, where the execution context belongs.

Authoring stopped building a fake SetupContext. The CLI, Python API and dry-run now share one run_descriptor path.

String dispatch in tasks went from 24 cases to zero. eval remained only inside runtime.

ArchKeel also rejected changes that came directly from the design plan. Three “move this verbatim” instructions would have introduced a forbidden construct or external dependency in the new owner. An inner contract rejected another planned move. The plan was not trusted more than the code.

Review found what the architecture result could not show

The overnight run reached the contract. It was still not merge-ready.

After a failed execution, the public Python API returned an empty captured result instead of the rows produced before the failure. The function signature still looked valid. The component dependencies were legal. No architecture rule could distinguish the broken behaviour.

A regression test could. It passed on the starting code, failed on the refactored code and passed again after the correction.

The second failure was in my gate script. It piped test output through tail without pipefail. A red test command could therefore leave the wrapper green.

ArchKeel itself failed closed. The shell around it did not.

The corrected gate now checks every command's own exit state, and a deliberately failing probe proves that the whole gate turns red.

This is the boundary I want to keep:

tests      -> behaviour
type check -> types
ArchKeel   -> architecture

ArchKeel should make the required behavioural checks part of the declared gate. It should not pretend to replace them.

I blamed the agents too quickly

The facade result looked like a clean Goodhart example.

At the start there were 475 imports bypassing component facades. At the end there were none. But three of the resulting facades were mostly re-export barrels:

model.api     114 names
domains.api    46 names
io.api         29 names
runtime.api     5 names

The import paths changed. Much of the coupling did not.

My first explanation was that the agents had found the cheapest way through the architecture gate.

The independent review corrected that explanation.

The implementation brief explicitly prescribed re-export-only facades. The design plan derived their export lists from the names existing consumers already imported. The agents implemented what I asked for. ArchKeel enforced what I specified.

The weak target was mine.

I had defined who may depend on whom:

runtime -> model

I had not defined what model should offer:

model provides:
  parse(...)
  element_definition(...)
  validate(...)

A component graph can enforce dependency direction. It cannot create a useful component API from arrows alone.

What the contract did not measure

Runtime ended in the correct place, but it grew by 28 percent. Context-typed parameters inside runtime rose from 105 to 126. SourceRouter became a 666-line class of static methods. The old registry had moved behind a better boundary. It had not become a better design.

Type quality moved only where a rule required it:

Any annotations       245 -> 247
broad except           56 -> 56
cast                   10 -> 10
type: ignore           27 -> 27

string dispatch
inside tasks           24 -> 0

The logical component graph also said nothing about the physical Python layout. At the end the package root still contained 22 packages and 8 modules. Model code remained spread over model, parsers, statements, constants and enums. Runtime remained spread over runtime, tasks, workers, contexts, services and product_storage.

The contract answered one question well:

dependency architecture
Who may depend on whom?

It did not yet answer two others:

physical architecture -> Where should the code live?

public compatibility -> Which old paths must keep working?

Those dimensions need separate rules and separate evidence. Combining them into one architecture score would hide the difference.

What changes in the next experiment

The next target needs operations before dependency arrows. A facade should declare the questions another component may ask, not re-export every internal name the current consumers happen to use.

It also needs budgets for properties that should shrink even when they cannot reach zero in one run:

• re-exported names per facade
• cross-component coupling width
• single-consumer exports
• component size
• context-typed parameters
• module cycle edges

ArchKeel already has the underlying measurements for some of these. It does not yet ratchet them.

The next physical refactoring should also run in the opposite order: move code into the target package layout first, keep required old paths as explicit compatibility shims, then introduce the real component APIs, then fix dependency direction. This run did the facade work first and touched many imports twice.

Several smaller gaps now have concrete reproductions: facade types need to follow re-exports and inspect fields, string-dispatch checks need to follow module-level constants, private attribute access through an untyped parameter needs to become visible, and facade lifecycle changes must stop requiring manual contract surgery.

The gate belongs in ArchKeel as well. A declared set of architecture, test and type checks should fail closed without depending on a shell script getting every exit code right.

The experiment supports the hypothesis I wrote before the run: a fixed deterministic contract can drive a multi-step refactoring without silently widening the target, while separate gates protect observed behaviour.

It does not support the stronger claim that a dependency contract produces a clean architecture.

I had not specified enough.

The next run will test whether a target that names operations, placement and budgets changes that result.