DEV Community: Alex

PHP2Shell: the patch still needs an owner

Alex — Fri, 23 Oct 2026 04:00:23 +0000

PHP operators: Apply the current PHP security update, restart or replace affected processes as required, and verify the runtime serving requests. Fixed upstream releases are listed below. A patch does not establish whether an earlier compromise occurred.

In 2024, a PHP developer found the first signs of an out-of-memory problem. A possible exploit was theorized, but there was no concrete attack path.

The PHP Foundation's account describes competing fixes: compatibility breaks, denial-of-service risks and changes across third-party extensions. Specialists had other responsibilities, and the consensus process struggled to settle on an approach. Work stalled and restarted.

The Foundation calls the delay a structural failure involving both decisions and resources.

Calif's report of an unconditional remote-code-execution path on 11 August 2026 made repair the Foundation security team's top priority.

According to OpenAI's account, Calif used GPT-5.6 Sol for its discovery, and Wiz independently reported both flaws. The Foundation also records an independent report of the same root cause from OpenAI and describes using AI to validate and group reports, with people responsible for decisions and patches.

The weakness was known. Fixing it without breaking something else was still a problem.

The patch was not the starting gun

On 21 October, PHP released the fixes. But AWS subsequently identified an exploitation attempt in its MadPot honeypot telemetry dating to 17 October, four days earlier.

AWS says it is unclear how the actor obtained the information or whether these attempts would have succeeded on real-world systems. An investigation into possible compromise needs to look before the public disclosure date.

On 23 October, PHP issued another advisory concerning CVE-2026-115560. The issue requires an unusual combination of session settings and application behavior. PHP recommends disabling "session.upload_progress.enabled" until the next regular bugfix release. That mitigation was already part of the 21 October guidance.

It's another reason to verify not just which version is running, but whether the relevant security guidance was actually applied.

The PHP reports and Sean Heelan's o3-assisted Linux finding in May 2025, CVE-2025-37899 in ksmbd, show what AI-assisted research can find. They do not tell us whether discovery is globally outpacing repair.

The part I recognize from client work

I've worked with medium-sized businesses and larger enterprises. One thing I have repeatedly noticed is how uneven technical competence can be, especially around system administration.

I don't take an IT department or a hosting contract as proof that someone understands the running system. I would ask the same question of an internal team and a provider: can you follow this advisory through to the PHP process serving a customer's request?

Even for a single rented server, I want to know which parts the provider maintains and which parts the customer has to handle.

In a 2026 government study, 57% of UK businesses lacked confidence in at least one of nine basic cybersecurity tasks. This was a self-report measure, not a practical assessment.

Tasks handled by external providers were not counted as skills gaps. The expertise did not have to be on the company's payroll.

What I would expect from that partner

For an incident like PHP2Shell, I would want clear answers to four questions:

Which services and runtimes are affected?
Who has the access, authority and time to deploy the fix?
How will we verify that services and workers use the fixed build?
Who will assess whether earlier exposure needs investigation?

Laravel Forge's advisory explains that its automatic security updates do not patch PHP: those packages come from a separate repository outside the automatic update scope. That maintenance process can be working as configured while PHP remains vulnerable. Updating the PHP packages restarts PHP-FPM, but long-running CLI workers still need restarting to use the fixed runtime.

The company still needs to agree the scope and authorize the work. A partner can take responsibility for a repair without being able to guarantee the security of the whole system.

I also would not put every failure on an administrator. A capable engineer can be blocked by insufficient staffing, an unsupported dependency or management that will not approve downtime. Competence needs resources and the authority to make decisions.

Can the repair work keep up?

By 21 October, researchers had documented the flaws and maintainers had released fixes. Whether a particular business was protected from these flaws still depended on what happened in its own deployment.

I would judge progress in AI-assisted security by whether more affected systems actually get repaired. That requires following the work beyond the research report, through the release and into the services people use.

What happens if our ability to find vulnerabilities grows faster than our ability to understand and fix them?

PHP update reference

The 21 October 2026 releases address CVE-2026-109618 and CVE-2026-109619:

PHP branch	Fixed upstream release
8.2	8.2.35
8.3	8.3.36
8.4	8.4.27
8.5	8.5.12

Follow your supplier's current guidance for distribution backports and older branches. Customer-built container images may need rebuilding and redeployment, in addition to replacing or restarting affected processes.

Sources checked on 23 October 2026. The technical findings and telemetry are the cited researchers' and providers' reports; I have not reproduced the exploits myself.

What ArchKeel caught in our DATAMIMIC EE refactoring

Alex — Wed, 21 Oct 2026 11:34:20 +0000

Sandcastle and mould: building against a declared architecture.

An Include helper in DATAMIMIC EE needed DescriptorLocation. It imported the class directly from an internal Errors module, although the same class was already available through the public API. That tiny choice is the kind of thing I want to catch while a refactoring is still taking shape.

# Before
from datamimic_ee.errors.context import DescriptorLocation

# After
from datamimic_ee.errors import DescriptorLocation

In that package review, 340 architecture tests and 270 contract tests passed. The focused candidate suite passed all 123 tests too. The import was one of three interface violations ArchKeel reported. The package wasn't accepted until they were resolved.

It was the same kind of boundary problem I described in the first article. That CE refactoring had also taught us to question our verification: some comparisons had loaded code from the wrong revision. We invalidated those results and checked the origins of the loaded modules.

A reader asked how many interface violations agents would catch and repair if we moved ArchKeel before commit handover. I can't give a reliable total: our ledger didn't consistently separate autonomous pre-commit repairs from findings raised during review. I can show the EE cases we recorded and what changed because of them.

EE Core parses and executes test-data descriptors and coordinates data sources, exporters and runtime state. We wanted to align its structure with CE, make component responsibilities and APIs explicit, and prepare Authoring and Error Handling for a later migration into CE. We merged the refactoring on October 18.

I develop both DATAMIMIC and ArchKeel. Agents did much of the implementation and additional review. This is our experience using our own tools on a real codebase.

The sandcastle mould

I like the moment when an architecture starts to make sense: you can follow the dependencies, the responsibilities have a home, and the interfaces tell you how to use the pieces. That is the shape I wanted for EE. The sandcastle mould is how I picture giving that shape to an agent before it starts building.

That meant writing down components, responsibilities, public interfaces and permitted dependencies in the contract. ArchKeel checks supported structural rules deterministically against collected source evidence. We can inspect a deviation, repair it and run the same check again.

Without a persistent contract, the decisions live partly in documentation, partly in earlier conversations and partly in whoever reviews the next change. A new agent session has to recover them. A working internal import is easy to accept when the architectural decision behind it isn't in context.

A caller bypassed the intended gate

The public interface was there; the helper had taken a path through the wall beside it. Changing that type-only import left the runtime AST unchanged, and the native interface finding disappeared.

The caller now uses the declared interface. This illustration explains the import boundary; the actual correction was type-only.

The other two findings exposed missing publications of two constants at their already-planned owner. We published those exact symbols through the enclosing interfaces. We didn't open whole packages or add dependency directions.

The archived test logs and review decisions record these corrections. The wider refactoring still had other unresolved findings at that point.

Another EE package exposed two whole-module imports where the contract published named operations, and a private function exposed through a public alias. Review held the package. We corrected the imports and the defining name, preserving the function bodies. The next review confirmed that all three native findings had disappeared without broader access grants.

What happens if those imports stay?

The immediate consequence of the Errors import was small. It didn't change runtime behavior. My concern is what happens as the next changes build on it.

An existing import is an example for the next developer or agent. If more callers copy it, the internal module acquires consumers outside its intended boundary. A later Errors refactoring then has to find and migrate those consumers too. Eventually, changing an implementation detail can become a compatibility decision. Whole-module access creates a similar problem: it exposes more operations than the component intended to offer.

That's the dependency problem I'd expect to prevent. We didn't measure its long-term cost in EE.

For longer refactorings, I want the same architectural decisions to survive many small changes and agent handovers. The contract survives a context reset. The next implementation is checked against it again. Changing the boundary becomes an explicit decision for review.

A hand-written test could catch the Errors import too. Someone would have to define the policy, implement the check and keep it correct through moves, re-exports and incomplete analysis. The EE package already had hundreds of architecture tests; their green result didn't establish the target's interface boundaries.

That is why I built ArchKeel: to reuse the machinery for checking architectural intent across projects. A project supplies its decisions. Counterexamples and checker fixes can then benefit other projects, instead of becoming another private scanner to maintain. EE exposed gaps in that machinery too.

The X-ray view

The Errors import had a clear repair: use the published interface. The target itself needed a harder review. At one checkpoint, CE and EE each had 151 declared components. When I opened the lower levels of the report, I couldn't follow the EE structure all the way through. Nine common scopes were missing, including parts of models, parsers, statements and tasks. Some responsibility fields were generic sentences produced by helper scripts.

I rejected that state. A populated field hadn't given us a specified design. I wanted to see which operation a caller needed, which data crossed the boundary, who owned mutable state and connections, and how failures were handled. The first expanded target still published helpers automatically and contradicted connection ownership. That needed correction too.

That is where the report became useful to me as an X-ray. I could follow the structure into the details and ask the agent about a specific connection or ownership decision. The target was open to that review too.

The highlighted passage is an illustration of a connection worth inspecting, not a recorded EE finding or an ArchKeel screenshot.

The mould and the checker both needed work

The read-only Scripting API gave us another reason to look past the declarations. Review reproduced mutation through returned sets and bytearrays while all thirteen existing API tests passed. We corrected the snapshot boundary and exercised it in 81 focused cases and three real descriptor scenarios. Calling a component read-only hadn't established that behavior; ArchKeel's structural checks couldn't establish it either.

I also had to ask whether we could maintain the mould we were building. The inventory check found stale paths and missing files. We removed 1,156 exact module selectors already covered by the same owner's package selector. Even after that cleanup, the target contained 25,671 physical JSON lines across 37 contracts. Those are lines of text, not a rule count, and they remain a maintenance cost. I want each decision defined once, close to the component that owns it.

We also had to investigate a timeout in ArchKeel's own collector. Profiling and synthetic controls led to an indexed lookup fix in PR #603. The architecture job then passed on the previously failing runner. The planned second stability replay wasn't executed, so that result doesn't establish repeated-run stability. We started on published ArchKeel 2.0.0 and finished on 4.0.1. A missing module detail page in an earlier report also needed a fix in our report packaging.

What we accepted

The final pipeline passed all 37 jobs, including demos, service-backed tests, ML and determinism checks. It recorded 9,231 passing tests.

The final architecture report showed zero declared violations. Its overall declared-rules status was still UNKNOWN. It retained 92 UNKNOWN analyzer records, and validation returned 427 advisories.

Separately, static call resolution was 75.77%. That measures how much of the observed call structure the analyzer could resolve, not how much of the architecture was verified.

For the parser comparison, I wanted to avoid repeating the CE mistake of comparing the wrong code. We checked the origins of all 516 loaded EE modules in each process and compared 1,520 descriptors on two late revisions. Accepted and rejected inputs matched, along with messages, issue codes, typed locations and ordered statement trees. That gives us parser parity for that correction interval. It doesn't establish parity for every generated output from the original baseline to the final merge.

We accepted the refactoring with those limitations visible. The passing behavior suites didn't close the gaps in the architecture evidence. I didn't want zero violations to be mistaken for proof that everything was correct.

Why I'd use it again

I can't put a productivity percentage on this. We didn't run a controlled comparison or keep a complete time ledger. Nor do these examples establish which defects a human reviewer would have found without ArchKeel.

The follow-up work is to identify which manual architecture gates can be replaced by native checks. ArchKeel #602 tracks the gaps. A replacement has to reject a forbidden change, accept a valid control and leave missing evidence unresolved. Runtime and compatibility tests retain their own job.

What I want to leave behind in EE is an architecture the next developer or agent can still understand: why a component owns a piece of work, which interface to use, and where a change belongs. I like building that kind of structure. ArchKeel gave me a way to keep checking it as the agents worked, and to send them back when the design itself wasn't ready.

Three weeks of agentic refactoring: could we trust the result?

Alex — Mon, 12 Oct 2026 16:01:10 +0000

We spent three weeks refactoring DATAMIMIC CE with coding agents. And at some point we discovered that parts of our verification were testing the wrong code.

We had built a before/after comparison to check that the refactoring hadn't changed existing behavior.

Except an editable-install finder registered in Python's sys.meta_path loaded submodules from the current checkout, even though the top-level package came from the original one. The supposedly old implementation was partly running new code. For those modules, the comparison could hide exactly the differences we wanted it to reveal.

We marked the affected results VOID: an early parser comparison covering all 931 XML inputs, plus early direct-run and service results. We kept them in the archive and fixed the setup. We removed the interfering editable-install finder and started checking the origins of loaded DATAMIMIC CE submodules, not just the root package's __file__.

That one was on us.

We use guardrails because coding agents make mistakes. But what happens when the verification itself is wrong? An architecture check doesn't establish which implementation a test process actually loaded. For much of this refactoring, generating code was easier than deciding what should change and establishing whether the result was correct.

I've been developing DATAMIMIC for years. Our DataSourceRegistry, for example, had grown to around 1,000 lines. Runtime contexts, XML statements, script evaluation, file formats, database access. Too much in one place.

We wanted to align Community Edition's architecture with Enterprise Edition, with coding agents doing much of the refactoring.

Agents can change code faster than I can review it. But they don't necessarily understand why we made certain decisions years ago.

With multiple agents working in separate sessions, each with its own context, small local decisions can slowly change the architecture without anyone agreeing to that change.

So we defined the target architecture first. Components, responsibilities, allowed dependencies and public interfaces. A versioned contract in Git, something the agents and I could work against. ArchKeel checks the actual code against that contract.

36 selected Git checkpoints, from 21 September to 12 October 2026. Each tile represents a Python file; rename identities are inferred by Git.

I also develop ArchKeel, so this wasn't an independent evaluation. We'd tested it and run some smaller experiments. I thought its Python support was ready for a larger codebase.

It wasn't. DATAMIMIC exposed weaknesses and edge cases we hadn't seen. Typed dictionaries, callback types, inherited methods. And findings that weren't clear enough to make a decision.

Sometimes we couldn't immediately tell whether the problem was in DATAMIMIC, the architecture contract or ArchKeel.

We were changing DATAMIMIC, our architecture contract and ArchKeel at the same time. I had underestimated how much work verifying ArchKeel would take.

On September 25, an agent working on locale error handling introduced four imports from an internal module:

from datamimic_ce.errors.factory import invalid_locale_error

The import worked. But we'd explicitly decided that services should use the public error interface:

from datamimic_ce.errors import invalid_locale_error

ArchKeel reported four new INTERFACES-ONLY violations. We fixed the imports and ran the check again. All four violations disappeared. No changes to the contract or baseline.

Original agent commit · Correction

It caught the change after the commit. The implementing agent had checked focused tests, lint and types; the architecture check ran later during integration. That left a gap in our agent workflow: this check belonged before the agent handed over its commit.

Four imports aren't a huge issue. But one reasonable local decision had crossed an agreed boundary.

We also had cases where the rule itself needed discussion. The SQL dispatch path, for example, exposed a conflict between our blanket cast ban and the existing dynamic-client contract. We checked the callers and documented a narrow exception to preserve that contract.

We still have to challenge the architecture. ArchKeel should check whether the implementation matches what we've agreed on.

DATAMIMIC is a test-data engine. Customers have existing XML models, and we didn't want the refactoring to silently change their behavior.

We had 931 XML inputs to account for. At one checkpoint we'd reviewed 180. Another 751 were still open.

We linked inputs to existing tests and recorded results, then added focused checks where evidence was missing.

And we had to distinguish between successful executions, expected rejections, known failures, diagnostics and missing local coverage. Some grouped negative tests timed out, so we split them into smaller checks.

Eventually all 931 inputs had an evidence classification. Of these, 874 had paired evidence from both revisions, including 12 with the same pre-existing error. Another 8 were covered by a passing parent run with readback, 1 had an approved behavior change, and 48 were classified as fixture or legacy debt, or had only module-level CI evidence. The technical report preserves the full breakdown.

That doesn't mean all 931 were successfully replayed. I didn't want to turn different kinds of evidence into the same green PASS.

That distinction mattered even more after the module-loading problem from the beginning. We'd challenged the agents' output. We hadn't challenged our own verification setup enough.

On October 12, we merged the refactoring into development. ArchKeel 2.0.0 reported zero declared architecture violations across 488 observed Python files: 471 core files and 17 under resources/.

The final CE component structure at the accepted revision.

The final PR CI passed, including architecture checks, product tests, static analysis, build checks and seeded replay. Some branch-conditioned E2E and release jobs were skipped.

We still had 286 architecture UNKNOWN records: unresolved cases such as inherited methods, property bindings and type aliases. These are separate from the 931 XML inputs.

Those were assessed through separate agent review tasks, but within the same model/provider ecosystem. That's another review pass, not independent external verification.

We also changed our acceptance criteria during the project. Originally the gate required zero material UNKNOWN positions. The final gate accepted zero declared violations with documented assessment of the remaining unknowns. I accepted that because unresolved analyzer cases needed explicit review; forcing the counter to zero could mean changing valid Python just to satisfy the checker. The assessments still record remaining code obligations, and the unknowns remain unknown.

We didn't meet the original gate. We met the revised one.

The codebase didn't get smaller either. Core Python files increased from 437 to 471, and lines of code went up around 3.8%. Some large modules still need work.

So was it successful?

I think responsibilities and boundaries are clearer now. And we have a concrete example where ArchKeel caught an unintended change against the declared architecture.

But that doesn't prove every architecture decision is correct or every existing behavior has been preserved.

What I'd keep is the shared view of the contract and its findings. The saved results identify the source, target and analyzer, so we could inspect the checks themselves rather than rely only on an agent's summary. On October 8, ArchKeel 1.0.0 reported 91 declared violations; on October 12, ArchKeel 2.0.0 reported zero. The checker and target changed in between, so that isn't a clean before-and-after. And zero didn't settle acceptance: the 286 remaining unknowns still needed explicit assessment.

Did the agents make us faster? I don't know.

We spent three weeks on this, but part of that time went into ArchKeel development and fixing the verification. We didn't have a control run or a complete cost ledger, and we didn't track those work streams separately enough to give a defensible percentage. I can't separate their effects well enough to claim a productivity gain.

Next is DATAMIMIC EE Core. We'll start with published ArchKeel 2.0.0, run the architecture check before an agent hands over its commit, and track refactoring work separately from checker development. I want to see how much of what we've learned transfers.

I'd like to spend more time challenging the architecture and less time debugging the checker.

The technical report has the evidence, including the invalidated results, remaining limitations and acceptance changes. As of October 12, the refactoring is merged, but isn't part of a published DATAMIMIC CE release yet.

Where have your own guardrails failed?

We Gave Developers AI Agents. But Kept Their Old Jobs.

Alex — Sun, 11 Oct 2026 05:44:53 +0000

I've been building solutions with software for decades. With today's coding agents I can sometimes get things done in hours that previously took weeks.

We've given our developers access to these tools too.

But the tools don't transfer my domain knowledge, architectural experience or understanding of our customers.

We spend so much time discussing AI capabilities and productivity gains. Much less time discussing whether our engineers and organizations are actually prepared for what comes with them.

We trained developers to close tickets

For years, many development teams worked in a pretty simple way.

Someone defines the requirements. Someone breaks them into tickets. Developers implement them. Tests pass, PR gets merged, next ticket.

Now we give these developers coding agents.

And suddenly we expect them to understand the business domain, challenge requirements, design architectures, guide agents and take responsibility for the outcome.

That's a different job.

Of course, good engineers have always done much of this. But plenty of organizations have spent years separating technical implementation from business decisions.

We wanted developers to focus on their tickets. And now we're surprised when they don't think beyond them?

I don't think every developer needs to become an architect or business analyst. But teams need to rethink where decisions are made and who owns the outcome.

In one of our own experiments, agents produced a buildable workflow. Looked promising. Except an important filter had disappeared.

Expected three rows. Got six.

The implementation looked fine. The business result was wrong.

You need domain knowledge to catch that. And you need to know what to verify.

No amount of generated code changes that.

And then there's the fear

I totally understand why some engineers are worried.

Imagine doing something for 10 or 15 years. You're good at it. You know your tools, your routines, how to approach problems.

Now your manager demonstrates that an agent can finish certain tasks in a fraction of the time.

What does that mean for your job?

AI will change how much engineering work organizations need and what kind of work they value. I don't know exactly how far that will go.

Telling everyone that AI will only make their jobs better would be dishonest.

And people don't change overnight.

Even in IT, where we work with new technology every day, we get comfortable with our routines.

Someone who spent years becoming a good implementation developer might not enjoy suddenly being asked to think about business strategy or architecture.

We can't expect people to make that shift just because we provide better tools.

And we shouldn't underestimate what fear of being replaced does to someone's willingness to experiment, make mistakes or challenge their manager.

What we're doing in our team

I'm in the middle of this myself. I don't have some perfect transformation strategy.

Our developers have access to Codex, Cursor and Claude.

We've also built a shared skill set in Git covering our architectures, standards, best practices and lessons from things we've repeatedly had to correct.

Knowledge that our engineers and agents can reuse instead of fixing the same problems over and over again.

It's part of our harness engineering approach.

In our regular catch-up sessions, I share my experiments. What worked, what failed, where agents produced convincing nonsense and where my expectations were simply unrealistic.

Now we're adding a weekly Agentic Engineering Lab.

60 minutes. Real workflows, hands-on experiments, failures and discussions.

I want to see how our engineers actually work with our harnesses. Where they struggle, what gets in their way and what they would do differently.

And I want critical feedback.

Not everyone agreeing with me because I'm the founder or because I've been doing this longer.

Ideally they'll find approaches better than mine.

We're just starting. We'll see how it works.

Changing how we decide what to build

This is another experiment I'm currently working on. And probably the harder one.

For years, most of our product direction came from me.

Customer feedback, market research, presentations, demos, discussions with customers. I would connect these things and decide what we should build next.

What creates value? What problems are worth solving? What belongs on the roadmap?

Now I'm trying to change how that works.

I still want to provide strategic direction. But I don't want to initiate and drive every feature myself.

I want engineers involved earlier. Understanding the problem, challenging the business value, proposing alternatives and taking more ownership.

We're experimenting with the whole pipeline, from an initial idea or customer problem to a finished feature and release.

Who identifies the problem? Who evaluates it? When do we prototype? Who makes the decisions? And who owns the result?

We're trying different approaches, gradually shifting responsibilities and seeing what works.

And this is where I have to challenge myself too.

I know our domain, the architecture, the history and our customers.

How much of my productivity with agents comes from AI, and how much from that accumulated knowledge?

How do I transfer it without becoming the person everyone still depends on?

And if I want engineers to challenge requirements, do I actually give them the authority to do so?

I also need to accept decisions I might have made differently.

That's something I have to learn as well.

Because giving people more responsibility while keeping all decisions with management doesn't make much sense.

Neither does expecting higher productivity while keeping the same processes, roles and measurements.

If engineers aren't allowed to challenge requirements, Codex won't change that.

So how are you dealing with this?

We're experimenting. I don't know yet what the right organizational setup will look like.

And there are questions I still don't have answers to. What happens when an experienced developer doesn't want this new role? How much ownership can we realistically distribute? And how do we measure whether any of this actually works?

I'm interested in hearing from both sides.

Engineering leaders: What are you changing beyond providing AI tools? Are you changing responsibilities and decision-making, or just expecting higher output?

Engineers: How are you preparing for this shift? Are you getting more ownership? Do you even want it? And what is your employer doing to support you?

Especially interested in what you've tried that didn't work.

ArchKeel After a 121-File Refactoring Experiment

Alex — Tue, 22 Sep 2026 04:36:24 +0000

Before the first coding agent touched DATAMIMIC CE, I froze the target architecture.

The current code disagreed with it in 613 places.

Six and a half hours later, the violation baseline was empty.

Then review found a public API regression and a gate script that could not fail.

This was not another test of whether ArchKeel can print architecture violations. I covered the idea behind the deterministic gate in the first article.

This experiment asked a different question:

Can a target architecture written first as a deterministic contract drive a multi-step refactoring by coding agents without silently moving the target?

I wrote that hypothesis before the target contract and before any refactoring step. The order was part of the experiment.

The protocol

The codebase was DATAMIMIC CE at a fixed commit. The checker was ArchKeel 0.5.1. One agent orchestrated the run, ten implementation-agent runs changed the code, and one separate agent advised on the design. Everything stayed local. No push and no CI.

Each accepted step had to pass four gates:

The target did not widen.
The known architecture debt only shrank.
Existing DSL descriptors kept their observable behaviour.
The relevant lint, type and test results stayed at least at the starting state.

The implementing agents could change code. They could not edit the architecture contract or add entries to the known-violation baseline.

A new violation meant the change was wrong. An unavoidable contract change needed a recorded amendment bound to the exact before and after versions.

The initial measurement found:

613 declared architecture violations
557 baseline entries

83 forbidden component edges
475 imports past component facades
28 violations in the inner model and runtime graphs
1 five-component cycle
24 string dispatches in tasks
1 eval outside runtime

The target described eight parts of DATAMIMIC and every dependency allowed between them.

The target was fixed. The implementation had to move.

The contract drove the work

The refactoring moved vocabulary to its owner, separated workers from tasks, removed runtime dependencies from model objects and exporters, moved source routing into runtime, replaced string dispatch with enums, introduced component facades and finally moved every cross-component import through them.

After each step ArchKeel checked two different things:

Did the known debt shrink?

Did somebody make the target easier to satisfy?

The baseline answered the first question. Comparing the new contract with the previous contract answered the second.

A shrinking violation count is weak evidence if the implementation can grant itself another dependency. The run therefore accepted four explicit amendments, but no new requires edge, relaxed rule or baseline entry.

The baseline moved like this:

613 -> 551 -> 513 -> 483 -> 444 -> 390 -> 353 -> 329 -> 182 -> 0

The run finished after 11 steps beyond the initial measurement, plus later review corrections.

121 product files changed
+2308 / -1788 lines, mostly moves
about 6.5 hours wall clock
0 declared violations
empty baseline
declared_rules: PASS

A full validation over roughly 450 files took around three seconds. The architecture gate was never the slow part.

Of the 907 pre-existing descriptors, 885 could be verified unchanged. The other 22 already varied between repeated executions of the same starting code, so I marked them unverified instead of treating unstable output as evidence.

Some architecture became genuinely better

The empty baseline was not only import rewriting.

The 23-module cross-package cycle between clients, contexts, data sources, exporters, storage, services and statements disappeared.

io, model and domain no longer take Context or SetupContext parameters. They started with 17, 4 and 3 of those parameters.

DataSourceRegistry went from roughly 1,000 lines to 382 lines of scalar loaders. Source routing moved to runtime, where the execution context belongs.

Authoring stopped building a fake SetupContext. The CLI, Python API and dry-run now share one run_descriptor path.

String dispatch in tasks went from 24 cases to zero. eval remained only inside runtime.

ArchKeel also rejected changes that came directly from the design plan. Three “move this verbatim” instructions would have introduced a forbidden construct or external dependency in the new owner. An inner contract rejected another planned move. The plan was not trusted more than the code.

Review found what the architecture result could not show

The overnight run reached the contract. It was still not merge-ready.

After a failed execution, the public Python API returned an empty captured result instead of the rows produced before the failure. The function signature still looked valid. The component dependencies were legal. No architecture rule could distinguish the broken behaviour.

A regression test could. It passed on the starting code, failed on the refactored code and passed again after the correction.

The second failure was in my gate script. It piped test output through tail without pipefail. A red test command could therefore leave the wrapper green.

ArchKeel itself failed closed. The shell around it did not.

The corrected gate now checks every command's own exit state, and a deliberately failing probe proves that the whole gate turns red.

This is the boundary I want to keep:

tests      -> behaviour
type check -> types
ArchKeel   -> architecture

ArchKeel should make the required behavioural checks part of the declared gate. It should not pretend to replace them.

I blamed the agents too quickly

The facade result looked like a clean Goodhart example.

At the start there were 475 imports bypassing component facades. At the end there were none. But three of the resulting facades were mostly re-export barrels:

model.api     114 names
domains.api    46 names
io.api         29 names
runtime.api     5 names

The import paths changed. Much of the coupling did not.

My first explanation was that the agents had found the cheapest way through the architecture gate.

The independent review corrected that explanation.

The implementation brief explicitly prescribed re-export-only facades. The design plan derived their export lists from the names existing consumers already imported. The agents implemented what I asked for. ArchKeel enforced what I specified.

The weak target was mine.

I had defined who may depend on whom:

runtime -> model

I had not defined what model should offer:

model provides:
  parse(...)
  element_definition(...)
  validate(...)

A component graph can enforce dependency direction. It cannot create a useful component API from arrows alone.

What the contract did not measure

Runtime ended in the correct place, but it grew by 28 percent. Context-typed parameters inside runtime rose from 105 to 126. SourceRouter became a 666-line class of static methods. The old registry had moved behind a better boundary. It had not become a better design.

Type quality moved only where a rule required it:

Any annotations       245 -> 247
broad except           56 -> 56
cast                   10 -> 10
type: ignore           27 -> 27

string dispatch
inside tasks           24 -> 0

The logical component graph also said nothing about the physical Python layout. At the end the package root still contained 22 packages and 8 modules. Model code remained spread over model, parsers, statements, constants and enums. Runtime remained spread over runtime, tasks, workers, contexts, services and product_storage.

The contract answered one question well:

dependency architecture
Who may depend on whom?

It did not yet answer two others:

physical architecture -> Where should the code live?

public compatibility -> Which old paths must keep working?

Those dimensions need separate rules and separate evidence. Combining them into one architecture score would hide the difference.

What changes in the next experiment

The next target needs operations before dependency arrows. A facade should declare the questions another component may ask, not re-export every internal name the current consumers happen to use.

It also needs budgets for properties that should shrink even when they cannot reach zero in one run:

re-exported names per facade
cross-component coupling width
single-consumer exports
component size
context-typed parameters
module cycle edges

ArchKeel already has the underlying measurements for some of these. It does not yet ratchet them.

The next physical refactoring should also run in the opposite order: move code into the target package layout first, keep required old paths as explicit compatibility shims, then introduce the real component APIs, then fix dependency direction. This run did the facade work first and touched many imports twice.

Several smaller gaps now have concrete reproductions: facade types need to follow re-exports and inspect fields, string-dispatch checks need to follow module-level constants, private attribute access through an untyped parameter needs to become visible, and facade lifecycle changes must stop requiring manual contract surgery.

The gate belongs in ArchKeel as well. A declared set of architecture, test and type checks should fail closed without depending on a shell script getting every exit code right.

The experiment supports the hypothesis I wrote before the run: a fixed deterministic contract can drive a multi-step refactoring without silently widening the target, while separate gates protect observed behaviour.

It does not support the stronger claim that a dependency contract produces a clean architecture.

I had not specified enough.

The next run will test whether a target that names operations, placement and budgets changes that result.

Tests green, architecture worse: a deterministic gate for coding agents

Alex — Tue, 15 Sep 2026 15:25:04 +0000

My coding agents kept the tests green. The architecture still got worse.

In the DATAMIMIC EE core the agents didn't break the build. They broke the structure. Utilities landed in whatever module was closest, not where they belonged. Code imported past the public interface of another component. And the one that hurt most: clients got imported in places that had no business touching them, above all in the communication between data sources and tasks. In a small task a reviewer catches that. In a large, nested task it hides in a diff that looks reasonable, with every test green.

The problem has two halves. The first is obvious: an architecture change that nobody declared. The second is easy to miss. A change can make the code harder to analyze, so the next report looks clean only because the analyzer sees less of the program. A gate that can't detect when its own visibility gets worse can't distinguish clean code from code it can no longer see.

Why instructions and an LLM judge weren't enough

We build the EE core spec driven. Architecture decisions live in ADRs, and we move them into the skills our agents load. On small tasks that works. On bigger tasks and longer sessions it doesn't hold. A model's blind spots shift with the seed and with how full the context is, and an AGENTS.md or a skill is a request to the agent, not a check on its output.

Asking a model to judge the pull request moves the same problem one level up. The answer changes when you rephrase the question, and an agent can argue with it. A gate that an agent can talk its way around isn't a gate.

Architecture tests aren't new either. ArchUnit, import-linter and dependency-cruiser check whether one snapshot of the code obeys a set of rules, and Archkeel does that too. What I needed on top was a comparison: the accepted state against the candidate, the change against what the agent said it would do, and a hard stop when the evidence itself got worse.

When the code gets blinder

Fixture A in the Archkeel repo is the smallest case. You can run it yourself.

 def run(key: str) -> int:
-    return first() + second()
+    handlers = {"first": first, "second": second}
+    return handlers[key]()

The tests pass. No new forbidden import, no cycle, no private access. A gate that only compares findings says nothing changed.

The architecture may still be fine. What changed is that a static analyzer can no longer prove it: two resolved calls became one unresolved call. For a deterministic gate, that's enough to reject an undeclared change.

expectation_fulfilled: FAIL
regression check failed in calls_unresolved: 0->1
regression check failed in unresolved_ratio: 0/2->1/1

The ratio is compared with integer cross-multiplication. No rounded percentages. No score.

What the gate checks

You describe a target architecture in a contract: components, the packages they own, the names each one makes public, and rules. Every ordered pair of components needs a decision, allowed or forbidden, each with a written reason. A pair nobody decided is an open decision, and validation stays red until it's gone.

Archkeel keeps its verdicts separate and never blends them into one number. Three of them carry the idea:

Verdict	Question
`observation_complete`	Did the scan see everything it claims to see?
`declared_rules`	Does the code obey the contract?
`expectation_fulfilled`	Did the change match what was declared, without regressions?

The third one is built for agents. Before an agent submits an implementation, it commits an expectation: what it intends to change in the architecture. Archkeel checks Git ancestry and the merge request history on the host to verify that the expectation was published before the first submission. An agent that writes the expectation afterwards gets rejected even when its code is clean. That's Fixture B.

Exit codes are boring on purpose. 0 passes, 1 rejects, 2 means Archkeel couldn't verify the input, always with a diagnostic that names the subject and a remedy. Unknown never becomes green.

The architect owns the target

My first onboarding got this wrong. init read today's imports and wrote them into the contract as the architecture. That report could only pass. It described the code, including every shortcut the agents had already taken.

In 0.3.0 init proposes components and their public interfaces, and it writes no dependency rule at all. Every pair is an open decision, listed by how many import sites use it. An import the code already has isn't a decision. If the target forbids it, the first report shows it red, and that's exactly what I want to see on day one.

Who decides? The architect. The packaged skill runs in two modes. In interview mode the agent reads the ADRs and architecture documents first, prepares every decision with a recommendation, confirms the overall picture once and then asks only about conflicts and gaps. When I choose against its recommendation, it asks why. In auto mode the agent decides alone: documents first, then documented principles, then its own judgment, labeled as such. Every rule records decided_by, and the report says it plainly: "156 of 159 rules decided by the agent, awaiting the architect."

We tried both on a field service app: Python 3.12, FastAPI, async SQLAlchemy on PostgreSQL with PostGIS, Redis and Taskiq for the background workers, OR-Tools for route planning. 13 components. The service's owner acted as the architect.

The first interview overwhelmed the architect. About 20 rounds of unranked questions over 156 component pairs. Some got answered in bulk, and three of those bulk answers contradicted the service's own architecture document. Nobody noticed until a second agent had decided the same 156 pairs blind, without seeing the architect's answers. It matched 140 of them, 89.7%. The three contradictions got fixed. Then the interviewing agent asked why the architect had gone against its recommendation on the remaining mismatches, and five more decisions changed.

That interview is why interview mode looks the way it does now. Recommendations with evidence, the overall picture confirmed once, questions only where there's a conflict or a gap.

The first report of the final target failed with 162 violations. 148 of them are one edge: the use cases import the persistence adapter directly, and both targets forbid that. The run also found five defects in Archkeel that no fixture had shown. One of them counted an import twice when a forbidden dependency and the interface boundary both caught it, and nearly doubled the headline. All five are fixed in 0.3.0. The anonymized contracts, both reports and the pair-by-pair comparison are in the repo under docs/evidence/internal-service/.

That agreement rate is one service, measured once. It's not a general accuracy of auto mode, and I won't sell it as one.

One verdict, two readers

The reviewer gets an HTML report with the decision first, then the evidence. The agent gets the same result as JSON, plus a packaged instruction file from uvx archkeel skill install claude (or codex). I don't want one truth for the machine and a friendlier story for the human.

For the drift itself there's a component flow view. Every card is a component from the contract, every line an import between components that Archkeel observed, drawn by the number of import sites. Teal lines conform to the contract. A dashed red line breaks a rule and carries the rule's id. A dotted amber line is a dependency the code uses that nobody has decided yet.

That's the small shop sample from Archkeel's fixtures, with violations planted on purpose.

It checks itself

Archkeel runs its own gate in CI. Its contract has 6 components, and I decided all 30 component pairs myself: 46 rules, none of them by an agent. It requires exactly one owner per module, no component cycles, and it forbids getattr, hasattr, cast, eval, exec, dynamic imports and type: ignore anywhere in the package. Those are my rules for Archkeel, not defaults you inherit. Every enforcing rule was proven by planting a violation and watching the check fail.

Determinism is measured, not assumed. A test runs the report three times on two clones in different paths, with different hash seeds, working directories, time zones and locales, and requires byte-identical output without normalization. That covers one machine and one Python build. Across operating systems and Python versions I haven't proven it yet.

What it can't do

It won't tell you that two competing implementations of the same idea exist, unless a rule or a regression exposes them. It checks private access through imports, so import pkg; pkg._member slips through. It proves publication order, not that nobody edited privately before publishing. Host evidence comes from GitLab merge requests today; there's no GitHub adapter yet.

About one call in five stays unresolved: 630 of 3,303 on Archkeel itself, 998 of 4,318 on the service. They're counted and reported, never guessed. Runtime behavior, data flow and performance aren't observed at all. They belong in the reason behind a decision, and Archkeel checks that a reason exists, not that it's true.

Why it's open source

Before I wrote a line of Archkeel, I tried what already exists on the EE core: static analyzers, code graph tools, and cleanup helpers with and without an LLM behind them. For what I needed they added overhead instead of focus. None of them gave me the comparison I was after: the accepted architecture against the candidate against the declared intent, with weaker evidence treated as a failure.

So Archkeel started as a guardrail for the agents working on the DATAMIMIC EE core. I extracted it because the problem isn't specific to our codebase. Components declare their public interface, and interface_boundary rejects any import from another component that reaches past it. That's the utility and client problem from the start of this post, turned into a red verdict with a file and a line.

It's MIT licensed. The code is on GitHub at rapiddweller/archkeel, the package on PyPI, and uvx archkeel --help is all it takes to start. Point it at a repository where agents write code and find a verdict that's wrong. Open an issue with it. Those are the cases I want.