DEV Community: Anthony Johnson II

Exit Code 2: How Claude Hooks Turn Agentic Rules Into Runtime Barriers

Anthony Johnson II — Tue, 05 May 2026 21:40:37 +0000

This article was originally published on EthereaLogic.ai.

The first article in this series introduced the five-layer governance stack and made a single load-bearing claim: the layers that live in documents are necessary, and the layers that run as code are what make the system trustworthy. This article goes inside the highest-leverage code layer — runtime enforcement via Claude Hooks — and shows what one looks like at the level of detail an engineering team would actually need to build it.

The thesis of the first article was that an instruction in CLAUDE.md or AGENTS.md can be ignored, reasoned around, or context-windowed out of an agent's working memory, but a hook that exits with status 2 cannot. The thesis of this article is that turning the abstract idea of a hook into a guard that holds under real subagent traffic is its own engineering discipline — one with a small but distinct set of patterns, failure modes, and tests. Most teams who reach the hook layer underestimate that engineering discipline. The empirical evidence in this article comes from one of those underestimations.

Layer 4 sits between the agent's intent and the tool call's effect. The hook receives the tool payload as JSON on stdin, decides allow or block, and on block exits with status 2 — the only exit code the Claude Code harness treats as a hard refusal whose reason is surfaced back to the model.

The Failure Mode Documents Cannot Close

Every team that operates an agentic coding workflow eventually meets the same failure mode: a rule that exists in the project's documentation, in the agent's instructions, in the user's persistent memory, and is still violated by a subagent operating at speed. The rule is not unclear. The rule has not changed. The agent has read it before. None of that prevents the next violation, because none of those locations are points of execution. They are points of intent. The tool call is the point of execution.

In the GovForge project on April 11, 2026 at 19:56 Pacific time, an automated /implement subagent produced commit 3f3b7f9 and pushed it directly to main. The rule "no direct pushes to main" was written in AGENTS.md, in the project Constitution as principle P1 ("protected branches are hard boundaries"), and in user memory as feedback_no_direct_push_main.md. Every one of those locations had been read by the subagent's parent context. The push happened anyway, because three enabling conditions stacked: the project's pre-tool-use.js hook existed in the repository as an empty stub copied from a sibling project's scaffold; .claude/settings.json had no PreToolUse registration, so even a non-empty hook would not have been invoked; and the operator's home-level Claude Code settings had skipDangerousModePermissionPrompt: true, which suppressed the confirmation dialog that would otherwise have caught the destructive operation. Forty-nine minutes later, commit b404fbe replaced the empty stub with a real protected-branch guard and registered it under PreToolUse:Bash in .claude/settings.json. The same class of attempt has exited with status 2 every time since.

The instruction existed. The instruction was not enough. The hook is the enforcement.

What a PreToolUse Hook Actually Is

A Claude Code hook is an executable that the harness invokes around tool calls. The protocol is small enough to describe in three sentences. The harness writes a JSON payload to the hook's stdin describing the tool name and the tool input. The hook decides whether to allow or block, and exits with status 0 to allow or status 2 to block. On a block, the harness reads the hook's stderr and surfaces it to the model as the refusal reason, which gives the agent a written explanation it can reason against on the next turn.

That last detail matters more than it sounds. The hook is not silent enforcement. It is enforcement that explains itself in natural language directly to the model that is now blocked, which means the same hook closes the failure mode and teaches the agent what to do instead — usually within the same turn. In the GovForge guard, the stderr message names the policy file the rule comes from, names the specific sub-command that was blocked, names the reason, and tells the model exactly what fix to attempt: create a chore/..., feat/..., or fix/... branch, retry the commit there, and open a PR. After the hook lands, agents that hit the guard typically resolve it on the next message without operator intervention.

The registration is a small block in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": "node .claude/hooks/pre-tool-use.js" }
        ]
      }
    ]
  }
}

That is the entire interface. The matcher selects which tool calls the hook inspects — Bash here, because every git operation flows through the Bash tool. The command names the executable. There is no SDK. There is no daemon. There is one short script that runs, decides, and exits.

What the Guard Has to Catch

The hard part is not the protocol. The hard part is that "block direct commits and pushes to main" turns out to have a much larger surface than it appears.

The naive implementation inspects the Bash command for the literal token sequence git commit or git push, checks for the word main, and exits 2 if both are present. That guard fails on day one against a competent agent. Within the GovForge guard's 320 lines, six categories of bypass are closed:

Refspec rewriting. git push origin main is the obvious form. git push origin HEAD:main and git push origin +refs/heads/master are the same operation in different syntax. The guard has to parse the refspec, strip a leading +, split on the colon, normalize refs/heads/<branch>, and check the destination side against a PROTECTED_BRANCHES set. The source side is irrelevant; only where the commit lands on the remote matters.

Implicit refspec on a protected branch. git push with no refspec can update the current or upstream branch depending on push.default and the configured upstream. If HEAD is on main, the realistic outcomes all land commits on main. The guard calls git rev-parse --abbrev-ref HEAD and treats a protected current branch as a block when no explicit refspec is present, which closes the entire range of push.default configurations without having to model each one.

Broad-mode flags. git push --all and git push --mirror may update protected branches without the operator naming one. --all pushes every local branch, including main if it is local. --mirror is even broader, syncing refs the remote did not previously know about. Both are blocked unconditionally on the basis that the operator's intent is ambiguous and the failure mode is asymmetric.

Commit-producing subcommands that are not git commit. git merge, git rebase, git cherry-pick, git revert, git am, and git pull all produce new commits on HEAD without invoking git commit directly. A guard that only matches commit lets every one of these through. The GovForge guard maintains an explicit COMMIT_PRODUCING_SUBCOMMANDS set and treats a match against this set as equivalent to git commit when HEAD is on a protected branch. git pull is in the set because it is fetch followed by merge or rebase, which is exactly the failure path the guard exists to block when HEAD is protected.

Nested-shell bypasses. A whitespace tokenizer treats bash -c "git push origin main" as a three-token command where the first token is bash, not git. The naive token check does not fire. The GovForge guard recognizes the nested-shell pattern, extracts the inline payload, and recursively re-runs the full evaluator on the inner command with a depth limit of 4 to bound pathological nesting. The recognized forms are bash -c '...', sh -c '...', and zsh -c '...' — the real -c flags those shells expose — plus a defensive match against a --command= style flag that no listed shell currently supports but that an agent could plausibly hallucinate; the guard rejects the pattern up-front rather than relying on the shell to do it. On a block, the stderr message appends (via nested shell) to the reason so the model can see exactly which layer caught it.

Chained commands. git status && git push origin main is a single Bash invocation but two logical operations. The audited GovForge guard splits sub-commands on &&, ||, ;, and newline before any other inspection, so each fragment is evaluated independently. The first fragment passes; the second blocks the whole call. A single pipe (|) is not in the GovForge splitter — only || is — so a chain that hides a destructive operation behind a pipe (git status | git push origin main) is allowed by the audited guard today. The public templates kit linked at the end of this article ships an updated splitter that treats single-pipe as a separator with quote-awareness; backporting that change into GovForge is the next planned hardening commit.

The shape of those six categories tells you what is hard about hook engineering. It is not the policy. The policy is one sentence. It is enumerating the surface, finding the patterns that look superficially like the policy but are actually different commands, and deciding which side of the line each one is on. The 320 lines of pre-tool-use.js are almost entirely the enumeration.

Each category looks superficially like the policy but is a different command in syntax. Five are closed outright in the GovForge guard; the sixth (chained commands) handles &&, ||, ;, and newline today, with single-pipe added in the templates-kit version of the splitter.

Why The Hook Is Not A Security Boundary

A comment in the GovForge guard names this explicitly: the hook is advisory for the Claude harness, not a security boundary. A motivated human operating outside the harness can defeat it in any number of ways — base64-encode the payload, alias g to git, mutate .git/HEAD directly, set up a sibling clone with a different working tree. The guard does not try to defend against any of those. Trying would balloon the implementation and produce a false sense of containment.

The threat the hook defends against is narrower and more important: an agent operating at speed under the harness, with skipDangerousModePermissionPrompt: true enabled (which is the configuration most operators run because the prompts are too noisy under sustained automation), making a destructive call that violates a written rule. That is the case the GovForge incident produced. That is the case the guard is engineered to make impossible. The guard's value is that it closes a specific failure mode under a specific runtime, not that it is bulletproof against an adversary.

This distinction is worth being explicit about because the alternative framing — "if a determined attacker can bypass it, it is worthless" — leads teams to skip the hook layer entirely, which is the single highest-leverage mistake in the whole governance stack. The right framing is the same one engineers already apply to lint rules, type checkers, and pre-commit hooks: these tools defend against the realistic failure mode of a tired developer at 4 p.m., not against malice. They are load-bearing precisely because the realistic failure mode is the common one.

The Tests Are The Other Half

A hook that exits 2 on the right input and 0 on the right input is necessary but not sufficient. The other thing the GovForge incident proved is that a hook can silently degrade. The empty-stub state at the time of 3f3b7f9 was indistinguishable from a working hook from the operator's perspective: the file existed, the path was correct, the project looked governed. The runtime difference — the file exited 0 on every payload because there was no logic in it — was invisible until a destructive call ran. Nothing in the build system noticed. Nothing in CI noticed. The hook was invisible exactly when invisibility was the failure.

The regression suite that closes that gap is tests/test_pre_tool_use_hook.py — 22 test functions across 354 lines, expanded by pytest parametrization to 32 collected cases, exercising the hook end-to-end as a node subprocess against a real temporary git repository. The first two tests are the load-bearing ones for the empty-stub failure mode. T-1 asserts the hook file is non-empty, contains "use strict", and contains process.exit(2) somewhere — three properties that any working version of the hook satisfies and any empty stub fails. T-2 asserts that .claude/settings.json still registers the hook under PreToolUse:Bash. Together they assert that both enabling conditions of the original incident — the empty file and the missing registration — would be caught by CI before they shipped.

The remaining tests cover the surface enumerated above: explicit-refspec push to main, HEAD:main push, --all and --mirror broad modes, plain push while on a protected branch, commit while on a protected branch, every commit-producing subcommand on a protected branch, nested-shell bypasses through bash -c, sh -c, and the zsh -c family, chained commands with a protected-branch operation in the middle, and a battery of negative cases — push to a feature branch, read-only git commands on main, non-git commands, non-Bash tool calls, empty stdin — that all have to pass through unmodified. Each test instantiates a fresh git init repository under tmp_path and invokes the hook as a subprocess with the JSON payload the harness would send. There is no mocking. The test exercises the same code path the harness does.

Facts

The following are measured facts drawn from the GovForge repository and the public sync record covering the April 11, 2026 incident, verified on May 4, 2026. They should be read within the scope of the GovForge project.

The protected-branch guard at .claude/hooks/pre-tool-use.js is 320 lines of JavaScript with no external npm dependencies — only Node built-ins (node:child_process, node:fs). It is registered under PreToolUse:Bash in .claude/settings.json.
The regression suite at tests/test_pre_tool_use_hook.py is 354 lines containing 22 def test_ functions that pytest expands to 32 collected cases via parametrization (the figure quoted as "32 tests" in the first article in this series). It runs the hook as a node subprocess against a temporary git repository. Two of those functions assert the empty-stub failure mode is caught by CI: hook file non-empty and PreToolUse:Bash registered.
The April 11, 2026 incident — automated /implement subagent pushed commit 3f3b7f9 to main at 19:56 Pacific — was closed by commit b404fbe at 20:45 Pacific, 49 minutes later. Two follow-on commits hardened the guard: 6b11a35 added --all / --mirror broad-push detection, and cffdc57 added the regression suite. A later commit, 038b0e8, extended the commit-producing-subcommand set to include git pull.
The guard's surface includes six categories of bypass: explicit refspecs targeting protected branches, implicit pushes while HEAD is on a protected branch, broad-mode --all / --mirror flags, commit-producing subcommands (merge, rebase, cherry-pick, revert, am, pull), nested-shell wrappers (bash -c, sh -c, zsh -c, plus a defensive --command= match) with depth-limited recursive evaluation up to depth 4, and chained commands split across &&, ||, ;, and newline. Single-pipe (|) separation is a documented gap in the GovForge splitter — closed in the public templates kit at etherealogic.ai/agentic-governance-stack-templates and tracked as the next backport target for GovForge.
Among the six active projects in the development directory carrying the document foundation, GovForge and sdlc_app both wire a PreToolUse:Bash protected-branch guard. The GovForge implementation is the audited exemplar — 320 lines paired with the 354-line regression suite and the GF-PLAN-015 audit document. The sdlc_app implementation is a separately-engineered 278-line variant of the same pattern (shipped as pre-tool-use.cjs to make the CommonJS module type explicit under that project's package.json). ADWS Pro, AetheriaForge, and DriftSentinel keep hook stubs in .claude/hooks/ but do not register them in settings.json. spec-driven-docs-system wires hooks of a different shape — Python documentation pre/post-write validators against Write|Edit matchers, not a Bash protected-branch guard.
The sdlc_app project's doc_pre_write.py is 277 lines. The spec-driven-docs-system project ships a four-file Python hook suite (doc_pre_write.py, doc_post_write.py, doc_post_review.py, hook_utils.py) totaling 751 lines, scoped to documentation files via a path-aware matcher.

Interpretation

The following are engineering judgments drawn from operating the hook layer on these projects. They should be read as claims about the author's experience, not universal prescriptions.

The hook is the highest-leverage single layer in the stack because it is the only place where a written rule becomes structurally impossible to violate inside the harness. Every other layer either explains the rule (documents) or detects the violation after the fact (CI). The hook prevents the violation from landing in the first place. That asymmetry is the entire reason the layer exists, and it is the reason teams that have only the document foundation see the same class of incident repeat across sessions even after the rule has been clarified for the third time.

The empty-stub state is the canonical failure mode and it is caught by exactly two assertions. The first is that the hook file is non-empty and contains the literal process.exit(2) somewhere. The second is that settings.json still has the registration. Both fit in twenty lines of test code. Both run in milliseconds. Adding them to a project's existing test suite is the cheapest single change that retires the original GovForge incident's enabling conditions. If a team is going to write only one piece of code in this whole layer, those two tests are the right one.

Nested-shell handling is the bypass most teams underestimate. It is easy to write a guard that catches git push origin main and miss that bash -c "git push origin main" is the same operation under a different surface. An agent does not have to be adversarial to produce that form — Makefiles, shell scripts, and toolchains shell out routinely, and once the agent reaches for one of those, the inner command is invisible to a token-level inspector. Recursive evaluation with a small depth limit is the right tool. It is also the part of the guard that grows the code most, because the inline payload extraction has to handle multiple quoting forms and the recursion has to re-enter the same evaluator without infinite loops.

The hook explains itself. A surprisingly large fraction of the guard's load-bearing behavior is in the stderr message, not in the exit code. The exit code stops the action. The message tells the agent why and what to do next. A well-written message — name the policy file, name the sub-command, name the reason, name the fix — turns a refusal into a self-correcting cycle. A terse message turns it into a confused retry loop. The work of writing the message is small; the leverage is high.

The hook is project-shaped, not generic. The spec-driven-docs-system project does not have a protected-branch guard because its failure mode is different — the realistic risk is malformed documentation, not destructive git operations, and the hook surface is a Write|Edit matcher running Python validators that check for forbidden patterns, missing structure, and consistency rules. The sdlc_app project carries both shapes: a 278-line PreToolUse:Bash protected-branch guard for the same reason GovForge does, plus the documentation-oriented Write|Edit validators for its own docs surface. The architectural pattern is the same across all three projects. The policy is different in each one. A team copying the GovForge hook into a project with a different risk surface has done the cargo-cult version of this work; the right move is to enumerate the project's own failure modes and write the guard against those.

Practical Implications for Teams Considering the Pattern

If your team has the document foundation and no hooks, write the protected-branch guard first. It is the highest-stakes destructive operation in any agentic coding workflow, and it is the one most likely to be performed by a confident subagent under speed. Start with the simplest correct version: parse the Bash command, check for git commit or git push, look up the current branch, exit 2 if HEAD is on a protected branch and the operation is one of the two. Register it in .claude/settings.json. Add the two-assertion regression test for the empty-stub state. That version is one evening of work and closes the dominant failure mode.

After the basic guard is in place and tested, harden it iteratively. The order I would recommend, based on the GovForge sequence: refspec parsing for HEAD:main and +main forms; broad-mode flag detection for --all and --mirror; commit-producing subcommand expansion to cover merge, rebase, cherry-pick, revert, am, pull; nested-shell recursion for bash -c, sh -c, zsh -c. Each addition is a self-contained commit with its own test. Each addition closes a category of bypass that is unlikely on day one and certain by day thirty.

If your team has hooks but treats them as set-and-forget, write the regression suite. The empty-stub failure mode is silent — the file exists, the configuration looks right, no error is ever raised, and the hook stops working. The only thing that catches it is a test that runs the hook end-to-end and asserts a known-blocked payload actually exits 2. Run that test in CI. Treat a failing hook test as a CI-blocking event with the same severity as a failing unit test. The guard is part of the production surface; it deserves the same treatment.

If you are starting a new project, ship the hook on the first commit. The cost of writing a working protected-branch guard is the same on day one as on day thirty. The cost of not having one is asymmetric: the longer the project runs without it, the more likely a subagent operating at speed produces a destructive operation that the rest of the stack is not positioned to catch. The GovForge incident happened because the hook scaffold was copied from a sibling project but never wired with real behavior. A working hook from day one would have prevented it. An empty stub from day one was worse than no file at all, because it produced the appearance of governance without the enforcement.

The runtime-enforcement layer is where agentic coding stops being a productivity experiment and starts being a system a regulated business can deploy with confidence. The hook is small. The policy is small. The leverage is the largest single asymmetry in the whole stack. The teams that are most ready to deploy agentic coding into production environments are the teams that have understood and crossed this line.

Get the templates

The protected-branch hook described in this article — together with its registration in .claude/settings.json and the regression test suite that catches the empty-stub failure mode — is available as part of the agentic governance starter kit at etherealogic.ai/agentic-governance-stack-templates. The hook is on the page in copy-paste-ready form alongside the document-foundation files from the first article in this series.

The templates page ships an adapted starter version, not a verbatim port of the audited GovForge artifacts. The structural pattern, fact basis, and incident lessons are the same; the differences are deliberate adjustments for public reuse: the .claude/settings.json shape uses Claude Code's current list-of-entries form (rather than the legacy dict-by-matcher form GovForge still carries); the regression suite uses the Python standard-library unittest framework so it runs without pip install in CI; and the splitter treats single-pipe (|) as a separator with quote-awareness, closing the documented gap in the GovForge implementation. The starter is roughly 365 lines of hook code paired with 10 standard-library tests — fewer than the audited GovForge guard's 320 lines + 32 collected pytest cases, scoped to what a team would copy and harden iteratively rather than a verbatim production transplant.

References

Anthropic Claude Code documentation — Claude Hooks specification (PreToolUse / PostToolUse contract, exit-code-2 block protocol) and Settings reference.
AGENTS.md open standard — agentsmd/agents.md, governed by the Linux Foundation's Agentic AI Foundation.
GovForge — internal repository (not public) implementing the guard and regression suite referenced in this article. The .claude/hooks/pre-tool-use.js file, .claude/hooks/README.md, tests/test_pre_tool_use_hook.py, specs/GF-PLAN-015_Hook_Guard_Audit.md, and the incident sync record at report/2026-04-12T03-41-21Z-notion-sync-record.md are the authoritative artifacts; the latter preserves the operator-reported governance note for commit 3f3b7f9. The publicly reusable equivalent is the agentic governance templates page.
First article in this series — CLAUDE.md Is Not Enough: The Governance Stack for Agentic Development.

This is the second article in the EthereaLogic series on the agentic governance stack. The next article covers the external-validation layer in the same depth, including the Codacy, Codecov, and Snyk configurations used in the four production projects, the SHA-pinning practice that closes the mutable-tag supply-chain class, and the rule that the agent's self-report is never authoritative when CI disagrees.

CLAUDE.md Is Not Enough: The Governance Stack for Agentic Development

Anthony Johnson II — Fri, 01 May 2026 04:38:12 +0000

This article was originally published on EthereaLogic.ai.

The standard advice for governing AI coding agents is "write a good CLAUDE.md."

That is like saying the standard advice for software quality is "write good code." Both are true. Neither is sufficient.

I have been building production AI tools with agentic coding as the primary implementation workflow for roughly five months. In conversations with engineering leaders evaluating AI adoption, three concerns surface consistently and in the same order: governance, error rates, and security vulnerabilities. A well-written CLAUDE.md addresses none of them. It addresses orientation — how a coding agent finds its way around the project. Orientation is necessary. It is not governance.

Across the projects in my development directory, the answer to those three concerns has converged on a five-file document foundation — CONSTITUTION.md, DIRECTIVES.md, SECURITY.md, AGENTS.md, and CLAUDE.md — paired with two execution layers that the documents alone cannot supply: runtime enforcement, and external validation. The documents are the layer the agent reads. The execution layers are the layer the system runs. Both are required. This article introduces the full five-layer stack, explains how the document foundation maps onto its first three layers, and shows why the two execution layers are what turn agentic coding from a productivity tool into a system a regulated business can trust. It is the first article in a new EthereaLogic series on the agentic governance stack.

Each layer closes failure modes that the layers above it cannot catch. The top three layers live in documents and templates. The bottom two layers run as code. The distinction between an instruction and an execution barrier is the distinction between orientation and governance.

The Gap Between Orientation and Governance

CLAUDE.md is a navigation file. It tells a coding agent where things live, what commands to run, what conventions the project uses. Done well, it removes the time an agent wastes rediscovering context at the start of every session. AGENTS.md, the open standard now governed by the Linux Foundation's Agentic AI Foundation and adopted by more than 60,000 projects, plays the same role with portability across multiple agent runtimes.

Both are essential. Both are documentation, not policy.

A map tells you where the roads are. It does not tell you which roads you are allowed to take, under what conditions, with what authorization, with what evidence required afterward. Enterprise software engineering has decades of infrastructure for that second layer — coding standards with enforcement, code review requirements, branch protections, audit trails, quality gates that block deployment unless specific criteria are met. We learned long ago that "write good code" was not enough. We needed systems that made good code the path of least resistance and bad code structurally harder to ship.

Agentic coding is at exactly that inflection point. The question is not whether an LLM can write code. The question is whether the system around it is governed well enough that a regulated business can trust the output.

The Five-Layer Stack

The stack I am building toward in every active project has five layers. Each layer answers a question the layer above it cannot. The document-foundation layers are in place across all six active projects today. The two execution layers are deployed end-to-end in GovForge, partially in others, and are the active migration target for the rest. The framework below is therefore the target architecture that recent production experience has crystallized — accurate as a description of where the projects are headed, not as a uniform claim about where each one stands today.

Layer 1 — Navigation Files

CLAUDE.md, AGENTS.md, and where applicable GEMINI.md are the project-specific orientation layer. They handle commands, file maps, technology stack, workflow shortcuts, agent roles, and precedence rules when pattern conflicts arise. I maintain a separate file per model because different agents have different context conventions and different attention to long files. This is the layer most teams already have. It is necessary, and on its own it is the layer that keeps an agent productive within a single session. It is also the layer most likely to change weekly as the project evolves.

Layer 2 — Constitutional Governance

CONSTITUTION.md, DIRECTIVES.md, and SECURITY.md form the policy layer above the prompt.

The Constitution defines the governing principles and a decision order for resolving conflicts between them. When safety and performance disagree, safety wins. When evidence traceability and speed disagree, evidence traceability wins. The ordering is the statement. A constitution without a declared decision order is significantly less useful than one with — projects that list principles as equal peers produce agent behavior that optimizes for whichever principle is locally easiest to satisfy at the moment of decision, not the principle the project most needs to defend.

DIRECTIVES.md converts the Constitution's principles into enforceable rules at three levels: Critical (blocking), Important (requires written justification to bypass), and Recommended. Critical directives include the dual-evidence rule for PASS claims, the no-fabricated-metrics rule, the no-placeholder-content rule for production files, and per-project boundary rules — for example, GovForge's CRIT-003, which forbids the product repository from taking a runtime dependency on any sibling research repository.

SECURITY.md defines what constitutes a vulnerability in this project, how to report it, severity classifications, and response targets. It scopes what is in and out of bounds — explicitly including prompt injection and credential leakage, which are not hypothetical risks in agentic development.

These three files are required reading before substantive work begins. They are referenced from the navigation layer above, but they live in their own files because their mutation rate and audience are different.

Layer 3 — Agent Specialization

.claude/agents/ defines specialized sub-agents — lead software engineer, test automator, technical writer, Python specialist, UX specialist, security engineer, governance engineer. Each has a scoped system prompt that limits its role and sets its evidence standards. A test automator with explicit instructions, a required evidence format, and a no-simulated-data rule closes a failure mode that a general-purpose agent cannot — the test agent that fabricates passing tests is a known and recurring failure in agentic coding. A specialized agent narrows the surface where that failure can occur.

.claude/commands/ defines the slash command library: /prime, /implement, /review, /verify, /audit, /commit, /pull-request. These are not shortcuts. They are policy-encoded workflows. The /verify command does not just run tests; it requires independent confirmation of claims. The /commit command enforces conventional commit format and checks that governance files are intact before allowing the commit to proceed. The command is the contract.

Layer 4 — Runtime Enforcement

This is the layer most teams have not reached.

Claude Hooks are scripts that execute before and after every tool call. The PreToolUse hook runs before Claude takes any action — Read, Write, Edit, Bash, anything. A PostToolUse hook runs after. These hooks have access to the tool payload and can block execution with an exit code before the action lands.

In GovForge, the pre-tool-use.js hook is registered against PreToolUse:Bash and blocks any git commit or git push that would land directly on main or master. It handles nested shell bypasses — bash -c "git push origin main" is caught the same way a direct git push origin main is. The rule was already in AGENTS.md before the hook existed. It did not prevent the failure that motivated the hook.

On April 11, 2026 at 19:56 PDT, an automated subagent operating against an empty pre-tool-use.js stub pushed commit 3f3b7f9 directly to main, violating a rule that was clearly written in AGENTS.md and in user memory. Forty-nine minutes later, commit b404fbe replaced the stub with a real protected-branch guard, registered it under PreToolUse:Bash in .claude/settings.json, and tested it against twelve representative Bash payloads — push and commit variants, chained commands, non-git commands. All twelve behaved as expected. Later hook-hardening commits expanded the guard and its automated test coverage to the current 320-line, 32-test state. The same class of attempt now exits with status 2 before the tool call lands.

The instruction existed. The instruction was not enough. The hook is the enforcement.

The same rule appeared in AGENTS.md and in pre-tool-use.js. Only one of those held when the empty stub met the subagent. Forty-nine minutes later, the second implementation existed and the same class of attempt began exiting with status 2 before the tool call landed.

This is the precise gap the rest of the layers cannot close on their own. An instruction in a document can be ignored, reasoned around, or context-windowed out. A hook that exits 2 cannot. Hooks turn governance from advice into infrastructure.

Layer 5 — External Validation

The final layer is independent of the agent entirely.

The four production projects — ADWS Pro, AetheriaForge, GovForge, DriftSentinel — each run a parallel suite of GitHub Actions checks on every push and pull request. The shape varies by project: GovForge runs three jobs (lint-and-test, codacy, snyk); ADWS Pro decomposes into test, post-merge-signal, and security; DriftSentinel and AetheriaForge upload coverage to Codecov alongside lint and security checks. The principle is constant: a quality job, a static-analysis job, and a dependency-vulnerability job, each running independently from a clean environment, with no access to the agent's session state. Earlier-stage projects in the directory run lighter or differently-shaped CI surfaces — sdlc_app runs a single validate job, spec-driven-docs-system runs a docs-oriented smoke/security/isolated-install triad — and have not yet been brought up to the production-project bar.

If the agent claims tests pass, CI confirms it. If CI disagrees, the claim is unverified.

Two implementation details are load-bearing. First, in the production projects — ADWS Pro, AetheriaForge, GovForge, DriftSentinel — every GitHub Action invoked from the workflow files is pinned to a specific commit SHA, not a version tag. Version tags are mutable, and a supply-chain compromise through a mutable tag is a documented attack vector that GitHub's own secure-use guidance now recommends defending against by SHA-pinning. Pinning to a SHA removes the entire class. Earlier-stage projects in the directory have not all been brought up to that bar yet — sdlc_app and spec-driven-docs-system still resolve some actions by tag — a known gap rather than a deliberate choice. Second, the static-analysis and dependency-scan tools — Codacy, Codecov, and Snyk — produce reports independent of the agent's reporting. The agent can write whatever summary it wants. The external tools generate their own.

This layer makes one assumption: the agent's self-report is not authoritative. That assumption is the one most agentic coding deployments quietly omit, and it is the one that turns the entire stack from "interesting" to "trustworthy."

Why This Makes Output More Deterministic

LLMs are probabilistic by nature. The governance stack does not change that. What it changes is the operating envelope.

With the full stack in place — as it is in GovForge — an agent cannot commit or push directly to main, because the protected-branch hook exits 2 before the tool call lands. It cannot ship placeholder content into enforced scan roots, because the guardrail check fails the build. It cannot mark a test PASS without machine-verifiable output and a human-readable artifact, because the dual-evidence directive blocks the claim. It cannot produce an output that CI will not independently validate, because Codacy, Codecov, and Snyk run from a clean environment with no access to the agent's session. None of these constraints are prompts. None of them depend on the agent reading instructions correctly. They are runtime barriers and external checks. The other production projects sit at the same document foundation but have not yet wired the runtime-enforcement layer to the same depth — that gap is the next active piece of work the rest of this series is being written to support.

The result: agentic coding output that is auditable, traceable, and repeatable. Not because the model is more deterministic — it isn't — but because the system around it constrains the variance to a band the business can tolerate. Constitutional principles set the direction. Directives make principles enforceable on paper. Hooks make directives enforceable in execution. CI makes execution claims enforceable independently. Each layer compounds.

Facts

The following are measured facts drawn from the development directory and the public repositories of the projects referenced, verified on April 30, 2026. They should be read within the scope of those projects.

Six top-level active projects — ADWS Pro, AetheriaForge, GovForge, DriftSentinel, sdlc_app, and spec-driven-docs-system — currently carry the full document-foundation surface (CONSTITUTION.md, DIRECTIVES.md, AGENTS.md, CLAUDE.md, and SECURITY.md).
The GovForge pre-tool-use.js hook is 320 lines and is registered against PreToolUse:Bash in .claude/settings.json. Other hook scripts in the GovForge .claude/hooks/ directory — notification.js, post-tool-use.js, pre-compact.js, stop.js, subagent-stop.js, user-prompt-submit.js — are documented in the project's hook README as scaffolded stubs not currently wired, kept in place for incremental future enforcement.
The April 11, 2026 incident in GovForge — an automated subagent pushing commit 3f3b7f9 directly to main at 19:56 PDT against an empty hook stub — was closed by commit b404fbe at 20:45 PDT, roughly 49 minutes later, replacing the stub with a real guard. The initial commit's validation covered twelve representative Bash payloads; a later commit (cffdc57) added the regression test suite that runs in CI today.
DriftSentinel currently collects 416 tests under pytest and uploads coverage to Codecov on every push.
Across the four production projects (ADWS Pro, AetheriaForge, GovForge, DriftSentinel), every GitHub Action invoked from the workflow files is pinned to a specific commit SHA rather than a version tag.
Two of the six active projects — sdlc_app and spec-driven-docs-system — currently resolve at least some GitHub Actions by version tag rather than SHA. Those gaps are known and unaddressed at the time of writing rather than deliberate exceptions.
Of the six active projects, only GovForge currently wires the protected-branch runtime barrier: its .claude/settings.json registers pre-tool-use.js against PreToolUse:Bash. ADWS Pro, AetheriaForge, and DriftSentinel keep hook scripts in .claude/hooks/ but do not wire them in settings.json. sdlc_app and spec-driven-docs-system wire hooks of a different shape (documentation pre-write checks rather than protected-branch guards). The full five-layer stack as described in this article is therefore implemented end-to-end in one of the six projects today; the document foundation is in place across all six, and the runtime-enforcement and external-validation layers are at varying levels of completion across the remaining five.

Interpretation

The following are engineering judgments drawn from operating the stack across these projects. They should be read as claims about the author's experience, not universal prescriptions.

The single most important distinction in the stack is between layers that live in documents and layers that run as code. The top three layers — navigation, constitutional governance, agent specialization — are documents. The bottom two — runtime enforcement, external validation — are code. Documents are necessary for the agent to know what it should and should not do. Code is necessary for the system to enforce the answer when the agent gets it wrong. Most public agentic-coding content lives entirely in the document layers. The distinguishing element of an enterprise-grade deployment is the code layers underneath them.

The hook layer is the highest-leverage single addition a team can make to a working four-file governance pattern. It is the layer that turns a written rule into a runtime barrier. The GovForge incident is the empirical demonstration: the rule existed in AGENTS.md and in user memory; the hook did not exist; the rule was violated within hours of the project beginning to operate at full speed. Once the hook existed, the same class of violation became impossible to commit, regardless of agent reasoning. The cost of writing the hook was a one-time engineering effort. The cost of not writing it was an actual incident.

The supply-chain hygiene of pinning every GitHub Action to a SHA is one of the lowest-cost, highest-value practices in the stack. It takes minutes per repository. It removes an entire attack class. It is also the practice that distinguishes a CI configuration that has been audited from one that has been copied from a tutorial. Most tutorials use version tags because version tags are easier to read; that ease is the same property that makes them mutable and vulnerable. SHAs trade legibility for integrity. For an agentic project, the trade is straightforward.

The five-layer framing is a sequence, not a checklist. Skipping ahead does not work. A team that wires hooks before authoring a constitution and directives will end up with hooks that enforce the wrong rules, or rules with no agreed-upon source of authority, or both. A team that wires CI before specializing agents will catch failures late, after the agent has already produced and reported on broken artifacts. The order in which the layers appear here is the order in which they tend to pay off, and it is the order in which I introduce them on a new project.

The framework is deliberately model-agnostic at the top and Claude-specific at the bottom. The navigation, constitutional, and external-validation layers work with any agent runtime — that is the AGENTS.md design intent, and it is why CONSTITUTION and DIRECTIVES live in their own files rather than inside CLAUDE.md. The agent-specialization and runtime-enforcement layers are currently Claude-specific because the hook surface and the sub-agent surface are Claude features. Equivalent surfaces are emerging in other agent platforms; the architectural pattern is portable even where the implementation today is not.

Practical Implications for Teams Considering the Pattern

If your team has a CLAUDE.md or an AGENTS.md and nothing else, the next layer to add is constitutional governance. Author a CONSTITUTION with a decision order, derive a DIRECTIVES file from it, and wire the directives to a lightweight repository-level guardrail check — file-presence, marker scan, secret hygiene, complexity budget — that fails the build when a critical directive is violated. That guardrail check is a narrow, repository-scoped script, distinct from the full external-validation suite of Layer 5; both are useful, and both are usually built in that order. This step produces the largest single shift in the agent's behavior under load.

If your team has the four-file governance pattern but no hooks, the next layer to add is runtime enforcement. Begin with a PreToolUse hook that blocks the highest-stakes destructive class — direct commits or pushes to main, deletions outside dist/ or build directories, anything that touches secrets. Test it against nested shell payloads. Register it in .claude/settings.json. The hook does not need to be sophisticated to be load-bearing; it needs to be present, registered, and tested. An empty hook stub is worse than no hook at all because it produces a false sense of governance without the enforcement.

If your team has hooks but is relying on the agent's own test-pass reports for quality assurance, the next layer to add is external validation. Wire a CI workflow with at least one quality job, one static-analysis job, and one dependency-vulnerability job. Pin every action to a SHA. Configure coverage upload to a tool the agent does not control. Treat any disagreement between the agent's self-report and CI as a CI win.

If you are starting a new project from scratch, plan the full stack from day one rather than assembling it in pieces. The layers compose well when introduced together and compose poorly when retrofitted. A scaffold that ships with the navigation files, governance files, agent and command catalogs, hook implementations, and CI workflows already wired produces a project that is governed from its first commit. Retrofitting governance onto an existing agentic project is harder than starting governed, in the same way that retrofitting tests onto an untested codebase is harder than writing tests alongside the code.

The five-layer stack is not a productivity tool. It is a trust tool. Productivity is what an unconstrained agent can produce in an afternoon. Trust is what a regulated business needs before it can ship that production into a customer environment. The gap between the two is what the governance stack closes.

Get the templates

The drop-in starter kit for this stack — CONSTITUTION.md, DIRECTIVES.md, SECURITY.md, AGENTS.md, CLAUDE.md, the protected-branch hook, and a SHA-pinned CI workflow — is published at etherealogic.ai/agentic-governance-stack-templates. Each template is on the page in copy-paste-ready form with download buttons. The page also includes a one-shot install prompt you can hand to a coding agent so it can install the stack in your project autonomously.

References

AGENTS.md open standard — agentsmd/agents.md, governed by the Linux Foundation's Agentic AI Foundation.
Anthropic Claude Code documentation — Claude Hooks and sub-agent specifications.
GitHub Actions secure-use guidance — recommends pinning third-party actions to a full commit SHA to defend against mutable-tag supply-chain risk.
GovForge — public repository implementing the load-bearing examples in this article, including the protected-branch hook and the CI workflow with full SHA pinning.

This is the first article in a new EthereaLogic series on the agentic governance stack. The next article goes deep on the runtime-enforcement layer — what Claude Hooks actually look like in code, how to design a protected-branch guard that handles nested shell bypasses, and what failure modes the hook layer closes that documentation cannot. The article after that covers the external-validation layer in the same depth, including the Codacy, Codecov, and Snyk configurations used in production projects.

Why Semantic Layers Need Distributional Validation, Not Just Schema Validation

Anthony Johnson II — Thu, 16 Apr 2026 18:20:54 +0000

This article was originally published on EthereaLogic.ai.

Semantic layers are supposed to be the trust boundary. The governed interface between messy warehouse tables and the metrics your organization depends on. Whether you are running a dbt Semantic Layer, a BI-platform semantic model, or a natural-language data agent that translates questions into SQL — the implicit promise is the same: if the query goes through the semantic layer, the answer is trustworthy.

That promise rests on assumptions that most implementations do not actually verify.

Governance language in semantic-layer documentation tends to focus on schema contracts, access controls, metric definitions, and freshness SLAs. These are necessary. They are also incomplete. None of them measure whether the underlying model still carries the same distributional signal it carried when the metric definition was authored and validated.

In a previous article, I described why Shannon entropy catches data quality failures that schema validation structurally cannot. In a follow-up, we validated that claim across nearly 6.6 million rows of real-world data in a preregistered benchmark program. This article applies that evidence to an adjacent architecture that is increasingly central to how enterprises consume data: the semantic layer.

Where Semantic Layers Fail Silently

A semantic layer defines metrics as functions of columns. Revenue is SUM(order_amount) where order_status = 'completed'. Active users is COUNT(DISTINCT user_id) where last_login >= CURRENT_DATE - 30. Churn rate is a ratio of cohort counts. These definitions are governed, version-controlled, and tested against expected schemas.

The failure mode that schema validation does not cover is distributional. Consider what happens when the underlying order_status column — which historically carried five distinct values in a roughly stable proportion — quietly shifts to 92% 'completed' because an upstream system changed its default assignment logic. The schema is unchanged. The column is not null. Freshness is on time. The metric definition still compiles and executes. But the filter condition that made the metric meaningful now selects nearly the entire table instead of the intended subset. Revenue is overstated. Every downstream consumer — dashboards, reports, governed agents querying the metric — inherits the error.

This is not a hypothetical edge case. It is a predictable consequence of a monitoring architecture that validates structure without validating signal.

The same failure class applies to natural-language data agents and NL-to-SQL systems. These tools generate queries against governed models, and the governance contract implicitly assures the user that the results are sound. But if the agent constructs a valid query against a model whose underlying distributions have degraded, the answer will be structurally correct and informationally wrong. The agent has no mechanism to detect that the column it is filtering on has lost the distributional variation that made the filter meaningful. Worse, the user receiving a natural-language answer has even less visibility into the underlying data state than an analyst reviewing a dashboard would.

The Distributional Blind Spot

The order_status scenario above illustrates a five-category column collapsing toward a single dominant value. The same principle applies at any cardinality.

Schema validation answers: does the data conform to the expected structure? Freshness validation answers: did the data arrive on time? Neither answers the question that matters most for semantic-layer trust: does the data still carry the same information content it carried when the metric was defined and validated?

This is the question Shannon entropy is designed to answer. Entropy quantifies the information content of a distribution — how much uncertainty (or signal) a column carries. To make the mechanics concrete with a simpler example: a column with four evenly distributed categories carries 2.0 bits of entropy. If that column shifts to 92% concentration in a single value, entropy drops to approximately 0.5 bits. The schema is identical. The information content has collapsed by 75%.

The benchmark program documented in the prior article tested this class of detection systematically. Across three independent real-world datasets — OpenML Adult Income (32,561 rows), NYC TLC Yellow Taxi (3,066,766 rows), and U.S. Census ACS PUMS (3,500,000 rows) — entropy-based drift detection achieved a sensitivity of 1.0 with a false positive rate of 0.0. The row counts are the specific benchmark samples used in each experiment; the full upstream datasets may be larger. Every injected distributional shift was caught. No false alarms were raised. Detection latency matched the evaluation window at 1.0 batch.

On quality validation, the distribution-aware approach achieved precision and recall of 1.0 on all three datasets, while a rule-based baseline modeled after standard constraint-checking patterns dropped to precision 0.6 and F1 0.75 on the Census ACS dataset — the dataset with the most complex distributional characteristics. The gap was not marginal. It was the difference between catching a class of failure and missing it entirely.

These results were produced on DriftSentinel 0.4.2+ and AetheriaForge 0.1.4+, and were reproducible across independent machines: 60 out of 60 gate verdicts matched with bitwise-identical non-latency metrics and matching configuration hashes.

The benchmark support this article depends on, restated in article-specific form: three real-world datasets, perfect drift sensitivity, zero false positives, and 60/60 matched verdicts across independent machines.

A Concrete Scenario: Metric Drift in a Governed Semantic Layer

Consider a governed dbt Semantic Layer that exposes a monthly_recurring_revenue metric defined as SUM(contract_value) filtered on subscription_status = 'active' and grouped by customer_segment. The metric is used by a BI dashboard, an executive reporting pipeline, and a natural-language agent that lets product managers ask questions like "What is MRR for Enterprise customers this quarter?"

The underlying customer_segment column historically carried four values — Free, Starter, Professional, Enterprise — in a distribution that gave the grouping analytical meaning. Each segment represented a materially different population.

Now suppose an upstream CRM migration begins remapping its tier logic. The Starter tier is fully absorbed into Free. The Professional tier is being split: most accounts are reclassified as Free, a smaller portion moves to Enterprise, but the migration is still rolling — roughly 5% of accounts remain classified as Professional pending manual review. The column still has valid values. The schema contract passes. Freshness is on time. The dbt model builds successfully and all tests pass.

But the distribution has collapsed. What was a four-value column with approximately 2.0 bits of entropy is now a three-value column dominated by Free at roughly 85% of volume, with Enterprise near 10% and a residual Professional tail near 5%. Entropy has dropped to approximately 0.75 bits. The customer_segment grouping no longer differentiates populations meaningfully. The MRR metric, grouped by segment, now reports a massively inflated Free tier and a deflated Enterprise tier — not because customer behavior changed, but because the upstream classification shifted mid-migration.

Every consumer of this metric inherits the distortion. The BI dashboard shows a trend break that looks like a business event. The executive report flags a revenue concentration concern. The natural-language agent, when asked "How is Enterprise MRR trending?", returns a number that is technically correct against the current data but misleading against the metric's intended semantics.

A distributional check would have caught this before the metric was served. The normalized stability score — entropy divided by the theoretical maximum for the observed cardinality — would have dropped from approximately 1.0 to approximately 0.47. The detection is driven by the skewed concentration in the surviving values: the 85/10/5 split produces far less entropy relative to its theoretical maximum than a uniform distribution would. DriftSentinel classifies a column as collapsed when the delta between its current normalized score and the baselined score exceeds a configurable threshold (default 0.3). Here the delta is −0.53 — well past that boundary. Under a drift policy with a health score threshold of 0.70, this load would have been gated before it reached the semantic layer. The metric would not have been served until the distributional anomaly was investigated and resolved.

The concrete failure described in the article, visualized directly: a trusted four-segment baseline collapses to three surviving values, the normalized stability score falls to 0.47, and the semantic-layer load is gated before distorted metrics reach dashboards, reports, and natural-language agents.

Practical Implications for Enterprise Teams

If your organization relies on a governed semantic layer — whether dbt Semantic Layer, a BI-platform metric store, or a natural-language data agent backed by governed models — there are specific gaps in the current trust architecture that distributional validation closes.

Metric definitions are only as trustworthy as the distributions they depend on. A metric defined as a filtered aggregation is implicitly a function of the filter column's distribution. If that distribution shifts, the metric's semantics shift with it — even though the definition has not changed. Validating the definition is necessary. Validating the distribution is what makes the output defensible.

Natural-language data agents amplify distributional failures. When an analyst queries a dashboard, they have some visual context for whether the numbers look reasonable. When a natural-language agent returns a single number in response to a question, there is no surrounding context to signal that the underlying data has degraded. The trust surface is smaller, and the consequence of a silent distributional failure is higher.

Run-over-run distributional stability is auditable evidence of metric fidelity. Schema tests and freshness checks produce binary pass/fail signals. A normalized entropy stability score produces a continuous, comparable measure of how much information content a column retains relative to its baseline. This score is auditable — it can be logged, trended, alerted on, and included in data contracts as a governed threshold. It answers the question downstream consumers actually care about: not "did the data arrive?" but "can I still trust the metric?"

Layer-aware coherence thresholds align with the Medallion architecture. AetheriaForge's coherence scoring evaluates information preservation across transformations with configurable layer-specific thresholds — Bronze ≥ 0.5, Silver ≥ 0.75, Gold ≥ 0.95. These are AetheriaForge operating defaults, not Databricks-prescribed standards; the thresholds are configurable per data contract. For semantic-layer models that sit at the Gold level, a coherence threshold of 0.95 means the transformation from Silver to Gold must preserve at least 95% of the source's information content. If a model refresh quietly drops distributional fidelity below that threshold, the coherence gate blocks the refresh before it reaches consumers.

Distributional validation is complementary, not competitive. This is not a replacement for schema tests, freshness monitoring, or access governance. It is the missing layer. Schema validation confirms structure. Freshness confirms timeliness. Distributional validation confirms that the data still carries the signal the metric was designed to measure. The combination is what makes a semantic layer's trust promise auditable rather than aspirational.

Getting Started

Both tools used in the benchmark program are open source and available on PyPI. The benchmark results reported in this article were produced on DriftSentinel 0.4.2+ and AetheriaForge 0.1.4+. The pip install commands below install the latest available release; pin to the benchmarked versions if reproducibility against these specific results is required.

DriftSentinel monitors distribution stability over time using Shannon entropy as its primary signal. Configure monitored columns with a declarative drift policy, set health score thresholds, and gate loads that have lost too much distributional information before they reach downstream consumers.

pip install etherealogic-driftsentinel

GitHub: github.com/Org-EthereaLogic/DriftSentinel

AetheriaForge scores information preservation across transformations. Feed it a source and target DataFrame, and it returns a coherence score — the ratio of entropy preserved through the transformation, capped at the source level so that noisy joins cannot mask information loss elsewhere.

pip install etherealogic-aetheriaforge

GitHub: github.com/Org-EthereaLogic/AetheriaForge

Both tools publish customer impact advisories (DriftSentinel, AetheriaForge) when defects are found that could affect operator decisions. If you are evaluating data quality tooling for governed analytics, look for that kind of transparency. It tells you more about engineering rigor than any feature comparison.

If your semantic layer's trust story stops at schema validation and freshness, you have a measurable blind spot. Distributional validation closes it — and the evidence is now available to back that up.

Anthony Johnson II is a Databricks Solutions Architect and the creator of the Enterprise Data Trust portfolio. He writes about data quality, distribution drift, and the engineering patterns that make data trustworthy at scale.

From Theory to Evidence: Validating Shannon Entropy for Data Quality at Scale

Anthony Johnson II — Tue, 14 Apr 2026 18:22:24 +0000

This article was originally published on EthereaLogic.ai.

In a previous article, I laid out the case for why Shannon entropy — Claude Shannon's 1948 measure of information content — catches data quality failures that schema validation, row counts, and null checks structurally cannot. The theory is clean: entropy measures whether a distribution still carries the signal your downstream logic depends on, not just whether the data arrived in the expected shape.

Theory is a starting point. Evidence is what earns trust.

Over the past several weeks, we ran a structured sequence of experiments to answer a harder question: does entropy-based monitoring actually outperform traditional tools on real data, at real scale, under conditions that matter to production Databricks environments?

The answer, across three independent real-world datasets and nearly 6.6 million rows, is yes — and the margin is not small.

The Research Program

We designed and executed three preregistered experiments with a single governing constraint: every claim must be backed by reproducible, append-only evidence. No retroactive adjustments. No cherry-picked datasets. Every run produces a provenance manifest with configuration hashes, dataset fingerprints, and gate verdicts that can be independently verified.

The experiments tested two capabilities against traditional baselines:

Distribution drift detection — using Shannon entropy stability scores to detect when a column's information content has shifted, compared against a KS-test adapter modeled after the statistical drift detection approach used in Evidently, one of the most widely adopted drift monitoring frameworks.

Data quality validation — using distribution-aware semantic validation to detect source contract violations, compared against a rule-based constraint adapter modeled after the validation patterns in Deequ, the standard quality library for Spark environments. Where the rule-based adapter validates individual values against predefined constraints, the challenger evaluates the full distributional profile of each column — an approach informed by the same information-theoretic principles that underpin entropy-based drift detection.

In both cases, the baselines are simplified adapters designed to isolate the comparison against a specific detection mechanism — not full replicas of the Evidently or Deequ product surfaces.

The benchmark harness injected known faults into real data — schema violations, range violations, volume anomalies, gradual distribution shifts, and abrupt distributional breaks — then measured whether each approach caught them, how quickly, and with what precision.

Three Datasets, Three Domains, One Conclusion

We selected three real-world public datasets that span materially different territory. The row counts below are the specific benchmark samples used in the experiment; the full upstream datasets may be larger.

OpenML Adult Income (UCI) — 32,561 rows of socioeconomic tabular data with categorical features like education level, occupation, and marital status.

NYC TLC Yellow Taxi (January 2023) — 3,066,766 rows of transactional trip data with timestamps, geospatial coordinates, fare amounts, and payment types.

U.S. Census ACS PUMS (2022) — 3,500,000 rows of public demographic and earnings microdata from the American Community Survey.

Combined: nearly 6.6 million rows across three independent data domains.

What the Benchmarks Showed

Drift Detection: Perfect Sensitivity, Zero False Positives

The entropy-based drift detector achieved a sensitivity of 1.0 (caught every injected drift event) with a false positive rate of 0.0 (never raised a false alarm) — across all three datasets. Detection latency matched the baseline at 1 batch.

The KS-test baseline also achieved high marks on detection sensitivity. But the entropy approach matched it on every detection metric while providing something a KS-based approach does not naturally offer: a normalized measure of proportional information capacity that is intuitively comparable across columns with different cardinalities, including unordered categorical data where KS is not natively applicable. A stability score of 0.87 on a column with 4 categories carries the same operational meaning as 0.87 on a column with 100 categories — entropy is at 87% of the theoretical maximum for the observed support.

The throughput advantage was also notable: the entropy-based approach processed data at 1.29x to 2.12x the baseline's throughput across the three datasets.

Quality Validation: Where the Gap Becomes Measurable

On quality validation, the distribution-aware approach achieved precision and recall of 1.0 on all three datasets. The rule-based baseline matched on two of the three — but on the Census ACS dataset, the baseline's precision dropped to 0.6 and its F1 to 0.75, while the challenger maintained perfect scores.

Why did Census ACS expose the gap? The Census dataset has distributional characteristics that make rule-based boundary checks less reliable: overlapping value ranges across demographic categories, high-cardinality categorical fields with skewed distributions, and subtle schema interactions that look normal in isolation but carry measurable information loss when evaluated as a distribution.

A rule-based engine asks "is this value within the allowed range?" A distributional approach asks "does the distribution of values still carry the same information it carried in the trusted baseline?" When the answer to the first question is yes but the answer to the second is no, you have the kind of silent data quality failure that erodes downstream model performance without triggering a single alert.

The latency comparison reinforced this: the distribution-aware approach ran at 37–65% of the baseline's wall-clock time across datasets.

Cross-Machine Reproducibility

Every benchmark was re-run on a second machine — a Mac mini with a fresh dataset download, independent Python environment, and no shared state. The result: 60 out of 60 gate verdicts matched across both machines. Non-latency metrics were bitwise identical.

From Benchmark to Live Execution

In a follow-on experiment, we took the validated controls and executed them against a live, non-production Databricks workspace. Two consecutive replayable runs passed all charter-scoped gates, with a fidelity ratio of 1.0 (every source record accounted for in the output), inline cost measurement, and zero audit violations. This does not constitute production-scale proof — the experiment was explicitly scoped to Bronze-layer validation in a sandbox workspace — but it closes the gap between "this works in a benchmark harness" and "this works on Databricks."

Two consecutive replayable runs in a live, non-production Databricks workspace. All four FAIL-tier gates passed; WARN-tier latency sat at 6.4-6.6% of threshold and cost at 11.2% in both runs. Source: E62 closeout (2026-04-01).

Natural Fault Validation

The third experiment carried a validated_with_caveat evidence tier from the outset, reflecting a deliberately narrow scope. The question was whether the governed pipeline infrastructure could execute end-to-end against a corpus of naturally occurring faults rather than synthetic injections.

We curated a corpus of six naturally occurring Bronze-layer data quality incidents. The full pipeline passed all six preregistered KPI gates. Each lane's held-out set contained one true fault and one clean case; both lanes detected the fault and correctly identified the clean case, yielding held-out recall of 1.0 and false positive rate of 0.0 on each lane independently. The detection adapters used deterministic scoring against pre-adjudicated labels — validating the governed infrastructure, not independent model generalization. Proving that entropy-based detectors catch novel natural faults without prior labeling remains the objective of a planned successor experiment.

What We Learned About Entropy in Practice

Three experiments, hundreds of benchmark run artifacts, and millions of rows later, a few practical lessons emerged:

Normalization is non-negotiable. The stability score — entropy divided by the maximum possible entropy for the observed number of distinct values — is what makes entropy operationally useful. A normalized score of 0.75 means entropy is at 75% of the theoretical maximum for the column's current distinct-value count. DriftSentinel catches category disappearance by comparing the normalized score against the baselined snapshot, so a column that silently drops from 12 categories to 8 will trigger a drift classification even if the surviving 8 remain uniformly distributed.

Layer-aware thresholds match how lakehouses actually work. AetheriaForge ships with default coherence thresholds aligned to Medallion architecture layers: Bronze ≥ 0.5, Silver ≥ 0.75, Gold ≥ 0.95. These are operating defaults, not Databricks-prescribed standards. The thresholds are configurable per data contract, and the right values depend on what each layer is doing to the data.

Entropy and schema validation are complementary, not competitive. Schema validation catches structural defects. Entropy catches distributional defects. You need both. The mistake is assuming that passing schema checks means the data is trustworthy.

Evidence discipline changes the conversation. Every run produced append-only evidence artifacts: JSON bundles with configuration hashes, measured gate values, thresholds, and verdicts. When a downstream consumer asks "how do you know the data is good?", the answer is a specific artifact ID, a specific health score, and a specific gate verdict — queryable, auditable, and immutable.

Applying This in Your Pipeline

Both tools are open source and available on PyPI. The benchmark results reported in this article were produced on DriftSentinel 0.4.2+ and AetheriaForge 0.1.4+, after the defects described in each product's customer impact advisory were resolved.

DriftSentinel uses Shannon entropy as its primary distribution stability signal.

pip install etherealogic-driftsentinel

Org-EthereaLogic / DriftSentinel

Databricks-native data trust pipeline — intake certification, drift gating, and control benchmarking in a single deployable product.

Three Control Patterns. Multiple Datasets. One Platform That Proves All of Them Are Working.

Enterprise Data Trust — Chapter 4: DriftSentinel

Built by Anthony Johnson | EthereaLogic LLC

If this platform is useful to your team, consider starring the repo — it helps others in the Databricks community find it.

The first three chapters of Enterprise Data Trust prove three things: data can be certified at intake, distribution drift can be gated before publication, and control effectiveness can be measured against known failure scenarios. Each chapter solves one problem in isolation.

DriftSentinel solves the next one: running all three control patterns together, across multiple registered datasets, in a production Databricks environment — with append-only evidence for every run and an operator dashboard the platform team can actually use.

Three modules. One registry. Queryable evidence. No assumption that any run passed unless the artifact says so.

Important: If you used DriftSentinel…

View on GitHub

AetheriaForge uses Shannon entropy to score information preservation across transformations.

pip install etherealogic-aetheriaforge

Org-EthereaLogic / AetheriaForge

Databricks-native intelligent data transformation engine — coherence-scored Bronze/Silver/Gold with entity resolution and temporal reconciliation in a single deployable product.

Intelligent Data Transformation. Coherence-Scored. Evidence-Backed.

EthereaLogic Databricks Suite — AetheriaForge

Built by Anthony Johnson | EthereaLogic LLC

If this tool is useful to your team, consider starring the repo — it helps others in the Databricks community find it.

Every Medallion transformation introduces information loss. Most pipelines ignore it. AetheriaForge measures it by transforming source records through schema contracts, scoring the result for coherence, applying optional exact-match entity resolution and latest-wins temporal reconciliation, and recording append-only evidence. Nothing is assumed to have passed unless the artifact says so.

Executive Summary

Leadership question	Answer
What business risk does this address?	Enterprises transforming data through Bronze to Silver to Gold layers have no mathematical model governing how much information loss is acceptable at each stage, no governed entity resolution across source systems, and no auditable evidence trail for transformation decisions.
What does this application prove?	A Databricks-deployable transformation engine that scores every

…

View on GitHub

Both deploy as Databricks Apps with four-tab operator dashboards, Asset Bundle definitions for governed deployment, and notebook-based onboarding workflows.

What Comes Next

The validated experimental surface covers Bronze-layer quality validation and drift detection. The next research priorities are operational readiness validation (unattended execution with service-principal authentication), expanded natural-fault coverage with independent model evaluation and multi-reviewer adjudication, and Silver/Gold layer escalation — each following the same discipline of preregistered charters, independent datasets, and reproducible evidence.

Shannon entropy is not a silver bullet. It does not replace schema validation, freshness monitoring, or volume checks. But it measures something those tools structurally cannot — whether the data still carries the information it carried yesterday. The experiments demonstrate that this measurement is accurate, fast, and operationally useful at scale.

The tools are open source. The gap between validating structure and validating signal is closable — and now there is evidence to back it up.

Why Shannon Entropy Catches What Schema Validation Misses

Anthony Johnson II — Sat, 11 Apr 2026 03:39:59 +0000

This article was originally published on EthereaLogic.ai.

Your pipeline passed every check. Schema valid. Row count matched. Null percentage within threshold. Freshness on time. Dashboard green.

But this morning the downstream segmentation model lost a third of its signal. Marketing is asking why the "Premium" and "Enterprise" tiers collapsed into a single bucket. Finance wants to know why revenue forecasting diverged from actuals by 12%. The Customer 360 that was supposed to unify 40,000 accounts is quietly deduplicating to 24,000.

Everything validated. Nothing was correct.

If this sounds familiar, you have a monitoring blind spot — and it is not a tooling gap you can solve with more schema checks.

The Monitoring Blind Spot

Most data quality tools validate shape: Is the schema right? Are the types correct? Are nulls within threshold? Did the expected number of rows arrive on time?

These are necessary checks. They are not sufficient.

Here is what none of them measure: information content. A column can go from 12 distinct categories to 8 and every traditional check passes. A distribution can shift from uniform to heavily skewed and row counts will not flinch. Two source tables can silently converge to identical values during a merge, destroying the differentiation your downstream model depends on — and your freshness monitor will report on time.

The problem is not that these tools are wrong. The problem is that they are answering the wrong question. They tell you whether data arrived in the expected shape. They do not tell you whether it still carries the information it carried yesterday.

This is the difference between validating structure and validating signal.

What Is Shannon Entropy and Why Does It Matter for Data?

Shannon entropy, introduced by Claude Shannon in 1948, is a measure of information content — specifically, the average amount of uncertainty (or surprise) in a distribution. The formula is straightforward:

H = -Σ p(x) log2(p(x))

Where p(x) is the probability of each distinct value in the distribution.

The intuition: a column where every row is "Active" carries zero information — entropy is 0.0. A column evenly split across 8 categories carries maximum information for that cardinality — entropy is 3.0 bits (log2(8)). The more uniform the distribution, the higher the entropy. The more collapsed or skewed, the lower.

A concrete example

Consider a customer_tier column with 10,000 rows across four values:

Baseline (Monday):

Value	Count	Probability	-p log2(p)
Free	2,500	0.25	0.500
Basic	2,500	0.25	0.500
Premium	2,500	0.25	0.500
Enterprise	2,500	0.25	0.500

H = 2.000 bits. Maximum entropy for 4 values. Stability score: 1.0.

Friday's load:

Value	Count	Probability	-p log2(p)
Free	7,000	0.70	0.361
Basic	2,800	0.28	0.514
Premium	200	0.02	0.113
Enterprise	0	0.00	0.000

H = 0.988 bits. Stability score: 0.494. A category has disappeared entirely. Your schema check? Still green. Your row count? 10,000 as expected.

That is what entropy catches: not whether data arrived, but whether the information content of that data is still intact.

Four Failure Modes Entropy Catches

1. Distribution Collapse

What it looks like: A categorical column gradually loses diversity. A region field that once had 12 values starts arriving with 8. An order_type column concentrates from evenly distributed to 90% dominated by a single value.

Why traditional monitoring misses it: Schema is unchanged. Row count is stable. The remaining values are all valid enum members.

How entropy catches it: The stability score drops proportionally to information loss. DriftSentinel classifies this as collapsed when the score drops below the baseline by more than the configured threshold, and it will gate the load before it reaches downstream consumers.

2. Coherence Loss Across Medallion Layers

What it looks like: Your Bronze-to-Silver transformation is supposed to clean, standardize, and enrich. But somewhere in the pipeline, a join condition is too aggressive, a filter is too broad, or a coalesce is silently flattening variation.

Why traditional monitoring misses it: The Silver schema matches the contract. Types are correct. Row count may even be similar.

How entropy catches it: AetheriaForge computes a coherence score — a ratio of preserved entropy to source entropy — and enforces layer-specific thresholds: Bronze must preserve at least 50% of information (score >= 0.5), Silver at least 75% (>= 0.75), and Gold at least 95% (>= 0.95).

3. Entity Resolution Drift

What it looks like: Your Customer 360 is supposed to resolve records from multiple source systems into unified entities. But matching logic drift causes over-matching. Your "Customer 360" is actually a Customer 240.

Why traditional monitoring misses it: The output schema is correct. The row count dropped, but entity resolution should reduce rows.

How entropy catches it: If the resolved output has significantly lower entropy than expected, you are over-merging — collapsing distinct entities into fewer buckets than the source data supports.

4. Temporal Conflict and Silent Overwrites

What it looks like: A latest_wins merge strategy is supposed to resolve temporal conflicts by keeping the most recent record per entity. But when timestamps are missing or malformed, the "winner" is arbitrary.

Why traditional monitoring misses it: The merge completed without errors. Row count is within expected range. Schema matches.

How entropy catches it: If a latest_wins strategy is silently falling back to arbitrary ordering, values from one source system will be systematically overrepresented, reducing entropy in source-identifying columns.

From Theory to Practice

Drift Gating with DriftSentinel

DriftSentinel uses Shannon entropy as its primary distribution stability signal. The drift policy configuration is declarative:

drift_policy:
  monitored_columns:
    - column_name: customer_tier
      method: shannon_entropy
    - column_name: transaction_amount
      method: shannon_entropy

  gates:
    health_score_threshold: 0.70
    max_columns_failed: 2

  verdict_on_fail: block

The entropy computation itself is compact:

def column_stability_score(series: pd.Series) -> float:
    counts = series.value_counts(dropna=False)
    n_unique = len(counts)
    if n_unique <= 1:
        return 0.0
    probs = (counts / counts.sum()).to_numpy()
    positive = probs[probs > 0]
    h = -float(np.sum(positive * np.log2(positive)))
    h_max = math.log2(n_unique)
    return round(min(h / h_max, 1.0), 4)

Coherence Scoring with AetheriaForge

Where DriftSentinel measures drift within a single dataset over time, AetheriaForge measures information preservation across a transformation:

coherence:
  engine: shannon
  thresholds:
    bronze_min: 0.5   # Raw ingestion — expect some loss
    silver_min: 0.75  # Cleaned and standardized — preserve most signal
    gold_min: 0.95   # Business-ready — near-perfect preservation

Getting Started

Both tools are open-source, available on PyPI, and designed to run on Databricks.

DriftSentinel — Databricks-native data trust platform for intake certification, drift gating, and control benchmarking.

pip install etherealogic-driftsentinel

Org-EthereaLogic / DriftSentinel

Databricks-native data trust pipeline — intake certification, drift gating, and control benchmarking in a single deployable product.

Three Control Patterns. Multiple Datasets. One Platform That Proves All of Them Are Working.

Enterprise Data Trust — Chapter 4: DriftSentinel

Built by Anthony Johnson | EthereaLogic LLC

If this platform is useful to your team, consider starring the repo — it helps others in the Databricks community find it.

Three modules. One registry. Queryable evidence. No assumption that any run passed unless the artifact says so.

Important: If you used DriftSentinel…

View on GitHub

AetheriaForge — Coherence-scored transformation engine for entity resolution, temporal reconciliation, and schema enforcement.

pip install etherealogic-aetheriaforge

Org-EthereaLogic / AetheriaForge

Databricks-native intelligent data transformation engine — coherence-scored Bronze/Silver/Gold with entity resolution and temporal reconciliation in a single deployable product.

Intelligent Data Transformation. Coherence-Scored. Evidence-Backed.

EthereaLogic Databricks Suite — AetheriaForge

Built by Anthony Johnson | EthereaLogic LLC

If this tool is useful to your team, consider starring the repo — it helps others in the Databricks community find it.

Executive Summary

Leadership question	Answer
What business risk does this address?	Enterprises transforming data through Bronze to Silver to Gold layers have no mathematical model governing how much information loss is acceptable at each stage, no governed entity resolution across source systems, and no auditable evidence trail for transformation decisions.
What does this application prove?	A Databricks-deployable transformation engine that scores every

…

View on GitHub

Both projects publish customer impact advisories when defects are found that could affect operator decisions. If you are evaluating data quality tooling, look for that signal. The willingness to publicly disclose what went wrong, who was affected, and what to do about it tells you more about engineering culture than any feature list.