We Audited Counterfact With Mneme. Mneme Could Enforce 0 of Its 15 Architectural Decisions.

#architecture #typescript #opensource #ai

We ran Mneme's architecture protection audit against Counterfact, an open-source TypeScript monorepo that simulates APIs from OpenAPI specs. Mneme scored 0%.

Counterfact itself scored 100%.

Metric	Value
Decisions catalogued	29
Protection-relevant decisions	15
Protected by Counterfact's own tooling	15 (100%)
Enforceable by Mneme today	0 (0%)
Would need a new Mneme rule type	15 (100%)

We're publishing the result because it tells us more about Mneme than a passing score would.

How we ran it

Counterfact doesn't keep a Mneme project memory, so we wrote one. We read the ADRs and source at commit 2a1dfb1, catalogued 29 architectural decisions, and marked 15 as protection-relevant: decisions where a violation would break something. Then we ran mneme audit (v0.9.2) against that memory and the repo.

For each decision the audit asks three questions. Does the repo already protect it? Can Mneme's current rule vocabulary express it? If not, what primitive is missing?

Three decisions Counterfact already enforces

1. One-way package dependencies, checked at the import level

Counterfact's packages form a strict direction: types → openapi → (generator, runtime) → (client, repl) → counterfact facade. scripts/check-package-boundaries.mjs declares an ADR_DEPENDENCY_ALLOWLIST and runs as yarn check:boundaries on every PR.

export const ADR_DEPENDENCY_ALLOWLIST = Object.freeze({
  "@counterfact/client": ["@counterfact/openapi"],
  "@counterfact/generator": ["@counterfact/openapi", "@counterfact/types"],
  "@counterfact/runtime": ["@counterfact/openapi", "@counterfact/types"],
  "@counterfact/types": [],  // leaf
  // ...
});

The checker compares package.json dependencies and TypeScript project references against the allowlist. It also tokenizes source files, extracts every import and export * from specifier, rejects disallowed cross-package imports, and flags cycles. A regex over the codebase can't do the same job.

2. Every package installs from its own tarball closure

scripts/test-package-closures.mjs runs npm pack on every workspace. It then installs each package into an empty temp directory using only the tarballs in that package's declared dependency closure. The script checks four things:

npm ls matches the declared closure exactly
every exports subpath imports
deep imports fail with ERR_PACKAGE_PATH_NOT_EXPORTED
tsc --noEmit passes on the installed declarations, and the documented example runs from the installed copy

If a package leaks a dependency it never declared, this test catches it before a user does.

3. Generated code never imports the generator

When Counterfact generates route handlers, it copies a counterfact-types/ template into the output. Generated files import types by relative path (../types/_.context.js) and never from @counterfact/types or any other workspace package. CI compares generated fixtures byte for byte. Your generated project keeps working even if you never install Counterfact's internals.

Why Mneme scored zero

Mneme's shipped rules check source text: forbid a literal, require a pattern, scoped to paths. Those rules work for decisions like "never shell out to git" or "don't import the ORM in the API layer." Counterfact's decisions need primitives Mneme doesn't have yet:

Counterfact decisions	Missing Mneme rule type
Package boundaries, exports maps, TS project references (4)	`dependency`: allowlist plus source-level import verification
Package closure tests, mutation baseline, release preflight (4)	`evidence-backed`: consume results from an external verifier
Request/response validation, immutable route builder, types from OpenAPI (3)	`protocol/API contract`
Multi-API spec config, shared store across hot reloads (2)	`state/lifecycle`
Generated-code self-containment (1)	`generated-vs-source`: constrain what generated artifacts may import
Degrade gracefully on bad input (1)	`behavioural`

The evidence-backed gap stood out most. Mneme currently assumes it runs every check itself. Counterfact already runs better, more specific verifiers than Mneme could write for it. A mature repo doesn't need Mneme to re-implement its boundary checker. It needs Mneme to read the checker's output and tie it to the decision that justifies it.

What we're taking from this

We read Counterfact as a reference implementation. Its boundary checker is a spec for a dependency rule type, and its closure harness is a spec for evidence-backed rules. The audit gave us a ranked list of primitives to build: dependency, evidence-backed, generated-vs-source, state/lifecycle.

It also sharpened who Mneme is for today. Counterfact already has CI enforcement for every decision that matters, so Mneme adds nothing there yet. The teams we built Mneme for have written ADRs and standards but no enforcement behind them, and they watch coding agents ignore those standards daily.

We store every audit with its commit SHA and Mneme version. As each new rule type ships, we'll re-run this exact audit and count how many of the 15 decisions move from "requires modelling" to "protected."

If you maintain a public repo with ADRs and want an audit run against it, open an issue on the Mneme repo. More on how Mneme prevents architectural drift at mnemehq.com.

DEV Community