Case Study: Catch Layering Violations With Static Analysis, Then Ask a Model Only for the Why

You should not ask a language model to invent your architecture. You should compute the import graph yourself, then ask a model only to explain edges that already break a written policy. This case study walks a small Python service from a layering YAML file to a pull-request-ready violation report. The static step stays deterministic, and the model step stays optional, cheap, and testable.

When generated code is inexpensive, new files arrive faster than informal layering rules. A request handler starts opening an ORM session. A shared util module starts importing your web framework. You usually notice during an incident, not during review, because the code still compiles and the tests still pass.

Background: cheap files, silent edges

This walkthrough uses a fixture service with four packages, not a production monorepo. You will treat that tree as the whole project so every command stays reproducible on a laptop. The goal is not a pretty architecture diagram. The goal is a short list of policy breaks you can defend in review.

The fixture layout looks like this:

• app/api/ may import app/services/ and app/schemas/
• app/services/ may import app/data/ and app/schemas/
• app/data/ may import only the standard library and approved drivers
• app/schemas/ may import only the standard library

Those four bullets are the entire architecture. If a model cannot see that YAML file, it should not be allowed to narrate your system.

Goal for the weekend project

You want a bot that fails closed when the graph is dirty, then writes human text only for rows the scanner already proved. You will keep the LLM off the critical path for detection. You will keep detection in AST parsing, policy loading, and a golden test against the fixture tree.

Success for this case study is narrow on purpose:

1. The scanner prints one JSON object per illegal import, with file, line, from-layer, and to-layer.
2. A unit test fails if a known illegal import disappears or a legal import is flagged.
3. An optional explainer turns each JSON row into three bullets a reviewer can accept or reject.
4. You can run the explainer on a small free server, or skip it and still ship the JSON.

If you cannot point to a policy file, stop. A model that invents layers is just another source of technical debt.

Implementation

1. Write the policy before any prompt

Save this as layers.yaml. Keep allowed edges explicit, and treat missing keys as denials.

# layers.yaml
layers:
  api: ["app/api"]
  services: ["app/services"]
  data: ["app/data"]
  schemas: ["app/schemas"]
allow:
  api: ["services", "schemas"]
  services: ["data", "schemas"]
  data: []
  schemas: []
ignore_prefixes:
  - app/tests/

You now have a contract a diff can break. Reviewers can argue about the YAML. They should not argue with a chat transcript.

2. Parse imports with the AST, not with vibes

The scanner below is a complete teaching script. Point it at the fixture tree and it emits JSON Lines you can snapshot in CI.

# scan_imports.py
from __future__ import annotations

import ast
import json
import sys
from pathlib import Path

import yaml


def load_policy(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    prefix_to_layer = {}
    for layer, prefixes in data["layers"].items():
        for prefix in prefixes:
            prefix_to_layer[prefix.rstrip("/")] = layer
    return {
        "prefix_to_layer": prefix_to_layer,
        "allow": {k: set(v) for k, v in data["allow"].items()},
        "ignore": tuple(data.get("ignore_prefixes", [])),
    }


def layer_for(path: Path, prefix_to_layer: dict) -> str | None:
    text = path.as_posix()
    matches = [prefix for prefix in prefix_to_layer if text.startswith(prefix + "/") or text == prefix]
    if not matches:
        return None
    return prefix_to_layer[max(matches, key=len)]


def imported_module(node: ast.AST) -> str | None:
    if isinstance(node, ast.ImportFrom) and node.module:
        return node.module
    if isinstance(node, ast.Import):
        return node.names[0].name
    return None


def module_to_path(root: Path, module: str) -> Path | None:
    candidate = root.joinpath(*module.split(".")).with_suffix(".py")
    if candidate.exists():
        return candidate
    package = root.joinpath(*module.split("."), "__init__.py")
    return package if package.exists() else None


def scan(root: Path, policy_path: Path) -> list[dict]:
    policy = load_policy(policy_path)
    findings: list[dict] = []
    for py in root.rglob("*.py"):
        rel = py.relative_to(root).as_posix()
        if rel.startswith(policy["ignore"]):
            continue
        src_layer = layer_for(Path(rel), policy["prefix_to_layer"])
        if src_layer is None:
            continue
        tree = ast.parse(py.read_text(), filename=rel)
        for node in ast.walk(tree):
            module = imported_module(node)
            if not module:
                continue
            target = module_to_path(root, module)
            if target is None:
                continue
            dst_rel = target.relative_to(root)
            dst_layer = layer_for(dst_rel, policy["prefix_to_layer"])
            if dst_layer is None or dst_layer == src_layer:
                continue
            if dst_layer in policy["allow"].get(src_layer, set()):
                continue
            findings.append({
                "file": rel,
                "line": getattr(node, "lineno", 0),
                "import": module,
                "from_layer": src_layer,
                "to_layer": dst_layer,
            })
    return findings


if __name__ == "__main__":
    root = Path(sys.argv[1])
    policy = Path(sys.argv[2])
    for row in scan(root, policy):
        print(json.dumps(row))

Run it against the fixture after you plant one illegal import in app/api/users.py:

python scan_imports.py ./fixture layers.yaml > violations.jsonl
wc -l violations.jsonl

You should see a row that names app/api/users.py, the import string, and the api -> data hop. If that hop is missing, your scanner is the bug, not the model.

3. Lock the scanner with a fixture test

Do not demo the explainer until this test is green. The test is the product; the prose is a comment.

# test_scan_imports.py
from pathlib import Path
from scan_imports import scan

ROOT = Path(__file__).parent / "fixture"
POLICY = Path(__file__).parent / "layers.yaml"


def test_flags_api_to_data_and_ignores_schemas():
    rows = scan(ROOT, POLICY)
    pairs = {(r["from_layer"], r["to_layer"], r["import"]) for r in rows}
    assert ("api", "data", "app.data.db") in pairs
    assert not any(r["to_layer"] == "schemas" and r["from_layer"] == "api" for r in rows)

python -m pytest test_scan_imports.py -q

If a later refactor renames app.data.db, this test should fail loudly. That failure is cheaper than a confident paragraph about a file that no longer exists.

4. Explain only proven rows

The explainer receives JSON Lines, not the repository. That constraint is the safety feature. A model that never sees unused files cannot invent a fifth layer.

# explain_violations.py
from __future__ import annotations

import json
import os
import sys
from pathlib import Path

import httpx

SYSTEM = """You explain import-layer violations.
You receive JSON only. Do not invent files, layers, or imports.
Return Markdown with exactly three bullets: why it matters, a safer hop, a test to add.
If the JSON is incomplete, say so and stop."""


def explain(rows: list[dict], endpoint: str, model: str, token: str) -> str:
    payload = {
        "model": model,
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": json.dumps(rows)},
        ],
        "temperature": 0,
    }
    headers = {"Authorization": f"Bearer {token}"}
    response = httpx.post(endpoint, json=payload, headers=headers, timeout=60)
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]


if __name__ == "__main__":
    rows = [json.loads(line) for line in Path(sys.argv[1]).read_text().splitlines() if line.strip()]
    print(explain(
        rows,
        endpoint=os.environ["LLM_ENDPOINT"],
        model=os.environ["LLM_MODEL"],
        token=os.environ["LLM_TOKEN"],
    ))

Wire the two stages so CI can skip the second one on a fork without secrets:

python scan_imports.py ./fixture layers.yaml > violations.jsonl
if [ -n "$LLM_TOKEN" ]; then
  python explain_violations.py violations.jsonl > report.md
else
  echo "scanner-only mode" && cat violations.jsonl
fi

If you need a place to run the explainer without standing up GPU hardware, MonkeyCode's free model access and free server option can host that second stage. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The scanner does not depend on that host; you can point the same JSON at any OpenAI-compatible endpoint you already trust.

5. Decision table for calling the model

Use this table before you spend a token. The left column is a scanner fact, not a feeling.

Scanner fact	Call the model?	What you ask it to do	What you forbid
Zero rows	No	Nothing	A tour of the codebase
One illegal hop with file and line	Yes	Three bullets for that hop	New files or layers
Many hops, same pair of layers	Yes, after you group	One explanation per pair	A unique essay per file
Import target is outside layers.yaml	No	File a policy ticket	Guessing a new layer name
JSON failed schema or parse	No	Fix the scanner	Retry with a looser prompt

Group before you prompt. Ten api -> data rows are one architectural story, not ten chat completions.

Fixture results, labeled as fixture results

The commands above were written against a four-package tree with two planted violations. Expected scanner output looks like this, not like a dashboard screenshot from a company you do not have:

{"file": "app/api/users.py", "line": 4, "import": "app.data.db", "from_layer": "api", "to_layer": "data"}
{"file": "app/schemas/user.py", "line": 7, "import": "app.services.users", "from_layer": "schemas", "to_layer": "services"}

A valid explainer response for the first row should mention the skipped services hop and a test that imports app.api.users while mocking app.services. If the model names app/workers/ or a domain layer, you throw the paragraph away. The JSON did not contain those strings, so the text is not a review comment; it is contamination.

Keep a golden Markdown file only after a human accepts it. Snapshot the scanner JSON in git. Do not snapshot model prose until you have a grader that rejects invented paths.

Limitations, and who should skip this

This approach will not design a system you have not already described. Dynamic imports, string-based loaders, and plugin entry points are invisible to the AST walk shown here. Cross-language repos need a different edge extractor, and this script will happily ignore them.

Do not use this workflow when any of the following is true:

• You have no written layer list, only a slide from last quarter.
• Your compliance rules forbid source snippets or import graphs from leaving the network.
• You want a model to propose a new architecture instead of enforcing the current one.
• Your "violations" are mostly generated client SDKs that you do not own and should ignore in policy, not explain.

A free server is a convenience for the explainer process, not a substitute for the golden test. If the scanner is wrong, cheaper completions only scale the wrong story.

Lessons from the small project

Write the allow-list first, because an unexplained graph is still a graph you can fail CI on. Keep detection boring and local, because reviewers trust line numbers more than tone. Pass the model the minimum JSON, because extra files become extra fiction. Group repeated hops, because architecture debt is a pattern, not a novel. Throw away any sentence that names a path the scanner did not emit.

If you extend the fixture, add one illegal import and one legal import in the same commit as the test. That pairing is the whole method. The report is optional commentary on a fact you already proved.

You can stop after violations.jsonl and still have a useful bot. Add the explainer only when a human is waiting for three bullets, not when you want the model to feel involved.