Case Study: A Doc-Drift Ledger That Only Sends Changed Symbols to a Model

Most documentation rot is invisible until someone follows a docstring into a TypeError. The fix I built for a small internal library is a ledger, not a chatbot: it hashes every public symbol's signature and docstring, then asks a model about the handful of symbols whose signature moved while the prose did not. The useful part is what never leaves your machine — unchanged symbols cost zero tokens, and every model reply is cached against a hash pair you can inspect in Git.

Background: the docstring that outlived its argument

The library in question exports roughly forty functions across a dozen modules, and three of them had quietly changed their parameter names during a cleanup. Nothing failed in CI, because docstrings are not assertions, and nobody reread the module by hand in that sprint. When a teammate wired fetch_user(uid=...) into a script and got a keyword error, the docstring was still advertising uid.

That is the shape of the problem worth solving: not "docs are bad", but "a narrow class of doc drift is mechanically detectable and cheap to detect". Full-file review passes are noisy and expensive, while a symbol-level diff gives you a short, defensible list you can hand to a reviewer or a model.

Goal and explicit non-goals

The goal was one command that answers a single question: which exported symbols changed shape without their documentation following along?

• In scope: Python modules, signature changes, missing or stale docstrings on public symbols, a machine-readable report, a non-zero exit code for CI.
• Out of scope: prose pages in a wiki, translation drift, tutorial code blocks, and any claim that a model can certify documentation as correct.
• Non-negotiable: the tool must run offline against a fake client, so the test suite never depends on a network call.

You should read that last point as the design constraint that made everything else simple. If the diffing layer is deterministic, the model becomes an optional annotation on top of a result you already trust.

Step 1: Extract symbols and hash them

Start with the extractor. It walks module-level definitions only, which keeps the ledger small and the false-positive rate low.

# drift/symbols.py
import ast
import hashlib
from pathlib import Path


def _signature(node: ast.AST) -> str:
    if isinstance(node, ast.ClassDef):
        bases = ", ".join(ast.unparse(b) for b in node.bases)
        return f"class {node.name}({bases})"
    args = [a.arg for a in node.args.args]
    if node.args.vararg:
        args.append(f"*{node.args.vararg.arg}")
    if node.args.kwarg:
        args.append(f"**{node.args.kwarg.arg}")
    return f"def {node.name}({', '.join(args)})"


def _digest(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def collect(root: Path) -> dict[str, dict[str, str]]:
    ledger: dict[str, dict[str, str]] = {}
    for path in sorted(root.rglob("*.py")):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        for node in tree.body:
            if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
                continue
            key = f"{path.as_posix()}::{node.name}"
            sig = _signature(node)
            ledger[key] = {
                "sig": sig,
                "sig_hash": _digest(sig),
                "doc_hash": _digest(ast.get_docstring(node) or ""),
            }
    return ledger

Commit the generated JSON next to your code, and treat it exactly like a lockfile. The diff between the committed ledger and the freshly collected one is your entire work queue.

Step 2: Turn two ledgers into a candidate list

The comparison rule matters more than the code around it. A symbol is a candidate only when its signature hash changed and its docstring hash did not, because a human who already edited the prose has answered the question.

# drift/diff.py
def candidates(base, head):
    out = []
    for key in sorted(set(base) | set(head)):
        old, new = base.get(key), head.get(key)
        if old is None:
            out.append({"key": key, "change": "added", "symbol": new, "ask": True})
        elif new is None:
            out.append({"key": key, "change": "removed", "symbol": old, "ask": False})
        elif old["sig_hash"] != new["sig_hash"] and old["doc_hash"] == new["doc_hash"]:
            out.append({"key": key, "change": "signature", "symbol": new, "ask": True})
    return out

Removals are reported but never sent to a model, since the correct answer there is a human decision about deprecation wording. Added symbols are also a weak signal, so I gate them behind a flag instead of asking about every new helper.

Step 3: Ask the model about candidates only

The prompt is deliberately boring, and it demands JSON so the reply can be validated before it touches the report. Note that the payload contains signatures and docstrings, not whole files.

PROMPT = """You review Python docstrings against a changed signature.
For each item answer only whether the docstring still describes the new signature.
Reply with JSON: {"verdicts":[{"key":str,"verdict":"ok"|"stale","reason":str}]}

Items:
{items}
"""


def build_prompt(cands):
    items = [
        {
            "key": c["key"],
            "signature": c["symbol"]["sig"],
            "docstring": c["symbol"].get("doc", ""),
        }
        for c in cands
        if c["ask"]
    ]
    return PROMPT.format(items=json.dumps(items, indent=2))

Keep the response schema strict and drop any verdict you cannot parse. A model that answers in prose has told you nothing you can put in a report, and silently repairing its output is how false confidence gets into a repo.

Step 4: Cache verdicts by hash pair

Caching is where the economics live. A verdict is keyed by (key, sig_hash, doc_hash), so a rerun after an unrelated commit performs zero model calls.

# drift/cache.py
def cache_key(cand):
    return f"{cand['key']}|{cand['symbol']['sig_hash']}|{cand['symbol']['doc_hash']}"


def pending(cands, cache):
    return [c for c in cands if c["ask"] and cache_key(c) not in cache]

Store that cache in CI artifacts or a small committed file, whichever your team already trusts. The committed option is easier to audit, because a reviewer can see exactly which verdicts were accepted and when.

Step 5: Make the whole pipeline testable with a fake client

The tool has to be provable without a network, so the model client is an injectable one-method interface. The test below asserts two things that actually matter: irrelevant symbols never enter the prompt, and a second run makes no calls at all.

# tests/test_drift.py
import json
from pathlib import Path
from drift.run import run


class FakeClient:
    def __init__(self, replies):
        self.replies = list(replies)
        self.prompts = []

    def complete(self, prompt: str) -> str:
        self.prompts.append(prompt)
        return self.replies.pop(0)


def write(path: Path, text: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(text, encoding="utf-8")


def test_only_changed_symbols_are_sent(tmp_path):
    src = tmp_path / "src"
    write(src / "a.py", 'def fetch(uid):\n    """Fetch by uid."""\n    return uid\n')
    write(src / "b.py", 'def ping():\n    """Ping."""\n    return 1\n')
    base = run(src, cache={}, client=None, write_ledger=tmp_path / "ledger.json")

    write(src / "a.py", 'def fetch(user_id):\n    """Fetch by uid."""\n    return user_id\n')
    reply = json.dumps({"verdicts": [{"key": "a.py::fetch", "verdict": "stale", "reason": "param renamed"}]})
    client = FakeClient([reply])
    report = run(src, cache={}, client=client, base_ledger=base, write_ledger=tmp_path / "ledger2.json")

    assert "b.py::ping" not in client.prompts[0]
    assert report["stale"] == ["a.py::fetch"]

Add a second test that reruns the same tree against the populated cache and asserts client.prompts == []. That pair of tests is the whole contract, and it runs in milliseconds on a laptop with no credentials configured.

Running it nightly without babysitting a laptop

A cron entry finishes the loop, because the interesting case is the commit that landed at 2 a.m. from someone else's branch.

17 3 * * * cd /srv/drift && /usr/bin/python -m drift nightly --root src --fail-on stale >> drift.log 2>&1

This is also where MonkeyCode's free model access and free server option became relevant to the project rather than a footnote. The nightly step needs a machine that stays awake and a model endpoint whose cost does not scale with how often you run the job, and the variant I deployed uses the free tier for both. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two practical notes on that. First, the free allowance advertised on the plan page at the time of writing was ten million tokens, and the operator-supplied framing matters: quotas and terms change, so verify the current page before you build a budget on it. Second, keep the endpoint in an environment variable rather than in code, so swapping providers is a config change and not a refactor.

export MONKEYCODE_API_KEY="..."        # never commit this
export MONKEYCODE_BASE_URL="..."       # from your workspace settings
export DRIFT_MODEL="..."               # whatever your workspace lists

If you want to try the pattern before provisioning anything, the candidate-only design works with any OpenAI-compatible client, so you can prototype locally and move the cron job later.

Results: what the ledger actually changed

On the fixture above, a signature rename produces exactly one candidate, one model call, and one cached verdict; a rerun produces zero calls. Those numbers come from the test suite, not from a benchmark, and I have not run a controlled cost study on a large repository.

The qualitative result is the one I would defend. Reviewers stopped reading forty-symbol reports and started reading a two-line queue, and the stale verdicts arrived with the offending signature attached, so the fix took seconds. The tool also caught a case I expected it to miss: a renamed keyword argument whose docstring example still used the old name in a code block.

Decision table

Situation	Use the ledger?	Why
Python library with docstrings on public symbols	Yes	Hashes are cheap and drift is real
Single-file script under 300 lines	No	Just read it
Docs live in separate Markdown pages	Not yet	You need anchor extraction first
You need legally exact wording	No	Model verdicts are advisory
Heavy churn on private helpers	No	Noise dominates; restrict to exports

Limitations and who should skip this

• Hash equality is not meaning equality. A docstring that was already wrong before the rename keeps its hash and stays invisible to the ledger.
• Python only, for now. The extractor is ast-based, and porting to TypeScript means a different parser plus your own signature stringifier.
• Renames look like deletion plus addition. Wire in git log --follow or accept one noisy commit per rename; I accepted the noise.
• Skip this if your team has no committed ledger discipline. A ledger in .gitignore gives you a tool that flags everything on every run.
• Skip this if your docstrings are generated. Generated prose is already consistent with signatures, so you would pay for verdicts nobody reads.

Lessons learned

The design that survived contact with a real repository was the least clever one: detect locally, ask rarely, cache aggressively, and test with a fake client. Determinism at the diffing layer is what made the model call optional, auditable, and cheap enough to run nightly on a free tier. If you rebuild this, start with the fake-client test first — the pipeline falls out of it almost mechanically.