How I Built an Agent That Keeps My Docs From Going Stale

TL;DR

I run an autonomous coding agent that ships changes to my projects without me watching every diff. For months, the docs quietly rotted — READMEs describing flags that no longer existed, setup steps for a config file I'd deleted three refactors ago. I fixed it by giving the agent a dedicated "does the doc still match reality" pass that runs after every change, with its own guardrails so it doesn't turn every commit into a documentation essay. Here's how it works and what broke along the way.

The Problem

Docs rot. Every engineer knows this. Normally it's a slow, human-paced rot — someone changes a function signature, forgets to update the README, and six weeks later a teammate wastes an hour following instructions that describe a version of the code that no longer exists.

Now put an autonomous agent in the loop, one that's making dozens of small changes a day without a human reviewing each diff before it lands. The rot doesn't happen over six weeks anymore. It happens in six days.

I noticed it first in a setup guide. The doc said "run configure.py with the --mode flag" — except two weeks earlier, my agent had refactored that script and renamed the flag to --profile as part of an unrelated change. The rename was correct. The code was correct. The tests passed. Nobody updated the doc, because updating docs wasn't anyone's job — the agent was focused on the task it was given, and I wasn't reading every diff closely enough to catch prose that didn't change when it should have.

The deeper problem: an autonomous agent optimizes for "the task I was told to do," not "the state of the repository as a whole." Code changes are the primary artifact. Docs are a side effect that only a human historically remembered to maintain. Remove the human from the loop and that side effect just... stops happening.

I had three options:

1. Review every diff for doc impact myself (defeats the point of automation)
2. Accept doc rot as a cost of running things unattended (untenable past a certain repo size)
3. Make "keeping docs in sync" a first-class step in the agent's own workflow

I went with option 3.

How I Solved It

Step 1: A dedicated sync pass, not an afterthought

The first version of this was embarrassingly naive: I added a line to my main prompt saying "also update documentation if needed." It did almost nothing. "If needed" is exactly the kind of soft instruction a model will quietly skip when it's focused on the primary task — the same way a busy human skips it.

What actually worked was splitting doc sync into its own pass, run as a separate step after the code change is done and tests pass:

1. Agent completes the code change, tests pass
2. Diff is captured (files changed, functions added/removed/renamed, config keys touched)
3. A second, narrowly-scoped agent call runs:
   - input: the diff + the current contents of any docs that reference the changed files
   - task: identify claims in the docs that are now false, and only those
4. If drift is found, patch the doc inline and re-run doc-specific lint/link checks
5. If no drift is found, do nothing — no docs are touched

That last point mattered more than I expected. Early on I didn't gate step 5 properly, and the doc-sync agent would find something to change on nearly every run, even when nothing meaningfully drifted. More on that in the guardrails section.

Step 2: Scoping "which docs are relevant" without reading the whole repo

You can't hand an agent your entire docs folder on every change and ask "does anything need updating" — it's slow, expensive, and worse, it invites the model to helpfully rewrite things that were already fine.

Instead, I built a lightweight reverse index: which doc files reference which source files, config keys, or CLI flags. It's not fancy — mostly grep for filenames, function names, and flag strings across the docs directory, built once and refreshed incrementally. When a diff touches configure.py and the flag --mode, the sync pass only looks at docs that already mention configure.py or --mode. Everything else is out of scope by construction.

def find_affected_docs(changed_symbols, doc_index):
    affected = set()
    for symbol in changed_symbols:
        affected.update(doc_index.get(symbol, []))
    return affected

This did two things: it made the sync pass fast enough to run on every change without becoming a bottleneck, and it kept the blast radius small. The agent literally cannot rewrite a doc it has no reason to touch, because that doc never enters its context window.

Step 3: Drift detection has to be about claims, not vibes

The first prompt I used for the sync agent asked it to "review the docs for accuracy." That's too vague — it produces overzealous rewrites, tone changes, and reformatting nobody asked for.

What worked was reframing the task as claim verification: extract concrete, checkable claims from the doc ("this script accepts a --mode flag," "config lives at settings.yaml," "step 3 is optional") and check each one against the current code. Only claims that are now false get touched. Everything else — style, structure, unrelated sections — is explicitly off-limits.

For each sentence in the affected doc section:
  - Is this a factual claim about code behavior, a file path, a flag, or a config key?
  - If yes: does the current diff make this claim false?
    - If false: propose a minimal edit that makes it true again
    - If still true: leave it untouched
  - If it's not a checkable claim (prose, rationale, examples): leave it untouched

This one change cut the sync agent's "false positive" edit rate dramatically — from touching something on almost every run to only touching docs when something had actually gone stale.

Step 4: Guardrails, because agents love to over-explain

Once the doc-sync step worked, it created a new problem: the agent liked writing docs. Give it an opening to touch a README and it would add a paragraph of context nobody asked for, restructure a list into a table, or expand a one-line note into three sentences. Technically none of this was wrong. All of it was scope creep.

I added explicit constraints to the sync agent's instructions:

• Maximum diff size per doc file (a few lines, not a rewrite)
• No new sections, no restructuring, no tone changes
• If more than N doc files would need changes for one code diff, stop and flag it for me instead of touching all of them — that's usually a sign the code diff was too broad, not that the docs are that stale
• Every doc edit gets logged with the specific claim it invalidated and the diff that caused it, so I can audit after the fact instead of reviewing before

That last point is the same "fail loud, not silent" principle I use everywhere else in this agent's design — I'd rather see a log entry I can spot-check than have a quiet, unreviewable rewrite land in my repo.

Lessons Learned

1. "Update docs if needed" is not an instruction, it's a coin flip. Soft, conditional instructions get skipped by agents under task pressure the same way they get skipped by busy humans. Doc sync needed to be its own scheduled pass with its own success criteria, not a footnote on the main task.

2. Scope the input before you scope the output. The single biggest win was building the reverse index so the sync agent only ever sees docs relevant to the actual diff. Every problem I had with over-eager rewrites got worse when the agent had more doc surface area in context, and better when I narrowed it.

3. Claim verification beats "review for accuracy." Turning the task into "check these specific checkable claims" instead of an open-ended accuracy review is what killed the scope-creep problem. Open-ended review prompts invite open-ended rewrites.

4. An agent's helpfulness is a liability without a cap. The doc-sync pass didn't fail by being wrong — it failed by being too willing. Diff size limits and a hard stop when too many files are implicated turned out to be more important than the drift-detection logic itself.

5. Silent doc edits are worse than no doc edits. Logging exactly which claim triggered which edit means I can audit a week's worth of doc changes in five minutes instead of re-reading every README from scratch. If you can't audit it after the fact, don't let it run unattended.

What's Next

Right now the sync pass only understands docs as prose with checkable claims — it doesn't touch diagrams, and it can't tell when an entire doc file has become obsolete rather than just inaccurate (that's still a "notify me" case). I'm working on extending the affected-docs index to catch "this whole guide describes a workflow that no longer exists" rather than just sentence-level drift, which is a fuzzier problem than flag-and-path checking.

Wrap-up

If you're running any kind of autonomous agent against a real codebase, docs are the first thing that'll silently rot once a human stops reviewing every diff. Treat doc sync as its own step with its own guardrails, not a hope tacked onto your main prompt.

If this was useful, follow me here for more build-in-public posts on running autonomous coding agents, and give Claude Code a try if you haven't yet — it's what I've been building all of this on top of.