Catch Docs Drift at the PR: A Free-Tier Bot That Drafts Impact Notes

A merged PR changed a function's signature. The README still showed the old call. Three weeks later, a user opened an issue with a stack trace. Nobody updated the docs because nobody knew the docs were affected.

That is the real failure mode of documentation drift. It is not a writing problem. It is a review problem — and it starts at the pull request, not at release time.

AI has turned every developer into a reviewer, and the side effect is predictable: the docs review step is the first one to get dropped. This article shows a workflow that makes docs review cheap enough to keep. It runs on a free server, uses free model access, and produces one artifact per merged PR: a docs impact note. The model drafts the note. A human decides what happens next.

What you need

• A GitHub repository you can read.
• A free server that can run a scheduled job (MonkeyCode's free server option).
• Free model access (MonkeyCode's free tier includes 10 million tokens as of this writing).

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The pipeline

The bot runs on a schedule, not on every push. That keeps token spend predictable and the server stateless.

1. Fetch all PRs merged since the last run.
2. Build a prompt from the PR title, the changed files, and a truncated diff.
3. The model returns a JSON classification: impact label, candidate doc files, and a short note.
4. The bot posts the note as a comment on the PR or opens a draft PR against the docs.
5. A human reviews the note and decides whether to edit the docs.

Step 5 is the point of the whole exercise. The bot never edits docs directly.

The prototype

The snippet below is a prototype outline, not production code. The model call is a placeholder — adapt it to the endpoint you actually use.

# docs_impact.py — prototype outline
import json

def merged_prs(since_sha: str) -> list[dict]:
    # gh pr list --state merged --json number,title,body,files
    # then fetch each diff and truncate it
    ...

def build_prompt(pr: dict) -> str:
    return f"""
You classify whether a merged PR affects user-facing documentation.
PR title: {pr['title']}
Files changed: {', '.join(pr['files'])}
Diff (truncated to 4000 chars): {pr['diff'][:4000]}

Return JSON only:
{{"impact": "public_api|behavior|example|internal|none",
  "doc_files": ["README.md"],
  "note": "One sentence on what may need updating."}}
"""

def call_model(prompt: str) -> dict:
    # Placeholder for the free model endpoint. Not executed here.
    ...

def post_comment(pr_number: int, note: str) -> None:
    # gh pr comment {pr_number} --body "{note}"
    ...

def main() -> None:
    for pr in merged_prs(since_sha="HEAD"):
        result = call_model(build_prompt(pr))
        post_comment(pr["number"], json.dumps(result, indent=2))

if __name__ == "__main__":
    main()

The truncation matters. A diff larger than the context window will be cut, and the model will miss cross-file effects. That is a limitation, not a bug — more on that below.

What the model may draft, what the human owns

Step	The model may draft	The human must own
Impact label	public_api, behavior, example, internal, none	Whether the change is intentional and user-facing
File mapping	Candidate doc files to check	Which docs actually matter to readers
Suggested note	A draft sentence about what changed	Tone, accuracy, versioning, deprecation wording
Priority	An urgency score	When the docs update ships

The division is deliberate. The model is good at pattern matching: a renamed export, a changed default, an updated example. It is unreliable at product judgment: whether a behavior change is a bug or a feature, and how to phrase it for the people who will read it.

Token accounting

Free model access still deserves a budget. Measure it per run instead of guessing.

def estimate_tokens(text: str) -> int:
    return len(text) // 4  # rough estimate; calibrate for your model

def log_run(pr_number: int, prompt: str, response: str) -> None:
    print(pr_number, estimate_tokens(prompt), estimate_tokens(response))

A typical small PR lands in the low thousands of input tokens. A large diff can exceed the context window and force truncation. Track the numbers for your own repos for a week before you trust any estimate — including this one.

Test the reviewer

A classifier you never test is a rumor. Before trusting the impact label, run the bot against a fixed set of PRs with known ground truth.

PR description	Expected label
Rename get_user to fetch_user across the public API	public_api
Add retry logic inside an internal helper	internal
Change the default timeout from 5s to 30s	behavior
Update the README example to the new syntax	example
Fix a typo in a code comment	none
Remove a deprecated endpoint	public_api

Run the bot over this set, count the agreements, and decide whether the label is trustworthy for your repo. This is the same discipline as auditing tool calls: the model is part of the pipeline, but the pipeline is what you measure.

Limitations and who should not use this

• Truncated diffs. Large PRs lose context. The note may miss effects in files outside the diff.
• No product intent. A change labeled internal can still need docs if it changes observable behavior. The model cannot know that.
• Data leaves your network. Code diffs go to a third-party model. Do not point this at proprietary code if that violates your policy.
• Free server constraints. Ephemeral storage, cold starts, no uptime guarantee. Store state in the repo or a remote, and make the job re-runnable.
• Who should not use this: teams producing regulated or security-sensitive docs, projects where docs are the product and need human authorship, and repos without a docs owner. The bot does not replace the owner; it makes the owner's job visible.

The takeaway

Docs drift is a review problem. A free-tier bot can draft the impact note, but the decision — what to update, when, and how to phrase it — stays with a human. That division is the entire workflow.

If you want to try the pattern, MonkeyCode's free model access and free server option are enough to run it end to end. The script above is the whole pipeline. The hard part is the review habit — and that part is yours.