DEV Community

Avery Lin
Avery Lin

Posted on

Stop Reviewing Every AI-Written Sentence: Let a Free Server Do the Triage First

You generated a 3,000-word doc with an AI model. Now a reviewer must read all of it. That is the trap.

Most sentences in a generated doc are safe. They describe what the code does, which you can check by reading the source. A handful of sentences are dangerous. They claim numbers, versions, security behavior, or hard rules. Those few sentences cause incidents. Full-document review wastes hours finding them. Skim review misses them. The fix is a triage layer that shows you only the dangerous sentences.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode in this workflow because it offers free models for drafting and a free server for running automation. Both are useful for a CI-style docs check. The technique itself works with any stack.

The Triage Workflow: Draft, Classify, Own

Your pipeline has three stages:

  1. Draft – Use a free model to convert code comments or function signatures into initial prose.
  2. Classify – Run a small Python script that splits Markdown into sentences and tags risk patterns.
  3. Own – Review only the tagged sentences. Approve or rewrite each one. Commit your decision.

The classifier does not check truth. It checks danger potential. A sentence with "supports 10,000 requests/s" gets flagged. A sentence that says "the function returns a list" does not. You decide what to verify.

The Classifier: 70 Lines of Python

Save this as risk_triage.py. It uses only the standard library, so it runs almost anywhere.

import re, json, sys
from pathlib import Path

# Extend these rules for your domain
CATEGORIES = {
    "performance": r"\b(speed|latency|throughput|overhead|fast|slow)\b",
    "quantitative": r"\b\d+(\.\d+)?\s*(%|ms|s|gb|mb|kb|requests|rps|users|times)\b",
    "version": r"\b(v?\d+\.\d+(\.\d+)?|latest|current|stable|deprecated)\b",
    "security": r"\b(secure|security|safe|encrypt|auth|permission|private|secret)\b",
    "compatibility": r"\b(compatible|supports|works with|integrates|requires)\b",
    "recommendation": r"\b(should|must|never|always|avoid|recommend|use)\b",
    "uncertainty": r"\b(may|maybe|could|probably|likely|usually)\b",
}

def sentences(text):
    text = re.sub(r'^---\n.*?\n---\n', '', text, flags=re.DOTALL)
    text = re.sub(r'```

.*?

```', '', text, flags=re.DOTALL)
    parts = re.split(r'(?<=[.!?]) +', text)
    return [p.strip() for p in parts if len(p.strip()) > 15]

def classify(path):
    text = Path(path).read_text(encoding='utf-8')
    out = []
    for s in sentences(text):
        hits = [cat for cat, pat in CATEGORIES.items() if pat.search(s, re.I)]
        if hits:
            out.append({
                "sentence": s,
                "categories": hits,
                "risk": "high" if len(hits) >= 2 else "medium"
            })
    return out

if __name__ == "__main__":
    results = []
    for pattern in sys.argv[1:]:
        for p in Path(pattern).glob("**/*.md"):
            results += classify(p)
    print(json.dumps(results, indent=2))
    high = sum(1 for r in results if r["risk"] == "high")
    print(f"\n{len(results)} flagged, {high} high-risk", file=sys.stderr)
Enter fullscreen mode Exit fullscreen mode

Run it locally first: python risk_triage.py docs/. You will see a JSON report. The next step is automating it so the report appears on every pull request.

Free Server: Where the Automation Lives

MonkeyCode’s free server option gives you a lightweight environment for exactly this kind of batch job. You upload the script, point it at your repo, and let it run after every push. No credit card needed. No always-on machine at home.

A free server matters because risk triage must happen frequently to be useful. If you run the script manually before a docs PR, you will forget half the time. If it runs automatically, the report becomes a natural part of your review checklist.

Here is a sample server-side runner that logs results and blocks merges when the high-risk count exceeds your threshold:

# run_on_server.py – executes inside the free server
import json, subprocess, sys, os

subprocess.run(["python", "risk_triage.py", "docs", "-o", "report.json"])
report = json.load(open("report.json"))
high = [r for r in report if r["risk"] == "high"]
limit = int(os.environ.get("HIGH_RISK_LIMIT", "5"))

for r in high:
    print("\nHIGH:", r["sentence"])

if len(high) > limit:
    print("\nBlocking: too many high-risk sentences.")
    sys.exit(1)
print("\nOk.")
Enter fullscreen mode Exit fullscreen mode

You now have a free, repeatable triage bot. It does not verify facts. It tells the human where to look. That is the division of labor that scales.

Decision Table: What the Human Owns

Use this table when the report arrives.

Risk level Trigger example Human action
High "The API handles 50k RPS with no cache" Verify with a benchmark or delete the claim.
High "Works with v2 and v3 of libfoo" Check the actual test matrix, then cite it.
Medium "You should enable TLS" Confirm the recommendation matches your policy.
Low / none "The function returns a list" Skip unless you are the code owner.

Write the human decisions directly into the doc as an <!-- reviewed: date, by --> comment. That gives the next reviewer a traceable record.

Limitations: When Not to Use This

This classifier is a heuristic. It misses subtle lies. A sentence like "This is eventually consistent" may pass your generic regexes unless you add "eventually" to your domain list. You must extend the rules for your own vocabulary. It also produces false positives, especially in docs with many numeric examples. And it is useless for non-English docs unless you translate every pattern.

Do not use this as a replacement for running your code examples. Execution catches syntax errors; the classifier catches risky assertions. Use both. Also skip this entire workflow if you maintain one short README. The triage overhead will outweigh the time saved.

Start With Two Files

Create risk_triage.py and the server-side runner. Run the classifier on an existing docs folder. Look at the high-risk items. You will probably find claims nobody can back up. That feeling—the one where you say "wait, who measured that?"—is the value. Now you know exactly which sentences need your signature. The free model drafted them. The free server filtered them. You decide what survives.

That is the only workflow worth keeping.

Top comments (0)