Anchor Troubleshooting Docs to Real Error Strings Before a Model Drafts Them

Troubleshooting tables are the only documentation most users read while something is already broken, which makes them the worst place for an invented cause. The cheapest gate I know is also the dullest one: require every row to quote an error string that exists in the repository today, and reject any row whose quote cannot be found. Everything below is about building that gate, not about prompting your way to better prose.

The failure mode is structural rather than stylistic. A model asked to "document common errors" will produce symptom and cause pairs that read well, because plausibility is exactly what the training signal rewards. A reviewer then has to decide, row by row, whether a paragraph describes something the code can actually do, and that judgement is expensive. String matching is not a judgement at all, so it should run first and absorb the cheapest failures.

Why the error string is the cheapest available anchor

Error messages are literals in source code, so they can be extracted and compared without any semantic understanding. That property makes them a better anchor than headings, product names, or ticket links, all of which drift silently. A row that quotes real text is at least connected to a code path that exists.

Quoting is also a deliberately lossy operation for a model. Paraphrasing a string breaks the match, and a model that has not read the source will paraphrase almost immediately. You are not measuring honesty; you are measuring whether the drafting step was actually grounded in repository content rather than in general knowledge about error handling.

The gate does not tell you whether the cause is correct, whether the remedy works, or whether the row belongs in the docs. Those stay human decisions, and pretending otherwise is how teams end up trusting a checker for something it never measured.

Step 1: extract an anchor index from source

The extractor below walks a source tree, keeps only single-argument error constructions, and writes a JSON index. It deliberately skips multi-argument calls, because positional arguments there are usually codes or context objects rather than the message a user sees.

# anchors.py
import json, pathlib, sys

QUOTE_CHARS = chr(39) + chr(34)  # single and double quote
TRIGGERS = ('raise ', 'throw new ', 'throw ')

def message_of(line):
    if '(' not in line:
        return None
    rest = line.split('(', 1)[1]
    end = rest.find(')')
    if end == -1:
        return None
    args = rest[:end]
    if ',' in args:
        return None
    msg = args.strip().strip(QUOTE_CHARS).strip()
    return msg or None

def index(root):
    rows = []
    for path in pathlib.Path(root).rglob('*'):
        if path.suffix not in {'.py', '.js', '.ts'} or not path.is_file():
            continue
        text = path.read_text(encoding='utf-8', errors='ignore')
        for lineno, line in enumerate(text.splitlines(), 1):
            stripped = line.strip()
            if not any(t in stripped for t in TRIGGERS):
                continue
            msg = message_of(stripped)
            if msg:
                rows.append({'anchor': msg, 'file': str(path), 'line': lineno})
    return rows

if __name__ == '__main__':
    print(json.dumps(index(sys.argv[1] if len(sys.argv) > 1 else '.'), indent=2))

Run it once per release and store the output next to the docs, not in the repository root where it will be mistaken for a config file.

python anchors.py ./src > docs/_generated/error_index.json
python -c "import json;print(len(json.load(open('docs/_generated/error_index.json'))))"

The second command is a sanity check, and it matters more than it looks. A sudden drop in count usually means the extractor stopped matching a refactored call style, not that the codebase became quieter. Treat a large delta as a bug in the gate before you treat it as a signal about the docs.

Step 2: constrain the draft to a row schema

Give the model the index and the diff, and ask for rows that use only the fields you can check. The contract below makes the human boundary explicit by requiring remedy to be null, which turns an accidental invention into a schema violation rather than a review question.

You are drafting rows for a troubleshooting table.
Inputs: error_index.json (verbatim strings), diff.txt (changed code).
Output: a JSON array. Every element must have exactly these keys:

  symptom        one sentence describing what the user observes
  error_anchor   copied verbatim from error_index.json, no edits
  likely_causes  0-3 strings, each prefixed with "hypothesis: "
  verify_command one shell command a reader can run to confirm the symptom
  remedy         always null

Rules: do not invent anchors, do not merge two anchors into one row,
and do not describe a cause you cannot tie to a line in diff.txt.

Three details do the real work here. Copying verbatim is a mechanical instruction that models can follow reliably, which keeps the rejection rate informative instead of noisy. The hypothesis: prefix stops a guess from being read as a diagnosis during review. And requiring a verify command forces each row to describe a reachable state rather than a mood.

Step 3: validate mechanically before any human reads it

Run the checker in continuous integration, fail the job on schema errors, and print a per-row reason for every rejection so authors can see what was dropped.

# check_rows.py
import json, re, sys

TEMPLATE = re.compile('[{][^}]*[}]')
REQUIRED = ('symptom', 'error_anchor', 'likely_causes', 'verify_command', 'remedy')

def normalize(value):
    return TEMPLATE.sub('<v>', str(value)).strip()

def load(path):
    with open(path, encoding='utf-8') as handle:
        return json.load(handle)

anchors = {normalize(row['anchor']) for row in load(sys.argv[1])}
rows = load(sys.argv[2])
accepted, dropped = [], []

for row in rows:
    missing = [key for key in REQUIRED if key not in row]
    if missing:
        dropped.append({'reason': 'missing-fields', 'keys': missing, 'row': row})
        continue
    anchor = normalize(row['error_anchor'])
    if anchor not in anchors:
        dropped.append({'reason': 'anchor-not-found', 'row': row})
    elif row['remedy'] is not None:
        dropped.append({'reason': 'remedy-must-be-null', 'row': row})
    else:
        accepted.append(row)

print(json.dumps({'accepted': len(accepted), 'dropped': len(dropped),
                  'rows': accepted, 'drop_log': dropped}, indent=2))
sys.exit(2 if any(d['reason'] == 'missing-fields' for d in dropped) else 0)

A rejection reason like anchor-not-found is useful output rather than an error, because it tells the next author which strings the model believed existed. Exit code 2 is reserved for schema damage, since a malformed array is a pipeline bug; a fabricated anchor is an ordinary, expected event that should not block a merge by itself.

Step 4: route the residue to humans with an ownership table

Most drafting workflows fail here, because the review step inherits every field without deciding who is accountable for each one. The table below is the one I would attach to the pull request template, so the split is visible before anyone opens the diff.

Field	Model may draft	Human must own
symptom	yes	approve wording and scope
error_anchor	copy only	confirm the error still fires
likely_causes	as labelled hypotheses	confirm or delete each one
verify_command	propose	actually run it once
remedy	never	author and test
severity, audience, ordering	never	author

The verify_command row is the one teams skip most often, and it is also the cheapest to enforce. If nobody has run the command, the row has not been shown to describe anything a reader can reach. One run per row is enough; a run per release is not, because commands rot faster than prose.

A three-fixture test plan for the gate

Before you trust the checker on real drafts, prove it rejects the two cases that matter. The fixtures below are short enough to keep in the same directory as the scripts, and label them clearly as fixtures rather than documentation.

Fixture	Input	Expected result
Happy path	one row quoting an anchor from error_index.json	accepted, exit 0
Invented anchor	one row quoting a plausible but absent string	dropped with anchor-not-found
Remedy leak	valid anchor, remedy filled in	dropped with remedy-must-be-null

If the invented-anchor fixture passes, your normalization is too loose or the checker is reading the draft file twice. That single test catches most of the ways this gate quietly stops working.

Limitations and who should not use this

Template literals, %s placeholders, and .format() calls produce anchors that will not match after normalization, so those rows will be rejected even when they are correct. Extend normalize() deliberately and keep the extension small, because a normalizer that strips too much will match unrelated strings and destroy the gate's value.

Errors raised inside dependencies, messages assembled at runtime, and failures that arrive as bare stack traces have no stable anchor at all. For those, this workflow yields nothing, and you should keep the existing human-only process instead of forcing a match. Teams whose messages are already user-facing prose should also skip it, since the anchor duplicates the symptom and adds review work without adding evidence.

The gate also says nothing about correctness of causes or remedies, and a high acceptance rate can coexist with a table full of wrong diagnoses. Read the accepted count as a measure of grounding, never as a measure of quality.

Where free model access and a free server change the loop

For full transparency: Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow above wants to run on every pull request rather than once per release, and that changes the cost profile of the drafting step. The operator of this account states that MonkeyCode offers free model access and a free server option, which is enough to keep a per-PR draft step and a small hosted validator in the loop without a budget conversation. I am describing the setup I would build, not publishing measured throughput, and you should verify current limits, model availability, and duration on the pricing page yourself before depending on them.

Keep the checker local or in your own CI regardless of where drafting happens, because the validation step must not depend on the service that produced the text. If you host the validator on the free server, give it the index and the draft as inputs and nothing else, so a compromise or an outage cannot corrupt the source of truth. The draft step is replaceable; the anchor index is not.

What to keep from a rejected row

Rejected rows are the most useful artifact this gate produces, because each one is a claim a model was willing to make without source support. Collect them per release, tag the recurring ones, and use them to fix prompts or extractors rather than to argue about model quality in the abstract.

If you try the anchor gate on one release, keep the drop log and compare it against the rows you eventually shipped. That comparison is the only number that tells you whether the gate is saving review time or just moving it.

# the whole loop, one release
python anchors.py ./src > docs/_generated/error_index.json
python check_rows.py docs/_generated/error_index.json draft_rows.json > reviewed.json