The Bug Was in the Docs: Building Regression Tests for Written Content

Our documentation had a bug. Not a typo — a bug, in the strict sense: a wrong value that propagated through everything downstream of it.

I work on a Brazilian card game platform that has been online since 2010 and has 3M+ registered players. We run Buraco, Canastra, Tranca, Truco, Sueca and Dominó. On a platform like this, the rules pages aren't marketing collateral. They are the reference implementation of the product, in prose. People read them mid-game to settle arguments.

In August we found out that one paragraph on one rules page had been wrong for a long time. And because that page was the source everything else was checked against, roughly forty other pieces of content had inherited the same wrong sentence: blog posts, FAQ entries, FAQPage JSON-LD, a published video, social captions, a help-center article.

Every one of those had been reviewed. The reviews all passed.

That's the part worth writing about.

Validation is only as good as your oracle

The wrong claim was about the Truco rule for hiding a card. The docs said hiding wasn't allowed in the first round. The actual game behavior is different: the player who opens a round can't hide their card; from the second player onward, hiding is allowed — in any round, including the first.

The client tells you this plainly. There's a hideCard flag that flips to true the moment any card hits the table in a round, and resets at the end of each round. The game even prints the correct rule in its own error message.

So the ground truth was sitting right there in the running product. But nobody was checking against the running product. Everybody was checking against the docs — including the people writing new docs.

We had a review step that read, in every checklist, as some version of:

✅ validated against the official rules

That step did exactly what it said. It compared new content to the source of truth and confirmed a match. The content was faithful. The source was wrong. Validating against a wrong oracle validates the error and stamps it as verified.

If you've ever had a test suite that was green because the fixture was generated from the buggy implementation, this is the same failure, with prose instead of assertions.

Content has a dependency graph

The instinct after an incident like this is "let's do a careful audit and fix all forty places." We did that. It's necessary and it's not sufficient, because it doesn't change anything structural. Six months later somebody writes a new post, looks up the rule, finds the corrected page — fine — but also finds three older assets we missed, or a cached snippet, or an LLM answer trained on the old page, and the claim walks right back in through a side door.

A retracted fact behaves like a regression. It gets reintroduced by someone who wasn't there for the fix and has no way to know the claim is dead.

The standard answer for reintroduced bugs in code is: write a test that fails on the bug, keep it forever. So we did the content version of that.

Three things we built

1. An errata ledger, append-only and dated

One markdown file. Every claim we've ever proven false gets an entry, and entries never get deleted — only added to.

## E001 · Truco · hiding a card

- **Wrong claim:** "hiding a card is not allowed in the first round"
- **Correct claim:** the player who OPENS the round cannot hide. From the
  second player onward, hiding is allowed — in ANY round, including the first.
- **How we found out:** reported 2026-08-31. Ground truth = actual game
  behavior (the `hideCard` flag, and the game's own error message).
  The official rules page was the ORIGIN of the error — all downstream
  content was "validated" against it and inherited the claim.
- **Date:** 2026-08-31
- **Fixed in:** PR #859 (merged) · 7 published posts · 1 re-rendered video ·
  structured data · social queue

…followed, in the same entry, by a fenced block tagged regex containing the
pattern that catches the wrong phrasing:

(is\s+not\s+allowed|cannot|can't)[^.]{0,60}hid(e|ing)[^.]{0,60}first\s+round

Three fields carry all the weight:

• How we found out — the provenance. Without it you can't tell an errata entry from someone's opinion.
• Date — so a future reader knows which of two contradictory sources is newer.
• Fixed in — so the cleanup is auditable, and so the next person can tell whether an old asset was missed or deliberately left alone.

2. A pre-publish check that reads the ledger

The fenced regex blocks aren't decoration. A checker parses them out and runs them against any draft before it ships.

import re
from pathlib import Path

TICKS = "`" * 3  # avoid nesting literal fences inside this file
FENCE = re.compile(TICKS + r"regex(-warn)?\n(.*?)" + TICKS, re.S)

def load_patterns(errata: Path):
    text = errata.read_text(encoding="utf-8")
    # split on entry headings so each pattern keeps the label it belongs to
    chunks = re.split(r"^## ", text, flags=re.M)[1:]
    for chunk in chunks:
        label = chunk.splitlines()[0].strip()
        for warn, body in FENCE.findall(chunk):
            yield label, bool(warn), re.compile(body.strip(), re.I)

def check(draft: Path, errata: Path) -> int:
    text = draft.read_text(encoding="utf-8")
    failed = False
    for label, is_warn, pattern in load_patterns(errata):
        m = pattern.search(text)
        if not m:
            continue
        kind = "WARN" if is_warn else "FAIL"
        print(f"{kind} [{label}] matched: {m.group(0)[:90]!r}")
        failed |= not is_warn
    return 1 if failed else 0

It runs in well under a second, costs nothing, and is completely deterministic. That last property is the point. I did consider having a model review drafts against the errata, and it's a reasonable second layer — but for this job I want something that blocks a known-bad string and has no opinions, no temperature, and no bill. A retracted claim is a string-matching problem, and string matching is a solved problem.

3. Two severities, because context-dependent claims exist

The first version only had FAIL and it was immediately annoying, because some claims are true in one game mode and false in another.

Example from our own ledger: the "500-point canastra" and "1000-point canastra" bonuses exist in three of our Buraco variants and do not exist in Canastra (Buraco Italiano) or Tranca. The sentence "the 500-point canastra is a full Ace-to-King run" is perfectly correct in a Buraco article and wrong in a Tranca one. A blind FAIL on that phrase would block correct content, and a check that blocks correct content gets disabled within a week.

So the ledger has two fence types:

• a fence tagged regex → FAIL. The phrasing is wrong in every context. Non-zero exit.
• a fence tagged regex-warn → WARN. The claim depends on which mode you're writing about. Prints, doesn't block.

The warn tier is what keeps the check credible enough that people leave it turned on. A linter everyone bypasses is worse than no linter, because it also produces false confidence.

The provenance rule

The other change had no code in it at all. We banned a sentence from our own process notes:

validated 100% against the official rules

and replaced it with:

checked against <file or URL> on <date>

Source and date, or it isn't a verification. This costs nothing and it's the single highest-leverage thing on the list, because it makes every past check re-runnable. When we found out the Truco page was wrong, the useful question was "which content was checked against that page, and when?" With the old phrasing, unanswerable. With the new one, it's a grep.

It also reframes what a check is. "Validated" sounds like a property of the content. "Checked against X on date D" is honest about being a property of the check — one that can expire when X turns out to be wrong.

If you ship content, steal this

You don't need our stack. The pattern is three properties:

1. The ledger is append-only and dated. Deleting a retracted claim destroys the reason the guard exists.
2. Patterns come from the sentence that actually shipped, not from an idealized version of the mistake. You're guarding against how the error phrases itself, and the wild form is the one that will recur.
3. Every entry records where it was corrected. Otherwise the next audit re-litigates the same forty files from scratch.

And the meta-lesson, which applies well beyond docs: when a check passes, ask what it compared against. A green result tells you the artifact matches the oracle. It tells you nothing whatsoever about the oracle.

Our rules pages — the corrected ones — are public at jogosdorei.com.br, including the Truco rule that started all of this. The errata entry for it will be there for as long as the platform is, which is the whole idea.