Snapshot Tests Lie About Markdown: Assert Structure Instead

A Markdown parser upgrade can pass every “does it render?” check and still change the document.

The dangerous regressions are quiet: a malformed link starts auto-linking a URL, a list changes nesting, an invisible character survives normalization, or a plugin turns previously literal text into an extension node.

An HTML snapshot will notice some of those changes. The problem is that it also notices quote style, optional closing tags, whitespace, attribute order, and renderer-specific formatting. Once a dependency upgrade changes hundreds of snapshots, “update all” becomes tempting—and that can approve the semantic regression with the noise.

I ran a small fixed-version experiment to separate structure from serialization.

The experiment

Environment:

• Node.js 25.3.0
• commonmark.js 0.31.2
• markdown-it 15.0.0 with the commonmark preset
• Marked 18.0.9 with default options

The corpus deliberately includes four classes of input:

const cases = [
  { id: "normal", source: "# Release\n\n- parse\n- render\n" },
  { id: "difficult", source: "Read [the *nested* case](https://example.com/a_(b)).\n" },
  { id: "failure", source: "Broken [link](<https://example.com\n" },
  { id: "edge-nul", source: "left\u0000right\n" },
];

Instead of first comparing the entire HTML string, the harness extracts the contract we actually care about:

function structuralFacts(html) {
  return {
    heading: /<h1(?:\s|>)/.test(html),
    list: /<ul(?:\s|>)/.test(html) && /<li(?:\s|>)/.test(html),
    link: /<a\s+href=/.test(html),
    replacement: html.includes("�"),
  };
}

All 12 parser-fixture assertions passed after the expected capability differences were made explicit.

The differences were the useful result

All three parsers produced the same structures for the normal document and the nested-emphasis link.

For the unclosed link destination, commonmark.js and markdown-it emitted literal text. Marked did not create the intended Markdown link either, but its default URL auto-linking created an anchor for the URL inside the broken syntax.

For the U+0000 case, commonmark.js and markdown-it emitted U+FFFD, matching CommonMark's input-character rule. The HTML string returned by Marked still contained U+0000.

That does not make differential testing a vote. Two implementations agreeing does not establish the product contract. The correct reference depends on the layer:

• CommonMark core: the pinned CommonMark spec examples
• GFM or another extension: that extension's spec and the enabled configuration
• product behavior: an explicit, reviewed project decision
• unresolved divergence: a report, not an automatically accepted snapshot

Start with upstream conformance fixtures

The CommonMark specification repository embeds more than 500 examples that act as conformance tests. Its test tool can dump them as JSON records containing the Markdown input, expected HTML, section, and example number.

That is a much stronger base than a hand-written “Markdown basics” file:

for (const example of commonmarkSpec.tests) {
  assert.equal(
    render(example.markdown),
    example.html,
    `CommonMark example ${example.number}`
  );
}

But conformance is not the whole product. Those fixtures do not cover your sanitizer, editor transactions, plugin ordering, export template, or platform-specific extensions.

Add one permanent fixture per real bug

Every parser bug should leave behind:

1. the smallest input that reproduces it;
2. parser, plugin, preset, and option versions;
3. the expected structural facts;
4. the issue or commit that explains why the expectation exists;
5. the boundary: spec requirement, extension, or product policy.

Do not delete that fixture when the implementation changes. If the intended behavior changes, update the contract in a reviewable change that explains why.

Use snapshots as evidence, not as the oracle

Raw HTML snapshots are still useful when exact serialization is a public API. Keep them beside structural assertions.

For most editor and publishing systems, stronger assertions target:

• node types and nesting;
• heading levels;
• link and image destinations;
• literal text preservation;
• source spans when diagnostics depend on them;
• sanitizer results after rendering;
• stable IDs or attributes that downstream code consumes.

When a table fixture is needed, I use a visual generator only to create the source, then test the generated Markdown in the target parsers. In a September 2 check, the MDFold Markdown Table Generator escaped Maya | UX as Maya \| UX; that proves the current generator's output for this fixture, not universal table support. The same page also stayed within a 390-pixel viewport without page-level horizontal overflow.

Property tests need real invariants

“Every parser must produce identical HTML” is not a valid property across Markdown dialects.

Better properties include:

• parsing bounded input never crashes;
• the same version and options produce a deterministic result;
• the AST contains no parent-child cycles;
• disabling raw HTML produces no raw-HTML nodes;
• bounded input size and nesting do not produce unbounded work.

Property-based testing tools combine generated inputs with predicates. Record the seed for every failure, shrink it, and add the minimal result to the permanent regression corpus. Otherwise the interesting failure disappears after the random run.

Keep pathological tests separate

Large unmatched delimiter runs, deep brackets, nested lists, and unclosed HTML comments test complexity rather than ordinary correctness. markdown-it's own repository separates CommonMark fixtures, implementation fixtures, and pathological tests; the pathological suite uses an isolated worker and a timeout.

That separation is important. Run fast semantic fixtures on every commit. Run larger complexity suites nightly or before releases, using a stable input scale and a generous budget. A noisy wall-clock threshold on shared CI is not a reliable algorithm test.

A practical test layout

fixtures/
  commonmark/      # upstream, versioned conformance data
  regressions/     # one directory per real bug
  extensions/      # GFM tables, task lists, footnotes...
  security/        # raw HTML, protocols, remote resources
  pathological/    # deep, long, and unclosed inputs

For dependency upgrades, generate a report with three sections: new differences, removed differences, and unclassified differences. Block the upgrade on the last section. That makes “update snapshots” a reviewed decision instead of a reflex.

Sources

• CommonMark 0.31.2 specification and examples: https://spec.commonmark.org/0.31.2/
• CommonMark spec test tooling: https://github.com/commonmark/commonmark-spec
• markdown-it test layout and pathological cases: https://github.com/markdown-it/markdown-it/tree/master/test
• fast-check property model: https://fast-check.dev/docs/core-blocks/properties/

What does your Markdown suite protect today: the document's meaning, or one renderer's current whitespace?