DEV Community: frank

Source Positions Are a Coordinate System, Not Just Line and Column

frank — Fri, 04 Sep 2026 06:24:16 +0000

A Markdown linter can detect the right problem and still highlight the wrong character.

The usual cause is not parsing. It is a coordinate-system mismatch: one component counts UTF-8 bytes, another counts Unicode code points, and JavaScript indexes UTF-16 code units.

I tested this with Node.js 25.3.0, unified 11.0.5 + remark-parse 11.0.0, commonmark.js 0.31.2, markdown-it 15.0.1 (commonmark preset), and Marked 18.0.11.

One emoji, three valid offsets

Consider this source:

# A😀B

The position immediately before B is:

Unit	Offset
JavaScript UTF-16 code units	5
Unicode code points	4
UTF-8 bytes	7

const source = "# A😀B";
const utf16 = source.indexOf("B");
const codePoints = [...source.slice(0, utf16)].length;
const bytes = Buffer.byteLength(source.slice(0, utf16), "utf8");

console.log({ utf16, codePoints, bytes });
// { utf16: 5, codePoints: 4, bytes: 7 }

None of those numbers is universally wrong. The bug appears when an API calls all of them offset.

Positions should be half-open ranges

The unist specification defines a position with start and end points. Lines and columns are one-based, offsets are zero-based, and end points to the first character after the source region. Its definition of a character is a UTF-16 code unit.

remark-parse produced this position for the A😀B text node:

{
  "start": { "line": 1, "column": 3, "offset": 2 },
  "end":   { "line": 1, "column": 7, "offset": 6 }
}

That makes source recovery unambiguous:

source.slice(node.position.start.offset, node.position.end.offset)

Treating end as inclusive introduces an off-by-one error.

Four parsers, four position capabilities

The same input produced materially different public metadata:

Parser	Block positions	Inline positions	Absolute offset
remark-parse 11.0.0	yes	yes	UTF-16
commonmark.js 0.31.2	yes	no	no
markdown-it 15.0.1	line ranges	no	no
Marked 18.0.11	no standard field	no	no

commonmark.js reported the heading as [[1,1],[1,6]], but its A😀B text node had no sourcepos. markdown-it reported [0,1] on the inline token, while every inline child had a null map. Marked exposed raw, not a unique source location.

This is a capability difference, not a rendering-quality ranking. Block ranges are enough for scroll synchronization. Character-accurate quick fixes need inline ranges and explicit offsets.

Do not reconstruct positions with indexOf(token.raw). Repeated text makes the result ambiguous. Accumulating raw.length also breaks when a parser normalizes newlines, ignores a BOM, or merges text nodes.

Test failure syntax, not only valid syntax

I used four fixtures:

const fixtures = {
  normal: "# Title\n\nA paragraph.",
  difficult: "# A😀B\n\nUse **e\u0301** and `code`.",
  failure: "# Broken [link](<oops\n\nTail",
  boundary: "\uFEFF# Zero\u200BWidth\r\n\r\nEnd",
};

The failed link never became a link node. remark kept the failed construct in one text node; commonmark.js split it into several text nodes; markdown-it still only identified the containing line. A diagnostic implementation cannot assume malformed syntax has the AST shape of successful syntax.

The boundary fixture exposed another divergence. With a leading BOM, remark still recognized the heading, while commonmark.js parsed the first line as a paragraph. The zero-width character remained part of the text, and CRLF consumed two UTF-16 code units.

When the first question is whether a real file is Markdown, plain text, or affected by encoding, this Markdown file structure guide is the contextual checklist I use. It does not imply that parsers normalize those boundaries consistently.

A practical range contract

For a JavaScript editor, I would make the unit part of the field name:

type SourcePoint = {
  offsetUtf16: number;
  line: number;
  columnUtf16: number;
};

type SourceRange = {
  start: SourcePoint;
  end: SourcePoint; // exclusive
  sourceVersion: string;
};

If a Rust or Go service returns byte offsets, call the field offsetUtf8Bytes and convert against the exact same source text. sourceVersion matters because every range becomes suspect after formatting or editing.

For occasional lookups, scan from the start to compute line and column. For many diagnostics, precompute every line-start offset and use binary search. Decide how CRLF is counted and encode that decision in tests.

What I would regression-test

an emoji before the target;
a decomposed character such as e\u0301;
LF and CRLF;
a leading BOM;
a zero-width character;
repeated identical text;
malformed links and emphasis;
generated AST nodes with no source range.

The invariant is stronger than a snapshot: slicing the original source with a node's half-open offsets should reproduce that node's source spelling whenever the node genuinely came from one continuous region.

Source positions are a protocol between parser, analyzer, editor, and formatter. Define the unit, range semantics, and source version before trusting the number.

After a formatter rewrites the document, would you invalidate every diagnostic and reparse, or maintain a source map from the previous version?

Snapshot Tests Lie About Markdown: Assert Structure Instead

frank — Wed, 02 Sep 2026 08:22:56 +0000

A Markdown parser upgrade can pass every “does it render?” check and still change the document.

The dangerous regressions are quiet: a malformed link starts auto-linking a URL, a list changes nesting, an invisible character survives normalization, or a plugin turns previously literal text into an extension node.

An HTML snapshot will notice some of those changes. The problem is that it also notices quote style, optional closing tags, whitespace, attribute order, and renderer-specific formatting. Once a dependency upgrade changes hundreds of snapshots, “update all” becomes tempting—and that can approve the semantic regression with the noise.

I ran a small fixed-version experiment to separate structure from serialization.

The experiment

Environment:

Node.js 25.3.0
commonmark.js 0.31.2
markdown-it 15.0.0 with the commonmark preset
Marked 18.0.9 with default options

The corpus deliberately includes four classes of input:

const cases = [
  { id: "normal", source: "# Release\n\n- parse\n- render\n" },
  { id: "difficult", source: "Read [the *nested* case](https://example.com/a_(b)).\n" },
  { id: "failure", source: "Broken [link](<https://example.com\n" },
  { id: "edge-nul", source: "left\u0000right\n" },
];

Instead of first comparing the entire HTML string, the harness extracts the contract we actually care about:

function structuralFacts(html) {
  return {
    heading: /<h1(?:\s|>)/.test(html),
    list: /<ul(?:\s|>)/.test(html) && /<li(?:\s|>)/.test(html),
    link: /<a\s+href=/.test(html),
    replacement: html.includes("�"),
  };
}

All 12 parser-fixture assertions passed after the expected capability differences were made explicit.

The differences were the useful result

All three parsers produced the same structures for the normal document and the nested-emphasis link.

For the unclosed link destination, commonmark.js and markdown-it emitted literal text. Marked did not create the intended Markdown link either, but its default URL auto-linking created an anchor for the URL inside the broken syntax.

For the U+0000 case, commonmark.js and markdown-it emitted U+FFFD, matching CommonMark's input-character rule. The HTML string returned by Marked still contained U+0000.

That does not make differential testing a vote. Two implementations agreeing does not establish the product contract. The correct reference depends on the layer:

CommonMark core: the pinned CommonMark spec examples
GFM or another extension: that extension's spec and the enabled configuration
product behavior: an explicit, reviewed project decision
unresolved divergence: a report, not an automatically accepted snapshot

Start with upstream conformance fixtures

The CommonMark specification repository embeds more than 500 examples that act as conformance tests. Its test tool can dump them as JSON records containing the Markdown input, expected HTML, section, and example number.

That is a much stronger base than a hand-written “Markdown basics” file:

for (const example of commonmarkSpec.tests) {
  assert.equal(
    render(example.markdown),
    example.html,
    `CommonMark example ${example.number}`
  );
}

But conformance is not the whole product. Those fixtures do not cover your sanitizer, editor transactions, plugin ordering, export template, or platform-specific extensions.

Add one permanent fixture per real bug

Every parser bug should leave behind:

the smallest input that reproduces it;
parser, plugin, preset, and option versions;
the expected structural facts;
the issue or commit that explains why the expectation exists;
the boundary: spec requirement, extension, or product policy.

Do not delete that fixture when the implementation changes. If the intended behavior changes, update the contract in a reviewable change that explains why.

Use snapshots as evidence, not as the oracle

Raw HTML snapshots are still useful when exact serialization is a public API. Keep them beside structural assertions.

For most editor and publishing systems, stronger assertions target:

node types and nesting;
heading levels;
link and image destinations;
literal text preservation;
source spans when diagnostics depend on them;
sanitizer results after rendering;
stable IDs or attributes that downstream code consumes.

When a table fixture is needed, I use a visual generator only to create the source, then test the generated Markdown in the target parsers. In a September 2 check, the MDFold Markdown Table Generator escaped Maya | UX as Maya \| UX; that proves the current generator's output for this fixture, not universal table support. The same page also stayed within a 390-pixel viewport without page-level horizontal overflow.

Property tests need real invariants

“Every parser must produce identical HTML” is not a valid property across Markdown dialects.

Better properties include:

parsing bounded input never crashes;
the same version and options produce a deterministic result;
the AST contains no parent-child cycles;
disabling raw HTML produces no raw-HTML nodes;
bounded input size and nesting do not produce unbounded work.

Property-based testing tools combine generated inputs with predicates. Record the seed for every failure, shrink it, and add the minimal result to the permanent regression corpus. Otherwise the interesting failure disappears after the random run.

Keep pathological tests separate

Large unmatched delimiter runs, deep brackets, nested lists, and unclosed HTML comments test complexity rather than ordinary correctness. markdown-it's own repository separates CommonMark fixtures, implementation fixtures, and pathological tests; the pathological suite uses an isolated worker and a timeout.

That separation is important. Run fast semantic fixtures on every commit. Run larger complexity suites nightly or before releases, using a stable input scale and a generous budget. A noisy wall-clock threshold on shared CI is not a reliable algorithm test.

A practical test layout

fixtures/
  commonmark/      # upstream, versioned conformance data
  regressions/     # one directory per real bug
  extensions/      # GFM tables, task lists, footnotes...
  security/        # raw HTML, protocols, remote resources
  pathological/    # deep, long, and unclosed inputs

For dependency upgrades, generate a report with three sections: new differences, removed differences, and unclassified differences. Block the upgrade on the last section. That makes “update snapshots” a reviewed decision instead of a reflex.

Sources

CommonMark 0.31.2 specification and examples: https://spec.commonmark.org/0.31.2/
CommonMark spec test tooling: https://github.com/commonmark/commonmark-spec
markdown-it test layout and pathological cases: https://github.com/markdown-it/markdown-it/tree/master/test
fast-check property model: https://fast-check.dev/docs/core-blocks/properties/

What does your Markdown suite protect today: the document's meaning, or one renderer's current whitespace?

Regex Cannot Sanitize Markdown: Put the Security Boundary After Parsing

frank — Mon, 31 Aug 2026 07:10:45 +0000

Deleting <script> from Markdown with a regular expression is not an HTML security boundary. The browser eventually consumes a DOM, not the original Markdown string. A regex that matches one attribute spelling can miss another spelling the HTML parser accepts, while an aggressive replacement can damage text that was supposed to remain inside a code span.

The reliable order is:

Markdown source
  -> pinned parser
  -> HTML sanitizer for the real output policy
  -> controlled DOM sink
  -> CSP / Trusted Types as defense in depth

A regex that passes the easy test

I tested this deliberately narrow cleaner:

function naiveRegexClean(html) {
  return html
    .replace(/<script\b[^>]*>[\s\S]*?<\/script>/gi, '')
    .replace(/\son\w+="[^"]*"/gi, '')
    .replace(/javascript:/gi, '');
}

It removes a plain <script> and a double-quoted onerror. It does not remove the same event attribute when it is unquoted or single-quoted:

Input	Result
`<img src=x onerror="alert(1)">`	event attribute removed
`<img src=x onerror=alert(1)>`	event attribute remains
`<img src=x onerror='alert(1)'>`	event attribute remains

Replacing javascript: also turns the URL into a different string. It does not prove that the resulting URL matches the application's protocol or navigation policy.

The point is not that every regex is short. The point is that string matching and browser HTML parsing operate at different abstraction levels.

The fixed-version test

I used Node.js 25.3.0, Marked 18.0.7, DOMPurify 3.4.12, and MDFold's current browser-side Markdown-to-HTML path. The fixture covered ordinary formatting, raw scripts, quoted and unquoted event handlers, Markdown and raw-HTML javascript: links, inline code, a remote image, and a fixed-position style.

Marked preserved the raw HTML in its generated output. That is expected: the Marked documentation explicitly warns that it does not sanitize output HTML.

The real browser preview then sanitized the parsed HTML. It:

preserved bold text and an HTTPS link;
removed the script element and event handler;
removed unsafe href values while preserving link text;
preserved the attack-shaped string inside inline code as escaped text;
preserved the remote image URL;
preserved the tested style attribute.

This is more informative than checking whether an alert happened to appear. The test inspected the resulting elements, attributes, URLs, and text nodes.

Parsing, sanitizing, and resource policy are separate

CommonMark defines how raw HTML participates in Markdown parsing. It does not define a web application's XSS policy. OWASP recommends output encoding when data should remain text and HTML sanitization when authors are allowed to provide HTML. DOMPurify works on a parsed, inert DOM and applies an allow-list to elements and attributes.

An application that does not need raw HTML can reject HTML nodes earlier. That reduces the surface but does not answer every downstream question. Images can still cause network requests. Links can still leave the site. Plugins can still generate HTML. The final sink still owns the final policy.

The two preserved values in my test show the remaining boundaries:

A sanitized remote image can still make a request. XSS prevention and privacy are different controls.
A permitted style can still obscure content. Script safety and visual-integrity policy are different controls.

Do not modify sanitized HTML afterward

This sequence destroys the boundary:

const clean = DOMPurify.sanitize(dirty);
target.innerHTML = clean.replace('USER', untrustedValue);

Every untrusted value needs the defense for the context where it is inserted. Prefer safe sinks such as textContent for plain text. If a CMS or template engine reparses or mutates the HTML later, validate at that boundary too.

A practical review checklist

Pin parser and sanitizer versions separately.
Disable raw HTML when the product does not need it.
When HTML is required, start with a narrow allow-list.
Define protocol rules for href and resource rules for src.
Decide whether inline styles belong in the content model.
Avoid post-sanitization string concatenation.
Inspect the final DOM and network behavior, not only the source string.
Retest in the destination CMS, browser, and mobile layout.

I reproduced the browser portion with the MDFold Markdown-to-HTML converter. At a 390-pixel viewport the tested preview stayed within the page width, but that result covers this fixture and current version only.

The engineering question is not "which regex catches every payload?" It is "which component owns the policy for this exact output sink?"

Primary sources

Invisible Unicode Characters Can Break Markdown Without Looking Different

frank — Sat, 22 Aug 2026 22:44:45 +0000

Two Markdown lines can look identical and still produce different HTML. Before blaming the parser, inspect the code points.

# heading
#heading

The second line contains U+200B ZERO WIDTH SPACE after #. CommonMark requires the ATX marker sequence to be followed by a space, tab, or line ending, so this is a paragraph rather than a heading.

Make the invisible input observable

const codePoints = value => [...value].map(char =>
  `U+${char.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')}`
)

console.log(codePoints('#\u200Bheading'))
// U+0023 U+200B U+0068 ...

When the source came from an unknown editor, first verify that it is actually a text file and inspect its encoding. This guide to opening Markdown files is a useful checklist; the important step is inspection, not conversion.

What three fixed parser versions produced

I tested Node.js 25.3.0 with Marked 18.0.7, markdown-it 14.1.0, and commonmark.js 0.31.2. These are observed outputs under default options, not claims about every Markdown platform.

Input	Marked	markdown-it	commonmark.js
`# heading`	heading	heading	heading
leading U+FEFF before `#`	paragraph, FEFF retained	paragraph, FEFF omitted in output	paragraph, FEFF omitted in output
U+200B after `#`	paragraph	paragraph	paragraph
U+00A0 after `#`	heading	paragraph	paragraph
fullwidth `＃` or `＊`	plain text	plain text	plain text

The NBSP result is the useful warning: a character that looks like a space creates a real cross-parser difference. “Whitespace” is not one universal Markdown category.

A BOM is positional

UTF-8 uses EF BB BF as its optional signature. After decoding, that is U+FEFF. Unicode says a recognized initial BOM should be removed before text processing. The same code point in the middle of text is not automatically a BOM and must not be globally deleted.

In the experiment, a leading U+FEFF prevented # from being the first character seen by the Markdown grammar. An internal alpha\uFEFFbeta remained present in all three outputs. Fix BOM handling at the decoding boundary, not with replaceAll('\uFEFF', '').

Zero width does not mean zero semantics

U+200B is a format control used to indicate a word or line-break opportunity. Its effect depends on position:

after #, it prevents ATX-heading recognition;
inside **bo\u200Bld**, all three parsers still create <strong>, while preserving U+200B in the text node.

So a blanket “remove invisible characters” rule is both too broad and too weak.

Look-alike punctuation is different punctuation

＃ is U+FF03, not U+0023. ＊ is U+FF0A, not U+002A. All three parsers treated the fullwidth examples as ordinary text.

NFKC normalization can map some compatibility characters to ASCII, but silently normalizing user source changes data. A safer editor warns, previews the proposed change, and lets the author approve it.

Bidirectional controls require a different review model

UAX #9 distinguishes logical order from display order. Directional controls can change what reviewers see while parsers continue reading logical code-point order.

With U+202E inside a Markdown URL, none of the three parsers simply discarded it. Marked and commonmark.js percent-encoded it; markdown-it serialized the host differently through IDNA processing. The safe review target is therefore not just the visible source. Inspect the code points, parsed href, and final resolved URL.

A practical ingestion policy

Handle an initial BOM during byte decoding.
Preserve raw bytes in regression fixtures.
Flag Cf characters and non-ASCII look-alike punctuation near Markdown delimiters.
Show code point, line, column, and Unicode name in diagnostics.
Preview repairs instead of silently rewriting source.
Validate links after parsing and URL resolution.

Invisible-character bugs are boundary bugs: the decoder, editor, Markdown grammar, HTML serializer, and browser may each make a different decision. Good tooling makes those decisions observable.

Would you prefer an editor that automatically cleans invisible characters, or one that always preserves the source and only warns?

Markdown Images Are Network Requests, Not Embedded Assets

frank — Thu, 20 Aug 2026 06:08:34 +0000

The line ![diagram](../assets/flow.png) does not embed an image. It creates an image node whose destination still has to be resolved, fetched, allowed, and rendered by another system.

That distinction explains why a document can parse perfectly and still show broken images—or make network requests its author did not account for.

Parsing and URL resolution are separate stages

I fixed the experiment to Node 25.3.0 and Marked 18.0.7 with GFM enabled:

const html = marked.parse('![diagram](../assets/flow.png "Flow")');
// <p><img src="../assets/flow.png" alt="diagram" title="Flow"></p>

Marked preserves the destination. A browser or publisher later resolves it against a base URL:

new URL("../assets/flow.png", "https://docs.example.test/guides/setup/").href
// https://docs.example.test/guides/assets/flow.png

new URL("../assets/flow.png", "https://docs.example.test/guides/setup.html").href
// https://docs.example.test/assets/flow.png

Same Markdown, different request. The parser did its job in both cases.

Six cases that need different policies

Input	Parser output	What the host must decide
`../assets/flow.png`	relative `src`	base URL and asset copy rules
`/assets/logo.svg`	root-relative `src`	which origin owns the root
`https://.../pixel.png?doc=42`	absolute `src`	remote trust and request privacy
`data:image/...`	data URL	sanitizer and CSP policy
empty alt	`alt=""`	decorative or accessibility defect
empty destination	`src=""`	reject, rewrite, or host behavior

CommonMark defines image syntax and maps the description to alt text. It does not upload files or promise that a URL exists.

Remote images are third-party dependencies

When HTML contains a remote <img src>, the browser requests that resource. The remote server observes the request; the page's Referrer Policy controls how much referring-page information is sent. Query parameters can also encode a document or campaign identifier.

Treat remote images like any other external dependency:

allow only reviewed schemes and hosts;
proxy or self-host assets when appropriate;
set a deliberate Referrer Policy;
restrict image sources with CSP img-src;
sanitize final HTML after AST transformations;
never put sensitive identifiers in image URLs.

Parser configuration cannot replace those controls.

Relative paths need a publishing contract

A stable workflow defines:

where the Markdown file lives;
what base URL the final page uses;
how referenced assets are copied;
whether root-relative paths are allowed;
what happens when an image is missing.

If you are debugging a real document, this Markdown image path guide provides a concrete path checklist. The important step is to test the published URL, not just the editor preview.

Data URLs trade path stability for other costs

Data URLs avoid a separate file lookup, but they make source files larger and harder to review, prevent independent caching, and may be blocked by a sanitizer or CSP. They are reasonable for small controlled assets, not a universal fix for image portability.

Validate the final request graph

My recommended pipeline is:

parse Markdown to an AST;
inspect image destinations;
resolve them against the declared publication base;
validate scheme, host, path, and query;
copy or rewrite managed assets;
sanitize rendered HTML and apply CSP/referrer policy;
open the public page and verify actual responses and layout.

The AST tells you what the author referenced. URL resolution tells you where it points. A browser test tells you what really happened.

Should Markdown hosts block cross-origin images by default, or preserve compatibility and require each application to opt into a stricter policy?

Primary references

Your Markdown Parser Is Not Your XSS Boundary

frank — Wed, 19 Aug 2026 02:36:27 +0000

A Markdown parser can produce exactly the right HTML and still leave your application exposed to XSS. Parsing answers what the input means. Sanitization decides which parts of that meaning are allowed to reach an HTML sink.

I tested that boundary with Node.js 25.3.0, Marked 18.0.7, DOMPurify 3.4.12, and jsdom 30.0.1. The important comparison is not a screenshot. It is the HTML before and after sanitization.

The smallest useful pipeline

const rendered = marked.parse(markdown)
const sanitized = DOMPurify.sanitize(rendered, {
  USE_PROFILES: { html: true },
  SANITIZE_NAMED_PROPS: true,
})

This deliberately keeps two responsibilities separate. Marked parses Markdown. DOMPurify applies an allow-list to the HTML structure that will approach the browser.

Five cases that expose the boundary

1. Normal content survives

# Hello

[Safe](https://example.com)

The heading and HTTPS link survive both stages. A sanitizer should preserve allowed document structure, not flatten every document to text.

2. Raw HTML is valid Markdown, not necessarily safe HTML

<img src=x onerror="alert(1)">

Marked 18.0.7 returns the element and its event attribute unchanged. DOMPurify returns:

<img src="x">

The parser did not fail. CommonMark supports raw HTML. The unsafe step would be treating syntactic validity as authorization to insert every attribute.

3. URL schemes need their own policy

[click](javascript:alert(1))

Rendered HTML:

<p><a href="javascript:alert(1)">click</a></p>

Sanitized HTML:

<p><a>click</a></p>

An element allow-list alone is insufficient. URL-bearing attributes need scheme validation.

4. Code examples must not be cleaned as attacks

```html
<img src=x onerror="alert(1)">
```

The parser escapes the payload inside pre > code. A regex that removes attack-looking source before parsing would damage legitimate security documentation. Context has to be established first.

5. XSS defenses extend beyond `script`

<form id="attributes"><input name="action"></form>

With SANITIZE_NAMED_PROPS, the result becomes:

<form id="user-content-attributes"><input name="user-content-action"></form>

This addresses DOM clobbering: attacker-controlled names can interfere with properties that application code expects to resolve normally.

Put sanitization after the last unsafe transform

A practical pipeline is:

untrusted Markdown
  -> parser
  -> Markdown/HTML AST transforms
  -> sanitizer
  -> serializer
  -> matching HTML sink

The rehype-sanitize documentation makes the ordering rule explicit: sanitize after the last unsafe operation, because a later plugin can reintroduce unsafe properties. DOMPurify's current threat model adds another constraint: do not sanitize and then freely post-process the result. The policy and the sink must stay aligned.

Turning off raw HTML is a useful reduction in attack surface, but it is not a universal sanitizer. Plugins, link protocols, generated IDs, and later transforms still deserve explicit policies.

What I verify in a conversion workflow

When checking Markdown to HTML, I separate three questions:

Did normal Markdown preserve the intended structure?
Did fenced examples remain inert code?
Is untrusted output safe for this application's sink and policy?

The first two are conversion checks. The third belongs to the embedding application. A converter producing structurally correct HTML does not automatically promise that arbitrary input is safe to inject into another site's DOM.

Engineering checklist

Treat external Markdown as untrusted by default.
Disable raw HTML where the product does not need it.
Sanitize the final HTML tree with a maintained allow-list sanitizer.
Model URL schemes, id, name, and styling capabilities explicitly.
Do not run arbitrary HTML-mutating plugins after sanitization.
Test AST shape, rendered HTML, and sanitized HTML separately.
Pin and update sanitizer versions; security fixes are part of the boundary.

The design question I keep coming back to is this: should Markdown renderers disable raw HTML by default, or should they expose it only behind an explicit host-supplied security policy?

Primary sources

Build a Markdown Heading Linter on the AST, Not With Regex

frank — Sat, 15 Aug 2026 22:13:26 +0000

A heading linter sounds like a one-line regex:

/^#{1,6}\s+(.+)$/gm

It works until a documentation page contains a fenced example:

# Real heading

```md
## This is code, not navigation
```

The regex finds two headings. A Markdown parser finds one heading and one code block.

That difference is why I prefer to put document automation behind an AST boundary:

source -> parser -> mdast -> transformer -> hast -> HTML

A small, testable plugin

For this experiment I pinned unified@11.0.5, remark-parse@11.0.0, remark-rehype@11.1.2, rehype-stringify@10.0.1, and unist-util-visit@5.1.0 on Node 25.3.0.

The plugin has two jobs:

collect headings for a table of contents;
report a diagnostic when heading depth jumps by more than one level.

function headingAudit() {
  return (tree, file) => {
    const headings = []
    let previousDepth = 0

    visit(tree, 'heading', node => {
      const title = textOf(node)

      if (previousDepth && node.depth > previousDepth + 1) {
        file.message(
          `Heading jumps from h${previousDepth} to h${node.depth}`,
          node
        )
      }

      headings.push({depth: node.depth, title})
      previousDepth = node.depth
    })

    file.data.headings = headings
  }
}

The parser decides what a heading is. The plugin only evaluates heading nodes. This keeps syntax decisions separate from project policy.

Four cases worth keeping in CI

1. Normal structure

# Guide

## Install

Result: two heading nodes, no diagnostic, and two TOC entries.

2. Inline markup and duplicate labels

# Guide

## *API* Reference

## API Reference

The first heading does not contain one flat text node. Its children include an emphasis node, so a robust text extractor must walk descendants. Both visible labels normalize to the same slug; the second needs a stable suffix such as api-reference-1.

3. Valid Markdown that violates a project rule

# Guide

### Internals

Parsing and HTML rendering succeed, but the plugin reports:

Heading jumps from h1 to h3

This is a useful distinction. A parser error means the syntax cannot be interpreted under the configured grammar. A linter diagnostic means the syntax is valid but conflicts with a documentation policy.

4. A heading marker inside code

# Real heading

```md
## Not a heading
```

The top-level mdast nodes are heading and code. The TOC contains only Real heading. No special-case regex is needed because the parser has already resolved the context.

Keep plugins on the right side of the bridge

Here is the complete processing order:

const file = await unified()
  .use(remarkParse)
  .use(headingAudit)
  .use(remarkRehype)
  .use(addHeadingIds)
  .use(rehypeStringify)
  .process(markdown)

headingAudit expects mdast and therefore runs before remarkRehype. addHeadingIds writes HTML properties and therefore runs on hast after the bridge. Plugin order is part of the contract, not cosmetic configuration.

When checking the same samples with a Markdown-to-HTML converter, I compare structural output—headings, code blocks, links—not only whether the preview looks plausible. Two pages can look similar while exposing different trees to a TOC generator, sanitizer, or editor.

What an AST does not solve

An AST removes a lot of context guessing, but it does not make a pipeline automatically safe:

two plugins can mutate the same nodes;
generating a TOC before another plugin rewrites headings creates stale data;
slug behavior is not defined by CommonMark and differs across platforms;
raw HTML still needs an explicit trust and sanitization policy;
syntax-extension plugins can change what inputs the parser accepts.

The practical rule I use is to document every plugin's input tree, output tree, mutations, diagnostics, and ordering constraints. Then test both the tree shape and the serialized output.

CommonMark 0.31.2 supports the underlying separation: block structure is resolved before inline structure. Unified and remark make that structure available as mdast and provide a plugin pipeline around it. The experiment above only claims the pinned versions and inputs listed here; it is not proof that every Markdown dialect shares the same tree or slug rules.

Where would you draw the boundary between plugin freedom and syntax stability in a long-lived documentation pipeline?

Markdown Dialects Need a Capability Matrix, Not Just a Name

frank — Sat, 15 Aug 2026 03:41:50 +0000

Two tools can both claim to support “Markdown” and still disagree about tables, task lists, bare URLs, and even the HTML element used for strikethrough.

That is usually not a parser bug. It is a contract problem.

“Markdown support” often collapses three separate layers into one label:

a base syntax such as CommonMark;
extensions such as GitHub Flavored Markdown;
platform-specific configuration, sanitization, and post-processing.

I ran a small differential test to make those boundaries concrete.

Fixed environment

Tested on August 15, 2026:

commonmark 0.31.2
markdown-it 14.3.0, default preset, no third-party plugins
marked 18.0.7, default options

The comparison records generated HTML, not screenshots. That distinction matters: two results can look similar while producing different DOM structures for accessibility, export, or later transforms.

Minimal setup:

import * as commonmark from "commonmark";
import markdownit from "markdown-it";
import { marked } from "marked";

const source = "before ~~deleted~~ after";
const reader = new commonmark.Parser();
const writer = new commonmark.HtmlRenderer();

console.log(writer.render(reader.parse(source)));
console.log(markdownit().render(source));
console.log(marked.parse(source));

These are versioned defaults, not permanent labels for the libraries. Options and plugins can change the result.

1. A pipe table is not base CommonMark

Input:

| A | B |
|---|---|
| 1 | 2 |

Observed structure:

Parser	Result
commonmark 0.31.2	One paragraph; pipes remain text
markdown-it 14.3.0	A `table` with `thead` and `tbody`
marked 18.0.7	A `table` with `thead` and `tbody`

CommonMark 0.31.2 does not define pipe tables. GFM adds them as an extension.

When I need a controlled table input for a compatibility test, I use a Markdown table generator to avoid accidental delimiter mistakes, then verify the output with the actual target renderer. Generating valid source is not the same as proving platform compatibility.

2. A pipe inside a code span is a hard case

Input:

| A | B |
|---|---|
| `x|y` | a\|b |

With the tested markdown-it and Marked defaults, the unescaped pipe inside the code span still splits the row. One cell receives the opening backtick plus x; the next receives y plus the closing backtick, instead of one code span.

GFM’s table examples require escaping that pipe even inside a code span:

| A | B |
|---|---|
| `x\|y` | a\|b |

This is why splitting a table row with source.split('|') is not a parser.

3. A table-like shape can fail correctly

Input:

| A | B |
| 1 | 2 |

All three parsers emitted a paragraph. Without the delimiter row, none guessed that the author intended a table.

For editor diagnostics, checking for a real table node is safer than checking whether the source contains pipes.

4. Strikethrough has two kinds of differences

For:

before ~~deleted~~ after

commonmark keeps the tildes as text;
markdown-it emits <s>;
Marked emits <del>.

For the boundary case:

before ~not deleted~ after

only Marked recognized strikethrough in this test.

There are two contracts here: whether the syntax is recognized, and which HTML structure represents it. Screenshot tests can miss the second one.

5. Task lists may be plain text or form controls

Input:

- [x] shipped
- [ ] pending

commonmark and the markdown-it default produced normal list items containing [x] and [ ]. Marked produced disabled checkbox inputs and marked the first one as checked.

Whether a host then makes those boxes interactive is a UI and security decision, not something the Markdown syntax alone can decide.

6. Angle-bracket autolinks and bare URLs are different features

All three parsers linked this CommonMark form:

<https://example.com/a_(b)>

But for:

Visit https://example.com/a_(b).

only Marked linked the bare URL under the tested defaults. It also excluded the trailing period from the destination.

markdown-it can add this behavior with linkify; that option was deliberately left off so the matrix describes the default preset.

7. Raw HTML policy is not HTML safety

Input:

<xmp>**not emphasis**</xmp>

commonmark and Marked preserved the xmp element. markdown-it escaped the tags because raw HTML is disabled by default. All three still parsed the emphasis inside.

GFM has a tag-filter extension for names such as script, style, iframe, and xmp. That filter is not a complete HTML sanitizer. Marked’s own documentation also warns that its output is not sanitized.

A production pipeline therefore needs two explicit answers:

What HTML can the parser generate?
What HTML does the sanitizer allow?

The resulting capability matrix

For these exact versions and defaults:

Capability	commonmark	markdown-it	Marked
Angle-bracket autolink	Yes	Yes	Yes
Pipe tables	No	Yes	Yes
`~~x~~` strikethrough	No	Yes	Yes
`~x~` strikethrough	No	No	Yes
Task-list checkboxes	No	No	Yes
Bare-URL autolinking	No	No	Yes
Raw HTML enabled by default	Yes	No	Yes
Complete HTML sanitization	No	No	No

“No” is not automatically a defect. Disabling raw HTML can be a deliberate safety boundary. markdown-it can gain more syntax through plugins. The important point is that version and configuration belong in the contract.

A more useful declaration

Instead of only storing:

{ "flavor": "gfm" }

an application could expose something closer to:

{
  "base": "commonmark-0.31.2",
  "extensions": {
    "tables": true,
    "strikethrough": "single-or-double-tilde",
    "taskList": true,
    "extendedAutolink": true,
    "tagfilter": false
  },
  "rawHtml": "parse-then-sanitize"
}

That object is testable, versionable, and much harder to misunderstand than a dialect name.

The open question is whether Markdown ecosystems should standardize a machine-readable capability manifest—or whether every platform will continue documenting its dialect through examples and surprises.

Why One Newline Renders Differently Across Markdown Tools

frank — Fri, 14 Aug 2026 03:37:51 +0000

Press Enter once in a Markdown editor and the preview may keep the text in one paragraph. Move the same file to a chat app and every source line may become a visible line break.

That is usually not a parser bug. Three separate things are being confused:

the file's line ending (LF, CRLF, or CR),
the Markdown node (softbreak or hard line break), and
the browser's final layout.

I tested the cases below on August 14, 2026 with fixed versions:

commonmark.js 0.31.2
markdown-it 15.0.0
Marked 18.0.9
CommonMark specification 0.31.2

One Enter is a soft break by default

Input:

alpha
beta

All three parsers produced one paragraph in their default configuration:

<p>alpha
beta</p>

CommonMark calls the line ending a soft break. A renderer may emit it as a newline character or a space. Under the browser's usual white-space: normal, that whitespace is collapsed, so the two words normally appear on one visual line unless the container wraps them.

This distinction matters: a newline in generated HTML source is not necessarily a visible line break.

Two spaces and a backslash create hard breaks

CommonMark supports two forms:

alpha  
beta

alpha\
beta

Both produced a hard-break node and a <br> element in all three parsers:

<p>alpha<br>
beta</p>

I prefer the backslash form when a break is semantically important. It is visible during review, while trailing spaces are easy for an editor or formatter to delete.

LF, CRLF, and CR normalized to the same result

I rendered these inputs separately:

alpha\nbeta
alpha\r\nbeta
alpha\rbeta

All three parsers treated them as the same paragraph with a soft break by default. Converting a repository from CRLF to LF should therefore not change this Markdown meaning by itself.

Trailing-space cleanup is a different story. Removing the second space from a hard break changes the document structure.

The `breaks` option changes platform behavior

With breaks: true, markdown-it and Marked converted a normal paragraph newline to <br>:

Configuration	Result for `alpha\nbeta`
commonmark.js default	soft break
markdown-it default	soft break
markdown-it `breaks: true`	`<br>`
Marked default	soft break
Marked `breaks: true`	`<br>`

That option explains many "works here, breaks there" reports. A chat renderer may intentionally preserve author-entered lines, while a documentation system follows CommonMark's paragraph behavior.

Treat the option as part of the platform's Markdown dialect. Recording only "we use Markdown" is not enough for a reproducible publishing pipeline.

Edge cases expose the real boundary

One trailing space is still soft

alpha 
beta

It does not create a hard break. The parsers differed slightly in the exact HTML whitespace they retained, but not in the resulting Markdown structure.

Two spaces at the end of a block do not create `<br>`

alpha  

next

CommonMark says a hard break separates inline content inside a block. It cannot occur at the end of the paragraph. commonmark.js and markdown-it removed the trailing spaces. Marked 18.0.9 retained them in the HTML string, but none of the three emitted <br>.

The page looks the same in a browser, yet byte-for-byte HTML snapshots differ. Decide whether your tests protect semantic structure or exact serialization.

Code spans use different whitespace rules

`alpha
beta`

All three produced:

<p><code>alpha beta</code></p>

The newline is normalized inside the code span. A backslash or two trailing spaces there do not become a hard break.

A practical testing strategy

For a multi-output documentation pipeline, test three layers:

preserve the exact fixture, including visible line-ending and trailing-space notation;
compare the parsed structure, especially soft-break and hard-break nodes;
compare the target output and inspect it under the real CSS.

If the final target is fixed-layout output, I also check a representative file against the Markdown-to-PDF formatting workflow. That does not replace parser tests; it catches the separate class of layout differences introduced after parsing.

My working rule is simple: use blank lines for paragraphs, a visible backslash for intentional inline breaks, and automatic wrapping for prose. Then lock parser versions and options wherever the same source must render consistently.

Who should own line-break semantics in a shared document system: the author in the source, or the rendering environment for each destination?

Markdown vs PDF: Stop Choosing One File for Every Stage

frank — Thu, 13 Aug 2026 06:24:26 +0000

Markdown vs PDF: use one for the source and one for delivery

The practical answer to Markdown vs PDF is not that one format wins. Use Markdown while content is being written, reviewed, translated, or reused. Use PDF when the page arrangement has been checked and the recipient needs a stable delivery copy for reading, printing, signing, or archiving.

For an important document, keep both: the .md file is the editable source; the .pdf file is a dated or versioned output. After changing the source, export and inspect a new PDF. Do not edit the two files independently and assume they still contain the same revision.

This guide is for writers, developers, researchers, and small teams deciding which file to keep, share, or convert. It owns the format decision and source-to-output workflow. The separate format-preservation guide covers detailed fixes for headings, links, code, and tables after you have already chosen PDF.

The core difference is structured text versus page-oriented output

CommonMark defines Markdown as a plain-text format for writing structured documents. The source uses visible markers—such as # for a heading or - for a list—and a renderer decides how that structure looks. A different theme can change fonts, spacing, colors, and line wrapping without changing the source words.

The Library of Congress PDF format description describes PDF as representing formatted, page-oriented documents. A PDF can contain text, images, graphics, and structure, but its defining job is to carry a composed page rather than the author's lightweight source notation.

That difference leads to a useful decision table:

Need	Prefer Markdown	Prefer PDF
Revise sentences and headings repeatedly	Yes	No
Review exact pages and line breaks	No	Yes
Reuse the same content on a website or in another build	Yes	No
Send a print-ready handout	No	Yes
Inspect small text changes	Yes	Possible, but less direct
Preserve a final visual snapshot	No	Yes
Guarantee accessibility without checking	No	No

The last row matters. Plain text is not automatically an accessible published experience, and fixed pages are not automatically well tagged. Format choice is only the start of accessibility work.

Our same-document test shows what each format preserves

We tested the current MDFold Markdown-to-PDF converter with a controlled 635-byte source. It contained one heading, a three-row decision table, a link, a checklist, a block quotation, and the final marker FORMAT-END-008.

First, we changed SOURCE-REV-A to SOURCE-REV-B and expanded one checklist item. The source editor and live preview both showed the new revision; the old revision disappeared from the preview. This is the Markdown advantage in concrete form: the author can change the underlying structured text directly and review the rendered result before producing a delivery file.

Next, we used the real PDF action and inspected the exact downloaded file. It was a valid one-page PDF of 23,268 bytes. Text extraction found SOURCE-REV-B, the revised checklist wording, and FORMAT-END-008. These checkpoints show that the tested output came from the edited source and reached the end of the document. The output also included an MDFold-branded converter section after the fixture. That added section is visible in the screenshot and means the current export is not a source-only handoff.

This test does not prove that every Markdown feature or every PDF viewer behaves identically. It proves a narrower, useful boundary: for this fixture and current converter, editing happened in the source, while the generated PDF captured that accepted revision as a one-page output with extractable text and an added branded section. Inspect whether that extra material is acceptable before using the file as a formal deliverable.

Choose Markdown while meaning and structure can still change

Markdown is the stronger working format when people need to change the document rather than merely comment on its appearance.

Use it for:

Drafting and restructuring. Headings, lists, quotations, code, and links remain visible in the source instead of being placed on fixed pages.
Reviewing text changes. Line-based tools can compare two text revisions. The official git diff documentation describes the mechanisms Git provides for showing changes between content states. You do not need Git to edit Markdown, but plain text makes this kind of review practical.
Reusing content. The same structured source can feed a web page, documentation site, or PDF workflow. Each renderer still needs its own acceptance check.
Automating repeatable output. A team can keep the source, rendering rules, and assets together, then rebuild the delivery format after an approved change.
Working with content-focused tools. Search, replacement, translation, and code review operate on the actual words and markers rather than reconstructed page objects.

Markdown is not a finished visual specification. A .md file does not carry a universal font, paper size, margin system, page count, or print result. CommonMark also does not define every extension that popular editors offer. Tables, task lists, equations, diagrams, and raw HTML can vary between renderers. Name the intended renderer when those features matter.

Choose PDF after the pages are ready to review and share

PDF is the stronger delivery format when the recipient cares about the composed pages more than the source notation.

Use it for:

a report whose page breaks, margins, and typography have been approved;
a printable handout, invoice, form, or signed record;
a review copy where everyone should refer to the same page arrangement;
a final snapshot that should not silently change when a website theme changes;
a recipient who expects a standard document viewer rather than a Markdown renderer.

“Fixed output” does not mean “impossible to edit.” PDF software can add comments, fill fields, redact content, or modify page objects. The important distinction is that PDF is not the cleanest source for revising the original logical document. If you revise only the PDF and leave the Markdown untouched, the two versions diverge.

PDF also needs final-file checks. A generated document can have clipped tables, awkward page breaks, substituted fonts, dead links, or missing structure even when the preview looked acceptable. Open the downloaded file independently and inspect the pages you will actually send.

A reliable workflow keeps one source of truth

Use this five-step handoff:

Write and revise in Markdown. Keep linked assets in a stable location and use a revision label for consequential documents.
Preview with the intended renderer. Check headings, lists, tables, links, code, images, and any renderer-specific extensions.
Export a new PDF. Do not overwrite a previously approved copy until the new output passes review.
Inspect the saved PDF. Confirm page count, final marker, links, text selection, table width, image presence, page breaks, and print size.
Label the relationship. Record which source revision produced the PDF. Store the Markdown as the master and the PDF as an output unless policy requires a different recordkeeping model.

A simple naming pattern prevents ambiguity:

release-notes.md                 editable source
release-notes-r3.pdf             reviewed output from revision 3

If a recipient edits the PDF, treat those edits as feedback to reconcile into the Markdown source. Then export revision 4. This avoids two competing masters.

Mobile viewing exposes the fixed-page tradeoff

We repeated the product check at 390 × 844 pixels. The MDFold PDF preview stayed within the browser viewport without document-level horizontal overflow and showed the same one-page result.

the PDF preview remained contained without document-level horizontal overflow; detailed reading still benefits from zoom. Captured August 7, 2026.")

Containment is not the same as comfortable reading. A fixed page scaled down to a phone can make table cells and body text small. Markdown rendered as a responsive web page can reflow more naturally, but the result depends on the renderer and its styles. If phone reading is the primary use, publish a responsive version and offer PDF as a secondary download. If printing or page references are primary, retain the PDF and test zoom and navigation on a phone.

Accessibility and privacy require separate checks

Neither file extension is an accessibility certificate. Markdown can express headings, links, lists, image descriptions, and table structure, but the published renderer must carry that meaning into an accessible experience. PDF can include document structure, yet the final file still needs correct reading order, meaningful links, alternative text, language, contrast, and keyboard behavior. Adobe's Reading Order tool guidance explains how page reading order, headings, form labels, alternative text, and table tagging may need to be checked and repaired; a .pdf extension alone does not establish those qualities.

For privacy, inspect the complete workflow rather than inferring safety from a format. In this MDFold test, we observed no request to an application upload or conversion endpoint while editing and generating the controlled document. The page still loaded ordinary application assets. This observation covers the tested route and fixture on August 7, 2026; it is not a promise about unrelated links, remote images, analytics, extensions, or other tools.

If the Markdown contains remote images, opening a preview may contact those hosts. If the PDF contains sensitive material, its portability makes access control and secure delivery important. Use non-sensitive test content until the workflow is approved.

Know what this comparison cannot guarantee

The controlled test used installed Google Chrome, the current MDFold converter, one 635-byte English fixture, one template, one desktop viewport, and one phone viewport. It covered a heading, table, link, checklist, quotation, and plain text. The current output added an MDFold-branded converter section after the fixture, so it is not suitable when a source-only or unbranded PDF is required. The test did not cover authenticated assets, complex images, equations, diagrams, embedded fonts, forms, signatures, encryption, archival profiles, screen-reader testing, every Markdown dialect, or every PDF viewer.

Do not generalize the one-page result into universal layout fidelity. A longer or more complex document can expose different page breaks and conversion limits. Do not infer that extractable text guarantees correct semantic tagging or reading order. And do not discard the Markdown merely because the PDF passed: the source is what makes the next controlled revision straightforward.

Final decision checklist

Choose Markdown now if the document still needs substantive edits, structured reuse, translation, line-by-line review, or output in more than one format.

Choose PDF now if the content has been accepted and recipients need reviewed pages for printing, formal delivery, signatures, or stable page references.

Keep both when the document has a lifecycle: Markdown as the maintained source, PDF as a versioned delivery artifact. Before sharing, verify that the PDF was generated from the intended source revision and inspect the actual saved file.

When PDF is the right next step, open the Markdown-to-PDF converter, render the source, download the result, and complete the final-file checks above.

Sources and testing basis

CommonMark 0.31.2 specification — Markdown's plain-text structured-document model.
Library of Congress: PDF format family — PDF's formatted, page-oriented representation.
Adobe Acrobat: Reading Order tool for PDFs — checking and repairing reading order, tags, headings, form labels, images, and tables.
Git documentation: git diff — text-change comparison mechanisms.
MDFold product test, August 7, 2026 — controlled source edit, live preview, exact one-page PDF, extracted-text checkpoints, request observation, and 390-pixel phone review.

Tested PDF-to-Markdown on Searchable, Scanned, and Table-Heavy PDFs

frank — Sun, 09 Aug 2026 10:36:56 +0000

PDF-to-Markdown conversion is often described as if it were a file-format rewrite:

Read the PDF, emit Markdown, done.

That model breaks as soon as the PDF contains anything more complicated than a single column of selectable text.

I tested four controlled cases in a browser-based converter:

a two-page searchable PDF with headings, prose, and a table;
a searchable PDF containing a grouped header, a blank cell, and a multiline row label;
an image-only scanned PDF;
the same kind of scan after OCR added a searchable text layer.

The results were consistent, but not simple:

searchable text was extracted;
page boundaries remained identifiable;
table values survived, but the table grid did not;
the image-only scan returned no text;
OCR made the scan extractable, but silently changed identifiers;
none of the tested outputs became trustworthy Markdown without review.

The important lesson is not that PDF-to-Markdown “works” or “fails.” It is that text recovery, reading order, visual structure, and character accuracy are separate problems.

The test setup

I ran the current MDFold PDF-to-Markdown tool in installed Google Chrome on August 9, 2026.

The controlled fixtures contained no personal or customer data.

Fixture	What it tested	Verification method
Two-page incident memo	Searchable prose, page order, a small table, exact values	Four unique checkpoints and two page markers
Complex regional table	Grouped heading, multiline label, deliberate blank cell	Nineteen labels and values plus visual comparison
Image-only field note	Behavior when no text layer exists	Expected zero extracted words
OCR-processed scans	Recognition errors in clean and difficult pages	Exact source-to-output token comparison

I repeated the complex table and difficult OCR tests at desktop size and at a 390 × 844 phone viewport. The extracted text matched between the two viewports, and the page itself did not overflow horizontally.

This is a controlled product test, not a universal benchmark. Different PDF generators, fonts, tags, languages, rotations, and OCR engines can produce different results.

Case 1: searchable text survived, but the table did not

The first fixture was a two-page incident-review memo. It contained:

a title and section headings;
case ID IR-317;
a three-column table;
the values Table overflow, 12 pages, and 2026-08-28;
a second page with scope and ownership notes.

All four checkpoints appeared in the extracted output. Two page markers also appeared in the correct order.

But the table did not become a Markdown pipe table. Its words were present as a flat sequence.

The controlled values and page order survived, but the source table was not reconstructed as Markdown table syntax. Captured August 7 and reproduced August 9, 2026.

This is the first distinction a converter UI can hide:

Complete text is not the same thing as complete structure.

A quick scan of the output might suggest success because every important word is visible. A program expecting | Check | Result | Evidence |, however, would find no table at all.

The correct next step was manual reconstruction based on the original PDF, not guessing columns from spaces in the extracted string.

Why a searchable PDF still has no Markdown structure

A PDF page describes positioned content. Markdown describes document structure.

Those are different models.

Mozilla’s PDF.js API exposes page text through getTextContent(). It exposes a tagged structure tree separately through getStructTree(), and that structure tree may be null. Even when text extraction succeeds, the extractor may not receive a reliable semantic instruction saying:

This line is an H2.
These nine text fragments form a 3 × 3 table.
This empty region is an intentionally blank cell.

Instead, the page may contain text fragments positioned at coordinates. The converter has to infer reading order and relationships from those fragments, unless the PDF contains usable tags or another structural layer.

That inference is where apparently successful conversions become dangerous. The text may be present while its relationships are wrong.

Case 2: every table value survived, but the blank cell became unknowable

The complex table fixture contained:

a grouped Quarter totals heading;
Q1 and Q2 subcolumns;
a multiline North America row label;
a deliberately blank Q2 cell for Europe;
a note explaining that blank means “not reported,” not zero.

The source looked like this:

The visual grid carries meaning that is not contained in the individual words. Project-created fixture captured August 5, 2026.

The converter retained all 19 controlled text checkpoints, including:

North America
1,240
1,318
Reviewed
Europe
980
Missing Q2
Asia Pacific
1,105
1,207
Provisional

The output still was not a table:

All labels and values survived, but grouped headers, cell boundaries, and the location of the blank cell did not. Captured August 5 and reproduced August 9, 2026.

From the flat text alone, a cleanup script cannot safely answer:

whether Quarter totals spans two columns;
whether North America is one cell or two;
whether Europe’s missing number belongs to Q1 or Q2;
whether the blank represents zero, missing data, or a layout gap.

The footnote helps a human recover the intent, but it does not restore the missing cell coordinate.

This is why table verification must be cell-oriented. Counting extracted words or checking that every number appears is insufficient.

Case 3: the image-only scan produced zero text

The third fixture looked like a normal one-page field note in a PDF viewer. It had headings, numbered steps, identifiers, and measurements.

But the PDF contained only page pixels. There was no searchable text layer.

The result in the converter was an empty output: zero words.

The file opened successfully, but there was no text layer to extract. Captured August 5 and reproduced August 9, 2026.

This is not evidence that the document contains no writing. It means the extraction method found no machine-readable text.

OCR must happen before text-based Markdown conversion. OCRmyPDF describes this as adding an OCR text layer to a scanned PDF while retaining the original page image. Once that layer exists, a text extractor has something to read.

The boundary matters: the tested MDFold workflow extracts an existing text layer. It does not perform OCR on the image-only PDF.

Case 4: OCR recovered editable text and introduced silent errors

I processed controlled scans with local OCR, created searchable PDFs from the recognized text, and ran those PDFs through the same converter.

The clear fixture produced 63 editable words. The difficult fixture produced 64.

That sounds successful until the identifiers are compared exactly.

Source	OCR output	Error type
`ZX-41`	`X-41`	Character omitted
`B0O-518`	`B00-518`	Letter O changed to zero
`III.`	`I.`	Roman-numeral characters omitted
`OI-1058`	`0I-1058`	Letter O changed to zero

The OCR also correctly retained Café, $1,084.70, 2026-08-05, and a faint REVIEWED 08/05 stamp.

OCR made the scan searchable, but several high-risk tokens changed without an error message. Captured August 5 and reproduced August 9, 2026.

This is a more difficult failure than empty output.

Empty output is visible. A plausible but incorrect asset tag can flow into a repository, search index, or LLM prompt without being noticed.

The safe question is not “Did OCR produce text?” It is:

Which characters would cause harm if one symbol changed?

For many documents, that includes:

invoice totals;
dates and times;
serial numbers;
case IDs;
chemical units;
version strings;
negative signs and decimal separators;
names containing accents;
checkboxes, list numbers, and status labels.

A six-step verification workflow

The following process worked better than treating conversion as one action.

1. Classify the source before converting

Try selecting text in the PDF viewer.

If normal text can be selected, begin with text extraction.
If only a rectangular image region can be selected, plan an OCR step.
If the file mixes searchable and scanned pages, test both page types.
If tables, columns, figures, or equations carry meaning, plan visual comparison even when the text is searchable.

2. Add checkpoints to controlled tests

For a production pipeline, create a representative fixture with unique values at the beginning, middle, and end.

For example:

{
  "required": ["IR-317", "Table overflow", "12 pages", "2026-08-28"],
  "expectedPages": 2,
  "expectedTables": 1
}

Checking only the opening paragraph will not catch a missing final page or an interrupted reading order.

3. Separate text assertions from structure assertions

These are different tests:

for (const value of requiredValues) {
  assert(markdown.includes(value));
}

assert(markdown.includes("| Check | Result | Evidence |"));

In my searchable fixture, the first assertion passed and the second failed. Reporting a single “conversion passed” result would have hidden the table failure.

4. Rebuild only structure you can prove

Do not infer a table from word order alone when a blank cell or merged header exists.

Open the original PDF beside the Markdown and reconstruct:

heading levels;
list boundaries;
table rows and columns;
figure captions;
page-sensitive notes;
code indentation.

If the structure cannot be proven, keep the content as clearly labelled text or choose a layout-aware extraction tool.

5. Verify exact high-risk tokens

Create a small change ledger:

| Source | Extracted | Decision |
| :--- | :--- | :--- |
| B0O-518 | B00-518 | Correct from source image |
| OI-1058 | 0I-1058 | Correct from source image |

Do this before polishing headings. Semantic cleanup is wasted if the underlying identifiers are wrong.

6. Retain provenance

Keep the PDF until review is complete. Page markers in Markdown can help a reviewer return to the source page.

For high-risk workflows, store:

the source filename or document ID;
conversion date and tool version;
page boundaries;
OCR engine and language when applicable;
manual corrections;
unresolved uncertainties.

Markdown is easier to edit, which is useful. It also makes undocumented corrections easier to hide.

Choose the method by failure mode

Source condition	Reasonable first method	Do not assume
Simple searchable prose	Local text extraction	Heading levels are correct
Searchable PDF with columns	Layout-aware extraction and reading-order review	Text sequence matches visual order
Simple table	Text extraction plus manual cell verification	Spaces reconstruct a grid
Merged or nested table	Table-specific extraction or manual reconstruction	Blank cells and spans survive
Image-only scan	OCR first, then text extraction	Opening the PDF means text exists
Mixed searchable and scanned pages	Per-page detection with OCR fallback	One method handled every page
Equations or diagrams	Visual or specialist extraction	Extracted characters preserve meaning

The best converter is not always the one that produces the prettiest Markdown. It is the one whose failure modes you can detect and whose output you can verify for the document in front of you.

Privacy is a property of the whole pipeline

During the current MDFold tests, I observed no request carrying the PDF to an application conversion endpoint. The extraction happened in the browser.

That is a narrow observation, not a claim that the page is offline. Ordinary analytics requests were still present, and remote page assets can still require a network connection.

The OCR step was separate. I used a local operating-system OCR engine for the controlled scans. A hosted OCR service would have a different data path, retention policy, and risk profile.

Before processing sensitive material, inspect every stage:

PDF storage
  → OCR engine, if needed
  → text extractor
  → cleanup editor
  → repository, search index, or AI system

A local converter cannot make a later cloud OCR or AI upload local.

What this test does not prove

The fixtures were small, English-language documents created for testing. They covered two pages, a grouped table, an intentional blank cell, clean and difficult scans, and a phone-sized browser view.

They did not cover:

encrypted PDFs;
handwritten pages;
right-to-left scripts;
vertical writing;
mathematical notation;
multi-page tables with repeated headers;
tagged-PDF accessibility quality;
embedded files or interactive forms;
every OCR engine or PDF producer.

The test also does not prove that browser-local extraction is more accurate than a hosted or AI-based service. It proves the observed boundary of one current workflow.

If you want the deeper OCR cleanup procedure, I documented the exact-token method in a separate scanned-PDF-to-Markdown guide. For table-specific reconstruction, the PDF table recovery guide includes the controlled blank-cell example.

The acceptance test matters more than the conversion button

After these tests, I no longer describe PDF-to-Markdown as a single conversion.

It is a pipeline:

classify the PDF
  → recover text
  → verify reading order
  → reconstruct proven structure
  → compare exact values
  → retain provenance

For the searchable fixtures, text recovery succeeded while table recovery failed. For the image-only scan, text recovery could not begin until OCR created a layer. For the OCR fixtures, extraction succeeded while character verification still found silent errors.

That leads to a more useful definition of success:

The Markdown is acceptable only when every important fact and relationship can be traced back to the source PDF.

If you have built a PDF ingestion or RAG pipeline, which failure has caused more damage in practice: missing text, wrong reading order, flattened tables, or plausible OCR errors?

Focused FAQ

Can a searchable PDF still produce bad Markdown?

Yes. Searchable text proves that characters can be extracted; it does not prove correct heading levels, reading order, table cells, columns, or figure relationships. Compare the Markdown with the visible PDF structure.

Why does a scanned PDF return an empty result?

An image-only scan has page pixels but no machine-readable text layer. Run OCR first, then verify the recognized text before rebuilding Markdown structure.

Does OCR make the converted Markdown accurate?

Not automatically. In this controlled test, OCR made the scans editable but silently changed four identifiers or list tokens. Exact values still need source comparison.

How should I test a PDF-to-Markdown pipeline?

Use representative fixtures with unique checkpoints, assert text and structure separately, include one image-only and one table-heavy case, compare high-risk tokens exactly, and retain page-level provenance.

Sources

PDF.js PDFPageProxy API — separate text-content and structure-tree access.
OCRmyPDF documentation — adding searchable OCR text layers to scanned PDFs.
Tesseract: Improving OCR output quality — scan quality, segmentation, rotation, and recognition factors.
W3C PDF6 — table structure in tagged PDF documents.
GitHub Flavored Markdown table extension — the target pipe-table structure used in the asse

I Tested What a Browser-Based DOCX-to-Markdown Converter Actually Preserves

frank — Sun, 02 Aug 2026 05:59:41 +0000

“Preserves formatting” is one of the least useful promises a document converter can make.

Which formatting?

A Word document can contain semantic heading styles, font sizes, colors, numbered lists, floating objects, page headers, merged table cells, tracked changes, comments, and embedded images. Markdown has no direct representation for many of those things.

So I stopped asking whether DOCX-to-Markdown conversion “works” and tested a narrower question:

Which document structures survive, which become plain text, and which require a deliberate cleanup decision?

I created two controlled .docx files and converted them with MDFold’s browser-based Word to Markdown tool. The first represented a well-structured technical document. The second deliberately used features that are easy to misinterpret or awkward to express in Markdown.

The result was not “everything preserved” or “formatting lost.” It was much more useful:

real Word heading styles became Markdown headings;
bold, italic, links, bullets, and numbered lists survived;
a large bold paragraph stayed bold instead of becoming a heading;
table text survived, but table structure did not;
headers and footers were omitted;
an embedded raster image became a self-contained data URI;
page layout, fonts, and visual positioning were not reproduced.

That is the behavior this article explains and turns into a repeatable workflow.

DOCX and Markdown describe different things

DOCX can describe both document meaning and page presentation. Markdown mostly describes lightweight document structure.

Microsoft’s explanation of WordprocessingML shows that a Word document is composed of structured elements such as paragraphs and runs, with styles and properties attached to them. A paragraph can be a real Heading 1, or it can be an ordinary paragraph that merely looks like a title because someone made it 22-point bold text.

Those two paragraphs may look almost identical in Word. They are not equivalent to a converter.

The conversion pipeline I tested uses Mammoth to interpret DOCX structure as HTML, then converts that HTML into Markdown. Mammoth’s own documentation is unusually honest about the boundary: DOCX and HTML have a large structural mismatch, and conversion works best when the Word document uses styles for their intended meaning.

That gives us the first rule:

Prepare Word for structural conversion, not visual imitation.

If your real goal is to reproduce page margins, floating text boxes, fonts, headers, or exact pagination, Markdown is the wrong destination. Use PDF for a fixed result or keep the document in DOCX for collaborative editing.

The controlled test documents

I used generated files rather than a convenient hand-picked customer document. This made every expected result explicit and kept personal information out of the screenshots.

Test A: semantic structure

The representative file contained:

a real Word Heading 1;
a real Heading 2;
one sentence containing bold and italic runs;
an external hyperlink with descriptive text;
two bullet items;
two numbered steps;
a distinctive final checkpoint sentence.

The expected Markdown was:

# DOCX Conversion Field Test

## Release checklist

The **owner** reviews the _generated Markdown_ before publishing.

[Open the release runbook](https://example.com/runbook)

- Verify the title
- Verify the final paragraph

1. Convert the DOCX
2. Review the Markdown

FINAL CHECKPOINT: names, numbers, and links still match.

Test B: features that need judgment

The difficult file contained:

a visually large, bold title that was still a normal Word paragraph;
underlined text;
a two-column table;
an embedded PNG with alternative text;
a page header and footer;
a distinctive final checkpoint sentence.

This second file was designed to prevent a false sense of success. A converter can return non-empty Markdown while still losing the distinction that matters.

The well-structured document converted cleanly

The representative file preserved every checkpoint we asserted: heading levels, emphasis, link destination, both list types, and the final sentence.

The generated Markdown was not character-for-character identical to my hand-written expectation. The converter used extra spaces after list markers:

-   Verify the title
-   Verify the final paragraph

1.  Convert the DOCX
2.  Review the Markdown

That difference is harmless in common Markdown renderers. It is also a useful reminder not to confuse stylistic normalization with information loss.

The important checks were structural:

Word input	Observed Markdown	Result
Heading 1 style	`# DOCX Conversion Field Test`	Preserved
Heading 2 style	`## Release checklist`	Preserved
Bold run	`owner`	Preserved
Italic run	`_generated Markdown_`	Preserved
External hyperlink	Markdown link with the same destination	Preserved
Bullet list	Hyphen list	Preserved
Numbered list	Ordered Markdown list	Preserved
Final checkpoint	Same sentence in the result	Preserved

This is the kind of Word document that converts well: the author used real structure and kept the layout linear.

A title that only looks like a title stays ordinary text

The difficult document began with VISUAL-ONLY TITLE in a large, bold font. It looked like a heading in Word, but it was not assigned a heading style.

The observed Markdown was:

**VISUAL-ONLY TITLE**

This is correct behavior.

Automatically turning every short bold paragraph into a heading would create false structure in labels, warnings, captions, and emphasized sentences. A converter should prefer the document’s semantic style over a guess based on appearance.

Before conversion, open Word’s Navigation pane. If an intended section title does not appear in the document outline, apply the correct Heading style. Do not merely increase the font size.

Microsoft’s Open XML paragraph-style documentation supports this distinction: paragraph style is an explicit document property, separate from the runs that carry text and visual formatting.

Table text survived, but the table did not

This was the most important limitation in the test.

The DOCX table contained:

Owner	Status
Platform	Ready

The observed output was:

Owner

Status

Platform

Ready

The words survived. The row-and-column relationship did not.

The difficult file produced valid Markdown text, but valid is not the same as structurally complete. The table must be rebuilt and checked against the source.

This matters because a flattened table can silently change meaning. Four values in the correct order may still be unusable when the reader cannot tell which header belongs to which value.

For a small table, rebuild it manually and compare every cell with Word. For a large table, use a converter that explicitly supports GitHub Flavored Markdown tables, export the data separately as CSV, or use MDFold’s Markdown table generator after verifying the source values.

Do not trust the presence of all words as proof that the table survived.

Embedded images can survive in an inconvenient form

The test PNG became Markdown with a data: URL:

![A generated status marker](data:image/png;base64,...)

That makes the image self-contained: no separate file is required. It can also make the Markdown enormous, create unreadable diffs, and fail in publishing systems that block data URLs.

For a repository or documentation site, my preferred workflow is:

extract the image into an assets directory;
give it a stable, descriptive filename;
replace the data URI with a relative path;
preserve or improve the alt text;
preview the page in the actual publishing system.

For example:

![Green deployment status marker](./assets/deployment-status.png)

Mammoth documents inline data URIs as its default image behavior. Other converters may drop images, produce placeholders, or extract separate files. “Images supported” does not tell you which of those contracts a tool provides.

Headers and footers were omitted—and that may be desirable

The difficult DOCX used both:

CONFIDENTIAL DRAFT HEADER
PAGE FOOTER — INTERNAL REVIEW

Neither appeared in the Markdown.

For a web article, this is often desirable. Repeating page furniture is not part of the article’s logical reading flow. For a policy, contract, or controlled document, however, the missing classification, revision, or page note may be important.

The right question is not “did the header survive?” It is “does any information in the header need a new structural home?”

Move essential revision data into front matter, a document metadata block, or the opening section before conversion. Do not rely on page headers to become Markdown automatically.

Underline is ambiguous in Markdown

The underlined sentence returned as plain text.

Markdown has no universal underline syntax, and underlining is often confused with hyperlinks on the web. Mammoth’s default mapping intentionally ignores underline rather than inventing a meaning.

Decide what the underline meant in the source:

emphasis → use *italic* or **bold**;
a section label → use a real heading;
an insertion → write explicit editorial text;
a link → create a real hyperlink;
decoration → remove it.

Conversion cannot make that editorial decision safely.

Convert DOCX to Markdown in a reviewable workflow

Here is the process I would use for a real document.

1. Prepare the Word file

Save legacy .doc files as .docx first.
Apply Heading styles to actual section headings.
Accept or reject tracked changes.
Resolve comments that should not become published content.
Move essential header or footer information into the document body.
Simplify merged tables where possible.
Add useful alternative text to meaningful images.

2. Convert the DOCX

Open MDFold Word to Markdown, choose the .docx file, and wait for editable Markdown to appear. In the version tested on August 2, 2026, the conversion ran in the browser and did not require a document upload endpoint.

3. Check high-risk structures first

Do not begin by proofreading ordinary paragraphs. Check the features most likely to change meaning:

H1, H2, and H3 hierarchy;
first and final paragraphs;
ordered-list numbering;
link destinations;
every table;
image references and alt text;
footnotes, comments, headers, and footers;
names, dates, percentages, and negative numbers.

4. Repair the Markdown for its destination

Different destinations support different Markdown extensions. GitHub Flavored Markdown supports pipe tables, while strict CommonMark does not define the same table extension. A data URI or HTML fragment may work in one renderer and fail in another.

Preview the result where it will actually live: GitHub, a documentation generator, a CMS, an Obsidian vault, or another target system.

5. Download and verify the final `.md` file

Open the downloaded file rather than assuming the editor state and saved artifact are identical. Search for the final checkpoint, inspect the Markdown source, and render it once in the target environment.

The phone-sized workflow worked, but desktop is better for review

I repeated the representative conversion at a 390 × 844 viewport. The file opened, the expected final checkpoint was present, the result remained editable, and the page had no document-level horizontal overflow.

Phone conversion worked in the test. A larger screen is still the better place to compare tables, long links, and image data.

Mobile support is useful for an urgent check. It does not remove the need for a careful desktop review when the source contains complex structure.

When to use a different conversion path

Use this browser workflow for ordinary .docx documents that are mostly headings, paragraphs, emphasis, lists, and links.

Consider Pandoc or another document pipeline when you need:

batch conversion;
repeatable command-line builds;
separate media extraction;
custom style mapping;
footnotes, citations, or richer table handling;
a conversion process that can be reviewed and versioned as code.

Keep Word or export to PDF when the requirement is visual fidelity rather than portable structure.

Privacy is part of the conversion choice

For the tested MDFold workflow, the DOCX was read by browser-side code and the Markdown appeared locally in the editor. That is narrower than claiming that every website conversion is private.

If you use another converter, determine whether the file is uploaded, retained, used for model processing, or logged. This matters for contracts, unpublished research, internal runbooks, customer data, and regulated material.

Local conversion still depends on the device and browser you trust. Extensions, shared computers, managed devices, backups, and downloaded files remain part of the security boundary.

A practical definition of “preserved”

After this test, I would not describe DOCX-to-Markdown quality with one percentage or one checkbox.

I would ask five questions:

Is every important sentence present?
Is the heading hierarchy still correct?
Are relationships—lists, links, tables, notes—still explicit?
Are images usable in the target system?
Did any page-level information need to be moved into document structure?

A converter can automate syntax. It cannot decide what a visual convention meant to the author.

That is why DOCX-to-Markdown conversion should be treated as a small content migration, not a file-extension swap.

DOCX-to-Markdown FAQ

Can I convert an old `.doc` file directly?

Not with the MDFold workflow tested here. Save the legacy file as .docx in Word, LibreOffice, or another compatible editor, then convert the new copy. Keep the original until you have checked the Markdown.

Why did my Word heading become ordinary text?

The paragraph probably looked like a heading without using a Word Heading style. Apply the correct semantic style and convert again; do not rely on font size or bold text alone.

Should I keep embedded images as base64 data?

Only when the destination supports data URLs and a self-contained file matters more than readable source. For repositories and documentation sites, extracted image files with stable relative paths usually produce smaller, reviewable Markdown.

If you have a difficult Word document, try the browser-based converter and share one reproducible feature that did not map cleanly. A merged table, floating text box, tracked-change sequence, or image-heavy page is more useful feedback than “formatting broke.”

DEV Community: frank

Source Positions Are a Coordinate System, Not Just Line and Column

One emoji, three valid offsets

Positions should be half-open ranges

Four parsers, four position capabilities

Test failure syntax, not only valid syntax

A practical range contract

What I would regression-test

Snapshot Tests Lie About Markdown: Assert Structure Instead

The experiment

The differences were the useful result

Start with upstream conformance fixtures

Add one permanent fixture per real bug

Use snapshots as evidence, not as the oracle

Property tests need real invariants

Keep pathological tests separate

A practical test layout

Sources

Regex Cannot Sanitize Markdown: Put the Security Boundary After Parsing

A regex that passes the easy test

The fixed-version test

Parsing, sanitizing, and resource policy are separate

Do not modify sanitized HTML afterward

A practical review checklist

Primary sources

Invisible Unicode Characters Can Break Markdown Without Looking Different

Make the invisible input observable

What three fixed parser versions produced

A BOM is positional

Zero width does not mean zero semantics

Look-alike punctuation is different punctuation

Bidirectional controls require a different review model

A practical ingestion policy

Markdown Images Are Network Requests, Not Embedded Assets

Parsing and URL resolution are separate stages

Six cases that need different policies

Remote images are third-party dependencies

Relative paths need a publishing contract

Data URLs trade path stability for other costs

Validate the final request graph

Primary references

Your Markdown Parser Is Not Your XSS Boundary

The smallest useful pipeline

Five cases that expose the boundary

1. Normal content survives

2. Raw HTML is valid Markdown, not necessarily safe HTML

3. URL schemes need their own policy

4. Code examples must not be cleaned as attacks

5. XSS defenses extend beyond script

Put sanitization after the last unsafe transform

What I verify in a conversion workflow

Engineering checklist

Primary sources

Build a Markdown Heading Linter on the AST, Not With Regex

A small, testable plugin

Four cases worth keeping in CI

1. Normal structure

2. Inline markup and duplicate labels

3. Valid Markdown that violates a project rule

4. A heading marker inside code

Keep plugins on the right side of the bridge

What an AST does not solve

Markdown Dialects Need a Capability Matrix, Not Just a Name

Fixed environment

1. A pipe table is not base CommonMark

2. A pipe inside a code span is a hard case

3. A table-like shape can fail correctly

4. Strikethrough has two kinds of differences

5. Task lists may be plain text or form controls

6. Angle-bracket autolinks and bare URLs are different features

7. Raw HTML policy is not HTML safety

The resulting capability matrix

A more useful declaration

Why One Newline Renders Differently Across Markdown Tools

One Enter is a soft break by default

Two spaces and a backslash create hard breaks

LF, CRLF, and CR normalized to the same result

The breaks option changes platform behavior

Edge cases expose the real boundary

One trailing space is still soft

5. XSS defenses extend beyond `script`

The `breaks` option changes platform behavior

Two spaces at the end of a block do not create `<br>`

5. Download and verify the final `.md` file

Can I convert an old `.doc` file directly?