Source Positions Are a Coordinate System, Not Just Line and Column

A Markdown linter can detect the right problem and still highlight the wrong character.

The usual cause is not parsing. It is a coordinate-system mismatch: one component counts UTF-8 bytes, another counts Unicode code points, and JavaScript indexes UTF-16 code units.

I tested this with Node.js 25.3.0, unified 11.0.5 + remark-parse 11.0.0, commonmark.js 0.31.2, markdown-it 15.0.1 (commonmark preset), and Marked 18.0.11.

One emoji, three valid offsets

Consider this source:

# A😀B

The position immediately before B is:

Unit	Offset
JavaScript UTF-16 code units	5
Unicode code points	4
UTF-8 bytes	7

const source = "# A😀B";
const utf16 = source.indexOf("B");
const codePoints = [...source.slice(0, utf16)].length;
const bytes = Buffer.byteLength(source.slice(0, utf16), "utf8");

console.log({ utf16, codePoints, bytes });
// { utf16: 5, codePoints: 4, bytes: 7 }

None of those numbers is universally wrong. The bug appears when an API calls all of them offset.

Positions should be half-open ranges

The unist specification defines a position with start and end points. Lines and columns are one-based, offsets are zero-based, and end points to the first character after the source region. Its definition of a character is a UTF-16 code unit.

remark-parse produced this position for the A😀B text node:

{
  "start": { "line": 1, "column": 3, "offset": 2 },
  "end":   { "line": 1, "column": 7, "offset": 6 }
}

That makes source recovery unambiguous:

source.slice(node.position.start.offset, node.position.end.offset)

Treating end as inclusive introduces an off-by-one error.

Four parsers, four position capabilities

The same input produced materially different public metadata:

Parser	Block positions	Inline positions	Absolute offset
remark-parse 11.0.0	yes	yes	UTF-16
commonmark.js 0.31.2	yes	no	no
markdown-it 15.0.1	line ranges	no	no
Marked 18.0.11	no standard field	no	no

commonmark.js reported the heading as [[1,1],[1,6]], but its A😀B text node had no sourcepos. markdown-it reported [0,1] on the inline token, while every inline child had a null map. Marked exposed raw, not a unique source location.

This is a capability difference, not a rendering-quality ranking. Block ranges are enough for scroll synchronization. Character-accurate quick fixes need inline ranges and explicit offsets.

Do not reconstruct positions with indexOf(token.raw). Repeated text makes the result ambiguous. Accumulating raw.length also breaks when a parser normalizes newlines, ignores a BOM, or merges text nodes.

Test failure syntax, not only valid syntax

I used four fixtures:

const fixtures = {
  normal: "# Title\n\nA paragraph.",
  difficult: "# A😀B\n\nUse **e\u0301** and `code`.",
  failure: "# Broken [link](<oops\n\nTail",
  boundary: "\uFEFF# Zero\u200BWidth\r\n\r\nEnd",
};

The failed link never became a link node. remark kept the failed construct in one text node; commonmark.js split it into several text nodes; markdown-it still only identified the containing line. A diagnostic implementation cannot assume malformed syntax has the AST shape of successful syntax.

The boundary fixture exposed another divergence. With a leading BOM, remark still recognized the heading, while commonmark.js parsed the first line as a paragraph. The zero-width character remained part of the text, and CRLF consumed two UTF-16 code units.

When the first question is whether a real file is Markdown, plain text, or affected by encoding, this Markdown file structure guide is the contextual checklist I use. It does not imply that parsers normalize those boundaries consistently.

A practical range contract

For a JavaScript editor, I would make the unit part of the field name:

type SourcePoint = {
  offsetUtf16: number;
  line: number;
  columnUtf16: number;
};

type SourceRange = {
  start: SourcePoint;
  end: SourcePoint; // exclusive
  sourceVersion: string;
};

If a Rust or Go service returns byte offsets, call the field offsetUtf8Bytes and convert against the exact same source text. sourceVersion matters because every range becomes suspect after formatting or editing.

For occasional lookups, scan from the start to compute line and column. For many diagnostics, precompute every line-start offset and use binary search. Decide how CRLF is counted and encode that decision in tests.

What I would regression-test

• an emoji before the target;
• a decomposed character such as e\u0301;
• LF and CRLF;
• a leading BOM;
• a zero-width character;
• repeated identical text;
• malformed links and emphasis;
• generated AST nodes with no source range.

The invariant is stronger than a snapshot: slicing the original source with a node's half-open offsets should reproduce that node's source spelling whenever the node genuinely came from one continuous region.

Source positions are a protocol between parser, analyzer, editor, and formatter. Define the unit, range semantics, and source version before trusting the number.

After a formatter rewrites the document, would you invalidate every diagnostic and reparse, or maintain a source map from the previous version?