- Book: AI That Reads
- The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Markdown is the most common thing teams put into a RAG system —
docs sites, wikis, READMEs, exported Notion pages — and it is the
format most badly served by generic chunkers, because markdown
carries structure that plain-text splitters cannot see.
Five strategies, in TypeScript, each with the thing it breaks. The
last one is not the winner by default.
The document under test
## Rate limits
Each API key is limited per minute. Exceeding the limit
returns 429 with a `Retry-After` header.
| Plan | Requests/min |
|-------|--------------|
| Free | 60 |
| Pro | 600 |
ts
const res = await fetch(url);
if (res.status === 429) {
await sleep(Number(res.headers.get("retry-after")) * 1000);
}
See [authentication](./auth.md) for key rotation.
A heading, prose, a table, a fenced code block, a relative link. Each
strategy below meets exactly this.
1. Fixed-size
export function fixedSize(text: string, size = 1000, overlap = 200) {
const out: string[] = [];
for (let i = 0; i < text.length; i += size - overlap) {
out.push(text.slice(i, i + size));
}
return out;
}
Predictable, trivially fast, and structurally blind. It cuts at
character 1000 whether that lands mid-word, mid-table row, or
mid-code-fence.
What it breaks: a split inside the fence produces one chunk with
an opening
``and no close, and another with a close and no open.sleep(...)`.
Both are syntactically broken markdown, and the second chunk now
looks to a model like prose that begins with
It is a reasonable baseline and a bad default for markdown.
2. Sentence-aware
Split on sentence boundaries, pack sentences up to a budget.
ts
export function bySentence(text: string, budget = 800) {
const seg = new Intl.Segmenter("en", { granularity: "sentence" });
const sentences = [...seg.segment(text)].map((s) => s.segment);
const out: string[] = [];
let cur = "";
for (const s of sentences) {
if (cur.length + s.length > budget && cur) {
out.push(cur.trim());
cur = "";
}
cur += s;
}
if (cur.trim()) out.push(cur.trim());
return out;
}
Intl.Segmenter is built into Node — no dependency, and it handles
abbreviations and non-English text far better than a regex on .
does.
What it breaks: it has no idea that a table row is not a
sentence. The table above has no sentence-ending punctuation at all,
so it either merges into a neighbouring chunk or forms one
run-on blob. Code fences are worse: if (res.status === 429) { ends
with a brace, and the segmenter treats the whole block as one
enormous sentence or shreds it on the periods in res.headers.get.
3. Heading-aware
Use the markdown AST. This is where TypeScript gets a real advantage,
because unified/remark gives you a typed tree with position
offsets.
ts
import { unified } from "unified";
import remarkParse from "remark-parse";
import type { Root, RootContent } from "mdast";
export function byHeading(md: string, maxLen = 1200) {
const tree = unified().use(remarkParse).parse(md) as Root;
const sections: { heading: string[]; nodes: RootContent[] }[] = [];
const stack: string[] = [];
for (const node of tree.children) {
if (node.type === "heading") {
stack.length = node.depth - 1;
stack[node.depth - 1] = toText(node);
sections.push({ heading: [...stack].filter(Boolean), nodes: [] });
continue;
}
if (!sections.length) {
sections.push({ heading: [], nodes: [] });
}
sections.at(-1)!.nodes.push(node);
}
return sections.flatMap((s) =>
packNodes(s.nodes, maxLen).map((body) => ({
text: `${s.heading.join(" > ")}\n\n${body}`,
heading: s.heading,
})),
);
}
Two things this buys. Chunks never straddle a heading, so a chunk is
always about one topic. And every chunk carries its heading path —
Rate limits, or API > Rate limits for nested ones — which is free
context that materially improves retrieval, because the body often
never repeats the words in the heading.
packNodes slices the node list on block boundaries and serialises
each node back to markdown, so a fence is either wholly in or wholly
out.
What it breaks: nothing structurally, but it distributes badly. A
document with one ## and forty paragraphs yields one huge section
that packNodes then has to split arbitrarily anyway. And a
reference page that is one giant table is a single node — you cannot
split it on block boundaries without splitting the table.
4. Recursive with overlap
Try progressively finer separators until the piece fits.
ts
const SEPARATORS = ["\n## ", "\n### ", "\n\n", "\n", " "];
export function recursive(
text: string,
max = 1000,
seps = SEPARATORS,
): string[] {
if (text.length <= max) return [text];
const [sep, ...rest] = seps;
if (!sep) return fixedSize(text, max, 0);
const parts = text.split(sep);
const out: string[] = [];
let cur = "";
for (const p of parts) {
const candidate = cur ? cur + sep + p : p;
if (candidate.length > max) {
if (cur) out.push(cur);
cur = p.length > max ? recursive(p, max, rest).join(sep) : p;
} else {
cur = candidate;
}
}
if (cur) out.push(cur);
return out;
}
This is the standard workhorse and it is decent. Prefer coarse
boundaries, fall back to fine ones only where needed.
What it breaks: the separator list has no notion of "inside a
fence." A long code block gets recursed down to "\n" and split
between lines, so if (res.status === 429) { ends one chunk and the
body starts the next. It also does not know a table from prose —
"\n" splits rows away from the header, and a row without its header
is a sequence of values with no labels.
The fix is to protect atomic regions before splitting: extract fences
and tables into placeholders, recurse over what remains, then restore.
That is a genuinely useful twenty lines and almost nobody writes it.
5. Semantic
Embed each sentence, walk the document, cut where consecutive
similarity drops below a threshold.
ts
export async function semantic(sents: string[], threshold = 0.82) {
const vecs = await embedAll(sents);
const out: string[] = [];
let cur = [sents[0]];
for (let i = 1; i < sents.length; i++) {
if (cosine(vecs[i - 1], vecs[i]) < threshold) {
out.push(cur.join(" "));
cur = [];
}
cur.push(sents[i]);
}
out.push(cur.join(" "));
return out;
}
Appealing in principle: boundaries land where the topic changes
rather than where a character count expires.
What it breaks: it costs an embedding call per sentence at ingest,
which for a large corpus is a real bill for a step you run before
you have any retrieval quality to justify it. The threshold is a
magic number that does not transfer between corpora. And it is
actively wrong on technical markdown — a code block has low semantic
similarity to the prose explaining it, so the splitter cuts exactly
between the explanation and the example, separating the two things
that most belong together.
What to actually do
Start with heading-aware, protect fences and tables as atomic units,
fall back to recursive within an over-long section, and attach the
heading path to every chunk.
ts
export function chunkMarkdown(md: string, max = 1200) {
return byHeading(md, max).flatMap((section) =>
section.text.length <= max
? [section]
: splitProtectingAtomics(section, max),
);
}
Then store the heading path as a field, not just as a prefix in the
text — it is useful for filtering and for rendering the citation.
Reach for semantic chunking only after you have measured that
boundaries are your retrieval problem. Usually they are not; usually
it is ranking, and that is a cheaper thing to fix.
The test worth writing
Assert the invariants rather than the output.
ts
it("never splits a fenced block", () => {
for (const c of chunkMarkdown(doc)) {
const fences = (c.text.match(/
```/g) ?? []).length;
expect(fences % 2).toBe(0);
}
});
Fence parity, table headers present wherever a row is, no chunk under
a floor length. Those three catch most of what goes wrong, and they
keep catching it when someone tunes max six months from now.
If this was useful
AI That Reads works through
the ingest side properly — chunking strategies against real document
formats, what metadata to carry, and how chunk design shows up later
as a retrieval problem.
The full series is at
xgabriel.com/ai-in-typescript.
Top comments (0)