- Book: AI That Reads
- The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
A user asks your docs bot: "Can I get a refund on an annual plan after 30 days?" Retrieval works. The top hit is the sentence "Annual plans are refundable within 30 days of purchase." The model reads it and says no. The next sentence in the same section, the one that says prorated refunds apply after day 30, never made it into the prompt. It lived in a different chunk, that chunk ranked ninth, and your retriever sends the top five.
Now flip it. You chunk by whole sections instead, 800 tokens each. The refund section covers refunds, invoices, tax IDs and currency changes. Its embedding is an average of four topics, and it loses to a short FAQ entry that happens to say "refund" twice.
That is the chunk-size trade-off, and LangChain's docs for its ParentDocumentRetriever put it plainly: you want small documents "so that their embeddings can most accurately reflect their meaning", and long enough documents "that the context of each chunk is retained". The fix they describe is to search the small pieces and return the big ones. People call it parent-document retrieval or small-to-big. Below is the whole thing in TypeScript, no framework, plus a check you can run against your own docs.
The shape: children that point at parents
Two tables. Parents are sections of your docs, split on headings. Children are runs of one or two sentences cut out of a parent, and each child stores its parent's ID. You embed only the children. At query time you rank children, map each hit to its parent, drop duplicate parents, and fill a token budget with whole sections.
Search the sentence, return the page it came from.
This is the same idea frameworks ship under other names. LlamaIndex's AutoMergingRetriever builds a three-level hierarchy (2048, 512 and 128 tokens by default) and merges leaf hits into their parent once enough of a parent's leaves match. Writing it yourself takes about a page of code and means you can see every decision.
Setup, on Node 20 or later:
npm init -y && npm pkg set type=module
npm i openai js-tiktoken
npm i -D tsx typescript
The openai client reads OPENAI_API_KEY from the environment. js-tiktoken counts tokens, and tsx runs the .ts files directly.
Splitting: sections and sentences
Parents come from headings. Children come from Intl.Segmenter with sentence granularity, which ships with Node, so there is no regex guessing where sentences end.
// chunk.ts
import { getEncoding } from "js-tiktoken";
const enc = getEncoding("cl100k_base");
export const countTokens = (s: string) =>
enc.encode(s).length;
export type Parent = {
id: string;
title: "string;"
text: string;
tokens: number;
};
export type Child = {
id: string;
parentId: string;
text: string;
};
Parents carry their token count so the budget check later is a lookup, not a re-encode. The two splitters go in the same file:
// chunk.ts (continued)
const seg = new Intl.Segmenter("en", {
granularity: "sentence",
});
export function splitSections(
doc: string,
file: string,
): Parent[] {
return doc
.split(/^(?=#{1,3} )/m)
.map((t) => t.trim())
.filter((t) => t.length > 0)
.map((text, i) => ({
id: `${file}#${i}`,
title: "text.split(\"\n\")[0],"
text,
tokens: countTokens(text),
}));
}
export function splitChildren(
p: Parent,
size = 2,
): Child[] {
const sentences = [...seg.segment(p.text)]
.map((s) => s.segment.trim())
.filter((s) => s.length > 0);
const out: Child[] = [];
for (let i = 0; i < sentences.length; i += size) {
out.push({
id: `${p.id}/${out.length}`,
parentId: p.id,
text: sentences.slice(i, i + size).join(" "),
});
}
return out;
}
cl100k_base is the encoding OpenAI's token-counting cookbook lists for the text-embedding-3 models. It is only used here to measure section size for the budget, so a close count is enough.
Two sentences per child is only a starting point. One sentence matches tightest; three survives pronouns like "this plan" better. The eval at the end is how you pick.
Embedding and indexing the children
OpenAI's embeddings endpoint takes an array of inputs, up to 2,048 items and 300,000 tokens summed per request, according to the API reference. Batches of 512 sentence-sized children stay well under both limits.
// embed.ts
import OpenAI from "openai";
const openai = new OpenAI();
const MODEL = "text-embedding-3-small";
export async function embed(
texts: string[],
): Promise<number[][]> {
const out: number[][] = [];
for (let i = 0; i < texts.length; i += 512) {
const res = await openai.embeddings.create({
model: MODEL,
input: texts.slice(i, i + 512),
encoding_format: "float",
});
res.data.sort((a, b) => a.index - b.index);
out.push(...res.data.map((d) => d.embedding));
}
return out;
}
export function dot(a: number[], b: number[]) {
let s = 0;
for (let i = 0; i < a.length; i++) s += a[i] * b[i];
return s;
}
A plain dot product works as cosine similarity here because OpenAI's embeddings guide states its embeddings are normalized to length 1. If you swap in a model that does not normalize, divide by the norms.
The index keeps parents in a Map and children in an array, each child carrying parentId and its vector:
// index.ts
import { readFile, readdir } from "node:fs/promises";
import { join } from "node:path";
import {
splitSections, splitChildren,
type Parent, type Child,
} from "./chunk.ts";
import { embed, dot } from "./embed.ts";
export type Indexed = Child & { vec: number[] };
export type Index = {
parents: Map<string, Parent>;
children: Indexed[];
};
export async function buildIndex(dir: string) {
const parents = new Map<string, Parent>();
const kids: Child[] = [];
for (const f of await readdir(dir)) {
if (!f.endsWith(".md")) continue;
const doc = await readFile(join(dir, f), "utf8");
for (const p of splitSections(doc, f)) {
parents.set(p.id, p);
kids.push(...splitChildren(p));
}
}
const vecs = await embed(kids.map((c) => c.text));
const children = kids.map((c, i) => ({
...c,
vec: vecs[i],
}));
return { parents, children } as Index;
}
In production the same shape is two tables: sections(id, title, text, tokens) and chunks(id, section_id, text, embedding), with the vector column in something like pgvector. The retrieval logic below does not change.
Retrieval: rank children, return parents within a budget
// index.ts (continued)
export function rank(ix: Index, q: number[], k = 40) {
return ix.children
.map((c) => ({ c, score: dot(q, c.vec) }))
.sort((a, b) => b.score - a.score)
.slice(0, k)
.map((h) => h.c);
}
export function smallToBig(
ix: Index,
q: number[],
budget = 1500,
): Parent[] {
const picked: Parent[] = [];
const seen = new Set<string>();
let used = 0;
for (const c of rank(ix, q)) {
if (seen.has(c.parentId)) continue;
seen.add(c.parentId);
const p = ix.parents.get(c.parentId)!;
if (used + p.tokens > budget) continue;
picked.push(p);
used += p.tokens;
}
return picked;
}
A parent's rank is the rank of its best child, because the loop visits children in score order and keeps the first time each parent appears. The continue on an over-budget section matters: a 1,400-token section ranked second should not block a 200-token section ranked third from fitting.
The baseline to compare against fills the same budget with child text only:
// index.ts (continued)
export function childOnly(
ix: Index,
q: number[],
budget = 1500,
): Indexed[] {
const picked: Indexed[] = [];
let used = 0;
for (const c of rank(ix, q)) {
const t = Math.ceil(c.text.length / 4);
if (used + t > budget) break;
picked.push(c);
used += t;
}
return picked;
}
Many child hits, a few parents, one token budget.
Things to watch. A section bigger than the whole budget can never be returned, so split long sections at index time (by subheading, or every ~800 tokens). The segmenter breaks at every newline, so a heading becomes its own segment and hard-wrapped markdown splits mid-sentence; if your docs are wrapped, join single newlines inside a paragraph (text.replace(/(?<!\n)\n(?!\n)/g, " ")) before segmenting. And childOnly estimates tokens as length divided by four to stay cheap; it is the baseline, so being roughly fair is enough.
Check it on your own docs
Whether small-to-big beats plain sentence chunks depends on your docs. Short self-contained FAQ answers may not need it. Long reference pages with exceptions two sentences after the rule usually do. So measure it.
Write 20 to 30 questions in eval.json. For each one, record the section ID that answers it and a needle: the shortest exact phrase from the docs that a correct answer depends on.
[
{
"q": "Can I get a refund on an annual plan after 30 days?",
"parent": "billing.md#3",
"needle": "prorated refund for the unused months"
}
]
Then score both strategies at the same budget:
// eval.ts
import { readFile } from "node:fs/promises";
import { buildIndex, smallToBig, childOnly } from "./index.ts";
import { embed } from "./embed.ts";
type Case = { q: string; parent: string; needle: string };
const norm = (s: string) =>
s.replace(/\s+/g, " ").toLowerCase();
const ix = await buildIndex("./docs");
const cases: Case[] = JSON.parse(
await readFile("eval.json", "utf8"),
);
const s = { bigSec: 0, bigNeedle: 0, kidSec: 0, kidNeedle: 0 };
for (const c of cases) {
const [q] = await embed([c.q]);
const big = smallToBig(ix, q);
const kids = childOnly(ix, q);
const n = norm(c.needle);
s.bigSec += +big.some((p) => p.id === c.parent);
s.kidSec += +kids.some((k) => k.parentId === c.parent);
s.bigNeedle += +norm(big.map((p) => p.text).join(" "))
.includes(n);
s.kidNeedle += +norm(kids.map((k) => k.text).join(" "))
.includes(n);
}
const pct = (x: number) =>
`${((100 * x) / cases.length).toFixed(0)}%`;
console.log("small-to-big section", pct(s.bigSec));
console.log("small-to-big needle ", pct(s.bigNeedle));
console.log("child-only section ", pct(s.kidSec));
console.log("child-only needle ", pct(s.kidNeedle));
Run it with npx tsx eval.ts. The two numbers per strategy answer different questions. "Section" asks whether retrieval found the right place at all. "Needle" asks whether the text the answer depends on is actually in front of the model. The gap between child-only's section and needle scores is the refund story from the top: the right section was touched, but the sentence that mattered stayed behind. Expect that gap to widen as the budget shrinks.
Limits. Questions you write while looking at the docs reuse the docs' wording, which flatters both strategies; real queries from your logs are a better source. The needle check is exact substring matching, so pick phrases that do not contain markdown formatting. And a 1,500-token budget is arbitrary: rerun at 800 and 3,000, because the comparison can flip as the budget changes. I haven't run this against any particular corpus, so I can't tell you which way yours will go. That is the point of the script.
If this was useful
Chunking is where a lot of RAG answers go missing, and small-to-big is one of several patterns for getting the right text in front of the model. AI That Reads is the RAG book in the AI in TypeScript series: embeddings, chunking, vector search and grounding answers in your own docs, all in TypeScript. If this post was the kind of problem you are fighting, the book is the longer version of it.
The rest of the series, from your first LLM call to agents in production, is at xgabriel.com/ai-in-typescript.
Top comments (0)