- Book: AI That Reads
- The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You have a docs/ folder with 300 Markdown files and someone asks for "chat with our docs". The usual first sketch has a hosted vector database or a Postgres instance with an extension, plus an ingestion service to feed it. For a few hundred files, that is a lot of infrastructure to answer "where do we configure the retry budget?"
Node already ships a SQLite driver. Since v22.13.0 and v23.5.0 it can load SQLite extensions, and sqlite-vec is an extension that adds a vec0 virtual table for vector search. Put them together and the whole retrieval side of a RAG pipeline lives in one rag.db file next to your script. No separate vector service, and nothing to compile.
One TypeScript file covers the whole loop, from indexing a folder of Markdown to an answer that cites its source files. It holds up into the hundreds of thousands of vectors, the practical ceiling sqlite-vec's author gives for its brute-force search.
What you need, by version
node:sqlite appeared in v22.5.0 behind --experimental-sqlite, lost the flag in v23.4.0 and v22.13.0, and became a release candidate (stability 1.2) in v25.7.0. The loadExtension() method and the allowExtension constructor option are what make sqlite-vec possible: loadExtension() only works if the database was constructed with allowExtension: true.
One naming change to know about: in v26.11.0 the class DatabaseSync was renamed to Database, and DatabaseSync stays as a documentation-only deprecated alias (DEP0210). The script below imports DatabaseSync so it runs on Node 24 and 26 alike. On 26.11+ you can switch the import to Database with no other change.
The script is a .ts file run directly by node. Type stripping is on by default since v23.6.0 and v22.18.0 and is marked stable since v25.2.0 and v24.12.0. So the floor is Node 22.18. I'd use 24, which is Active LTS and scheduled to move to maintenance on October 20, 2026, or 26, which becomes LTS on October 28, 2026, per the Node release schedule.
sqlite-vec's current stable release on npm is 0.1.9. It ships prebuilt binaries as optional dependencies for macOS (x64, arm64), Linux (x64, arm64) and Windows x64, so npm install downloads a .dylib, .so or .dll instead of compiling anything. For embeddings and the answer this uses the OpenAI Node SDK with text-embedding-3-small (1,536 dimensions by default, per the embeddings guide) and gpt-6-luna, which OpenAI calls its most efficient current model. Swap in any provider; the SQLite side does not care where the floats come from.
mkdir docs-rag && cd docs-rag
npm init -y
npm pkg set type=module
npm install sqlite-vec openai
export OPENAI_API_KEY=...
type=module matters because the script uses top-level await.
Chunks, vectors and the query all live in one file next to the script.
Open the database and create the tables
// rag.ts
import { DatabaseSync } from "node:sqlite";
import { readdir, readFile } from "node:fs/promises";
import { join } from "node:path";
import * as sqliteVec from "sqlite-vec";
import OpenAI from "openai";
const EMBED_MODEL = "text-embedding-3-small";
const DIMS = 1536;
const CHAT_MODEL = "gpt-6-luna";
const MAX_CHARS = 2000;
const openai = new OpenAI(); // reads OPENAI_API_KEY
const db = new DatabaseSync("rag.db", { allowExtension: true });
sqliteVec.load(db);
db.enableLoadExtension(false);
db.exec(`
CREATE TABLE IF NOT EXISTS chunks (
id INTEGER PRIMARY KEY,
path TEXT NOT NULL,
heading TEXT NOT NULL,
body TEXT NOT NULL
);
CREATE VIRTUAL TABLE IF NOT EXISTS vec_chunks USING vec0(
embedding float[${DIMS}] distance_metric=cosine
);
`);
allowExtension: true is off by default, and the Node docs note that if you leave it off you cannot turn extension loading on later. sqliteVec.load(db) resolves the platform binary and calls db.loadExtension() on it. Calling enableLoadExtension(false) straight after turns extension loading off again for this connection, so SQL that runs later cannot call load_extension() unless your code switches it back on.
Two tables, joined by id. chunks is a normal table with the text and where it came from. vec_chunks is the vec0 virtual table holding one 1,536-float vector per chunk, with rowid equal to chunks.id. distance_metric=cosine switches KNN from the default L2 distance to cosine, as the KNN docs describe. vec0 also supports +aux columns if you would rather keep the text inside the virtual table; a plain table is easier to inspect with any SQLite tool.
Chunk the Markdown by heading
type Chunk = { path: string; heading: string; body: string };
function chunkMarkdown(path: string, text: string): Chunk[] {
const chunks: Chunk[] = [];
let heading = "(top)";
let lines: string[] = [];
let inFence = false;
const flush = () => {
let cur = "";
for (const para of lines.join("\n").split(/\n\s*\n/)) {
if (cur && cur.length + para.length > MAX_CHARS) {
chunks.push({ path, heading, body: cur.trim() });
cur = "";
}
cur += para + "\n\n";
}
if (cur.trim()) chunks.push({ path, heading, body: cur.trim() });
lines = [];
};
for (const line of text.split("\n")) {
if (line.startsWith("`".repeat(3))) inFence = !inFence;
if (!inFence && /^#{1,3} /.test(line)) {
flush();
heading = line.replace(/^#+ /, "").trim();
}
lines.push(line);
}
flush();
return chunks;
}
Each chunk is one section under an H1 to H3 heading. Long sections are split on blank lines into pieces of about 2,000 characters, so a paragraph is never cut in half. The inFence flag stops a # comment inside a bash code block from being read as a heading. A single paragraph longer than MAX_CHARS stays whole; text-embedding-3-small accepts up to 8,192 tokens per input (embeddings guide), so that is fine for normal docs.
Finding the files is one recursive readdir:
async function markdownFiles(dir: string): Promise<string[]> {
const entries = await readdir(dir, { recursive: true });
return entries
.filter((p) => p.endsWith(".md"))
.map((p) => join(dir, p));
}
Embed and store
async function embed(texts: string[]): Promise<Float32Array[]> {
const res = await openai.embeddings.create({
model: EMBED_MODEL,
input: texts,
});
return res.data
.sort((a, b) => a.index - b.index)
.map((d) => new Float32Array(d.embedding));
}
const toBlob = (v: Float32Array) =>
new Uint8Array(v.buffer, v.byteOffset, v.byteLength);
const embedText = (c: Chunk) =>
`${c.path} > ${c.heading}\n\n${c.body}`;
Prepending the path and heading to the embedded text (embedText) costs a few tokens and gives the vector some context when many sections share a generic heading like "Configuration".
async function index(dir: string) {
const chunks: Chunk[] = [];
for (const path of await markdownFiles(dir)) {
const text = await readFile(path, "utf8");
chunks.push(...chunkMarkdown(path, text));
}
const vectors: Float32Array[] = [];
for (let i = 0; i < chunks.length; i += 64) {
const batch = chunks.slice(i, i + 64).map(embedText);
vectors.push(...(await embed(batch)));
}
const insChunk = db.prepare(
"INSERT INTO chunks (path, heading, body) VALUES (?, ?, ?)",
);
const insVec = db.prepare(
"INSERT INTO vec_chunks (rowid, embedding) VALUES (?, ?)",
);
db.exec("BEGIN");
try {
db.exec("DELETE FROM chunks; DELETE FROM vec_chunks;");
chunks.forEach((c, i) => {
const r = insChunk.run(c.path, c.heading, c.body);
insVec.run(BigInt(r.lastInsertRowid), toBlob(vectors[i]));
});
db.exec("COMMIT");
} catch (err) {
db.exec("ROLLBACK");
throw err;
}
console.log(`indexed ${chunks.length} chunks`);
}
Three details here come straight from sqlite-vec's own node:sqlite example. Vectors go in as raw bytes: a Float32Array wrapped in a Uint8Array, because node:sqlite binds typed arrays as BLOBs and vec0 accepts a vector as a BLOB of raw float32 values. The rowid goes in as a BigInt, which is how the official example binds it. And there is no .transaction() helper in node:sqlite, so it is BEGIN, COMMIT and ROLLBACK by hand.
All network calls happen before the transaction opens. If the embeddings API fails halfway through, the old index is untouched. The trade-off is that every index run re-embeds every chunk. For a few hundred files that is a small bill; past that, store a hash per chunk and only embed what changed.
Query: KNN, then a cited answer
type Hit = Chunk & { distance: number };
async function ask(question: string, k = 5) {
const [q] = await embed([question]);
const hits = db.prepare(`
WITH knn AS (
SELECT rowid, distance FROM vec_chunks
WHERE embedding MATCH ? AND k = ?
)
SELECT c.path, c.heading, c.body, knn.distance
FROM knn JOIN chunks c ON c.id = knn.rowid
ORDER BY knn.distance
`).all(toBlob(q), BigInt(k)) as Hit[];
const sources = hits
.map((h, i) => `[${i + 1}] ${h.path} > ${h.heading}\n${h.body}`)
.join("\n\n---\n\n");
const res = await openai.responses.create({
model: CHAT_MODEL,
instructions:
"Answer only from the numbered sources. Cite them like " +
"[2]. If the sources do not answer it, say so plainly.",
input: `Sources:\n\n${sources}\n\nQuestion: ${question}`,
});
console.log(res.output_text);
hits.forEach((h, i) =>
console.log(`[${i + 1}] ${h.path} (${h.distance.toFixed(3)})`),
);
}
The last lines of rag.ts pick the command:
const [cmd, arg] = process.argv.slice(2);
if (cmd === "index") await index(arg ?? "./docs");
else if (cmd === "ask" && arg) await ask(arg);
else console.log('usage: node rag.ts index <dir> | ask "<q>"');
Run it:
node rag.ts index ./docs
node rag.ts ask "Where do we configure the retry budget?"
The WHERE embedding MATCH ? AND k = ? pair is the vec0 KNN syntax. sqlite-vec also accepts LIMIT on SQLite 3.41+, but k = ? works everywhere. The KNN runs in a CTE and joins back to chunks afterwards, so the virtual table only has to answer "which rowids are closest".
The model sees numbered sources with their file paths and is told to cite them, and the script prints the paths and distances after the answer. That printout is your debugging tool. When an answer is wrong, look at the five paths first: if the right section is not among them, no prompt change will fix it. One more caution: your docs are now model input. If anyone outside your team can edit them, treat retrieved text as untrusted.
vec0 compares the query against every stored vector, so cost grows with the corpus.
When one file stops being enough
sqlite-vec's stable release does exact, brute-force search. At the v0.1.0 launch its author wrote that sqlite-vec "is currently focused on really fast brute-force vector search" and that "it just didn't make sense to include a complex ANN solution in this initial version." Every query compares against every vector. In that post's benchmarks, 100,000 float vectors at 1,536 dimensions took about 105 ms per query, and at 1 million vectors "none of the float vectors at any dimension pass the 100ms smoke test." The author's own estimate of the practical ceiling for latency-sensitive apps was "probably in the 100's of thousands depending on your dimensions/quantization techniques." Those are 2024 numbers, so read them as a rough order of magnitude and measure on your own hardware.
For a docs folder that is a long runway. 300 files at a few chunks each is a few thousand vectors, far below where the scan starts to hurt. If you get closer to the limit, you have three cheap levers before migrating:
- Ask for fewer dimensions. The
-3embedding models accept adimensionsparameter, and the same benchmark had all dimensions from 1,024 down under 75 ms at 100,000 vectors. ChangeDIMSto match, then re-create the table and re-index. - Use binary quantization. The post reports a 32x storage cut and roughly 10x faster queries for "possibly only a 5-10% loss of quality."
- Partition. vec0 partition keys restrict a KNN query to one tenant or product, with the docs' warning to keep hundreds of vectors per key.
The ANN indexes are coming: the 0.1.10 alphas add rescore and DiskANN indexes, but they are pre-releases, so I wouldn't put them under production traffic yet.
Graduate to pgvector when SQLite itself becomes the limit. That means several app servers writing to the index at once (SQLite allows one writer at a time), filters joined against tables that already live in Postgres, or millions of vectors where you want an HNSW or IVFFlat index and are fine with approximate results. pgvector does exact search by default too; the index is what buys you speed at that scale.
Until then, a single rag.db file is easy to copy, check in as a test fixture, ship inside a container and delete when you are done.
If this was useful
This post is the retrieval loop with nothing around it. AI That Reads is the book I wrote for what comes next in TypeScript: RAG over your own docs, embeddings, vector search, and grounding answers so the model sticks to what your sources say. If you just built rag.ts and want the full picture behind each of those steps, that is the book.
It is book 2 of AI in TypeScript, five books from your first LLM call to agents in production: the whole series is here.
Top comments (0)