Iurii Rogulia

Posted on Jul 17 • Originally published at iurii.rogulia.fi

Adding Semantic Search to Internal Docs in 200 Lines

#typescript #node #postgres #ai

Someone on your team asks in chat: "how do I cancel a customer's order after it shipped?" Your wiki has the answer. It's a page titled Returns and reversals — post-fulfilment procedure. Nobody finds it, because they searched for "cancel order" and the document never uses the word "cancel." So they ping a colleague, who re-explains a process that was already written down two years ago.

That gap is what semantic search over internal docs closes. Keyword search matches strings; people ask questions in their own words. This article is a compact, working recipe — chunk, embed, query by meaning — built on Postgres and pgvector, in roughly two hundred lines. It is strictly about retrieval: finding the right document. Not generating an answer on top of it. That distinction is the whole point, and I'll come back to why it matters.

Why Keyword Search Runs Out

LIKE '%cancel%' and even Postgres full-text search both match tokens. They are excellent when the searcher and the author happen to use the same words. They fall apart the moment they don't:

"how do I cancel a shipped order" vs. a doc titled "cancellation policy" — the policy page might never contain the verb the user typed.
"the app is slow after login" vs. "performance degradation on session initialization" — zero shared content words, same meaning.
"expense reimbursement" vs. "travel claims" — synonyms a keyword index treats as unrelated.

You can paper over some of this with synonym dictionaries and stemming, and full-text search with a good tsvector configuration genuinely helps. But you are maintaining a hand-curated thesaurus forever, and it still misses paraphrases nobody anticipated. Semantic search attacks the problem from the other side: it compares the meaning of the query to the meaning of each document, not the surface words.

The mechanism is embeddings. An embedding model maps a piece of text to a vector — a list of numbers — such that texts with similar meaning land close together in that space. "Cancel a shipped order" and "post-fulfilment reversal procedure" end up near each other even with no shared words. Search then becomes: embed the query, find the nearest document vectors, return those documents.

The Pipeline in Three Steps

The entire system is three moves:

Chunk each document into pieces of a sensible size, with a little overlap.
Embed every chunk once, at ingest time, and store the vectors.
Query: embed the incoming question, find the top-k nearest chunks by cosine distance, return them with a link back to the source.

That's it. There's no model being asked to write prose, no agent loop, no streaming. Just "given this question, here are the five most relevant passages and where they came from."

Storage: Postgres + pgvector

If you already run Postgres, you do not need a separate vector database to start. pgvector is a Postgres extension that adds a vector column type and distance operators. It keeps your documents and their embeddings in the same database you already back up, query, and monitor — which is reason enough for most internal-tooling cases. (I keep the rest of my Postgres habits in PostgreSQL production patterns; the same indexing discipline applies here.)

The schema. I'm using OpenAI's text-embedding-3-small, which produces 1536-dimensional vectors, so the column is vector(1536):

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
  id          BIGSERIAL PRIMARY KEY,
  source      TEXT NOT NULL,        -- file path, page ID, or ticket URL
  title       TEXT NOT NULL,        -- for citation in results
  url         TEXT,                 -- link back to the original
  chunk_index INTEGER NOT NULL,     -- position within the source doc
  content     TEXT NOT NULL,        -- the raw chunk text, returned to the user
  embedding   VECTOR(1536) NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

The vector index is the part people get wrong. pgvector offers two index types. ivfflat partitions vectors into lists and is fast to build but needs you to set the list count and probes at query time. hnsw builds a graph, is slower to build and uses more memory, but gives better recall at a given speed and needs no list tuning. For an internal corpus that fits comfortably in memory, I default to hnsw. Crucially, the index operator class must match the distance operator you query with — cosine distance is <=>, so the index uses vector_cosine_ops:

CREATE INDEX ON doc_chunks
  USING hnsw (embedding vector_cosine_ops);

A note on honesty: for a corpus of a few hundred chunks, you don't even need an index — or pgvector — at all. A sequential scan over a few hundred vectors is milliseconds, and you could hold the whole thing in memory and compute cosine similarity in plain TypeScript. The Postgres path earns its keep when the corpus grows, when you want the data sitting next to the rest of your application state, and when concurrent queries matter.

Chunking

Chunking turns each document into the units you'll actually retrieve. The goal: each chunk should be a coherent, self-contained passage, big enough to carry meaning, small enough that its embedding represents one topic rather than a blur of several.

I split on structure first (headings, then paragraphs) and only fall back to a hard token cap when a section is too long. Overlap carries a little context across boundaries so a sentence split across two chunks still retrieves. This is illustrative — paragraph-aware, not exhaustive — but it's the shape I actually use:

// chunk.ts
import { encoding_for_model } from "tiktoken";

const MAX_TOKENS = 400; // target chunk size
const OVERLAP_TOKENS = 60; // carry-over between adjacent chunks

const enc = encoding_for_model("text-embedding-3-small");

function tokenLength(text: string): number {
  return enc.encode(text).length;
}

export interface Chunk {
  content: string;
  index: number;
}

export function chunkDocument(markdown: string): Chunk[] {
  // Split on blank lines (paragraphs / heading blocks) first.
  const blocks = markdown
    .split(/\n{2,}/)
    .map((b) => b.trim())
    .filter(Boolean);

  const chunks: string[] = [];
  let current: string[] = [];
  let currentTokens = 0;

  for (const block of blocks) {
    const blockTokens = tokenLength(block);

    // A single oversized block: hard-split it on token count.
    if (blockTokens > MAX_TOKENS) {
      if (current.length) {
        chunks.push(current.join("\n\n"));
        current = [];
        currentTokens = 0;
      }
      chunks.push(...splitOversized(block));
      continue;
    }

    if (currentTokens + blockTokens > MAX_TOKENS) {
      chunks.push(current.join("\n\n"));
      // Start the next chunk with a tail of the previous one for overlap.
      current = overlapTail(current, OVERLAP_TOKENS);
      currentTokens = tokenLength(current.join("\n\n"));
    }

    current.push(block);
    currentTokens += blockTokens;
  }

  if (current.length) chunks.push(current.join("\n\n"));

  return chunks.map((content, index) => ({ content, index }));
}

function splitOversized(block: string): string[] {
  const tokens = enc.encode(block);
  const out: string[] = [];
  for (let start = 0; start < tokens.length; start += MAX_TOKENS - OVERLAP_TOKENS) {
    const slice = tokens.slice(start, start + MAX_TOKENS);
    out.push(new TextDecoder().decode(enc.decode(slice)));
  }
  return out;
}

function overlapTail(blocks: string[], targetTokens: number): string[] {
  const tail: string[] = [];
  let count = 0;
  for (let i = blocks.length - 1; i >= 0 && count < targetTokens; i--) {
    tail.unshift(blocks[i]);
    count += tokenLength(blocks[i]);
  }
  return tail;
}

Embedding and Ingesting

Embed each chunk once, at ingest, and store the vector alongside its source metadata. Batch the calls — the embeddings endpoint accepts many inputs per request, which is both faster and cheaper than one call per chunk:

// ingest.ts
import OpenAI from "openai";
import { Pool } from "pg";
import { chunkDocument } from "./chunk";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

const EMBEDDING_MODEL = "text-embedding-3-small";

interface SourceDoc {
  source: string;
  title: string;
  url?: string;
  body: string;
}

async function embedBatch(texts: string[]): Promise<number[][]> {
  const res = await openai.embeddings.create({
    model: EMBEDDING_MODEL,
    input: texts,
  });
  return res.data.map((d) => d.embedding);
}

export async function ingest(doc: SourceDoc): Promise<void> {
  const chunks = chunkDocument(doc.body);
  const embeddings = await embedBatch(chunks.map((c) => c.content));

  const client = await pool.connect();
  try {
    await client.query("BEGIN");
    // Re-ingest cleanly: drop the old chunks for this source first.
    await client.query("DELETE FROM doc_chunks WHERE source = $1", [doc.source]);

    for (let i = 0; i < chunks.length; i++) {
      await client.query(
        `INSERT INTO doc_chunks (source, title, url, chunk_index, content, embedding)
         VALUES ($1, $2, $3, $4, $5, $6)`,
        [
          doc.source,
          doc.title,
          doc.url ?? null,
          chunks[i].index,
          chunks[i].content,
          // pgvector accepts a vector literal: '[0.1,0.2,...]'
          `[${embeddings[i].join(",")}]`,
        ]
      );
    }
    await client.query("COMMIT");
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

The DELETE-then-INSERT per source makes re-ingesting a single edited document idempotent: change a wiki page, re-run ingest for that one source, and its old chunks are replaced. For a large corpus where most documents are unchanged between runs, hash the content and skip embedding when the hash matches the stored one — embedding calls are cheap individually but add up across thousands of chunks. I cover that and the rest of the embedding-cost surface in reducing OpenAI API costs in production.

Query: Nearest Neighbours by Cosine Distance

At query time, embed the question with the same model used at ingest, then ask Postgres for the nearest chunks. The <=> operator is cosine distance — smaller is closer — so you ORDER BY embedding <=> $1 and LIMIT k:

// search.ts
import OpenAI from "openai";
import { Pool } from "pg";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

export interface SearchHit {
  title: string;
  url: string | null;
  source: string;
  content: string;
  distance: number; // 0 = identical direction, 2 = opposite
}

export async function search(query: string, k = 5): Promise<SearchHit[]> {
  const res = await openai.embeddings.create({
    model: "text-embedding-3-small", // must match the ingest model
    input: query,
  });
  const queryVector = `[${res.data[0].embedding.join(",")}]`;

  const { rows } = await pool.query<SearchHit>(
    `SELECT title, url, source, content,
            embedding <=> $1 AS distance
     FROM doc_chunks
     ORDER BY embedding <=> $1
     LIMIT $2`,
    [queryVector, k]
  );

  return rows;
}

That is the whole retrieval system. Chunking, ingest, and search together land in the low hundreds of lines — call it the order of two hundred — because the hard parts of a full RAG stack are deliberately absent. No answer generation, no reranker, no UI, no streaming. The output is a ranked list of real passages with a title, a url, and a distance you can show as a relevance hint. Wire it to a search box and a list of links, and people find documents by meaning.

Chunking Is Where the Quality Lives

If you take one thing from this article: the embedding model is rarely your bottleneck. Chunking is. A worse model with good chunks beats a great model with bad ones.

Chunks that are too large blur multiple topics into one vector, so the chunk matches everything weakly and nothing strongly. Chunks that are too small lose the context that made them meaningful — a sentence retrieved without its surrounding paragraph is often useless to whoever reads it. Split a procedure in the middle of a numbered list and the retrieved fragment answers half a question.

The levers that move relevance more than model choice:

Split on structure, not character count. Honour headings and paragraph boundaries. A chunk that maps to one section of a doc retrieves far better than one cut at an arbitrary 500-character mark.
Carry metadata. Store title and url with every chunk. You need them to cite the source and link back — the entire value proposition is "here's the document," not "here's an orphan paragraph."
Tune overlap. A small overlap (10–15% of chunk size) keeps boundary-straddling ideas retrievable. Too much overlap inflates storage and returns near-duplicate hits.

There is no universal chunk size. 400 tokens is a reasonable default for prose handbooks; dense reference material or short FAQ entries want different settings. You find yours by running real queries from your team against the index and reading what comes back.

slug="ai-integration"
text="Have the docs but can't find them by meaning? I build semantic search over internal wikis, handbooks, and ticket history — Postgres-native, retrieval-first, no hallucinated answers."
/>

Where Semantic Search Falls Short — Honestly

Pure vector search is a tool, not a search oracle. The places it disappoints are predictable, so plan for them.

It loses to keyword matching on exact terms. Error codes, function names, SKUs, ticket IDs, acronyms — embeddings generalize, which is exactly wrong when the user wants ERR_2043 and not "errors that feel similar." For any corpus where exact tokens matter, the answer is a hybrid: run vector search and keyword (Postgres full-text or BM25) in parallel and combine the rankings. Add a reranking step when the stakes justify the extra latency and cost. If your internal docs are full of identifiers, treat hybrid as the baseline, not an upgrade.

Chunking is fragile and has no settled answer. Too coarse and you retrieve noise; too fine and you retrieve context-free fragments. The right size depends on your content, and you only learn it by testing against your own queries. Expect to re-tune it after you see real usage.

Embeddings go stale, and re-embedding has a cost. Edit a document and its old chunks no longer match the new text — you must re-embed that source. Worse: if you switch embedding models, vectors from the old model and the new one live in incompatible spaces and cannot be compared. Changing models means re-embedding the entire corpus. Pin your model version and treat a model change as a migration, not a config tweak.

It returns similar, not correct. Nearest-neighbour search gives you the passages closest in meaning to the query. "Closest" is not "right." The top hit can be a confidently-worded but outdated policy; the embedding has no notion of which document is authoritative or current. You must validate ranking quality on real queries from your team and keep the source documents trustworthy — garbage in, confidently-ranked garbage out.

Sometimes you don't need it at all. A few dozen documents? Ctrl+F and Postgres full-text search are simpler, free, and good enough — adding an embedding pipeline is over-engineering. When exact matching matters more than meaning, keyword search is the right primary, not a fallback. Reach for semantic search when the corpus is large enough that browsing fails and paraphrase is the actual problem.

Every one of these is manageable. None of them is hidden if you go in expecting it.

Search Returns Documents. A Chatbot Writes Answers.

This is the line worth drawing clearly. Everything above retrieves — it hands back real passages and links, and it never invents anything, because there's no generation step to invent with. That's a feature: a search box that returns the actual returns-policy page cannot hallucinate a returns policy. For "nobody can find the doc," retrieval alone solves the problem.

If you want a bot that reads those passages and writes a natural-language answer on top of them — "To cancel a shipped order, create a reversal in the admin panel, then…" — that's the next layer: retrieval-augmented generation. It adds an LLM, a confidence threshold so it escalates instead of guessing, source citation, and a streaming UI, plus a new failure mode the search-only version doesn't have (a model can phrase a wrong answer fluently). I built that end to end for an e-commerce support desk — across 25 languages, resolving 70% of tickets without a human — in the RAG chatbot architecture write-up. The semantic search here is the retrieval half of that system, useful entirely on its own.

Most teams I talk to think they need the chatbot. They need the search. Build the retrieval layer first, watch what people actually ask, and only add generation if returning documents turns out not to be enough.

Takeaways

Keyword search matches words; people ask by meaning. Semantic search closes that gap by comparing the meaning of the query to the meaning of each chunk.
The pipeline is three steps: chunk with overlap, embed once at ingest, query by cosine distance (<=>) for the top-k nearest chunks. On Postgres + pgvector it's roughly two hundred lines.
Chunking, not the model, decides quality. Split on structure, carry title/url metadata for citation, tune overlap, and test against your own queries.
Go hybrid for exact terms. Error codes, SKUs, and acronyms need keyword/BM25 alongside vectors; pin your embedding model, because changing it means re-embedding everything.
Retrieval ≠ generation. Search returns documents and cannot hallucinate. A RAG chatbot writes an answer on top and can. Build search first.

If you're sitting on a wiki, a handbook, and years of tickets that your team can't search by meaning, that's exactly the kind of AI integration I do — retrieval-first, on the Postgres you already run, with the honest limits built in from the start rather than discovered later.