How paragraph-aware chunking, embeddings, and PostgreSQL pgvector can make AI retrieval more efficient.
When you build an AI application that needs to work with large documents, one problem shows up quickly: how do you give an LLM the information it needs without sending the entire document with every request?
Sending more text isn't always the answer. It can increase token usage, introduce irrelevant context, and make retrieval harder to manage.
I explored a different approach: a vector search engine built with Node.js, TypeScript, and Supabase.
The idea is simple: turn document content into embeddings, store those vectors, and retrieve the passages most relevant to a user's query.
The architecture
The retrieval pipeline follows this flow:
- Ingest the source documents.
- Split the content into meaningful paragraph-based chunks.
- Preserve headings and contextual metadata.
- Generate 768-dimensional embeddings.
- Store the chunks and embeddings in Supabase with pgvector.
- Embed the incoming query and rank relevant chunks using cosine similarity.
- Pass the selected context to the LLM.
This approach forms the retrieval layer of a retrieval-augmented generation (RAG) system.
1. Chunking documents without losing context
A common mistake is to split a document every fixed number of characters without considering its structure.
That can separate a paragraph from its heading or break a thought in the middle.
Instead, I use paragraph-aware chunking and preserve contextual headers wherever possible.
For example, a chunk might retain this structure:
- Document: Vector Search Architecture
- Section: Similarity Scoring
- Content: How cosine similarity ranks query and document embeddings.
This metadata helps make each chunk more meaningful on its own.
For production systems, chunk size and overlap should be tested against the document type and retrieval quality rather than chosen arbitrarily.
2. Storing 768-dimensional embeddings with Supabase
Once the content is chunked, an embedding model converts each chunk into a vector containing 768 numerical dimensions.
These vectors represent semantic features learned by the embedding model. Similar meanings can be close in vector space, even when the exact words differ.
Supabase provides a PostgreSQL database, while the pgvector extension supports vector storage and similarity operations.
A simplified table could look like this:
create extension if not exists vector;
create table document_chunks (
id bigint generated always as identity primary key,
content text not null,
metadata jsonb default '{}'::jsonb,
embedding vector(768) not null
);
Important: The embedding model must actually produce 768-dimensional vectors. The database dimension needs to match the model output, and query and document embeddings must come from a compatible embedding space.
3. Ranking results with cosine similarity
After a user submits a query, the application generates a query embedding and compares it with the stored document vectors.
With pgvector, cosine distance can be calculated using the <=> operator:
select
id,
content,
1 - (embedding <=> $1::vector) as similarity
from document_chunks
order by embedding <=> $1::vector
limit 5;
Here, $1 represents the query vector supplied by the application.
The query returns the five nearest chunks according to cosine distance. The similarity expression is a convenient ranking score, not a calibrated probability that a result is correct.
For larger datasets, vector indexes, query plans, filtering, and recall-versus-latency trade-offs become important.
4. Keeping retrieval fast
A vector search system is only useful if it can return relevant context within the application's latency budget.
My performance goal for this project is sub-200ms semantic search responses.
To evaluate that properly, I would measure query latency separately from embedding generation, network overhead, and the subsequent LLM response. I would also track p50 and p95 latency across representative queries.
The actual result depends on the dataset size, index configuration, hardware, query patterns, and network conditions.
Fast retrieval also needs to preserve relevance. Returning the wrong passages quickly doesn't solve the original problem.
5. Why this architecture matters
This design is useful for applications such as:
- Documentation and codebase search
- Internal knowledge assistants
- Semantic search across long documents
- Retrieval-augmented generation pipelines
- Multimodal applications, when compatible text and non-text embeddings are supported by the selected model
One distinction matters: a 768-dimensional vector is not automatically multimodal. Multimodal retrieval requires an embedding model that supports the relevant modalities and produces compatible vectors.
Final thoughts
Building a vector search engine taught me to treat retrieval as its own engineering problem rather than an afterthought in an LLM application.
Good chunking preserves meaning. Embeddings make semantic comparison possible. pgvector makes retrieval practical inside PostgreSQL. And proper benchmarking tells you whether the architecture meets its performance goals.
The next challenge is balancing retrieval quality, latency, and cost as the dataset grows.
If you're building a RAG pipeline, I'd be interested to know what has worked best for you: smaller chunks, larger chunks, hybrid search, or reranking?
Tech stack: Node.js, TypeScript, Supabase, PostgreSQL, pgvector, vector embeddings, RAG.
Top comments (1)
read more