DEV Community

Cover image for How to build index with text embeddings
Linghua Jin for CocoIndex

Posted on Edited on Originally published at cocoindex.io

How to build index with text embeddings

This post is also on cocoindex.io, with interactive figures. The code uses CocoIndex v1.

Keyword search only finds the words you typed. Ask "how do I log back in?" and a page titled "recover your account access" never comes up, because it shares no words with your question. A page about log files does come up, because it contains "log".

Semantic search matches on meaning, so a question finds its answer even when the words are different. This guide explains the 4 ideas behind it (embeddings, similarity, chunks and an index) and then builds a semantic search index over a folder of Markdown with CocoIndex v1 and Postgres. You do not need to know anything about embeddings to start.

Every number in the figures is real. We computed it for this post with the model the example uses, all-MiniLM-L6-v2, and the 3 sample files that come with the example: the Transformer paper (1706.03762v7.md), the BERT paper (1810.04805v2.md) and the JSON spec, RFC 8259 (rfc8259.md). The full code is in the CocoIndex v1 repo.

What is a text embedding?

An embedding is a list of numbers (a vector) that a model gives to a piece of text. The model is trained so that texts with similar meanings get similar lists. all-MiniLM-L6-v2 gives every text 384 numbers, whether it is 3 words or a full paragraph.

No single number means "password" or "account". The meaning is in the pattern of all 384 together. You can think of the list as coordinates: it puts the text at one point in a space with 384 directions. Texts about the same thing end up near each other in that space.

You cannot draw 384 directions, but you can flatten them to 2 with PCA (principal component analysis), which keeps the largest differences between the points. Here are 10 everyday phrases after that flattening:

We gave the model no labels or groups. It still put "my cat sleeps all day" next to "kittens love to nap" (they share no words), and it put all the ways of saying "I can't get into my account" next to the question.

How close is close?

Search needs a number for "how close". The usual one is cosine similarity: it compares the directions of 2 vectors. 1 means the same direction, 0 means unrelated, and a value below 0 points the other way. This model scales every vector to length 1, so cosine similarity equals the dot product: multiply the 2 lists number by number and add the results.

Postgres with the pgvector extension computes this for you. Its <=> operator returns the cosine distance, which is 1 minus the similarity, so a smaller distance means a closer match. A search is then: embed the question, sort the rows by distance, keep the first few.

Why split files into chunks?

Our sample files are long. The Transformer paper in the sample folder is 46,145 characters, which is 12,746 tokens (the word pieces a model reads). all-MiniLM-L6-v2 reads at most 256 tokens. If you embed the whole file as one text, the model uses the first 2% and drops the rest without a warning.

Even a model with a longer input limit gives a file only 1 vector. A paper about many things gets 1 point somewhere between all of them, and a question about one section has to match that average. When you split the file into chunks and embed each chunk, every passage gets its own point, and the search can return the exact passage that answers the question.

The example uses CocoIndex's RecursiveSplitter. With language="markdown" it picks split points from the Markdown structure, and it aims for chunks of about 2,000 bytes (for English text, about 2,000 characters). Neighboring chunks can overlap by up to 500 bytes, so text near a boundary often appears in both. The 3 sample files give 106 chunks.

The pipeline: 3 files in, 106 rows out

The index is a short pipeline: read the files, split each one into chunks, embed each chunk, and store one row per chunk in Postgres.

Keeping the index correct is the hard part

You could write this pipeline as a plain loop, and it would embed every file once. Keeping the index correct as files change takes more code: on every run you must find what changed, embed only the new text, and make the table match.

CocoIndex is declarative: you describe what the table should contain, not the steps to update it. You write the target as a function of the source, target_state = transformation(source_state), and its Rust engine keeps the 2 in sync. You write the transformation (chunk, embed, and declare the row each chunk should have). The engine detects changes, embeds only the text it has not seen, and applies the inserts and deletes, including cleanup when a file is removed.

Build it

Prerequisites

Clone the CocoIndex repo and work in examples/text_embedding; the commands below expect its main.py and markdown_files/ folder.

Install Postgres with the pgvector extension; that is where the embeddings live. Set the connection string:

export POSTGRES_URL="postgres://cocoindex:cocoindex@localhost/cocoindex"
Enter fullscreen mode Exit fullscreen mode

Install CocoIndex and the packages the example uses:

pip install -U "cocoindex[postgres,sentence_transformers]" asyncpg pgvector numpy python-dotenv
Enter fullscreen mode Exit fullscreen mode

Step 1: define the row

Each chunk becomes one row. A plain dataclass describes the columns. The embedding field is annotated with the embedder, so CocoIndex gets the vector size from the model:

```python title=main.py
import os
import pathlib
from dataclasses import dataclass
from typing import Annotated, AsyncIterator
from numpy.typing import NDArray

import cocoindex as coco
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder

DATABASE_URL = os.getenv("POSTGRES_URL", "postgres://cocoindex:cocoindex@localhost/cocoindex")
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
EMBEDDER = coco.ContextKeySentenceTransformerEmbedder

@dataclass
class DocEmbedding:
id: int
filename: str
chunk_start: int
chunk_end: int
text: str
embedding: Annotated[NDArray, EMBEDDER]




This is one real row from the sample files, the chunk that holds section 4 of the Transformer paper:

The database connection and the embedder are created once when the app starts (in the `coco_lifespan` function) and shared through [context](https://cocoindex.io/docs/programming_guide/context/):



```python title=main.py
import asyncpg
from cocoindex.connectors import postgres

PG_DB = coco.ContextKey[asyncpg.Pool]("text_embedding_db")

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    async with asyncpg.create_pool(DATABASE_URL) as pool:
        builder.provide(PG_DB, pool)
        builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
        yield
Enter fullscreen mode Exit fullscreen mode

detect_change=True on EMBEDDER tells CocoIndex to watch the embedder itself. If you change EMBED_MODEL, the next run embeds every chunk again with the new model.

Step 2: chunk and embed

process_chunk embeds one chunk and declares its row. process_file reads a file, splits it into chunks, and runs process_chunk on each one. memo=True turns on memoization: CocoIndex caches the result, and when a file's content and the function's code are unchanged, it skips the file on the next run.

```python title=main.py
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.resources.chunk import Chunk
from cocoindex.resources.file import FileLike
from cocoindex.resources.id import IdGenerator

_splitter = RecursiveSplitter()

@coco.fn
async def process_chunk(
chunk: Chunk,
filename: pathlib.PurePath,
id_gen: IdGenerator,
table: postgres.TableTarget[DocEmbedding],
) -> None:
table.declare_row(
row=DocEmbedding(
id=await id_gen.next_id(chunk.text),
filename=str(filename),
chunk_start=chunk.start.char_offset,
chunk_end=chunk.end.char_offset,
text=chunk.text,
embedding=await coco.use_context(EMBEDDER).embed(chunk.text),
),
)

@coco.fn(memo=True)
async def process_file(
file: FileLike,
table: postgres.TableTarget[DocEmbedding],
) -> None:
text = await file.read_text()
chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language="markdown")
id_gen = IdGenerator()
await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)




> **Match chunk_size to your model.** `chunk_size=2000` comes from the upstream example. With `all-MiniLM-L6-v2`, most of those chunks are longer than 256 tokens (the median is 433 tokens in the sample files), so the model reads only the start of each one. A smaller `chunk_size` keeps more of each chunk inside the limit: at 1,000 bytes about 3 in 4 chunks of the sample files fit, and at 600 bytes almost all do. A model with a longer input limit can use larger chunks.

`declare_row` does not write to the database at once. It declares a [target state](https://cocoindex.io/docs/programming_guide/target_state/), a row that should exist. CocoIndex compares the declared rows with the ones it declared on the last run and writes only the difference.

### Step 3: wire up the app

`app_main` sets up (mounts) the Postgres table, adds a vector index so Postgres can find the closest vectors quickly, walks the folder of Markdown files, and runs one `process_file` per file:



```python title=main.py
from cocoindex.connectors import localfs
from cocoindex.resources.file import PatternFilePathMatcher

@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    table = await postgres.mount_table_target(
        PG_DB,
        table_name="doc_embeddings",
        table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=["id"]),
        pg_schema_name="coco_examples",
    )
    table.declare_vector_index(column="embedding")

    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
        live=True,
    )
    await coco.mount_each(process_file, files.items(), table)

app = coco.App(
    coco.AppConfig(name="TextEmbeddingV1"),
    app_main,
    sourcedir=pathlib.Path("./markdown_files"),
)
Enter fullscreen mode Exit fullscreen mode

coco.mount_each gives each file its own processing component: a unit of work that CocoIndex runs, tracks and cleans up on its own. That is what lets CocoIndex update one file without touching the others.

Step 4: build the index

cocoindex update main
Enter fullscreen mode Exit fullscreen mode

The first run embeds all 106 chunks of the sample files. Add -L (cocoindex update -L main) to keep watching the folder and update the index as files change (live mode).

Query the index

Search mirrors indexing: embed the question with the same model, then ask Postgres for the nearest vectors. In main.py, the pool is created with asyncpg.create_pool(DATABASE_URL, init=register_vector) (from pgvector.asyncpg), so asyncpg can send a vector as a query parameter. A different model puts text in a different space, so its vectors cannot be compared with the stored ones.

```python title=main.py
async def query_once(pool, embedder, query: str, *, top_k: int = 5) -> None:
query_vec = await embedder.embed(query)
async with pool.acquire() as conn:
rows = await conn.fetch(
"""
SELECT filename, text, embedding <=> $1 AS distance
FROM "coco_examples"."doc_embeddings"
ORDER BY distance ASC
LIMIT $2
""",
query_vec,
top_k,
)
for r in rows:
print(f"[{1.0 - float(r['distance']):.3f}] {r['filename']}")
print(f" {r['text']}")




Run it with a question in plain language:



```sh
python main.py "what is self-attention?"
Enter fullscreen mode Exit fullscreen mode

$1 LIMIT 5. The top 5 of 106 chunks by score, with the chunk's first line and where it is: 0.494, #### Attention Visualizations, 1706.03762v7.md 45126 to 46144; 0.445, ### 3.2.3 Applications of Attention in our Model, 13016 to 14977; 0.419, ### 4 Why Self-Attention, 17982 to 19559; 0.343, ## 2 Background, 4905 to 6513; 0.329, ### 3.2.1 Scaled Dot-Product Attention, 9743 to 11378. Score = 1 minus distance."} title={"A search"} width={960} height={452} detailScale={1.4} minWidth={840} />

All 5 hits come from the Transformer paper, and none of them comes from the BERT paper or the JSON spec. The section 4 row from Step 1 is the third hit. We computed this ranking with the same model; <=> gives the same order, because distance is 1 minus similarity. With the vector index, Postgres may return a slightly different top 5, because that index finds nearest neighbors approximately.

A re-run redoes only what changed

Run cocoindex update main again and CocoIndex does only the work that the change needs:

2 mechanisms make this work. memo=True on process_file skips files that did not change. The embedder's embed() is memoized too, so in an edited file only the chunks with new text go to the model. On the write side, each row's id comes from IdGenerator.next_id(chunk.text), so a chunk with the same text keeps the same id across runs. Rows for chunks that are gone are deleted, new chunks are inserted, and rows whose text moved get their new offsets without a new embedding.

Where to go next

You now have the core of semantic search: chunk, embed, store, and query by nearest neighbor. The same pipeline can read from Amazon S3, Azure Blob or Google Drive instead of a local folder, write to a vector database such as Qdrant instead of Postgres, or pass the passages it finds to an LLM for RAG (retrieval-augmented generation: the LLM answers from the passages you retrieved). Because the pipeline is declarative and incremental, none of that changes how the index stays fresh.

For code instead of Markdown, Index your codebase uses the same flow with syntax-aware chunking.

Top comments (0)