I built vector search by hand out of curiosity, and my toy caught a mistake I did not know I had made

TL;DR: I shipped a "second brain" app at a hackathon that searched notes by meaning, and I did not understand the search part. So I rebuilt it by hand in numpy, 20 fake products in 8 dimensions. The mechanism turned out to be about ten lines. The interesting part was that the toy quietly told me two of my four categories overlapped, which I had done to myself and never noticed. That is the thing real embeddings will never let you see.

The question that started it

At a hackathon I built second-brain, which stores notes as memories and searches them by meaning instead of exact words. It runs sentence-transformers (all-MiniLM-L6-v2) to turn text into 384-dimensional vectors, and Qdrant to store and search them. It worked. You could type a rough idea and get back the note you meant, with none of the same words in it.

But I had wired that up without knowing what it did. It was an embed() call and a Qdrant query, and it felt like magic. Magic in my own project bothers me, so I wrote down the actual question:

What happens between embed(query) and the results coming back?

Not "what is an embedding". I could recite that. I wanted the mechanism, the part that takes a pile of vectors and picks the close ones. So I built one. No model, no vector database, no framework. Just numpy, 20 made-up products across four categories (electronics, clothing, cooking, stationary), each clustered around a hand-picked center in 8-dimensional space with noise added so they were not identical.

import numpy as np

# each category is a cluster center in 8-dimensional space
electronics_center = np.array([0.9, 0.1, 0.2, 0.08, 0.1, 0.3, 0.79, 0.2])
clothing_center    = np.array([0.1, 0.8, 0.09, 0.1, 0.09, 0.2, 0.1, 0.89])
cooking_center     = np.array([0.2, 0.3, 0.95, 0.2, 0.1, 0.9, 0.3, 0.1])
stationary_center  = np.array([0.8, 0.2, 0.86, 0.9, 0.08, 0.1, 0.2, 0.1])

# real embeddings vary, so simulate that with noise around each center
noise = 0.1
electronics = electronics_center + noise * np.random.randn(5, 8)
# ...same for the other three categories

Using fake vectors was the whole point, not a shortcut. I did not want to learn how a transformer produces a vector. I wanted to isolate the half I did not understand, which is what you do with the vectors once you have them. Remember the four centers above. They come back later and they are the best part of this post.

Question 1: how do you actually compare two vectors?

The answer is short enough to be annoying. Normalize every vector to unit length, then rank by dot product. For unit vectors, the dot product is cosine similarity, because the denominator in the cosine formula is just the two magnitudes, and you already made both of them 1.

def normalize(vectors: np.ndarray) -> np.ndarray:
    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
    norms = np.where(norms == 0, 1e-10, norms)  # avoid divide by zero
    return vectors / norms

class VectorIndex:
    def __init__(self):
        self.vectors = None
        self.labels = None

    def add(self, vectors: np.ndarray, labels: list):
        self.vectors = normalize(vectors)   # normalize once, at index time
        self.labels = labels

    def search(self, query_vector: np.ndarray, top_k: int = 3):
        query = normalize(query_vector.reshape(1, -1))
        scores = np.dot(self.vectors, query.T).flatten()  # cosine similarity
        top = np.argsort(scores)[::-1][:top_k]
        return [(self.labels[i], float(scores[i])) for i in top]

That is the entire engine. It ran, and every query came back with the right category on top:

Query: 'Cooking'
  1. [0.9948] Ceramic pan
  2. [0.9816] Iron cast pan
  3. [0.9749] wodden spoon

Query: 'Clothing'
  1. [0.9935] Silk saree
  2. [0.9808] Oversized black t-shirt
  3. [0.9775] Women's crop top

Two things clicked here that no amount of reading had done. Normalizing is not a ritual, it is what buys you the cheap dot product later. And it belongs in add(), not search(), because you index once and you query forever.

The other thing I finally understood is why it is cosine and not plain Euclidean distance. Cosine only looks at direction, so it ignores magnitude. For text that matters, because a long document and a short one about the same topic can differ a lot in magnitude while pointing the same way. Distance would call them far apart. Cosine calls them the same thing.

Qdrant adds a great deal on top of this, and I will get to what, but the core of what my second-brain runs on is those ten lines.

Question 2: can I see the space?

8 dimensions, so no. Not directly. I projected down to 2D with PCA and plotted the products as dots and the queries as stars. The four clusters separated cleanly and each query star landed inside its cluster. Very satisfying.

Then I printed how much of the variance those two components actually captured:

PC1: 45.92%
PC2: 26.45%
Total (PC1 + PC2): 72.36%

So the picture I was happily trusting is showing me about 72 percent of the structure. More than a quarter of what distinguishes these vectors is not on that plot at all, and 8 dimensions is a toy. Real embeddings are 384 or 1536. Whatever a t-SNE or UMAP plot of those looks like, the honest reading is that it is a heavily lossy sketch, and two points sitting on top of each other in the picture may not be near each other in the space.

That was the first thing I got out of the build that I did not expect: a healthy distrust of every embedding visualization I had ever nodded along to, including my own.

Question 3: is a high score proof the result is right?

Top-3 always looked perfect, which is exactly the problem, because top-3 is all a search UI ever shows you. So I stopped looking at the top and scored the query against all twenty products, then sorted.

That is when the run got interesting. Here is the tail of each query, the highest-scoring wrong-category item:

Query	Worst in-category score	Best wrong-category score	Margin
Clothing	0.9572	0.4457 (wodden spoon)	0.51
Electronics	0.9441	0.6542 (Parker pen)	0.29
Cooking	0.9649	0.6689 (Glue stick)	0.30
Stationary	0.9704	0.7110 (Nakiri knives)	0.26

A cutoff around 0.8 separates right from wrong for every query here, so the toy did not fail. But look at the margins. Clothing has twice the breathing room that stationary does. Same code, same noise, same everything. Why would one category be twice as easy as another?

The thing I did not go looking for

The queries were generated near the cluster centers. So if some categories are getting confused with each other, maybe the centers themselves are the problem. I had never thought to check them, because I picked those numbers by hand, in about thirty seconds, to "look different". So I measured the cosine similarity between the four centers:

electronics  vs clothing     0.3318
electronics  vs cooking      0.5287
electronics  vs stationary   0.6202
clothing     vs cooking      0.3872
clothing     vs stationary   0.2911
cooking      vs stationary   0.6475

There it is. Clothing sits between 0.29 and 0.39 from everything else, off in its own corner. Stationary is 0.62 from electronics and 0.65 from cooking, wedged right between them. Go back and look at the centers I hand-wrote: stationary is high on dimension 0 (0.8), and so is electronics (0.9). Stationary is also high on dimension 2 (0.86), and so is cooking (0.95). I built a category that shares a strong axis with two others without noticing, and then I built the two other categories it overlaps with.

And the search told me. The two most-confused pairs in the search results, stationary and cooking bleeding into each other at 0.71 and 0.67, are exactly the two highest-similarity pairs in that table. The retrieval quality was a direct readout of a geometry mistake I had made and could not see.

That is the payoff of using fake vectors, and it is the opposite of what I expected. I thought the toy would teach me the algorithm and the algorithm was the boring part. What the toy actually gave me is the only setting where I get to see the ground truth of the space and the search behavior at the same time, and compare them. With all-MiniLM-L6-v2 I get 384 numbers I cannot interpret, and if two of my note categories overlap in that space, nothing tells me. The search just quietly gets worse and I blame the threshold.

What this changed about my actual app

The retrieval in second-brain is not pure similarity, and building the naive version showed me why it cannot be:

payload = {
    "vector": embed(query),         # semantic similarity
    "limit": limit,
    "score_threshold": 0.5,         # drop "near but irrelevant" hits
    "with_payload": True,
    "filter": {                     # hard constraints, not similarity
        "must": [
            {"key": "memory_type", "match": {"value": "knowledge"}},
            {"key": "importance", "range": {"gte": 0.3}},
        ]
    },
}
requests.post(f"{QDRANT_HOST}/collections/knowledge_memory/points/search", json=payload)

Cosine similarity does exactly one thing: find the nearest vector. Nearest is not "correct", and it is definitely not "allowed". So the similarity finds candidates and everything else enforces the rules. "Notes from last week tagged work" is a database query wearing a search box, not a similarity question, and no amount of embedding quality will turn it into one.

The same logic explains the failure I have hit most in practice: exact matches get worse, not better. Search a file name, an ID, or a tag, and semantic search will confidently hand you something close in meaning and bury the exact hit. Plain string matching wins there and it is not close.

I also now think that score_threshold: 0.5 is a number I made up. It works, in the sense that the app stopped returning nonsense. I have no principled reason for it, and the margin table above is why that bothers me. The gap between right and wrong moved by 2x across four categories in a dataset I built myself. In a real embedding space I have no idea what it is.

The questions I left with

The mechanism took an afternoon. These did not:

How do you pick a score threshold honestly? Mine is a vibe. Doing it properly seems to need labelled query and result pairs and a look at where the score distributions actually cross, which means I need ground truth for my own notes, which I do not have. This is the one I want to solve next.

How does the geometry change at 384 dimensions? Everything above is 8 dimensions, where I can reason about a single axis being shared. I have read that similarity scores bunch up as dimensions increase, so everything starts looking moderately similar to everything. I have not tested it, and my whole intuition here was built in a space small enough to be unrepresentative.

How does a real index avoid comparing everything? My search() is a dot product against every vector, O(n) per query, which is fine for 20 products and absurd for a million. Qdrant uses HNSW to approximate. So it can be wrong, on purpose, for speed. I would like to build that next and find out what "approximate" actually costs, because right now I take that trade on faith.

Filtering and approximate search must fight each other. If HNSW is walking a graph to find near neighbors, and I also demand importance >= 0.3, does it filter before it walks or after? Filter after and you can get back fewer results than you asked for. Filter before and it is not clear how the graph still works. I know Qdrant solves this. I do not know how.

How do you route a query to the right search? The exact-match problem has a known answer, hybrid search: run keyword and vector search, merge the rankings. What I do not have is a clean way to decide which path a query deserves, or whether you just always run both and fuse. This is where I am stuck.

If you have built hybrid retrieval in production, I would like to know how you make that routing decision, and whether you calibrated your score threshold or picked it the way I did. That is the honest state of it.

Second-brain is here:

CoffeeAurCode / second-brain-ai_MVP

second-brain-ai (qdrant-based)

this project is a minimal Second brain AI system using qdrant as the primary vector memory store

How to Run

1. Start Qdrant locally docker run -p 6333:6333 qdrant/qdrant

2. Install dependencies: pip install -r requirements.txt

3. Run the any of the demo file: python Demo_1.py/Demo_2.py/Demo_3.py

The demo will store knowledge, retrieve relevant memories, apply memory evolution, and retrieve again.

Dependencies

This project uses Qdrant as the primary vector search engine. Qdrant is run locally using Docker, which is the official and recommended deployment method.

Docker is required only to start the Qdrant service. All application logic runs in Python.

Memory Architecture

The system uses Qdrant as the primary memory store.

Three typed memory collections are defined:

• knowledge_memory: long-term semantic knowledge(hard disk)
• context_memory: short-lived working context(RAM)
• interaction_memory: conversation traces and user intent(EQ)

All future retrieval and memory evolution operates exclusively on these collections.

Embeddings

Text is converted into semantic vector representations…

View on GitHub