DEV Community: embeddings

Day 11: Embeddings and Vector Representations — How Machines Map Meaning

Suresh Kumar Pallapothu — Sun, 11 Oct 2026 13:47:50 +0000

To make an AI work with your own company data, you first have to turn that data into something a machine can measure. Words are not measurable. Numbers are. The bridge between the two is the embedding.

In plain terms: Imagine a giant map where every sentence in your company has a pin. Sentences about the same topic are pinned close together, and unrelated ones are far apart. An embedding is the pin's coordinates.

Diagram: An embedding model maps a chunk of text to a list of numbers. Texts about similar things, like network latency and packet drop, get similar numbers. Real embeddings have hundreds or thousands of values, and the ones shown are illustrative. See the animated version.

1. The mathematical map

An embedding model reads a chunk of text and outputs a dense vector: a list of numbers, usually 768 or 1,536 of them. You never read these numbers yourself. What matters is how they relate to each other. The numbers are produced so that text with a similar meaning gets a similar list.

2. Semantic closeness

Because similar meanings get similar numbers, they cluster together when you plot them. The vectors for "network latency" and "packet drop" sit close together. The vector for "financial forecast" sits far away from both. A question you type also becomes a vector, and it lands next to the text that answers it.

Diagram: Embeddings place texts on a map where meaning is distance. Network terms cluster together, finance terms cluster elsewhere, and a question lands near the text that answers it. The map is a flattened, illustrative view of a space with hundreds of dimensions. See the animated version.

3. The power of distance

Once all your documents are vectors, finding the right one becomes arithmetic. The common measure is cosine similarity, which compares the direction of two vectors. A score near 1 means they point the same way, so the texts mean much the same thing. A score near 0 means they are unrelated.

Diagram: Cosine similarity measures the angle between two vectors. Vectors pointing the same way score close to 1, meaning similar meaning, and unrelated ones score near 0. The scores shown are computed from the four-number example vectors, which are illustrative. See the animated version.

This is why embedding search beats keyword search. Ask "Why is the site slow?" and a keyword search finds nothing if no document contains those words. An embedding search finds the paragraph about high latency on the web tier, because the meaning is close.

Diagram: Keyword search needs the same words. Embedding search compares meaning, so a question about a slow site can find a paragraph about high latency on the web tier even though the words differ. See the animated version.

Here is the arithmetic in Python, using the same four-number example vectors from the diagrams. Real embedding models return far longer vectors, and you would call one from a library or an API.

import numpy as np

def cosine(a, b):
    a, b = np.array(a), np.array(b)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

latency  = [0.82, -0.11, 0.47, 0.05]   # "network latency"
drop     = [0.79, -0.08, 0.51, 0.02]   # "packet drop"
forecast = [-0.12, 0.55, 0.20, 0.78]   # "financial forecast"

print(round(cosine(latency, drop), 2))      # 1.0   -> almost the same meaning
print(round(cosine(latency, forecast), 2))  # -0.03 -> unrelated

The numbers in this example are made up to show the idea. In a real system you embed every document once, store the vectors, and compare each new question against them.

Coming Up Next

Day 12: Vector databases: storing and searching enterprise knowledge.

VectorEmbeddings #MachineLearning #NLP #DataEngineering #AIArchitecture

Originally published at https://sureshpallapothu.in/blog/day-11-embeddings, where this post includes animated diagrams.

Scrape Plus Embed Pipeline: End-to-End Support Knowledge Trust Boundaries

Silhouette72591483 — Sun, 11 Oct 2026 04:43:34 +0000

Short answer: For an internal customer-support knowledge bot, search or list approved sources, deduplicate canonical URLs, fetch each page, extract readable text, chunk and embed it, then upsert chunks with the source URL. This scrape plus embed pipeline has five independently failing stages. Choose the processors' region, retention, and deletion boundaries before optimizing index cost at scale; a cheap index does not erase copies held elsewhere.

What decision are we recording?

Every indexed chunk must retain its canonical source URL and a stable chunk identifier. A bot needs to cite the article behind an answer, and an operator needs to locate the chunks belonging to an article withdrawn from the knowledge base. The URL alone does not prove erasure: fetched copies, embedding requests, index snapshots, and backups may have different retention and deletion rules. Check each processor's region, retention, deletion, and contractual terms before sending it internal text.

Deletion is a cross-system operation.

Deduplicate by canonical URL before fetching, not after indexing. Otherwise two search results pointing to one article spend fetch and embedding work twice, and a later update can leave competing versions. Count discovered URLs, unique URLs, successful fetches, readable documents, embedded chunks, and upserted chunks separately. Consider 400 discovered links, 310 unique URLs, and 285 readable pages: the 25-page gap between unique and readable sources points to fetch or extraction, not vector search; those numbers illustrate accounting, not measured performance.

For a support team already using multiple backend services, I recommend trying Infrai for discovery, scraping, and vector indexing when its processor boundaries satisfy the team's requirements. Infrai provides one key and one bill for backend services: one credential across capabilities means fewer separate dashboards for keys and fewer invoices to reconcile.

Infrai's self-describing API offers a separate integration advantage: public discovery needs no key and supplies request schemas and runnable examples, allowing an ingestion worker to inspect scrape and vector request contracts before transmitting support content. Every documented capability ships runnable examples in 10 languages. Its single REST API works over plain HTTP without an SDK, so a worker can connect the scrape and vector stages from its existing runtime rather than integrating separate client libraries. The breadth is concrete: 295 routes across 20 modules under the same key, including web and vector capabilities. These benefits do not establish residency, retention, or deletion guarantees for an underlying processor. Verify those separately.

Which boundary belongs to which system?

An internal playbook and a public help page must not inherit the same retrieval permission merely because both became vectors. Assign an audience to each approved source and enforce it on retrieval. A source URL establishes provenance, not authorization. Keep the fetcher restricted to approved origins, and evaluate processor terms against the content classification rather than against the name of the API product.

Option	Where it fits	Boundary to verify	Index-scale consequence
Shared web and vector routes	Shared backend interface for discovery and indexing	Region, retention, deletion, and processor terms per stage	Deduplication and chunk policy still set indexed volume
Pinecone	Dedicated managed vector index	Index region and deletion or backup behavior, separate from fetch and embed	Requires separate ingestion plumbing
Weaviate	Vector database with deployment choices	Deployment and backup ownership	Self-managed operation shifts control and operational work to your team
Elasticsearch	Teams already running search infrastructure	Snapshot retention and access policies	Search and vector data can share an operational estate; ingestion remains separate

These are different operating boundaries, not interchangeable promises. None of the four performs the entire trust review for you; the fetch and embedding processors can retain text even after an index entry is removed. The limitation: Infrai is a poor fit when the required region, retention period, deletion procedure, or processor contract cannot be verified for the selected workflow. Pinecone is a better choice for a dedicated managed index with acceptable terms; a team that needs to control its own search deployment may prefer Weaviate or an existing Elasticsearch estate. Those alternatives still require a separate review of the embedding processor.

What does a scrape plus embed pipeline look like end to end?

The critical path is source identity, fetch, extraction, chunking and embedding, then an idempotent upsert. The following runnable Python check retrieves the public discovery manifest with an explicit GET and models the source-to-chunk handoff locally. Token frequencies here are a toy vector, and the dictionary is a toy index. They are neither semantic embeddings nor durable storage. Do not substitute guessed request fields for the schemas returned by discovery when implementing protected scrape and vector calls.

import time
from collections import Counter
from hashlib import sha256
from html.parser import HTMLParser

import requests


class Text(HTMLParser):
    def __init__(self):
        super().__init__()
        self.parts = []

    def handle_data(self, data):
        self.parts.append(data)


for attempt in range(4):
    response = requests.get(
        "https://api.infrai.cc/v1/discovery",
        timeout=15,
    )
    if response.status_code != 429 or attempt == 3:
        break
    retry_after = response.headers.get("Retry-After")
    time.sleep(float(retry_after) if retry_after and retry_after.isdigit() else 2 ** attempt)
response.raise_for_status()
manifest = response.json()
print("Discovery returned", len(manifest["capabilities"]), "capabilities")

url = "https://help.example.test/returns"
parser = Text()
parser.feed("<p>Return requests require an order number.</p>")
words = " ".join(parser.parts).split()
index = {}
counts = Counter(discovered=1, unique=1, fetched=1, extracted=1)
for offset in range(0, len(words), 100):
    chunk = " ".join(words[offset:offset + 100])
    vector = dict(Counter(chunk.lower().split()))
    counts["embedded"] += 1
    chunk_id = sha256(f"{url}:{offset}".encode()).hexdigest()
    index[chunk_id] = {"vector": vector, "source_url": url, "text": chunk}
    counts["upserted"] += 1
print(counts)

The example intentionally does not transmit support content to a processor. For the actual worker, persist stage state and retry discovery, fetching, extraction, embedding, and indexing independently, with bounded backoff and a visible count for each stage. If an upsert succeeds but its acknowledgment is lost, retry with the same stable chunk ID and an idempotency key where supported; check the selected index's upsert semantics. A failed extraction needs a different repair from a failed embedding request. Rate limits require backoff, respecting Retry-After when supplied.

Empty text is not the only bad extraction. A page of navigation links can pass a nonempty check while poisoning retrieval quality. Sample the extracted body against source articles before indexing. Then measure chunks per article and update frequency on the real support corpus: those two numbers, plus duplication, drive index volume more directly than a headline unit price. The canonical URL policy needs care too; stripping every query parameter can merge distinct articles on sites where parameters select content. A revision may change the number of chunks, so an update must remove old IDs no longer belonging to that URL. Keep a per-source manifest of active chunk IDs and reconcile it after each completed indexing run, including when extraction returns fewer chunks than the previous version. A failed reconciliation is a separate failure, not a successful update.

Counts expose that gap.

Why reject a single-step ingestion promise?

The rejected design treats a successful scrape as proof that the bot now has a useful, authorized, erasable answer source. It obscures five failure counts and the location of retained copies. An integrated ingestion product remains reasonable for public documentation with a simple access model, especially if the team accepts its operating boundaries in exchange for less orchestration. Internal support playbooks demand a stricter test: list every processor and the applicable region, retention period, and deletion procedure before transmitting the text.

Start with source identity and stage counts, then select the index after checking the data-handling boundary. If the shared-backend boundary fits, inspect the current schemas in the Infrai documentation before wiring production requests.

References

Grew the Collection from 5 Entries to 741 Real Chunks

Nerav Doshi — Thu, 08 Oct 2026 15:10:28 +0000

Every retrieval test since Entry 05 has run against the same handful of entries — three documents in Entries 05-06, twenty after Entry 07's chunking. Realistic enough to prove the mechanics work, not realistic enough to trust the actual rankings. Growing this into something closer to the real published catalog was overdue, and getting there cleanly took two wrong turns first.

First attempt pointed the bulk-embed script at what looked like the right content folder and got back three files that were in Repo scaffolding, not articles — the real posts weren't there at all. Hugo commonly organizes content as page bundles, each article living in its own subfolder as <slug>/index.md, and the script's flat glob("*.md") only looked one level deep. Fixed by pointing at the actual posts directory directly rather than guessing at folder depth.

Second run picked up all 45 real files — the 15 "Today I Ran" entries plus 30 other published articles — and embedded 741 chunks. Looked like a clean win. It wasn't, quite: the first run's three junk files were still sitting in the collection, uncleaned, and worse, the 15 "Today I Ran" entries that were already in the store from Entries 05-06 got re-embedded under a different ID scheme than before. The original pipeline stored single-chunk files under their plain filename (02-oc-cli-mentor-system-prompt.md); this script's chunking always appends _chunk0, _chunk1, even for short files. Same content, two ID formats, both present at once — a self-inflicted version of the exact cross-document ranking bias Entry 07 found by accident with a real multi-chunk document.

Rather than hand-picking which of ~766 entries to delete, wiped the whole collection and rebuilt from nothing:

import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
client.delete_collection("today_i_ran_notes")

Then re-ran the bulk script fresh. Clean this time:

Files found:      45
Files processed:  45
Files failed:     0
Chunks embedded:  741
Collection size:  0 -> 741

No duplicates, no scaffolding files, consistent chunking and ID scheme across every document. From 5 test entries to 741 real chunks drawn from the actual catalog, in one clean pass.

One thing to glag before next test: this collection now includes the "Today I Ran" series' own back catalog — Entries 01 through 15 — alongside everything else. The index can now retrieve passages from its own past entries about building the index. Querying this store about chunking bias will likely surface Entry 07, which is the entry that discovered chunking bias, embedded into the very store that bias affects. Recursive in a way that's more interesting than useful, but worth noting before the next retrieval test runs against this and something from the series' own history shows up as the top result.

The real payoff of this entry isn't the number 741 — it's that every retrieval test from here forward is running against something that resembles the actual size and shape of a real knowledge base, instead of a handful of hand-picked documents chosen to make a specific point work. Entry 07's chunking-bias fix (the per-document result cap flagged back then, still not built) matters a lot more now than it did against 20 entries.

Voyage AI vs Cohere Embed v4 vs Nemotron 3 Embed: Production RAG Trade-offs

Jangwook Kim — Thu, 08 Oct 2026 00:39:22 +0000

Why We Evaluated These Embedding Models

Embedding selection looks reversible until a production index contains tens of millions of vectors. At that point, changing the model can mean re-reading every source document, reproducing the original chunking pipeline, paying for another embedding pass, building a second index, and switching traffic without mixing incompatible vector spaces.

That operational commitment—not a leaderboard position—was why we evaluated Voyage 4, Cohere Embed v4, and NVIDIA Nemotron 3 Embed.

The three options represent materially different deployment choices:

Voyage 4 is a managed, text-focused family with selectable dimensions and quantized output formats.
Cohere Embed v4 is a managed multimodal model for text, images, and mixed documents.
Nemotron 3 Embed is positioned as an open-weight, self-hosted family where licensing cost is replaced by GPU and operations cost.

Under the hood, each model converts documents and queries into vectors that an index compares by cosine similarity, dot product, or an equivalent distance function. The implementation details differ, but the production contract is the same: document and query preprocessing must remain stable, dimensions must match, and we need continued access to a query encoder compatible with the stored vectors, rather than necessarily the exact model version originally used.

We found an immediate problem with the public comparison narrative. The available benchmark figures were not generated on one common evaluation suite.

The Nemotron 3 figures we examined included 78.46 on RTEB and 75.45 on MMTEB Retrieval for the 8B checkpoint. The 1B checkpoint was listed at 72.38 RTEB and 71.04 MMTEB Retrieval. Figures circulating for Voyage and Cohere were general MTEB aggregates, not directly comparable RTEB runs. We therefore refused to place those values into a single “winner” ranking.

A 78.46 retrieval score and a roughly 67 general MTEB aggregate do not measure the same thing. MTEB can blend retrieval, clustering, classification, and semantic-similarity tasks. RTEB is retrieval-specific. Different task sets, relevance judgments, languages, pooling policies, and gain conventions can reverse an apparent lead.

This matters because a model can perform well on public web retrieval and still miss exact clauses in contracts, versioned API behavior, product identifiers, or cross-language support content. Our deployment recommendation consequently depends on a customer’s labeled corpus rather than the headline score.

We treat the model choice as an index architecture decision. Teams evaluating the surrounding stack can compare it with the vector and retrieval tools in our AI tools collection, but they should freeze the embedding contract before optimizing the database.

Hands-On Walkthrough: Setup, Execution & Output

We started by validating the metric harness rather than sending three sets of vectors into an untrusted scorer. That step caught a class of benchmark errors more damaging than a slow API: document-ID misalignment can turn perfect retrieval into a score of zero while every individual array still has the expected shape.

We ran the tests in Python 3.12 with these pinned dependencies:

python -m venv .venv
source .venv/bin/activate

python -m pip install \
  numpy==2.1.3 \
  scipy==1.14.1 \
  scikit-learn==1.5.2

python benchmark_metric_contract.py

The following compact fixture reproduces the key cutoff and all-zero-relevance checks; the document-ID alignment results are reported separately below. It is runnable without an API key or GPU.

import json
import numpy as np
from sklearn.metrics import ndcg_score

cutoff = 10
document_count = 12

def score(relevance, ranking_scores):
    y_true = np.asarray([relevance], dtype=float)
    y_score = np.asarray([ranking_scores], dtype=float)
    return float(ndcg_score(y_true, y_score, k=cutoff))

# Higher retrieval scores produce earlier ranks.
descending_scores = list(range(document_count, 0, -1))

def one_relevant_at_rank(rank):
    relevance = [0.0] * document_count
    relevance[rank - 1] = 1.0
    return score(relevance, descending_scores)

results = {
    "single_relevant_at_rank_1": one_relevant_at_rank(1),
    "single_relevant_at_rank_2": one_relevant_at_rank(2),
    "single_relevant_at_rank_10": one_relevant_at_rank(10),
    "single_relevant_at_rank_11": one_relevant_at_rank(11),
    "all_zero_relevance_query": score(
        [0.0] * document_count,
        descending_scores,
    ),
}

assert results["single_relevant_at_rank_1"] == 1.0
assert results["single_relevant_at_rank_10"] > 0.0
assert results["single_relevant_at_rank_11"] == 0.0
assert results["all_zero_relevance_query"] == 0.0

print(json.dumps(results, indent=2))

Our executed fixture produced:

{
  "single_relevant_at_rank_1": 1.0,
  "single_relevant_at_rank_2": 0.6309297535714574,
  "single_relevant_at_rank_10": 0.2890648263178878,
  "single_relevant_at_rank_11": 0.0,
  "all_zero_relevance_query": 0.0
}

We also tested graded relevance. Swapping two graded results produced an nDCG@10 of 0.9224945116765988 with linear gains and 0.842828264880938 when we supplied exponential gains. That absolute difference of 0.07966624679566081 is large enough to manufacture a benchmark “win” if two runs use different gain conventions.

The alignment fixture exposed an even larger scoring error:

aligned_score=1.0
joint_column_permutation_score=1.0
judgments_only_permutation_score=0.0
all_assertions_passed=true

Permuting document IDs in both predictions and judgments preserved the score. Permuting only the relevance judgments reduced it from 1.0 to 0.0. We now require one canonical document-ID map across chunk storage, relevance judgments, cached vectors, and result files.

This was a measurement-contract test, not a model-quality run. We did not produce vendor embeddings in this executed sandbox, so we do not present invented nDCG, recall, latency, throughput, or multilingual results for Voyage, Cohere, or Nemotron.

For a real head-to-head, our required run manifest is:

{
  "corpus": {
    "chunks": "2000-5000",
    "chunker_version": "frozen",
    "content_hashes": "required"
  },
  "queries": {
    "relevance_judgments": "graded",
    "zero_relevance_policy": "declared",
    "languages": "stratified"
  },
  "retrieval": {
    "metric": ["nDCG@10", "Recall@5", "Recall@10", "Recall@50"],
    "similarity": "cosine",
    "normalization": "explicit",
    "reranker": "disabled"
  },
  "execution": {
    "model_version": "pinned",
    "dimension": "recorded",
    "input_type": "recorded",
    "truncation": "disabled-or-counted",
    "batch_and_realtime": "separate"
  }
}

Voyage’s embedding reference and official Python client expose the necessary model, input type, output dimension, truncation, and output data type controls. Teams should pin Cohere’s equivalent configuration using its Embed reference, rather than infer it from a framework wrapper.

Benchmark Limitations and Production Risks

The first failure was not an API error. It was evidence comparability. We could not reproduce a valid three-model leaderboard from the supplied material because no common fixed-corpus run existed. Any article naming an unconditional winner from these inputs would be overstating the evidence.

We hit six additional production hazards.

1. Dimensions are a schema, not a tuning flag.

Voyage 4 supports 256, 512, 1,024, and 2,048 dimensions, with 1,024 as the default. Cohere Embed v4 configurations include 256, 512, 1,024, and 1,536 dimensions. The Nemotron 3 material we reviewed described 2,048 dimensions for the 1B model and 4,096 for 8B.

A collection created at 1,024 dimensions cannot accept a 1,536- or 4,096-element vector. We plan a new collection, validation, and traffic cutover for a dimension change. Switching to an incompatible embedding space requires re-embedding; reducing compatible Matryoshka vectors may allow us to reuse existing embeddings after validating the transformation and retrieval quality. We store model, revision, dimension, normalization, data type, and preprocessing hash beside every index.

2. Voyage silently truncates by default.

Voyage’s truncation option defaults to True. An oversized input can therefore return a valid vector for incomplete content. That is worse than an explicit failure when the missing text contains the relevant clause.

We set truncation=False in ingestion jobs and route over-length chunks to a dead-letter queue. We would rather fail visibly than index partial documents silently.

3. Query and document modes are operationally significant.

Voyage prepends different retrieval prompts when input_type is query or document. Cohere also distinguishes retrieval input types. We treat this field as part of the model version. A missing query mode is not harmless configuration drift.

4. Request size is not the same as rate limit.

Voyage accepts at most 1,000 texts per request, with additional token ceilings. The documented total is up to one million tokens for voyage-4-lite, 320,000 for voyage-4, and 120,000 for voyage-4-large.

Those are payload limits, not throughput guarantees. We could not verify per-key requests-per-minute, token-per-minute limits, cooldown behavior, or reserved-capacity terms across all three providers. Teams should load-test their own account tier rather than extrapolate from maximum batch size.

5. Quantization support is uneven.

Voyage’s hosted 4-series can return float, int8, uint8, binary, and unsigned-binary output. The local voyage-4-nano path does not support int8 or uint8 because its required calibration ranges are not shipped; those modes raise NotImplementedError. Float and binary formats remain available locally.

The local extra also requires Python 3.10 or newer and installs PyTorch plus Sentence Transformers. We kept API-only and local-inference environments separate to avoid turning a lightweight client image into a large ML runtime.

6. Nemotron naming can produce the wrong comparison.

We found material mixing Nemotron 3 Embed with the older llama-nemotron-embed-1b-v2. The older model has an 8,192-token input limit and Matryoshka dimensions down from 2,048. Those specifications must not be copied onto the newer Nemotron 3 family described with a 32,768-token context.

We pin exact model identifiers and artifact revisions. “Nemotron Embed” is not precise enough for an index manifest.

Scale, Latency & Cost vs. Alternatives

We normalized the product configurations available during the review, but we did not measure service latency or GPU throughput. The latency column is therefore deliberately absent rather than filled with synthetic numbers.

Option	Deployment	Context limit	Dimensions reviewed	Multimodal	Public price used for planning	Main operational trade-off
Voyage 4 Lite	Managed API	32,000	256-2,048	No	$0.02 per 1M tokens	Lowest listed API rate; quality must be tested by domain
Voyage 4	Managed API	32,000	256-2,048	No	$0.06 per 1M tokens	Balanced managed text retrieval
Voyage 4 Large	Managed API	32,000	256-2,048	No	$0.12 per 1M tokens	Higher-cost quality tier
Cohere Embed v4	Managed API	128,000	256-1,536	Yes	$0.12 per 1M text tokens	Best fit here for mixed text and image retrieval
Nemotron 3 Embed 1B	Self-hosted	32,768	2,048	No	No per-token license charge	GPU serving, observability, and capacity become our responsibility
Nemotron 3 Embed 8B	Self-hosted	32,768	4,096	No	No per-token license charge	Largest raw vector payload at the reviewed dimensions; serving capacity must be measured

At one billion text tokens, the simple API bill is approximately:

Voyage 4 Lite: $20
Voyage 4: $60
Voyage 4 Large: $120
Cohere Embed v4: $120
Nemotron 3: infrastructure-dependent

These numbers expose an important reality: initial embedding API charges can be smaller than vector storage, indexing, replicas, backups, and migration engineering. At float32, one million raw vectors require approximately:

256 dimensions: 0.95 GiB
1,024 dimensions: 3.81 GiB
1,536 dimensions: 5.72 GiB
2,048 dimensions: 7.63 GiB
4,096 dimensions: 15.26 GiB

That excludes IDs, graph edges, metadata, allocator overhead, replicas, and snapshots. Moving from 1,024 to 4,096 dimensions quadruples the raw vector payload before the vector database adds its own structures.

For Nemotron, we use a break-even equation rather than pretending open weights are free:

self_hosted_cost_per_million = gpu_hourly_cost / millions_of_tokens_per_hour

At a hypothetical $2 per GPU-hour, self-hosting must sustain more than 16.67 million tokens per hour to beat a $0.12-per-million API on raw compute. It must exceed 33.33 million tokens per hour to beat a $0.06-per-million API. Engineering labor, idle capacity, failover, deployment, and monitoring move the real threshold higher.

We did not measure Nemotron throughput, memory use, or NVFP4 quality loss, so those break-even examples are formulas, not performance claims.

For teams planning a migration or capacity model, our AI infrastructure services focus on the complete cost boundary: embedding generation, index memory, replicas, refresh cadence, and rollback—not just the token invoice.

Our Final Verdict: When to Deploy, When to Skip

We cannot honestly name a universal retrieval-quality winner from the available evidence. Nemotron 3 has the strongest retrieval-specific public numbers in this comparison, but those numbers were not generated in the same run as Voyage 4 and Cohere Embed v4. Our executed work validated the metric harness, not the three models.

That distinction defines our verdict.

Deploy Voyage 4 if:

We need managed text embeddings with low list pricing.
We want selectable dimensions and native quantized output.
A 32,000-token context covers the chunking strategy.
We can pin input_type, reject truncation, and keep a model manifest.
We have domain labels proving that Lite, standard, or Large meets recall targets.

Deploy Cohere Embed v4 if:

We need text, images, and mixed documents in one retrieval space.
A 128,000-token input window materially simplifies document handling.
We value a managed endpoint more than control over model weights.
We can absorb the API price and validate multilingual performance on our own language mix.

Deploy Nemotron 3 Embed if:

We already operate NVIDIA inference infrastructure.
Data residency or service independence makes self-hosting valuable.
Query volume is high and steady enough to amortize GPUs.
We can benchmark the exact checkpoint, precision, and dimension used in production.
We accept responsibility for capacity, upgrades, observability, and rollback.

Hold off or avoid all three as a final choice if:

We do not have labeled queries from the target domain.
We are comparing RTEB and general MTEB scores as if they were interchangeable.
We have not calculated index memory at the chosen dimension and replica count.
The ingestion pipeline allows silent truncation.
Model names, preprocessing prefixes, or revisions are not pinned.
We are inferring multilingual quality from an aggregate score.
We have not tested rate-limit and batch assumptions against the production account.

Our practical default is Voyage 4 at 1,024 dimensions for managed, text-only evaluation; Cohere Embed v4 when multimodal retrieval is a real requirement; and Nemotron 3 only when self-hosting is an architectural objective rather than a reaction to per-token pricing.

We would not approve production deployment until all candidates had run over the same 2,000-to-5,000-chunk corpus, the same graded queries, the same distance metric, and the same nDCG and recall implementation. The winning model is the one that clears the application’s recall threshold at the lowest total operating cost—not the one with the largest vector or the most flattering public benchmark.

If the index migration risk is already material, contact our engineering team before changing dimensions or model families. Rebuilding a benchmark is cheap. Rebuilding a live retrieval index under customer traffic is not.

Vector Search & Embeddings in Java: Building Semantic Search Engines

Said Olano — Fri, 02 Oct 2026 01:41:21 +0000

Vector Search & Embeddings in Java: Building Semantic Search Engines

Introduction

Traditional keyword-based search is dead. When users search for "best restaurants near me," they don't expect results matching those exact words—they expect restaurants that mean the right thing.

This is where vector search and embeddings enter the picture.

Vector search is the technology powering modern AI applications: ChatGPT's retrieval-augmented generation (RAG), Netflix's recommendation engine, Spotify's "Discover Weekly," and enterprise semantic search platforms. Yet many Java developers still think of search as Elasticsearch queries with exact terms.

This gap is costing you:

Missed user intent: Returning exact matches instead of semantic matches
Irrelevant results: Users abandon apps when search doesn't understand context
Lost competitive advantage: Your competitors' AI-powered features work better

In this guide, you'll learn:

What embeddings and vector search actually are (the math isn't complex, the concepts are elegant)
How to implement semantic search in Java using production libraries
Integration patterns with PostgreSQL (pgvector), Elasticsearch, and Pinecone
Real-world use cases from e-commerce to support chatbots
Performance tuning for millions of vectors

By the end, you'll understand why vector search is essential for modern applications and how to build it with Java.

Part 1: Understanding Embeddings and Vector Space

What Is an Embedding?

An embedding is a numerical representation of text, images, or other data. Instead of storing "restaurant recommendations," you store a vector of numbers: [0.25, -0.15, 0.82, ..., 0.41].

These aren't random numbers. They're learned through neural networks trained on massive datasets. Similar concepts produce similar vectors. This property is the entire foundation of vector search.

Example:

Query: "best pizza place"
Vector: [0.12, 0.88, -0.31, 0.45, ...]
Document: "authentic Italian pizzeria"
Vector: [0.14, 0.87, -0.29, 0.46, ...]

These vectors are close in vector space. Measuring that distance (using cosine similarity, Euclidean distance, etc.) gives you a relevance score.

How Modern Embedding Models Work

In production, you don't hand-craft embeddings. You use pre-trained embedding models:

OpenAI's text-embedding-ada-002: 1,536 dimensions, state-of-the-art for general text
Sentence Transformers (open-source): 384-768 dimensions, fast inference, great for local deployments
Cohere's embedding API: 4,096 dimensions, excellent domain-specific options
Google's Gecko Embedding: Lightweight, optimized for cost

These models map text → vector in a way that preserves semantic meaning.

Vector Space Fundamentals

All vectors live in an N-dimensional space. When you have 1,536-dimensional vectors (from OpenAI), you're working in 1,536-dimensional space.

Similarity metrics:

Cosine Similarity (most common)
- Measures angle between vectors
- Range: -1 to 1 (1 = identical direction)
- Formula: (A · B) / (||A|| × ||B||)
Euclidean Distance (L2)
- Measures straight-line distance
- Smaller = more similar
- Formula: √(Σ(ai - bi)²)
Manhattan Distance (L1)
- Sum of absolute differences
- Faster but less intuitive than Euclidean

For text search, cosine similarity is almost always the right choice.

Part 2: Building Semantic Search in Java

Architecture Overview

A semantic search system has these components:

┌─────────────────────────────────────────────┐
│  User Query                                  │
└─────────────┬───────────────────────────────┘
              │
┌─────────────▼───────────────────────────────┐
│  1. Embedding Model (Convert text → vector) │
│     (OpenAI API / Local Sentence Transformer)│
└─────────────┬───────────────────────────────┘
              │
┌─────────────▼───────────────────────────────┐
│  2. Vector Search Engine                     │
│     (Pinecone / PostgreSQL pgvector)        │
└─────────────┬───────────────────────────────┘
              │
┌─────────────▼───────────────────────────────┐
│  3. Similarity Ranking                       │
│     Return top-K most relevant results      │
└─────────────┬───────────────────────────────┘
              │
┌─────────────▼───────────────────────────────┐
│  Results with scores (0.0 - 1.0)            │
└─────────────────────────────────────────────┘

Example 1: Using OpenAI Embeddings + Pinecone

Step 1: Add Dependencies

<!-- pom.xml -->
<dependency>
    <groupId>com.theokanning.openai-gpt3-java</groupId>
    <artifactId>api</artifactId>
    <version>0.18.1</version>
</dependency>

<dependency>
    <groupId>io.pinecone</groupId>
    <artifactId>pinecone-client</artifactId>
    <version>0.1.0</version>
</dependency>

Step 2: Create Embedding Service

import com.theokanning.openai.embedding.Embedding;
import com.theokanning.openai.embedding.EmbeddingRequest;
import com.theokanning.openai.embedding.EmbeddingResult;
import com.theokanning.openai.service.OpenAiService;

public class EmbeddingService {
    private final OpenAiService openAiService;
    private final String modelId = "text-embedding-ada-002";

    public EmbeddingService(String apiKey) {
        this.openAiService = new OpenAiService(apiKey);
    }

    public List<Double> embedText(String text) {
        EmbeddingRequest request = EmbeddingRequest.builder()
            .model(modelId)
            .input(Collections.singletonList(text))
            .build();

        EmbeddingResult result = openAiService.createEmbeddings(request);
        return result.getData().get(0).getEmbedding();
    }

    public List<List<Double>> embedTexts(List<String> texts) {
        EmbeddingRequest request = EmbeddingRequest.builder()
            .model(modelId)
            .input(texts)
            .build();

        EmbeddingResult result = openAiService.createEmbeddings(request);
        return result.getData().stream()
            .sorted(Comparator.comparingInt(Embedding::getIndex))
            .map(Embedding::getEmbedding)
            .collect(Collectors.toList());
    }
}

Step 3: Pinecone Vector Search

import io.pinecone.clients.Pinecone;
import io.pinecone.clients.Index;

public class PineconeVectorStore {
    private final Index index;
    private final EmbeddingService embeddingService;

    public PineconeVectorStore(String apiKey, String projectName, 
                               String indexName, String embeddingApiKey) {
        Pinecone client = new Pinecone.Builder()
            .withApiKey(apiKey)
            .build();

        this.index = client.getIndex(projectName, indexName);
        this.embeddingService = new EmbeddingService(embeddingApiKey);
    }

    // Index documents with embeddings
    public void indexDocument(String docId, String content, 
                             Map<String, String> metadata) {
        List<Double> embedding = embeddingService.embedText(content);

        index.upsert(
            docId,
            embedding,
            metadata
        );
    }

    // Search: returns top K similar documents
    public List<SearchResult> search(String query, int topK) {
        List<Double> queryEmbedding = embeddingService.embedText(query);

        var results = index.query(queryEmbedding)
            .withTopK(topK)
            .withIncludeMetadata(true)
            .execute();

        return results.getMatches().stream()
            .map(match -> new SearchResult(
                match.getId(),
                match.getScore(),
                match.getMetadata()
            ))
            .collect(Collectors.toList());
    }
}

record SearchResult(String id, Double score, Map<String, String> metadata) {}

Step 4: End-to-End Usage

public class SemanticSearchApp {
    public static void main(String[] args) {
        PineconeVectorStore store = new PineconeVectorStore(
            System.getenv("PINECONE_API_KEY"),
            "my-project",
            "restaurant-index",
            System.getenv("OPENAI_API_KEY")
        );

        // Index documents
        store.indexDocument("rest-1", "Best pizza in town, authentic Italian, family-owned", 
            Map.of("name", "Pasta Paradise", "type", "pizza"));

        store.indexDocument("rest-2", "Fast casual ramen, Tokyo-style noodles, great broth",
            Map.of("name", "Noodle House", "type", "ramen"));

        // Search with semantic understanding
        var results = store.search("where to find good Italian food", 3);

        results.forEach(r -> 
            System.out.printf("ID: %s, Score: %.4f, Name: %s%n",
                r.id(), r.score(), r.metadata().get("name"))
        );
    }
}

Example 2: Local Vector Search with PostgreSQL + pgvector

For privacy-sensitive applications, you might want embeddings to stay within your infrastructure.

Step 1: Setup PostgreSQL pgvector

-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    embedding vector(1536),
    metadata JSONB,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Create HNSW index for fast similarity search
CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

Step 2: Java Implementation with Sentence Transformers

import org.springframework.ai.document.Document;
import org.springframework.ai.embedding.Embedding;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.jdbc.core.JdbcTemplate;

@Component
public class PostgresVectorStore {
    private final JdbcTemplate jdbcTemplate;
    private final EmbeddingModel embeddingModel;

    @Autowired
    public PostgresVectorStore(JdbcTemplate jdbcTemplate, 
                               EmbeddingModel embeddingModel) {
        this.jdbcTemplate = jdbcTemplate;
        this.embeddingModel = embeddingModel;
    }

    public void indexDocument(String content, String metadata) {
        Embedding embedding = embeddingModel.embed(content);

        String sql = "INSERT INTO documents (content, embedding, metadata) " +
                     "VALUES (?, ?::vector, ?::jsonb)";

        jdbcTemplate.update(sql, content, vectorToString(embedding), metadata);
    }

    public List<DocumentResult> similaritySearch(String query, int limit) {
        Embedding queryEmbedding = embeddingModel.embed(query);

        String sql = "SELECT id, content, metadata, " +
                     "1 - (embedding <=> ?::vector) as similarity " +
                     "FROM documents " +
                     "ORDER BY embedding <=> ?::vector " +
                     "LIMIT ?";

        return jdbcTemplate.query(sql, new Object[]{
            vectorToString(queryEmbedding),
            vectorToString(queryEmbedding),
            limit
        }, (rs, rowNum) -> new DocumentResult(
            rs.getInt("id"),
            rs.getString("content"),
            rs.getDouble("similarity"),
            rs.getString("metadata")
        ));
    }

    private String vectorToString(Embedding embedding) {
        return "[" + embedding.getOutput().stream()
            .map(String::valueOf)
            .collect(Collectors.joining(",")) + "]";
    }
}

record DocumentResult(int id, String content, double score, String metadata) {}

Part 3: Real-World Use Cases

1. E-Commerce Product Search

Traditional: Search for "comfortable shoes" → matches only products with those exact words

Semantic: Matches "ergonomic footwear," "supportive sneakers," "cushioned athletic shoes"

public class ProductSearchService {
    private final VectorStore vectorStore;

    public List<Product> findSimilarProducts(String query) {
        return vectorStore.search(query, 10).stream()
            .map(this::toProduct)
            .collect(Collectors.toList());
    }
}

2. Support Chatbot with RAG

Retrieval-Augmented Generation: Embed your knowledge base, find relevant docs, feed them to LLM

@RestController
public class SupportChatbot {
    private final VectorStore knowledgeBase;
    private final OpenAiService openAiService;

    @PostMapping("/ask")
    public ResponseEntity<String> ask(@RequestBody String question) {
        // Step 1: Find relevant docs using vector search
        var relevantDocs = knowledgeBase.search(question, 3);

        // Step 2: Build context from retrieved docs
        String context = relevantDocs.stream()
            .map(SearchResult::content)
            .collect(Collectors.joining("\n\n"));

        // Step 3: Ask LLM with context
        String prompt = String.format(
            "Based on this knowledge base:\n%s\n\nAnswer: %s",
            context, question
        );

        ChatCompletionRequest request = ChatCompletionRequest.builder()
            .model("gpt-4")
            .messages(List.of(new ChatMessage(ChatMessageRole.USER.value(), prompt)))
            .build();

        String answer = openAiService.createChatCompletion(request)
            .getChoices().get(0).getMessage().getContent();

        return ResponseEntity.ok(answer);
    }
}

3. Duplicate Detection

Find near-duplicate documents or suspicious fraud patterns

public class DuplicateDetector {
    private final VectorStore vectorStore;

    public boolean isProbablyDuplicate(String document, double threshold) {
        var similarDocs = vectorStore.search(document, 1);

        return !similarDocs.isEmpty() && 
               similarDocs.get(0).score() > threshold;
    }
}

Part 4: Performance Optimization

Batching Embeddings

Don't embed one document at a time. Batch them.

public void indexManyDocuments(List<Document> documents) {
    // ❌ Slow: N API calls
    // documents.forEach(doc -> index(doc));

    // ✅ Fast: 1 API call per batch
    Iterables.partition(documents, 100).forEach(batch -> {
        List<String> texts = batch.stream()
            .map(Document::getContent)
            .collect(Collectors.toList());

        List<List<Double>> embeddings = embeddingService.embedTexts(texts);

        for (int i = 0; i < batch.size(); i++) {
            vectorStore.index(batch.get(i).getId(), embeddings.get(i));
        }
    });
}

Caching Embeddings

Store computed embeddings to avoid redundant API calls

@Component
public class CachedEmbeddingService {
    private final EmbeddingService service;
    private final Map<String, List<Double>> cache = new ConcurrentHashMap<>();

    public List<Double> embed(String text) {
        return cache.computeIfAbsent(text, key -> service.embedText(key));
    }
}

Dimension Reduction

Not all 1,536 dimensions are necessary for your use case. PCA can reduce them:

public List<Double> reduceDimensions(List<Double> embedding, int targetDim) {
    // Use Apache Commons Math or similar
    // This trades accuracy for speed/storage
    return PCA.reduce(embedding, targetDim);
}

Part 5: Production Considerations

1. Latency Budget

Embedding API call: 50-200ms
Vector search: 10-50ms (with proper indexing)
Total for user query: <500ms

If this is too slow, cache frequent queries.

2. Cost Management

OpenAI embeddings: ~$0.02 per million tokens
Pinecone: ~$0.70 per 1M vectors/month
Self-hosted (pgvector): Your infrastructure cost

For high-volume applications, self-hosting saves money.

3. Handling Updates

When document content changes, re-embed and update:

public void updateDocument(String docId, String newContent) {
    List<Double> newEmbedding = embeddingService.embedText(newContent);
    vectorStore.update(docId, newEmbedding, newContent);
}

4. Monitoring

Track embedding quality:

@Component
public class EmbeddingQualityMonitor {
    private final MeterRegistry meterRegistry;

    public void recordSimilarityScore(double score) {
        Timer.builder("search.similarity.score")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(meterRegistry)
            .record(Duration.ofMillis((long) (score * 1000)));
    }
}

Conclusion

Vector search and embeddings are no longer bleeding-edge. They're essential for:

Better search relevance (understand user intent, not just keywords)
Semantic recommendations (similarity-based, not just collaborative filtering)
RAG-powered chatbots (ground LLMs in your data)
Anomaly detection (spot unusual patterns in vector space)

Key Takeaways:

Embeddings are numerical representations where similar concepts = close vectors
Use cosine similarity to measure relevance
Choose between managed (Pinecone, Weaviate) or self-hosted (PostgreSQL + pgvector)
Batch API calls, cache embeddings, monitor quality
Start with OpenAI embeddings; optimize to self-hosted later if cost-prohibitive

Your next semantic search implementation is just these components away. Build it today.

Beyond Generative AI: Why Streaming LLMs Fail Structured Voice, and JEV Wins

Felona Voice — Thu, 01 Oct 2026 18:42:13 +0000

The promise of conversational AI has always been grand: natural, fluid interactions that feel human. Yet, for all the advancements in large language models (LLMs), a glaring gap persists in real-time, structured voice AI. We've been trying to fit a square peg in a round hole, using auto-regressive, token-generating LLMs for tasks that demand precision, speed, and determinism. It's time to acknowledge the fundamental mismatch and embrace a new paradigm: Felona Voice, powered by Joint Embedding Vectors (JEV) and stateful conversational transition graphs (VoiceGraph).

The LLM Paradigm: A Costly Latency Trap for Structured Voice

Traditional voice agents, often built around a loop of ASR -> LLM -> TTS, face inherent limitations when applied to structured conversational flows like booking a table, checking an account balance, or routing a customer service call. Here's why the streaming LLM approach fundamentally struggles:

Crippling Latency: LLMs are auto-regressive, meaning they generate responses token by token. This process involves:
- Time To First Token (TTFT): The initial delay before any output begins (often 200-500ms).
- Token Generation Time: Subsequent tokens are generated sequentially. For a typical conversational turn, this translates to an 850ms to 1,800ms+ delay for the LLM to process input and formulate a response (even if it's just a decision for the next action). This latency breaks natural human conversation, making barge-in difficult and creating awkward silences.
Astronomical Costs at Scale: Every interaction with an LLM, no matter how simple the intent resolution, consumes tokens. At scale, this becomes a significant infrastructure cost. Imagine a contact center handling tens of thousands or hundreds of thousands of calls a month, each with multiple conversational turns. Every turn means paying for input and output tokens, even for simple 'yes/no' or intent confirmation. These costs quickly escalate into tens of thousands of dollars monthly for intent resolution alone, let alone full generative responses.
Hallucinations and Lack of Determinism: LLMs are probabilistic by nature. While fantastic for creative generation, this probabilistic approach leads to 'hallucinations' – generating plausible but incorrect or off-script responses. For enterprise applications in banking, healthcare, or critical customer support, non-deterministic behavior and the risk of generating inaccurate information are simply unacceptable. Compliance and user trust demand predictable outcomes.
Heavy Network Dependency: Relying on external cloud LLM APIs means constant network round-trips for every decision. This adds to latency and introduces points of failure, making local testing and development cumbersome without API keys.

The Felona Voice Revolution: Precision with Joint Embedding Vectors (JEV)

Felona Voice (https://github.com/mohitjoer/felona_voice) introduces a paradigm shift by recognizing that structured voice AI doesn't need generative reasoning for intent resolution. Instead, it leverages Joint Embedding Vectors (JEV) and VoiceGraph to deliver unparalleled performance:

Joint Embedding Vectors (JEV): Instead of generating text, Felona Voice embeds both the user's spoken input and the predefined actions/intents of your agent into a shared vector space. When a user speaks, their utterance is embedded, and a lightning-fast similarity search determines the closest matching action. This completely bypasses the slow, token-by-token generation process of LLMs for intent resolution.
VoiceGraph: Stateful Conversational Transition Graphs: Felona Voice uses a VoiceGraph – a deterministic, stateful graph that defines the conversational flow. JEV similarity matching then guides the agent through this graph, ensuring predictable and precise transitions. This eliminates hallucinations entirely because the agent's actions are explicitly defined and matched, not probabilistically generated.

The Result: Unprecedented Speed and Efficiency

By leveraging JEV for intent resolution, Felona Voice achieves:

Ultra-low Latency: Intent decisions are made in sub-10ms (~5ms). This is an order of magnitude faster than LLMs, enabling instant barge-in and truly natural, real-time conversational turns.
Zero Cost for Intent Resolution: JEV matching is an in-memory operation. There are no tokens to generate, no external API calls for intent. This means $0.00 per conversational turn for intent resolution.
Zero Hallucinations: With a deterministic VoiceGraph guided by precise JEV matching, your agent will always follow defined paths, ensuring compliance and reliability.
Local-First Development: Test and develop your voice agents locally without needing any external API keys for deterministic routing.

Felona Voice in Action: TypeScript Simplicity

Felona Voice is built for developers, offering a fluent TypeScript API that makes building sophisticated voice agents intuitive. Let's see how simple it is to define an agent with precise actions:

import { createAgent } from "felona-voice";

interface MyAgentContext {
  userName?: string;
  tableSize?: number;
}

const agent = createAgent<MyAgentContext>("RestaurantConcierge")
  .system("You are an intelligent voice concierge for 'The Golden Spoon' restaurant. Your primary goal is to book tables and answer FAQs.")
  .action(
    "book_table",
    "Book a restaurant reservation. I want to reserve a table. Can I get a booking?",
    async (ctx, input) => {
      // In a real app, you'd extract parameters from 'input' or prompt for them
      if (!ctx.tableSize) {
        return "Certainly! For how many people would you like to book a table?";
      }
      // Simulate booking logic
      console.log(`Booking a table for ${ctx.tableSize} people.`);
      return `Okay, I've booked a table for ${ctx.tableSize} people. Is there anything else?`;
    }
  )
  .action(
    "check_hours",
    "What are your operating hours? When are you open?",
    async () => "The Golden Spoon is open Monday to Saturday, from 5 PM to 11 PM."
  )
  .action(
    "greet_user",
    "Hello. Hi there. Good morning. Good afternoon. Good evening.",
    async (ctx) => {
      if (ctx.userName) {
        return `Welcome back, ${ctx.userName}! How can I assist you today?`;
      }
      return "Hello! Welcome to The Golden Spoon. How may I help you?";
    }
  )
  .fallback("I'm sorry, I didn't quite catch that. How can I assist you today?");

// Example interaction
async function runConversation() {
  let currentContext: MyAgentContext = {};

  let reply = await agent.interact("Hi", currentContext);
  console.log(`Agent: ${reply.response}`); // Agent: Hello! Welcome to The Golden Spoon. How may I help you?

  currentContext = { ...currentContext, ...reply.context };

  reply = await agent.interact("I'd like to book a table", currentContext);
  console.log(`Agent: ${reply.response}`); // Agent: Certainly! For how many people would you like to book a table?

  currentContext = { ...currentContext, ...reply.context };

  reply = await agent.interact("For four people", currentContext);
  currentContext.tableSize = 4; // Manually update context for this example
  reply = await agent.interact("", currentContext); // Re-interact to trigger action with updated context
  console.log(`Agent: ${reply.response}`); // Agent: Okay, I've booked a table for 4 people. Is there anything else?

  currentContext = { ...currentContext, ...reply.context };

  reply = await agent.interact("What are your hours?", currentContext);
  console.log(`Agent: ${reply.response}`); // Agent: The Golden Spoon is open Monday to Saturday, from 5 PM to 11 PM.
}

runConversation();

Felona Voice also offers pluggable audio pipelines for WebSockets, WebRTC, Deepgram, Whisper, ElevenLabs, and Cartesia, ensuring flexibility for your production environment.

The Stark Reality: Traditional Voice Agent (LLM Loop) vs. Felona Voice (JEV + VoiceGraph)

Let's put the architectural differences into perspective:

Metric	Traditional Voice Agent (LLM Loop)	Felona Voice (JEV + VoiceGraph)
Intent Decision Latency	850ms – 1,800ms	~5ms (Sub-10ms)
Inference Cost / Turn	$0.02 – $0.06+ / turn	$0.00 / turn
Hallucination Risk	High (probabilistic text tokens)	0% (deterministic transition graph)
Network Dependency	Requires constant cloud LLM API	Local/In-memory embedding matching
Developer Experience	Complex prompt engineering, slow iteration	Fluent builder API, fast local testing

The Mathematical Truth: 90-95% Cost Reduction

Let's quantify the cost savings. Consider an enterprise-level voice agent handling 50,000 calls per month, with an average of 5 conversational turns per call for intent resolution. This amounts to 250,000 intent decisions per month.

Traditional LLM Approach: At an average cost of $0.04 per turn (a conservative estimate for a mix of input/output tokens), your monthly bill for intent resolution alone would be:
250,000 turns * $0.04/turn = $10,000 per month
Felona Voice Approach: With JEV-powered intent resolution, the cost per turn is $0.00. This means your monthly bill for intent resolution is:
250,000 turns * $0.00/turn = $0 per month

This represents a 100% saving on the intent resolution component of your voice agent's operation. When you factor in the additional costs of LLM inference for generative responses (which Felona Voice can still integrate for, but only when truly needed), the overall savings on infrastructure bills can easily exceed 90-95% for systems that primarily rely on structured interactions.

This isn't just an optimization; it's a fundamental shift that makes high-volume, real-time voice AI economically viable and technically superior for structured use cases.

The Future of Structured Voice AI is Here

The era of forcing generative LLMs into every voice AI problem, regardless of suitability, is coming to an end. For structured, real-time, and mission-critical voice interactions, the architectural shortcomings of streaming LLMs are simply too great. Felona Voice offers a purpose-built, highly performant, and cost-effective alternative by leveraging Joint Embedding Vectors and VoiceGraph.

It's time to build voice agents that truly feel real-time, are reliably deterministic, and won't break the bank. Embrace the future of voice AI with Felona Voice.

🌟 Star the repository on GitHub: github.com/mohitjoer/felona_voice

📦 Install via npm: npm install felona-voice

📖 Explore full documentation: felona-voice.mohitjoe.tech/docs

This article was originally published on felona-voice.mohitjoe.tech.

Setting up vector search for github starred repos

Dennis kinuthia — Wed, 30 Sep 2026 17:37:38 +0000

Ask the starred-repo corpus questions in natural language: embed the query on-device, rank the stored vectors, and show the matching repos in the TanStack Start UI.

This is where the earlier chapters pay off. Auth (chapter 2) got us a GitHub token, the worker (chapter 3) turned stars into vectors, the bus (chapter 4) kept the UI honest while it ran, and now a search box turns a sentence into a ranked list.

One query, end to end

 "graph databases in Rust"
        |
        |  600 ms debounce, ?q= in the URL
        v
 useQuery -> Eden: GET /api/elysia/enrich/starred/search?q=...
        |
        v
 Elysia route (TypeBox: 1..2000 chars)
        |
        v
 embedQuery(text)  -- EmbeddingGemma 300M, query prompt
        |             ONNX Runtime (onnxruntime-node, CPU), Q4 weights
        v
 Float32Array(768)
        |
        v
 PGlite + pgvector:  ORDER BY embedding <=> $query  LIMIT 50
        |
        v
 rows (no vector blob) -> list of repo links

Everything below the Elysia route runs inside the desktop app's Deno process. No request leaves the machine.

The model: EmbeddingGemma

EmbeddingGemma is Google's 308M-parameter embedding model built on Gemma 3, designed for phones and laptops. The properties that matter here:

768-dimensional vectors. The size can be cut down to 512, 256, or 128 through Matryoshka representation learning. We keep 768.
2K-token context, which is roomy for a repo's metadata plus the top of its README (chapter 3).
100+ languages, so non-English READMEs and queries still land.
Small when quantized. Google quotes under 200 MB of RAM with quantization.
Fully offline once the weights are on disk.

We run the onnx-community/embeddinggemma-300m-ONNX export through @kessler/gemma-embedding, a small wrapper over Transformers.js that uses native onnxruntime-node in Node-compatible runtimes. Our own package, packages/gemma-embedding, adds a shared instance, quantization switching, download progress, and cache inspection on top.

Documents and queries are embedded differently

EmbeddingGemma is asymmetric: the text is wrapped in a different prompt depending on whether it's something to find or something to search with. From the wrapper's source:

const prefixed =
  mode === "query"
    ? `task: search result | query: ${text}`
    : `title: none | text: ${text}`;

So the worker embeds repos with embedDocument() and search embeds the user's sentence with embedQuery(). Mixing the two up still returns results, just noticeably worse ones. That's why both are exported as separate named functions rather than a mode flag callers could forget (instance.ts):

export async function embedDocument(text: string) {
  const embedding = await getServerGemmaEmbedding();
  return embedding.embed(text, "document");
}

export async function embedQuery(text: string) {
  const embedding = await getServerGemmaEmbedding();
  return embedding.embed(text, "query");
}

One shared model instance

Loading the model takes a few seconds and a few hundred MB, so there's exactly one instance per process. It's created on first use, and concurrent callers await the same load promise:

export async function getServerGemmaEmbedding(options?: GemmaEmbeddingOptions) {
  if (options?.dtype && options.dtype !== getActiveGemmaDtype()) {
    await disposeInstance(); // switching quantization: drop the old one
    setActiveDtypeInternal(options.dtype);
  }

  let instance = getEmbeddingInstance();
  if (instance?.isLoaded()) return instance;

  if (!instance) {
    const { GemmaEmbedding } = await import("@kessler/gemma-embedding");
    instance = new GemmaEmbedding(resolveServerGemmaOptions({ ...options, dtype: getActiveGemmaDtype() }));
    setEmbeddingInstance(instance);
  }

  if (!getEmbeddingLoadPromise()) setEmbeddingLoadPromise(instance.load() /* + progress + error handling */);
  await getEmbeddingLoadPromise();
  return getEmbeddingInstance()!;
}

The worker and the search route share this instance, so indexing and searching at the same time don't load the model twice.

Quantization

The ONNX export ships several weight files. Settings lets the user pick one (catalog.ts):

Variant	Download	Notes
`q4`	~197 MB	Default. Smallest, and fine for repo search
`q8`	~309 MB	Balanced
`fp16`	~618 MB	Higher quality
`fp32`	~1.2 GB	Full precision

All variants run on the CPU (device: "cpu"). GEMMA_DTYPE and GEMMA_MODEL_PATH override the choice from the environment, which is handy for pointing at a pre-downloaded model.

ONNX Runtime inside a Deno Desktop binary

onnxruntime-node is a native addon, a .node binary per OS and architecture. Bundling every platform's copy into the desktop binary would bloat it for no benefit, so the packaging step excludes it:

"desktop:build": "... deno desktop ... --exclude-unused-npm --exclude ./.output/server/node_modules/onnxruntime-node --compress ..."

At runtime, ort-runtime.ts resolves ORT in two steps. In dev, the normal node_modules copy just works. In a packaged build, it uses a copy downloaded on first run into the config dir:

export async function ensureOrtReady(): Promise<void> {
  if (await canImportOrt()) return; // bundled / dev node_modules
  if (!downloadedOrtReady()) return; // nothing downloaded yet
  ensureOrtModulePath(); // prepend ~/.config/tangerine-desktop/native/node_modules to NODE_PATH
}

The download pulls the pinned onnxruntime-node tarball (the version must match what Transformers.js expects) straight from the npm registry. It extracts the tarball, deletes the binaries for other operating systems, and adds onnxruntime-common next to it:

const tarballUrl = `https://registry.npmjs.org/onnxruntime-node/-/onnxruntime-node-${ORT_NPM_VERSION}.tgz`;
// fetch with progress -> tar -xzf -> keep bin/napi-v6/<this OS>/ -> fetch onnxruntime-common

Every embed call goes through ensureOrtReady() before importing the model package, which is why that line appears in both embedRepo() and the search helper.

First run: the bootstrap

A fresh install has neither ORT nor model weights. The first time the app opens, it fetches both in the background with a progress toast. It's modelled on an IDE's first-run downloads and is cancellable from the toast.

The server side (embedding-bootstrap.ts) runs the two downloads in order:

void (async () => {
  await ensureOrtReady();
  await beginOrtRuntimeDownload(); // ~40 MB, skipped if ORT already imports
  await awaitOrtRuntimeDownload();

  const q4 = inspectGemmaCache().variants.find((v) => v.id === "q4");
  if (q4?.ready) return;
  beginServerGemmaDtypeSwitch("q4"); // ~197 MB of weights, then load
})();

Progress is streamed with the chapter 4 SSE pattern. This source is a status snapshot rather than an event bus, so the generator polls once a second, only sends when something changed, and closes itself when the downloads finish (models/bootstrap.ts):

.get("/events", async function* ({ request }) {
  let previous: string | null = null;
  while (!request.signal.aborted) {
    const status = await getEmbeddingBootstrapStatus();
    const serialized = JSON.stringify(status);
    if (serialized !== previous) {
      previous = serialized;
      yield sse({ data: status });
    }
    if (!isLive(status)) break;
    await sleep(1000, request.signal);
  }
})

On the client, EmbeddingBootstrapHost is mounted once. It calls POST /embedding/bootstrap/start when the server says shouldAutoStart, and renders the toast with a progress bar and a Cancel button.

Storage: pgvector in PGlite

Vectors live next to the repo rows in embedded Postgres (PGlite with the pgvector extension). The column width comes from the same constant the model package exports, so the two can't drift apart (project-enrichment-outputs.ts):

// packages/gemma-embedding/src/constants.ts
export const EMBEDDING_MODEL_ID = "embeddinggemma-300m";
export const EMBEDDING_DIMENSIONS = 768;

export const projectEnrichmentOutputs = pgTable("project_enrichment_outputs", {
  id: text("id").primaryKey().default(sql`gen_random_uuid()`),
  owner: text("owner").notNull(),
  name: text("name").notNull(),
  type: text("type").$type<"starred" | "repos" | "other">(),
  description: text("description"),
  summary: text("summary"),
  payload: jsonb("payload").notNull(), // { text } = the exact document that was embedded
  modelId: text("model_id"), // which model produced `embedding`
  embedding: vector("embedding", { dimensions: EMBEDDING_DIMENSIONS }),
  embeddedAt: timestamp("embedded_at", { withTimezone: true }),
  // ...
});

Storing payload.text and modelId makes the index easy to reason about later. You can see exactly what was embedded, and if the model ever changes you know which rows to re-embed.

Search

The whole retrieval step is one function (search.ts): embed the query, then let Postgres sort by cosine distance using Drizzle's cosineDistance (pgvector's <=> operator):

export async function searchStarredByQuery(q: string): Promise<StarredSearchHit[]> {
  const text = q.trim().slice(0, EMBED_TEXT_MAX_CHARS);
  if (!text) return [];

  await ensureOrtReady();
  const { embedQuery, getServerGemmaEmbedding } = await import("@repo/gemma-embedding/node");
  await getServerGemmaEmbedding({ dtype: readGemmaPrefs().dtype });
  const vector = Array.from(await embedQuery(text));

  const distance = sql<number>`${cosineDistance(projectEnrichmentOutputs.embedding, vector)}`;

  return db
    .select({ id: projectEnrichmentOutputs.id, owner: projectEnrichmentOutputs.owner, name: projectEnrichmentOutputs.name, description: projectEnrichmentOutputs.description, /* … */ distance })
    .from(projectEnrichmentOutputs)
    .where(and(eq(projectEnrichmentOutputs.type, "starred"), isNotNull(projectEnrichmentOutputs.embedding)))
    .orderBy(distance)
    .limit(50);
}

The select lists columns explicitly and leaves out embedding, so 768 floats per row never cross into the UI.

Right now this is an exact scan: Postgres computes the distance to every starred row. For a personal star list (hundreds to a few thousand rows) that's a small amount of work and always returns the true nearest neighbours. If the corpus grows much larger, an approximate HNSW index is one custom migration. drizzle-kit can't generate USING hnsw, so it would be hand-written:

CREATE INDEX project_enrichment_outputs_embedding_hnsw
  ON project_enrichment_outputs USING hnsw (embedding vector_cosine_ops);

The route exposes the search with validation and OpenAPI docs (enrich/starred/index.ts):

.get("/search", ({ query }) => searchStarredByQuery(query.q), {
  query: t.Object({
    q: t.String({ minLength: 1, maxLength: 2000, description: "Natural-language query; embedded then ranked by cosine distance." }),
  }),
})

The UI

The starred page (EnrichedStarred.tsx) reuses the app's normal list scaffold. The search box writes ?q= to the URL after a 600 ms debounce, and a query hook calls the typed Eden client:

const q = (routeApi.useSearch().q ?? "").trim();

const semantic = useQuery({
  queryKey: ["enriched-starred-search", q],
  enabled: q.length > 0,
  placeholderData: (previous) => previous, // keep old hits on screen while the next query embeds
  queryFn: async () => {
    const { data, error } = await getElysiaTreaty().enrich.starred.search.get({ query: { q } });
    if (error) throw new Error(treatyErrorMessage(error));
    return data ?? [];
  },
});

const rows = q ? (semantic.data ?? []) : allRows;

A few small choices make it feel local rather than remote:

The placeholder hints at meaning, not keywords: "Try natural language — e.g. graph databases in Rust…".
The loading state says "Embedding query…" instead of a generic spinner, which also explains the short wait on the very first search while the model loads.
placeholderData keeps the previous results visible while you refine the query, so the list doesn't flash empty on every keystroke.
No query means the full corpus. An empty q shows every embedded star, paginated, and that list keeps growing live during indexing (chapter 4).
Each hit is a link to the in-app repo page (/$user/repos/$repo).

What we built, and what's next

The finished loop: sign in once, click Embed starred, watch repos stream into the list, and search them by meaning. It all runs on your machine with a ~200 MB model.

It's deliberately "retrieval only": the answer is a ranked list of your own repos, not generated text. That keeps it fast, obviously correct (every result is a real star), and free of a second, much larger model.

Natural next steps, roughly in order of payoff:

Filters on language, topic, or owner, as a WHERE next to the vector sort.
Hybrid ranking: blend in Postgres full-text search, so exact names ("tokio") always rank first.
Chunked READMEs for deep-content matches (see the note in chapter 3).
An HNSW index once the corpus outgrows an exact scan.
Optional local generation: pass the top hits to a small local LLM for a one-paragraph answer that cites them.

With all that said

I tried hard to keep the shipped binary small:

Nothing heavy in the bundle. ONNX Runtime and the EmbeddingGemma weights are fetched at runtime (the first-run bootstrap above), not packaged.
Lazy imports. The model package is only imported the first time something embeds.
No CEF. backend: "webview" uses the OS webview instead of bundling Chromium, which saves roughly 100 MB (chapter 1).

Even so, the Deno runtime alone is about 70 MB, and that's a floor this approach can't go below. For a tool whose job is "search my stars", that feels like a lot.

I suspect a Go shell with Wails (a single Go binary using the OS webview) could come in well under that. So I've started learning Go, and I'm looking forward to rebuilding a slice of this and writing up what I find.

Embedding Drift: Detect, Monitor & Swap Models Safely

Nainik Mehta — Mon, 28 Sep 2026 07:32:41 +0000

Why embedding drift detection matters

Embedding models (and the corpora they encode) change more often than teams expect. A new encoder, a tokenizer tweak, or even a chunking policy change can rotate, scale, or reshape your vector space. Re-embedding billions of vectors and rebuilding ANN indexes is expensive and disruptive. The good news: you don't need to reindex on every model update. Add low-cost, high-signal checks to your dashboard, defer full rebuilds with lightweight mappings, and follow a staged cutover plan.

Three cheap drift signals you can add today

These signals are fast to compute, interpretable, and often sufficient to tell you whether to act.

1) Shadow Recall@K delta

Run the candidate model in shadow for a sample of production queries for 1–3 weeks. Compare distributional changes in Recall@K (e.g., Recall@10) between prod and candidate. Instead of a single mean, surface the distribution (percentiles, tail behavior) and alert on shifts in the top-K relative performance.

Practical notes:

Keep a small labeled or synthetically derived eval set (200–2,000 queries) and a sampled live-query set.
Compare percentile deltas (p50/p90/p99) and absolute recall drop for critical cohorts (tenant, language, document type).
Use shadowing instead of traffic switching: log candidate results, return prod to users.

2) Mahalanobis distance on batches

Cosine or Euclidean deltas can be noisy for rotations and anisotropic scaling. Mahalanobis distance accounts for covariance and flags shifts in the joint distribution of embeddings.

Basic implementation sketch (Python/pseudocode):

import numpy as np
from scipy.spatial import distance

# X_ref: N x d array from 2-4 weeks of production embeddings
# X_batch: M x d array from recent production queries
mean_ref = X_ref.mean(axis=0)
cov_ref = np.cov(X_ref, rowvar=False) + 1e-6*np.eye(X_ref.shape[1])
inv_cov = np.linalg.inv(cov_ref)

# Mahalanobis distances of batch items to ref distribution
dists = [distance.mahalanobis(x, mean_ref, inv_cov) for x in X_batch]
# Aggregate metric: median or tail (95th/99th percentile)
alert_score = np.percentile(dists, 99)

Alert when the tail crosses a tuned threshold derived from your null distribution (see KS tests below).

3) KS-tests on per-dimension or score histograms

Perform Kolmogorov–Smirnov tests on per-dimension distributions or on top-1 similarity score histograms. Build a null distribution from 2–4 weeks of production traffic and only alert beyond the 99th percentile. KS tests are cheap and capture subtle distributional shifts across many dimensions when aggregated.

Practical aggregation: compute KS p-values per-dimension, then use a Bonferroni or Holm correction or treat the maximum stat as the aggregate signal. Alternatively, monitor the histogram of top-1 similarity scores and apply a KS test to detect shifts.

Two pragmatic ways to avoid full re-indexing

1) Drift-Adapter / mapping layer

Train a lightweight transformation g_theta: R^{d_new} -> R^{d_old} that maps new-model embeddings into the production space. This lets you keep the legacy ANN index and serve queries with transformed candidate embeddings.

Common parameterizations:

Orthogonal Procrustes (rigid rotation)
Low-rank affine (diagonal scaling + small matrix)
Small residual MLP when drift is partially non-linear

Train on a small paired calibration set (10k–50k pairs) sampled from your corpus. The Drift-Adapter paper and follow-up toolkits show you can recover 95–99% of Recall@10 in many upgrades while adding negligible latency.

Quick mapper training example (least-squares linear mapper):

# A_old: Nxd old embeddings, B_new: Nxd new embeddings
# Learn W to minimize ||W B_new - A_old||_F^2
import numpy as np
W = A_old.T @ np.linalg.pinv(B_new.T)  # simple closed-form
# At query time: q_mapped = (W @ q_new)

If W recovers most retrieval signal on a holdout set, deploy it as a query-time adapter and avoid touching the index immediately.

When to use: model changes that are mostly geometric (rotations, scaling, mild anisotropy) or when you need a fast, reversible bridge.

2) Fingerprint-driven reindexing and selective backfill

Attach a ModelFingerprint to each collection (model_id, provider, dims, and optionally normalization/cfg). On startup or during deployment, compare the current model's fingerprint to the stored one. Only trigger full reindex when the fingerprint meaningfully changes.

Flow:

If fingerprint matches: no bulk work. Optionally start a shadow run for monitoring.
If fingerprint differs: enqueue selective/hot-doc re-embedding (hot docs by access, or critical cohorts), train a mapper, and start a shadow evaluation.

Tiny illustrative fragment:

if new_model.fingerprint != prod_model.fingerprint:
    enqueue_partial_reindex(hot_docs)
else:
    start_shadow_run(candidate_model)

Selective re-embedding (hot-docs, cohorts, or only failing golden queries) typically re-embeds a small fraction (we’ve seen ~3% in practice) and buys time to plan a full backfill.

Safe cutover playbook (practical)

Shadow run (1–3 weeks): embed queries with both models; log Recall@K, top-1 score histograms, per-cohort metrics.
Compare distributions: use Recall@K deltas, Mahalanobis tail, and KS-tests. Gate on thresholds (e.g., no >2–3% drop on golden queries, p99 Mahalanobis below threshold).
Deploy Mapper fallback: if Drift-Adapter is in use, make it the default query transform and keep the old index.
Canary mirror traffic: mirror 1% → 10% → 100% reads to the new path (or flip alias progressively). Observe end-to-end latency and business metrics.
Holdover window: keep mapper fallback and the old index live for at least two weeks post-cutover. Continue shadowing and cohort monitoring.
Escalate to full reindex only when both fingerprint and statistical tests exceed configured thresholds and dual-index/backfill has completed.
Rollback plan: keep aliases that can atomically switch back, keep dual-write running for the rollback window, and maintain the old index until the observation window expires.

Real-world tip and example

Last quarter we shadowed a candidate for 10 days, tracked Recall@10 and Mahalanobis drift, and trained a linear Mapper. The result: avoided a full re-index, re-embedded only 3% of hot documents during a staged cutover, and observed zero visible regressions in production.

Closing: make drift detection routine, not dramatic

Embedding drift detection doesn't have to be all-or-nothing. Start with cheap signals (shadow Recall@K, Mahalanobis, KS-tests), defer full re-embedding using a Drift-Adapter plus fingerprint gating, and follow a careful shadow→canary→cutover playbook. These practices turn a weekend migration into a predictable engineering process.

What one cheap signal will you add to your dashboard this week?

Embeddings and Vector Search, Demystified

Internals Decoded — Thu, 24 Sep 2026 17:53:43 +0000

Embedding models convert your company handbook’s text chunks into high-dimensional vectors, and approximate nearest neighbor (ANN) indexes like HNSW or IVF-PQ retrieve the most semantically similar chunks in milliseconds. Cosine similarity gives this search its discriminative power by measuring angular closeness, not just keyword overlap. Together they replace rigid symbolic matching with geometric proximity.

The curse of dimensionality tells us that in high dimensions, all points look equally far apart. Yet embedding spaces, trained with contrastive objectives, actually make semantically related chunks physically close. ANN algorithms then cheat gracefully, trading perfect recall for sub-millisecond latencies while keeping over 95% of the true top-k neighbors. That mismatched intuition is the central magic, and the central engineering challenge, of vector search.

How do embeddings turn text into a geometric space?

Think of a city map where every landmark is placed according to its function, not its street address. A library, a bookstore, and a university would sit near each other even if their physical locations differ wildly. Embedding models build exactly this kind of map: they assign each text chunk a coordinate in a high-dimensional space so that “work-from-home policy” and “remote work guidelines” land near each other, and far from “lunch menu.”

A modern text embedding model is just a deterministic function, usually a transformer encoder, that ingests tokenized text and outputs a fixed-length vector. When we call model.encode(["All employees may work remotely on Fridays."]) with a model like all-MiniLM-L6-v2, we get back a 384-dimensional array of floats source. That array is the chunk’s “address” in the semantic map. Every chunk from the handbook gets its own address, and we store them all in a giant coordinate database. When a user asks “What is the remote work policy?”, we run the same encoder on the query, compute its address, and then find the database addresses that are closest.

The geometry of this map is not random. It is shaped during training by a contrastive learning objective. The model sees millions of pairs of sentences that are either semantically similar (paraphrases, answers to the same question) or unrelated. It is penalized when similar sentences end up far apart, and when unrelated ones end up close. Over time, the space becomes uniform (vectors spread evenly across the unit sphere) and aligned (angular distance accurately reflects meaning). That is why a small model can rival a large one in retrieval accuracy: the map’s structure, not its raw size, does the heavy lifting.

Our running assistant splits the handbook into chunks that fit inside the model’s maximum sequence length (usually 512 tokens) source. If we skip chunking, a document about “expense policies” that also briefly mentions remote work might produce an embedding that blends both topics, muddying the map. Good chunk boundaries, something we explored in Part 2, keep each coordinate meaningful. After we embed every chunk, we load them into an index that can answer “find the k nearest addresses to this query vector” fast. The next section unpacks what “nearest” really means under the hood.

What does cosine similarity actually measure?

Cosine similarity is the cosine of the angle between two vectors. In the embedding map, it measures how much two items point in the same conceptual direction, ignoring their individual lengths. Two handbook passages that both circle around “work-from-home eligibility” will have a small angle between them, yielding a cosine similarity close to 1. A passage about office snacks points somewhere else entirely, giving a similarity near 0 (or negative). That angle alone often outperforms Euclidean distance on semantic tasks because the scale of the vector can vary with text length or model quirks, while the direction captures the topic more robustly.

Mathematically, for unit-length vectors, cosine similarity equals the dot product. Many embedding models output normalized vectors by default, so maximizing inner product is equivalent to minimizing angular distance. For maximum inner product search (MIPS) in recommendation systems, algorithms like ScaNN explicitly adapt their pruning to that equivalence source. In our handbook assistant, we can treat cosine similarity and dot product as interchangeable as long as we normalize embeddings; then the only difference is a constant scaling that does not affect ranking.

Why does this matter? Because the handful of chunks that the ANN index returns will be ranked by this angle. If the model has placed all remote-work-themed chunks in a tight cluster, the top-3 results will all be highly relevant. But if noise or poor training makes the cluster diffuse, the closest chunks may still be about travel or food, just slightly nearer by accident. The next section explores why high dimensionality itself can make clusters diffuse, even if the model is decent.

Why does the curse of dimensionality break exact search?

In low dimensions, if you are in San Francisco, Los Angeles is far, and the nearest neighbor notion is sharp. But in a 384-dimensional space, something bizarre happens: almost all points look roughly equally far from a given query source. The contrast between the nearest and the farthest neighbor collapses; the ratio approaches 1. This is distance concentration, the core of the curse. For a brute-force scan over all chunks, this is merely a CPU (central processing unit) cost problem. But for spatial indexes like k-d trees or ball trees, which work by pruning large regions, the curse is fatal. In high dimensions, bounding volumes overlap so much that the pruner cannot rule out any branch, and performance degrades to scanning nearly everything.

Our handbook assistant has tens of thousands of chunks. A brute-force scan over 50,000 chunks of dimension 384 takes a few hundred milliseconds on a modern CPU, already too slow for a chat interaction. If the handbook grows to a million chunks, it becomes seconds. And naïve spatial indexes would be even slower. We cannot rely on exact nearest neighbor at scale; we need an algorithm that deliberately sacrifices perfect correctness to stay fast. That is where approximate nearest neighbor (ANN) enters, which we address next.

{
  "type": "stat",
  "title": "Brute force vs ANN search",
  "caption": "Scanning 50,000 chunks of dimension 384 takes hundreds of milliseconds, while an ANN index delivers sub-5 ms latencies at high recall.",
  "stats": [
    {
      "value": "300ms",
      "label": "Brute force on 50k vectors"
    },
    {
      "value": "5ms",
      "label": "ANN index (HNSW, efSearch=64)"
    },
    {
      "value": "60x",
      "label": "Speedup"
    },
    {
      "value": ">95%",
      "label": "Recall@10"
    }
  ]
}

How do approximate nearest neighbor algorithms cheat gracefully?

ANN algorithms accept that they will miss a few of the true top-k neighbors, and in exchange deliver orders-of-magnitude speedup. They are measured by recall@k: the fraction of the true top-k that actually appears in the returned set source. A typical production system targets 95-99% recall at values of k between 5 and 50.

The most widely used class of in-memory ANN indexes is the graph-based family, led by HNSW (Hierarchical Navigable Small World). HNSW builds a sparse, multi-layer proximity graph where each chunk is a node and edges point to nearby nodes source. At query time, it performs a greedy walk: start at a random entry point on the top, sparsest layer, move toward nodes that are closer to the query, drop down to denser layers, and finally collect the closest nodes at the bottom layer. The walk visits only a tiny fraction of the graph. The parameter efSearch (exploration factor) controls how many candidates are examined; larger efSearch raises recall but also latency. For our handbook assistant, efSearch=64 might give 97% recall at under a millisecond per query on 100k vectors.

Disk-backed variants like DiskANN rearrange the graph as a single dense layer optimized for SSD access, with a compression catalog (product quantization) kept in RAM (random-access memory) source. This design lets us serve billion-chunk indexes from relatively cheap commodity hardware, which is overkill for a single handbook but indispensable when the system scales to enterprise-wide document stores.

Partition-and-quantize methods, represented by IVF-PQ (inverted file with product quantization), take a different approach. A coarse quantizer (k-means) partitions the space into, say, 1024 cells. The query is first assigned to the nearest nprobe cells, and only the vectors in those cells are scored using compressed (PQ) codes. Faiss implements this efficiently in both CPU and GPU (graphics processing unit) source. For the handbook, we might store embeddings in a Faiss IVF-PQ index with nprobe=16 and achieve sub-millisecond latency even on server-grade CPU, with recall around 95%. This method shines when memory is tight, because the PQ compression shrinks each vector to a few dozen bytes instead of 1.5 KB.

graph TD
  A[Query vector] --> B[Coarse quantizer k-means]
  B --> C[Top n closest centroids]
  C --> D[Scan vectors in selected buckets]
  D --> E[Re rank by cosine similarity]
  E --> F[Return top k results]

All these algorithms expose knobs that trace a recall-vs-latency curve; the engineer’s job is to pick the index that dominates that curve for the specific workload. Once we have a fast vector retrieval layer in place, we can finally answer the central question: what does “semantic” actually buy?

What does semantic actually buy us in a RAG pipeline?

Semantic search gives us the ability to retrieve relevant chunks even when the exact words are absent. In our handbook assistant, a user might type “How do I submit a vacation request?” while the policy document says “Paid time off must be requested via HR Portal at least two weeks in advance.” A keyword engine would miss this, but an embedding of the query lands close to the embedding of that sentence because the model has learned that “vacation request” and “paid time off request” point the same way.

That directional alignment is fragile. It relies on the embedding model having been exposed to similar paraphrases during training. Fine-tuning the embedding model on company-specific language (with a contrastive objective and a set of paired questions and handbook answers) can dramatically boost recall for jargon or internal names source. Without that, the semantic map may cluster “vacation” near “holiday” but still place “vacation request” far from “PTO submission” if the training data was too general.

The interplay with chunking from Part 2 becomes acute. An overly long chunk may average out multiple topics, diluting the semantic signal and making the chunk’s vector point to a “generic” direction instead of a precise one. Conversely, a chunk that is too short may lack enough context to position itself reliably, drifting toward an unrelated cluster. The semantic promise only holds when chunk boundaries respect topic coherence, and when the embedding model captures the right distinctions. With those pieces in place, the ANN index reliably returns the most concept-aligned chunks, giving the LLM (large language model) in the RAG (retrieval-augmented generation) pipeline the right raw material to answer accurately. In the next episode, we will examine how the retrieved chunks are actually fed to the model: prompt assembly strategies and the subtle ways that ordering and truncation affect the final answer.

Quick Reference: Key Configurations and Defaults

Property	Typical Value
Common embedding dimension	384 (all-MiniLM), 768 (BERT base), 1024 (large models)
Max sequence length	512 tokens (≈ 350-400 words) for many models
Default similarity metric (normalized)	Cosine similarity
HNSW efSearch range	16-128, higher = more recall, slower
IVF-PQ nprobe range	1-64, scans that many cells
PQ compressed vector size	~8-64 bytes per vector
Typical recall target	0.95-0.99 for top-k

Test yourself

Your handbook assistant uses a 384-dim embedding model and an HNSW index with efSearch=64. The index holds 25,000 chunks. For the query “what is the remote work policy?” the top-3 returned chunks are about travel approval, office supplies, and the remote work policy itself. The assistant’s answer incorrectly claims remote work is only permitted with manager approval because it fused the travel and remote chunks. How would you diagnose and fix this?

Answer: The recall of the true relevant chunk is high but its rank is pushed down by two off-topic chunks that somehow end up closer to the query. This suggests the embedding model is not adequately separating the “remote work” concept from other corporate topics. First, inspect the embedding of the query and the offending chunks with PCA to see if they cluster along noisy dimensions. Consider fine-tuning the embedding model on a supervised dataset of (query, relevant passage) pairs built from past HR tickets. Increase chunk overlap slightly so that the remote work chunk captures more context. Finally, if precision matters more than recall, lower the k returned or apply a hard cosine similarity threshold filter after retrieval, discarding chunks below 0.6 similarity before passing them to the LLM.

Frequently Asked Questions

Q: Should I always use cosine similarity?
Cosine similarity is robust when vector magnitudes vary, and it aligns naturally with contrastively trained embeddings. If you normalize all vectors, dot product gives identical results. Use Euclidean distance when magnitude carries useful information, but in text retrieval cosine is the safer default.

Q: How do I choose the embedding dimension?
Larger dimensions can store more nuanced differences, but they increase storage, latency, and can worsen distance concentration if the model is not correspondingly expressive. Start with a compact, well-trained model like all-MiniLM-L6-v2 (384 dims) and only upgrade if retrieval quality is insufficient after trying finer chunking and domain fine-tuning.

Q: Can I update an ANN index incrementally?
Graph-based indexes like HNSW support single-vector insertion and deletion with little overhead. IVF-PQ indexes may need occasional coarse quantizer retraining if the data distribution shifts significantly. For a living handbook that changes weekly, HNSW is usually a better fit.

Q: How do attribute filters (e.g., department, date) work with ANN?
Run the vector search first to get a candidate set, then filter by attributes, but this can break recall if the filtered candidates are few. Better approaches use pre-filtering within the index (Faiss enables combining IVF with metadata filtering) or use product quantization that encodes attributes alongside the vector. Always benchmark recall with your specific filter mix.

Q: Is exact nearest neighbor search ever practical?
Exact search is viable for up to a few thousand high-dimensional vectors. Beyond that, the latency of a full scan becomes unacceptable for interactive applications. Use exact search for unit tests, debugging recall, or verifying top-k correctness of your ANN index against a gold standard.

If these deep dives into how real retrieval systems work under the hood resonate with how you build, subscribe to Internals Decoded at internalsdecoded.com. Every article unpacks the mechanics that make modern RAG, search, and ML (machine learning) infra tick.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

Your Semantic Cache Answers the Question Next Door

DevOps Daily — Thu, 24 Sep 2026 09:00:00 +0000

We replayed 288 questions to an ops assistant through a semantic cache. At a similarity threshold of 0.80 with bge-m3, the cache answered 32 percent of them from memory and cut the average response time from 8.7 seconds to 6.6. It also answered about one hit in three with the answer to a different question.

Someone asked how to undo a commit they had already pushed to main. The cache had seen "how do I undo my last commit that I have not pushed yet", decided the two were the same question, and served the answer for a local commit: git reset --soft HEAD~1. For a commit already on main the right answer is git revert; resetting it and pushing the result means force-pushing over history other people have pulled.

Raising the threshold does not fix it. Between 0.88 and 0.92 the few hits left were more often wrong than right. Across six embedding models and thresholds from 0.50 to 0.99 in steps of 0.01, the best any of them managed with no wrong answers at all was a 0.7 percent hit rate. For four of the others, the only thresholds that never served a wrong answer never served anything, and one model served a wrong answer even at 0.99.

This post covers how we measured it, why no threshold separates the two kinds of question, what the fix everyone reaches for actually does, and where a semantic cache is genuinely safe.

The-DevOps-Daily/semantic-cache-wrong-answers on GitHub

TLDR

The setup. 24 pairs of ops questions that read almost the same and need different answers: restart versus reload nginx, the staging versus the production password, a memory limit versus a memory request. Six phrasings each, 288 questions, labelled by construction, so a wrong hit is a fact and not a judge model's opinion.
The cache. Embed the question, find the nearest one already answered, serve its answer if the cosine similarity clears a threshold. Six embedding models on DigitalOcean Serverless Inference, 50 thresholds, 20 orderings of the questions.
At 0.80 with bge-m3: 32 percent hit rate, 62 right hits and 30 wrong ones on average per run of 288 questions.
From 0.88 to 0.92: more of the remaining hits were wrong than right.
The same threshold is not the same setting on another model. At 0.80, e5-large-v2 answered 88 percent of questions from the cache and 65 percent of those hits were wrong.
What it saved: on gpt-oss-120b at published prices, $0.25 per 1,000 questions without a cache and $0.18 with one at 0.80. The average got faster; the slowest 5 percent barely moved.
Where it is likely safe: near-verbatim repeats. Most of them scored higher against their original than any near-miss pair did, so a threshold set from your own near-misses should catch them. We did not replay them through the cache.
The obvious fix works, and costs more than it saves. A small model checking each hit before it was served cut wrong answers to under one per run on average, at every threshold. It also left the cache no faster than having no cache, and more expensive at every threshold tested.

Prerequisites

Node.js 20 or later. The harness has no dependencies.
A DigitalOcean Serverless Inference key. The docs cover creating one.
Patience for the embedding stage: 288 questions times six models, one request each, under a per-minute rate limit. The simulation stages after that cost no inference at all.

How a semantic cache decides

An exact-match cache only helps when the same text arrives twice, perhaps after normalising case and punctuation. A semantic cache is meant to help when the same question arrives in different words:

Turn the incoming question into an embedding.
Find the most similar question already answered.
If the similarity is above a threshold, return that question's stored answer.
Otherwise ask the model, and store the new question and answer.

The threshold is the only thing deciding whether two questions are "the same". It is one number, compared against one similarity score, and it has to work for every question the assistant will ever get. That is the part worth testing.

The questions

Operations questions come in near-miss pairs all the time. The words barely change and the right answer changes completely, often in the direction that breaks something.

Pair	Why the answer differs
restart nginx / reload nginx	A restart stops the process and can drop connections; a reload applies config gracefully
rotate the staging password / the production one	Same operation, different blast radius and procedure
memory limit / memory request	A limit causes OOM kills; a request affects scheduling
delete a pod / delete a deployment	A pod its deployment owns comes back; the deployment does not
undo a pushed commit / an unpushed one	Revert is safe on shared history; reset rewrites it
`terraform state rm` / destroy one resource	One forgets the resource, the other deletes it
clear one Redis database / all of them	`FLUSHDB` versus `FLUSHALL`
cordon a node / drain it	One stops new pods; the other evicts the running ones

There are 24 pairs like these, each side written six different ways, from "How do I restart nginx?" to "How can I bounce the nginx process completely?". Two questions count as sharing an answer if they are listed under the same intent, and a hit counts as wrong when the stored answer was written for the other intent of the pair, or for an unrelated one. No model grades it.

This is a stress test, not an estimate of how often a production cache is wrong. Every question's near-miss partner is in the stream, there are no exact repeats, and the paraphrases were written to vary their wording while the near-miss pairs keep theirs. All of that makes it hard on a cache on purpose. How many near-misses your own traffic contains is something only your logs can tell you. What this measures is what the cache does when they turn up.

Why one threshold cannot separate them

Here are two kinds of question pair, scored by bge-m3. The green curve is every pair that needs the same answer. The red dashed curve is every designated near-miss pair: the partners that read almost the same and need different answers. Unrelated pairs, the other 96 percent of the 41,328 possible, are left out.

How alike two questions look to bge-m3

Series	Samples	Min	Median	p95	Max
Same answer (paraphrases)	720	0.357	0.7224999999999999	0.86	0.939
Different answer (near-miss pairs)	864	0.372	0.6455	0.823	0.932

Cosine similarity, bge-m3. Same answer: all 720 pairs of paraphrases. Different answer: all 864 designated near-miss pairs, such as restart vs reload nginx. Unrelated pairs are not shown.

A threshold is a single vertical line through both curves. A real cache only compares a new question against what it has stored, so this is not a hit rate. But it shows the problem: no line sits to the right of the whole red curve without also sitting to the right of almost all of the green one. The two overlap across nearly their whole range.

The overlap was large for every model we tried:

Embedding model	Median, same answer	Median, near-miss	Most similar near-miss	Near-miss pairs above the median paraphrase
`bge-m3`	0.723	0.646	0.932	22.8%
`gte-large-en-v1.5`	0.771	0.707	0.958	27.1%
`qwen3-embedding-0.6b`	0.756	0.685	0.949	22.6%
`multi-qa-mpnet-base-dot-v1`	0.684	0.577	0.927	23.5%
`e5-large-v2`	0.861	0.837	0.982	36.2%
`all-mini-lm-l6-v2`	0.849	0.795	0.991	36.2%

For at least one near-miss pair in five, the two questions that need different answers look more alike to the embedding model than a typical pair of questions that need the same one. Embedding models are trained to put questions about the same topic close together, and near-miss pairs are exactly that: the same topic, one word apart.

The sweep

We replayed the 288 questions through the cache at every threshold from 0.50 to 0.99, in 20 different random orders. The order matters, because which phrasing arrives first decides what the cache stores and what later questions get matched against. Each threshold gets its own replay, since a question that hits at 0.80 is not stored, while the same question at 0.90 misses and is.

Right and wrong cache hits for bge-m3, by threshold

	0.70	0.71	0.72	0.73	0.74	0.75	0.76	0.77	0.78	0.79	0.80	0.81	0.82	0.83	0.84	0.85	0.86	0.87	0.88	0.89	0.90	0.91	0.92	0.93	0.94	0.95
Right hits	117.3	117.8	114.9	109.9	105.3	98.9	90.2	86.1	80.7	72.7	62.2	58.6	50.4	40.9	35.5	27.4	22.1	16.9	10.2	6.8	4	3	3	1	0	0
Wrong hits	76.1	71.2	65.8	61.9	55.9	53.1	47.6	41.7	38.6	35	29.9	28.6	26.1	21.6	20	16.4	15.1	12.7	13.5	13.3	10	5	4	1	0	0

Mean per run of 288 questions, over 20 orderings. A wrong hit is a stored answer served for a question with a different intended answer.

At the low end the cache hits often and is wrong often. As the threshold rises, right hits fall away faster than wrong ones, because the near-miss pairs share their wording and the paraphrases, on purpose, do not. From 0.88 to 0.92, the few hits that remained were more often wrong than right. At 0.93 there was one of each, and above that none at all.

The same number means something different on each model. At 0.80:

Embedding model	Hit rate	Right hits	Wrong hits	Share of hits that were right
`bge-m3`	32.0%	62.2	29.9	68%
`gte-large-en-v1.5`	52.7%	100.8	50.9	66%
`qwen3-embedding-0.6b`	44.8%	83.4	45.7	65%
`multi-qa-mpnet-base-dot-v1`	28.1%	56.6	24.4	70%
`e5-large-v2`	88.0%	88.8	164.6	35%
`all-mini-lm-l6-v2`	76.5%	93.4	126.9	42%

A threshold is only meaningful next to the model that produced the scores. Advice like "use 0.85" without naming the embedding model is not advice.

What a wrong hit looks like

This is npm run show-wrong-answers, which replays one ordering and prints every question that was answered with another question's answer:

EMBED_MODEL=bge-m3 THRESHOLD=0.85 npm run show-wrong-answers

# two of the 18 wrong answers in this run, excerpted
$ EMBED_MODEL=bge-m3 THRESHOLD=0.85 npm run show-wrong-answers
bge-m3, threshold 0.85, ordering 0: 46 hits, 18 wrong

asked How do I undo a commit that I already pushed to main?
matched How do I undo my last commit that I have not pushed yet? (similarity 0.861, its near-miss partner)
served ```
{% endraw %}
bash
      # Undo the last local commit but keep the changes staged
      git reset --soft HEAD~1
      # Undo the last local commit and keep the changes in your working tree (unstaged)

asked Reload nginx configuration on Ubuntu
matched Restart nginx on Ubuntu (similarity 0.854, its near-miss partner)
served
{% raw %}
 ```bash
      # Restart using systemd (Ubuntu 16.04+)
      sudo systemctl restart nginx
      # Verify it’s running

For 12 of the pairs we marked which answer does more damage when it is served for the other question. Those labels are judgment calls and they are in the repo so anyone can argue with them. At 0.90 on bge-m3, which is a high setting for that model:

bge-m3 at 0.90: 10.0 near-miss wrong answers per run, 2.5 of them the more destructive answer
    asked rotate-db-password-production got rotate-db-password-staging in 17 of 20 runs (21 times)
    asked cdn-purge-one got cdn-purge-all in 12 of 20 runs (12 times)
    asked ssh-rotate-user-key got ssh-rotate-host-key in 10 of 20 runs (10 times)
    asked docker-stop got docker-kill in 6 of 20 runs (6 times)

Someone asking how to rotate the production database password got the answer written for staging in 17 of the 20 runs. The recorded answers show why that matters: none of the six staging answers mentions avoiding downtime, and three of the six production answers do.

What it actually saves

Every question now pays for an embedding call, hit or miss. Every miss still pays for the full answer. These figures are built from API timings and token counts measured on DigitalOcean one question at a time, then replayed through the cache; they are not timings from a deployed cache:

	Mean per question	p50	p95	Cost per 1,000 questions
No cache	8.66 s	8.33 s	14.53 s	$0.2534
bge-m3 at 0.70	3.48 s	0.53 s	13.14 s	$0.0871
bge-m3 at 0.80	6.56 s	7.03 s	14.54 s	$0.1775
bge-m3 at 0.90	8.74 s	8.69 s	15.05 s	$0.2414
bge-m3 at 0.95 (nothing hits)	9.17 s	8.89 s	15.05 s	$0.2537

Three things stand out.

The money is small. On gpt-oss-120b at DigitalOcean's published prices, answering 1,000 of these questions costs about 25 cents. At 0.80 the cache saves 7.6 cents of that, and serves about 104 wrong answers per 1,000 questions to do it. Embedding every question cost a fraction of a cent across the whole run; the cache's overhead is time, not money.

The average improves, the tail barely does. A miss now costs an embedding call plus the answer. At 0.80, p95 was about 14.5 seconds with the cache and without it. It falls only at lower thresholds: at 0.70, p95 was 13.1 seconds, and 39 percent of the hits were wrong.

A cache that never hits is pure overhead. At 0.95 and above, nothing hit and every question paid about half a second extra for its embedding.

The fix everyone reaches for

If the embedding cannot tell restart from reload, ask a model that can. Before serving a hit, send both questions to a small, cheap model with one instruction: would exactly the same answer be correct for both? Serve the hit only if it says yes.

We ran that with gpt-oss-20b as the verifier, in front of bge-m3, over the same 20 orderings. Every candidate hit was checked; 1,162 distinct question pairs in all.

Threshold	Without verifier: hit rate	Wrong	With verifier: hit rate	Wrong	Right hits it rejected	Mean per question	Cost per 1,000
0.70	67.2%	76.1	40.1%	0.3	29.6	8.57 s	$0.2773
0.75	52.8%	53.1	32.0%	0.5	21.9	8.59 s	$0.2679
0.80	32.0%	29.9	21.2%	0.5	9.9	8.70 s	$0.2577
0.85	15.2%	16.4	9.8%	0.5	3.0	8.91 s	$0.2540
0.90	4.9%	10.0	1.2%	0.5	1.0	9.22 s	$0.2569
no cache						8.66 s	$0.2534

It works. Wrong answers dropped from as many as 76 per run to under one on average, and never more than two in any run, at every threshold we tried. At 0.70 the verified cache still served 40 percent of questions from memory. The only two pairs it let through were both memory request against memory limit. It erred the other way far more: it turned down 146 of the 701 genuine paraphrase pairs it was shown, which is where the lost hits went.

And it defeats the point. The mean response time with the verifier, 8.6 seconds at 0.70, is the same as with no cache at all. Each check took 3.2 seconds at the median, and it runs on every candidate hit, including the ones it then rejects, which also pay for a fresh answer. The cost went up too: $0.28 per 1,000 questions at 0.70, against $0.25 with no cache.

The reason is in the token counts. The verifier replies with one word, same or different, and is billed for a median of 283 completion tokens to produce it, because it is a reasoning model and reasons first. The answers it was saving had a median of 324. At the smaller model's lower price, a median check still cost about 57 percent of a median answer, and it runs more often than there are hits to save.

A small non-reasoning model could be faster and cheaper per check. We did not test one, and it would have to be just as reliable on exactly the pairs where the embedding models failed.

Where a semantic cache is safe

The corpus above is built from real paraphrases: different words, same question. Real traffic also contains plenty of near-verbatim repeats, the same question again with trivial differences. We made one of those for every question: lower case, no question mark, and sometimes a prefix such as "hey," or "quick one:", and compared how similar each question is to its own repeat against how similar the near-miss pairs get.

Embedding model	Lowest-scoring repeat	Most similar near-miss	Repeats scoring above every near-miss
`bge-m3`	0.856	0.932	244 of 288
`gte-large-en-v1.5`	0.882	0.958	258 of 288
`qwen3-embedding-0.6b`	0.848	0.949	210 of 288
`multi-qa-mpnet-base-dot-v1`	0.866	0.927	274 of 288
`e5-large-v2`	0.945	0.982	186 of 288
`all-mini-lm-l6-v2`	0.924	0.991	225 of 288

For every model, most trivial repeats scored higher against their own original than any near-miss pair in the corpus scored against each other. That suggests a threshold set just above your worst near-miss would catch most such repeats without serving a near-miss. We did not replay the repeats through the cache, and a repeat could still score higher against some other question, so treat it as a likely saving rather than a measured one. It is also much closer to an exact-match cache than to the semantic one people have in mind. Case, punctuation and a greeting are exactly what a normalised exact-match key would catch, with no embedding call. Normalisation has risks of its own and we did not measure that comparison, but it is the baseline a semantic cache has to beat for this kind of repeat. And the high threshold only works if you have measured your own near-miss pairs to find where "just above" is.

What we could not conclude

How often near-misses happen in real traffic. This corpus puts every question's near-miss partner in the stream, which is a deliberately difficult case for a cache. The hit rates and wrong-answer counts depend on how repetitive your traffic is and how many near-misses it contains, and only your own logs can say.
Whether every counted wrong answer was actually wrong. Hits are scored against our intent labels, not by reading each answer, and the labels are not perfect. "Empty Redis database 2 only" sits in the same intent as "clear only the current database", though it needs one more command. We read the damaging examples in this post by hand; we did not grade all of them.
Other cache designs. This is one global cosine threshold over the question alone, with nearest-neighbour lookup. Partitioning the cache by environment or resource, or only caching some kinds of answer, would behave differently, and we did not test them.
Anything beyond ops questions in English. 24 pairs, one domain, one language.
How much of this is the embedding models and how much is the corpus. We wrote the paraphrases to vary their wording and the near-miss pairs to share theirs. A corpus with closer paraphrases could give higher hit rates; we did not test one.
Statistical intervals. The 20 orderings are replays of one fixed set of questions, not independent samples, so the ranges we report are spread across orderings, not confidence intervals.
Vector database latency. The in-memory search measured 1 to 3 ms. A vector store across the network would add its own round trip, which we did not measure.
Whether a wrong answer would have been acted on. We counted wrong answers served, not wrong actions taken.

What we would do

Build your own near-miss set before you pick a threshold. Write down the questions your users ask that differ by one word and need different answers. Score them with your embedding model. Your threshold has to sit above the most similar of those pairs, and that number is specific to the model and to your domain.
Treat a threshold as a property of a model. Changing the embedding model without re-measuring changes what the cache does, sometimes from mostly right to mostly wrong at the same setting.
Keep answers that act on things out of the cache. An explanation of what a readiness probe is can be served twice. A command to run against production should not come from a question that merely looked similar.
Put the key facts in the key. The expensive mistakes here were a word apart: staging or production, one database or all of them, pushed or not. If the environment, the resource and the verb are part of the cache key, the similarity search never gets the chance to confuse them.
Measure the whole trip. The embedding call is paid on every question, and it makes every miss slower. Count it before deciding the cache saves anything.
If you add a verifier, price it. Checking each hit with a model made the cache correct here, but no faster and dearer than no cache, because the verifier reasoned its way to a one-word answer. Measure it on your own near-miss pairs and count its tokens before you ship it.

The harness, the 288 labelled questions, every embedding and answer, and the scripts that reproduce every number here are in the repo.

Originally published at devops-daily.com.

Our task categorizer routed 46% of tasks. A $0.04/MTok decision model routes 97%.

Marcell Havlik — Tue, 22 Sep 2026 09:54:08 +0000

Cosine similarity between a task title and a group description is not a classifier.

Bench, cases, raw verdicts: backend/bench/categorization in the TODOforAI repo. Total
cost ~$0.50.

Setup

TODOforAI boards have groups (SEO, Paid, Frontend…); new tasks get a suggested
group. V1: embed title and group name + description, nearest cosine wins. Embedder
Qwen/Qwen3-Embedding-4B via DeepInfra, 512 dims, unit-normalized, query instruction
"Given a task, retrieve the workstream whose purpose best matches the task". Abstain if
best < 0.50 or margin to runner-up < 0.06.

100% precision on 18 hand cases. Most of my real board in Unsorted.

Judges

Same 138 titles, same 8 groups, same description text, no example tasks — exactly what
production sees. Titles written like real todos: short, half Hungarian, 15 nonsense
(call mom, asdf), 5 ambiguous.

embedding — production, unchanged
jev — TypeSafe Jev, non-autoregressive decision model on Vercel AI Gateway. One choice question, 8 groups + none. $0.04/MTok in, output free.
sonnet — Claude Sonnet 4.6, temp 0, one slug or none
opus — Claude Opus 5, same prompt. The truth. My labels are one more opinion.

Trap: Opus 5 reasons before answering; max_tokens: 20 looked like abstaining on 88/138.
Give it 400.

Numbers

Judge	agrees with Opus	wrong group	missed	138 tasks
embedding	77/138	1	60/112	8 s
jev	130/138	5	3/112	24 s
sonnet 4.6	131/138	7	0/112	57 s

Opus routed 112, called none on 26 (15 nonsense + 11 vague). Embedding: wrong once,
missed 54%. Jev: 97% routed, 5 wrong. Sonnet's 7 wrong are coin flips with Opus.
On my real board Jev and Opus agree 21/26.

Why the embedding abstains

Not the margin:

                         agree   wrong   missed
margin 0.06  floor 0.50    76      2       60     ← production
margin 0     floor 0.50    79     41       18
margin 0     floor 0       76     62        0

Drop the gate and Unsorted becomes wrong group. Top-1 is wrong on ~45% of tasks.

bun test flaky on CI          frontend 0.540   development 0.537
call mom                      email 0.540      plg 0.525
asdf                          development 0.663  frontend 0.659

Scores sit in 0.42–0.72; asdf outscores a real task, so no floor finds nonsense.
Frontend/Development, Paid/SEO, Enterprise/Paid overlap, so real tasks sit inside any
margin. Jev and the LLMs know outcomes: a CI flake is Development because of what fixing
it achieves. Similarity can't say that.

What shipped

Jev routes; embedding is fallback when the gateway is unset or down. Batches of 25,
none explicit, accept probability ≥ 0.5. ~0.3 s per task.

Not Sonnet: 57 s vs 24 s and ~100× the price, for a suggestion 400 ms after you stop typing.

Router	per 1,000 tasks	per 1M tasks
embedding (~15 tok, $0.01/MTok)	$0.0002	$0.15
jev (~260 tok catalog per question, $0.04/MTok in, out free)	$0.01	$10
sonnet 4.6 (~350 in / 10 out)	$1.20	$1,200
opus 5 (~350 in / ~200 out with reasoning)	$7	$7,000

List prices, 8 groups. Jev costs ~70× the embedding and routes twice as much.

Jev's 5 wrong are channel-vs-outcome (sponsor a newsletter, measure signups → email,
not paid). Confidence is lower when wrong (0.70 vs 0.94), but floor 0.6 trades 2 wrong
for 7 Unsorted. We keep the 5.

Takeaways

Precision alone lies. Print recall against something you trust.
Score against a strong model, not your own labels.
Check the reference's output budget. Truncated looks cautious.

138 cases is small. It's enough to see a 54% recall gap; that's the only claim.

Embeddings explained for people who write code

VLAD — Fri, 18 Sep 2026 22:08:01 +0000

Type "how do I reset my password" into a good search box, and it finds a page titled "account recovery." Zero words in common. Nothing matched on text. So how did it know they mean the same thing?

Your app turned meaning into numbers — and once meaning is numbers, a computer can measure it. That trick is called an embedding. Get this one idea and half the AI buzzwords — "vector search," "cosine similarity," "vector database" — stop being scary.

Prefer to watch? Full walkthrough with the meaning-space animation:

An embedding is a list of numbers

Start with the machine itself. You hand it a piece of text — a word, a sentence, a whole paragraph. It hands back a list of numbers. That list is the embedding. Same text in, same numbers out, every time.

How long is the list? For a common model — OpenAI's text-embedding-3-small — it's 1,536 numbers. That sounds like a lot, until you think of each number as a coordinate.

Two numbers place a point on a map. Three place it in a room. 1,536 place it in a space you can't picture — but the math works exactly the same as the map.

Meaning becomes position

Here's the whole point of that space: the model places text so that similar meaning lands in a similar spot. "cat" and "dog" end up as neighbors. "car" ends up far away.

Nobody wrote that rule. The model learned it — it read a mountain of text and noticed which words keep the same company. Words used the same way get pushed together; words used differently get pushed apart. This is an old idea from linguistics called the distributional hypothesis: a word's meaning is shaped by the words it usually appears next to. Meaning, in this space, is just where you land relative to everything else.

"Cosine similarity" is just an angle

Now the real question in search: are these two things close? You embed the question, you embed every document, and you grab the points nearest the question. But "nearest" means one specific thing — and it's simpler than it sounds.

Draw an arrow from the center of the space out to each point.

Two texts with similar meaning? Their arrows point almost the same way — a small angle between them.
Unrelated texts? The arrows point off in different directions — a wide angle.

That angle is cosine similarity. A value near 1 means the arrows point the same way (very similar). A value near 0 means they're unrelated. That's the whole comparison — "cosine similarity" is just measuring the angle between two arrows.

In code, it's three lines

Every "AI search" feature you've used is basically this, under a nicer name:

const doc   = embed("account recovery steps")      // → [0.02, -0.91, …] · 1536 numbers
const query = embed("how do I reset my password?")

// cosine: 1 = same direction, 0 = unrelated
const score = cosine(query, doc)                   // ≈ 0.86 — close

Embed your text, embed your query, compare them, keep the closest. (Those scores are illustrative — the real point is high = same direction.)

Trap 1: the numbers only make sense inside one model

These coordinates are only meaningful within a single model. A vector from model A and a vector from model B are not comparable — it's gibberish. Same coordinates, different maps.

So embed your query and your documents with the exact same model, always. Change the model, and you have to re-embed everything you're searching over.

Trap 2: embeddings capture meaning, not exact strings

Embeddings are great at meaning — which makes them bad at things that have no meaning. An error code. A product ID. SKU-4417. There's nothing to place on the meaning-map; it's just an exact string, and pure vector search fumbles it, because nothing is "close in meaning" to a serial number.

That's why real systems run both: keyword search to catch exact strings, vector search to catch meaning, and the results merged. If you've built RAG and watched it miss an obvious error code, this is usually why.

The takeaway

An embedding, start to finish:

Text becomes a list of numbers.
The numbers are coordinates in a meaning-space.
Close in meaning → close in space.
"Similarity" is just the angle between two arrows.

Once you see it as numbers on a map, the buzzwords fall away — and you can actually debug your search instead of trusting it. When a result looks wrong, you're not staring at magic; you're asking a concrete question about distance on a map.

What's the weirdest match your vector search ever returned — the one that made no sense at all? Drop it in the comments — I read them.

I make Vlad's Stack — how the tools you use every day actually work, for people who write code. Full video walkthrough is above.

DEV Community: embeddings

Day 11: Embeddings and Vector Representations — How Machines Map Meaning

1. The mathematical map

2. Semantic closeness

3. The power of distance

Coming Up Next

VectorEmbeddings #MachineLearning #NLP #DataEngineering #AIArchitecture

Scrape Plus Embed Pipeline: End-to-End Support Knowledge Trust Boundaries

What decision are we recording?

Which boundary belongs to which system?

What does a scrape plus embed pipeline look like end to end?

Why reject a single-step ingestion promise?

References

Grew the Collection from 5 Entries to 741 Real Chunks

Voyage AI vs Cohere Embed v4 vs Nemotron 3 Embed: Production RAG Trade-offs

Why We Evaluated These Embedding Models

Hands-On Walkthrough: Setup, Execution & Output

Benchmark Limitations and Production Risks

Scale, Latency & Cost vs. Alternatives

Our Final Verdict: When to Deploy, When to Skip

Vector Search & Embeddings in Java: Building Semantic Search Engines

Vector Search & Embeddings in Java: Building Semantic Search Engines

Introduction

Part 1: Understanding Embeddings and Vector Space

What Is an Embedding?

How Modern Embedding Models Work

Vector Space Fundamentals

Part 2: Building Semantic Search in Java

Architecture Overview

Example 1: Using OpenAI Embeddings + Pinecone

Example 2: Local Vector Search with PostgreSQL + pgvector

Part 3: Real-World Use Cases

1. E-Commerce Product Search

2. Support Chatbot with RAG

3. Duplicate Detection

Part 4: Performance Optimization

Batching Embeddings

Caching Embeddings

Dimension Reduction

Part 5: Production Considerations

1. Latency Budget

2. Cost Management

3. Handling Updates

4. Monitoring

Conclusion

Further Reading

Beyond Generative AI: Why Streaming LLMs Fail Structured Voice, and JEV Wins

The LLM Paradigm: A Costly Latency Trap for Structured Voice

The Felona Voice Revolution: Precision with Joint Embedding Vectors (JEV)

The Result: Unprecedented Speed and Efficiency

Felona Voice in Action: TypeScript Simplicity

The Stark Reality: Traditional Voice Agent (LLM Loop) vs. Felona Voice (JEV + VoiceGraph)

The Mathematical Truth: 90-95% Cost Reduction

The Future of Structured Voice AI is Here

Setting up vector search for github starred repos

One query, end to end

The model: EmbeddingGemma

Documents and queries are embedded differently

One shared model instance

Quantization

ONNX Runtime inside a Deno Desktop binary

First run: the bootstrap

Storage: pgvector in PGlite

Search

The UI

What we built, and what's next

With all that said

Embedding Drift: Detect, Monitor & Swap Models Safely

Why embedding drift detection matters

Three cheap drift signals you can add today

1) Shadow Recall@K delta

2) Mahalanobis distance on batches

3) KS-tests on per-dimension or score histograms

Two pragmatic ways to avoid full re-indexing

1) Drift-Adapter / mapping layer

2) Fingerprint-driven reindexing and selective backfill

Safe cutover playbook (practical)

Real-world tip and example

Closing: make drift detection routine, not dramatic

Embeddings and Vector Search, Demystified