👉 Full index:
Global Fix Map README
Why embeddings pipelines keep breaking
If you’ve worked with vector search or semantic retrieval, you’ve probably hit this:
the embeddings look fine, the index builds without errors, but search results come back empty or irrelevant.
It’s not because FAISS, pgvector, or Milvus are “broken.”
It’s because the pipeline contracts drift silently:
- Normalization skipped during ingestion.
- Tokenization rules diverge between ingestion and query.
- Casing treated inconsistently across environments.
- Chunk overlaps misaligned.
- Embedding dimensions silently change after a model upgrade.
On dashboards, everything looks “green.” In reality, the vectors no longer live in the same space.
Common failure modes
- Normalization gaps — raw vs. normalized vectors mixed in the same store.
- Casing drift — uppercase vs. lowercase text creates different embeddings.
- Tokenizer mismatch — ingestion uses one tokenizer, query uses another.
- Overlapping chunks — off-by-one errors duplicate or skip parts of text.
- Silent dimension shift — embedding size changes (e.g. 1536 → 3072) without index rebuild.
What’s actually breaking
These aren’t one-off bugs. They’re systematic mismatches:
- Retrieval assumes normalized embeddings → ingestion skipped it.
- Queries lowercased → stored vectors weren’t.
- Tokenizers evolve silently between library versions.
- Different stride/window logic causes missing spans.
- New embedding model doubles the dimension size but index schema isn’t updated.
Result: the math collapses. Cosine similarity and recall degrade silently.
Minimal fixes
To stabilize an embeddings pipeline, enforce guardrails before generation:
- Normalize always: L2 normalize at both ingestion and query.
- Casing contract: freeze casing rules (lowercase everything, or not).
- Tokenizer lock: pin tokenizer version, verify checksum at runtime.
- Chunk contract: assert identical stride + window size across pipelines.
- Dimension guard: validate embedding size matches index schema, fail fast if not.
Acceptance targets
- Cosine similarity drift (raw vs. normalized) ≤ 0.02.
- Duplicate/missing chunk rate ≤ 1% across corpus.
- Tokenizer checksum drift = 0 across environments.
- Dim mismatch detection = 100% before index build.
How to use
- Open the Global Fix Map README.
- Go to Embeddings Pipeline section.
- Apply the minimal fix checklist.
- Validate against the acceptance targets above.
Next Episode (6): Multi-agent orchestration — why agents deadlock, overwrite each other’s memory, or spin forever.
Top comments (0)