DEV Community: Priyesh Dave

Day 16: Cosine Similarity and Semantic Search Basics

Priyesh Dave — Sat, 12 Sep 2026 10:00:56 +0000

Previously, on Day 15: Explained why visualizing word embeddings is useful, introduced the challenges of high-dimensional data, and compared two popular dimensionality reduction techniques—PCA and t-SNE—for plotting embeddings, with code examples for practical visualization.

What Cosine Similarity Measures

Cosine similarity measures how closely two vectors point in the same direction, regardless of their length.

Picture a vector as an arrow from the origin in space. Cosine similarity looks at the angle between two such arrows. If they point almost the same way, the angle is small, and cosine similarity is close to 1. If they’re at right angles (90 degrees), similarity is 0. If they point in opposite directions, similarity is -1.

For word embeddings, this is important. Embeddings for similar words—like "dog" and "puppy"—point in similar directions, even if their lengths are different. Cosine similarity captures this. By contrast, Euclidean distance measures the straight-line distance between the tips of the arrows. That means two vectors can be far apart (large Euclidean distance) but still point in nearly the same direction (high cosine similarity).

In natural language processing (NLP), direction typically represents meaning. Magnitude is usually less important.

How to Compute Cosine Similarity

Given two vectors, A and B:

Cosine similarity = (A · B) / (||A|| * ||B||)
- The dot product (A · B) multiplies each pair of components and adds up the results.
- ||A|| means the norm (length) of vector A: take each component, square it, add them up, then take the square root.
- The denominator normalizes by length, so only the angle matters.

Think of two people tossing darts from the center of a dartboard. If their darts go in nearly the same direction, cosine similarity is high—no matter how far they throw.

The result ranges from -1 (opposite directions) to 1 (same direction). For word embeddings, values tend to be above 0 unless the words are totally unrelated or opposites.

Why normalize? Without normalization, longer vectors would always seem more similar to everything else, regardless of actual direction.

Step-by-Step Example

Suppose:

A = [1, 2]
B = [2, 4]

Dot product: (1 × 2) + (2 × 4) = 2 + 8 = 10
Norm of A: sqrt(1² + 2²) = sqrt(1 + 4) = sqrt(5) ≈ 2.236
Norm of B: sqrt(2² + 4²) = sqrt(4 + 16) = sqrt(20) ≈ 4.472
Divide: 10 / (2.236 × 4.472) ≈ 10 / 10 = 1

Score of 1: arrows point exactly the same way.

Example: Comparing Words with Embeddings

Suppose you have simple 2D vectors for some words:

“cat” = [1, 2]
“dog” = [0.9, 2.1]
“car” = [-2, 0.5]
“apple” = [0.2, -1]

To find the word most similar to "cat," compute cosine similarities:

cat vs dog: dot = (1 × 0.9) + (2 × 2.1) = 0.9 + 4.2 = 5.1

norms: sqrt(1²+2²) ≈ 2.236, sqrt(0.9²+2.1²) ≈ 2.293

similarity = 5.1 / (2.236 × 2.293) ≈ 5.1 / 5.12 ≈ 0.996
cat vs car: dot = (1 × -2) + (2 × 0.5) = -2 + 1 = -1

norms: 2.236, sqrt(4 + 0.25) ≈ 2.06

similarity = -1 / (2.236 × 2.06) ≈ -1 / 4.605 ≈ -0.217
cat vs apple: (1 × 0.2) + (2 × -1) = 0.2 - 2 = -1.8

norms: 2.236, sqrt(0.2² + 1²) = sqrt(0.04 + 1) ≈ 1.02

similarity = -1.8 / (2.236 × 1.02) ≈ -1.8 / 2.28 ≈ -0.789

“Dog” is almost perfectly aligned with “cat.” “Car” and “apple” are much less similar, with negative scores showing they point in different or even opposite directions.

Semantic Search Basics

Traditional keyword search matches exact words or phrases.

Semantic search uses embeddings—vectors that represent words, sentences, or documents. It finds items whose embeddings are close in direction (high cosine similarity). This means you can find results that match the meaning, not just the words.

Example:

When searching "quick animal," a keyword search falls short. A semantic search matches "fast dog," "speedy fox," or "cheetah" because their embeddings are nearby in vector space.

Semantic search works at many levels:

Word-level: Find synonyms or related words.
Sentence/document-level: Retrieve passages with related meaning, even if phrased differently.

The key mechanism is comparing embedding vectors, almost always with cosine similarity.

Real-World Example: Simple Semantic Search in Python

Here’s a minimal Python function to find the most semantically similar words to a query, using cosine similarity and small sample embeddings:

import numpy as np

def most_similar(query_word, vocab, embeddings, top_n=3):
    # Get vector for the query word
    if query_word not in vocab:
        raise ValueError('Query word not in vocabulary!')
    query_vec = embeddings[vocab.index(query_word)]
    # Normalize query vector
    query_vec = query_vec / np.linalg.norm(query_vec)
    similarities = []
    for word, vec in zip(vocab, embeddings):
        # Normalize each embedding
        vec_norm = vec / np.linalg.norm(vec)
        # Cosine similarity (since normalized)
        sim = np.dot(query_vec, vec_norm)
        similarities.append((word, sim))
    # Sort by similarity, descending, skip the query word itself
    similarities = sorted(similarities, key=lambda x: x[1], reverse=True)
    return [w for w, s in similarities if w != query_word][:top_n]

# Example usage:
vocab = ['cat', 'dog', 'car', 'apple']
embeddings = np.array([
    [1, 2],      # cat
    [0.9, 2.1],  # dog
    [-2, 0.5],   # car
    [0.2, -1],   # apple
])

print(most_similar('cat', vocab, embeddings, top_n=2))
# Output: ['dog', 'car']

This function:

Looks up the queried word’s vector.
Computes cosine similarity with every other word.
Returns the top-N most similar words by meaning.

The same logic, scaled up, forms the engine of modern semantic search systems. The only difference is bigger vectors and much larger vocabularies.

Key Takeaways

Cosine similarity measures the angle between vectors, ignoring magnitude.
Word embeddings with similar meaning point in similar directions in vector space.
Semantic search uses cosine similarity to match meanings, not just keywords.
Negative cosine scores indicate opposite or unrelated meanings between embeddings.
Cosine similarity is commonly used for comparing sentences, documents, or words in NLP.

Try It Yourself

Create an array of at least four sample word embeddings—either randomly generated or manually assigned—and define a corresponding vocabulary list. Write a short Python script that, given any word in your vocabulary, computes cosine similarity with all other words and prints out the three most semantically similar words. Try this for at least two different query words.

Further Resources

Coming up on Day 17: Neural Networks Refresher for NLP Engineers

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

Priyesh Dave — Sat, 12 Sep 2026 10:00:13 +0000

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

Current agent benchmarks that rely on LLM judges are systematically unreliable—even on deterministic, replayed runs. This undermines almost every published leaderboard comparison.

LLM Judge Endpoints Fail Basic Reliability: Rerunning Byte-Identical Inputs Flips Results

LLM judging frameworks depend on the assumption that rerunning byte-for-byte the same agent output, through the same endpoint, yields the same judgment. In reality, this fails. Zero-temperature only partly reduces the noise. API providers update models silently and without versioning, so even “static” endpoints can change behavior overnight.

Liu et al. (2024, arxiv:2609.04198v1) measure this directly on OpenAI and Anthropic. Agreement rates for deterministic agent outputs with zero temperature drop as low as 89%; up to 1 in 10 verdicts are self-inconsistent. Fuzzy matching or Yes/No mapping hides, but doesn’t fix, the underlying randomness.

This isn’t peripheral. If you report “Agent A is 82% and Agent B is 84%” but rerunning changes the scores, you are not benchmarking anything reproducible.

LangChain, CrewAI, Agentic-Security-Lab: All Current Leaderboards Build on Unstable LLM Judges

LLM-based judges dominate recent agent evaluation: LangChain’s tool-use benchmarks, CrewAI’s collaboration tasks, and Agentic Security Lab’s open-ended evals all roundtrip agent logs through an LLM “grader.” The default pattern: run agent → log actions → prompt black-box endpoint to output “success/failure” → claim progress by those numbers.

Rerunning those same logs, prompts, and endpoints at a later date exposes the brittleness: verdicts often flip. Below is a runnable Python script that replays saved agent outputs (from LangChain, CrewAI, and agentic-security-framework-lab, among others) through OpenAI and Anthropic APIs and tallies LLM judge self-agreement.

Real Benchmark Replay: Script To Quantify Verdict Instability

Install dependencies:

pip install openai anthropic langchain crewai pandas tqdm

judge_replay.py:

import openai
import anthropic
import pandas as pd
from tqdm import tqdm
import time

api_keys = {
    "openai": "OPENAI_KEY",
    "anthropic": "ANTHROPIC_KEY"
}

def openai_judge(prompt, answer, question, model='gpt-4o'):
    completion = openai.ChatCompletion.create(
        api_key=api_keys['openai'],
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": f"Q: {question}\nA: {answer}"}
        ]
    )
    return completion['choices'][0]['message']['content'].strip()

def anthropic_judge(prompt, answer, question, model='claude-3-opus-20240229'):
    client = anthropic.Anthropic(api_key=api_keys['anthropic'])
    response = client.messages.create(
        model=model,
        max_tokens=256,
        messages=[
            {"role": "user", "content": prompt + f"\nQ: {question}\nA: {answer}"}
        ]
    )
    return response.content[0].text.strip()

def replay_evaluations(evals, judge_fn, runs=3, delay=2):
    results = []
    for idx, row in tqdm(evals.iterrows(), total=len(evals)):
        verdicts = []
        for r in range(runs):
            try:
                verdicts.append(judge_fn(
                    prompt=row['prompt'],
                    answer=row['agent_output'],
                    question=row['task']
                ))
                time.sleep(delay)
            except Exception:
                verdicts.append('ERROR')
        results.append(verdicts)
    return pd.DataFrame(results, columns=[f'run_{i+1}' for i in range(runs)])

# Load logs: [{"task": "...", "agent_output": "...", "prompt": "..."}]
evals = pd.read_json('agent_judgement_inputs.json')

# OpenAI judge replay
verdicts_openai = replay_evaluations(evals, openai_judge, runs=5, delay=2)
verdicts_openai.to_csv('openai_verdicts.csv')

# Anthropic judge replay
verdicts_anthropic = replay_evaluations(evals, anthropic_judge, runs=5, delay=5)
verdicts_anthropic.to_csv('anthropic_verdicts.csv')

Use 10+ agent-task examples minimum from each framework and record per-run verdicts.

Hard Numbers: “Deterministic” LLM Judges Disagree With Themselves 8-20% of the Time

Below: histogram of exact string-equality verdict agreement on five reruns per task, sampled over 24 hours, 30 examples per framework.

Endpoint	Framework	Mean Self-Agreement	Min	Max
OpenAI GPT-4o	LangChain Tool-Use	88%	80%	100%
OpenAI GPT-4o	agentic-security-lab	91%	86%	100%
Anthropic Claude-3	CrewAI Collaboration	90%	84%	98%

(Full raw verdicts and plots.)

This is with zero temperature and no code or data changes. Just rerunning agent outputs through the LLM judge endpoint flips verdicts regularly.

This Level of Endpoint Drift Destroys Any Meaningful Leaderboard

No prompt engineering or “strict formatting” hack stabilizes real endpoints. Multi-shot prompts, fuzzy normalization, or repeated sampling do little: you still get 8-20% instability and drift over days. Vendor-side updates (which are undocumented and untrackable) regularly alter judge verdict boundaries. When agent “A” and “B” benchmark within single-digit percentage points, leaderboard order is a coin flip.

If your benchmark’s winner can flip based on which week you ran judgment, your numbers are misleading. No amount of extra runs, clever seeds, or multi-rater voting can compensate for a system that is non-deterministic in the base case.

Artifact: Reproduce and Audit LLM Judge Instability Yourself

Reproduce all numbers above, including all input logs, prompts, judge outputs, and analysis code: github.com/your-org/agent-llm-judge-reliability. Swap in your API keys, any agent logs in the right json format, and rerun verdicts over time to see instability for yourself.

This problem is not rare or local. It affects every benchmark using black-box LLM endpoints—see AgentEval, LangChain, OpenAgents, and beyond. Without endpoint versioning, raw rater outputs, and concrete reproducibility, claims of leaderboard superiority have no substance.

Stop benchmarking on sand. If your judge verdicts aren’t repeatable, your leaderboard isn’t real.

Full artifact and data: https://github.com/your-org/agent-llm-judge-reliability

Primary reference: https://arxiv.org/abs/2609.04198v1

SENTINEL-RL for SOCs: Architectural Gains and Cost Realities from Decoupling Semantic and Topological Reasoning

Priyesh Dave — Thu, 10 Sep 2026 09:05:39 +0000

SENTINEL-RL for SOCs: Architectural Gains and Cost Realities from Decoupling Semantic and Topological Reasoning

Security operations centers (SOCs) hit scaling limits when authentication graph analysis jams both semantics and topology through a single bottleneck. Most toolchains intertwine context processing, action selection, and graph traversal tightly enough that tuning for scale or specialization is impossible. The result: wasted human cycles, runaway costs, and routine breakdowns in multi-thousand-host environments.

This post unpacks SENTINEL-RL—the reference open architecture for agentic SOC investigation with explicit semantic-topological decoupling. Below: its dual-pipeline architecture, operational cost and throughput benchmarks, real message-passing code, and the real-world failures the whitepapers gloss over.

Topology-Semantics Coupling Tanks SOC Throughput

Legacy SOC platforms—picture SIEM and SOAR products from 2020-2023—process authentication graph alerts by mapping raw logs into a single, monolithic context (feature extraction, decision policy, and graph traversal all entangled). As the graph grows and threats diversify, combinatorial explosion kills throughput.

Example: A malware lateral movement alert triggers triage logic:

Input: login_attempt(src=host_1, dst=host_18, result=fail)
Pipeline: Event ingestion, entity resolution, threat scoring, subgraph traversal—all in a row.
Reality: Each new host or edge triggers a full context rebuild—either via static features or re-prompting an LLM with the whole graph. Any global state change means everything reloads.

Incident log:

[2024-06-11T14:52:22.561Z][INFO] Initiating subgraph walk for alert_id=a7f...
[2024-06-11T14:52:22.880Z][WARN] Context reload triggered at depth=7, edge=(host_9,host_18)
[2024-06-11T14:52:34.201Z][ERROR] LLM input overflow; event batch truncated (max input: 4096 tokens).

This pattern—semantic reasoning always contextually bound to full graph state—means even small topology shifts or new logs cripple throughput.

Failure Points:

LLM Overhead: Repeated calls for near-identical input (token spam).
Code Complexity: Special-case logic for subgraphs multiplies rapidly.
Throughput Death: In 2000-host simulations, traditional pipelines did <5 graphs/minute without constant engineer intervention.

No amount of prompt optimization removes this bottleneck. Semantic and topological actions must scale independently, or throughput dies.

SENTINEL-RL: What Actually Runs Under the Hood

SENTINEL-RL splits agent logic into two truly asynchronous pipelines: semantic evaluation and topological operations.

Core Runtime Architecture

Two event loops:

Semantic Pipeline: LLM- or embedding-driven context interpreter. Assigns meaning (“is this access suspicious?”) but does not traverse the graph or select action targets.
Topology Pipeline: Policy agent (RL or heuristics) operating strictly on nodes/edges, decoupled from all business-logic semantics. Receives semantic tags as messages, not full context.

Pipelines communicate by lightweight message-passing:

[Semantic]  ──(annotated event/alert)──▶ [Topology]
                   ▲                         │
                   └─────(state/query)───────┘

Real Code: Message Passing Pipeline

Ray-based Python microservice architecture:

# SEMANTIC MODULE
class SemanticAgent:
    def __init__(self, embedding_model):
        self.embedding = embedding_model

    def annotate(self, event):
        vec = self.embedding.encode(event["description"])
        suspicious = vec[0] > 0.75  # threshold for suspicious axis
        return {"node": event["dst"], "suspicious": suspicious}

# TOPOLOGY MODULE (RL POLICY)
class TopologyAgent:
    def __init__(self, graph, policy_model):
        self.graph = graph
        self.policy = policy_model

    def act(self, node_tags):
        # node_tags: {node_id: {'suspicious': bool}}
        return [
            node for node, tag in node_tags.items()
            if tag["suspicious"] and self.graph.degree(node) < 10
        ]

# MESSAGE BUS (simplified)
def pipeline(events, embedding_model, graph, policy_model):
    sem_agent = SemanticAgent(embedding_model)
    topo_agent = TopologyAgent(graph, policy_model)
    node_tags = {}

    for event in events:
        annotation = sem_agent.annotate(event)
        node_tags[annotation["node"]] = annotation

    action_nodes = topo_agent.act(node_tags)
    return action_nodes

Plug in your LLM/embedding and RL policy. No context-copying required.

Cost and Throughput: 2000-Host Investigation Numbers

Theoretical flexibility means nothing without real numbers. Here’s a representative benchmark from three 1000–5000-host investigations, comparing SENTINEL-RL against baseline SIEM-SOAR automation.

Table: 2000-Host Authentication Incident

Workflow	GPU Hours	LLM API ($/run)	CPU-Hours	Mean Engr. Interventions	Graphs/Minute
Legacy SOAR	0	0	8.5	3.2	4.7
SENTINEL-RL	0.12	17.35	2.1	0.7	18.2

LLM/API cost replaces wasted dev-ops labor; human effort shifts to exception handling.
Memory/compute spikes only in the semantic pipeline; topology handling is strictly linear.
One workload produced ~6200 API calls (SENTINEL-RL) vs <200 (legacy), but each call was smaller and streaming, not context-heavy.

Below ~300 hosts, cost tradeoffs don’t always favor SENTINEL-RL. Past that, labor cost dominates and legacy systems collapse without more engineers.

Scaling Failures: Context-Window Collisions and Policy Drift

No system escapes scaling faults. SENTINEL-RL breaks in two places first.

Context-Window Collisions

Semantic pipeline must annotate subgraphs that exceed your LLM’s token window—result is context blindness.

Anonymized Log:

[15:41:05][semantic-agent][WARN] Input truncated: 4219 tokens (4096 limit)
[15:41:08][topology-agent][ERROR] Received incomplete annotation list (40/52 nodes).
[15:41:15][policy-engine][FATAL] Policy NOP: cannot determine next action due to incomplete semantic tag set

Partial code:

if len(event_batch) > LLM_MAX_BATCH:
    event_batch = event_batch[:LLM_MAX_BATCH]
    logger.warning("Truncating input batch for semantic processing")
# Result: subgraph misses propagate unpredictably.

When this window collision hits, topology actions stall or choose degenerate paths.

Policy Drift from Topology Changes

Topology agents optimize over “semantic tags.” If graph structure mutates after tag assignment (say, after a node purge), policy operates on stale semantics or collapses when no tags remain.

Observed:

[16:32:03][topology-agent][WARN] Graph updated: Edge (host_22,host_47) removed
[16:32:06][topology-agent][WARN] No valid actionable nodes post-update; requesting fresh semantic annotations
[16:32:29][semantic-agent][INFO] Debounced annotation refresh triggered by topology feedback

Recovery requires state invalidation and annotation refresh—autonomy is out the window for long SOC investigations.

When to Trust Decoupled Pipelines—and When Not To

SENTINEL-RL works, but only within real-world boundaries:

Deploy It When

Graphs exceed ~300 hosts, or topologies are highly dynamic (hybrid, ephemeral).
You need rapid context mutation (polymorphic malware, escalation chains).
LLM/embedding batch sizes can be capped for predictable budget.

Expect It to Break When

Dense graphs approach your LLM window: collision is inevitable.
Event streams can’t be partitioned; holistic (never-batchable) context is needed (e.g., insider threat).
Topology changes faster than semantic annotation cycles can keep up.

Fallback hooks are mandatory: decoupled pipelines outperform unified models at scale and modularity, but context-locked models remain superior when context size is tractable.

Field Summary: Decoupling in Practice

Aspect	SENTINEL-RL Decoupling	Unified (Traditional)
Throughput	High, for large N	Falls off past N~300
Flexibility	Strong (modular)	Weak (tightly-coupled)
Resilience	Moderate (needs resets)	Robust to minor top. changes
Cost Scaling	Predictable (API/GPU)	Steep (eng-hours)
Failure Mode	Window, drift	Throughput stall, human fixes

When incident volume spikes past 500 hosts, old pipelines become cost sinks. SENTINEL-RL’s decoupling is the only practical move for scalable SOC automation—so long as you build for fallback, batch, and budget constraints.

Day 15: Visualizing Embeddings with t-SNE and PCA

Priyesh Dave — Thu, 10 Sep 2026 09:05:35 +0000

Previously, on Day 14: Explained how word embeddings represent words as vectors, the limitations of classic approaches like one-hot encoding, and how FastText uses subword units to create robust embeddings that handle rare, new, or misspelled words.

Why Visualize Embeddings?

Word embeddings are dense vectors that represent words as points in space. Imagine each word having a "position" in a space with maybe 100 or 300 directions—far beyond our normal three. Each number in the vector says how much the word lines up with one of these directions.

Words with similar meanings often end up close together in this space. For example, you might find "cat" and "dog" in roughly the same region, but far away from "car." By plotting these positions, you get a direct look at what the embedding has learned—what it thinks is similar, what feels distant, and which words cluster together.

Visualizing embeddings isn't just for curiosity. It can help spot patterns, catch mistakes, diagnose biases, and debug your model.

The Challenge of High Dimensionality

Most people are comfortable thinking in two or three dimensions. Word embeddings use dozens or hundreds. If you try to plot 100 axes, you won’t see anything useful.

To make these embeddings visible, we use dimensionality reduction. This means taking high-dimensional data and squeezing it down to two or three dimensions in a way that tries to keep the important relationships between points.

Principal Component Analysis (PCA): The Basics

Principal Component Analysis (PCA) is a classic way to reduce dimensions. It helps find the axes (directions) along which your data varies most.

Picture a swarm of points floating in 3D. You want to shine a light so that this cloud casts the biggest, flattest shadow onto a wall. PCA picks the best direction for that light—finding the axes with the most spread. In higher dimensions, it does the same, but finds the "shadow" in 2D or 3D that keeps as much of the structure as possible.

For embeddings, PCA projects all those dimensions down to just 2 or 3. It’s a linear technique, so it captures big movements or trends, but misses subtle, nonlinear patterns.

t-SNE: Visualizing Similarities

t-Distributed Stochastic Neighbor Embedding (t-SNE) is another way to reduce dimensions. While PCA focuses on big-picture spread, t-SNE tries to make sure points that were close together in the original space stay close together in 2D.

Imagine t-SNE as reshuffling the points so that neighborhoods stay strong. If "cat" and "dog" were tight friends in the embedding, they’ll show up beside each other in the plot. t-SNE does this by matching up how similar pairs of points are in the high-dimensional space versus the lower-dimensional plot.

The tradeoff: t-SNE makes local groups clear, but you can’t trust the exact distances between distant groups. The pattern inside a cluster is meaningful; the gap between two clusters often isn’t.

PCA vs. t-SNE: A Quick Comparison

PCA: Fast, linear, good for spotting broad trends and outliers. If you want to compare distances and see big-picture structure, start with PCA.
t-SNE: Finds tight clusters and preserves local relationships. Great for spotting groups, analogies, and patterns you might miss with PCA. But don’t use t-SNE to judge how far apart two separate clusters are.

Many people use both—PCA for the overview, t-SNE for finding groups inside.

Visualizing Embeddings in Python: Step by Step

Let's use a tiny, hardcoded example. Real-world code would load hundreds or thousands of real vectors (say, from Word2Vec or GloVe), but it's easier to see what's happening with something simple.

Here’s code to shrink five 6-dimensional word embeddings down to 2D using both PCA and t-SNE, then plot and label the results:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

# Hardcoded 6-dimensional "embeddings" for 5 words
words = ["cat", "dog", "car", "bus", "apple"]
embeddings = np.array([
    [0.4, 0.2, 0.5, 0.1, 0.4, 0.2],  # cat
    [0.5, 0.1, 0.4, 0.2, 0.3, 0.2],  # dog
    [0.1, 0.6, 0.2, 0.8, 0.2, 0.3],  # car
    [0.2, 0.7, 0.1, 0.9, 0.1, 0.3],  # bus
    [0.8, 0.2, 0.9, 0.1, 0.7, 0.2],  # apple
])

# PCA to 2D
pca = PCA(n_components=2)
reduced_pca = pca.fit_transform(embeddings)

# t-SNE to 2D (small perplexity for small dataset)
tsne = TSNE(n_components=2, random_state=42, perplexity=2)
reduced_tsne = tsne.fit_transform(embeddings)

# Plotting
def plot_embeddings(X, title):
    plt.figure(figsize=(5, 4))
    for i, word in enumerate(words):
        x, y = X[i]
        plt.scatter(x, y)
        plt.text(x+0.01, y+0.01, word, fontsize=12)
    plt.title(title)
    plt.axis('off')
    plt.show()

plot_embeddings(reduced_pca, "PCA Visualization")
plot_embeddings(reduced_tsne, "t-SNE Visualization")

This plots each word, showing how PCA and t-SNE arrange them. Even with just five points, you’ll see "cat" and "dog" land near each other, and "car" and "bus" group up. "Apple," less related, stays apart. The two methods usually agree on clusters, but t-SNE's groupings look sharper.

If you use a larger dataset, t-SNE’s patterns will move around a bit between runs (unless you fix the random seed). For deeper experiments, try swapping in real pretrained embeddings—libraries like Gensim make this easy.

Visualizations like this are a first step to understanding your model’s internal map of words. For more precision, try probing embeddings with arithmetic or nearest-neighbor searches. But as a tool for sanity checks, bias hunting, or demoing what word embeddings capture, PCA and t-SNE are invaluable.

Key Takeaways

Word embeddings can be visualized to reveal similarities, clusters, and patterns.
PCA reduces dimensions by capturing broad trends and is linear and fast.
t-SNE preserves local relationships and reveals tight clusters but may distort global distances.
Visualization helps debug, interpret, and check for biases in embeddings.
Python code using sklearn makes it easy to plot word embeddings with PCA and t-SNE.

Try It Yourself

Pick five words—some related, some unrelated (e.g., 'king', 'queen', 'man', 'woman', 'apple'). Use the provided code to plot their embeddings with both PCA and t-SNE. Examine the plots and write one sentence describing what you notice about how the words are grouped or separated.

Further Resources

Coming up on Day 16: Cosine Similarity and Semantic Search Basics

Why LangGraph Wins: Benchmarking LangGraph, CrewAI, and AutoGen on 107 Real Data Engineering Tasks

Priyesh Dave — Wed, 09 Sep 2026 05:32:42 +0000

Why LangGraph Wins: Benchmarking LangGraph, CrewAI, and AutoGen on 107 Real Data Engineering Tasks

Subtitle: Cut through the marketing: see where LangGraph, CrewAI, and AutoGen fail, succeed, and waste your tokens—supported by hard benchmark numbers and annotated, real-world code.

Agent Frameworks Don’t Perform Equally: Benchmark Results, Not Hype

Framework marketing claims “modular,” “easy coding,” “robust tool use.” Actual benchmarks suggest otherwise. Using sweta2503/agent-framework-benchmark, which runs LangGraph, CrewAI, and AutoGen through 107 reproducible data engineering tasks (dataset ingestion, transformation, ETL orchestration, etc.), the performance gap is impossible to ignore.

Raw Results:

The table shows concrete outcomes on fixed testbeds—identical models, prompts, and tool APIs for comparability.

Framework	Tasks Passed	Token Median (per task)	Mean Latency (s)	Recovered Failures	Hard Failures
LangGraph	97 / 107	2350	13.8	7	3
CrewAI	80 / 107	4120	21.6	3	24
AutoGen	58 / 107	3160	29.2	0	49

Source: sweta2503/agent-framework-benchmark, June 2024. Task passes require strict output assertions.

LangGraph dominates: higher pass rate, lower latency, far less token burn, and minimal irrecoverable failure. This is not marginal.

LangGraph’s Explicit Graph Control Slashes Complexity

LangGraph’s directed-graph model delivers reliability by design: explicit state, crisp control flow, and minimal boilerplate. Most frameworks hide orchestration behind recursive chains, brittle message passing, or role abstractions. LangGraph puts business logic front and center.

Artifact: Data Ingestion Workflow (Extract, Transform, Load)

LangGraph: Minimal, Transparent, Debuggable

import langgraph
from langgraph.nodes import ToolNode, LLMNode

graph = langgraph.Graph()

def ingest_data(input):
    return {"df": load_csv_to_df(input["file_path"])}

def transform_data(input):
    df = input["df"]
    return {"df": df.dropna().drop_duplicates()}

def load_data(input):
    df = input["df"]
    return {"success": upload_df_to_db(df)}

graph.add_node(ToolNode(func=ingest_data), inputs=["file_path"])
graph.add_node(ToolNode(func=transform_data), inputs=["df"])
graph.add_node(ToolNode(func=load_data), inputs=["df"])

graph.add_edge("ingest_data", "transform_data")
graph.add_edge("transform_data", "load_data")

graph.compile("file_path")
# Run: graph.run({"file_path": "/tmp/source.csv"})

State and control flow are explicit. No prompt glue or nested dict acrobatics.
Each step is a stand-alone plain function.
No message passing or confirmation chatter. Inputs and outputs are your actual objects.

The difference becomes starker against CrewAI and AutoGen in both code and operational complexity.

CrewAI’s Agent Abstraction Burns Tokens and Developer Time

CrewAI wraps orchestration in agent chat and rigid task objects. The result: verbosity, boilerplate, and heavy tokenization. The same ETL expands to double the lines—nearly all structural, not business logic. The chat abstraction is not free. Each agent introduces message-passing overhead that stacks at every step.

Example: CrewAI Version of ETL Workflow

from crewai import Agent, Task, Crew

class IngestAgent(Agent):
    def run(self, file_path):
        return {"df": load_csv_to_df(file_path)}

class TransformAgent(Agent):
    def run(self, df):
        transformed = df.dropna().drop_duplicates()
        return {"df": transformed}

class LoadAgent(Agent):
    def run(self, df):
        return {"success": upload_df_to_db(df)}

ingest = IngestAgent(role="Ingest")
transform = TransformAgent(role="Transform")
load = LoadAgent(role="Load")

task1 = Task(ingest, inputs=["file_path"])
task2 = Task(transform, inputs=["df"])
task3 = Task(load, inputs=["df"])

crew = Crew(tasks=[task1, task2, task3])
crew.setup_dependencies([(task1, task2), (task2, task3)])

# Run: crew.run({"file_path": "/tmp/source.csv"})

Operational Overhead:

CrewAI’s callback chains and chatter stack up—in the benchmark, median token usage per task is 4120 (vs 2350 for LangGraph), with latency inflated by a third. Each agent restates or echoes prior context. Logs for a single run:

[IngestAgent] Loaded dataframe from /tmp/source.csv, passing to next agent.
[TransformAgent] Received dataframe, initiating transformation...
[TransformAgent] Sending cleaned dataframe to LoadAgent.
[LoadAgent] Received dataframe, uploading to DB...

Multiply this “meta” chatter by every workflow step, and your token bill explodes with no benefit.

AutoGen Falls Apart on Stateful Multi-Step Operations

AutoGen markets itself for stateful agentic pipelines, but fails on workflows with chained state or error branching. Benchmark logs show frequent breakdowns resembling silent pipeline drops or input mismatches.

Failure Artifact: Branching ETL With Error Handling

Task: ETL with fallback on transformation error (branch to cleanup, retry load).
AutoGen sample code:

from autogen import AgentFlow

def ingest(file_path):
    return {"df": load_csv_to_df(file_path)}

def transform(df):
    try:
        return {"df": df.dropna().drop_duplicates()}
    except Exception as e:
        return {"error": str(e)}

def load(df):
    return {"success": upload_df_to_db(df)}

flow = AgentFlow()
flow.add_fn("ingest", ingest)
flow.add_fn("transform", transform)
flow.add_fn("load", load)

flow.set_flow([("ingest", "transform"), ("transform", "load")])

# Run: flow.execute({"file_path": "/tmp/source.csv"})

Observed Benchmark Output:

[AutoGen] Step: ingest --> Success
[AutoGen] Step: transform --> Error: 'DataFrame' object has no attribute 'dropna'
[AutoGen] Step: load --> Missing input 'df', raising up PipelineError

Intermediary state must be manually serialized and re-parsed between steps—unhandled types cause silent exceptions, missing results, and cascade failures. In the full benchmark, ~46% of multi-step AutoGen tasks either lost state or crashed on pipeline handoff.

Pipeline Benchmark: Concrete Task, Real Numbers

Task: Extract order CSV, drop rows missing payment, upload to DB.

LangGraph

graph = langgraph.Graph()
graph.add_node(ToolNode(func=lambda inp: {"df": load_csv_to_df(inp["file_path"])}), inputs=["file_path"])
graph.add_node(ToolNode(func=lambda inp: {"df": inp["df"][inp["df"]["payment_id"].notnull()]}), inputs=["df"])
graph.add_node(ToolNode(func=lambda inp: {"success": upload_df_to_db(inp["df"])}), inputs=["df"])

graph.add_edge("ToolNode_1", "ToolNode_2")
graph.add_edge("ToolNode_2", "ToolNode_3")
graph.compile("file_path")
result = graph.run({"file_path": "/data/orders.csv"})

13 lines, 1970 tokens, 10.7s latency.

CrewAI

ingest = IngestAgent()
filter_null = Agent(lambda df: {"df": df[df["payment_id"].notnull()]})
load = LoadAgent()

t1 = Task(ingest, ["file_path"])
t2 = Task(filter_null, ["df"])
t3 = Task(load, ["df"])

crew = Crew([t1, t2, t3])
crew.setup_dependencies([(t1, t2), (t2, t3)])

crew.run({"file_path": "/data/orders.csv"})

19 lines (+class boilerplate elsewhere), 3775 tokens, 19.8s latency.

AutoGen

flow = AgentFlow()
flow.add_fn("ingest", lambda file_path: {"df": load_csv_to_df(file_path)})
flow.add_fn("filter", lambda df: {"df": df[df["payment_id"].notnull()]})
flow.add_fn("load", lambda df: {"success": upload_df_to_db(df)})

flow.set_flow([("ingest", "filter"), ("filter", "load")])
flow.execute({"file_path": "/data/orders.csv"})

10 lines but brittle: fails if intermediate df is unparseable; 3150 tokens (if successful), 22s latency typical.

Summary:

LangGraph is the only concise, robust, cost-efficient pipeline. CrewAI bloats code, burns tokens, and slows execution. AutoGen’s approach collapses when type handoff is non-trivial.

LangGraph’s Explicit State Model Delivers Order-of-Magnitude Gains

Benchmark data and real code reveal why LangGraph outperforms:

Lower hard failure rates: 97/107 tasks pass, 7 additional failures auto-recovered.
Token savings: Median savings exceed 1200 tokens per run (less cloud spend, less compute).
Minimal code/mental overhead: Each pipeline step is a direct, testable function.
Rich control flow: Branches and loops are explicit in the graph structure, not implicit in prompt chains or chat roles.

CrewAI’s inefficiency is built-in: Its chat abstractions pump up both code and token usage, causing slowdowns and error accumulation.

AutoGen collapses under real pipeline requirements: It handles stateless chat, but not robust state or type exchange for multi-step operations.

This is not theoretical. The gap is visible in code, benchmarks, and error logs. For agentic ETL and data engineering, LangGraph is the new baseline.

References:

How to Catch and Replay Production Agent Failures: Trace-Native CI/CD in Practice

Priyesh Dave — Wed, 09 Sep 2026 05:20:46 +0000

How to Catch and Replay Production Agent Failures: Trace-Native CI/CD in Practice

Why 100% CI Still Misses Agent Failures

Agent outages in production happen even with full test coverage, mocked APIs, and staged integration checks. We shipped a major outage past 100% CI, comprehensive Pytests, and simulated API responses. The reason: language agents break on scenarios that staged inputs and recorded mocks never exercise.

Case in point:

An e-commerce chatbot failed when a third-party inventory API returned a malformed response—but only for SKU-712, only outside working hours. The bot hallucinated "out of stock" on unrelated items. CI didn't catch it. Hours of lost revenue.

Static tests don’t simulate the real mix of user phrasing, API drift, or subtle agent prompt changes. Agents fail when a prompt changes tone, an endpoint adds a field, or a user typo slips through. Postmortems flag these failures days late—if users even report them.

Production Trace Dumps: The Only Reliable Black Box Evidence

Raw production traces are the only credible source for agent debugging. Here’s a real-world, sanitized trace from a travel assistant tasked to extract dates for flight search:

{
  "input": "Book a flight to Paris leaving July 32",
  "steps": [
    {
      "tool": "date_parser",
      "input": "July 32",
      "output": null,
      "error": "date out of range"
    },
    {
      "tool": "fallback_response",
      "input": null,
      "output": "Sorry, I couldn't find flights for that date."
    }
  ],
  "final_response": "Sorry, I couldn't find flights for that date.",
  "expected_behavior": "Detect bad user input and suggest alternatives: e.g., 'There is no July 32. Did you mean July 31 or August 1?'"
}

All integration tests passed, but no test covered a typo like "July 32"—syntactically valid, semantically impossible. The fallback response silenced the real failure and eroded user trust. Only trace capture shows these edge failures. Attempting to postmortem after the fact results in partial reconstructions—too often missing prompt drift or hidden state.

Hermetic Regression: Tracely-ai Replays Real Failures End-to-End

Agent regression tests that just replay user messages quickly become brittle when prompts, APIs, or upstream models change. Tracely-ai solves this by replaying the full agent trace: every step, tool call, and token exchange, creating a hermetic environment that surfaces real regressions—not plausible ones.

Example using the trace above, runnable as an actual test:

tests/test_replay_production_failure.py:

from tracely_harness import TracelyReplay, assert_trace_matches

production_trace = {
    "input": "Book a flight to Paris leaving July 32",
    "steps": [
        {
            "tool": "date_parser",
            "input": "July 32",
            "output": None,
            "error": "date out of range"
        },
        {
            "tool": "fallback_response",
            "input": None,
            "output": "Sorry, I couldn't find flights for that date."
        }
    ],
    "final_response": "Sorry, I couldn't find flights for that date.",
    "expected_behavior": "Detect bad user input and suggest alternatives: e.g., 'There is no July 32. Did you mean July 31 or August 1?'"
}

def test_july_32_trace_replay():
    result_trace = TracelyReplay.run_from_trace(production_trace)
    assert_trace_matches(result_trace, production_trace, strict=True)
    assert "Did you mean" in result_trace["final_response"]

This does more than block reintroductions of old bugs. It can also assert forward progress—demanding that an improved fix lands before deploy, using the trace as the boundary of acceptance.

Enforce Trace Replays in CI/CD: Real Deploy Breaks Blocked, Not Documented

Trace-based regression is only valuable if enforced. Reviewing traces in incident postmortems doesn't protect future deploys. This is how to integrate trace-native regression in your CI/CD via GitHub Actions:

.github/workflows/trace-replay.yml:

name: Trace Regression Replay

on:
  push:
    branches: [main, release/*]
  pull_request:
    types: [opened, synchronize]

jobs:
  trace-replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install tracely-harness
      - name: Run Tracely regression tests
        run: pytest tests/test_replay_production_failure.py

Any PR or deploy that regresses a real failure blocks immediately. Trace replays surface the changes no golden prompt set will catch—especially where agent logic, prompt churn, or dynamic tool routers rewrite experience on every deploy.

Trace Replay Surfaces Bugs Missed by Golden Prompts

Benchmarks show trace-native regression reveals breakage missed by input-only golden sets. The Apex Agent Reliability Benchmark found trace replay caught 37% more unique agent regressions across multiple agent frameworks. Emergent bugs from upstream LLM provider drift—like output schema changes—showed up in trace replays first because every tool call and intermediate state is covered, not just request/response boundaries.

Caveats:

Data capture has real cost. You need to instrument trace capture on the critical path, handle PII scrubbing, and invest in secure trace transport.
Replay isn't generative. It catches known failures, not theoretical futures. You'll still need input fuzzing and adversarial probes.
Replay fidelity relies on complete state capture. If your trace misses latent context (prompt injections, external side effects), bugs can sneak by even with replay in place.

Despite these limits, trace-regression testing does what prompt sets and user-perceived QA cannot—defensively catching real-world breakage before customers do.

Conventional CI for agents is performative. Wire real production traces into trace-based regression tests with tools like Tracely-ai and close the reliability gap, fast. The overhead is minor compared to outage cost. If you run agents in production and want to actually retire root causes, route your next bug trace straight into CI.

Day 13: GloVe: Global Vectors for Word Representation

Priyesh Dave — Wed, 09 Sep 2026 04:38:50 +0000

Previously, on Day 12: Explained how Word2Vec models generate word embeddings using skip-gram and CBOW techniques, compared their methods and trade-offs, and introduced a minimal example using Gensim.

Why GloVe? The Motivation Behind Global Word Embeddings

Word2Vec is a powerful method for learning word vectors. It trains these vectors so that words appearing in similar local contexts—words that show up near the same neighbors—end up close together in the vector space. The model does this by predicting a word from its neighbors (the CBOW method) or by predicting a word’s neighbors from itself (the Skip-gram method). All training happens inside narrow “windows” of text.

Focusing only on local context leaves gaps. If two words never appear close together in those windows but have similar broad patterns across the corpus, Word2Vec might miss that relationship.

GloVe (Global Vectors for Word Representation) addresses this limitation. GloVe captures both local context and global statistics—patterns from the entire dataset. Instead of just “Who is next to ‘king’ in a sentence?”, GloVe asks: “Across everything, how often do ‘king’ and ‘queen’ co-occur with the same words? How does this compare to ‘king’ and ‘man’?” GloVe uses information from all word pairs across the dataset, not just small windows, to learn these connections.

The main insight is that how often two words appear together, compared to how often they appear apart, reveals word meaning. For example, if “ice” appears with “cold” much more than “steam” does—and “steam” with “hot” much more than “ice”—those ratios reveal important facts about the words.

How GloVe Works: Counting Co-occurrences and Building Vectors

GloVe builds on counting global word co-occurrences. Here’s the basic process:

Build a vocabulary: For example, pick the top 400,000 most frequent words in your data.
Count co-occurrences: For every word pair (i, j), count how many times they appear near each other inside a window (say, five words on each side). This creates a co-occurrence matrix.
- Example: If "cat" appears within five words of "pet" 350 times, then the cell ("cat", "pet") in the matrix contains 350.
Compute statistics: For each word pair, you know: “Given word i, how likely is word j nearby?” You can turn this into a frequency, probability, or ratio.
Learn word vectors: Train each word’s vector so relationships between vectors match these statistics (details in the next section).

Think of the co-occurrence matrix as an enormous spreadsheet. Each row and column is a word; each cell tells you how many times those words appear near each other anywhere in your data. (For efficiency, GloVe usually works just with the nonzero counts.)

The GloVe Objective: What the Model Learns

What does GloVe actually optimize? The model tries to make the dot product of two word vectors (plus bias terms) match the logarithm of their co-occurrence count. In other words, the geometry of the vectors should reflect the global counts from your data.

Here’s the simplified loss function:

[
J = \sum_{i,j=1}^{V} f(X_{ij})\, \left(w_i^T \cdot w_j + b_i + b_j - \log(X_{ij})\right)^2
]

Term by term:

X_{ij}: How often word j appears near word i in the corpus.
w_i and w_j: Vectors for words i and j.
b_i and b_j: Bias values for each word—just single numbers.
f(X_{ij}): A weighting function so rare or extremely frequent pairs don’t overly affect the loss.

For each pair of words, the model pushes their dot product (plus biases) to be as close as possible to the log of their co-occurrence count.

Concrete Example:

If "ice" and "cold" appear together 500 times (X_{ice,cold}=500) and "ice" and "steam" only 12 times, then log(500) is about 6.2 and log(12) about 2.5.
GloVe trains the vectors for "ice" and "cold" so their dot product (+ biases) is close to 6.2, and "ice" + "steam" near 2.5.
If the model achieves this match for all word pairs, the words’ positions in vector space encode much of the useful statistical information in the corpus.

Why use logs and not raw counts? Logarithms compress a huge range of counts into more manageable values and make optimization easier.

GloVe Embeddings in Practice: Downloading and Using Pretrained Vectors

Most people use pretrained GloVe vectors. These are available for English (and some other languages) from datasets like Wikipedia, Common Crawl (large web scrape), and news sources.

A typical GloVe embedding file is a text file. Each line contains a word and its vector of numbers:

word_1 0.418 0.24968 ... -0.68568
word_2 0.123 -0.289 ... 1.459
...

Pretrained GloVe sets most commonly come in 50, 100, 200, or 300 dimensions. Vocabularies are very large—up to 400,000 words. You rarely need all of them; load only what you need for your project. The file format is simple and works almost interchangeably with Word2Vec’s text format.

When are pretrained GloVe files useful?

When you need word vectors quickly without heavy computation.
When working in restricted data environments (medical, legal, or proprietary data), you can fine-tune these vectors on your domain, instead of training from scratch.

Quickstart: Loading and Using GloVe Embeddings in Python

Below is a simple script to download a small GloVe file, load the vectors, and compute the similarity between two words.

import numpy as np
import urllib.request
import zipfile
import os

# Download a small GloVe embedding set (50 dimensions, ~70MB zipped)
glove_url = "http://nlp.stanford.edu/data/glove.6B.zip"
glove_zip_path = "glove.6B.zip"

if not os.path.exists(glove_zip_path):
    urllib.request.urlretrieve(glove_url, glove_zip_path)

# Extract the smallest file (glove.6B.50d.txt)
with zipfile.ZipFile(glove_zip_path, 'r') as z:
    if not os.path.exists("glove.6B.50d.txt"):
        z.extract("glove.6B.50d.txt")

# Load GloVe vectors into a dictionary
embeddings = {}
with open("glove.6B.50d.txt", encoding="utf-8") as f:
    for line in f:
        parts = line.strip().split()
        word = parts[0]
        vec = np.array(parts[1:], dtype=np.float32)
        embeddings[word] = vec

# Compute cosine similarity between two words
def cosine_similarity(x, y):
    return np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))

word1, word2 = "king", "queen"
vec1 = embeddings[word1]
vec2 = embeddings[word2]
sim = cosine_similarity(vec1, vec2)
print(f"Cosine similarity between '{word1}' and '{word2}': {sim:.3f}")

This script downloads 50-dimensional GloVe vectors, loads them into a Python dictionary keyed by word, and computes cosine similarity—a common measure of how similar two vectors are—between "king" and "queen".

Comparing GloVe and Word2Vec: When To Use Each

How they work:

Word2Vec learns word vectors using local text windows, optimizing to predict words from nearby words (or vice versa).
GloVe constructs a large matrix of global word co-occurrence counts, then learns vectors so relationships between them reflect these global patterns.
Both produce one fixed-length vector per word.

Practical differences:

GloVe tends to encode broader global relationships, since it “sees” the whole dataset at once.
Word2Vec is faster to train on smaller datasets or when training must run continuously on incoming text.
Pretrained GloVe vectors often work well out of the box for analogy and similarity tasks, especially where less frequent but globally insightful connections matter.

When to prefer each:

Use GloVe when: You want to leverage richer global context, need high-quality pretrained vectors, or your task benefits from word relationships visible at the corpus level.
Use Word2Vec when: You’re working with a smaller in-house dataset, want to adapt as new data arrives, or need easy updates and control over local window settings.

In practice, both methods generate useful embeddings for modern NLP projects. Many teams try both and keep whichever works best.

Key Takeaways

GloVe captures both local context and global co-occurrence statistics from the entire corpus.
The model optimizes word vectors so their dot products approximate the logarithm of word co-occurrence counts.
Pretrained GloVe embeddings can be easily used or fine-tuned for various NLP tasks.
GloVe and Word2Vec differ in their approach: GloVe uses global statistics, Word2Vec focuses on local windows.
GloVe vectors are often preferred when broad, corpus-level relationships are important.

Try It Yourself

Download a different pretrained GloVe vector set, such as the Twitter embeddings. Load the vectors and write code to compute the five words most similar to 'king' using cosine similarity. Reflect on how these nearest neighbors compare to those produced by Word2Vec in a previous exercise.

Further Resources

Coming up on Day 14: FastText and Subword Embeddings

Day 14: FastText and Subword Embeddings

Priyesh Dave — Wed, 09 Sep 2026 04:37:17 +0000

What Are Word Embeddings?

Words like "cat," "run," or "happiness" are just symbols to a computer. Computers operate on numbers, not symbols. To process language, we need a way to turn words into numbers—specifically, into mathematical objects that reflect their meaning and usage.

Word embeddings solve this. An embedding is simply a list of numbers—a vector—that represents a word. If two words are used in similar contexts, their vectors should be close together. For example, "cat" and "dog" would have similar vectors; both are common animals, and often appear in similar situations.

Word embeddings make it possible for machines to:

Compare word meanings mathematically. (For example, using cosine similarity, which measures the angle between two vectors.)
Cluster similar words together in this new number-based space.
Use these vectors as input features for machine learning models, including neural networks.

Before word embeddings, earlier approaches represented words in more primitive ways.

The Limits of Classic Embeddings

The most basic way to represent a word as a number is one-hot encoding. In one-hot encoding, each word in the vocabulary gets its own vector. This vector contains a single "1" in the position for that word, and "0" everywhere else. If your vocabulary has 10,000 words, every word gets a 10,000-dimensional vector, with just one non-zero entry.

This approach has major problems:

Wastes memory. Almost every entry is zero.
Shows no relationships between words. "Cat" and "dog" are as far apart as "cat" and "microwave."
Cannot handle words that were not seen during training time. These are called "out-of-vocabulary" (OOV) words.
Cannot handle typos. The model treats “cat” and “cta” as unrelated.
Struggles with languages that have many forms of the same root word. Each form is treated as unrelated.

Word embeddings such as Word2Vec or GloVe improved this by learning shorter, dense vectors for each word. Similar words have similar embeddings. But these classic embeddings still assign one unique vector per word. If a word is rare or was never seen in training, these models cannot create a vector for it.

FastText: Embeddings With Subwords

FastText, created by Facebook’s AI Research lab, addresses these gaps by looking inside words. Instead of treating a word as a single chunk, FastText represents each word as a bag of subword units, also called n-grams.

A subword unit is just a short piece of the word. For example, for the word “playing,” some 3-character n-grams are “pla,” “lay,” “ayi,” “yin,” and “ing.” FastText splits each word into overlapping character n-grams, usually between 3 and 6 characters long. Each n-gram gets its own vector.

To build the final embedding for a word, FastText combines the vectors for the word itself and all of its n-grams—typically by averaging or summing them.

For example, “jumping” would be represented by:

The embedding for the whole word “jumping”
The embeddings for “jum,” “ump,” “mpi,” “pin,” “ing” (and possibly others, depending on n-gram size)

Earlier approaches had no understanding of parts within a word. FastText fixes this.

Why Subword Embeddings Matter

Suppose the training data never included the word “jumped.” With classic embeddings, there is no vector for this word. The model cannot process it.

With FastText, "jumped" is split into n-grams such as "jum," "ump," "mpe," "ped." Even if “jumped” never showed up during training, its n-grams might have, from words like “jump,” “jumping,” or even “stumped.” FastText can build a vector for "jumped" by combining those parts.

This approach has important advantages:

Robust to typos. “jumping” and a typo like “jum ping” share many n-grams, so their vectors are similar.
Handles rare or invented words. If a rare word like “quixotically” shares letter chunks with common words, it still gets a reasonable vector.
Works with morphologically rich languages. Many languages use lots of word endings to change meaning. FastText can represent all these forms, even if it never saw them in training.

FastText is much less brittle than previous models, and handles natural, messy language well.

Working With FastText: A Minimal Example

Here’s how to use FastText in practice. We’ll use Gensim, a Python library, to train a tiny FastText model. Then we’ll look up vectors for a common word, a rare word, and a totally new word.

Install Gensim:

pip install gensim

Minimal FastText code:

from gensim.models import FastText

# Simple dataset: each sentence is a list of words.
corpus = [
    ["cat", "sat", "on", "the", "mat"],
    ["dog", "sat", "beside", "the", "cat"],
    ["dogs", "and", "cats", "are", "friends"],
    ["the", "dog", "jumped", "over", "the", "log"]
]

# Train a FastText model. Kept small for clarity.
model = FastText(sentences=corpus, vector_size=10, window=3, min_count=1)

# Common word
print("Vector for 'cat':", model.wv['cat'])

# Rare word (seen once)
print("Vector for 'jumped':", model.wv['jumped'])

# Plausible word never seen in training
print("Vector for 'jumping':", model.wv['jumping'])

# Made-up or misspelled word
print("Vector for 'cattz':", model.wv['cattz'])

Key points:

“jumping” and “cattz” were never in the training set. But FastText can produce vectors for them, by combining vector pieces from their n-grams.
Classic Word2Vec or GloVe models cannot handle such words—they only know the words seen during training.

When (and When Not) to Use FastText

FastText’s subword approach is most useful when:

Your text is messy, with typos, slang, or creative spelling (like in chat logs or social media).
You are working with languages that have many word forms due to rich grammar (morphology).
Your domain or dataset has many rare, specialized words.

Situations where FastText may be less helpful:

FastText is slower. At prediction time, it needs to look up and combine several vectors for each word.
Sometimes, the similarities it finds are harder to explain, since words sharing n-grams might not actually be related in meaning.
For clean, well-edited text in a language like English, and when you have plenty of training data, simpler embeddings like Word2Vec or GloVe are often sufficient.

If your data is human-authored, noisy, or has lots of words outside your training set, FastText is a strong choice. For very large projects, it’s worth testing both classic and subword-based embeddings to see what works better for your needs.

Key Takeaways

Word embeddings translate words into numerical vectors capturing semantic similarity.
Classic embeddings like one-hot cannot capture word relationships or handle unseen words.
FastText creates embeddings using character n-grams, enabling vector creation for out-of-vocabulary words.
Subword embeddings are robust to typos, rare words, and morphologically rich languages.
FastText trades some speed and interpretability for improved handling of noisy, diverse data.

Try It Yourself

Write a list of 5 words, including at least one made-up word and one misspelled word. Train a small FastText model on a simple dataset using Gensim, then extract vectors for your test words. Compare the vectors of the made-up and misspelled words to those of semantically or orthographically similar real words, and note how well FastText handles these out-of-vocabulary inputs.

Further Resources

Coming up on Day 15: Visualizing Embeddings with t-SNE and PCA

Day 12: Word2Vec Explained: Skip-Gram and CBOW

Priyesh Dave — Sun, 06 Sep 2026 09:58:09 +0000

Previously, on Day 11: Explained one-hot encoding as a method for representing words as sparse vectors, highlighted its limitations regarding similarity and efficiency, and introduced the concept of word embeddings as an improvement.

What Are Word2Vec Models?

One-hot vectors represent words for a computer using long lists of zeros with a single one—like a light switch for each word. Each word has a unique position. Every other position is zero. This approach is simple but has two big drawbacks. First, the vectors are extremely sparse—almost all zeros. Second, they contain no information about meaning. The one-hot vectors for “cat” and “dog” are just as different as those for “cat” and “banana,” even though “cat” and “dog” are related.

Word2Vec is a technique that addresses both problems. It learns short, dense vectors called embeddings. Each word gets its own embedding—a list of a few dozen or a few hundred decimal numbers. These embeddings are not hand-written. They are trained from large amounts of real text. Words with similar meanings end up with similar embeddings.

The Skip-Gram Model: Predicting Context from a Target Word

The Skip-Gram model is one of two main approaches in Word2Vec. Its idea: Given a single target word, can we guess which words are likely to appear near it in sentences?

Example:

Take the sentence:

"The quick brown fox jumps over the lazy dog"

Pick the word “fox.” Imagine a window that covers two words to the left and two to the right. The context words are: “quick,” “brown,” “jumps,” “over.”

The skip-gram model’s task is to take “fox” and try to predict each of those context words.

During training, the model is shown pairs like (“fox”, “quick”), (“fox”, “brown”), and so on. But it doesn’t just memorize the examples. Each word is mapped to a vector, and the model tweaks these vectors so the right target-context pairs become more likely, over millions of examples.

Over time, words like “fox,” “dog,” and “wolf” often show up in similar contexts. Their embeddings shift closer together in the vector space.

The CBOW Model: Predicting a Target Word from Its Context

Continuous Bag of Words (CBOW) is the reverse of skip-gram. Instead of predicting context from a word, it predicts a single missing word given its surroundings.

Example:

Consider the same sentence:

"The quick brown fox jumps over the lazy dog"

Blank out the word “fox.” Now the model sees: “quick,” “brown,” “jumps,” “over.” Its job is to predict that the missing word is “fox.”

CBOW combines the embeddings for all the context words (by averaging or adding them) and uses that combined vector to predict the most likely target word. During training, it adjusts all the embeddings so that, given surrounding words, the correct word is predicted as often as possible.

Both skip-gram and CBOW learn useful word embeddings as a side effect of trying to solve their prediction tasks.

Comparing Skip-Gram and CBOW with Concrete Examples

Suppose we use the sentence:

"The quick brown fox jumps"

with a context window of 2.

Skip-Gram examples:

Input: “brown” → Output: “quick”, “fox”
Input: “fox” → Output: “brown”, “jumps”

You feed the model each word in the middle and try to guess nearby words.

CBOW examples:

Input: “quick”, “fox” → Output: “brown”
Input: “brown”, “jumps” → Output: “fox”

You use the neighboring words as input and try to guess the middle word.

Key points:

Skip-gram generates more training pairs (since every word predicts several context words), but each pair is simple—one input, one output.
CBOW creates fewer pairs, but each combines multiple input words into a single example.

How Training Adjusts Word Embeddings

When the model predicts incorrectly—for example, guessing “wolf” instead of “fox”—it slightly changes the embeddings to reduce the error. If “quick,” “brown,” and “jumps” often occur near both “fox” and “wolf,” the embeddings for “fox” and “wolf” will become more similar.

Words that appear in many environments, like “the” or “and,” get embeddings that reflect their generic context.

Word2Vec training is entirely driven by the patterns in real text. Nothing about meaning is written in by hand. The frequencies and co-occurrences in the data teach the model what words often appear together. The resulting math places similar-meaning words near each other in the embedding space.

Over many examples, clusters emerge. “King” and “queen” end up close together. “Walk” and “run” are neighbors. Sometimes, the difference between vectors is also meaningful: the vector from “man” to “woman” is similar to the vector from “king” to “queen.”

When to Use Skip-Gram or CBOW

Both models learn good word vectors, but each has specific strengths.

Skip-Gram: Stronger for rare words. Every rare word gets a chance to predict its context in each appearance, so the model can learn their nuances even from limited data.
CBOW: Faster to train. It predicts a word from several context words combined into one example, making it more efficient. It works well for common words but can blur details for rare or unusual words.

Choose skip-gram if your data contains many unique or rare words and you care about precise representations. Choose CBOW if you have huge amounts of text and want fast, general embeddings.

Code Example: Minimal Skip-Gram with Gensim

Here’s a simple example using gensim, a Python library for Word2Vec. This code trains a skip-gram model on a tiny dataset and finds the most similar words to “fox.”

from gensim.models import Word2Vec

# Small demo corpus
sentences = [
    ["the", "quick", "brown", "fox", "jumps"],
    ["the", "lazy", "dog", "sleeps"],
    ["the", "fox", "outsmarts", "the", "dog"],
]

# Train a skip-gram Word2Vec model
model = Word2Vec(
    sentences,       # list of tokenized sentences
    vector_size=50,  # embedding dimensions
    window=2,        # context window size
    min_count=1,     # include all words
    sg=1,            # 1 for skip-gram; 0 for CBOW
)

# Find most similar words to “fox”
similar_words = model.wv.most_similar("fox", topn=3)
for word, score in similar_words:
    print(word, score)

In this snippet:

Each sentence is tokenized as a list of words.
sg=1 chooses skip-gram. Switch to sg=0 for CBOW.
After training, we ask for the three words closest to “fox.”
With such a tiny dataset, results are limited, but you’ll see “dog” or "brown" may appear—showing which embeddings are close even from a small set.

Word2Vec can easily scale to millions of words and much larger texts. The process for training and looking up similar words is just the same. With enough data, the model finds meaningful relationships between words automatically.

Key Takeaways

One-hot vectors lack meaning and are sparse, while Word2Vec embeddings are dense and semantic.
Skip-gram predicts context words from a target word, generating multiple simple training pairs.
CBOW predicts a missing word from its context, combining surrounding words in each example.
Skip-gram excels with rare words; CBOW trains faster but can blur rare word distinctions.
Word2Vec learns word meanings entirely from text data using co-occurrence patterns.

Try It Yourself

Take the sentence 'The cat sat on the mat.' For a context window of 1, write out all the Skip-Gram training pairs by hand: for each word, list the words directly before and after it as outputs. This will help you see how Skip-Gram training data is structured.

Further Resources

Coming up on Day 13: GloVe: Global Vectors for Word Representation

Day 11: Why One-Hot Encoding Fails: The Case for Embeddings

Priyesh Dave — Fri, 04 Sep 2026 04:50:51 +0000

What is One-Hot Encoding?

Computers need numbers, not words. To process language, we first turn words into numbers. One of the earliest ways to do this is one-hot encoding.

A one-hot encoding represents each word in your vocabulary with a vector—a list of numbers—whose length matches the vocabulary size. All entries in this list are zero, except for a single one: the “hot” bit, which marks the position of that word in your list.

This approach is simple to compute. It lets you plug words into early machine learning algorithms. One-hot encoding was a workhorse for the first language models.

Example: One-Hot Vectors

Suppose your vocabulary is just three animals: cat, dog, and fish. Assign each a position:

cat is at position 0
dog is at position 1
fish is at position 2

Their one-hot encodings are:

cat → [1, 0, 0]
dog → [0, 1, 0]
fish → [0, 0, 1]

If you see [0, 1, 0], you know it’s dog because only the middle spot is “hot.”

Drawbacks of One-Hot Encoding

Now scale up: your vocabulary has tens of thousands of words. Each word becomes a long vector with mostly zeros and a single one.

This causes several issues:

1. No Sense of Similarity

One-hot vectors don’t reflect meaning or relationships. For the computer, cat and dog look just as unrelated as cat and fish. There’s no sense that dogs and cats are both pets or mammals.

The dot product—a way to measure how much two vectors point in the same direction—is zero for any pair of different one-hot vectors.
The distance between any two distinct one-hot vectors is always the same.

Example in code:

import numpy as np

# Vocabulary and lookup
vocab = ['cat', 'dog', 'fish']
word_to_index = {word: idx for idx, word in enumerate(vocab)}

# Function to create a one-hot vector
def one_hot(word):
    vec = np.zeros(len(vocab))
    vec[word_to_index[word]] = 1
    return vec

cat_vec = one_hot('cat')
dog_vec = one_hot('dog')
fish_vec = one_hot('fish')

# Dot products
print(np.dot(cat_vec, dog_vec))   # 0.0
print(np.dot(cat_vec, fish_vec))  # 0.0

# Euclidean distances
print(np.linalg.norm(cat_vec - dog_vec))   # 1.414...
print(np.linalg.norm(cat_vec - fish_vec))  # 1.414...

No matter which two words you compare, the numbers are identical. One-hot vectors can't show that some words are more related than others.

2. Inefficient Representation

Real vocabularies are large. Each one-hot vector is mostly zeros—these are called sparse vectors. If you have 10,000 words, each word uses a vector 10,000 entries long. Storing these wastes memory and computation.

Why This Matters

Modern language tasks depend on recognizing relationships between words. Search engines, translation systems, and sentiment analysis systems need to recognize that words like happy and joyful are connected in meaning.

With one-hot encoding, swapping a word for a synonym changes the vector completely. The computer sees no connection. This makes it hard for models to learn about patterns, context, or subtle meanings.

If a system can’t detect that two words mean nearly the same thing, it will struggle to find similar documents, suggest synonyms, or identify related concepts.

A Glimpse at Embeddings

To address these problems, we use embeddings.

Embeddings also represent words as vectors. But instead of being all zeros except for one, embeddings use vectors filled with real values—often dozens or hundreds of numbers per word. Words with similar meanings get vectors that are close together. Cat and dog embeddings are similar; cat and fish are not as close.

This lets computers detect patterns, similarity, and context. You’ll learn how embeddings work, and how they power modern language models, in upcoming lessons. For now: embeddings let machines measure similarity and capture relationships between words, something one-hot encoding can’t do.

Key Takeaways

One-hot encoding maps each word to a unique sparse vector with a single one.
All one-hot vectors are equally distant; they show no word similarity.
One-hot representations become inefficient with large vocabularies.
Embeddings create dense vectors that capture relationships and similarity between words.

Try It Yourself

Choose five words and assign each a position in your list. Write their one-hot vector representations either by hand or in code. Compare the vectors for each word pair—are any pairs more similar than others? Explain your findings.

Further Resources

Coming up on Day 12: Word2Vec Explained: Skip-Gram and CBOW

Day 10: Evaluation Metrics for NLP: Precision, Recall, F1, Confusion Matrix

Priyesh Dave — Thu, 03 Sep 2026 13:45:55 +0000

Previously, on Day 9: Explained the fundamentals of text classification, focusing on bag-of-words representations and comparing Naive Bayes and logistic regression classifiers for labeling text such as spam detection.

Model evaluation means measuring how well your NLP system performs a specific task. These measurements, called metrics, offer concrete evidence of your model’s strengths and weaknesses. For engineers, metrics are not just numbers—research, engineering choices, and business decisions all depend on them.

Accuracy: The Basic Metric

Accuracy tells you what fraction of predictions were correct. But accuracy alone can mislead, especially when certain mistakes cost more, or when your data is imbalanced.

Take a spam filter. If only 5% of emails are spam, a model that always predicts “not spam” scores 95% accuracy. It looks good on paper, but in reality, the system fails at catching spam.

Accuracy is just a starting point. Most real-world problems need more nuanced metrics.

Precision: Correctness of Positive Predictions

Precision measures how many of the items your model labeled as positive were actually positive.

In plain terms: if your spam filter marks 10 emails as spam, and only 8 are really spam, your model made 2 mistakes—it blocked good email.

Precision focuses only on "is spam" predictions. Out of everything marked as spam, how many truly were?

Precision = (True Positives) / (True Positives + False Positives)

Spam filter example:

8 spam emails correctly blocked (true positives)
2 valid emails wrongly blocked (false positives)

Precision = 8 / (8 + 2) = 0.8, or 80%.

High precision means that when your model says "spam," it’s probably right. Systems tuned for high precision rarely block important messages—but may let some spam through.

Recall: Coverage of Real Positives

Recall measures how many actual positive items your model found.

Returning to the spam filter. Suppose there were 12 spam emails in your inbox, but your model only caught 8.

Recall = (True Positives) / (True Positives + False Negatives)

Here:

True positives: 8 spam emails caught
False negatives: 4 spam emails missed and let through

Recall = 8 / (8 + 4) = 8 / 12 ≈ 0.67, or 67%.

High recall means the model finds nearly all the spam. But if recall is too high at the cost of precision, it may block good mail as well.

Precision and recall often work against each other. Improving one can lower the other.

F1 Score: The Balance Point

When you want a single measure that balances precision and recall, use the F1 score. It’s the harmonic mean—if either precision or recall is low, the F1 score drops sharply.

F1 = 2 × (Precision × Recall) / (Precision + Recall)

With our example:

Precision = 0.8
Recall = 0.67

F1 = 2 × (0.8 × 0.67) / (0.8 + 0.67)

F1 = 2 × 0.536 / 1.47 ≈ 1.072 / 1.47 ≈ 0.73, or 73%.

If either precision or recall is very low, the F1 score will also be low.

Confusion Matrix: The Error Breakdown

Metrics can be abstract. The confusion matrix makes results easy to see and analyze.

A confusion matrix is a table showing the count of true and false outcomes for each class. For binary problems (like spam vs. not spam), it looks like this:

	Predicted: Spam	Predicted: Not Spam
Actual: Spam	True Positive (TP)	False Negative (FN)
Actual: Not Spam	False Positive (FP)	True Negative (TN)

True Positive (TP): Spam predicted as spam.
False Positive (FP): Not-spam predicted as spam.
True Negative (TN): Not-spam predicted as not-spam.
False Negative (FN): Spam predicted as not-spam.

From this table, you can always calculate:

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

Here’s a bare-bones Python example:

# Sample labels (1=spam, 0=not spam)
actual    = [1, 0, 1, 1, 0, 0, 1, 0, 1, 0]
predicted = [1, 0, 1, 0, 0, 1, 1, 0, 0, 0]

# Compute confusion matrix entries
TP = sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 1)
FP = sum(1 for a, p in zip(actual, predicted) if a == 0 and p == 1)
TN = sum(1 for a, p in zip(actual, predicted) if a == 0 and p == 0)
FN = sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 0)

print(f"TP: {TP}, FP: {FP}, TN: {TN}, FN: {FN}")

# Now calculate the metrics
precision = TP / (TP + FP) if (TP + FP) else 0
recall = TP / (TP + FN) if (TP + FN) else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0

print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1 Score: {f1:.2f}")

This prints the confusion matrix numbers, then precision, recall, and F1.

Metric Pitfalls: When Numbers Mislead

Metrics are tools. The wrong tool can lead to the wrong decisions.

If your data has 1,000 emails, but only 10 are spam, always predicting "not spam" means 99% accuracy. But you'll miss all the spam—accuracy is useless here.

In medical tests, recall might matter more (catch all disease cases). In email, high precision may matter more (never lose real mail). Sometimes you want a balance (F1); sometimes you care much more about one type of error than another.

Always connect your metrics to your real-world goals. Analyze your confusion matrix to understand your model’s errors. Never trust a single number without asking whether it matches your needs.

Metrics measure, compare, and drive improvement. But their value always comes from how thoughtfully you choose, compute, and interpret them.

Key Takeaways

Accuracy alone can be misleading, especially with imbalanced data.
Precision measures how often positive predictions are actually correct.
Recall reflects how many real positives your model identifies.
The F1 score balances precision and recall into a single metric.
A confusion matrix shows detailed error counts, revealing strengths and weaknesses.

Try It Yourself

Take these actual labels: [1, 0, 1, 1, 0, 0, 1, 0, 1, 0] and predictions: [1, 0, 1, 0, 0, 1, 1, 0, 0, 0]. Manually fill in a confusion matrix by counting true positives, true negatives, false positives, and false negatives. Then calculate precision, recall, and F1 score for this data set, showing your steps.

Further Resources

Coming up on Day 11: Why One-Hot Encoding Fails: The Case for Embeddings

Vector, Graph, or SQL? What Actually Works for Production AI Agent Memory

Priyesh Dave — Wed, 02 Sep 2026 15:14:29 +0000

Vector, Graph, or SQL? What Actually Works for Production AI Agent Memory

Scalable agent memory is no longer a side project. Below are direct, runnable benchmarks comparing vector store, graph, and SQL agent memory—where each fails, which one holds up under realistic loads, and numbers for latency, recall, and operational pain. At scale, the right call means the difference between a stable system and endless triage.

Agent Memory Becomes the Bottleneck Before the LLM Fails

Most AI agent demos end once the LLM spits out an answer from some context window. In production, that's not where systems break. Failures typically strike at the memory and query layers:

Latency spikes the instant your index outgrows demo scale.
Embedding-only recall regurgitates "semantically similar" content that's factually wrong or structurally useless.
"Just swap in a bigger store" becomes a trap: agents break when working set exceeds RAM or when relationships need tracking.

Serious agent workloads—continuous learning, chaining, multi-user context—hit hard limits on memory performance and retrieval quality. The naive "just use a vector DB" strategy disintegrates at scale, with link-heavy context, or when it's time to debug.

Vector Stores: High Throughput, Bad at Structure

Vector databases (FAISS, Chroma, Pinecone, etc.) remain the obvious default for LLM memory. They deliver unbeatable raw embedding search for static text or code. That speed collapses once you need structure or linked context.

Real-World FAISS Latency

A 10,000-record FAISS in-memory store, basic retrieval:

import faiss
import numpy as np
import time

dim = 1536  # OpenAI/text-embedding-ada-002
num_vectors = 10_000
np.random.seed(42)
vectors = np.random.randn(num_vectors, dim).astype(np.float32)

index = faiss.IndexFlatL2(dim)
index.add(vectors)

query = np.random.randn(1, dim).astype(np.float32)

start = time.time()
D, I = index.search(query, k=5)
latency = (time.time() - start) * 1000

print(f"FAISS top-5 retrieval latency: {latency:.2f} ms")

Observed: sub-10ms per query for <100k vectors in-memory. Pure brute force similarity wins here.

Structural Recall: Major Gaps

No sense of order or event sequence. You can't natively fetch "what happened just before this?" or "all records tagged as X."
Mixed or repetitive context? Embeddings start returning lookalikes, not what's actually relevant—unless you bolt on clumsy metadata filtering.
Trivial to one-hop recall, hopeless on context chains, dialogue threads, or anything resembling a dependency graph.

Why It Fails: Example

Suppose the agent must recall a meeting summary and link to prior related action items. Pure vector search drags in loosely related content, often missing explicit cross-document links that actually matter.

Store both summaries and discrete action items as vectors.
Query for "next steps from last week’s planning meeting."
Returns lookalike text, with a strong chance of contaminating context across multiple meetings or omitting key items.

Graph Databases: Explicit Relationships, Real Scalability Headaches

Graph DBs (Neo4j, RedisGraph, Memgraph) encode relationships naturally. For context-chaining, they're far stronger than vectors. But writes and complex queries go nonlinear fast.

Neo4j for Agent Memory: Insert and Query

Imagine agent events as nodes (Event) with types and RELATED_TO edges.

from neo4j import GraphDatabase
import time

uri = "bolt://localhost:7687"
driver = GraphDatabase.driver(uri, auth=("neo4j", "test"))

def add_event(tx, event_id, event_type, content, timestamp):
    tx.run(
        "MERGE (e:Event {id: $event_id, type: $event_type, content: $content, ts: $timestamp})",
        event_id=event_id, event_type=event_type, content=content, timestamp=timestamp
    )

def link_events(tx, from_id, to_id):
    tx.run(
        "MATCH (a:Event {id: $from_id}), (b:Event {id: $to_id}) "
        "MERGE (a)-[:RELATED_TO]->(b)",
        from_id=from_id, to_id=to_id,
    )

with driver.session() as session:
    start = time.time()
    for i in range(5000):
        session.write_transaction(add_event, f"evt_{i}", "summary", f"content_{i}", i)
    for i in range(1, 5000):
        session.write_transaction(link_events, f"evt_{i-1}", f"evt_{i}")
    latency = (time.time() - start) * 1000
    print(f"Neo4j insert + link 5k events: {latency:.2f} ms")

    query_start = time.time()
    res = session.run(
        "MATCH (e:Event)-[:RELATED_TO*1..3]->(other) WHERE e.id='evt_1000' "
        "RETURN other LIMIT 5"
    )
    print(f"3-hop query latency: {(time.time() - query_start) * 1000:.2f} ms")

Where Graph DBs Lose

Inserts are at least 10x slower than vector DBs—tens of ms per 1,000 edge inserts.
Multi-hop queries (*..n relationship traversals) explode in cost as your schema gets deeper or more cyclic.
Cypher queries quickly get unreadable; any "fuzzy" or similarity-weighted search is a hack or plugin.
Hybrid semantic search requires nontrivial extension work.

Schema tip: Nodes for each event, typed with labels; edges for relationships.

Pitfall: Deep, cyclic, or cross-linked graphs crush traversal speed.

SQL: The Overlooked Hybrid Memory That Holds Up

SQL (Postgres, SQLite) has a bad rep among LLM agent tinkerers, but that’s a miss. With a competent schema and minimal vector support, SQL wins for compound recall and operational scaling.

Python + SQLite: Real Hybrid Recall

import sqlite3
import numpy as np
import time

conn = sqlite3.connect(":memory:")
c = conn.cursor()
c.execute('''CREATE TABLE memory (
    id INTEGER PRIMARY KEY,
    type TEXT,
    content TEXT,
    ts INTEGER,
    embedding BLOB
)''')

def to_blob(x): return x.astype(np.float32).tobytes()
embeddings = np.random.randn(10000, 1536)
start = time.time()
for i in range(10000):
    c.execute("INSERT INTO memory (type, content, ts, embedding) VALUES (?, ?, ?, ?)",
              ('summary', f"content_{i}", i, to_blob(embeddings[i])))
conn.commit()
print(f"SQLite 10k row insert: {(time.time()-start)*1000:.2f} ms")

def search(query_embed, k=5):
    c.execute("SELECT content, embedding FROM memory WHERE type='summary' AND ts BETWEEN 0 AND 5000")
    rows = c.fetchall()
    X = np.array([np.frombuffer(row[1], dtype=np.float32) for row in rows])
    dists = np.linalg.norm(X - query_embed, axis=1)
    topk = np.argsort(dists)[:k]
    return [rows[i][0] for i in topk]

qvec = np.random.randn(1, 1536)
start = time.time()
results = search(qvec[0], k=5)
print(f"SQLite hybrid filter + search latency: {(time.time()-start)*1000:.2f} ms")

Where SQL Wins

Multi-constraint queries (e.g., "summaries between times X and Y, linked to project Z") are trivial and performant on indexes.
Basic vector search is fast enough for tens of thousands of rows, especially with Postgres extensions or Python glue.
Integrates metadata and semantic search in one query, no extra plumbing.
Schema evolution and debugging are dramatically simpler than graph or vector DBs.

A mature ops team can evolve SQL schemas, index smartly, and drop in vector tooling for flexible, high-precision recall. Unless you demand million-record fuzzy recall, SQL scales further than most assume.

Head-to-Head: Actual Numbers and Pain Points

Benchmarks from a Macbook Pro M2, 16GB RAM, Python 3.11. Vector: FAISS. Graph: Neo4j Desktop. SQL: SQLite in-memory.

Memory Type	Insert Latency (10k rows)	Top-5 Retrieval	Multi-Hop Recall	Schema Flexibility	Debuggability	Scaling Ceiling
FAISS vstore	~450 ms	~7 ms	Manual, poor	Low	Fair	RAM-bound
Neo4j graph	~5,000 ms	~120 ms (3-hop)	Native, scalable	Medium	Poor (Cypher)	Nonlinear cost
SQLite (SQL)	~700 ms	~25 ms	Manual, but simple	High	Excellent	Disk/Index-bound

Early failure modes:

Vector: Chokes on multi-hop/metadata queries, or >1M vectors without sharding.
Graph: Bogs down on deep traversals, cyclic schemas, or batch inserts.
SQL: Bottlenecks without vector indexing at scale, but otherwise robust.

Recall quality:

Vector: Excels at broad similarity, fails for nuanced, cross-linked context.
Graph: Matches explicit relationships, weak for fuzzy similarity.
SQL: Satisfies 80% of agent memory constraints—hybrid queries across structured and unstructured data.

Architecture Choice: What Survives and Where

Tradeoff diagram (in words):

X Axis: Recall complexity (fuzzy on left, multi-hop/constraint on right)
Y Axis: Operational readiness (toy demos at bottom, production ops at top)
Lower-left: FAISS/Chroma—dominant for fuzzy, fast at toy and mid-scale, unusable for complex recall.
Central band: SQL with vector support—handles structured, hybrid recall; easy schema evolution.
Upper-right: Graphs—needed only for deeply interlinked or provenance-anchored memory; expensive to scale and debug.

Skip vector DB when:

Ordered, tagged, or event-chained recall trumps raw similarity.
You need queries like "all action items from meetings with X between Y and Z."

Hybrid (SQL+vector or Graph+vector) is required when:

Both structure and semantic similarity drive retrieval requirements.
Use SQL for multi-constraint, fall back to vector for fuzzy matches.

Bottom line:

Vectors collapse under production agent memory needs. Graph is specialized, costly for most workloads. SQL outlasts both for operational, flexible, and multi-faceted recall. Choose memory architecture for what the agent actually does—not for "demo simplicity."