DEV Community: MANGESH MANDLIK

You Probably Don't Need an LLM: ML vs Deep Learning vs Generative AI

MANGESH MANDLIK — Sat, 19 Sep 2026 12:21:06 +0000

Picture a planning meeting where someone says, "We should use AI to catch fraudulent transactions." Everyone nods.

But one engineer is thinking about a gradient-boosted model trained on last year's transaction table. Another is imagining a neural network. A third has already opened the docs for a hosted LLM API. Same sentence, three completely different systems, with different costs, latencies and failure modes. Nobody in the room noticed, because "AI" has become a word that stretches from a 1990s chess engine to whatever is writing your pull request descriptions this week.

The words AI, machine learning, deep learning and generative AI are not synonyms. They aren't competitors either. They're nested, and once you have that picture in your head, a surprising number of architecture decisions get easier.

I also built an animated walkthrough of this hierarchy, ML vs Deep Learning vs Generative AI on SeeItFlow, if you'd rather watch it move. What follows is the written version.

Four circles, one inside the other

Think of four circles, each sitting inside the previous one.

The biggest is AI: any system that mimics intelligent behaviour. That includes hand-written rules, statistical methods and neural networks. A chess engine from 1997 counts.

Inside it sits machine learning, where the system learns patterns from data instead of following rules a human wrote down. Inside that is deep learning, which is machine learning done with neural networks that have many layers. And in the middle is generative AI, which is deep learning that creates new content (text, code, images, audio) rather than just labelling what you give it.

That nesting gives you a one-way rule. Every generative AI system is a deep learning system, and every deep learning system is a machine learning system. The reverse is not true. Your spam filter is machine learning but not deep learning. An image classifier that sorts photos into a thousand categories is deep learning but not generative. When someone says "AI" and means "a chatbot," they've skipped three circles and landed on the innermost one, and that's where a lot of bad project decisions start.

Why the layers exist at all

These categories didn't appear together. Each one showed up when the previous approach ran into a wall.

For decades, roughly the 1960s through the 90s, building intelligent software meant writing rules. Expert systems and chess engines ran on explicit if/else logic that people typed in by hand. This works until the world gets messier than your rulebook. A spam filter with 500 hardcoded rules is out of date the week a new spam pattern shows up, and someone has to notice and write rule 501. Manual rules don't scale.

Machine learning flipped the workflow. Instead of writing the rules, you show the system a pile of labelled examples and let it find the rules itself. Spam filters, credit scoring and recommendation systems all moved this way.

But classic ML has a catch that's easy to miss: a human still decides what the model gets to look at. That step is called feature engineering. On a table of transactions, a domain expert knows to compute something like "this amount compared to the customer's usual spending." On a photograph, though, nobody knows which combination of pixel values means "cat." Experts spent years hand-designing image features, and the results plateaued.

Then came 2012. AlexNet, a deep convolutional network, cut the error rate on the ImageNet benchmark from about 26% to about 15% in a single year. That was a gap large enough to change where the whole field put its money. Deep learning had been possible for a long time. What made it practical was a combination of things arriving together: GPUs, which are very good at the thousands of parallel matrix multiplications that training a neural network consists of, big datasets like ImageNet and Common Crawl, and training improvements like ReLU activations, dropout and batch normalisation. By the middle of the decade it was the dominant approach for images, audio and language.

Generative AI is the newest layer, from about 2020 onward. It didn't replace what came before it. It asked a different question, which we'll get to.

Classic ML: the workhorse nobody tweets about

Here's the whole idea of machine learning in a few lines:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(TfidfVectorizer(), LogisticRegression())
model.fit(train_emails, train_labels)          # labelled examples go in

model.predict(["Congratulations, you won a free cruise"])   # a label comes out

You give it labelled data, it learns a mapping from inputs to outputs, and then it predicts on data it hasn't seen. Spam detection, fraud detection, demand forecasting and ranking items in a recommendation feed all follow this pattern.

The practical profile of classic ML is what makes it so useful. It typically needs thousands to millions of rows, not billions. It usually trains and runs fine on a CPU. Many of the models, like linear regression and decision trees, are interpretable, so when the model flags a transaction you can often say why. And on structured, tabular data, gradient boosting libraries like XGBoost and LightGBM have a long track record of beating fancier approaches, including in Kaggle competitions on tabular problems.

That last point is the one I'd tattoo on the inside of every AI roadmap. A large share of the AI running in production at real companies is still this: boring, cheap, debuggable classic ML. It's not glamorous, and it works.

Deep learning: for when you can't describe the features

Deep learning is machine learning with multi-layer neural networks, and the "deep" just means many layers. Each layer learns a more abstract representation of the input than the one before it. In an image model, early layers pick up edges, middle layers pick up textures and shapes, and later layers pick up things like "wheel" or "face."

The important shift is that the network discovers the features itself. You don't hand-craft them. That's exactly why deep learning took over problems where nobody could write down what to look for: image classification, object detection with models like YOLO, speech recognition, machine translation.

The trade-offs are the flip side of that power. You need much more data. You usually need GPUs. Training is expensive, and the resulting model is often a black box, so explaining an individual prediction gets much harder. If your data is a spreadsheet, you're probably paying all those costs for no gain. If your data is pixels or audio waveforms or raw text, you're paying them because you have no better option.

Generative AI asks a different question

This is the layer people mean when they say "AI" in 2026, and it's worth being precise about what's different.

A classic ML model, and most deep learning models, answer the question: which category does this input belong to? They output a label or a number. A generative model answers a different question: what should come next, given everything so far? It outputs new content.

For text and code, the mechanism is autoregressive generation. The model predicts the next token, appends it to the context, and repeats until it's done. Every response, however long, is built that way, one token at a time. (Image and video generators like Midjourney and Sora use other techniques, diffusion models being the common one, but the spirit is the same: a deep network trained to produce new content that resembles what it learned from.)

That difference is fundamental, not cosmetic. A fraud model gives you a probability between 0 and 1. A language model gives you a paragraph, and the paragraph might be brilliant, or plausible and wrong. You've moved from a system with a narrow, checkable output to a system with an open-ended one, and everything about how you test, monitor and budget for it changes.

It's also worth being honest about what these models are. They're extremely good at predicting statistically likely continuations of text. Whether that amounts to "understanding" is a real debate, and some behaviour at scale looks a lot like reasoning. But as an engineer you should design as though the model is a powerful pattern-matcher over token sequences, not as though it has a reliable model of the world. That assumption will save you from a lot of confidently wrong outputs in production.

So which one do you reach for?

The fastest way I know to sort a problem is to look at the input and the output.

Take "is this email spam?" The input is structured features plus text statistics, the output is a label, and you have labelled data. That's classic ML. "Predict tomorrow's sales" is a structured time series with a number as the output, and gradient boosting will often beat a neural network there. "Detect credit card fraud" is tabular data with numeric features, and an XGBoost-class model is the standard answer.

Now change the input. "Recognise objects in photos" has raw pixels as the input, and no one can handcraft pixel features at scale, so that's deep learning. "Transcribe spoken audio" is raw waveforms turned into text, which needs learned acoustic representations, so also deep learning.

Change the output and you get the third case. "Write a Python function," "generate marketing copy," "summarise a 50-page document": the output is new content, and only generative models can produce it.

If you want it as a rule of thumb:

Is the output a label or a number?         -> start with classic ML
Is the input raw images, audio or text?    -> deep learning
Is the output new content?                 -> generative AI

Read it top to bottom and stop at the first one that fits. The ordering is deliberate. It pushes you toward the simplest tool that could work.

What this changes about how you build

Once you see these as layers with different cost profiles, a few practical things follow.

An LLM is not a free upgrade. It adds latency, since generation is sequential and each response takes as many forward passes as it has tokens. It adds cost, since you're billed per token. And it adds unpredictability, because the same prompt can produce different outputs and there's no simple accuracy number to watch. For a spam filter, an inventory forecast or a click-through prediction, a small ML model is faster, cheaper, more predictable and frequently more accurate. Reaching for an LLM there is like hiring a novelist to fill in a spreadsheet.

Build the dumb baseline first. Before you design anything with GPUs in it, train a logistic regression or a gradient-boosted model on the data you have. It takes an afternoon. If it hits your target, you're done, and you've saved months. If it doesn't, you now have a number that any fancier approach has to beat, which keeps the conversation honest.

Debuggability is a feature. When a decision tree misclassifies something you can walk the tree and see why. When a deep network does, you get a shrug and some saliency maps. When an LLM does, you get a fluent explanation that may or may not reflect what actually happened. In regulated or high-stakes systems, that difference can decide the architecture on its own.

Mixing layers is normal. The strongest systems I see described tend to combine them: a deep model turns messy input like text or images into something structured, and a classic model makes the final scored decision on top. Nothing says a project has to live in one circle.

Where I'd land

None of these layers is better than the others. Each one is a response to a specific limit in the one before it: rules didn't scale, so we learned from data; hand-built features stalled, so we let networks learn them; classification wasn't enough, so we taught models to generate.

The mistake isn't using generative AI. The mistake is using it because it's the most visible circle, when the problem in front of you lives in an outer one. The next time someone in a meeting says "let's use AI," it's worth asking the boring questions first: what goes in, what comes out, and what's the simplest thing that could possibly work? That five-minute conversation is usually worth more than the model choice itself.

Explore It Visually

If you learn better by seeing the pieces move, I turned this whole hierarchy into an 8-scene animated walkthrough, from the confusion at the start to a complete side-by-side comparison. You can watch it straight through or jump between scenes: ML vs Deep Learning vs Generative AI on SeeItFlow.

I'm curious how it goes on your side. Have you ever seen a project reach for an LLM when a simple model would have done the job, or the reverse? Tell me in the comments.

Attention Mechanism Explained Visually: The Innovation That Made Modern AI Possible

MANGESH MANDLIK — Sat, 19 Sep 2026 12:07:42 +0000

Imagine reading a book where you forget everything you've read after every sentence.

You might still understand individual words, but following a story would be almost impossible.

Characters would appear and disappear. References wouldn't make sense. Important context would constantly be lost.

For a long time, AI models faced a similar problem.

They could process text, but maintaining context across long sequences was difficult. As sentences became longer, important information gradually faded away.

Then came a simple idea that changed the trajectory of AI:

Attention.

Today, attention sits at the heart of ChatGPT, Claude, Gemini, Copilot, and most modern AI systems. Without it, the current wave of generative AI probably wouldn't exist.

Let's see why attention became such a big deal.

The Problem Before Attention

To understand why attention matters, it's worth looking at what came before it.

Before Transformers, language models primarily relied on architectures such as RNNs and LSTMs.

These models processed text one token at a time.

Conceptually:

Token 1 → Token 2 → Token 3 → Token 4 → Token 5

Each token passed information to the next token in the sequence.

That sounds reasonable until the sequence becomes long.

Consider this sentence:

The database server crashed because it ran out of memory.

When you read the word it, you instantly know it refers to the database server.

An older sequence model has a harder job.

It has to carry information about "database server" through every intermediate step before reaching "it". As sequences grow longer, that information becomes harder to preserve.

This became known as the long-range dependency problem.

The further apart two related words were, the harder it became for the model to connect them.

The Question Every Token Wants Answered

When humans read, we don't give equal importance to every word.

Consider this sentence:

The customer could not log in because the authentication service was unavailable.

When trying to understand why the login failed, your brain naturally focuses on:

authentication service
unavailable

and largely ignores less relevant words.

We instinctively pay attention to the parts that matter.

The breakthrough idea behind attention was surprisingly simple:

What if every token could decide which other tokens are most important?

Instead of forcing information through a long chain, let each token directly look at the rest of the sequence and determine what matters.

That's attention.

Attention Is Basically Smart Context Lookup

A useful way to think about attention is as a search system running inside the model.

Every token asks:

Which other tokens should I pay attention to?

For example:

The database server crashed because it ran out of memory.

When processing:

it

the model might assign high importance to:

database
server

and lower importance to:

because
out
of

The result is that the word "it" receives information from the most relevant parts of the sentence.

Instead of treating every word equally, the model learns where to focus.

A Library Analogy

The terminology used in attention can sound intimidating at first:

Query
Key
Value

The names become easier to understand if you think of a library.

Imagine you're looking for information about PostgreSQL replication.

Your search request is the:

Query

Every book contains metadata describing what it covers.

Those descriptions act as:

Keys

Once a matching book is found, the actual content inside the book becomes the:

Value

The same idea exists inside attention.

Each token creates:

a Query (what am I looking for?)
a Key (what information do I contain?)
a Value (what information can I provide?)

The model compares Queries and Keys.

Strong matches receive more attention.

The corresponding Values contribute more information.

That's the core mechanism.

Watching Attention Happen

Let's revisit the earlier sentence:

The database server crashed because it ran out of memory.

When processing the word:

it

the model computes relevance scores against every other token.

Conceptually:

database      → high relevance
server        → high relevance
crashed       → medium relevance
because       → low relevance
out           → low relevance
memory        → medium relevance

These scores are converted into weights.

The final representation of "it" becomes a weighted combination of information from the most relevant tokens.

This allows the model to understand that "it" refers to the database server rather than some other word in the sentence.

Why This Was Revolutionary

The real breakthrough wasn't just better context.

It was that attention removed the sequential bottleneck.

Older models worked like this:

Token 1 → Token 2 → Token 3 → Token 4

Transformers work more like this:

Token 1
Token 2
Token 3
Token 4
Token 5

↓
All Tokens Attend To Each Other
↓
Processed In Parallel

Every token can interact with every other token at the same time.

This unlocked two enormous advantages:

Better Context Understanding

A token can directly access information from anywhere in the sequence.

Not just nearby words.

Massive Parallelism

Modern GPUs can process large numbers of tokens simultaneously.

Training becomes dramatically faster and more scalable.

This was one of the key reasons Transformers rapidly replaced older architectures.

Multi-Head Attention: Multiple Specialists Looking At The Same Sentence

One type of attention isn't always enough.

Language contains many different relationships.

For example:

The senior engineer fixed the production issue after reviewing the logs.

One attention pattern might focus on:

engineer ↔ fixed

Another might focus on:

issue ↔ logs

Another might focus on:

production ↔ issue

Instead of using a single attention mechanism, Transformers use multiple attention heads running in parallel.

Each head learns to focus on different relationships.

You can think of it as several specialists examining the same sentence from different perspectives.

The outputs are then combined to create a richer understanding of the sequence.

This is known as Multi-Head Attention.

Why Attention Scales So Well

One of the most important discoveries in modern AI is that larger Transformer models often become more capable when given:

More data
More parameters
More compute

Attention made that scaling possible.

Without attention, training massive language models would have been significantly more difficult.

The architecture was flexible enough to scale from millions of parameters to hundreds of billions of parameters.

That's one reason why the 2017 paper Attention Is All You Need is considered one of the most influential papers in AI history.

Where You Use Attention Every Day

Even if you've never heard the term before, you're already benefiting from attention.

ChatGPT

Attention helps connect your current prompt with relevant parts of the conversation history.

GitHub Copilot

Attention helps relate the code you're writing to functions, classes, and files elsewhere in the project.

Translation Systems

Attention links words in one language to their most relevant counterparts in another.

Semantic Search

Attention-powered models create embeddings that capture contextual meaning.

AI Agents

Attention helps combine user goals, retrieved information, previous actions, and tool outputs into a coherent response.

Why Engineers Should Care

You don't need to train foundation models to benefit from understanding attention.

Many practical AI engineering decisions become easier once you understand how models focus on information.

It helps explain:

Why context placement matters
Why retrieval quality affects RAG performance
Why some prompts work better than others
Why important information can sometimes be overlooked in long contexts

A surprising number of AI engineering problems ultimately come down to guiding attention toward the right information.

Final Thoughts

The idea behind attention is remarkably simple.

Instead of forcing information through a long chain of intermediate steps, let every token decide which other tokens matter most.

That one change transformed natural language processing.

It enabled Transformers.

Transformers enabled modern LLMs.

And modern LLMs gave us systems like ChatGPT, Claude, Gemini, and Copilot.

Not bad for a concept that started with a simple question:

Which parts of the input should I pay attention to?

Explore It Visually

Reading about attention helps.

Watching information flow between tokens makes it click much faster.

If you'd like to see self-attention, multi-head attention, token interactions, and Transformer internals animated step-by-step, check out the interactive walkthrough:

https://seeitflow.com/ai/ai-foundations/attention-mechanism

Because some engineering concepts are easier to understand when you can actually see them work.

Transformer Basics: The Architecture Behind ChatGPT, Claude and Gemini

MANGESH MANDLIK — Sat, 19 Sep 2026 11:48:15 +0000

Before 2017, most language AI systems relied on Recurrent Neural Networks (RNNs) and LSTMs.

They worked, but they had a fundamental limitation: they processed information sequentially.

Every token depended on the previous token's hidden state, making training slow, difficult to parallelize, and increasingly ineffective for long contexts.

Then came a research paper from Google:

Attention Is All You Need (Vaswani et al., 2017)

The paper introduced the Transformer, an architecture that would eventually power GPT, ChatGPT, Claude, Gemini, Copilot, and most modern AI systems.

Within a few years, Transformers replaced RNNs as the dominant architecture for natural language processing.

The Problem With RNNs

Consider this sentence:

The database server crashed because it ran out of memory.

To understand what "it" refers to, a model must connect the word it with database server several tokens earlier.

RNNs attempt to carry this information through every intermediate step.

As sequences grow longer:

Information gradually degrades
Training becomes slower
Long-range relationships become harder to preserve
GPU parallelism becomes difficult

This sequential bottleneck limited how large and capable language models could become.

The Core Idea Behind Transformers

The Transformer introduced a radically different approach.

Instead of processing one token at a time, it processes all tokens simultaneously.

Every token can directly interact with every other token.

Rather than passing information through a chain of hidden states, tokens communicate through a mechanism called self-attention.

Conceptually:

Token A  ←→ Token B
   ↑         ↓
Token C  ←→ Token D

Every token can examine the entire sequence and decide which other tokens matter most for understanding its meaning.

This creates a global view of context.

Why This Changed Everything

Processing tokens simultaneously unlocked massive parallelism.

Instead of:

Token1 → Token2 → Token3 → Token4

Transformers perform:

Token1
Token2
Token3
Token4
   ↓
Processed Together

This made it possible to:

Fully utilize modern GPUs
Train on internet-scale datasets
Build models with billions of parameters
Learn richer contextual relationships

The result was a dramatic increase in model capability.

RNNs vs Transformers

Feature	RNN / LSTM	Transformer
Processing	Sequential	Parallel
Long-context handling	Weakens over distance	Global context
GPU utilization	Limited	Excellent
Training speed	Slow	Fast
Scalability	Limited	Scales with compute
Context awareness	Local	Global

Transformers are computationally expensive because attention grows roughly with sequence length squared, but the gains in capability far outweigh the cost.

The Scaling Law Discovery

One of the most important discoveries after Transformers emerged was that performance improves predictably with scale.

Increasing:

Training data
Compute
Model parameters

consistently produced stronger models.

This observation became known as Scaling Laws.

Instead of relying solely on architectural breakthroughs, researchers discovered that simply scaling Transformer models generated substantial improvements.

This insight fueled the modern AI race.

A Short Timeline

Year	Milestone
2017	Transformer introduced in Attention Is All You Need
2018	BERT popularizes Transformer-based NLP
2019	GPT-2 demonstrates large-scale text generation
2020	GPT-3 introduces powerful few-shot learning
2022	ChatGPT brings LLMs to mainstream users
2023	GPT-4, Claude and Gemini expand multimodal capabilities
2024+	Open-source models continue closing the gap

Where You Encounter Transformers Today

Most modern AI products rely on Transformer architectures.

Examples include:

ChatGPT
Claude
Gemini
GitHub Copilot
Translation systems
Semantic search engines
Retrieval-Augmented Generation (RAG) systems

Even when the product looks different, the underlying architecture is often still a Transformer.

Why Transformers Matter

Transformers solved the sequential bottleneck that limited previous language models.

By allowing every token to interact with every other token while processing entire sequences in parallel, they unlocked the scaling behavior that made modern AI possible.

Whether you're using ChatGPT, Claude, Gemini, semantic search, RAG pipelines, or AI coding assistants, you're almost certainly benefiting from the Transformer architecture introduced in 2017.

See It In Action

Want to explore Transformer architecture visually?

https://seeitflow.com/ai/ai-foundations/transformer-basics

Vectors & Similarity Search: How AI Finds Meaning Instead of Keywords

MANGESH MANDLIK — Sat, 19 Sep 2026 11:37:37 +0000

Imagine searching for:

How do I scale PostgreSQL?

Now imagine the best document in your knowledge base is titled:

Horizontal Scaling of Relational Databases

A traditional search engine might completely miss it.

The document is clearly relevant. A human would immediately make the connection.

But keyword search doesn't think like a human.

It doesn't understand that:

PostgreSQL is a relational database
Scaling and horizontal scaling are related concepts
Sharding, replication, and scaling often appear in the same discussions

It only sees words.

Modern AI systems work differently.

Instead of searching for words, they search for meaning.

That's where vectors and similarity search come in.

They're the reason ChatGPT can retrieve relevant documents, GitHub Copilot can find related code, and AI-powered search feels dramatically smarter than traditional keyword search.

The Missing Piece After Embeddings

In the previous article, we looked at embeddings.

Embeddings convert text into vectors — long lists of numbers that capture meaning.

For example:

"PostgreSQL Replication"
→ [0.23, -0.81, 0.44, 0.67, ...]

"Database Sharding"
→ [0.19, -0.77, 0.51, 0.61, ...]

"Pizza Recipes"
→ [0.71, 0.33, -0.28, -0.52, ...]

The actual numbers aren't important.

What matters is where those vectors end up.

Concepts with similar meanings tend to be positioned close together.

Unrelated concepts end up further apart.

That raises an interesting question:

If meaning becomes a location, can we search by location instead of words?

That's exactly what similarity search does.

Think of It Like Google Maps

Imagine opening Google Maps and searching for a coffee shop nearby.

Google doesn't search for every coffee shop in the world.

Instead, it looks for locations that are close to your current coordinates.

Similarity search works the same way.

Except instead of cities and roads, we're dealing with ideas.

In this semantic map:

PostgreSQL is near replication
Replication is near scaling
Scaling is near sharding

Meanwhile:

Pizza recipes
Football scores
Travel itineraries

end up somewhere completely different.

The closer two vectors are, the more related their meanings tend to be.

Searching for Meaning Instead of Words

Let's revisit our query:

How do I scale PostgreSQL?

The search system first converts that query into an embedding.

Now it has a vector representing the meaning of the question.

Instead of scanning documents for exact words, it searches for nearby vectors.

The results might include:

PostgreSQL replication strategies
Database sharding techniques
Read replica architectures
Partitioning large databases

Notice something interesting.

The exact phrase:

scale PostgreSQL

might not appear anywhere.

Yet the results are still relevant.

That's the power of semantic search.

Why Keyword Search Isn't Enough

Keyword search is fantastic for certain problems.

If you're searching for:

Order ID 928372
User account mangesh
Error code ERR_403
Invoice number INV-1024

you want exact matches.

But humans rarely communicate using identical words.

We paraphrase.

We use synonyms.

We describe concepts in different ways.

Someone might search:

How can I speed up my database?

while the document says:

Improving PostgreSQL query performance

The meaning is similar.

The wording is not.

Keyword search struggles here.

Similarity search thrives.

What Is a Vector, Really?

The word vector sounds intimidating, but the idea is simple.

A vector is just a list of numbers.

For AI systems, those numbers represent learned patterns.

Think of a vector as a coordinate in a giant multidimensional map.

If two pieces of information are related, their coordinates tend to be nearby.

If they're unrelated, they tend to be further apart.

The embedding model learns those relationships during training.

By the time the vectors are created, similar concepts have naturally clustered together.

The Geometry Behind AI Search

One of the coolest things about embeddings is that semantic search becomes a geometry problem.

The system isn't asking:

Which document contains these words?

It's asking:

Which vectors are closest to this vector?

That's a fundamentally different approach.

Distance becomes a proxy for meaning.

Near vectors imply related concepts.

Far vectors imply unrelated concepts.

Search suddenly becomes less about language and more about mathematics.

Measuring Similarity

Of course, we need a way to determine whether two vectors are close together.

Several approaches exist, but the most common is Cosine Similarity.

Rather than measuring physical distance, cosine similarity measures how closely two vectors point in the same direction.

Conceptually:

Similar meanings
      ↘
       ↘
        ↘

High similarity score

Vectors pointing in similar directions tend to represent similar concepts.

This works surprisingly well for text, code, and many other forms of data.

Why This Becomes Difficult at Scale

Now imagine storing:

100 million document embeddings
1,536 dimensions each

A straightforward approach would compare the query vector against every single vector.

For every search.

That's a lot of math.

Even modern hardware would struggle to deliver fast results at that scale.

We need a smarter approach.

Enter Approximate Nearest Neighbor Search

This is where things get interesting.

Most AI systems don't search every vector.

Instead, they use algorithms designed to find vectors that are probably the closest.

This technique is called:

Approximate Nearest Neighbor (ANN) Search

The tradeoff is simple:

Slightly less precision
Dramatically faster retrieval

In practice, ANN search is so effective that it powers most production vector databases today.

Users get results in milliseconds instead of seconds.

Why Vector Databases Exist

Traditional databases were built for:

Transactions
Joins
Aggregations
Structured queries

They weren't designed for questions like:

Find the 20 most semantically similar documents.

Vector databases were built specifically for that problem.

Their job is to:

Store embeddings
Build efficient indexes
Run similarity searches
Filter results using metadata
Return relevant matches quickly

Popular options include:

Pinecone
Qdrant
Weaviate
Milvus
pgvector

Each approaches the problem slightly differently, but the goal remains the same:

Find nearby meaning as quickly as possible.

Where You Already Use Similarity Search

Even if you've never built a vector database, you're probably using similarity search every day.

ChatGPT and RAG Systems

When an AI assistant retrieves documents before answering a question, similarity search is often doing the retrieval.

The system embeds:

The user's question
The stored documents

and finds the closest matches.

GitHub Copilot

Copilot searches your codebase for relevant functions, files, and examples before generating suggestions.

Enterprise Search

Tools like Notion AI, Slack AI, and Confluence AI use semantic retrieval to search large collections of internal knowledge.

Recommendation Engines

Netflix, Spotify, Amazon, and many other platforms use embeddings to find content similar to what users previously enjoyed.

Different application.

Same core idea.

Find nearby vectors.

Why This Matters for RAG

Many developers first encounter similarity search while building Retrieval-Augmented Generation systems.

RAG works because:

Documents are converted into embeddings.
User questions are converted into embeddings.
Similarity search finds relevant chunks.
Those chunks are sent to the LLM.
The LLM generates a response using retrieved context.

Without similarity search, RAG would largely fall back to keyword matching.

The quality of retrieval would drop significantly.

In many RAG systems, retrieval quality matters as much as model quality.

Sometimes more.

The Bigger Idea

One of the most important shifts in modern AI is that we're increasingly moving away from exact matching.

Instead of asking:

Does this text contain these words?

we ask:

Does this information mean something similar?

Vectors and similarity search make that possible.

They allow machines to retrieve concepts rather than keywords.

That's why modern AI systems feel so much more intuitive than traditional search systems.

They're operating on meaning, not just text.

Final Thoughts

Embeddings turn meaning into numbers.

Similarity search turns those numbers into retrieval.

Vector databases make that retrieval practical at scale.

Together, they form the foundation of semantic search, RAG systems, recommendation engines, enterprise search platforms, and many of the AI products we use every day.

Once you understand vectors and similarity search, a lot of AI starts feeling less magical.

It's still impressive.

But you can finally see the engineering underneath.

Explore It Visually

Reading about vector search is useful.

Watching a query move through embeddings, vector space, ANN retrieval, and semantic search makes the whole process much easier to understand.

I've created an interactive visual walkthrough here:

https://seeitflow.com/ai/ai-foundations/vectors-similarity-search

Because some concepts click faster when you can actually see them work.

Embeddings Explained: How AI Turns Meaning Into Numbers

MANGESH MANDLIK — Sat, 19 Sep 2026 09:25:57 +0000

In the previous post, we looked at tokens — the units language models use to process text. But tokens alone don't carry meaning. To a model, they're just IDs.

That's where embeddings come in.

Embeddings are one of the most important ideas in modern AI. They power semantic search, RAG systems, recommendation engines, AI memory, code search, and much more. If you've ever searched for something and found relevant results even when the exact words didn't match, embeddings were probably involved.

Let's break down how they work.

The Problem: Computers Don't Understand Meaning

Humans naturally understand that these ideas are related:

Dog
Puppy
Golden Retriever

And we instantly know they're unrelated to:

PostgreSQL sharding
Redis caching
Kafka partitions

Computers don't have that intuition.

Traditional software only sees text as symbols. It can match exact words, but it doesn't inherently understand relationships between concepts.

That's a problem because people rarely search using the exact words that appear in documents.

Imagine a user searches:

How do I scale PostgreSQL?

Your documentation contains:

PostgreSQL Horizontal Scaling
Database Sharding Strategies
PgBouncer Connection Pooling
Scaling Postgres Databases

A keyword search might miss some of the most useful results simply because the wording differs.

Humans see the connection immediately.

Computers need a way to represent meaning mathematically.

What Is an Embedding?

An embedding is a vector of numbers that represents the meaning of some input.

That input could be:

Text
Code
Images
Audio
Documents

A simple example might look like:

"I love pizza"

↓

[0.12, -0.44, 0.89, 0.31, -0.72, ...]

Real embeddings are much larger.

For example:

Model	Dimensions
OpenAI text-embedding-3-small	1536
OpenAI text-embedding-3-large	3072
Google text-embedding-004	768
Sentence-BERT (all-MiniLM-L6)	384

The important thing isn't any individual number.

The meaning comes from the entire pattern across the vector.

Think of it like GPS coordinates.

A single coordinate doesn't tell you much. Together, all coordinates define a location.

Embeddings do the same thing for ideas.

From Tokens to Meaning

In the previous article, we saw that tokenization converts text into token IDs.

"The quick brown fox"

↓

[791, 4062, 14198, 39935]

Those IDs contain no semantic meaning.

An embedding model takes those token representations and transforms them into a dense vector that captures relationships, context, and meaning.

Conceptually:

Text
 ↓
Tokens
 ↓
Token IDs
 ↓
Embedding Model
 ↓
Vector Representation

The result is a numerical representation where similar concepts end up near each other.

The Magic: Similar Ideas End Up Close Together

This is the key property that makes embeddings useful.

Suppose we embed thousands of technical documents.

Documents about:

PostgreSQL
Database scaling
Replication
Sharding

naturally cluster together.

Documents about:

Redis
Caching
CDN strategies

form another cluster.

Documents about:

Recipes
Pizza
Baking

end up somewhere completely different.

The model was never explicitly told:

PostgreSQL and sharding are related.

It learned those relationships from patterns in training data.

As a result, semantic similarity becomes geometric distance.

In embedding space:

Near = similar meaning
Far = unrelated meaning

That's the entire foundation of semantic search.

Finding Similar Content

Once everything is represented as vectors, search becomes a geometry problem.

Instead of asking:

Which documents contain these exact words?

we ask:

Which vectors are closest to this vector?

When a user searches:

How do I scale PostgreSQL?

the query is converted into an embedding.

Then the system searches for nearby vectors among stored documents.

The closest vectors are usually the most relevant results.

Even if none of those documents contain the exact phrase "scale PostgreSQL."

This is why semantic search feels much smarter than keyword search.

How Similarity Is Measured

To compare embeddings, we need a distance metric.

The most common approaches are:

Method	Measures	Common Use
Cosine Similarity	Angle between vectors	Text similarity
Dot Product	Direction + magnitude	Normalized embeddings
Euclidean Distance	Straight-line distance	Images and some multimodal systems

In practice, cosine similarity is the most widely used for text embeddings.

It focuses on whether vectors point in the same direction rather than how large they are.

That's useful because a short document and a long document about the same topic should still be considered similar.

Why Embeddings Matter for RAG

Embeddings are one of the core building blocks of Retrieval-Augmented Generation (RAG).

The workflow looks like this:

User Question
      ↓
Create Query Embedding
      ↓
Search Vector Database
      ↓
Retrieve Relevant Chunks
      ↓
Send Context To LLM
      ↓
Generate Response

Without embeddings, retrieval would mostly rely on keyword matching.

With embeddings, retrieval becomes semantic.

That's why RAG systems can often find relevant information even when users don't use the same terminology as the source documents.

Real-World Applications

Embeddings quietly power many products people use every day.

Semantic Search

Instead of matching words, search systems match meaning.

Examples:

Enterprise knowledge bases
Documentation search
Internal company search

RAG Systems

Knowledge is embedded, stored, retrieved, and injected into LLM prompts.

Examples:

AI chatbots
Customer support assistants
Internal copilots

Recommendation Engines

Products, movies, songs, and articles can all be represented as vectors.

Examples:

Netflix recommendations
Spotify music discovery
Amazon product suggestions

AI Memory

Conversations can be embedded and stored.

Later, the system retrieves semantically relevant memories rather than searching exact text.

Code Search and Copilots

Tools like GitHub Copilot use embeddings to find relevant code context before generating suggestions.

Common Misconceptions

"Embeddings store the original text."

They don't.

Embeddings store a numerical representation of meaning, not the original text itself.

You generally cannot reconstruct the original document from an embedding vector.

"Nearby vectors mean identical meaning."

Not necessarily.

Nearby vectors indicate similar meaning.

Two documents can be closely related without being identical.

"Embeddings understand concepts like humans."

Not really.

Embeddings capture statistical patterns from training data.

They don't possess human understanding.

"More dimensions always means better embeddings."

Not always.

A well-trained 384-dimensional model can outperform a poorly trained 3000-dimensional model.

Training quality matters more than raw dimensionality.

"Embeddings are unbiased."

They aren't.

Embeddings inherit patterns and biases from the data they were trained on.

Scaling to Billions of Vectors

A small demo might compare a query against a few thousand vectors.

Production systems often store millions or billions.

Comparing every vector one by one becomes too slow.

That's why vector databases use Approximate Nearest Neighbour (ANN) algorithms such as:

HNSW
IVF
PQ

Popular vector databases include:

Pinecone
Weaviate
Qdrant
pgvector
Milvus

These systems can search enormous vector collections in milliseconds.

Key Takeaways

Embeddings convert meaning into vectors of numbers.
Similar meanings become geometrically close in embedding space.
Semantic search works by finding nearby vectors.
RAG systems depend on embeddings for retrieval.
Cosine similarity is the most common way to compare text embeddings.
Vector databases make large-scale similarity search practical.

What's Next?

Embeddings solve one problem:

How do we represent meaning as vectors?

The next challenge is:

How do we efficiently search billions of those vectors in milliseconds?

That's where vector databases come in.

If you've ever used a RAG system, AI memory feature, semantic search engine, or recommendation system, a vector database was probably working behind the scenes.

I also created an interactive visual walkthrough of embeddings and semantic search on SeeItFlow:

https://seeitflow.com/ai/ai-foundations/embeddings-explained

I'm curious — when did embeddings finally click for you?

Was it semantic search, RAG, recommendation systems, or something else?

Your LLM Has Never Read a Word: Tokenization Explained for Developers

MANGESH MANDLIK — Sat, 19 Sep 2026 09:03:40 +0000

You type a sentence into ChatGPT and it looks like normal English.

The model doesn't see it that way.

In fact, it never sees words at all. Before the model processes anything, your text is converted into a sequence of numbers. That's what the model actually works with.

When I first started learning how LLMs work, I kept hearing terms like tokens, context windows, and token limits. I understood them individually, but I didn't really appreciate how much of modern AI engineering revolves around tokens until I started building visual explanations for them.

Once you understand tokenization, a lot of things suddenly make sense:

Why AI APIs charge per token
Why long prompts cost more
Why context windows have limits
Why RAG systems need chunking
Why responses are generated gradually instead of all at once

Let's start from the beginning.

What Is a Token?

A token is the basic unit of text that a language model processes.

People often assume a token is a word, but that's not quite true. In practice, tokens are usually pieces of words.

For example, before the model sees:

Hello, world!

it gets transformed into something like:

["Hello", ",", " world", "!"]

Those tokens are then mapped to integer IDs:

[15496, 11, 995, 0]

At that point, the original text is gone. The model only sees those numbers.

A few terms are worth knowing:

Token — a text fragment that maps to an integer
Tokenizer — the component that splits text into tokens
Vocabulary — the collection of all known tokens
Context Window — the maximum number of tokens a model can process at once

The analogy that helped me most was this:

Tokens are to LLMs what bytes are to computers.

We interact with files, images, and videos, but computers ultimately operate on bytes. In the same way, we interact with words and sentences, but language models operate on tokens.

One subtle but important detail: the tokenizer is a separate preprocessing step. It is not part of the neural network itself. The same tokenizer used during training must also be used during inference.

Why Not Just Use Words?

At first, giving every word its own ID sounds reasonable.

The problem is that language is messy.

Think about words like:

play
played
playing
player

Should each be stored separately?

Then there are:

Typos
URLs
Programming code
Emojis
Newly invented words

A pure word-based vocabulary quickly becomes enormous.

The opposite approach is character-level tokenization:

c a t

That solves the vocabulary problem, but creates a different one. Sequences become much longer, making training and inference significantly less efficient.

Modern tokenizers use a compromise.

The Middle Ground: Subword Tokenization

Most modern models use some form of subword tokenization.

Common words often remain intact, while uncommon words are broken into smaller reusable pieces.

For example:

unbelievable

might become:

["un", "believ", "able"]

This allows a relatively small vocabulary to represent almost any text.

The exact algorithm varies between models. You'll often hear names like:

BPE (Byte Pair Encoding)
WordPiece
SentencePiece

The details differ, but the underlying idea is the same: reuse smaller pieces instead of storing every possible word.

See It Yourself

One of the easiest ways to understand tokenization is to experiment with it.

Using OpenAI's tiktoken library:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

text = "The quick brown fox"

ids = enc.encode(text)

print(ids)
print([enc.decode([i]) for i in ids])

Now try:

A long technical term
Source code
Emojis
Hindi text
Japanese text

You'll quickly notice that token counts vary much more than most people expect.

A sentence with the same number of characters can produce very different token counts depending on the content.

Context Windows Are Measured in Tokens

This is one of the most common misunderstandings around LLMs.

When a model advertises a large context window, that number refers to tokens, not words.

Everything shares that budget:

System prompts
Conversation history
Retrieved documents
User messages
Model responses

Once the limit is reached, something has to be removed or the request fails.

Modern models support context windows ranging from hundreds of thousands to over a million tokens, but the exact numbers change frequently as new models are released.

As a rough rule of thumb:

100 tokens ≈ 75 words
1,000 tokens ≈ 750 words
100,000 tokens ≈ 75,000 words

This becomes particularly important when you're working with large documents. A PDF that looks manageable to a human can easily consume a significant portion of the available context.

Why Tokens Matter in Production

For most engineers, tokenization becomes important when real-world constraints appear.

Cost

Most AI APIs charge based on token usage.

More input tokens increase prompt costs.

More output tokens increase generation costs.

Small inefficiencies can become surprisingly expensive when requests scale.

Latency

Longer prompts require more computation before the model can start generating a response.

Longer outputs require more generation steps.

More tokens generally means more waiting.

Memory

Long contexts require larger attention caches and more GPU memory.

This becomes one of the major engineering challenges behind large-context models.

Throughput

If each request contains fewer tokens, the same infrastructure can handle more requests.

That's why production teams often track token usage as closely as API latency.

Responses Are Generated One Token at a Time

This was probably the biggest surprise for me.

When ChatGPT produces a paragraph, it isn't generating the entire paragraph at once.

It predicts a single token.

Then another.

Each new token becomes part of the context for predicting the next one.

Conceptually:

"The capital of France is" → "Paris"
"The capital of France is Paris" → "."
"The capital of France is Paris." → " It"
"The capital of France is Paris. It" → " is"

Streaming makes the process feel instant, but under the hood it's a sequence of individual predictions.

That's one reason longer responses take longer to generate.

The 100-Page PDF Problem

This is where tokenization stops being theory and becomes a practical engineering problem.

Suppose you have a 100-page PDF and want an LLM to answer questions about it.

You can't simply send the entire document every time.

The document may contain tens of thousands of tokens, leaving little room for prompts or responses.

This is why RAG systems exist.

Instead of sending everything:

Split the document into chunks
Retrieve only relevant sections
Send those sections to the model

Once you start thinking in tokens rather than pages or words, these design decisions become much easier to reason about.

Problem	Token-Related Cause	Typical Fix
PDF won't fit	Too many tokens in one request	Chunk into token-sized pieces
API bill too high	Too many tokens per request	Compress prompts, cache, smaller models
Slow responses	Long output sequences	Stream and cap output length
Context keeps truncating	Conversation history grows	Summarize older turns
RAG results don't fit	Retrieved chunks exceed context	Keep top-K within token budget

Common Misconceptions

One Token Equals One Word

Not necessarily. Some words are one token. Others may be several.

Characters and Tokens Are the Same Thing

They're related, but not equivalent. Five characters can be one token, while a single emoji may become multiple tokens.

The Model Reads Text Directly

The model ultimately operates on numerical representations derived from token IDs.

Long Prompts Are Basically Free

Prompt length affects cost, latency, memory usage, and throughput.

Final Thoughts

The more I learn about LLMs, the more I feel that tokenization is one of the foundational concepts people skip over too quickly.

It's not just a preprocessing step.

It influences pricing, context limits, performance, retrieval systems, and even the way responses are generated.

Once you start looking at AI systems through the lens of tokens, many of the engineering trade-offs become much easier to understand.

While building the AI Foundations section on SeeItFlow, I created an animated walkthrough showing the entire flow:

Text → Tokens → Token IDs → Embeddings → Transformer → Response

Instead of reading about tokenization, you can watch the process happen step by step and see how the same input text gets transformed before reaching the model.

https://seeitflow.com/ai/ai-foundations/tokens-explained

The next logical question is:

If token IDs are just numbers, how do they become something that captures meaning?

That's where embeddings come in.

I'll cover that in the next article.

I built an interactive visualization of this entire process on SeeItFlow, where you can watch text become tokens, tokens become IDs, and responses get generated step by step.

https://seeitflow.com

What was the moment tokenization finally clicked for you? Let me know in the comments.