DEV Community: Joshua Nwachinemere

The Hidden Complexity of RAG

Joshua Nwachinemere — Sat, 25 Jul 2026 23:45:47 +0000

TL;DR

A production RAG system is not one model call wrapped around vector search. It has two pipelines—knowledge preparation and runtime retrieval—plus cross-cutting evaluation, authorization, observability, freshness, and cost controls.

If the final answer is wrong, inspect the stages separately. The root cause may be a stale source, damaged parsing, a bad chunk boundary, missed retrieval, an incorrect filter, weak reranking, noisy context, or generator overreach.

A basic retrieval-augmented generation demo can fit on a whiteboard:

Load → chunk → embed → retrieve → generate

It is a useful abstraction. It is also where many bad production decisions begin.

Imagine an employee asks:

Can contractors expense international travel in 2026?

The application retrieves a policy paragraph and the model returns a confident answer with a citation. The response looks good. It may still be wrong because:

the indexed policy expired last year;
the PDF parser separated a table from its heading;
the chunk omitted the sentence that applies specifically to contractors;
dense search missed the exact policy code;
a permissions filter exposed an internal exception;
the retrieved passages contradicted one another;
the answer added a claim that no source supported.

All of those failures can produce the same visible symptom: a fluent wrong answer.

That is the hidden complexity of RAG. The model is only the final stage in a much larger system. Production RAG is a knowledge supply chain, a search engine, a context-construction layer, an evaluation program, and a security boundary.

The difficult part is not making the pipeline run. It is knowing whether the right evidence survived every step.

RAG is better understood as two pipelines—knowledge preparation and runtime retrieval—surrounded by evaluation, security, and operational controls.

RAG starts before retrieval

Before a user asks a question, an offline pipeline has already made decisions that constrain every possible answer.

It decided which sources to trust. It parsed files. It removed or preserved structure. It split content into chunks, attached metadata, generated embeddings, and wrote records into an index. It also needs to update or delete those records when the source changes.

This is not clerical preprocessing. It is part of the model’s effective knowledge system.

Parsing is an information-loss problem

A parser can extract every visible word from a document and still destroy its meaning.

Consider a policy PDF with:

section headings that define scope;
footnotes that list exceptions;
a table whose column headers appear on another page;
scanned signatures or handwritten dates;
repeated headers and navigation text;
diagrams with labels that do not appear in the text layer.

A plain-text extractor may flatten that structure into a plausible-looking stream of words. The pipeline succeeds technically, but the relationship between those words is gone.

The fix is not always a better prompt. It may require layout-aware extraction, OCR, table handling, media-specific parsing, or a review queue for documents that fail quality checks.

The practical rule is simple:

If the ingestion pipeline loses the evidence, the retriever cannot recover it later.

Chunking is a retrieval policy

Chunk size is often treated like a configuration value to copy from a tutorial. In practice, it determines what the system can retrieve as one unit.

Chunks that are too small can separate a claim from its qualifier, title, date, or exception. Chunks that are too large can mix several topics, weaken retrieval precision, and waste the context budget.

There is no universally correct number of tokens. The right strategy depends on document structure, query patterns, retrieval method, and what counts as sufficient evidence.

Useful approaches include:

preserving headings and section paths;
keeping tables with their titles and column labels;
storing parent-child relationships;
adding controlled overlap where ideas cross boundaries;
attaching source, version, date, author, and access metadata;
using structure-aware or semantic splitting when fixed windows lose meaning.

Microsoft’s current RAG design guidance treats chunking as an experimental phase that should be evaluated against representative documents and queries, not chosen by folklore.

Chunk boundaries determine which facts, qualifiers, metadata, and table labels can be retrieved together.

Freshness requires a deletion story

Adding documents is easy. Synchronizing reality is harder.

A production knowledge pipeline needs to answer:

How quickly should a changed source appear in search?
How are renamed and duplicated documents detected?
What happens when a page is deleted?
Can an old embedding survive after its source has been revoked?
Which source wins when two versions conflict?
Can an answer show the version and effective date it used?

Without stable document identifiers, versioning, change detection, and deletion propagation, a RAG system can become a polished interface to stale information.

Temporal retrieval is harder than simply preferring the newest document. A question may refer to when an event happened, when a source was published, or when a rule was valid. The 2024 TempRALM study is useful evidence that semantic relevance and temporal relevance need separate treatment—but its reported gains come from specific temporal benchmarks, not a universal production recipe.

Retrieval is a ranking pipeline, not a database lookup

After ingestion, the online system has a different job: translate a user’s language into a ranked set of useful evidence.

That involves more than nearest-neighbor search.

Dense retrieval is useful, not magical

Embedding search is good at matching related meanings even when a query and a document use different words. It can still miss exact identifiers, product names, acronyms, dates, and rare terms.

If the user asks about policy TRV-2026-07, lexical search may recognize the exact string more reliably than a semantic vector alone. If the user says “the overseas contractor expense rule,” dense retrieval may be more helpful.

This is why hybrid retrieval remains important. Current Azure AI Search documentation describes running full-text and vector queries in parallel and merging their results with Reciprocal Rank Fusion. The larger lesson is vendor-independent: lexical and semantic signals fail differently, so combining them can be stronger than assuming one replaces the other.

Filters are part of relevance—and security

Metadata filters narrow the candidate set by attributes such as region, product, language, document type, effective date, or tenant.

They can improve quality, but they can also remove the correct answer when metadata is missing or wrong. The timing of a filter—before or after vector search—can also affect recall and latency.

Current Azure vector-filter guidance makes that tradeoff explicit: filter mode, selectivity, index structure, and candidate count can change both recall and performance. This is implementation-specific guidance, but the general lesson travels well—filters belong in retrieval evaluation, not only in application logic.

Permission filters are more serious. They must ensure that content the user cannot access does not enter the retrieved candidate set. Telling the model “do not reveal confidential information” is not access control.

If an unauthorized chunk reaches the prompt, the security failure has already happened.

Reranking is a separate model decision

A first-stage retriever is optimized to find plausible candidates quickly. A reranker spends more computation comparing the query with a smaller candidate set and reorders those results.

That can improve relevance, but it introduces new choices:

How many candidates enter the reranker?
Which fields does it see?
How much latency and cost does it add?
Does it work for the domain and language?
Does improved ranking actually improve final answers?

Current semantic-ranking documentation makes those constraints concrete: reranking operates over an initial ranked set and has input limits. “Add a reranker” is therefore not a free quality upgrade. It is another stage to benchmark.

Query rewriting can help—or quietly change the question

Real questions are messy. They contain typos, pronouns, missing context, and several requests in one sentence.

A query layer may expand synonyms, rewrite the question, decompose it into subqueries, or use conversation history. This can improve coverage for multi-part and multi-hop questions. It can also drift from the user’s intent, retrieve too broadly, or multiply latency and cost.

Current agentic retrieval guidance describes decomposition, parallel subqueries, reranking, and source tracking. It also states that agentic retrieval adds latency, and some capabilities remain preview features.

Agentic RAG is not simply “advanced RAG.” It is a trade: more adaptive retrieval in exchange for more planning, nondeterminism, cost, failure modes, and observability work.

Good retrieval narrows a broad, authorized candidate set into a small evidence set while measuring the tradeoffs at every transition.

Context construction decides what the model is allowed to see

Retrieval produces candidates. Those candidates should not automatically become the prompt.

A context-construction layer may need to:

remove duplicates and near-duplicates;
keep source diversity;
attach neighboring or parent sections;
preserve titles, dates, and provenance;
detect conflicting versions;
fit evidence within a token budget;
order passages intentionally;
create stable citation anchors;
exclude low-confidence or unauthorized material.

This matters because more context is not always better.

The “Lost in the Middle” study found that language models can use relevant information less reliably depending on where it appears in a long input. Separately, an EMNLP 2024 comparison of RAG and long-context models found a real quality-cost tradeoff rather than a universal winner.

A larger context window does not remove the need to select, organize, and evaluate evidence. It merely raises the amount of information the model can potentially process—and the amount of noise you can send it.

Generation cannot repair missing evidence

Once the context reaches the language model, prompting still matters. The model should know what task it is performing, what sources it may use, what output structure is required, and when it must abstain.

But prompting cannot make absent evidence appear.

Useful generation controls include:

require claims to be supported by supplied evidence;
return stable citation identifiers;
distinguish quoted facts from inference;
state when sources conflict;
abstain when evidence is insufficient;
validate required fields and allowed values;
verify that cited passages actually support the answer;
route consequential outputs to human review.

None of these is a perfect “hallucination detector.” Groundedness checks and LLM judges can also make mistakes. For high-impact tasks, deterministic rules, source verification, and human review still matter.

The same fluent wrong answer can begin in the source, parser, chunker, index, retriever, context builder, or generator.

Evaluation must tell you where the failure happened

A single end-to-end score is not enough.

Suppose the correct answer is absent from the response. Several explanations are possible:

The source never contained it.
The parser dropped it.
The chunker separated it from necessary context.
The retriever failed to find it.
A filter removed it.
The reranker pushed it down.
Context assembly excluded it.
The model ignored or misread it.
The evaluator marked a correct answer as wrong.

If evaluation only inspects the final text, the team may tune the prompt for a retrieval failure or replace the embedding model for a parsing failure.

Recent work reflects this need for diagnosis. The 2024 RAGChecker paper separates retrieval and generation diagnostics. MIRAGE, published in Findings of NAACL 2025, evaluates dimensions including noise vulnerability, context acceptability, context insensitivity, and context misinterpretation.

A useful evaluation program works at several levels.

Ingestion

Were all expected files processed?
Were headings, tables, dates, and relationships preserved?
Is metadata accurate?
Did updates and deletions propagate?

Retrieval

Does the candidate set contain the evidence?
How often does the correct passage appear in the top k?
Are exact terms and semantic paraphrases both handled?
Do filters preserve relevant authorized results?
Does reranking improve measured relevance?

Metrics such as Recall@k, Precision@k, mean reciprocal rank, and nDCG can help, depending on the task and labels. None is sufficient by itself.

Context and generation

Does the assembled context cover the required evidence?
Is it redundant, noisy, or contradictory?
Are answer claims supported by the context?
Are citations correct?
Does the system abstain when the corpus cannot answer?

System behavior

Task success with real users
Latency percentiles rather than averages alone
Cost per successful answer
Freshness and deletion service levels
Permission failures and security events
Failure rates by document type, language, tenant, and query class

The test set matters as much as the metric. It should include answerable, unanswerable, ambiguous, exact-identifier, multi-hop, conflicting-source, freshness-sensitive, permission-sensitive, and adversarial questions.

End-to-end scores show whether the system failed; stage-level contracts, versions, and traces help explain where and why.

LLM-based evaluators are useful for scaling review, but they are models—not ground truth. Calibrate them against human labels, monitor disagreement, and keep deterministic checks where possible.

Security is not a filter you add at the end

RAG connects a model to data that may be private, user-generated, or externally controlled. That makes retrieval a security boundary.

Two risks are especially easy to underestimate.

Indirect prompt injection

A retrieved document can contain instructions aimed at the model rather than information for the user. Those instructions may be hidden in webpages, files, tickets, emails, or poisoned knowledge-base content.

OWASP’s current prompt-injection guidance explicitly covers indirect injection from external sources. Retrieved text should therefore be treated as untrusted data, not trusted system instructions.

Embedding and index weaknesses

An attacker—or a simple ingestion bug—can introduce misleading records, manipulate similarity, leak sensitive relationships, or break tenant isolation. OWASP’s vector and embedding guidance discusses poisoning, unauthorized access, and data leakage in RAG-related systems.

The risk is not merely theoretical. PoisonedRAG, published at USENIX Security 2025, demonstrates targeted knowledge-base poisoning under specific experimental threat models. Its attack rates should not be generalized to every corpus or retriever, but the study establishes that retrieved knowledge can be an attack surface—not just a quality problem.

Practical controls include:

source allowlists and provenance;
retrieval-time authorization;
tenant isolation;
sanitization and content classification;
audit logs for sources and citations;
permission-aware caches;
index update and deletion controls;
adversarial evaluation;
output validation and human approval for consequential actions.

Then comes the operational tax

A production request may involve authentication, policy checks, query rewriting, several searches, embedding calls, reranking, context assembly, generation, citation validation, and logging.

Each stage adds latency and a new way to fail.

The system needs more than an overall timer. Trace each stage so a slow answer can be attributed to the search service, a model call, a retry, an oversized candidate set, or an external source.

The same applies to cost. Track the contribution from:

parsing and OCR;
embedding new or changed content;
storage and indexing;
query embeddings;
multiple retrieval calls;
reranking;
generation tokens;
evaluation and monitoring.

Caching can help, but a cache key that ignores tenant, permissions, source version, or model configuration can return stale or unauthorized information.

Version the components that affect behavior: parser, chunker, embedding model, index schema, retrieval settings, reranker, prompt, generator, and evaluation dataset. Otherwise, a quality regression becomes an archaeological investigation.

A diagnostic matrix for wrong answers

Symptom	First place to inspect	Useful evidence
Correct passage never appears	Parsing, chunking, retrieval	Parse samples, chunk boundaries, Recall@k
Exact code/name is missed	Lexical and hybrid retrieval	BM25 baseline, hybrid result set
Relevant passage ranks too low	Candidate depth and reranker	Before/after rankings, nDCG or MRR
Wrong tenant or restricted text appears	Authorization and filters	ACL evaluation, retrieval trace, cache key
Answer ignores strong evidence	Context order and generation	Final context, citation support, model trace
Old answer persists after update	Sync and deletion pipeline	Source version, index version, deletion log
Quality drops after a release	Component versioning	Dataset run by parser/index/prompt/model version

A practical build order

The safest way to build RAG is not to add every advanced technique. It is to add complexity only when evidence identifies a failure it can solve.

Define the task. Specify authoritative sources, users, permissions, freshness requirements, and what a successful answer means.
Build the evaluation set early. Use representative documents and questions, including questions the corpus cannot answer.
Make ingestion observable. Inspect parsing, chunk boundaries, metadata, versions, and deletions before tuning retrieval.
Establish retrieval baselines. Test lexical and dense retrieval, then measure whether hybrid retrieval improves the actual query set.
Add reranking only after measurement. Record quality, latency, and cost changes.
Engineer context deliberately. Handle duplicates, source conflicts, ordering, citations, and abstention.
Evaluate components and the whole system. A good retriever can still feed a model that overreaches; a grounded generator cannot use evidence it never receives.
Test security and failure behavior. Include permission boundaries, poisoned content, timeouts, partial failures, and stale data.
Introduce agentic retrieval selectively. Use it for demonstrated query classes that need decomposition, iteration, or dynamic source selection.
Re-run evaluation after every material change. A new parser, embedding model, index, prompt, or generator changes the system.

The real mental model

RAG is often presented as a way to give an LLM access to your data. That description hides the engineering responsibility.

A better definition is:

RAG is a controlled process for turning changing, permissioned sources into evidence that a generative model can use—and for measuring whether the resulting answer remains supported.

The hard part is not the connection between the model and the database. It is maintaining a trustworthy path from source to evidence to answer, while being able to explain exactly where that path failed.

20 Most Important AI Concepts Explained in Just 20 Minutes

Joshua Nwachinemere — Sat, 25 Jul 2026 21:42:37 +0000

A plain-English guide to the ideas behind ChatGPT, recommendation systems, AI agents, and the models changing how software is built.

Artificial intelligence can feel harder than it is because the same conversation often mixes computer science, statistics, product design, and science fiction.

One person says "AI" and means ChatGPT. Another means a fraud detector. A researcher may be talking about a training method, while a startup founder is describing an API wrapped in a polished interface.

You do not need a mathematics degree to separate these ideas. You need a map.

The 20 concepts below form that map. They explain how modern AI systems learn, how tools such as ChatGPT generate answers, why those answers can be wrong, and what engineers do to make AI useful in the real world.

Read one section per minute. By the end, most AI conversations should sound much less mysterious.

The quick map

Foundations: AI, machine learning, deep learning, neural networks, training, and inference
Ways models learn: supervised, unsupervised, self-supervised, and reinforcement learning
Modern model stack: foundation models, generative AI, LLMs, transformers, tokens, and context windows
Building real applications: embeddings, prompting, RAG, fine-tuning, multimodal AI, agents, evaluation, and guardrails

The concepts are easier to remember when you see how they connect.

1. Artificial intelligence

Artificial intelligence is the broad field of building machines that perform tasks we associate with human intelligence.

Those tasks include recognizing speech, understanding language, spotting patterns, planning actions, making predictions, and generating images or text. The definition is intentionally broad. A chess engine, a spam filter, and a voice assistant can all be called AI even though they work in very different ways.

AI does not automatically mean consciousness or human-like reasoning. Most systems are narrow: they perform a bounded task under specific conditions. A model that detects tumors in scans cannot drive a car unless someone builds and trains a separate system for driving.

Think of AI as the name of the entire field. The remaining concepts describe different ways of building systems inside it.

2. Machine learning

Traditional software follows rules written by a programmer. Machine learning takes a different route: the programmer gives the computer examples, and an algorithm learns a pattern from them.

Suppose you want to identify fraudulent card transactions. Writing a complete rulebook is nearly impossible because fraud changes constantly. With machine learning, you provide historical transactions and indicate which were fraudulent. The model studies relationships among amount, location, timing, device, and other signals. It then estimates the likelihood that a new transaction is suspicious.

The result is not a database of memorized answers. It is a mathematical function shaped by data.

Machine learning works best when useful patterns exist in the examples. If the data is incomplete, biased, mislabeled, or unrelated to the problem, the model learns the wrong lesson with impressive efficiency.

3. Deep learning

Deep learning is a branch of machine learning built around neural networks with many processing layers.

Earlier machine-learning systems often depended on people to choose the important features. An engineer building an image classifier might manually describe edges, shapes, and colors. A deep-learning model can learn useful representations directly from raw pixels. Early layers may respond to edges, later layers to textures and parts, and deeper layers to larger visual patterns.

This ability to learn representations helped drive major progress in computer vision, speech recognition, translation, and generative AI. It also comes at a cost. Deep models usually require more data, computing power, and energy than simpler methods. Their internal decisions can be difficult to interpret.

Deep learning is powerful, but it is not always the right answer. A small, well-structured dataset may be handled better by a simpler model that is cheaper and easier to explain.

4. Neural networks

A neural network is a collection of connected mathematical units arranged in layers. Each connection has a number called a weight.

An input enters the network. The layers transform it, using the weights, until the network produces an output. During learning, the system compares that output with the desired result and adjusts the weights to reduce the error.

Imagine a network deciding whether an email is spam. Words, links, sender information, and formatting become numerical inputs. The network combines those signals through its layers and returns a probability. A result of 0.97 might mean the model estimates a 97% chance that the message is spam.

The word "neural" comes from a loose analogy with biological neurons, but modern neural networks are mathematical systems, not digital brains. The analogy is useful historically and often misleading technically.

5. Training and inference

Every deployed model has two distinct phases: training and inference.

Training is when the model learns. It processes examples, measures its errors, and updates its internal weights. Training a large model can involve enormous datasets, thousands of specialized chips, and weeks or months of computation.

Inference is when the trained model is used. You provide a new input and the model produces a prediction or response. Each time you send a prompt to a chatbot, ask a phone to transcribe audio, or use an AI image generator, you are running inference.

The distinction matters because the engineering problems differ. Training focuses on data quality, optimization, compute, and reproducibility. Inference focuses on speed, cost, memory, reliability, and handling many users at once.

When a chatbot learns something from the current conversation, it usually has not retrained its weights. It is using the conversation as temporary context during inference.

6. Supervised learning

Supervised learning trains a model with examples that include correct answers, called labels.

A dataset might contain house details paired with sale prices, medical images paired with diagnoses, or customer messages paired with categories such as billing, cancellation, and technical support. The model learns a mapping from each input to its label.

Two common supervised-learning tasks are classification and regression. Classification chooses a category: spam or not spam, cat or dog, approved or rejected. Regression predicts a number: tomorrow's energy demand, a delivery time, or a home's likely price.

The hard part is often not choosing the algorithm. It is collecting reliable labels. Human annotators disagree. Historical decisions may encode discrimination. Labels can become outdated as the world changes.

A supervised model can reproduce those problems because it learns what happened in the dataset, not what should have happened.

7. Unsupervised and self-supervised learning

Unsupervised learning looks for structure in data that has no human-provided labels.

A retailer might group customers by purchasing behavior without deciding the groups in advance. A security system might flag activity that looks different from normal traffic. Common techniques include clustering, dimensionality reduction, and anomaly detection.

Self-supervised learning is closely related but creates a learning task from the data itself. A language model can hide part of a sentence and learn to predict it, or predict the next token from everything that came before. An image model can hide sections of an image and learn to reconstruct them.

This matters because unlabeled data is abundant. The internet contains far more text, images, video, and audio than humans could manually label.

Much of modern generative AI starts with self-supervised learning at scale, followed by additional training that makes the model more useful for particular tasks.

8. Reinforcement learning

Reinforcement learning trains an agent through actions and rewards.

The agent observes an environment, chooses an action, receives feedback, and updates its strategy. A game-playing agent may receive a positive reward for winning. A warehouse robot may be rewarded for moving items efficiently while avoiding collisions.

Unlike supervised learning, the correct action is not necessarily supplied for every situation. The agent must explore and discover which sequences of actions lead to better outcomes. This creates a difficult tradeoff between trying what already works and testing something new.

Rewards must be designed carefully. An agent will optimize the measurable objective, even when that objective is a poor substitute for what people intended. Give a cleaning robot points for collecting dirt and it might learn that spilling dirt creates more opportunities to score.

That failure pattern is called reward hacking. It is one reason reinforcement learning in the real world needs constraints and monitoring.

9. Foundation models

A foundation model is trained on broad data and can be adapted to many tasks.

Instead of building one model for summarization, another for translation, and another for question answering, developers can start with a general model and adapt it through prompts, retrieval, fine-tuning, or additional tools.

Large language models are one type of foundation model. Others work primarily with images, audio, video, or several forms of data together.

The word "foundation" does not mean the model is complete or universally intelligent. It means other applications can be built on top of it. A foundation model is usually one component in a larger system that also contains instructions, business rules, databases, APIs, user interfaces, and safety controls.

This reuse makes powerful capabilities accessible, but it also concentrates risk. A weakness inherited from one foundation model can affect many products that depend on it.

10. Generative AI

Generative AI creates new content by learning patterns from existing data.

Depending on the model, the output may be text, code, images, speech, music, video, or structured data. The system does not normally retrieve a finished answer from a giant archive. It generates an output step by step according to patterns encoded in its parameters and the information supplied at inference time.

"New" does not mean wholly original. Generated content can resemble training examples, repeat common clichés, or reproduce biases found in the underlying data. Models may also memorize and reveal fragments in some circumstances.

Generative systems are probabilistic. Give the same prompt twice and you may receive different results. Settings such as temperature alter how conservative or varied the output is.

This variability is useful for brainstorming and creative work. It is a liability when a system must return the same verified answer every time.

11. Large language models

A large language model, or LLM, is trained to model patterns in language, usually by predicting the next token in a sequence.

Given "The capital of France is", the model assigns probabilities to possible continuations. "Paris" receives a high probability because the training process has shaped the model's weights around patterns found across enormous amounts of text.

Next-token prediction sounds simple, but doing it at scale forces a model to learn grammar, style, facts, code patterns, and some relationships among concepts. This produces surprisingly general behavior: summarization, translation, drafting, question answering, and programming assistance can emerge from the same base model.

An LLM still does not consult a guaranteed internal database before answering. It produces a plausible continuation based on learned patterns and current context. Fluency is therefore evidence of language skill, not proof of factual accuracy.

12. Transformers and attention

Most modern LLMs use an architecture called the transformer, introduced in the 2017 paper "Attention Is All You Need."

Its central mechanism, attention, lets the model weigh relationships among different parts of an input. In the sentence "The trophy would not fit in the suitcase because it was too large," attention helps the model connect "it" with "trophy" rather than "suitcase."

Transformers process many positions in parallel during training, unlike older sequence models that handled text mainly one step at a time. This made them easier to scale on modern hardware.

Attention is not human concentration. It is a numerical operation that calculates which representations should influence one another and by how much.

The same architecture now appears beyond text. Transformers process images, audio, biological sequences, and combinations of several data types.

13. Tokens and context windows

Language models do not read text as words. They process tokens.

A token may be a whole word, part of a word, punctuation, or a short character sequence. "Unbelievable" might be represented as one token or split into pieces, depending on the tokenizer. Code, uncommon names, and some languages may require more tokens than ordinary English prose.

The context window is the amount of tokenized information a model can consider in one request. It may include your prompt, previous messages, retrieved documents, tool results, and the model's response.

A larger context window allows a system to handle longer documents and conversations, but it does not guarantee perfect recall. Important details can still be overlooked, drowned in irrelevant material, or contradicted elsewhere in the prompt.

Tokens also affect cost and speed because model providers usually measure usage by the number of input and output tokens processed.

14. Embeddings and vector search

An embedding is a list of numbers that represents the meaning or other useful properties of an item.

Text with similar meaning tends to have nearby embeddings even when it uses different words. "How do I reset my password?" and "I cannot log in because I forgot my credentials" may be close in embedding space.

Vector search compares these numerical representations to find similar items. It powers semantic search, recommendation systems, duplicate detection, clustering, and document retrieval.

A vector database stores embeddings and searches them efficiently. Despite the name, it does not understand meaning by itself. The usefulness of the results depends on the embedding model, the source data, how documents were divided, metadata filters, and the definition of similarity.

Embeddings turn fuzzy questions about meaning into geometry that software can compute, but the geometry remains an approximation.

15. Prompting

A prompt is the input that tells a generative model what to do and provides the context it should use.

Good prompts reduce ambiguity. They state the task, relevant background, constraints, desired format, and sometimes examples. "Summarize this" is weaker than "Summarize this incident report for an engineering manager in five bullets. Separate confirmed facts from unresolved questions."

Prompt engineering is useful, but it is not magic. A prompt cannot guarantee facts the model does not have, enforce permissions, or replace deterministic validation. If an application requires valid JSON, the system should parse and validate the output. If a user is not allowed to refund a payment, access control should block the action regardless of what the prompt says.

Prompts guide model behavior. Software controls must enforce the rules that matter.

16. Retrieval-augmented generation

Retrieval-augmented generation, usually shortened to RAG, gives a model relevant information before it answers.

A typical RAG system splits documents into searchable chunks, converts them into embeddings, retrieves the chunks closest to a user's question, and adds them to the model's context. The model then writes an answer using that material.

This helps when the required information is private, specialized, or newer than the model's training data. A support assistant can search product manuals. An internal tool can answer questions from company policies without retraining the underlying model every time a policy changes.

RAG does not automatically eliminate incorrect answers. Retrieval may return the wrong passage, miss an important document, or surface outdated material. The model can misunderstand good evidence.

Reliable RAG systems test retrieval and answer quality separately, preserve citations, track document versions, and allow the model to say when the evidence is insufficient.

17. Fine-tuning

Fine-tuning continues training an existing model on a smaller, targeted dataset.

It can teach a model a consistent output format, specialized vocabulary, a particular style, or better behavior on a narrow task. Because the starting model already knows broad patterns, fine-tuning usually requires far less data and compute than training from scratch.

Fine-tuning is often confused with adding knowledge. It can help a model internalize recurring patterns, but it is a poor way to maintain frequently changing facts. Updating a policy document in a RAG index is much easier than retraining a model whenever the policy changes.

A useful rule is:

Use prompting when instructions are enough.
Use retrieval when the model needs external or changing knowledge.
Use fine-tuning when behavior must change consistently across many examples.

Real systems often combine all three.

18. Multimodal AI

A multimodal model can process or generate more than one type of data, such as text, images, audio, or video.

A multimodal assistant might inspect a chart, discuss it in text, listen to a spoken follow-up question, and answer aloud. A document-processing system can combine page layout, printed words, handwriting, and tables rather than treating a PDF as plain text.

The difficult part is connecting representations across modalities. The model must learn that pixels showing a red traffic light, the phrase "red light," and the sound of someone saying those words refer to related ideas.

Multimodal does not mean equally capable in every mode. A model may describe photographs well but struggle with tiny labels, spatial measurements, long videos, or noisy audio. Each modality needs separate evaluation because a strong language score says little about visual accuracy.

19. AI agents and tool use

An AI agent combines a model with tools, state, and a loop for deciding what to do next.

A plain chatbot returns text. An agent may search the web, read files, query a database, run code, call an API, inspect the result, and continue until it reaches a goal.

The model usually acts as the planner, while ordinary software performs the actions. This distinction matters. The model can suggest "send the refund," but a payment API with real credentials is what moves the money.

Agents become risky when a probabilistic model controls consequential tools without tight boundaries. A production agent needs least-privilege permissions, input validation, timeouts, duplicate-action protection, logs, spending limits, and human approval for irreversible steps.

The impressive part of an agent demo is often its autonomy. The useful part of a deployed agent is controlled autonomy: enough freedom to finish the job, but not enough to create a new one.

20. Evaluation, hallucinations, and guardrails

A hallucination is an output that sounds plausible but is unsupported or false. Language models are especially prone to this because their basic objective rewards likely continuations, not verified truth.

Evaluation measures how well an AI system performs. A useful evaluation set reflects the real task, includes difficult and failure cases, and scores what users care about. That may mean factual accuracy, retrieval quality, latency, cost, bias, safety, or whether an agent completed an action without harmful side effects.

Guardrails are the controls around the model: schema validation, content filters, permission checks, source requirements, rate limits, human approval, and monitoring. They reduce risk but do not turn an unreliable model into a guaranteed one.

The application, not just the model, must be evaluated. A strong model can fail inside a poor retrieval pipeline. A weaker model can perform well when given clean data, narrow tools, and clear checks.

Treat model output as untrusted until the system has verified everything that matters.

How the pieces fit together

In production, the model is one component inside a controlled software system.

Consider an internal support assistant.

A foundation model provides the language capability. The application converts company documents into embeddings and stores them for vector search. When an employee asks a question, a RAG pipeline retrieves relevant policy passages and places them inside the model's context window. A prompt tells the model to answer only from those passages and include citations.

If the assistant needs to check an order, an agent tool calls the order database with read-only permission. The final response passes through validation and logging. Evaluations measure whether retrieval found the correct policy, whether the answer matched the evidence, and whether the tool respected access rules.

No single concept makes that system work. The value comes from how the parts are connected.

A mental model worth keeping

AI is the broad field. Machine learning learns patterns from data. Deep learning uses layered neural networks. Foundation models provide general capabilities. Transformers and attention power many of today's language systems. Tokens and context define what a model can process at once. Embeddings and retrieval connect models to external information. Fine-tuning changes recurring behavior. Agents connect models to actions. Evaluation and guardrails determine whether the complete system is trustworthy enough to use.

You do not need to memorize every term. Remember the boundaries:

A fluent answer is not necessarily a factual answer.
A large context window is not perfect memory.
Retrieval is not guaranteed understanding.
Fine-tuning is not a live database.
An agent is not safe merely because its instructions say "be careful."
A model demo is not the same as a reliable product.

Once those distinctions are clear, AI becomes easier to reason about. It stops looking like one mysterious technology and starts looking like what it is: a collection of models, data pipelines, software controls, and human decisions.