If you're coming into AI agents from software engineering, you've probably run into LLM, Embedding, Vector Database, RAG, Tool Calling, LangChain, LangGraph, and MCP as separate topics. Different blog posts, different tutorials, different mental models. That's the problem. In a real system, all of these are pieces of one architecture, and until you see how they connect, you're just memorizing framework APIs.
Here's the piece that matters most: building an agent is not a prompting trick. It's a systems engineering problem. The concerns that already define backend work don't disappear because there's an LLM involved. You just get one more component in the system, and this one is non-deterministic.
The LLM is a processing service with a limited input
Strip away the hype and an LLM is close to a service call:
Input → LLM → Output
Except the input isn't a plain string. The model works in tokens, and there's a hard limit on how many tokens it can process per request. You can't dump an entire database or every file in a company into a prompt and hope for the best. That's the first real constraint on any AI system's architecture:
Application → Context → LLM
The job becomes deciding what context the model actually gets.
Context window is working memory, and managing it is architecture
Think of the context window like a process's working memory. In a real request it usually has to hold the system prompt, conversation history, the user's query, retrieved documents, and tool results, all at once. If that exceeds the model's capacity, you can't send everything. So managing context becomes its own design problem:
Conversation → Select relevant history → Retrieve relevant documents → Build context → LLM
This is the seed of everything that follows. RAG and agents are both, in different ways, answers to "what goes in the context window."
Embeddings and vector search: finding things by meaning
An LLM is good at generating text, but it's not the right tool for searching a large body of information by meaning. That's what embeddings are for: they turn text into a numeric vector. "How do I reset my password?" becomes something like [0.12, -0.44, 0.81, ...]. Two pieces of text with similar meaning end up with vectors that are close together: "How can I reset my password?" and "I forgot my password, what should I do?" land near each other even though the wording is different. That's the whole basis of semantic search.
Once you have embeddings for a large set of documents, you need somewhere to store and query them — a vector database. The structure is simple: each document has its text, its metadata, and its embedding. A user query gets embedded the same way, and vector search returns the documents closest to it in that space. You're no longer matching keywords; you're matching meaning.
Chunking, and why the boundary between chunks matters
You don't embed a 200-page PDF as one vector. You split it into chunks — an employee handbook becomes a leave policy, a remote work policy, a salary policy, a security policy, each with its own embedding.
The one detail worth remembering: if you cut a document at an arbitrary point, you can lose context right at the boundary. Say one chunk ends with "Employees can request annual leave after completing their probation period..." and the next starts with "The probation period is normally three months..." Split cleanly, and the connection between those two facts disappears. That's why chunks usually overlap a bit at the edges, so information doesn't get lost exactly where you cut.
RAG is a pipeline, not a database
Now the pieces connect. Retrieval-Augmented Generation means: before the LLM answers, go find relevant information from an external source and add it to the context.
User Query → Embedding → Vector Search → Relevant Chunks → Prompt → LLM → Answer
Someone asks "how many days of annual leave do employees get?" The system embeds the question, searches the vector store, finds the leave policy chunk, adds it to the prompt, and the LLM answers from that context instead of guessing.
It's worth being precise here, because the terms get used interchangeably and they shouldn't be. A vector database is storage plus vector search. Nothing more. RAG is a pipeline: embed, retrieve, build context, prompt, generate. A vector DB can be part of a RAG system, but the two aren't the same thing.
Prompt engineering is application logic
Even with the right document retrieved, you still have to hand it to the model correctly — system instructions, retrieved context, then the question. The basic patterns here are zero-shot (no examples, just "classify this ticket as billing, technical, or account"), one-shot (one example first), and few-shot (several examples before the real input). None of this is exotic, but it's worth treating seriously: the prompt isn't a string you write once and forget. It's part of the application's logic, the same way a SQL query or a validation rule is.
An LLM alone is not an agent
This is where the real distinction starts. A plain LLM call is input → LLM → output, once. An agent runs a loop:
User Request → LLM → Decide what to do → Tool → Tool Result → LLM → Decide again → ... → Final Answer
An agent isn't "an LLM with a better prompt." It's a system that decides, based on its current state and goal, what action to take next.
Take a question like "what's the current price of this product?" The LLM doesn't know that. It has no live access to your database. So you give it a tool: get_product_price(product_id), or search_database(query), or send_email(...). The LLM decides it needs the price, calls the tool, gets a result, and uses that to answer. The LLM is the decision maker; the tool does the actual work.
Zoomed out, an agent is an observe → reason → act loop. For something simple, like checking the weather in Tehran, that's: understand the request, pick the weather tool, call it, get the result, respond. For something harder, it might be: search a database, analyze the result, call another API, compare results, produce a final answer. That's when "agent" stops being a buzzword and starts being an actual engineering problem.
LangChain, Chains, and LangGraph
As these systems grow, wiring every piece together by hand (LLM, retriever, vector DB, tool, parser, memory) gets unwieldy. LangChain is a set of abstractions for building this kind of application so you're not connecting everything from scratch. It's worth being clear about what it is, though: LangChain isn't AI. It's a framework for building LLM-based applications.
A Chain is a pipeline of fixed steps — question → prompt → LLM → parser → result, or question → retriever → documents → prompt → LLM → answer. That's fine when the path is mostly known in advance. Agents usually aren't like that; you don't know the next step ahead of time.
That's what LangGraph is for. Once a workflow branches — analyze, then either search a database or call an API depending on what's needed, then decide, then respond — a simple chain isn't enough. You have state, nodes, and edges. LangGraph models this as something closer to a state machine or a graph: state, a node processes it, the state updates, a conditional edge decides where to go next. Understanding that model matters more than memorizing LangGraph's API surface.
MCP: standardizing how an agent reaches tools
If an agent needs to talk to a calculator, a weather API, a database, a filesystem, GitHub, and some internal API, and each one needs its own custom integration, the architecture gets messy fast. The Model Context Protocol standardizes that connection. An MCP server exposes a set of tools — a Weather server might expose get_weather(city) and get_forecast(city); a Company Knowledge server might expose search_documents(query) and get_document(id). The agent doesn't need to know how the tool is implemented internally.
From an architecture standpoint, MCP is a standard boundary between the AI application and everything external to it. That separation is what lets you add capabilities without touching the agent's core logic: today it's a calculator and a weather API, tomorrow it's GitHub, Postgres, the filesystem, and a handful of internal services, and the reasoning loop itself doesn't have to change.
Putting it together: an org knowledge search example
Say a company has 100,000 documents — HR policies, engineering docs, contracts, product docs, security policies. Someone asks: "do I need my manager's approval for this kind of request?"
The system embeds the question, searches the vector database for the top relevant chunks, builds a prompt from the system instructions, the question, and those chunks, and sends it to the LLM. If answering fully requires something the documents don't have — say, the employee's own approval level — the agent can call a tool, hit an employee service, get the result, and feed it back to the LLM. That's a plain RAG system turning into an agent the moment it needs to act instead of just retrieve.
The difference between RAG and agent is worth stating plainly. RAG says: find relevant information and hand it to the model. Agent says: based on the goal and current state, decide what to do. RAG can be one of an agent's capabilities — retrieval is just one action among several it might choose.
An agent is not a magical entity: it's a small distributed system
It's easy to talk about an agent as if it's an independent thing that "does stuff." It isn't. It's a set of components: LLM, prompt, state, memory/context, tools, retrieval, external services, control flow. An agent is closer to an application architecture than to a model.
That framing is the useful one for a software engineer, because it means the classic problems show back up: authentication, authorization, retries, timeouts, rate limiting, logging, observability, caching, state management, error handling, idempotency, security, cost control. None of that disappears because there's an LLM in the loop. If anything, it matters more, because now one of your components is non-deterministic.
This is, honestly, close to what I've spent the last while building at work, minus the LLM. An approval workflow with a real task inbox, an append-only audit log, a scheduler driving both background jobs and workflow timing, human-in-the-loop suspend/resume — that's state management, idempotency, and control flow, the exact same concerns an agent needs, just without a model deciding the next step. Swap a human approver for an LLM making the routing decision and the shape of the problem barely changes.
A reasonable learning path
You don't need to learn every framework at once. A logical order looks something like: LLM → tokens and context window → embeddings → vector search → RAG → prompt engineering → tool calling → agents → LangChain → LangGraph → MCP → production concerns. Build something small at each step — a chat app for the LLM stage, a document search for RAG, a tool-calling agent for a calculator or weather API, a multi-step workflow for LangGraph, the same tools behind an MCP server for the last stage.
The mental model underneath all of it:
| Component | Responsibility |
|---|---|
| LLM | Understands and generates language, decides based on context |
| Context window | What's actually available to the model |
| Embedding | Turns meaning into a vector |
| Vector DB | Stores and searches vectors |
| RAG | Brings relevant information into context |
| Prompt | Defines context and instructions |
| Tool | Performs a specific action |
| Agent | Decides and executes across multiple steps |
| LangChain | Abstractions for building LLM applications |
| LangGraph | Models workflow and state as a graph |
| MCP | Standardizes how the application reaches tools/servers |
If you skip these concepts, you end up memorizing framework calls instead of understanding what you're building. The real question isn't "how do I build an agent." It's "how do I build a system where an LLM can use real data and real tools in a way that's controllable, observable, and trustworthy." That's the point where AI engineering just becomes software engineering again.
Top comments (0)