DEV Community: PromptMaster

Outcome vs. Process: Evaluating Multi-Step Agents

PromptMaster — Sat, 08 Aug 2026 14:58:26 +0000

Judging only an agent's final answer misses most of what can go wrong. An agent plans, calls tools, and reasons across steps — and can reach a good answer by luck through a broken process that fails on the next input.

Evaluate the trajectory, not just the destination: outcome evaluation (was the result right?) and process evaluation (were the steps sound?) together.

The trajectory is what makes an agent an agent

A single model call has one output to judge. An agent has a trajectory — it plans, calls tools, observes results, reasons, and acts, often over many steps. That in-between is exactly what separates evaluating an agent from evaluating a single model call, and it's where the leverage and the failures both hide. If you only look at final answers, you're evaluating the agent as though it were a model, and missing the dimension that makes it an agent.

Outcome versus process

There are two complementary questions. Outcome evaluation asks whether the final result was correct — necessary, but blind to how it was reached. Process evaluation asks whether the steps were sound: did the agent plan sensibly, call the right tools, recover from errors, avoid needless loops? An agent that gets the right answer through a wrong process will eventually get a wrong answer, so process evaluation is what catches problems before they surface as failures.

A right answer from a wrong process is a latent bug.

What to check along the trajectory

Planning — did the agent break the task down sensibly, or thrash?
Tool selection — did it choose the right tools and call them correctly?
Error recovery — when a step failed, did it adapt, or spiral?
Efficiency — did it reach the goal in a reasonable number of steps, or loop and wander?

Free Agent Evaluation QuickStart — the whole loop (define, measure, test, trust) on a few pages. Download it free.

Step-level and end-to-end together

The strongest evaluation combines both levels. End-to-end checks that the whole agent accomplishes real tasks; step-level checks pinpoint where a failing agent goes wrong, so you can fix the specific step rather than guessing. End-to-end tells you that something broke; step-level tells you what. You want both, because each answers a question the other cannot.

This is why tracing matters

You can only evaluate a trajectory you can see. Capturing the full record of what the agent did — every plan, tool call, and intermediate result — is the precondition for process evaluation. Without it, a failing agent is a black box and you're left re-running a non-deterministic failure blind. Trajectory evaluation and tracing go together.

Going deeper? AI Agent Evaluation & Testing: The Complete Guide is the full reference — 40 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.

FAQ

How do you evaluate a multi-step agent?

Evaluate the whole trajectory, not just the final answer: combine outcome evaluation (was the result correct?) with process evaluation (were the planning, tool calls, and recovery sound?). End-to-end checks the whole task; step-level pinpoints where it broke.

What's the difference between outcome and process evaluation?

Outcome evaluation judges whether the final result was correct, independent of how it was reached. Process evaluation judges whether the steps along the way were sound. An agent can reach a right answer through a wrong process — a latent bug.

Why isn't the final answer enough to evaluate?

Because an agent can reach a good answer by luck through a broken process that will fail on the next input. Judging only the outcome rewards luck and hides process failures until they surface as real failures later.

What should I check in an agent's trajectory?

Planning (did it break the task down sensibly?), tool selection (right tools, correct calls?), error recovery (did it adapt when a step failed?), and efficiency (reasonable number of steps, or looping?).

Do I need tracing to evaluate trajectories?

Effectively yes. Process evaluation requires seeing the full sequence of steps — plans, tool calls, intermediate results. Without tracing, a failing agent is a black box and the trajectory can't be evaluated.

LLM-as-Judge: How to Use a Model to Evaluate a Model

PromptMaster — Sat, 08 Aug 2026 14:57:47 +0000

LLM-as-judge uses a capable model to score or compare agent outputs against a rubric — filling the gap where quality is open-ended and human judgment doesn't scale.

Done well it approximates human judgment cheaply; done carelessly it produces confident nonsense. The keys: a specific rubric, pairwise over absolute scoring, and validating the judge against human labels.

Why use a model as a judge

Many of the qualities that matter most — helpfulness, faithfulness, reasoning quality — have no formula, and human judgment does not scale to thousands of cases on every change. Using a capable model as a judge fills that gap: you ask a model to score or compare outputs against a rubric. Done well it approximates human judgment at a fraction of the cost and effort. Done carelessly it produces confident, systematic nonsense — which is why the details below matter.

Why a model can judge at all

It can seem circular to use a model to evaluate a model — if the judge could reliably tell good from bad, why not use it as the agent? The resolution is that judging is easier than doing. Recognizing whether an answer is faithful to a source is narrower and more constrained than producing the faithful answer, the way it's easier to check a proof than to find one. The judge is handed the input, the output, and a rubric, and asked only to assess against that rubric.

The rubric is everything

The quality of the judgment depends almost entirely on the rubric. A vague instruction to "rate this 1-10" yields noise; a specific rubric that defines each level and what to look for yields something usably consistent. The judge's reliability comes from the structure the rubric provides.

judgment = judge_model(
    task=original_input,
    response=agent_output,
    rubric="""Score faithfulness 1-5.
    5 = every claim supported by the sources.
    3 = mostly supported, minor unsupported detail.
    1 = key claims not supported / contradicted.""",
)

Free Agent Evaluation QuickStart — the whole loop (define, measure, test, trust) on a few pages. Download it free.

Pairwise beats absolute scoring

Models are more reliable at comparing than at scoring in the abstract. Asking "which of these two responses is better?" tends to be far more consistent than "rate this 1-10," because absolute scores drift and cluster while comparisons are anchored. Whenever you can frame evaluation as a comparison — against a reference, or between two versions of the agent — you get more reliable signal.

Ask which is better, not how good.

The biases to defend against

Position bias — judges can favor whichever answer comes first; swap the order and average to cancel it.
Verbosity bias — judges often prefer longer answers regardless of quality; call it out in the rubric.
Self-preference — a judge may favor outputs from its own model family; be aware when judge and agent share a model.
Validate the judge — check it against human labels on a sample. An unvalidated judge is an opinion, not a measurement.

FAQ

What is LLM-as-judge?

Using a capable model to score or compare agent outputs against a rubric, in place of human judgment at scale. It's the workhorse for evaluating open-ended qualities like helpfulness and faithfulness that have no formula.

Is LLM-as-judge reliable?

It can be, with care: a specific rubric, pairwise comparison over absolute scoring, and validation against human labels. Without those it produces confident but systematic errors. An unvalidated judge is an opinion, not a measurement.

Why is pairwise scoring better than 1-to-10?

Models are more consistent comparing two outputs than scoring one in isolation. Absolute scores drift and cluster; comparisons are anchored. Framing evaluation as 'which is better, A or B?' yields more reliable signal.

What biases affect LLM judges?

Position bias (favoring the first answer), verbosity bias (favoring longer answers), and self-preference (favoring the judge's own model family). Counter them by swapping order and averaging, calling out length in the rubric, and validating against humans.

How do I know if my judge is accurate?

Validate it against human labels on a sample of cases. If the judge's scores track human judgment, you can trust it at scale; if not, fix the rubric. An unvalidated judge may be systematically wrong in ways you can't see.

How to Build a Test Set for Your AI Agent

PromptMaster — Sat, 08 Aug 2026 14:57:05 +0000

The single most valuable thing you'll build isn't the agent — it's the test set you evaluate it against. It's the ground truth every version is measured on, and it survives model swaps, framework changes, and rewrites.

Start with ten real cases, each paired with a verdict for what good looks like, and grow the set with every failure you find.

Why the test set is the asset

An evaluation is only as good as the cases it runs. A test set is a collection of scenarios — inputs paired with some notion of what a good response looks like — that represents the situations your agent must handle. It is the ground truth against which every version of the agent is measured, and it is the one asset that survives model changes, framework changes, and rewrites. Build it well and it pays off on every future decision.

Models change. Frameworks change.
The test set endures.

What a good test set contains

Representative cases — the common situations your agent actually faces, so the score reflects real performance.
Edge cases — the rare, tricky, and adversarial inputs where agents break, because these are what production surfaces.
Known failures — every bug you've found, captured as a case, so it can never silently return.
A verdict per case — an expected answer, a checklist, or a rubric. A case without a verdict can't evaluate anything.

Where test cases come from

The best test cases come from reality. Real user interactions — especially the ones that went wrong — are gold, because they represent situations that actually happen. Every production failure should become a test case. You can supplement with synthetic cases the model or your team generates to cover situations you haven't seen yet, but the core of a strong test set is drawn from real usage, curated over time.

Free Agent Evaluation QuickStart — the whole loop (define, measure, test, trust) on a few pages. Download it free.

A case is an input plus a verdict

Be precise about what a single case is: an input the agent will receive, and a way to decide whether the response was good. That verdict takes different forms — an exact expected answer, conditions the response must satisfy, a rubric a judge applies, or a reference to compare against. The discipline of writing the verdict for every case is what turns a pile of examples into an actual test set.

{
  "input": "I was charged twice, I want a refund",
  "expects": {
    "task_success": "acknowledges double charge, checks policy",
    "tool": "issue_refund only if within policy",
    "faithfulness": "cites the refund policy",
  },
}

Start small, grow deliberately

A common mistake is waiting to build a huge test set before evaluating anything. Start with ten cases that capture what matters, and grow the set as you learn where the agent fails. Twenty well-chosen scenarios that cover your real risks beat a thousand generic ones. The test set is a living asset that grows with every bug found and every new situation encountered.

FAQ

How do I build a test set for an AI agent?

Collect scenarios — inputs paired with a verdict for what good looks like. Include representative cases, edge cases, and every known failure. Pull from real usage, write a verdict for each, and start with about ten rather than waiting for a huge set.

How many test cases do I need?

Start with about ten well-chosen cases that cover what matters, then grow as you find failures. Twenty scenarios covering your real risks beat a thousand generic ones. The set is meant to grow over time, not be complete on day one.

Where do good test cases come from?

From reality — real user interactions, especially the ones that went wrong. Every production failure should become a case. Synthetic cases can supplement coverage, but the core comes from actual usage.

What makes a test case complete?

An input plus a verdict: some way to decide whether the response was good. The verdict can be an exact answer, a checklist of conditions, or a rubric. Without a verdict, a case can exercise the agent but can't evaluate it.

Why is the test set so important?

It's the ground truth every version of the agent is measured against, and it outlives models, frameworks, and rewrites. It's the accumulated definition of what good means for your task — the asset that makes every future change safer.

How to Evaluate an AI Agent (When There's No Single Right Answer)

PromptMaster — Sat, 08 Aug 2026 14:56:17 +0000

You can't test an AI agent the way you test normal software. Agents are non-deterministic (same input, different outputs), open-ended (no single right answer), and multi-step (they can reach a good answer through a broken process).

The answer is evaluation: a repeatable loop — define what good means, build a test set, measure with fitting metrics, and gate every change — that turns "it seemed to work" into "we measured it."

Why agents break traditional testing

Traditional software testing assumes a known answer: given this input, assert that output. Agents shatter that assumption on three fronts at once. They are non-deterministic — the same input can produce different outputs, so you cannot assert exact equality. They are open-ended — most real tasks have no single correct answer, only better and worse ones. And they are multi-step — an agent plans, calls tools, and reasons across many turns, any of which can go wrong in ways the final answer hides. The techniques you know for ordinary software simply do not transfer.

What evaluation actually is

Evaluation is a repeatable method for asking "does this agent do what we need, across the situations that matter?" and getting an answer you can act on. It replaces the guesswork most teams run on — a working demo, a few manual tries, and a hope — with evidence. Without it, every change to an agent is a guess and every deploy is a hope; with it, every change becomes a measured step.

A demo tests the cases you thought of.
Production is the cases you didn't.

The evaluation loop

Every evaluation is the same loop, and once you see its shape every eval system reads as a variation of it. Define what good means, measure the agent against that definition, test on every change to catch regressions, and — having earned it — trust what you ship while continuing to measure.

DEFINE    decide what "good" means for the task
MEASURE   score the agent against that definition
TEST      run it on every change; catch regressions
TRUST     ship knowing it works — and keep measuring

What to measure

No single number captures whether an agent is good. Choose the few dimensions that matter for your task and accept that they trade off. A strong starting set: task success (did it accomplish what the user wanted?), faithfulness (is the answer grounded, or made up?), safety (does it avoid harmful or out-of-scope actions?), and cost and latency (is it fast and cheap enough to use?). Measuring one axis alone hides the trade you are making.

Free Agent Evaluation QuickStart — the whole loop (define, measure, test, trust) on a few pages. Download it free.

Evaluate before you optimize

You cannot improve what you cannot measure, and you cannot tell whether an "improvement" helped without a baseline. The first move on any serious agent is to build an evaluation that captures what good looks like. Only then does optimization become meaningful — otherwise you are changing things and trusting your gut, which is exactly the guesswork evaluation exists to eliminate.

Where to start

Start small: pick two or three dimensions, write ten real test cases, score them, and grow from there. A handful of well-chosen scenarios that cover your real risks beats a thousand generic ones. The evaluation is a living asset that grows with every bug found — and it is the thing that lets you improve an agent on purpose instead of by hope.

FAQ

How do you evaluate an AI agent?

With a repeatable loop: define what good means (the quality dimensions that matter), build a test set of real scenarios, measure the agent with metrics that fit each dimension, and run the evaluation on every change to catch regressions. It replaces guesswork with evidence.

Why can't you test agents like normal software?

Because agents are non-deterministic (the same input gives different outputs), open-ended (no single right answer), and multi-step (they can reach a good answer through a broken process). Exact-output assertions, the basis of normal testing, don't apply.

What should you measure when evaluating an agent?

The few dimensions that matter for your task: task success, faithfulness (grounding), safety, and cost/latency. No single score captures agent quality, and the dimensions trade off, so measure them separately.

What is the evaluation loop?

Define what good means, measure the agent against it, test on every change to catch regressions, and trust what you ship while continuing to measure. Every evaluation system is a variation of this loop.

Do I need evaluation before optimizing my agent?

Yes. Without a baseline you can't tell whether a change helped or hurt. Building the evaluation first turns optimization from guesswork into measured steps — keep the change if the number improved, revert if not.

How to Estimate Tokens for RAG (and Why Character Counts Mislead)

PromptMaster — Tue, 04 Aug 2026 09:39:37 +0000

Models and pricing count tokens, but chunking libraries usually count characters — and the two don't map cleanly. The rough rule is ~4 characters per token for English prose, but it varies with content, code, and language.

Character counts mislead because a 600-character chunk isn't a fixed number of tokens. To budget context and cost accurately, you need the token count, not the character count.

Why tokens, not characters

Everything that matters downstream is measured in tokens: the context window the model can hold, the embedding cost, the generation cost. But the chunking step usually operates on characters, because that's what's easy to split on. This mismatch is a quiet source of surprises — you set a character size, and the token reality turns out different from what you assumed.

The rough rule, and where it breaks

For English prose, a token is roughly four characters — so ~600 characters is ~150 tokens, give or take. It's a useful rule of thumb, but it breaks down in exactly the cases you care about. Code tokenizes differently from prose. Numbers, punctuation, and rare words split into more tokens. Other languages diverge from the English ratio entirely. The rule is a starting estimate, not a guarantee.

~4 characters per token — until it isn't.

Why the mismatch costs you

Context budget — if you assume 600 characters is fewer tokens than it is, you can overflow the context window you planned.
Cost estimates — embedding and generation are priced per token, so a character-based estimate can be off by a wide margin.
Retrieval tuning — 'top-k = 5 chunks' means very different token loads depending on real chunk token sizes.

Free RAG Chunk Visualizer — see your chunks, token counts, and quality flags in the browser. Try it free.

Estimating tokens without the exact tokenizer

The precise answer comes from the exact tokenizer your model uses. But for planning, a good subword estimate — one that accounts for word length, punctuation, and numbers rather than just dividing characters by four — tracks real tokenizers closely enough to budget confidently. The point is to get away from raw character counts, which are the least accurate signal.

See tokens per chunk

The RAG Chunk Visualizer estimates tokens for every chunk and the whole document, using a subword heuristic rather than a crude character divide. You can see immediately whether your chunks land near your token target and how the total maps to embedding and generation cost — the character-to-token guesswork removed.

Go further: the Full Edition adds overlap control, cost-model presets, top-k modelling, strategy comparison, JSON export, and vector-DB record preview. Get the Full Edition. Built as a companion to RAG: The Complete Guide.

FAQ

How many tokens is a RAG chunk?

For English prose, roughly one token per four characters — so a 600-character chunk is about 150 tokens. But it varies: code, numbers, punctuation, and other languages tokenize differently, so character counts are only a rough estimate.

Why not just count characters?

Because models and pricing run on tokens, and characters don't map cleanly to tokens. A character-based estimate can overflow your context budget or throw off cost estimates, especially for code, numbers, or non-English text.

How do I estimate tokens without the model's tokenizer?

Use a subword heuristic that accounts for word length, punctuation, and numbers rather than dividing characters by four. It tracks real tokenizers closely enough for budgeting context and cost.

How many characters per token?

About four characters per token for English prose, as a rule of thumb. It breaks down for code (which tokenizes densely), numbers, punctuation, and other languages, so treat it as a starting estimate, not a fixed rate.

How can I see tokens per chunk?

A chunk visualizer can estimate tokens for each chunk and the whole document using a subword heuristic, showing whether chunks land near your token target and how the total maps to cost — without the character-to-token guesswork.

Chunk Overlap in RAG: What It Is and How Much You Need

PromptMaster — Tue, 04 Aug 2026 09:39:00 +0000

Chunk overlap repeats the tail of each chunk at the start of the next one, so a fact that spans a boundary isn't split in half and lost to retrieval.

A little overlap helps; too much wastes tokens and storage. A common range is 10–20% of chunk size — but the right amount depends on how your information sits relative to your boundaries.

The problem overlap solves

When you split a document into chunks, some facts land right on a boundary — the setup in one chunk, the payoff in the next. Retrieve either chunk alone and the fact is incomplete. Overlap fixes this by repeating a slice of the end of each chunk at the beginning of the following one, so boundary-spanning information appears whole in at least one chunk.

How overlap works

If your chunk size is 600 characters and your overlap is 60, each chunk shares its last 60 characters with the next chunk's first 60. The chunks still advance through the document, but with a repeated seam between them. That seam is cheap insurance against splitting a key sentence exactly where it mattered.

chunk_size = 600
overlap    = 60          # ~10% of chunk size
step       = chunk_size - overlap   # advance 540 chars per chunk
# each chunk shares its last 60 chars with the next chunk's first 60

How much overlap?

More overlap means fewer facts get split, but also more repeated text — which costs more to embed, more to store, and can surface near-duplicate chunks at retrieval time. A common starting range is 10–20% of chunk size. Below that you risk splitting facts; well above it you're mostly paying to store the same text twice.

Overlap is insurance.
Buy enough to cover the seams, not more.

Free RAG Chunk Visualizer — see your chunks, token counts, and quality flags in the browser. Try it free.

Structure-aware chunking needs less

Overlap matters most with fixed-size chunking, where cuts fall arbitrarily. Structure-aware chunking already splits on natural boundaries, so fewer facts get cut and less overlap is needed. The two knobs interact: the better your boundaries, the less overlap you have to buy.

See the overlap and its cost

Overlap is easiest to reason about when you can see it. The RAG Chunk Visualizer marks the overlapping region between chunks and counts the overlap tokens, so you can see exactly how much repeated text a given setting produces — and what it adds to your indexing cost — before you commit. Overlap control is part of the Full Edition.

FAQ

What is chunk overlap in RAG?

Repeating the tail of each chunk at the start of the next one, so a fact that spans a chunk boundary appears whole in at least one chunk instead of being split in half and lost to retrieval.

How much chunk overlap should I use?

A common range is 10–20% of chunk size. Below that, boundary-spanning facts risk getting split; well above it, you're mostly paying to store and embed the same text twice. The right amount depends on your documents.

Does chunk overlap increase cost?

Yes. Overlapping text is embedded and stored more than once, so more overlap means higher indexing cost and storage, and it can surface near-duplicate chunks at retrieval time. It's a trade-off against losing boundary-spanning facts.

Do I need overlap with structure-aware chunking?

Less than with fixed-size. Structure-aware chunking splits on natural boundaries, so fewer facts get cut and less overlap is needed. The better your boundaries, the less overlap you have to buy.

How do I see how much overlap I'm using?

A chunk visualizer can mark the overlapping region between chunks and count overlap tokens, showing exactly how much repeated text a setting produces and what it adds to indexing cost before you commit.

Fixed-Size vs. Structure-Aware Chunking: Which Should You Use?

PromptMaster — Tue, 04 Aug 2026 09:37:53 +0000

Fixed-size chunking cuts every N characters — simple, but it slices through sentences, paragraphs, and sections. Structure-aware chunking splits on the document's own boundaries (headings, paragraphs), keeping each chunk coherent.

Structure-aware usually wins for documents with real structure; fixed-size is fine for unstructured text. The fastest way to decide is to see both on your own documents.

The two strategies

Chunking strategies fall on a spectrum, but two anchor the ends. Fixed-size chunking cuts the text every N characters or tokens, ignoring what's there — simple, predictable, and completely blind to meaning. Structure-aware chunking splits on the document's natural boundaries — headings, paragraphs, sections — so each chunk is a coherent unit. The choice between them shapes how retrievable your chunks are.

What fixed-size gets wrong

Fixed-size chunking's weakness is that it cuts wherever the character count runs out — often mid-sentence, sometimes mid-word. A definition gets separated from the term it defines; a list gets split down the middle; the first half of an idea lands in one chunk and the second half in another. Each broken chunk retrieves worse, because neither half is fully meaningful on its own.

Fixed-size chunking is blind to meaning.
It cuts where the counter runs out.

What structure-aware gets right

Structure-aware chunking respects the document's own organization. A heading stays with its section; a paragraph stays whole; a list stays together. Because documents are usually organized so that related information sits together, splitting on those boundaries tends to produce chunks that are each about one thing — exactly what retrieval wants.

When fixed-size is actually fine

Structure-aware isn't always worth it. If your text has no meaningful structure — a wall of uniform prose, transcripts with no sections, scraped text with markup stripped — there are no boundaries to respect, and fixed-size is simpler with no real downside. The strategy should match how structured your documents actually are.

Free RAG Chunk Visualizer — see your chunks, token counts, and quality flags in the browser. Try it free.

See both on your own text

The decision is easy once you can see it. The free RAG Chunk Visualizer shows both strategies on your own document, side by side, and even scores how many chunks end on a clean boundary versus mid-sentence. Paste a representative document and the right choice is usually obvious in seconds — no need to argue about it in the abstract.

A sensible default

For most real documents — docs, articles, knowledge bases, anything with headings and paragraphs — start with structure-aware chunking and fall back to fixed-size only where structure is genuinely absent. Let the document decide, and verify by looking at the cuts rather than trusting the strategy name.

FAQ

What is fixed-size chunking?

Splitting text every N characters or tokens regardless of content. It's simple and predictable but blind to meaning — it often cuts through sentences, paragraphs, and sections, producing chunks that retrieve worse.

What is structure-aware chunking?

Splitting on a document's natural boundaries — headings, paragraphs, sections — so each chunk is a coherent unit. It keeps definitions with their terms and paragraphs whole, which tends to improve retrieval.

Which chunking strategy is better?

Structure-aware usually wins for documents with real structure (docs, articles, knowledge bases). Fixed-size is fine for unstructured text with no boundaries to respect. Match the strategy to how structured your documents actually are.

How to Choose the Right Chunk Size for RAG (Without Guessing)

PromptMaster — Tue, 04 Aug 2026 09:37:07 +0000

Chunk size is the single decision that caps RAG quality, and most people guess at it. Too small and each chunk loses the context that makes it meaningful; too large and retrieval gets imprecise and expensive.

There's no universal number — it depends on your documents and your model. But you don't have to guess: you can see exactly how a given size splits your text, counts tokens, and affects cost before you commit.

Why chunk size decides everything downstream

In a RAG pipeline, you split documents into chunks, embed them, and retrieve the most relevant ones at query time. Every later step inherits the chunking decision. Get it wrong and no amount of reranking or prompt-tuning fully recovers: if the right information was split across two chunks, or buried in one that's too large to retrieve precisely, the model never sees it cleanly. Chunk size is upstream of everything, which is why it deserves more thought than a copied default.

The trade-off, in both directions

Too small — each chunk loses the surrounding context that made it meaningful. A sentence retrieved without its paragraph can be ambiguous or misleading.
Too large — retrieval gets imprecise (one chunk covers many topics, so similarity is diluted) and expensive (you send more tokens into generation than you need).
The sweet spot — large enough to be self-contained, small enough to be about one thing. Where that lands depends on your content.

There's no universal number

Dense technical documentation chunks differently from chatty support transcripts; a legal contract differs from a product FAQ. The right size depends on how information is distributed in your specific documents and on the model you'll retrieve for. Anyone who gives you a single magic number hasn't seen your data. The reliable approach is to try a size, look at what it actually produces, and adjust.

The right chunk size is the one that fits your documents.
Not the one in a tutorial.

See it before you build it

The fastest way to reason about chunk size is to watch it happen. Paste a representative document, pick a size, and see exactly where the cuts land, how many chunks you get, and how many tokens each holds. That's precisely what the free RAG Chunk Visualizer does — the guesswork becomes a thing you can look at and adjust in seconds, instead of a number you hope is right.

Free RAG Chunk Visualizer — see your chunks, token counts, and quality flags in the browser. Try it free.

What to watch for once you can see it

Chunks that cut mid-sentence — a sign the size or strategy is fighting your document's structure.
Wildly uneven chunk sizes — some tiny, some huge — usually means fixed-size splitting on structured content.
Token counts far from your target — the character size you picked may not map to the token budget you assumed.

From guessing to tuning

Once you can see chunks and their token counts, choosing a size stops being a guess and becomes a quick loop: try, look, adjust. Start around a few hundred tokens for prose, watch where the cuts fall, and nudge from there based on what your documents actually do — not on a number you copied.

FAQ

What is the best chunk size for RAG?

There's no universal number — it depends on your documents and model. Dense technical text chunks differently from conversational transcripts. A common starting point is a few hundred tokens for prose, then adjust based on how your specific documents split. Seeing the result before committing beats guessing.

What happens if chunks are too small?

Each chunk loses the surrounding context that made it meaningful. A sentence retrieved without its paragraph can be ambiguous or misleading, and the model never sees the full picture cleanly.

What happens if chunks are too large?

Retrieval gets imprecise — one chunk covers many topics, so similarity scores are diluted — and generation gets more expensive, since you send more tokens than needed into the prompt.

How do I know if my chunk size is right?

Look at what it produces: whether chunks stay self-contained, whether they cut mid-sentence, and whether token counts match your target. A chunk visualizer lets you see all three before you build the pipeline.

Should chunk size be measured in characters or tokens?

Tokens are what models and pricing actually use, but chunking libraries often operate on characters. The two don't map cleanly (~4 characters per token for English prose, but it varies), so it helps to see the token count for a given character size.

Writing Agent Memories: What to Store and What to Skip

PromptMaster — Sat, 01 Aug 2026 14:53:41 +0000

Not everything an agent sees is worth remembering. Store too little and it forgets what mattered; store everything and the memory fills with noise that drowns the signal at retrieval time.

The extraction step — deciding what to keep and distilling it into clean, self-contained facts — is the most consequential and most overlooked part of a memory system.

Memory begins with a decision

Of everything that just happened, what's worth keeping? This is the first and most important question a memory system answers, and it's easy to get wrong in both directions. Store too little and the agent forgets the thing that mattered. Store everything and the memory fills with noise, because every irrelevant memory is a candidate to surface at the wrong moment and drown the useful one.

What deserves to be remembered

Stable facts about the user — preferences, context, recurring needs that will matter in future sessions.
Decisions and outcomes — what was chosen, what worked, what failed, so the agent doesn't relitigate settled ground.
Corrections — when a human fixes the agent, that correction is high-value memory: it must stick.
Durable state — the status of ongoing work, entities the agent tracks, anything with a life beyond one turn.

What to leave out

Transient chatter, one-off details with no future relevance, and anything that will be stale by next session are noise. The discipline of not storing them is as important as the discipline of storing what matters — a memory full of irrelevant fragments retrieves worse than a lean one.

Good memory is curated, not recorded.

Free Agent Memory Quick-Start — the four kinds of memory and the whole loop (store, retrieve, reflect, forget) on a few pages. Download it free.

Turn experience into clean facts

Raw conversation is messy; stored memories should be clean. Extraction usually distills an interaction into concise, self-contained statements — "user prefers X," "project deadline is Y" — rather than dumping the transcript. A common pattern is to use the model itself to summarize what's worth remembering, producing tidy semantic facts from noisy episodic experience.

memories = extract(conversation)
# -> [{'content': 'prefers email over phone',
#      'subject': 'user_1837', 'type': 'semantic'},
#     {'content': 'had billing issue, resolved by refund',
#      'subject': 'user_1837', 'type': 'episodic'}]
store.add(memories)

Self-contained is the key property: a memory that only makes sense with its original context will confuse retrieval later.

When to extract

There's also a question of timing. The common choices are at the end of a session, when the whole interaction can be distilled at once, or incrementally as significant things happen, so nothing is lost if a session is abandoned. End-of-session is simpler and cheaper but risks losing an interaction that never cleanly ends; incremental captures more reliably at the cost of running more often. Many systems do both.

Going deeper? AI Agent Memory: The Complete Guide is the full reference — 41 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.

FAQ

What should an AI agent store in memory?

Stable facts about the user (preferences, context), decisions and outcomes, corrections from humans, and durable state about ongoing work. The test is whether it will matter in a future session.

What should an agent NOT store?

Transient chatter, one-off details with no future relevance, and anything stale by next session. Not storing noise is as important as storing signal — irrelevant memories surface at the wrong time and degrade retrieval.

How does memory extraction work?

It distills a messy interaction into concise, self-contained facts — 'user prefers X' — rather than storing the raw transcript. A common approach uses the model itself to summarize what's worth remembering into clean semantic facts.

When should extraction run?

At the end of a session (simpler, cheaper) or incrementally as important things happen (more reliable, more frequent). Many systems do both — capturing clearly important facts immediately and distilling the rest at the end.

Why is storing less sometimes better?

Because every stored memory is a candidate to surface at retrieval time. A store full of irrelevant fragments retrieves worse than a lean one, burying the memory that mattered under ones that didn't.

Memory vs. Context Window vs. RAG: What's the Difference?

PromptMaster — Sat, 01 Aug 2026 14:52:59 +0000

The context window is where the model works right now — fast, immediate, temporary, and bounded. It's the agent's working memory, nothing more.

RAG is how relevant information gets fetched into that window — a retrieval technique.

Memory is the persistent state that outlives the session. Memory uses RAG to retrieve, and always lands in the context window — but it's the same as neither.

Three words, constantly confused

Memory, context, and RAG get used almost interchangeably, and that imprecision causes real architectural mistakes. They're related but distinct: the context window is where information lives while the model works, RAG is a technique for fetching relevant information, and memory is the persistent state that outlives any single request. Memory often uses RAG to do its job, and always ends up in the context window — but it isn't the same as either.

The context window is working memory, not memory

The context window holds what the model can see right now. It's fast and immediate but temporary and bounded — it's the agent's working memory, and nothing more. Anything not in it is invisible to the model, and everything in it vanishes when the session ends. Persistent memory exists precisely because the context window can't be the place knowledge lives long-term.

RAG is how memory gets retrieved

Retrieving relevant memories for the current moment is, mechanically, a retrieval problem. You embed memories, store them, and at query time find the ones most relevant to what's happening now — exactly the RAG pipeline. Memory retrieval is RAG applied to the agent's own past rather than to a document corpus.

The difference is what sits in the store. Classic RAG retrieves from a fixed body of documents. Memory retrieves from an evolving record of the agent's own experience — a store that grows, changes, and must be curated as the agent lives. Same retrieval mechanism, different and more dynamic source.

RAG retrieves from a fixed corpus.
Memory retrieves from a living past.

Free Agent Memory Quick-Start — the four kinds of memory and the whole loop (store, retrieve, reflect, forget) on a few pages. Download it free.

Memory is the persistent state

Memory is the whole discipline: deciding what to store, how to store it, when to retrieve it, how to keep it accurate as it grows, and when to forget. RAG is one component of that — the reading step. The context window is where the result lands. Memory is the system that spans all of it and persists between sessions.

A quick test to keep them straight

When you're unsure which of the three you're dealing with, ask where the information lives and how long it survives. If it exists only for this request and vanishes after, it's the context window. If it's a technique for fetching relevant information into that window, it's retrieval. If it persists across sessions and represents accumulated state, it's memory. The test tells you which layer owns the problem you're debugging.

Going deeper? AI Agent Memory: The Complete Guide is the full reference — 41 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.

FAQ

What's the difference between memory and the context window?

The context window is temporary working memory — what the model sees right now, wiped when the session ends. Memory is persistent state that outlives sessions. Memory feeds the context window, but isn't the same as it.

Is agent memory just RAG?

No. Memory uses RAG for retrieval — the reading step — but also covers writing (deciding what to store), consolidation, and forgetting, which RAG doesn't. And memory retrieves from the agent's evolving past, not a fixed document corpus.

How do memory, context, and RAG work together?

Memory is the persistent store. RAG is how relevant memories are retrieved from it. The context window is where those retrieved memories land so the model can use them. Memory uses RAG to fill the context window.

Why not just make the context window bigger?

Bigger windows help but don't replace memory: cost grows with everything you resend each turn, models attend poorly to huge contexts, and the window still vanishes at session end. Memory is persistent and selective by design.

Which one should I reach for?

Ask where the information lives and how long it survives. Vanishes after this request → context window. A technique to fetch relevant info → RAG. Persists across sessions as accumulated state → memory.

The Four Types of AI Agent Memory, Explained

PromptMaster — Sat, 01 Aug 2026 14:49:34 +0000

Agents need four distinct kinds of memory: working (what it's doing now), episodic (what happened), semantic (what it knows), and procedural (how it does things).

They map onto human cognition because the same constraints produce the same solutions. Confusing them is the single most common memory design mistake — almost every design question is really "which of these four am I talking about?"

Why one word isn't enough

"Give the agent memory" sounds like one feature. It's four. Working, episodic, semantic, and procedural memory each have a different job, a different lifetime, and a different place to live. Treating them as one undifferentiated thing is why so many memory systems end up muddled — storing the wrong things, retrieving badly, and growing into noise.

The vocabulary comes straight from cognitive science, and that's not decoration. A mind that had to hold every experience in immediate awareness would be paralyzed, so human cognition evolved separate systems. An agent faces the same pressure, and arrives at the same answer.

Working memory

Working memory is what the agent is holding right now — the current task, the last few turns, intermediate results. It maps directly onto the context window: fast, immediate, and small, wiped when the task ends. This is the memory an agent already has by default. Everything else is the memory it doesn't.

Episodic memory

Episodic memory is the record of specific past events — what happened, when, in which session. "The user reported this bug last Tuesday." "We tried this approach and it failed." It's memory of experiences, tied to time and context, and it's what lets an agent refer back to particular past interactions rather than starting fresh each time.

Semantic memory

Semantic memory is general knowledge distilled from experience, stripped of when or where it was learned. "This user prefers concise answers." "The refund window is thirty days." Where episodic memory remembers the event, semantic memory remembers the fact the event taught. Much of building memory is turning episodes into durable semantic facts.

Episodic remembers the event.
Semantic remembers what it taught.

Free Agent Memory Quick-Start — the four kinds of memory and the whole loop (store, retrieve, reflect, forget) on a few pages. Download it free.

Procedural memory

Procedural memory is how to do things — the skills, routines, and patterns an agent follows. In practice this often lives in the agent's instructions, tools, and learned workflows rather than a fact store. It's the least discussed of the four and the one most often baked into the system rather than stored dynamically.

The distinction that saves you

Almost every memory design question resolves to "which of these four am I talking about?" Storing a user preference? Semantic. Recording what happened in a session? Episodic. Managing the current conversation? Working. Teaching a repeatable skill? Procedural. Naming the kind tells you where it lives, how long it should last, and how to retrieve it.

Going deeper? AI Agent Memory: The Complete Guide is the full reference — 41 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.

FAQ

What are the four types of AI agent memory?

Working (the current task, held in the context window), episodic (specific past events tied to time), semantic (general facts distilled from experience), and procedural (how to do things — skills and workflows).

What's the difference between episodic and semantic memory?

Episodic memory records specific events — what happened, when. Semantic memory records the general fact an event taught, stripped of when it was learned. Ten episodic sessions of a user asking for brevity become one semantic fact: 'prefers concise answers.'

Is the context window the same as working memory?

Effectively yes. Working memory is what the agent holds right now — the current task and recent turns — which lives in the context window. It's fast and immediate but small and wiped when the task ends.

Where does procedural memory live?

Usually in the agent's instructions, tools, and learned workflows rather than a separate fact store. It's the 'how to do things' memory, and it's most often baked into the system rather than stored and retrieved dynamically.

Why does the type of memory matter?

Because each kind has a different lifetime, retrieval pattern, and storage need. Naming the kind tells you where a memory should live and how to handle it — and confusing them is the most common memory design mistake.

AI Agent Memory: Why Your Agent Forgets, and How to Fix It

PromptMaster — Sat, 01 Aug 2026 14:43:23 +0000

A language model is stateless — it forgets everything the moment a conversation ends. For an agent meant to work over time, for the same people, that's disqualifying.

Agent memory is the layer that fixes it: a persistent store, separate from the model, that captures what happened, keeps it past the context window, and surfaces the relevant pieces back when they're needed.

It's built from four kinds of memory and one repeating loop — and once you see the shape, every memory system reads as a variation of it.

Why statelessness is the whole problem

A language model is trained once and then frozen. Each request is answered in isolation, with no record of what came before beyond what happens to sit in the current context window. Close the session and reopen it, and the model has no idea it ever spoke to you. For a one-shot question that's fine. For an agent — something meant to work on related tasks over time, for the same people, in the same domain — it's disqualifying.

No continuity — every session starts from zero; the agent re-asks what it was already told.
No learning — corrections don't stick; next session, the same mistake.
No personalization — the agent can't adapt to anyone, because it retains nothing.

What memory actually is

Memory is a persistent store, separate from the model, that captures what happened, holds it beyond the life of a single context window, and surfaces the relevant pieces back into context when they're needed again. The model stays stateless; the system around it remembers.

The tempting shortcut — paste the entire history into every prompt — fails on three fronts at once: the context window has a limit you'll hit, cost grows with every turn as you resend everything, and models attend poorly to information buried in a huge context. Memory is a dedicated component precisely so the agent holds only the relevant few facts in context, not its entire past.

A model is stateless.
Memory is the state.

The four kinds of memory

"Memory" isn't one thing. Agents need several distinct kinds, and confusing them is the root of most memory design mistakes. The field borrows its vocabulary from human cognition, because the same constraints produce the same solutions.

Working memory — what the agent holds right now: the current task and recent turns. This is the context window.
Episodic memory — specific past events, tied to time: "the user reported this last Tuesday."
Semantic memory — general facts distilled from experience: "this user prefers concise answers."
Procedural memory — how to do things: skills and workflows, often held in instructions and tools.

The memory loop

Every memory system is the same loop running around the stateless model: an interaction happens, something is extracted from it, the result is stored, later it's retrieved, the agent uses it, and periodically the store is consolidated and pruned.

# writing (after an interaction)
memories = extract(conversation)
store.add(memories)  # with subject, type, time, source

# reading (at the start of the next one)
relevant = store.retrieve(subject=user, query=now, top_k=5)
answer = agent(prompt(now, relevant))

Free Agent Memory Quick-Start — the four kinds of memory and the whole loop (store, retrieve, reflect, forget) on a few pages. Download it free.

Memory retrieval is RAG, plus time

Retrieving the right memory for the current moment is, mechanically, a retrieval problem — the same pipeline as RAG. Embed the current situation, find the closest memories, optionally rerank. What memory adds is time: recency and importance matter, so yesterday's correction should outrank a year-old passing remark. If you know retrieval, you already hold the reading half of memory.

Where memory sits in the stack

Memory is one layer of modern AI engineering. It's a key source for context engineering (what enters the window). Its reading half is RAG (retrieval pointed at the agent's own past). And it's what turns an agent from a stateless responder into one that learns. Master the loop and you hold the piece that makes agents feel like they know you.

Going deeper? AI Agent Memory: The Complete Guide is the full reference — 41 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.

FAQ

What is AI agent memory?

A persistent store, separate from the language model, that captures what happened in past interactions and surfaces the relevant pieces back into context when needed. It's what lets an agent remember across sessions, since the model itself is stateless.

Why do AI agents forget?

Because language models are stateless — each request is answered in isolation, with no memory of previous ones beyond what's in the current context window. Close the session and the model retains nothing. Memory is the external layer that fixes this.

What are the types of agent memory?

Four: working (the current task, in the context window), episodic (specific past events), semantic (general facts distilled from experience), and procedural (how to do things). Confusing them is the root of most memory design mistakes.

Is agent memory the same as RAG?

Memory uses RAG. Retrieving the right memory is a retrieval problem — the same pipeline. But memory also covers writing, consolidation, and forgetting, which RAG doesn't. RAG is the reading half of memory.

Can't I just put everything in the prompt?

Not for long. The context window has a limit, cost grows as you resend everything each turn, and models attend poorly to huge contexts. Memory exists so the agent holds only the relevant few facts in context.