My search agent reads 20 messages to answer one question

TL;DR: I am at the very beginning of building agent infrastructure. I built Lumina, a search agent that streams cited answers, and gave it memory with one line: messages.slice(-20).
It worked, for a while. Then I noticed it citing sources that did not exist. Chasing that one bug took me through token budgets, a hidden formatting marker, an ordering bug, and eventually a question I hadn't actually answered for myself: does a search agent need "real" memory at all? You do not need a vector database to give an agent memory. You do need to know exactly what you are putting inside the window you send it, and that turned out to be the harder problem.

---

I was building Lumina, a search agent that streams cited, sourced answers back to the user (think a small, self-hosted version of the "ask a question, get an answer with [1][2] next to it" pattern). Solo project, no infra budget, no team to argue with about architecture. When I asked around about how to give an agent memory, almost everyone said some version of the same thing: embeddings, a vector store, retrieval over past turns. That's the default answer now. It's treated less like a design decision and more like a checkbox.

Which sounded to me like: you need a second piece of infrastructure before your agent is allowed to remember a follow-up question.

So I ignored that and wrote the simplest thing I could defend:

// backend/src/agent/agent-runner.ts
export const MAX_HISTORY_MESSAGES = 20;

export function historyFromDbMessages(messages) {
  return messages.slice(-MAX_HISTORY_MESSAGES).map((m) => ({
    role: m.role === "User" ? "user" : "assistant",
    content: m.content,
  }));
}

Last 20 messages, sent straight through. No ranking, no similarity search, no separate memory store. I asked a question, then a follow-up, and it resolved correctly. I asked a second follow-up. Also fine. I assumed memory was, more or less, a solved problem for this project, and I moved on to streaming and citation UI.

A few days later I was reading through a longer thread and noticed the model citing [1] and [2] on a turn where it had not called web_search at all.

That stopped me. Not "why is this slow," not even "why is this wrong," specifically: where did that citation come from? The agent hadn't retrieved anything this turn. It had no sources this turn. And yet there they were, formatted exactly like real citations, sitting in the answer.

That question took over the next few days. This is what I found.

Question 1: Why cap the history at all?

The obvious answer is tokens. That's true, but it's not the whole answer, and it wasn't actually the reason I picked 20 specifically.

Lumina's agent loop is tool first: on every turn, the model looks at the system prompt, the conversation history, and a web_search tool definition, and decides for itself whether it needs to search or can answer from what's already there. It makes that decision by reading the history directly. There's no separate classifier or router deciding "this needs a search," it's the same LLM call making the judgment inline.

That means an unbounded history doesn't just cost tokens. It changes the decision surface. A ten turns ago tangent sitting in context can quietly shift whether the model thinks the current question is "already answered" or needs fresh information. Rough math: if an average exchange (user question plus assistant answer) runs somewhere around 150 to 300 tokens once you include the citation formatting, a 20 message window lands around 3,000 to 6,000 tokens of history before the current question is even added. That's a manageable, predictable slice to reason over. Send the full history of a long running thread instead, and you're asking the model to weigh a hundred turn old detail against the actual question in front of it, on every single turn, forever.

So the cap isn't really a memory size decision. It's a decision about how much of the past gets a vote in what the agent does right now. Twenty was a guess, enough turns to resolve a follow-up cleanly, not so many that old context starts leaking into a decision it has no business influencing.

That part turned out to be fine. It wasn't where the bug was. But it's worth stating plainly, because it's the assumption everything else in this post sits on top of: the window itself was never the problem. What was inside it, was.

Question 2: Where were the fake citations actually coming from?

Here's where I lost real time.

Every assistant message Lumina saves to Postgres gets a hidden marker appended to the end of it. The reason is mundane: the frontend needs to re render citation cards when you reload a conversation, and rather than add a second table and join it back to messages on every read, I just stuffed the source data into the message itself, wrapped in an HTML comment so it wouldn't render:

${answer}\n\n<!--SOURCES:${JSON.stringify(sources)}-->

A real saved message looks something like this on disk:

Ice is less dense than water[1][2], which is why it floats instead of sinking.

<!--SOURCES:[{"index":1,"title":"Why Ice Floats","url":"https://...","domain":"..."},{"index":2,"title":"Density of Water","url":"https://...","domain":"..."}]-->

Invisible in the UI. Just a comment. I didn't think twice about it when I wrote it, because comments are, by definition, meant to be skipped.

That's true for a browser parsing HTML. It is not true for a language model reading a plain text message history. The model has no concept of "this part is a comment, ignore it." There's no DOM, no parser, just tokens. And those particular tokens happen to contain a JSON array with fields called index, title, and url, which is structurally identical to the citation data the model is supposed to be producing on its own.

So a few turns later, mid follow-up, the model had, sitting right there in its own context, indistinguishable from anything else, a fully formed, entirely plausible citation list from three turns earlier. It used it. The [1][2] pattern matched what it had produced before, syntactically. It just didn't correspond to anything registered in the current turn's actual source list, because no search had happened this turn.

It wasn't forgetting. If anything it remembered too accurately. It remembered formatting that was only ever meant for my frontend to parse, and treated it as legitimate conversational content because, from where it was sitting, there was no reason not to.

The test that convinced me

I didn't fully trust this explanation until I could reproduce it on demand, so I ran the same three turn conversation twice: once with the marker left in the history as is, once with it stripped out before the messages were sent to the model.

	Marker left in history	Marker stripped from history
Turn 1, search + answer	citations [1][2] correct, match this turn's sources	citations [1][2] correct, match this turn's sources
Turn 2, follow-up with new search triggered	citations [1][2] correct, match this turn's sources	citations [1][2] correct, match this turn's sources
Turn 3, follow-up with no new search	cites [1][2] again, but they resolve to turn 1's sources, which no longer exist in this turn's registry	no stale citations. Model either answers without citing or asks a clarifying question

Same model, same three prompts, in the same order. The only variable was whether that one hidden comment made it into the message list the LLM actually received. That's the point where I stopped calling this a hallucination and started calling it what it actually was: correct model behavior on input I hadn't bothered to clean.

I ran it a second time with a longer, five turn conversation to make sure it wasn't a fluke specific to a three turn setup, and got the same pattern: any turn that didn't trigger a fresh search would occasionally reuse whatever source indices happened to be sitting in the most recent marker still inside the 20 message window.

The fix was one line, applied before anything goes back into history:

const content = m.content
  .replace(/\n*\n?<!--SOURCES:[\s\S]*?-->\s*$/, "")
  .trimEnd();

Strip the marker, keep the visible answer text, then hand it to the model. No new tools, no new infrastructure. Just don't let the model see something it was never meant to read.

Question 3: Why did truncating before stripping still cause problems?

I got the regex right almost immediately. I got the order of operations wrong for longer than I'd like to admit.

My first pass truncated to the last 20 messages first, then stripped markers from whatever remained:

// what I had, roughly
const recent = messages.slice(-MAX_HISTORY_MESSAGES);
const cleaned = recent.map((m) => stripMarker(m.content));

That looks harmless. On short test threads, it was. On a couple of longer, real threads, a broken comment fragment started leaking through into what the model received. Not because .slice() cuts a message in half, it doesn't, message boundaries stay intact, but because of how I'd chained the two transformations, a message sitting right at the 20 message boundary occasionally had its marker only partially matched by the regex, depending on what else was on either side of it after truncation.

The practical effect: on certain threads, the model would receive a message ending in something like ...ice is less dense than water[1][2]. followed by a stray, unclosed <!--SOURCES: fragment with no closing -->. That's not just noise. It's a dangling, structurally broken tag sitting in the model's context, which is arguably worse than the clean, well formed version of the same bug, because now the model is trying to make sense of malformed markup instead of just ignoring it.

Reordering, strip the marker first, from the full message set, then truncate to the last 20, removed the ambiguity entirely, because truncation now happens after every message is already clean. Small fix, one line moved above another. But it's exactly the kind of bug that hides from every quick manual test you'll run while building the feature, and only shows up once a real conversation is long enough to actually reach the boundary you capped it at.

Question 4: Does a search agent need "real" memory at all?

This is the question that made me stop feeling behind on architecture, and it took the citation bug to get me to actually ask it properly instead of assuming the answer.

A vector store retrieves by semantic similarity. It ranks past messages by how related they seem to the current query, and pulls back the top few. For a follow-up like "what about the second one," the message that actually matters isn't the most topically similar one across the whole conversation. It's specifically the literal previous turn, in order, regardless of how "similar" it scores. Semantic retrieval can rank an earlier, more keyword heavy message above the one that actually matters, which would make a follow-up agent worse, not better. A sliding window doesn't have this failure mode, because it doesn't rank anything, it just preserves order.

The other half of what people usually mean by "agent memory," not re searching something you've already answered, turned out to be a prompt level decision, not a retrieval level one. Lumina's system prompt instructs the model not to call web_search again for follow-ups that are answerable from prior turns in the visible history. That decision lives entirely in prompts/prompt.md and the instructions appended in prompt-loader.ts, not in any retrieval layer. No embeddings involved, no similarity threshold to tune.

Where I do think retrieval based memory would actually matter is a fundamentally different shape of product: something that has to remember a specific user across sessions, days apart, where the fact you need genuinely isn't in the last 20 messages because it isn't even in this conversation. That's a real, well studied problem. It's what tools like mem0 and long term memory layers in agent frameworks are built for. Lumina isn't that product right now. It's a single thread search agent. Reaching for that architecture before I had that specific problem would have meant building and maintaining infrastructure to solve something I don't currently have.

Things I got wrong, collected in one place so you can skip them

"Agent memory means a vector database, that's the default now." Not for a single thread, tool first search agent. A sliding window solved the actual failure mode I had, which was resolving the previous turn correctly, not long horizon recall.

"What I store for my own UI doesn't matter to the model." It does. Anything that ends up back inside the message array is language, as far as the model is concerned, formatting conventions, HTML comments, and all.

"Truncate first, clean up second, it's just an ordering detail." It isn't just a detail. Cleaning up after truncating leaves you exposed to whatever happens to sit exactly at the cutoff, and that's the one case a quick manual test will never happen to hit.

"This is the model hallucinating, models do that." It looked exactly like that from the outside. It wasn't. The model was doing correct, expected inference on input that I had failed to sanitize.

"Twenty is an arbitrary number I should tune later." It's a real design decision about how much of the past gets to influence the agent's next action, not just a token budget knob. I hadn't actually thought about it that way until I had to defend it to myself while writing this.

What actually helped

A short list, in case you're building something similar and want to skip some of the detours I took.

Read the loop itself first

• docs/AGENT_LOOP.md: the internal architecture document I ended up writing while debugging this, including the full sequence diagram from HTTP request to streamed, cited answer, and the state machine the loop runs on.

Read when you're deciding how much memory you actually need

• Anthropic's writing on context and retrieval, for thinking clearly about what actually belongs inside a model's context window versus what should just live in your own storage layer.
• LangChain's memory documentation, mainly for the general window versus retrieval framing, useful even if you don't end up using their implementation, which I didn't.
• mem0's docs, as a reference point for what a genuine cross session memory layer looks like when you actually need one, so you can tell the difference between that problem and the one I had.

Read as code

• backend/src/agent/agent-runner.ts in the Lumina repo: the entire "memory system" discussed in this post is about fifteen lines inside that one file.
• backend/src/agent/agent-loop.ts: for how the tool first turn cycle actually decides, per turn, whether to search or answer, which is the mechanism the history window feeds into.

Where I'm taking this next

Right now, truncation is a hard cutoff. Message 21 stops existing to the model, silently, with no summary and no warning. The next thing I want to try is summarizing whatever falls out of the window instead of dropping it outright, so a very long thread degrades gracefully instead of the agent abruptly forgetting how the conversation started.

I also want to write a test that pushes a conversation past the 20 message boundary automatically as part of CI, so an ordering bug like the one in this post shows up in a failing test instead of in a production screenshot three weeks later, after I've forgotten exactly why I wrote the truncation logic the way I did.

Longer term, if Lumina ever needs to remember something about a specific user across separate sessions, not within a thread, but genuinely days apart, that's the point where a retrieval layer stops being premature and starts being the right tool. I don't think I'm there yet. But now I at least have a clearer test for when "yet" arrives: the question isn't "do agents typically have memory," it's "is the fact I need actually inside the window I'm already sending."

I am a builder, not an infra expert, and I've been wrong at least five documented times in this post alone. If something here is off, I'd rather be corrected than confident, tell me.