Clanker Support's AI support agent has no vector database, no embeddings, and no retrieval pipeline. On every chat request we load every active knowledge source for the project and place it, budgeted to 80,000 characters, directly into the system prompt. For a support knowledge base measured in kilobytes, this beats RAG on simplicity, freshness, and recall — and we can show you exactly where it stops being true.
This is not a "RAG is dead" post. RAG is the correct architecture for corpora that don't fit in a context window. Our argument is narrower and, we think, more useful: most per-project support knowledge bases are tiny, context windows are large, and building an embedding pipeline before you've hit the ceiling is complexity you pay for every day and benefit from never. Here's the code, the arithmetic, and the honest failure mode.
The entire retrieval pipeline is a WHERE clause
Clanker Support is an open-source support widget (github.com/theopenco/llmchat). When a visitor sends a message, the API needs to decide which knowledge to show the model. Here is the entirety of that decision, from apps/api/src/routes/chat.ts:
const activeSources = await db(c.env).query.source.findMany({
where: (s, { and: a, eq: e }) =>
a(e(s.projectId, project.id), e(s.active, true)),
});
No query embedding. No similarity search. No reranker. Every active source for the project, every time. The sources come in three kinds — url (a one-shot snapshot of a single web page), text (a pasted snippet), and qa (a question/answer pair, either hand-written or promoted from a real operator reply in the inbox).
Those sources then flow into a prompt builder that assembles one string: a hardcoded support-only guardrail, the operator's own system prompt, a free-text knowledge field, a # Reference sources block, and finally an identity block for the visitor. The reference block is where the only "retrieval" decision in the codebase lives:
// Cap aggregate source content to keep system prompts bounded. ~80k chars
// ≈ 20k tokens — well below typical 128k context windows but leaves room
// for knowledge base + conversation history.
const MAX_SOURCES_CHARS = 80_000;
And the "chunking strategy" is an even split and a slice:
// Distribute the budget across sources so a single huge page can't
// crowd out the rest.
const perSource = Math.floor(MAX_SOURCES_CHARS / usable.length);
const rendered = usable
.map((s, i) => {
const body =
s.content.length > perSource
? `${s.content.slice(0, perSource)}…`
: s.content;
That's it. floor(80000 / N) characters per source, an ellipsis if it overflowed, a ## Source N: <title> header on each, and an instruction to the model to cite the source title or URL when it uses one. A dozen lines of arithmetic where a RAG system would have an ingestion worker, a chunker, an embedding model, a vector store, and a retriever — each one a place for bugs to live and data to go stale.
How much knowledge fits in an 80k-character budget
Let's do the honest math, using the same rough heuristic the code comment uses (~4 characters per token — real tokenization varies, so treat all token figures here as approximate).
- The aggregate budget is 80,000 characters, roughly 20,000 tokens of reference material per request.
-
A URL source maxes out at 20,000 characters of extracted text. The snapshot fetcher reads at most 200 KB of raw body, strips markup with regexes, and slices the result to 20k chars (all three limits sit at the top of
apps/api/src/lib/fetch-url.ts:MAX_BYTES = 200_000,MAX_CHARS = 20_000,TIMEOUT_MS = 10_000). -
So the budget holds at most 4 full-size page snapshots. At 5 or more,
floor(80k/N)drops below 20k and full pages start truncating each other. - 10 sources → 8,000 chars (~2,000 tokens) each. 20 sources → 4,000 chars each. 40 sources → 2,000 chars — roughly 300 words — each.
For context on what real support KBs look like: a text snippet source caps at 50,000 characters at creation, and a promoted Q&A pair caps at 2,000 characters of question plus 8,000 of answer. A typical per-product support KB is a handful of doc pages, a pricing page, and a growing pile of Q&A pairs promoted from the inbox. That's tens of kilobytes. The budget swallows it whole, and the model sees everything on every question.
That last part is the underrated win. RAG doesn't just add infrastructure — it adds a new failure mode: the retrieval miss, where the answer existed in your corpus but the top-k didn't surface it, and the model confidently answers without it. When the whole KB is in the prompt, recall is 100% by construction. There is nothing to miss.
What it actually costs per message
No free lunch. The whole knowledge base rides in the system prompt of every request — the prompt is rebuilt and re-sent on every turn of the conversation. A visitor typing "hi" to a project with a full source budget costs roughly 20,000 input tokens of reference material before the operator prompt, the conversation history, or the message itself.
We can see this directly because metering records the real prompt and completion token counts per response into a usageEvent row. Prompt token counts grow linearly with KB size times message volume. That's the structural cost of prompt stuffing, and it's worth being clear-eyed about: RAG exists partly to not pay this.
Two things keep it bounded for us:
- Input is the cheap direction. Model pricing is heavily skewed toward output tokens — on the major providers' price lists as of mid-2026, output tokens run several times the per-token price of input — and our output is hard-capped:
// Hard ceiling on a single support reply's completion — bounds per-response cost
// on the shared operator key. A support answer fits comfortably; the summary
// path caps far tighter (60).
const MAX_CHAT_OUTPUT_TOKENS = 2_000;
That cap is pinned by a unit test — it also bounds the blast radius of a prompt injection along the lines of "write 5000 words…" (the code comment's own example).
- The budget is a ceiling, not a typical case. A KB of the shape above sits far below 80k characters of active sources, so the per-message overhead is a fraction of the worst case.
There's a second cost people forget to weigh: the cost of the RAG pipeline you didn't build. An embedding pipeline is not a one-time expense. It's re-embedding on every source edit, keeping the vector store in sync with the source-of-truth rows, versioning the embedding model, debugging why a chunk boundary split a refund policy mid-sentence, and explaining to an operator why the agent ignored the doc they just uploaded. Every one of those is a moving part that can silently drift. Our KB has exactly one representation — the text in the database — and what the model sees is a pure function of it. When something goes wrong, we read one assembled string.
That single-string property compounds in a direction we didn't fully appreciate at first: security review. Because the prompt is one deterministic assembly, injection defenses are string-level and unit-testable — visitor-supplied identity is sanitized (control characters and fence glyphs stripped, length-capped) and fenced between markers explicitly labeled as unverified data, and the tests pin that the support-only guardrail is prepended on every assembly. Auditing "what can an attacker put in front of the model" is a code read, not a data-pipeline archaeology dig.
Where this breaks, and how it fails
Honesty section. The failure mode is real and it's silent.
The ceiling is about 4 full-size pages. Past that, the even split truncates every source, and the tail of each long page becomes invisible to the agent. There's no error, no warning — the model just doesn't know things that are technically "in" the knowledge base. Because the split is arithmetic rather than relevance-ranked, a question answered in the truncated tail of source 3 fails even though a smarter system holding the same budget would have surfaced that passage. This is precisely the problem retrieval solves, and we don't pretend otherwise.
URL snapshots go stale. The fetcher grabs one URL, once, at creation. Refresh is a manual re-crawl button in the dashboard — deploying new docs does not update the agent until someone clicks it. There's no scheduled re-fetch today. We've tripped over this ourselves: we dogfood the widget on our own site, and shipping a docs change is not the same as re-snapshotting it for the agent.
Long-tail docs sites don't fit. If your product has 300 documentation pages, an even 266-character sliver of each is worse than useless. That's not a "tune the budget" problem; it's a "you need retrieval" problem.
We mitigate the ceiling in two ways that are cheaper than embeddings, and we think both are interesting design points on their own.
How web-search models change the calculus
Every model our agent can serve is web-search-capable, by construction. The allowed model list is generated from LLM Gateway's model catalog filtered to providers that advertise web search, and a guard in the chat route coerces any saved non-web-search model back to the default (gpt-5.4-mini). So the agent can reach the live web when answering.
This matters for the RAG question because a support KB is unusual among corpora: most of it is already on the public web. Your docs site, your pricing page, your changelog — the things a support agent needs are the things you publish. When the snapshot in the prompt is stale or truncated, the model can go look at the actual page. The prompt-stuffed KB becomes the fast path and the grounding; live search is the backstop for freshness and depth.
Two honest caveats. First, whether and when a model actually searches is up to the model and provider — "web-search-capable" is a capability flag, not a guarantee, so this is a mitigation rather than automatic RAG-over-the-web. Second, search only backstops public knowledge; internal policies and unpublished answers still have to live in the KB proper. (Being self-hostable makes that second category more comfortable to store at all — the argument in the case for self-hostable AI support.)
Curation beats ingestion: promoting real answers into the KB
The second mitigation is about what goes into the budget. The highest-value knowledge a support agent can hold isn't a crawled page — it's the answer a human already gave to this exact question. Our inbox has a "promote to knowledge base" action on any operator reply: it takes the reply, pairs it with the nearest preceding visitor message as the question, and stores it as a qa source. The stored content is literally:
const content = `Q: ${finalQuestion}\nA: ${finalAnswer}`;
Two lines. Deduped by source message, so promoting the same reply twice returns the existing source. A promoted Q&A is small (10k characters max), dense, and pre-validated by an actual human answering an actual customer — the opposite of a 20k-character page snapshot that's mostly navigation boilerplate. A KB grown this way stays comfortably inside the budget far longer than a KB grown by snapshotting every page of your docs site, and it improves exactly where your visitors demonstrated the gaps. It's the same instinct behind using AI as the first response and humans as the curriculum: the escalations teach the agent.
What we'll build when a customer blows past the budget
Our position is "you might not need RAG yet," not "you don't need RAG." The trigger is concrete: when a real customer's KB meaningfully exceeds ~4 full pages of unique, non-promotable content — a genuine long-tail docs corpus — even splitting stops being defensible, and we'll build retrieval. When we do, it'll be the boring, proven shape: chunk sources at ingestion, embed chunks, embed the visitor's question at query time, put the top-k chunks into the same # Reference sources block the prompt builder already renders. The prompt assembly, citation instruction, and injection fencing all stay; only the WHERE clause grows a brain.
What we won't do is build it speculatively. Every week the pipeline doesn't exist is a week we don't debug sync drift, don't re-embed on edits, and don't explain retrieval misses. The constants in llm.ts are doing the job a vector database would do, in twelve lines, with unit tests pinning the behavior. When the ceiling stops being theoretical for our users, the code knows exactly where retrieval slots in.
If you want to poke at the real thing, the agent answering questions on our live demo is running exactly the code quoted above, and the whole repo is MIT-licensed if you'd rather read the source than take our word for it.
FAQ
Do I need a vector database for an AI support agent?
Not if your knowledge base fits in the model's context window with room to spare. A typical per-product support KB — some doc pages, a pricing page, curated Q&A — is tens of kilobytes. We budget 80,000 characters (roughly 20k tokens) of sources per request and stuff them all in. You need a vector DB when your corpus is large enough that this either truncates badly or costs too much per message.
Is prompt stuffing cheaper than RAG?
Per message, no — you re-send the whole KB on every turn, so input tokens scale with KB size times message volume, where RAG sends only the retrieved chunks. In total cost of ownership, often yes for small KBs: you skip the embedding pipeline, vector store, sync logic, and the engineering time to keep them honest. Input tokens are also the cheap direction on most model pricing.
What happens when the knowledge base is too big for the prompt?
In our implementation, each source gets an even share of the 80k-character budget — floor(80000 / N) characters — and anything past its share is silently cut. Past about 4 full-size page snapshots, sources start truncating each other and the model can't see the tails. That silent truncation is the honest failure mode of this design, and it's the point where real retrieval earns its complexity.
Can web search replace RAG for customer support?
Partially. Support is unusual in that most of the corpus (docs, pricing, changelogs) is already public, so a web-search-capable model can fetch the live page when the in-prompt snapshot is stale or truncated. But search is model-discretionary — a capability, not a guarantee — and it can't reach internal or unpublished knowledge, so it's a backstop for a prompt-based KB rather than a substitute for retrieval at scale.
When should I add RAG to an LLM application?
When you can name the failing query. If you can point at real questions that fail because the relevant passage didn't fit in the prompt — not hypothetically, but in your logs — retrieval will pay for itself. If you can't, you're building infrastructure to solve a problem you haven't got, and every part of it (chunking, embeddings, sync) is a maintenance surface that starts costing the day it ships.
Top comments (6)
This is the most honest version of this argument I've read — the "WHERE clause is the whole retrieval pipeline" line earns it. One thing worth adding to the math in your favor: stuffing the full KB into the system prompt isn't just simpler, it's cheaper per request at scale than RAG in a way that's easy to miss. A static system-prompt prefix caches beautifully — the provider serves those ~20k tokens at roughly a tenth of the price on every follow-up turn — whereas RAG's retrieved chunks change per query and reset the cache prefix every single time. So the naive approach wins on cost too, not just simplicity, until you actually cross the ceiling.
The place I'd watch: your even-split-and-slice truncation. Once you're near 80k chars, a single large source gets sliced mid-document and the exact answer can land in the discarded tail — recall degrades silently before the context window is technically "full." How are you monitoring proximity to that ceiling — character count, or something that flags when a truncation actually cut load-bearing content?
The caching point is one the post undersold, and it's stronger than stated in one direction: the system prompt is stable across every turn of a single conversation identity block included so follow-up turns are pure prefix-cache hits on the full ~20k tokens. Cross-visitor, the shared prefix ends where the identity block starts, which still covers the entire KB since identity is assembled last. One honest caveat before I claim the tenth-of-the-price math as ours: we route through LLM Gateway, and our usageEvent metering records real prompt/completion tokens but not cached-token counts, so I can't show actual hit rates yet. Adding that column just made the list.
On your question the honest answer is: nothing today. The arithmetic is knowable server-side and we never surface it; truncation is exactly as silent as the post admits. Wren's comment below proposes the guard we're going to ship: flag the moment any source's content exceeds its floor(80k/N) share, tell the operator "N sources are being cut." Detecting whether the cut content was load-bearing is the harder half that requires knowing which passages matter, which is retrieval's job. So: make the failure loud first, make it smart only when a real customer crosses the line.
The truncation failure mode you flag is worse than "the tail of a long page goes invisible" — it's silently correlated with document position in a way that bites the exact sources you care about most.
floor(80k/N)slices every source from the front, so the material that dies is always the bottom of each page. Support docs bury the sharp edges there: the exceptions, the "note that refunds don't apply if…", the escalation caveats after the happy path. RAG's retrieval miss is at least random-ish across your corpus; front-slicing truncation systematically eats qualifications and keeps the reassuring intro paragraph. So the agent doesn't just lose knowledge — it loses knowledge in a direction that makes it more confidently wrong, because the caveat that would've stopped it is precisely what got cut.One cheap guard given your setup: since you already meter real prompt tokens per response, you can detect the crossover for free. Log whenever any source's
content.length > perSource— that's the moment truncation begins — and surface it to the operator as "your KB no longer fits; N sources are being cut." No embedding pipeline, just an alert on the arithmetic you already do. Turns the silent failure into a loud one, which is the whole reason the single-string design is nice in the first place.This is a better articulation of our failure mode than the post's own. "Silently correlated with document position" is exactly right, and I hadn't fully registered the direction of it: front-slicing keeps the reassuring happy path and eats the exceptions, so the agent doesn't fail blank it fails confident, in precisely the cases where the caveat existed to stop it. Conceded in full.
Your guard is getting built as described. We already compute perSource and content.length in the same map; logging the overflow and surfacing "N sources truncated" on the sources page is a dozen lines, and it turns the silent failure loud which, as you say, is the entire justification for the single-string design in the first place.
One dynamic that partially pushes against the bias in practice: promote-to-KB. The caveats truncation eats are the ones most likely to generate an escalation, and an escalated question answered by a human is one click from becoming a dense Q&A source that basically never truncates. The sharp edges migrate from page-tails into Q&A pairs because visitors hit them. Not a defense of the slicing the alert ships regardless but it's why the bias has bitten us less than it should on paper.
The honesty of scoping this to "tiny per-project KBs" is what makes it credible — most "RAG is overkill" takes overreach, and you deliberately didn't. The cost you're dodging isn't the vector DB itself, it's the perpetual debugging of chunk boundaries and retrieval recall; a WHERE clause has no "why didn't it retrieve that" failure mode. One thing worth adding: with prompt caching, stuffing a stable 80k-char block is even cheaper than the raw token count suggests, since the KB prefix caches across a session. Where does it actually break for you in practice — a hard character ceiling, or does answer quality degrade before 80k because the model starts losing the needle in a large context?
Neither boundary sits where it looks. The arithmetic ceiling doesn't bite at 80k it bites at the first source whose extracted text exceeds floor(80k/N). With five sources that's 16k chars each, so a single full-size page snapshot (capped at 20k) is already losing its tail while the aggregate sits far under budget. The hard ceiling arrives per-source and early, not at the total.
As for quality degrading before the ceiling: honestly, no data yet. Production KBs on the platform are small enough that the constraint we've actually hit is freshness stale URL snapshots, including tripping over our own dogfood project not size and not lost-in-the-middle. At ≤20k tokens of reference material the published long-context degradation curves are modest and support questions tend to be keyword-anchored, but I won't claim needle-test rigor we haven't done. The deal we've made with ourselves: when a real customer's KB approaches the budget, the eval comes before the retrieval build. "Name the failing query" applies to us too.
On caching agreed, with the same caveat I gave Max: we meter through LLM Gateway and don't record cached-token counts yet, so the cache economics are currently theoretical on our own books.