DEV Community: Evgenii Slepinin

RAG is for finding. Full context is for deciding.

Evgenii Slepinin — Fri, 17 Jul 2026 20:39:39 +0000

The advice in 2026 is settled: chunk your documents, embed them, retrieve top-k, feed those to the model. Don't waste context. Don't waste tokens.

I built a system that deliberately does none of that. Every query gets the expert's entire CV — 69 projects, publications, education, languages. Roughly 34,000 characters, in full, every time.

Here's the reasoning, because it wasn't laziness.

The task decides the architecture

The system finds UN tenders for one specific expert. Every day it crawls 28 sources, normalises and deduplicates thousands of procurement notices, then answers one question per tender: does this person fit?

That looks like retrieval. It isn't. It's exclusion.

Retrieval asks: what in this corpus is most similar to the query?
Exclusion asks: is there anything here that disqualifies this?

Those are opposite operations. The first wants the strongest matches. The second needs the weakest link — and by definition, the weakest link is the thing least similar to everything else.

Why similarity would have failed

A real pattern from the data.

A tender is titled Public Financial Management. The expert has a dozen public-finance projects. Cosine similarity between tender and profile: very high.

# what a RAG pipeline does here
chunks = retrieve(
    query=tender.description,
    corpus=cv_chunks,
    k=3
)
# returns: finance_2019, finance_2021, education_economics
# verdict: strong match

The tender is for an environmental officer. It says so once, on page five, in the responsibilities section.

Or: the role fits perfectly, but requires a national consultant with a mandatory local language. One sentence, buried in eligibility.

The disqualifying detail is never the semantically closest passage. That's exactly what makes it disqualifying — it's the part that doesn't match the pattern. A retrieval system is optimised to not surface it.

What chunking destroys

Split a CV into chunks and you keep the facts but lose the shape.

Retrieve three chunks about public finance and the model sees an expert in public finance. Send all 69 projects and the model sees which countries he's actually worked in, which languages he actually has, what seniority he operates at, and — critically — what he has never done.

Absence is information. And absence cannot be retrieved. There is no chunk that says "this person has no environmental background". You only learn it by reading everything and noticing nothing is there.

RAG can tell you what's in a document. It can't tell you what isn't.

The economics I was told to fear

34,000 characters is roughly 9,000 tokens of context per judgement. Multiplied by hundreds of tenders a day, this is supposed to be the expensive mistake.

It isn't. The AI and the proxies together run on a few dollars a month.

Two reasons:

1. The prefilter does the cheap work first. Stop-words, theme, territory, language, deadline, expired — all in code, no model calls. The AI only ever sees candidates that survived the rules.

# rules run first, model runs last
candidates = [t for t in tenders if passes_rules(t)]
# cheap: no model calls

verdicts = [judge_with_full_cv(t) for t in candidates]
# expensive, but few

2. The CV is byte-identical on every call. Same prefix, over and over — the ideal shape for prompt caching.

The panel tracks cost per run and cost per found tender, so this isn't a guess. I watch the number.

The generalisable lesson: token cost is a real constraint, but it's one people apply from memory rather than measurement. Prices moved. Most RAG pipelines I see were designed for an economics that no longer exists.

What the model returns

Not prose. A structured verdict, logged in full:

{
  "fits": false,
  "short_reason": "national consultant required",
  "long_reason": "Role matches PFM background, but
    eligibility restricts applicants to nationals
    with working Portuguese.",
  "country": "Mozambique",
  "tor_attached": true
}

It judges the substance of the role, not its title. That's the entire point.

When I would use RAG — and do

This is not an argument against RAG. It's an argument against defaulting to it.

RAG is right when:

the corpus is unbounded — a knowledge base that grows forever, docs, support history
latency matters
the task genuinely is retrieval: find the passage, answer from it

I use RAG on other projects for exactly those reasons. Different task, different shape, different answer.

The honest caveat on my side: long context isn't free of problems either. Models attend less reliably to the middle of a long prompt than to its start and end. 34k characters sits comfortably inside what current models handle well. At 300k I'd be having a different conversation, and I'd probably be building a hybrid.

The rule I ended up with

Ask what the model has to do with the document.

Find something → retrieve. The corpus can be enormous, and finding is what embeddings are good at.
Judge something → give it everything. Judgement needs the whole picture, including the parts that aren't relevant, because irrelevance is half the verdict.

Most business tasks people are wiring RAG into are judgement tasks wearing a retrieval costume. Does this candidate fit. Is this contract risky. Should we bid on this. In every one of those, the thing that decides the answer is the detail nobody would have retrieved.

Your page ranks #1 on Google and is invisible to ChatGPT. Both are true.

Evgenii Slepinin — Fri, 17 Jul 2026 19:59:08 +0000

There's an assumption sitting quietly inside most engineering teams: if Google can crawl and render our site, AI crawlers can too.

It's wrong. And it's the single most expensive wrong assumption in technical SEO right now.

The gap

Googlebot renders JavaScript. It has done for years — the page goes into a render queue, headless Chromium executes the scripts, the result gets indexed. Deferred, but it works.

AI crawlers don't render anything.

A server-log study by Vercel and MERJ, published 17 December 2024, found that none of the major AI crawlers execute JavaScript. That includes OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic's ClaudeBot. A separate analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution.

The detail I find most telling: these crawlers do download JavaScript files. GPTBot in roughly 11.5% of requests, ClaudeBot in roughly 23.84%. They fetch the scripts and never run them.

There's no partial credit here. Either your content is in the initial HTML response, or it isn't.

The one exception is Google Gemini, which rides on Googlebot's Web Rendering Service and inherits its ability to execute JS — along with all its queue delays.

So both things are true at once

Your React site ranks on page one. Google rendered it, indexed it, ranks it.

Then a customer asks ChatGPT the exact question your page answers, and you're not in the response. Nothing is wrong with your rankings. The crawler received an empty shell:

<body>
  <div id="root"></div>
  <script type="module" src="/assets/index-a1b2c3.js"></script>
</body>

You are simultaneously first and invisible.

The trap inside the trap: structured data

This is where it gets specific.

You build a React site. You add react-helmet. You inject JSON-LD through it — Organization, LocalBusiness, FAQPage, the full graph:

<Helmet>
  <script type="application/ld+json">
    {JSON.stringify(schema)}
  </script>
</Helmet>

You open DevTools, expand <head>, and there it is. Beautiful markup. Job done.

It isn't done. DevTools shows you the rendered DOM — the page after JavaScript has run. That is not what a crawler receives.

Open view-source: instead. Not the Elements panel. If your JSON-LD only exists in DevTools and not in view-source, then GPTBot, ClaudeBot and PerplexityBot never see it.

Google will still find it, because Google renders. That's why this failure is so quiet — every tool you'd normally check with says you're fine.

One nuance worth knowing: content doesn't have to be visible prose to count. Data embedded in the initial HTML as inline JSON or server-rendered payload is in the raw response and readable. What's missed is specifically what the browser builds after load.

Why this compounds

Around 92% of ChatGPT agent queries rely on Bing's index — and Bingbot's JavaScript rendering is limited. So a client-side rendered SPA risks being absent from Bing and from every AI crawler that queries it. Two failures, one root cause.

The test takes thirty seconds

curl -s https://you.com/page | grep -c "ld+json"
curl -s https://you.com/page | grep "your headline text"

If grep returns nothing, neither does the crawler.

Do it on your pricing page, your service pages, your best article. Those are the pages with the most to lose.

The fix is not new

Server-side rendering, static generation, or prerendering. Next.js, Nuxt, Angular Universal, Django SSR — the framework matters less than the principle: put the real content in the HTML the server sends.

On our own platform the JSON-LD is rendered directly into the DOM rather than injected through Helmet, and key pages are prerendered. Not because it's elegant — because it's the difference between having schema and having schema anyone can read.

This is the same advice that's been correct for ten years. What changed is the cost of ignoring it. It used to mean waiting on Google's render queue. Now it means being absent from answers entirely, in the fastest-growing discovery channel there is.