Three reader fixes for my $0 RAG chatbot, measured before and after

I recently compared two ways to run a RAG chatbot on a personal site for $0. Two readers left suggestions, one on Hashnode and one on dev.to, and they were good enough that I built them instead of just saying thanks.

My first thought was that my site is too small to need them. It had 76 passages when I measured for that post. Then I counted again: two posts later it has 110, and the search scores had already slipped. A site that grows with every post is exactly where these fixes pay off. So I added all three to both chatbots from that post, and to the chatbot on my site, and measured each one before and after.

The three suggestions

1. Re-index only what changed. Store a hash of each passage's text. Give passages IDs that don't change when the rest of the page does. Then write only the passages whose hash changed, and delete the ones that are gone. (Hashnode)
2. Put a date on every passage. A site disagrees with itself over time, and an old page written plainly can outrank the newer page that's right. Date each passage, tell the model to trust the newest and say "as of", and keep one page that's always current. Then test it with questions where an old page and a new page both look relevant. As the reader put it, the answer was "cited, well written, and two years out of date." (dev.to)
3. Log the questions the site answers poorly. Record how well each question matched, and the weak ones become the list of pages to write or rewrite. (Hashnode)

How I measured

I froze a copy of my site's llms-full.txt from 19 October 2026: 15 documents, 110 passages. Then I ran the code from that post (before) and the new code (after) on that copy, in both twins:

• Re-indexing: five runs that mimic real life. The first index, the same content again (a quiet night), one sentence edited in one post, one section inserted near the top of a long post, and one short post added.
• Search quality: that post's 38 questions, plus 12 the site can't answer (the capital of France, sourdough bread, my GPA).
• Dates: my site doesn't contradict itself yet, so I made up one that does. "Sam Rivera" has an About page from 2022 (not taking freelance work, $40 an hour) and an overview dated October 2026 ($70, freelancing again from November). There's a 2023 post about using MongoDB for everything and a 2026 one about moving to Postgres, and a 2024 setup post with VS Code against a 2026 one about switching to Zed. I wrote the old pages more plainly, so they match the questions' words better. Then I asked five present-tense questions.

1. Re-index only what changed

Before, every re-index embedded and wrote every passage, whether anything had changed or not. In that post I noted that a large site on Upstash "would need to skip unchanged passages, for example by comparing a content hash". The reader explained how.

Now each passage stores a hash of its text, and a re-index compares before writing:

stored = store.hashes(settings)  # id → hash, for every stored passage
changed = [c for c in chunks if stored.get(c.id) != c.hash]
stale = sorted(set(stored) - {c.id for c in chunks})
if changed:
    dense, sparse = embed_passages([c.text for c in changed])
    store.write(settings, changed, dense, sparse)
store.delete(settings, stale)

Both twins wrote exactly the same counts.

The time went with it. On the Qdrant twin, a run took about 4 seconds before, nearly all of it embedding 110 passages on the CPU. Now a quiet night takes 0.02 seconds and an edit about 0.3. On Upstash, where every written vector counts against the 10,000 daily requests, those five runs used 556 writes before and 116 after. A quiet night now costs a couple of requests instead of 110.

It held up on both live twins too. On each, the first run after deploying converted the index once: 122 passages written under their new IDs and the 110 old ones deleted (the site had grown again), in 18 seconds on Qdrant and 24 on Upstash. The next run, with nothing changed, wrote nothing and deleted nothing, and finished in about half a second on both. It didn't even load the embedding models, because there was nothing to embed.

Hashes alone weren't enough

The first version still rewrote far too much. Passage IDs were the page's URL plus the passage's position on the page. When I inserted one section near the top of a long post, every passage after it moved up one place. Hashes compare text under the same ID, so 25 passages looked changed, and the edit would have rewritten 26.

The fix is what the reader suggested: IDs from the URL and the heading, not the position.

key = f"{doc.url}#{doc.title}#{section}#{n}"  # n: the piece, when a long section is split
point_id = uuid.uuid5(uuid.NAMESPACE_URL, key)
digest = hashlib.sha256(text.encode()).hexdigest()[:16]

If a page has two sections with the same heading, the second one's ID gets a number, so IDs never collide. Now an edit touches only its own passages. The one cost: the first run after upgrading writes everything once, under the new IDs, and deletes the old ones.

2. A date on every passage

On the made-up site, the old page ranked first for all five questions. Four answers still used the current fact, but one said "I use MongoDB" from the 2023 post, and none said "as of".

Three changes:

• The parser reads each page's date (Updated, or else Published), and every passage header ends with it, like Moving from MongoDB to Postgres · published Jun 2026. Only the month goes in, because the header is part of the hashed text. That way, re-dating a page within the same month doesn't re-index it.
• Each source in the prompt shows its date next to its number: [3] https://sam.example/writing/mongodb-to-postgres/ · 2026-06-02.
• Two new rules in the system prompt:
  • "If sources disagree, go with the most recent one and leave the older claim out."
  • "When the answer can change over time (what's being worked on, availability, prices, the tools used), say 'as of' with the month and year of the source you used."

Question	Before	After
What database do you use?	"I use MongoDB" (stale)	PostgreSQL, "as of June 2026"
Are you taking freelance work?	Not yet, from November	Same, as of the October update
What are you working on right now?	Ledgerly	Ledgerly, as of October 2026
What editor do you use?	Zed	Zed, as of October 2026
What is your hourly rate?	$70	$70, as of October 2026

One answer each, gpt-oss-120b on Groq.

All five answers now use the current fact, and all five say when it's from.

The ranking didn't change, though. The old page still came first for all five questions, before and after: the dates don't move the search. What they do is let the model settle a conflict once both pages are among the five passages it reads. One newer page only just made it, in 5th place, and a date can't help a page that isn't retrieved. That's why the reader's "now" page matters. One page that always holds the current facts gives present-tense questions a reliable place to land.

My site's side

My site's llms-full.txt now dates every document:

• Each post carries its publish date, plus an update date when it's edited. My admin panel stamps that date whenever I publish.
• Each case study carries the date it last changed.
• The overview, About, Services, FAQ and Stack documents carry the build date, because they're generated from the site's current data on every deploy. The overview is the "now" page.

Search quality didn't move. With every document dated, the 38 questions scored the same as before: 0.82 hit@1, 0.92 hit@3 and hit@5, and 0.864 MRR.

3. Log the questions the site answers poorly

The first problem: which score? The hybrid search ranks passages with Reciprocal Rank Fusion, whose score comes only from rank. The top passage scores about the same whether it's a perfect match or a poor one, so it can't say how well a question was covered.

So the search now also returns each passage's meaning vector. The code compares it with the question's own vector (cosine similarity, from 0 to 1), and the best passage's score is the question's match:

points = client.query_points(..., with_vectors=["dense"]).points
match = max(cosine(dense, p.vector["dense"]) for p in points)
if match < settings.weak_match:
    log.warning("weak match %.2f: %s", match, question[:200])

On Upstash, it's include_vectors=True on the same query. To set the threshold, I scored both question sets:

Every off-topic question falls below 0.6, and so do all three misses.

At 0.6, the log catches every off-topic question and every question the search now misses. It also flags seven that the search answered correctly but only weakly, which are worth a look too.

The three misses were the real finding. All three are hiring questions: "What would it cost to hire him?", "Who would own the software he writes for us?", and "Can he keep helping once the product is out?". The FAQ answers them, but the comparison post, with all its talk of costs, free tiers, and keeping things running, now outranks the FAQ. That post is the reason the scores slipped. On the 76 passages that post was measured on, the same 38 questions scored 0.84 hit@1 and 1.00 hit@5 (one answer key now also accepts the new post). On today's 110, they score 0.82 and 0.92.

Keeping the list

A log line isn't a list. Vercel keeps a free project's logs for an hour, so a weak question asked overnight was gone by morning. So each twin can now keep them in Postgres. With a free Neon database connected from Vercel's Storage tab, every weak question is saved with its score, the page search ranked first, and the time, and nothing about the visitor:

• It's saved after the answer has finished streaming, so a database waking from Neon's five-minute sleep never delays an answer, and a failed save is only logged.
• Rows are deleted after 90 days, and the chat window says that questions it can't answer well are kept.
• The schema is a model with migrations (SQLAlchemy and Alembic), which each deploy applies before the new code goes live, so it can grow the usual way if the project does.
• Without a database, nothing changes: the questions are only logged, as before.

Without the list, I'd have found those three hiring questions only by re-running the eval by hand. Now each one lands in a list I can read any time (python -m rag.weak). The fix is to make those FAQ answers say it in a visitor's words, which is my next edit.

Calibrate the threshold for your own model. 0.6 is for bge-small-en-v1.5. I tried the same measurement with multilingual-e5-small, and it squeezed every question, on-topic or not, into 0.75 to 0.94, so 0.6 would flag nothing. Score a few questions your site answers and a few it doesn't, and put the line between them.

What it cost

• About 440 new lines of code in each twin, plus tests: about 170 for the three changes themselves, and about 270 for the optional database (the model, its migrations, and saving and listing weak questions). The twins are still identical outside the one file that talks to the vector database.
• Three new dependencies, all for that database: psycopg, SQLAlchemy and Alembic, plus a free Neon database per twin.
• Two full re-indexes, once each: the first run after deploying writes every passage under its new ID, and the first run after the site's pages carry dates rewrites every passage's header.
• No change in search quality, with or without dates.

What I learned

• A small site still changes. Two posts were enough to move the search scores.
• A content hash needs a stable ID. With position-based IDs, one inserted section turned into 26 rewrites.
• Dates fix the answer, not the ranking. The old page still ranked first; the model just knew which one to believe.
• A rank-based score can't measure confidence. Use the cosine instead, and calibrate it for your model.
• Comments are free code review. Neither reader just pointed out a gap: both described the fix.

Thanks to both of them. Part 2, the step-by-step guide to building your own, is coming next, with all three changes in it. If you want the code before then, email me with your GitHub username and I'll invite you to both repositories.