RAG Chatbot Development for Small Business: What Actually Changed When We Swapped in Claude Sonnet 5

Claude Sonnet 5 became the default free and Pro model on July 1. We had it running inside a client's retrieval pipeline four days later — not a toy demo, a production RAG chatbot for a small business client with about 40,000 support documents and a strict "don't make things up" requirement.

Here's what actually changed, and what didn't.

The setup

The client came to us the way a lot of small businesses do when they want to hire a developer to build a RAG system on their own data: a pile of PDFs, a shared drive, and a support inbox getting the same 30 questions on repeat. No enterprise budget, no in-house ML team, just a real need to stop answering the same email every day.

Our stack for this one is boring on purpose: pgvector for retrieval, a thin re-ranking step, and whatever LLM sits at the end answering with citations. That last part is a config value, not a rewrite. Swapping providers or model versions should be a one-line change. If it isn't, your architecture is the actual problem, not the model.

const generationModel = {
  provider: 'anthropic',
  model: 'claude-sonnet-5', // was claude-sonnet-4-6
  maxTokens: 1024,
  temperature: 0.1,
};

What got better

Two things stood out in the first 48 hours of testing against our existing eval set (about 120 real support questions with known-good answers):

• Fewer confident wrong answers. Sonnet 5 was noticeably more willing to say "the documents don't cover this" instead of blending two unrelated policy sections into something plausible-sounding. That's the failure mode that actually costs small businesses money — not the chatbot being unhelpful, but the chatbot being wrong with a straight face.
• Better use of retrieved context under pressure. When we deliberately fed it 8-10 chunks instead of 3-4 (a common real-world scenario when queries are vague), the answer quality held up better than the previous model. Less "picks the first chunk and ignores the rest."

Latency was roughly flat. Cost per query was flat. Neither was the reason to move.

What didn't change

The model swap did not fix bad chunking, and it did not fix a retrieval step that was already returning the wrong documents. We still spent most of our week on the boring stuff: chunk size, metadata filters, and re-ranking thresholds. That's the part of RAG chatbot development for small business clients that never makes it into vendor pitch decks, because it's not a feature you can screenshot.

There's a real backlash brewing right now about AI-written code quality — a widely cited 153-million-line analysis this week shows duplication up roughly 4x and rising churn where teams lean hard on agents without review. We see the same pattern in RAG systems: teams swap in a shinier model and expect it to paper over a retrieval layer nobody actually designed. It won't. The model is maybe 20% of why a RAG chatbot hallucinates. The other 80% is what you feed it.

The actual takeaway

If your RAG pipeline is architected so the generation model is a config value, model releases are good news you can act on in days, not a migration project. If it's not — if "switch models" means touching prompt templates in six different files — that's worth fixing before you worry about which model is smartest this quarter.

We're a founder-led software studio, not a faceless agency, and we test new model releases against real client pipelines because that's the only way to know if they matter. If you're trying to hire a developer to build a RAG system on your own data — a small business, not an enterprise budget — see how we scope it at duskel.com/services or write to [e-mail].