Why I stopped trusting model recall and built retrieval instead

Farzam — Tue, 22 Sep 2026 05:36:11 +0000

Most AI assistants answer from what the model "remembers" from training. That's fine for trivia. It's not fine when the answer has to match a company's actual, current material, and the model just confidently makes something up.

The problem
A coaching platform needed an assistant that answered strictly from their own content, not general knowledge, and not a hallucinated approximation of it. They also needed to keep editing that content in the tools they already used, not migrate into a new CMS just to feed an AI.

The approach
I embedded the source content into a vector index and retrieved it per query, so every answer is grounded in the client's actual corpus rather than model recall. The assistant runs across multiple model providers behind one interface, so no single provider is a point of failure. Content syncs directly from the client's existing operational tooling, their non-technical team keeps editing where they already work. A scheduler drives recurring prompts with no human in the loop.

Stack: Python, FastAPI, LangChain, OpenAI, Anthropic, Pinecone, Postgres, Docker

Result: answers grounded in real content, delivered in Slack, running unattended on a schedule, with zero migration friction for the non-technical team maintaining it.

The lesson: if your AI's accuracy depends on the model remembering correctly, you don't have an accuracy problem. You have a retrieval problem.

Full breakdown → https://farzamazhar.com/#work

What it actually takes to keep an LLM pipeline running for a year, unattended

Farzam — Thu, 06 Aug 2026 10:11:34 +0000

Most AI document-processing demos fall over on the third weird PDF. Here's a contract-intelligence pipeline I built that's been running in production for over a year the architecture decisions that made that possible.

The problem
A legal team was reading thousands of sponsorship contracts by hand. The obvious fix; throw an LLM at each document breaks in production for a specific reason: a wrong extraction early in the chain propagates into every downstream report, and a job that dies late in a long chain can't afford to re-burn every prior stage of tokens.

The architecture
I modeled each document as a work unit with an explicit status enum, moving through named stages; extraction, citation, deal typing, vendor identification, overlap detection, amendment detection, redaction checks — as separate, idempotent background jobs. A stage only enqueues the next on success. If a job dies mid-chain, it resumes exactly where it broke instead of restarting from zero.

Every model call routes through a provider gateway with bounded retries, a hard timeout, and typed handling for read/connect timeouts so one provider's bad afternoon doesn't take the whole pipeline down.

Before anything reaches reporting, the pipeline writes a QA record of the model's disagreements and open issues, then halts at a human review gate. Only the flagged item blocks; the rest of the queue keeps draining.

Stack: Ruby on Rails 7.2, Hotwire, Sidekiq, Portkey, OpenAI, Google Drive API, MySQL

Result: live and unattended for over a year, processing documents in batches with a 98%+ classification confidence rate before anything hits human review.

The pattern that generalizes: write the failure path before the happy path. If you can't answer "what happens when this specific stage fails at 2am," you don't have a production system yet, you have a demo.

More on how I build these → https://farzamazhar.com/#work

DEV Community: Farzam

Why I stopped trusting model recall and built retrieval instead

What it actually takes to keep an LLM pipeline running for a year, unattended