Safaricom PLC has published financial results every year since 2008. That is 19 years of annual reports, results booklets, and press commentaries, and almost none of it is searchable in any real sense. If you want to know what M-PESA revenue looked like in FY19, you can find a number in a spreadsheet somewhere. If you want to know why it grew the way it did, you are opening a 100 plus page PDF and skimming.
I built a project to close that gap: a conversational assistant that can answer both kinds of questions from one place, cite its sources, and honestly say "I do not know" when it does not know.
Try it here: rag-app-1003744998459.africa-south1.run.app
Code here: github.com/Derrick-Ryan-Giggs/safaricom-financial-rag
This post is less "look at my architecture diagram" and more "here is what actually went wrong while building this, and what I learned fixing it." The architecture matters, so I will cover it, but the real value of a project like this comes from the bugs, not the happy path.
What it actually does
Ask it something like "What was M-PESA revenue in FY2025, and how did that compare to the year before?" and it routes to BigQuery, generates SQL against the real mart tables, validates that SQL against the live schema before running it, and gives you a number.
Ask it "What factors drove M-PESA growth that year?" and it switches to a hybrid search over roughly 2,100 chunks pulled from 19 years of PDFs, reranks the results, and answers with a citation like (FY19, p.3) linked straight to the source PDF.
Ask it something neither source covers, and it falls back to a live web search, clearly labeled as unverified against a primary source rather than pretending it is.
Three paths, one interface, and a router deciding which one a question actually needs.
How it is put together
The unstructured side starts with unstructured[pdf] in hi_res mode, with table structure inference and Tesseract OCR, since a chunk of this corpus is scanned or image heavy investor presentations from the earliest years. Extracted elements get chunked at 500 tokens with 50 token overlap, sentence boundary aware, then embedded with an ONNX build of all-MiniLM-L6-v2 (384 dimensions, CPU only, since pulling in a GPU build of PyTorch for a project with no GPU made no sense).
Retrieval is hybrid: minsearch handles keyword matching, Qdrant handles vector similarity, and the two get combined with alpha weighted Reciprocal Rank Fusion. I grid searched that alpha weight across a 6,219 question raw ground truth set and found 0.6 (leaning slightly keyword heavy) narrowly beat the naive 0.5 default on both Hit Rate and MRR at k=20. The top 20 fused candidates then go through a cross-encoder reranker before the top 10 reach the generation step, since RRF rank and actual relevance are not the same thing.
Generation runs on Groq, using openai/gpt-oss-120b for RAG answers, SQL generation, and web fallback synthesis, and the smaller openai/gpt-oss-20b for the lightweight jobs, routing classification and answer judging during evaluation, since neither needs the bigger model's reasoning depth and the smaller model has a friendlier free tier rate limit.
The whole thing sits on Streamlit, deployed on Cloud Run with min instances set to zero, so it costs close to nothing at personal demo traffic. Conversation history is Firestore backed and session isolated, which matters more than it sounds like, since Cloud Run recycles containers and a local SQLite file simply does not survive that.
The part nobody puts in the architecture diagram: everything that broke
Filenames lied about which fiscal year they were
Safaricom's own filenames are not consistent across 19 years. Some use the starting calendar year of a fiscal year span, some use the ending year, some are two digit, some are four digit, and one 2008 era file used a single digit. Trusting the filename alone got five documents wrong, most notably three press commentaries named as if they spanned two years each, which turned out to actually be FY15, FY16, and FY17 respectively once I checked what each document's own "YEAR ENDED" statement said. I wrote a small script that checks document content against the filename derived label and only corrects the ones that actually disagree, rather than guessing on anything ambiguous.
Even after that fix, the same year was being spelled multiple ways across the corpus. FY_8 and FY8 both existed for 2008. That is not a wrong year, it is an inconsistent one, and it is a much sneakier bug: an exact match filter for a specific fiscal year would silently return zero results for whichever spelling it did not happen to check, with no error thrown. It looks exactly like "no results for this year" instead of an obvious crash. Normalizing every variant to one canonical form at load time fixed it everywhere at once, retrieval, filtering, and citations included.
A library upgrade quietly broke hybrid search
qdrant-client 1.18 removed .search() in favor of .query_points(), with a renamed parameter and a response object that wraps results differently. Nothing about the retrieval logic was wrong, the client library underneath it had just moved. This is the kind of bug that only shows up the first time you actually run something against real data rather than trusting that last month's code still matches this month's dependencies.
Qdrant Cloud enforces a rule a local instance does not
Filtering on a payload field, in my case the fiscal year, worked perfectly on a local Qdrant instance and threw a 400 error the first time the exact same code ran against Qdrant Cloud. Cloud requires an explicit index on a field before you can filter on it. Local does not. Same query, genuinely different server behavior between the self hosted and managed tiers. I fixed it by creating that index as part of collection setup itself, so any future rebuild of the collection gets it automatically instead of needing a one off manual fix every time.
The SQL path substituted a wrong answer for a right question
The first real query against BigQuery failed with "must be qualified with a dataset," because the model was writing bare table names and BigQuery needed either a fully qualified reference or a default dataset set on the query job. Easy fix. The more interesting discovery came right after: four of the seven mart tables named in my original project plan simply did not exist under those names. Not a typo, they were never built that way. I confirmed this with a direct schema check rather than assuming, and trimmed the SQL path down to the three tables that actually exist.
That led to a subtler failure mode worth calling out specifically: a real, live case of the SQL path answering a narrow question with a broader column, a total standing in for a specific category, because the specific column it actually needed was not available. A schema validation guard that rejects hallucinated table or column names does not catch this, since the substituted column was real, just wrong for the question asked. Fixing it needed an explicit instruction telling the model not to substitute a broader metric when the specific one is missing, on top of the schema guard.
The headline number looked bad until I split refusals from wrong answers
Early evaluation runs reported roughly a third of answers as "relevant," which sounds mediocre. It turned out most of those "not relevant" verdicts were the model honestly declining to answer when retrieval had not surfaced enough evidence, not the model getting something factually wrong. My first refusal detector, a set of regex patterns, undercounted these too: phrasing variations like "do not mention" instead of "does not mention" slipped straight past it as if they were substantive wrong answers. Broadening that detector and re splitting the same results, with no new model calls needed, showed the real picture: the system refuses roughly a third of the time when it genuinely lacks evidence, and is right or partly right about 97 percent of the time it actually attempts an answer.
Ground truth itself was not immune either. One reference answer for "Who are the owners of Safaricom?" came back as "ESSAR Communications," which is actually a competitor named in a "Competitive Landscape" section of the source document, not an owner at all. The question generation model had misattributed a fact to the wrong entity. Caught by manually reading failure cases rather than trusting verdicts blindly, which is a habit I would recommend to anyone building an LLM evaluation pipeline: read the actual failures yourself, at least some of them, before you trust any aggregate number.
A Docker layer was silently re-pushing 146MB per code change
Any code only change, a one line fix in retrieval/, was re-pushing the same roughly 146MB image layer as a full rebuild. Two compounding causes: a .dockerignore entry meant to exclude raw PDFs was pointing at a directory structure that did not actually exist in this repo, so it excluded nothing, and a single blanket COPY . . in the Dockerfile meant any file change invalidated one giant layer containing source code, roughly 2,100 embedded chunks, the ONNX model, and the (supposedly excluded) raw PDFs all bundled together. Splitting that into explicit per directory COPY instructions means a source only change now only rebuilds and re-pushes that one small directory's layer.
Reranking improved answers and then quietly started crashing the container
Adding a cross-encoder reranker on top of the existing embedding model meant two ONNX models living in the same process. Shortly after that shipped, Cloud Run's error dashboard started showing "Memory limit of 2048 MiB exceeded with 2249 MiB used" on the production service. Nothing about the code was wrong, the container simply needed more headroom than it used to. Bumped the memory limit, watched the errors clear.
The freshest one: a PDF table extraction bug that survived a prompt fix
This is the one I was still debugging as I wrote this post, and it is a good example of a lesson worth its own paragraph: a well written system prompt can only patch a symptom, it cannot retrieve information that genuinely was not captured correctly.
A question about FY23's net taxation payable kept coming back wrong, first citing a completely different line item's number, then, after I tightened the RAG system prompt to be more careful about ambiguous tables, citing a different wrong line item instead of the right one. Both times, a real number from the source table, just the wrong row.
The actual root cause: the PDF extraction step already computes a properly structured version of every table it detects, row by row, cell by cell, since table structure inference was already turned on. But the code reading that output was using the plain linearized text version instead, which for tables with a certain visual layout reads all the row labels first and all the numeric values second, with nothing left to tie a specific label to its specific value. The fix was not a prompt at all, it was reading the structured table representation that had been computed and discarded all along, and reconstructing each row properly before it ever reaches embedding. Prompt engineering is a real and useful tool, but it cannot fix a chunk that never had the right information paired together in the first place.
What the numbers actually say
Grid searching the hybrid search weighting across a large raw ground truth set found alpha 0.6 as the best keyword versus vector balance, narrowly ahead of an even 0.5 split, measured on Hit Rate and MRR at k=20 to match the actual width of the candidate pool the reranker sees in production.
Before this round of retrieval improvements, running the full pipeline against 500 held out questions and judging the answers against reference answers showed refusals at just under a third of all questions, meaning the system declined to answer roughly that often when it genuinely lacked evidence rather than guessing. Of the questions it did attempt, 97 percent were judged at least partially correct, with under 3 percent genuinely wrong.
A fresh, larger evaluation against the current pipeline, alpha tuning, reranking, and fiscal year filtering all included, is running now against an expanded 1,000 question benchmark. It has already caught the table extraction bug above mid run, which is honestly the best possible outcome for an evaluation harness: finding a real bug instead of just producing a score.
What is still not great, on purpose stated plainly
The reranker is a generic model, not tuned to financial document language, and several fiscal years reuse near identical marketing phrasing that can crowd out more useful, differently worded chunks in the same top 10. The BigQuery mart tables only cover a subset of the full 19 year range with some years incomplete. The web search fallback is intentionally the least trustworthy path and says so in its own answers. None of these are hidden. A project like this is more useful with its limitations stated than with them quietly smoothed over.
Why I built it this way
Financial history should not require choosing between a spreadsheet with no context and a stack of PDFs with no search. Structured data is good at the "what," annual report narrative is good at the "why," and a single assistant that knows which one to reach for, and knows when to admit neither one has the answer, is more useful than either alone.
If you want to see it live, ask it something: rag-app-1003744998459.africa-south1.run.app
If you want to see how it is actually built, including the parts covered above: github.com/Derrick-Ryan-Giggs/safaricom-financial-rag
Top comments (0)