Part 2 of a series on building a production banking AI chatbot.
By the time the classifier and the semantic cache were both live, I genuinely thought the hard part was behind us. Routing was fast. Caching worked. The demo looked clean.
Then the AI team asked me a question that sounded almost administrative.
AI Lead: "Can you send over the loan policy PDFs? We need to load them into the knowledge base."
"Sure," I said. "Give me an hour."
That hour turned into three weeks.
Chapter 1 — The PDF That Broke Everything
The first document I opened was a loan eligibility policy. Twelve pages. Looked simple enough on screen — headings, a couple of tables, some fine print at the bottom of each page.
I ran it through a basic PDF-to-text extractor, the kind that takes minutes to wire up. Skimmed the output.
It was garbage.
Not wrong garbage — worse. It was confidently wrong. Two columns of a table had been extracted side by side into one long, meaningless sentence. A header that said "Eligibility Criteria" had merged directly into the paragraph below it with no space, so it read Eligibility Criteriaapplicants must be. Numbers from a rate table were scattered through the text with no column headers attached, so a 7.2% interest rate sat next to a completely unrelated clause with nothing telling the retriever they weren't related.
Me, in the team channel: "Uh. Has anyone actually looked at what comes out the other side of PDF extraction?"
AI Lead: "...No. Why?"
I sent a screenshot. Nobody replied for a while.
A PDF looks like a document to a human because a human's eyes fill in the layout — the columns, the spacing, which number belongs to which row. To a naive extractor, a PDF is just characters scattered across a page with x-y coordinates. It has no idea "5.5%" and "Senior Citizen FD" are supposed to be read together. It just knows they happened to be near each other on page 4.
If the extraction was garbage, everything built on top of it — chunking, embeddings, retrieval — was going to be garbage too. Expensive, well-engineered garbage, but garbage.
Chapter 2 — Tables Are Where Hope Goes to Die
Loan documents live and die by their tables. Interest rate slabs. Tenure ranges. Eligibility by income bracket. If a table gets mangled during extraction, the chatbot doesn't just give a slightly worse answer — it gives a confidently wrong number, in a domain where wrong numbers have real consequences.
I tried three different extraction approaches that week.
The first flattened every table into plain text, row by row, with no structure. Technically all the numbers were "in there somewhere." Practically, a retriever chunking that text had no way to know which rate belonged to which tenure, because the structure that made it a table was gone.
The second tried to preserve table structure as markdown. Better — until a table spanned a page break, and the tool split it into two "tables" with no memory that they were originally one.
Me: "Why does the last row of page 3 not know it's connected to the first row of page 4?"
Teammate on the AI side: "Because as far as the tool's concerned, page 3 and page 4 don't know each other exist."
That one sentence explained about half of my problems that month.
The fix that actually worked wasn't a clever algorithm — it was slower and less exciting than that. We moved to a proper document-intelligence extraction pipeline that understood table boundaries and layout, ran it page by page, and then explicitly stitched tables back together when a table on one page continued onto the next. Not glamorous. Just correct.
PDF Page
│
▼
Layout-Aware Extraction
│
▼
Table detected?
│
├── Yes → Preserve rows/columns → Check: does it continue on next page?
│ │
│ Yes ───┴──► Merge with next page's table
│
└── No → Extract as normal text
Chapter 3 — Then Someone Uploaded a Scanned Document
Just as the table pipeline started behaving, someone from the client side uploaded a batch of older circulars.
Scanned. Photocopied. A couple of them slightly tilted, like they'd been placed on the scanner glass in a hurry.
Our extractor returned almost nothing. Because there was no text to extract — a scanned PDF is just an image wearing a PDF's file extension. No characters, no layout, nothing for a text extractor to find.
Me, back in the channel: "These aren't PDFs. They're photos of paper that happen to be saved as PDFs."
We had to add OCR into the pipeline — reading the image, recognizing the characters, and only then handing text over to the same extraction logic we'd already built. It worked, mostly. The tilted ones needed a de-skew pass first, or the OCR engine would misread entire lines. And a couple of circulars had someone's handwritten note scribbled in the margin — "revised, see attached" — which the OCR engine dutifully transcribed as gibberish sitting right next to a rate table, polluting the chunk it landed in.
We ended up filtering low-confidence OCR output separately instead of trusting it blindly. If the OCR engine wasn't confident about a line, we didn't feed it to the retriever at all. Better to have a small gap in the knowledge base than a hallucinated number with a straight face behind it.
Chapter 4 — The Chunking Argument That Lasted Two Days
Once extraction was in reasonable shape, the next fight was about chunking — how to split a long document into pieces small enough to embed and retrieve usefully.
My first pass just split every 500 characters, cleanly, mechanically.
Me, feeling good about it: "Chunking's done. 500 characters, done."
It took one bad retrieval to kill that confidence. A user asked about the eligibility criteria for a senior citizen FD. The retriever pulled back a chunk that started mid-sentence — "...must be above 60 years of age and hold a" — with the actual noun, "savings account," sitting in the next chunk that never got retrieved. The LLM answered from half a sentence and got the eligibility rule subtly wrong.
AI Lead: "It's not wrong. It's confidently half-right, which is worse."
We spent two days arguing about the right way to chunk — by fixed size, by paragraph, by heading section, with overlap or without. Fixed-size was simple but sliced sentences in half at random. Splitting strictly by heading kept sections whole but produced wildly uneven chunk sizes — some sections were one line, others were three pages. What actually worked was a hybrid: split along headings and paragraph boundaries first, and only fall back to a size limit if a section was too large to embed as one piece, with a small overlap between neighboring chunks so a sentence never got orphaned at a boundary again.
Document
│
▼
Split by heading / paragraph
│
▼
Section too large?
│
├── Yes → Split further, with overlap at the edges
│
└── No → Keep as one chunk
Not exciting. But it's the difference between a retriever finding a whole thought versus half of one.
Chapter 5 — Metadata, or, Why "Interest Rate" Isn't Just "Interest Rate"
Around this point, the client shared something that made everyone in the room go quiet for a second: the loan policy document we'd already ingested had three versions floating around — one from last year, one revised in March, and one marked "draft, pending approval" that had somehow ended up in the same shared folder.
Without knowing which document a chunk came from, or when it was published, our retriever had no way to tell an outdated interest rate from a current one. It would happily retrieve whichever chunk was semantically closest to the question, regardless of whether that chunk had been superseded six months ago.
Me: "So the bot could confidently quote us a rate that stopped being true in March."
AI Lead: "Yes. And nothing in the pipeline would know."
That's when metadata stopped being an afterthought. Every chunk needed to carry more than just its text — source document, version, effective date, and a status flag for anything not yet approved. Retrieval had to filter on that metadata before ranking by similarity, not after, so an outdated or draft document could never outrank a current one just because its wording happened to match the question more closely.
Query
│
▼
Retrieve candidates
│
▼
Filter by metadata (status = approved, latest version only)
│
▼
Rank remaining candidates by similarity
│
▼
Return top result
We also had to handle re-indexing — what happens when a document gets updated. The naive approach, deleting and re-embedding the whole knowledge base on every update, worked fine at ten documents and became unusable somewhere past a few hundred. We moved to updating only the chunks belonging to the changed document, keyed off that same metadata, instead of rebuilding the world every time someone fixed a typo in a PDF.
What Three Weeks of Documents Taught Me
I went into this thinking retrieval was the interesting part — embeddings, similarity search, reranking, all the pieces with actual papers written about them.
It turned out almost none of our real problems lived there. They lived one layer earlier, in the boring, unglamorous work of turning a messy real-world document into something a machine could actually reason about. A table that silently lost its structure. A scanned circular with no text at all. A sentence sliced in half at a chunk boundary. Three versions of the same policy sitting in one folder with nothing to tell them apart.
None of that shows up in a RAG architecture diagram. The diagram just says "Documents → Embeddings" and moves on, like that arrow isn't doing three weeks of work.
By the end of it, I had a new rule I still hold onto: a RAG system is only as trustworthy as the ugliest document you fed it. You can have the best retriever and the best model in the world, and none of it matters if the thing you handed them to read was quietly broken from the start.
Next up: the part where one "simple" user question turns out to be traveling through six different services before it ever reaches an answer.
Top comments (0)