Building an AI Document Intelligence System: Architecture, LangChain, and Production Lessons

How I built a RAG system for document Q&A - and what years of working on document-heavy digital services taught me.

---

The problem: documents don't answer their own questions

Every organisation I've worked with has the same problem. Information sits inside PDFs, Word files, and scanned reports that nobody has time to read fully. A caseworker needs one line from a 40-page policy document. An analyst needs one number from page 22 of a report. The document is "available," but nobody can actually find anything in it quickly.

I've spent several years working on large, document-heavy digital services. I saw this problem again and again, in different teams and different industries. Teams either pay for a closed-source document AI tool with unclear pricing and unclear data rules, or they just keep searching by hand.

So I built AI-DocumentIntelligence - an open-source, self-hosted RAG (Retrieval-Augmented Generation) system for uploading documents and asking questions about them in plain language. I didn't want to build just another wrapper around an API. I wanted to build something simple enough that any team handling sensitive documents could trust it: open, swappable, and easy to check.

The solution: upload, chunk, embed, ask

At its core, the app does four things:

1. Takes in PDF, DOCX, or TXT documents
2. Splits them into small, meaningful chunks
3. Turns those chunks into embeddings and stores them in PostgreSQL using pgvector
4. Answers questions about the document in plain language, using either OpenAI or Anthropic Claude - you can switch between them with one setting

That last point matters more than it sounds. Most tutorial-style RAG projects lock you into one LLM provider. In real projects, that's a problem - you often need to switch providers for cost, rules, or availability, without rewriting the app. So LLM_PROVIDER is just a setting in this project, not something buried deep in the code.

Tech stack

Layer	Technology
Frontend	React 18 + TypeScript
Backend	Node.js + Express + TypeScript
AI / orchestration	LangChain
Vector store	PostgreSQL + pgvector
LLMs	OpenAI or Anthropic Claude (you choose)
Dev tooling	Docker, ts-node, dotenv

I kept the stack simple on purpose. Nothing here needs a team to learn a whole new way of deploying software. If you can already run a Node.js service and a PostgreSQL database, you can run this.

Architecture

 ┌────────────┐      ┌──────────────────┐      ┌────────────────────┐
 │  React UI   │ ───► │  Express API      │ ───► │  Document Processor │
 │ (upload+chat)│      │ (TypeScript)       │      │ (chunking / split)   │
 └────────────┘      └──────────────────┘      └────────────────────┘
                                │                          │
                                ▼                          ▼
                      ┌──────────────────┐      ┌────────────────────┐
                      │  LangChain Layer   │ ◄──► │ PostgreSQL + pgvector│
                      │ (OpenAI / Claude)    │      │  (embeddings store)   │
                      └──────────────────┘      └────────────────────┘

The chunking and embedding step runs once, when a document is uploaded. After that, each question follows a simple path: turn the question into an embedding, search pgvector for the closest chunks, send those chunks to the chosen LLM along with the question, and return the answer plus the chat history for that document.

Nothing fancy here. That's on purpose. The value isn't in a clever new architecture - it's in getting the simple parts right: choosing the right provider, splitting text well, and handling errors properly when a key or setting is missing.

Code walkthrough (the useful bits, not the whole repo)

Switching LLM providers is just reading a setting, not digging through the code:

// backend/src/llm/provider.ts (simplified)
const provider = process.env.LLM_PROVIDER; // "openai" | "anthropic"

export function getChatModel() {
  if (provider === "anthropic") {
    return new ChatAnthropic({
      apiKey: process.env.ANTHROPIC_API_KEY,
      model: process.env.ANTHROPIC_MODEL,
    });
  }
  return new ChatOpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });
}

Chunking uses LangChain's recursive splitter, tuned for Q&A rather than raw storage:

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 150,
});
const chunks = await splitter.splitDocuments(rawDocs);

The overlap matters more than most tutorials say. Too little overlap, and an answer gets cut in half across two chunks. Too much overlap, and you pay to store and search the same sentence three times. 150 overlap on a 1000-character chunk worked best across the mixed PDF and DOCX files I tested.

Retrieval and generation are kept separate. This means I can test each part on its own. I can check that retrieval finds the right chunks without spending API credits on generation - which matters a lot when you're working on a free-tier OpenAI account.

Lessons learned

What went well:

• Making the LLM provider swappable from day one paid off fast. I built and tested most of the retrieval logic using free, local embeddings before I ever touched a paid API. This kept my costs close to zero while building.
• Keeping chunking/embedding separate from the question-answering step made debugging much easier. If something looked wrong, I could quickly tell whether it was a retrieval problem or a generation problem.
• Using Docker Compose for the whole stack (backend + frontend + Postgres) meant anyone else could clone the repo and have it running in minutes, not an afternoon of dependency problems.

What was genuinely hard:

• Building without paid API credits. Building a RAG system without a paid OpenAI or Claude account is a real limit, not a small detail. I ended up testing retrieval with free, local embedding models first, then adding the paid generation step later. I'd recommend this to anyone building RAG on a tight budget, in any field.
• Getting chunk size right is unglamorous but very important. No amount of prompt tuning fixes a bad chunk split. I underestimated this at first, and fixing it gave the single biggest improvement in the whole project.
• "Works on my laptop" and "works when someone else runs docker compose up" are two different things. Getting the second one right - a clear .env.example, clear database name settings, a documented fix for port conflicts - took longer than writing the actual RAG logic. But it's the part that decides whether anyone else can actually use the project.

Where this fits into a bigger picture

I'm building this alongside a small set of open-source AI projects - a policy Q&A RAG system for citizen-facing services, an AI code review agent, a voice-to-Agile-user-story tool. They all follow the same idea: using LLM orchestration (LangChain/LangGraph) to solve real workflow problems I've seen in large-scale digital delivery work, not just building demos. If my background in production delivery plus this kind of open-source AI work is useful context for anyone looking at this project, that's a big part of why I'm writing this.

Try it / contribute

The repo is open source, and you can run it locally in a few commands:

git clone https://github.com/Srameshgitnow/AI-DocumentIntelligence.git
cd AI-DocumentIntelligence
docker compose up --build

Full setup steps, environment variables, and troubleshooting tips are in the README.

If you try it, find a bug, or have thoughts on the provider-switching approach, I'd like to hear about it - issues and pull requests are open. And if you find the project useful, a ⭐ on the repo really helps other people find it:

👉 github.com/Srameshgitnow/AI-DocumentIntelligence

---

I'm a full-stack / AI engineer (React, Node.js, LangChain/LangGraph) with a background in large-scale digital delivery. I write about applied AI engineering and open-source tools - follow along for the next post in this series.