DEV Community

Ramesh S
Ramesh S

Posted on

Building a Policy Q&A System: Architecture, RAG, and Practical Lessons

#ai #architecture #langchain #agentskills

How I built a citizen-facing chatbot that answers policy questions with real citations - and what I learned trying to make it trustworthy, not just fluent.

The problem: policy documents are technically public, practically unreadable

Public services publish huge amounts of policy information - benefit rates, application requirements, eligibility rules. All of it is technically available online. Almost none of it is easy to actually use if you're not a specialist. A person just wants to know "how much am I entitled to" or "what documents do I need," and instead they get a 30-page policy document written for caseworkers, not citizens.

The result is predictable: people call a support line to ask something the answer to which is already published somewhere, or they give up and guess. Having worked on large, document-heavy digital services, I saw this gap constantly - not a lack of information, but a lack of a way to actually ask it a question.

So I built policy-qa-system - an open-source RAG (Retrieval-Augmented Generation) chatbot that lets people ask everyday questions about public policy in plain language, and get an answer that's grounded in an actual source document, with a citation attached. Not a general-purpose chatbot guessing at policy - a system that only answers from documents it's actually been given.

The solution: retrieve, read, answer, cite

When someone asks a question, the system:

  1. Searches a store of indexed policy documents for the most relevant sections
  2. Reads those sections alongside the question
  3. Generates an answer in plain language, not policy jargon
  4. Cites the exact source, so the answer can be checked, not just trusted

That last step is the one most chatbot projects skip, and it's the one that matters most here. An answer about benefit entitlement or application requirements is only useful if someone can verify it. So every response comes with a source reference, and the system is built to say "I'm not sure" rather than guess when nothing relevant is found.

A plain-language example

Question: "How much is the weekly payment for two children?"

Answer:

For 2 children: first child gets the standard rate, second child
gets the additional-child rate (combined weekly and yearly totals
shown together). An income-based charge may apply above a set
household income threshold.

Source: Official rates document, section on child payments
Enter fullscreen mode Exit fullscreen mode

The exact numbers depend on which country's policy documents are loaded - the system itself is not tied to any one country's rules. It's a pattern for citizen Q&A, not a fixed dataset.

Tech stack

Layer Technology
Frontend React 18 + TypeScript + Tailwind CSS
Backend Node.js + Express
AI / RAG LangChain + LangGraph
Vector store PostgreSQL + pgvector
Deployment Docker

Nothing exotic here either. A team running this for real would replace the sample document set with their own official policy documents, and everything else - retrieval, citation, plain-language generation - stays the same.

Architecture 

 User question
      │
      ▼
 React UI ──► Express API
      │
      ▼
 LangChain retrieval ──► PostgreSQL + pgvector (indexed policy docs)
      │
      ▼
 LLM answer generation (grounded in retrieved sections)
      │
      ▼
 Answer + citation + confidence score
Enter fullscreen mode Exit fullscreen mode

The confidence score matters as much as the answer. If the retrieval step doesn't find anything close enough to the question, the system is designed to say so, rather than let the LLM fill the gap with a plausible-sounding guess.

Code walkthrough (the useful bits)

Every answer is required to carry a source, not just generated freely:

const response = await chain.invoke({
  question,
  context: retrievedChunks,
});

return {
  answer: response.answer,
  sources: retrievedChunks.map(c => ({
    document: c.metadata.document,
    section: c.metadata.section,
  })),
  confidence: response.confidence,
};
Enter fullscreen mode Exit fullscreen mode

A confidence threshold decides whether to answer or say "not found":

if (topMatchScore < CONFIDENCE_THRESHOLD) {
  return {
    answer: "I couldn't find a reliable answer to that in the indexed documents.",
    sources: [],
  };
}
Enter fullscreen mode Exit fullscreen mode

This one check is small in terms of code, but it's the difference between a chatbot people can trust and one that just sounds confident.

Lessons learned

What went well:

  • Making citations a required part of every answer, not an optional extra, forced better retrieval quality from the start - a vague retrieval result becomes obvious as soon as you have to show where it came from.
  • The confidence threshold, even a simple version of it, cut down on confident-sounding wrong answers a lot. It felt like a small addition but had one of the biggest effects on trust.
  • Keeping the document set swappable meant the same system could, in theory, work for very different policy areas just by re-indexing a different set of documents.

What was genuinely hard:

  • Plain-language translation without losing accuracy. Policy documents are precise for a reason - simplifying the wording without dropping an important condition (an income threshold, an eligibility rule) took real care, and I don't think it's fully solved yet.
  • Deciding when to say "I don't know." Set the confidence threshold too low, and the system answers questions it shouldn't. Set it too high, and it refuses questions it could actually answer well. I ended up tuning this by hand against a small test set of real questions.
  • This is a starting point, not a finished, ready-to-deploy system. It needs a real document ingestion pipeline, ongoing review of the sample answers, and proper testing against edge cases before anyone should point real users at it. I'd treat it as a solid base to build on, not a drop-in, ready-to-launch tool.

Where this fits into a bigger picture

This is one of a small set of open-source AI projects I'm building - a document Q&A RAG system, an AI code review agent, a voice-to-Agile-user-story tool. The common thread is using LLM orchestration (LangChain/LangGraph) on real workflow problems, with a strong bias toward answers that can be checked rather than answers that just sound right.

Try it / contribute

The repo is open source and quick to try locally:

git clone https://github.com/Srameshgitnow/policy-qa-system.git
cd policy-qa-system
npm install
npm run dev
Enter fullscreen mode Exit fullscreen mode

Full setup steps, environment variables, and how to load your own documents are in the README.

If you try it, find a gap in the citation logic, or have ideas on the confidence threshold, I'd like to hear about it - issues and pull requests are open. And if the project is useful to you, a ⭐ on the repo helps other people find it:

👉 github.com/Srameshgitnow/policy-qa-system

I'm a full-stack / AI engineer (React, Node.js, LangChain/LangGraph) with a background in large-scale digital delivery. I write about applied AI engineering and open-source tools - follow along for the next post in this series.

Top comments (0)