Candidate Compliance Agent: Building a Multilingual RAG System for Tamil Nadu Election Affidavits

Hari Babu — Tue, 07 Jul 2026 01:57:29 +0000

The Problem Nobody Talks About

Every election cycle in India, candidates are not easily accessible to many so they don't the many information about candidate's they are voting for.

What I Built

Candidate Compliance Agent is an AI agent that lets anyone query Tamil Nadu election candidate affidavits in plain English or Tamil. Ask it about assets, criminal records, income declarations, education — it answers from the actual documents.

Live demo: https://affidavit-rag-frontend.vercel.app

The Technical Challenge

This wasn't a clean dataset project. The raw materials were:

Scanned PDFs (~6MB each)
Mixed Tamil and English text
Stamp paper backgrounds with notary seals overlaid on text
Inconsistent formatting across candidates

Architecture Overview

React Frontend (Vercel)
↓
Flask Backend (Railway)
↓
Hindsight Memory (recall past context)
↓
Intent Router
↓
Structured Query OR Semantic RAG
↓
Groq LLM (Llama 3.3 70B)
↓
Hindsight Memory (retain new context)
↓
Response + Follow-up Questions

Technical Feature Implemnentaion Walkthrough

OCR Pipeline

The first challenge was extracting text from scanned PDFs. I used Tesseract with Tamil + English language models:

python

text = pytesseract.image_to_string(
    image,
    lang="eng+tam",
    config="--psm 6"
)

--psm 6 treats each page as a uniform text block — critical for structured affidavit forms.

Each PDF was converted to images at 300 DPI using pdf2image, then OCR'd page by page. Output was stored as JSON with metadata:

{
  "text": "...",
  "metadata": {
    "candidate": "MKStalin",
    "party": "DMK",
    "constituency": "KOLATHUR",
    "page": 3
  }
}

Semantic Document Building

The biggest architectural decision was how to chunk the data.
Naive approach: Split each page into 500-token chunks with overlap. This gives you 600+ noisy chunks where assets, criminal records, and education are all mixed together.
Better approach: Build one semantic document per topic per candidate.
Each candidate gets 6 documents:

Identity and contact
Education
Criminal cases (structured)
Criminal detail (raw OCR text)
Income tax
Assets and liabilities

def build_docs(candidate, raw_pages):
    docs = []
    docs.append({"section": "criminal_cases", "text": f"""
Candidate: {name}
Party: {party}
Section: Criminal Cases

Pending Cases: {pending}
Convicted Cases: {convicted}
Summary: {"No criminal record" if pending == 0 else f"{pending} pending"}
"""})
    # ... repeat for each section
    return docs

35 candidates × 6 sections = 210 clean semantic documents instead of 600+ noisy chunks.
Result: When someone asks "does certain candidate have criminal cases?" the retrieval hits the criminal_cases document directly instead of a random page slice.

Embeddings and Vector Store

I used paraphrase-multilingual-MiniLM-L12-v2 for embeddings — a sentence transformer model that handles Tamil and English in the same vector space.

model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
embeddings = model.encode(texts, batch_size=32).tolist()

collection.add(
    documents=texts,
    embeddings=embeddings,
    metadatas=[{
        "candidate": d["candidate"],
        "party": d["party"],
        "section": d["section"],
    } for d in all_docs],
    ids=[f"{d['candidate']}_{d['section']}" for d in all_docs]
)

Intent Router

there are 5 intent category based on that the query is passed through

def detect_intent(query):
    q = query.lower()
    for party in PARTIES:
        if party in q:
            return "party"
    for pattern in EDUCATION_PATTERNS:
        if re.search(pattern, q):
            return "education"
    for word in SEMANTIC_KEYWORDS:
        if word in q:
            return "semantic"
    return "unknown"

Hindsight Memory Integration

Hindsight is a persistent memory service that stores and recalls context across sessions. When a user comes back the next day and asks a follow-up question, the agent remembers what they asked before.

from hindsight_client import Hindsight

client = Hindsight(
    base_url="https://api.hindsight.vectorize.io",
    api_key=os.getenv("HINDSIGHT_API_KEY")
)

def recall_memory(session_id, query):
    result = client.recall(
        bank_id="candidate-rag",
        query=f"Session: {session_id}\nQuestion: {query}"
    )
    return "\n".join([m.text for m in result.results])

def retain_memory(session_id, memory):
    client.retain(
        bank_id="candidate-rag",
        content=memory,
        metadata={"session_id": session_id}
    )

USING THE MEMORY STORED IN HINDSIGHT

1. Recall past context

past_memory = recall_memory(session_id, query)

2. Inject into RAG prompt

prompt = f"""
Previous context: {past_memory}

Context from affidavits: {retrieved_context}

Question: {query}
"""

3. Get answer

answer = ask_groq(prompt)

4. Store this exchange

retain_memory(session_id, f"User asked: {query}. Answer: {answer[:200]}")

Why this matters: A voter researching a candidate can ask multiple questions across multiple sessions and get progressively better, context-aware answers.

Contextual Follow-up Questions

After every answer, the LLM generates 4 follow-up questions based on the query, retrieved context, and memory:

ANSWER:
Poornima has no pending criminal cases.

FOLLOW_UP_QUESTIONS:

What are Poornima's total declared assets?
What is Poornima's declared annual income?
What is Poornima's educational qualification?

These appear as clickable buttons — clicking fills the input box directly.

Deployment

Backend: Flask + gunicorn on Railway
Frontend: React on Vercel
Vector DB: ChromaDB (persistent, committed to GitHub)
LLM: Groq API (Llama 3.3 70B Versatile)
Memory: Hindsight

What I Learned

OCR quality determines everything downstream. Bad OCR produces bad embeddings. The --psm 6 config change improved Tamil extraction significantly.
Semantic chunking beats token chunking. 210 topic-focused documents retrieve better than 600+ random page slices.
Persistent memory changes the interaction model. Without Hindsight, every conversation starts cold. With it, the agent builds understanding over time.
Real-world data is always messier than tutorials suggest. Stamp paper backgrounds, notary seal overlays, mixed scripts — none of this appears in RAG tutorials.

What's Next

Scale to all 234 Tamil Nadu constituencies (~4500 PDFs)
Add proactive alerts when new affidavits are filed
Support voice queries in Tamil

DEV Community: Hari Babu