The "Product & Impact"

rohan kovvuri — Mon, 28 Sep 2026 06:33:51 +0000

GramSetu AI: Architecting Context-Aware Multilingual Civic Tech with Episodic Agent Memory

Building conversational AI systems for civic infrastructure in rural India sounds straightforward until you run your first user test in production. Most LLM applications are designed around neat, single-session chat windows where a user asks a well-formatted question, gets an answer, and closes the tab. Rural public service delivery works nothing like that.

When a rural micro-entrepreneur interacts with GramSetu AI—our platform designed to provide hyper-local business advisory, financial structuring, and government scheme routing—they don't type crisp, self-contained English prompts. They send voice notes in regional Telugu dialects spaced two weeks apart, omit crucial context because "you already know me," and expect the system to remember their previous business idea and funding status across disparate sessions.

Early in our architecture design, we hit a wall that every AI engineer eventually faces: standard RAG (Retrieval-Augmented Generation) and naive context windows fail completely when state persistence spans weeks. Here is how we built GramSetu AI, the challenges we faced fine-tuning models for hyper-local Indian datasets, and how integrating episodic memory allowed us to build a reliable, production-ready civic assistant that operates within strict zero-cost user constraints.

What GramSetu AI Does and How It Hangs Together

GramSetu AI serves as an intelligent administrative bridge between rural citizens and complex public administration systems. At its core, the platform processes natural language voice and text queries in regional Indian languages (starting with Telugu), resolves administrative intent, evaluates local business feasibility, structures micro-financial plans, and routes users to relevant government loan schemes.

The technical architecture consists of four distinct operational layers:

[ Voice / Text Ingestion (Whisper) ]
│
▼
[ Localized NLU & Intent Engine (Fine-tuned Llama 3) ]
│
▼
[ Agent Orchestrator ]
├── Government RAG Store (Static Rules/GOs)
└── Hindsight Memory (Episodic Context)
│
▼
[ Tool Execution & Civic APIs ]

Localized Ingestion & Speech Pipeline: Powered by OpenAI's Whisper model, this layer converts low-bandwidth, noise-heavy audio streams into normalized text while preserving critical domain-specific nomenclature.
Intent Parsing & Tool Router: An asynchronous FastAPI engine backed by a locally hosted, fine-tuned Llama 3 model. It analyzes incoming messages to determine if the request requires database lookups, static document retrieval, or application state updates.
Dual Knowledge Layer:
- Static Domain Knowledge (RAG): Dense vector indexes containing official state policy documents, hyper-local market indicators, and district-level statistics collected from the Government of India.
- Dynamic Episodic Memory: Powered by Hindsight, this layer stores user-specific entities, historical interactions, business profile details, and resolved past issues across arbitrary time gaps.
Execution Core: Schema-validated tool wrappers that interface with financial calculators and scheme routing logic.

The Core Technical Story: Zero-Cost Constraints and Context Explosion

A hard constraint for GramSetu AI was cost. Our target users—rural micro-entrepreneurs—cannot afford per-token API charges or subscription tiers. The system had to be completely free for the end-user, meaning we had to aggressively optimize our infrastructure costs while running a capable LLM.

In our initial prototype, we relied on standard vector similarity search combined with a rolling ConversationBufferMemory. The strategy was simple: store all past conversation turns in Postgres, fetch the top-k semantic matches for the current prompt, and feed them into the system prompt.

This approach failed catastrophically:

Semantic Misdirection: If a user asked "Did my subsidy go through?", a vector search against past chat logs returned every instance where the user mentioned "subsidy"—including queries from six months ago regarding a completely different business idea.
Context Window Bloat: Passing whole chat histories into long context windows degraded response latency and skyrocketed our internal inference costs.
Temporal Confusion: LLMs struggle to infer timeline sequences from unordered retrieved context blocks. If a citizen reported acquiring a new loan last month, the model often hallucinated using their older, pre-loan financial profile.

We realized we needed a structured understanding of long-term agent memory architecture. We needed a system capable of extracting facts, recognizing entity updates over time, and recalling episodic history on demand without stuffing thousands of raw tokens into every inference call.

To solve this, we integrated the Hindsight open-source agent memory store. Hindsight allows GramSetu AI to treat memory as an active, queryable graph of facts and events rather than a passive string of past chat messages.

The Hardest Part: Data Collection and Fine-Tuning Llama 3

While fixing the memory architecture was crucial for UX, the hardest technical hurdle was data collection. To provide accurate hyper-local business advisory, the LLM needed to understand niche, regional context that base models completely lack.

We spent weeks manually scraping, cleaning, and verifying hyper-local data sets from official Government of India portals—district-level statistics, scheme eligibility guidelines, and local market indicators.

We then used this data to fine-tune a localized Llama 3 model. The fine-tuning process involved creating synthetic conversational datasets where the assistant had to reason about local competition, pricing potential, and seasonal factors specific to Indian rural markets.

Training the AI engine to output maximum accuracy required constant iteration. We had to implement strict guardrails to prevent the model from hallucinating non-existent government schemes or giving dangerous financial advice.

Code-Backed Implementation: Integrating Episodic Memory

Let's look at how we decoupled memory from inference to keep costs low and accuracy high.

Whenever a citizen sends a message, after running speech-to-text, we immediately digest the user turn into Hindsight. This decouples fact extraction from our main response generation path.