I Built a Voice-First AI Khata for a Shopkeeper Who Shouldn’t Have to Type

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built Khata Ledger, a voice-first AI credit ledger for Family Friend, who runs Baba gernal store in Ludhiana.

Small neighbourhood shops often give groceries and other essentials on credit. These transactions are usually written in a notebook, saved in scattered messages, or simply remembered.

That works until the shop becomes busy.

A customer’s name gets forgotten, a payment is recorded incorrectly, or someone promises to pay on Friday without there being a useful reminder when Friday arrives.

Most accounting applications expect the shopkeeper to stop serving customers, open a form, type several fields, and understand formal bookkeeping terminology.

I wanted the interaction to feel as natural as speaking across the counter:

“Ramesh bhai ko 850 rupaye ka ration diya, Friday tak dega.”

Khata Ledger extracts:

• Customer: Ramesh
• Amount: ₹850
• Transaction: credit given
• Context: ration
• Due date: Friday

It then safely adds the transaction to the shopkeeper’s private ledger.

The shopkeeper can:

• create a private shop account;
• record a voice note directly in the browser;
• see a real waveform and recording duration;
• discard and re-record an incorrect note;
• convert natural Hindi or Hinglish into structured ledger data;
• review uncertain entries before they affect a balance;
• see every customer’s outstanding amount;
• record repayments;
• search and filter the complete passbook;
• receive reminders only when payment is due;
• generate a message containing the customer’s dated account history.

The application does not blindly trust its AI model. If an amount is missing or extraction confidence is below 0.6, the note is placed in an owner review queue instead of being added to the financial ledger.

AI can assist with bookkeeping, but uncertainty should never quietly become financial truth.

Demo

Live application: [https://khata-ledger-3deb.onrender.com]

GitHub repository:

https://github.com/amritkang165/khata-ledger

The main flow is:

Shopkeeper speaks
        ↓
Browser records temporary audio
        ↓
ElevenLabs Scribe transcribes it
        ↓
FastAPI sends the transcript to local Gemma
        ↓
Gemma extracts validated ledger fields
        ↓
Low confidence? → Owner review queue
Confident?      → Private SQLite ledger
        ↓
Balances, passbook and payment reminders

Example notes:

Ramesh bhai ko 850 rupaye ka ration diya, Friday tak dega
Sunita owes 420 rupees for school books, payment by Monday
Vanshika ne 500 rupaye wapas de diye
Deepak ko aaj 333 rupaye ka school samaan diya

Credit given increases the customer’s outstanding balance. A received payment reduces it.

Code

amritkang165 / khata-ledger

Voice-first, local-first AI credit ledger for small shopkeepers

Khata Ledger

Bol ke likho. Hisaab simple rakho.

Khata Ledger is a voice-first credit ledger for kirana stores and other small shops. A shopkeeper can speak a natural note such as:

“Ramesh bhai ko 850 rupaye ka ration diya, Friday tak dega.”

The app transcribes the audio, extracts the customer, amount, transaction type and due date, then safely adds the entry to that shopkeeper's private ledger.

Why this exists

Many neighbourhood shopkeepers still manage credit in notebooks, memory, or scattered chat messages. Traditional accounting products often expect careful typing, formal bookkeeping language, and constant connectivity.

Khata Ledger is designed around the way a shopkeeper already works:

• speak naturally in Hindi, Hinglish, Punjabi-accented English, or everyday shop language;
• review uncertain entries instead of silently saving bad financial data;
• keep customer balances separated by shop account;
• see exactly who owes money and when a reminder is actually due;
• retain the real financial…

View on GitHub

The project is built with:

• FastAPI
• SQLite
• Gemma 3 4B
• Ollama
• ElevenLabs Scribe v2
• MongoDB Atlas Vector Search
• Sentry Agent Tracing
• Tinker
• Vanilla HTML, CSS, and JavaScript
• Docker

The repository contains the complete application, tests, synthetic datasets, Atlas scripts, Tinker evaluation assets, Docker configuration, and Render Blueprint configuration.

To run it locally:

git clone https://github.com/amritkang165/khata-ledger.git
cd khata-ledger

cp .env.example .env
uv sync --extra dev

ollama pull gemma3:4b
./start.sh

Then open:

http://localhost:8000

How I Built It

Voice transcription with ElevenLabs

The browser records a temporary audio file and sends it to the FastAPI backend.

The backend forwards that audio to ElevenLabs Scribe v2. Khata Ledger uses the returned transcript but does not save the uploaded recording.

This gives the project accurate speech transcription while keeping its financial system of record separate.

Local extraction with Gemma and Ollama

The transcript is processed by Gemma 3 4B running locally through Ollama.

Gemma converts everyday shop language into validated JSON:

{
  "customer_name": "Ramesh",
  "amount_rupees": 850,
  "due_day": "Friday",
  "context": "ration",
  "transaction_type": "credit_given",
  "confidence": 0.95
}

Natural shop language can be ambiguous.

Ramesh ko 500 rupaye ka ration diya

This means the shopkeeper gave goods on credit.

Ramesh ne 500 rupaye de diye

This means the customer repaid money.

A wrong interpretation would reverse the customer’s balance.

Along with the model prompt, I added deterministic transaction-direction safeguards for important phrases such as:

• ration diya
• udhaar diya
• Friday tak dega
• payment kar diya
• jama kar diya
• paise wapas diye

This combines the flexibility of an open model with predictable financial safeguards.

A review gate for uncertain entries

The model is not allowed to write every answer directly into the ledger.

An entry requires manual review when:

• the amount is missing; or
• model confidence is below 0.6.

The shopkeeper can correct the customer, amount, context, due date, or transaction type before saving it.

Duplicate-entry protection

People double-tap buttons, mobile connections fail, and browsers retry requests.

In a credit ledger, one repeated request could create a false debt.

Every submission receives an idempotency key. SQLite enforces that the same key cannot create two transactions for the same shop owner.

The button also locks while a request is processing, but correctness does not depend only on the frontend.

Local authentication and isolated ledgers

I deliberately kept authentication beside the ledger instead of requiring a hosted identity provider.

Accounts, sessions, customers, and transactions live in the same local SQLite database.

Passwords use salted PBKDF2-SHA256 hashes. Raw session tokens are not stored, sessions expire on the server, and every query is scoped to the authenticated owner.

Each registered shop therefore receives an isolated ledger.

Useful reminders rather than instant pressure

My first implementation immediately displayed a collection message for every outstanding balance.

That felt wrong.

If someone received groceries today, the app should not immediately pressure the shopkeeper to contact them.

The updated rule shows a reminder only when:

• the promised due date has arrived; or
• the credit has remained open for at least three days.

The application never sends a message automatically.

The shopkeeper taps Send message, reviews the draft, and can copy or share it. The message contains the current balance and the complete dated history of credits and payments.

Synthetic-only MongoDB Atlas search

I used MongoDB Atlas Vector Search to retrieve similar repayment patterns.

However, Atlas only contains explicitly synthetic customer profiles. Real customers, real transactions, and the SQLite ledger are never uploaded.

This allowed me to explore vector retrieval without breaking the project’s privacy boundary.

Sentry Agent Tracing

I added Sentry spans around the extraction pipeline.

The traces record:

• model name;
• extraction latency;
• token usage;
• fallback reason;
• parsing failure;
• whether manual review was required.

Parsed financial output is redacted by default.

This helped me confirm that the local model call dominated request time while FastAPI and SQLite work remained small.

Tinker fine-tuning experiment

I also ran a separate extraction experiment using Tinker and Qwen 3.5 4B.

The Tinker model catalog available during development did not expose a Whisper or audio fine-tuning path, so I measured transcript-to-ledger extraction instead.

The reproducible synthetic dataset contains:

• 160 training examples
• 40 fixed held-out examples

Metric	Baseline	Fine-tuned
Valid JSON	100%	100%
Customer exact match	100%	100%
Amount exact match	100%	100%
Transaction direction	80%	100%

This result measures text extraction accuracy, not speech-recognition word error rate. The application’s normal runtime continues to use Gemma through Ollama.

Why Does Open Innovation Matter?

Open innovation is central to Khata Ledger because this application handles personal financial relationships.

The ledger stays under the shopkeeper’s control

The real financial ledger remains in a local SQLite database. Gemma runs on the shopkeeper’s machine through Ollama.

Customer names, balances, and transaction history do not need to be stored by a hosted LLM provider.

The model can be inspected and replaced

The extraction prompt, validation schema, language rules, and fallback logic are all visible in the repository.

I can improve the prompt, change the model, add another language, or run a different open-weight model without rebuilding the product around another provider.

Local inference reduces recurring cost

After the model has been downloaded, extraction can run locally without paying for every ledger entry.

That matters for a product intended for small shops, where a recurring AI fee could cost more than the value provided by the application.

The application can degrade safely

If Ollama becomes unavailable, Khata Ledger uses a deterministic local fallback and reports that fallback in its response.

It does not silently pretend the model succeeded.

Privacy affected the architecture

“Local-first” was not added as marketing text after development.

It changed where I placed:

• authentication;
• model inference;
• financial storage;
• observability;
• vector retrieval;
• review and validation.

The application is not completely offline because high-quality voice transcription currently uses ElevenLabs. However, its financial system of record and structured AI extraction remain local.

My Agent Session

I did not record a DevRelay session for this build.

Instead, I kept the Git history intentionally divided into small, timestamped commits so that each backend, authentication, safety, and UI decision can be traced independently:

https://github.com/amritkang165/khata-ledger/commits/main/

Prize Categories

I am entering Khata Ledger in the following applicable categories:

Best Use of Gemma

Gemma 3 4B is the open-weight model at the centre of the transaction-extraction pipeline. It runs locally through Ollama and converts natural Hinglish shop notes into validated ledger data.

Best Use of ElevenLabs

ElevenLabs Scribe v2 transcribes voice notes before the transcript is processed by the local Gemma model. Audio is not persisted by Khata Ledger.

Best Use of Tinker

I used Tinker to fine-tune and evaluate transaction extraction on a reproducible synthetic dataset. Transaction-direction accuracy improved from 80% to 100% on the fixed held-out split.

Best Use of MongoDB Atlas

Atlas Vector Search retrieves similar repayment profiles from an explicitly synthetic-only collection while real financial data remains in local SQLite.

Best Use of Sentry Agent Tracing

Sentry traces show the complete extraction request, including model latency, token usage, parsing status, review requirements, and downstream HTTP work, while financial output remains redacted.

Building for one person

After trying the project, Hardeep Singh said:

“He'll most definately use this”

Their feedback directly changed UX had to make it more simple and easily readable.

Building for one real person made every decision more concrete.

The project began with one question:

What if maintaining a credit ledger felt as natural as speaking across the counter?

Open-weight local AI made that idea practical without asking the shopkeeper to surrender control of their financial records.

Comments

[Muneer Alam]

Great project. Would be definitely jashil for a lot of people. Especially the regional language support I think would help out many and decrease the friction to use this platform