DEV Community

Blastted
Blastted

Posted on

LedgerGone: Local WhatsApp Khata & Receipt Auditor for Kirana Stores using Multimodal Gemma 3 and TigerData

#hf26challenge #python #opensource #ai

Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Submission 💸

What I Built

Small Kirana (mom-and-pop grocery) and hardware shopkeepers in India receive dozens of supplier invoices, handwritten paper slips (parchis), UPI payment screenshots (PhonePe, GPay), and WhatsApp voice notes ("Sharma ji delivered 10 boxes of soap for Rs 4,500, paid 1,500 cash, 3,000 balance pending") every day.

At closing time, tallying up scattered notes is a nightmare. Yet, shopkeepers strongly resist uploading their sensitive margin, customer credit (udhaar), and tax data to third-party cloud accounting SaaS.

LedgerGone is a 100% local-first, privacy-preserving Khata and receipt auditor for Kirana and small retail stores. It combines:

  1. Multimodal Gemma 3: Ingests receipt photos, handwritten parchis, and WhatsApp voice notes directly (with zero external OCR pipelines) to extract structured line items, GST, payments, and pending balances.
  2. TigerData (TimescaleDB + pgvector): Stores time-series financial events in partitioned hypertables for cash-flow analytics and indexes line items using pgvector embeddings for semantic natural language inventory search.
  3. 1-Click WhatsApp Khata Statements: Automatically generates polite, audited WhatsApp payment reminders with Click-to-Chat deep links (wa.me) and downloadable PDF statements.
  4. Pure CPU Optimization: Runs seamlessly on standard laptops without requiring a dedicated GPU.

Why Open Innovation Matters (The "Side Hustle" Edge)

Hacktoberfest Week 2 asks builders: Why does open innovation matter for your project, and why would someone pay for it?

  1. Unit Economics That Actually Work: Kirana stores operate on tight 4–8% margins. Traditional AI apps charging per-token cloud API bills destroy the economics of a small business tool. Because Gemma 3 and FastEmbed run locally on the shopkeeper's CPU, the cost to audit a receipt is ₹0.00. This enables LedgerGone to be sold as an affordable one-time setup or ₹299/month local utility with 100% gross margins.
  2. Absolute Data Sovereignty & Privacy: Tax invoices, cash entries, and supplier discounts never leave the merchant's machine. Shopkeepers don't have to worry about data harvesting or tax exposure from cloud accounting providers.
  3. Offline Resilience: In tier-2/3 Indian cities and rural markets where internet connectivity fluctuates, a local-first system guarantees the shop's counter operations never stall.

System Architecture 

flowchart TD
    subgraph Inputs["Kirana Counter Inputs"]
        A1["📷 Supplier GST Bills"]
        A2["📝 Handwritten Parchis"]
        A3["📱 UPI Screenshots"]
        A4["🎙️ WhatsApp Voice Notes"]
    end

    subgraph AI["Local AI Engine (CPU Optimized)"]
        B1["🧠 Multimodal Gemma 3 (Ollama)"]
        B2["⚡ FastEmbed ONNX (384-dim Vectors)"]
    end

    subgraph Database["TigerData (TimescaleDB + pgvector)"]
        C1[("⏰ TimescaleDB Hypertables<br/>Daily Cash Flow & Time-Series Audit")]
        C2[("🔍 pgvector HNSW Index<br/>Semantic Line Item & Price History")]
    end

    subgraph Outputs["Merchant Actions"]
        D1["📊 Streamlit Web Dashboard"]
        D2["💬 1-Click WhatsApp Khata Reminders"]
        D3["📄 Audited PDF Statements"]
    end

    Inputs --> B1
    B1 -->|Structured JSON| B2
    B2 -->|Vectors & Items| Database
    B1 -->|Audited Ledger| Database
    Database --> D1
    D1 --> D2
    D1 --> D3

Key Features

1. Direct Multimodal Auditing (No OCR Preprocessing)

Gemma 3's native multimodal capabilities process messy handwritten parchis and thermal receipts directly in a single pass. It extracts:

  • Vendor/Party name
  • Line items (item, quantity, rate, total)
  • GST & tax breakdown
  • Paid amount vs. pending udhaar balance
  • Payment mode (UPI / Cash / Credit)

2. Time-Series Khata Analytics (TimescaleDB Hypertables)

Every audit entry is committed as a hypertable event. LedgerGone uses Timescale's time_bucket('1 day', created_at) to calculate daily sales, collections, and outstanding credit lines in sub-millisecond queries.

3. Semantic Inventory Search with pgvector

Shopkeepers frequently ask questions like "When did we last buy basmati rice and at what rate?" or "Show me all cement and hardware bills". With 384-dimensional vector embeddings stored in TigerData's pgvector index, LedgerGone enables instant semantic similarity matching (<=> cosine distance).

4. 1-Click WhatsApp Reminders & PDF Export

Merchants can select any customer or supplier to generate a polite, formatted WhatsApp summary. Clicking "Send via WhatsApp" opens https://wa.me/ with the pre-filled statement, ready to send. An official PDF Khata statement can also be downloaded with one click.

Tech Stack

  • Language: Python 3.12+
  • Frontend: Streamlit
  • Vision & NLP: Gemma 3 Multimodal (via Ollama)
  • Time-Series & Vector DB: TigerData (TimescaleDB PostgreSQL 18 + pgvector)
  • Embeddings: FastEmbed (ONNX CPU Runtime, BAAI/bge-small-en-v1.5, 384 dimensions)
  • PDF Engine: ReportLab

Partner Track Qualifications

  • 🏆 Best Use of Gemma ($200): Leverages Gemma 3 as an end-to-end multimodal document and voice auditor for Indian retail accounts.
  • 🐯 Best Use of Tiger Data ($100): Integrates TimescaleDB hypertables for time-series cash flow aggregation alongside pgvector for semantic line-item search.

Code & Links

Top comments (0)