HustlePilot

HustlePilot: an AI copilot for small businesses that is not allowed to invent numbers

Draft DEV post for Hacktoberfest 2026. AI disclosure: this project and this draft were built with agent assistance (some_ai).

Small sellers — a plant nursery in this case — already have data: WhatsApp order notes, a sales CSV, a products sheet. What they don't have is time to read it every morning. HustlePilot is a local-first AI copilot that turns three CSVs into a ranked morning briefing: what's running low, what's slowing down, which inquiries went cold, and which listings are thin.

The interesting part isn't the LLM. It's the rules around it.

The one design rule: code computes, the LLM writes

Every number you see in HustlePilot — stock cover, sales rate ratios, days waiting on an inquiry — is computed by deterministic pandas rules (backend/app/services/analytics.py). The LLM's only job is to turn that evidence into a sentence a human would write.

And it may only reuse numbers that appear in the evidence. Before any narrative reaches the UI, a grounding validator (backend/app/services/grounding.py) extracts every number from the LLM's text and checks it against the evidence dict:

def safe_narrative(opp_type, evidence, llm_text, extra=None):
    """Return (narrative, source, rejected_numbers)."""
    if llm_text:
        bad = ungrounded_numbers(llm_text, evidence, extra)
        if not bad:
            return llm_text.strip(), "llm", []
        return TEMPLATES[opp_type], "template", sorted(bad)
    return TEMPLATES[opp_type], "template", []

If the model says "you sold 94 units last week" and the evidence says 14, the whole narrative is replaced with a boring-but-true template, the rejection is counted, and the UI labels the source (llm vs template). The model can't quietly delete a bad number — the entire text is forfeited. That's on purpose: partial edits are harder to audit than all-or-nothing fallback.

This mirrors a pattern the community keeps re-learning: search-enabled or retrieval-enabled LLMs still get numbers wrong, because retrieval fixes access to facts, not faithfulness in restating them (Why Search-Enabled LLMs Still Get Numbers Wrong 👤 @yany-henry). A validator at the output boundary is the part that actually enforces it (Why your AI agent keeps hallucinating financial data (and how to fix it) 👤 @valyuai).

Screenshots

Architecture in one screen

• Backend: FastAPI + SQLite (single file, tenant-scoped rows with a business_id, demo API-key auth — it's a single-tenant demo, not multi-tenant SaaS).
• LLM: any Ollama-compatible endpoint (gemma3:4b locally) or a deterministic mock. Every provider response is labeled real/mock in the UI. No GPU required; inference runs CPU-bound on a laptop.
• Frontend: React + Vite + Tailwind. Dashboard, CSV import with validation reports, opportunity feed, listing studio, inquiry inbox, approvals queue (draft → approved → exported), history timeline.
• Eval harness (eval/run_eval.py): five stages — CSV-defect injection, a hand-computed rules fixture, grounding unit checks, labeled inquiry classification vs. a keyword baseline, and latency.

The approval queue matters more than it sounds. Generated drafts never send themselves: an inquiry reply or listing draft sits as submitted until the owner approves, edits, or dismisses it. Every state change is audited.

What the eval actually measured

Honest numbers, with sample sizes:

Stage	Result
CSV validation, 30 injected defects	30/30 detected, 0 errors on clean files
Rules fixture (hand-computed expectations)	14/14 checks pass
Grounding unit checks	7/7 (fabricated numbers rejected, ratio edge cases, URL flagging)
Backend tests (pytest, mock providers)	35 passed
API smoke (end-to-end through HTTP)	36/36 checks pass
Real-model latency (gemma3:4b, CPU, n=1)	analyze ≈ 77 s · listing ≈ 50 s · inquiry ≈ 56 s

What was not measured, and I'm not going to pretend otherwise: listing-copy quality has no human rating yet; classification accuracy against the labeled fixture is a small-n, single-domain test; and the before/after comparison view in the app is correlational — nothing in HustlePilot claims an action caused a sale. The rules carry honest limitations text on every opportunity for exactly this reason.

Small things that saved the demo

• AS_OF is derived from max(sale_date), not today's clock — demos are reproducible, and "days waiting" doesn't drift between rehearsals.
• Slow-mover analysis refuses to run with fewer than recent + baseline days of history (112 by default) and says so in a notice, instead of quietly computing nonsense on two weeks of data.
• Mock everything, label everything. LLM_PROVIDER=mock and SEARCH_PROVIDER=mock make the full stack testable in CI, and the UI never lets a mock output masquerade as a real one.
• >20% invalid rows rejects the whole CSV with an explanation, rather than importing the silent majority.

Run it

# backend
cd backend && pip install -r requirements.txt
cp ../.env.example ../.env   # LLM_PROVIDER=ollama, SEARCH_PROVIDER=mock
uvicorn app.main:app --reload

# frontend
cd frontend && npm install && npm run dev   # proxies /api → :8000

# tests + eval
LLM_PROVIDER=mock SEARCH_PROVIDER=mock pytest -q
python eval/run_eval.py --stage validation --llm mock
python eval/run_eval.py --stage classification --llm real --limit 12

Hit Import sample data in the UI to load a fictional Bengaluru plant nursery (8 products, 619 sales, 5 inquiries).

TL;DR

An LLM that writes your morning briefing is a liability. An LLM that writes your morning briefing under a numeric grounding check, behind an approval queue, with every fallback visible in the UI is a tool. Code computes; the LLM narrates; the human decides.

Built for Hacktoberfest 2026. Feedback, PRs, and "you should have used X instead" all welcome.