This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
Expense Auditor is a private AI bookkeeper that runs entirely on your own laptop. You chat with it like a person ("How much did I spend on software last month?") and it answers with numbers that are always exactly right, because the AI is never allowed to do the maths.
The problem
Anyone who earns money on the side (freelancers, small shop owners, online sellers, consultants) has to track expenses. It's tedious, and getting it wrong costs real money at tax time.
AI chat assistants look like the perfect helper, but they have two serious flaws:
- AI is bad at arithmetic. A language model predicts the next word; it doesn't calculate. Ask it to total 40 receipts and you'll often get a confident, wrong number.
- Your finances end up on someone else's servers. Every receipt, vendor and amount you upload to a cloud AI leaves your control.
The solution
Expense Auditor splits the work between two very different "workers":
- 🤖 The AI understands your question and writes the friendly answer. That's all it does.
- 🧮 Ordinary code looks up your records, adds up the numbers, converts currencies and works out dates, with exact precision.
So when you ask "How much did I spend in May?", the AI doesn't guess. It asks the calculator tool, gets back exactly $78.57, and tells you. Under every answer there's a "Verified by tool calls" panel showing the exact records and date range used, so you can check the AI's work yourself.
What it can do
- 💬 Chat about your spending: totals, averages, biggest expenses, breakdowns by category.
- 🧾 Read any receipt: drop in a PDF invoice, a phone photo of a paper receipt or a screenshot, and it fills in the vendor, date, category and total by itself.
- 💱 Handle US dollars and Nigerian naira: everything is stored in dollars. A naira receipt is converted automatically (at 1 USD = ₦1,400), and you can ask for any answer in naira.
- 📅 Understand everyday date language: "May", "August 1 to 15", "last quarter", "since March".
- 🔒 Stay 100% private: the AI and the database both live on your computer. Nothing is uploaded anywhere.
- 📊 Show a dashboard: total spend, a chart by category and a table of every record.
Who would pay for it?
- Freelancers and side-hustlers who want quick, trustworthy answers without learning accounting software.
- Small businesses in Nigeria and other countries with two currencies, where expenses arrive in naira but reports are needed in dollars.
- Bookkeepers and accountants who handle many clients' receipts and can't legally or ethically upload them to a cloud AI.
- Privacy-conscious professionals (lawyers, therapists, consultants) whose expense records reveal who their clients are.
The pitch is simple: an assistant that's as easy as ChatGPT, as accurate as a calculator, and as private as a notebook in your drawer. Because it runs on hardware the customer already owns, there's no cloud bill per question. That makes a low one-off licence or a small monthly fee viable.
Demo
🎥 Screen recording:
https://drive.google.com/file/d/1npxs7l64g2gMtJamKkeA4OE4Fv1UHEZE/view?usp=sharing
Code
https://github.com/Gamey001/expense-auditor-with-calculator.git
The repository includes:
- a step-by-step setup guide written for non-experts
- 124 automated tests that run in about 2 seconds without needing the AI
- a sample-data script so you can try it straight away
The code is organised like this:
| File | What it does |
|---|---|
backend/agent.py |
Talks to the local AI and decides when to use tools |
backend/tools.py |
The "accountant": database queries and exact maths |
backend/periods.py |
The calendar tool: turns "May" into May 1 to May 31 |
backend/currency.py |
Dollar and naira conversion |
backend/receipts.py |
Reads PDFs and photos |
frontend/app.py |
The chat window and dashboard |
How I Built It
The open-source AI stack
| Piece | What I used | Its job |
|---|---|---|
| Local AI runtime | Ollama | Runs AI models on my own laptop |
| Chat model | Meta Llama 3.1 (8B), open-weight | Understands questions and picks the right tool |
| Vision model | Alibaba Qwen2.5-VL (3B), open-weight | Reads photos and scanned receipts |
| Backend | FastAPI (Python) | Runs the tools and connects everything |
| Interface | Streamlit (Python) | The chat window and dashboard |
| Database | SQLite | One small file on your computer |
The key idea: "the AI talks, the code counts"
Modern open models support tool calling. Instead of answering straight away, the model can say "please run this tool with these settings." I gave the model six tools:
- Query and calculate: finds expenses and computes a sum, average, count, minimum or maximum.
- Breakdown by category: a total per category.
- Calendar lookup: turns phrases like "last quarter" into exact dates.
- Convert currency: dollars ↔ naira.
- List categories: so the AI uses the category names that actually exist.
- Add expense: so you can say "log a ₦14,000 taxi ride today".
The AI's only job is to pick the right tool and describe the result. Every number comes from code, using exact decimal maths, so 0.10 + 0.20 is always 0.30 and never 0.30000000000000004.
What testing taught me (the honest part)
The most valuable part of this project was watching the AI fail and building guardrails so it couldn't fail the same way again.
🐛 Bug 1: "May" became October. I asked "What transactions did I make in May?". The AI said I had 26, then listed made-up transactions with the wrong amounts and dates. The real answer was one. Here's what had happened:
- The AI sent two contradictory instructions to the tool: "May 1 to May 31" and "this month".
- My code obeyed "this month", so it returned October's records.
- The AI then wrote "May" on top of October's data and filled in details it invented.
- In another test, it guessed the year 2023 for "August".
The fix: a calendar tool. I took date maths away from the AI completely, just as I'd already done with arithmetic:
- The AI now passes the user's own words ("May") to the calendar tool, and code turns them into exact dates.
- Any instruction the AI invents outside the tool's definition is ignored and flagged.
- Every answer must state the date range that was actually used, and the screen shows it.
After the fix, the same question returned the one correct transaction. I added an automated test that replays the AI's exact wrong instructions, so this bug can't come back.
🐛 Bug 2: photos of receipts crashed. The first vision model I tried (llama3.2-vision) wouldn't load on my version of Ollama. I switched to Qwen2.5-VL, which is smaller (3.2 GB instead of 7.8 GB) and better at reading documents. It read my test receipts in about 6 seconds and correctly picked the final total rather than the subtotal. It also read the Nigerian date format (15/10/2026) correctly.
🐛 Bug 3: naira was treated as dollars. A ₦32,235 grocery receipt would have been saved as $32,235. Now the AI only spots the currency, and code does the conversion: ₦32,235 is saved as $23.03, with the original naira amount kept for the records.
The lesson: don't trust the model with anything that has a single right answer. Let it handle language, and let code handle facts.
Why Does Open Innovation Matter?
For this product, open-source AI isn't a nice extra. The product couldn't exist without it.
- 🔒 Privacy is guaranteed, not promised. With a closed cloud AI, every receipt (who you paid, how much, when) goes to a company's servers, and you have to trust their privacy policy. With open models running locally, the data physically never leaves the laptop. For accountants handling client records, that's the difference between "allowed" and "not allowed".
- 💸 No per-question bill. A cloud AI charges for every question and every receipt scanned. Here, the model runs on hardware the customer already owns, so the thousandth question costs the same as the first: nothing. That's what makes a cheap side-hustle product possible.
- ✈️ It works offline. No internet, no outage, no rate limits, no API key to leak.
- 🔧 I could swap models when one failed. When the first vision model wouldn't load, I switched to another open model in minutes by changing one setting. With a closed API you get the provider's model, and you can't move to another one.
- 🧭 Every layer can be inspected. The models, the runtime and the code are all open. A cautious customer, or their auditor, can check exactly what happens to their data.
- 🌍 It fits local needs. Open tools let me build in Nigerian naira, local date formats and local receipt styles without waiting for a large company to prioritise my market.
Closed AI would have given me a smarter chatbot. Open AI let me build a trustworthy, private, affordable one, and those are the qualities people actually pay for in a financial tool.
My First User
https://drive.google.com/file/d/1PUcd4IlFDC5NfZWIdGX1syxjSyWxyfmw/view?usp=sharing
Top comments (0)