ReceiptPilot: Turn a Pocket Full of Receipts into Organized Expenses with Gemma

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Small purchases create surprisingly large amounts of paperwork.

For freelancers and people running a side business, a receipt is easy to collect and easy to forget. Eventually, someone has to type its details into a spreadsheet.

I built ReceiptPilot, a receipt and expense organizer that uses Gemma to turn receipt images and PDFs into editable expense records.

The workflow is straightforward: upload, review, verify, export.

ReceiptPilot can:

Extract merchant, date, currency, subtotal, tax, total, category, and line items.

Accept JPEG, PNG, WebP, and PDF files, including scanned PDFs.

Display the original receipt beside editable fields.

Flag missing information and inconsistent totals.

Search and filter expenses by review status, category, and currency.

Export all saved expenses to CSV.

Detect uploads of the exact same receipt.

Every AI extraction starts as a review. The user checks the original, corrects any mistakes, and explicitly verifies the expense.

My intended customer is a freelancer or solo business owner who wants help organizing expenses without adopting a full accounting platform. The paid opportunity I want to explore is a lightweight tool that reduces repetitive entry and makes handing records to an accountant easier. Willingness to pay still needs validation.

Demo

Demo: https://recipe-pilot-livid.vercel.app/

Code

The source code and setup instructions are available here:

https://github.com/charan2r/recipe-pilot.git

How I Built It

ReceiptPilot uses React, TypeScript, and Vite for the interface; FastAPI and Pydantic for the backend; and SQLite for persistent storage.

The AI component is Gemma, an open-weight model accessed through Google’s hosted API. The backend sends receipt page images with instructions to return structured JSON matching the expense schema.

Before extraction, the app checks file content, prepares images, and renders PDF pages locally. Uploaded images are oriented and resized for the model request.

The extraction prompt asks Gemma to leave uncertain fields empty instead of inventing values. Server validation then checks the returned structure and flags issues such as missing fields or inconsistent subtotal, tax, and total amounts.

A few implementation decisions matter to the everyday workflow:

Human review is required. Successful extraction never automatically verifies an expense.

Currencies stay separate. The dashboard does not combine different currencies into a misleading total.

Credentials stay on the backend. The browser does not receive the provider API key.

Manual entry remains available. Organizing expenses does not depend on model availability.

The current version is a single-user application designed to run on the user’s computer. Expenses and original receipts are stored on the backend’s disk. AI extraction requires an internet connection and sends receipt images to Google after user consent.

The repository includes API, provider-contract, and frontend utility tests. Mocked provider responses check integration behavior; they do not establish live extraction accuracy. Measuring accuracy on varied real receipts is an important next step.

Why Does Open Innovation Matter?

Open-weight AI makes it possible to build a focused product around a model while retaining a path to greater control over inference.

For ReceiptPilot, that matters because receipts vary across merchants, languages, layouts, and currencies. Access to model weights creates possibilities for experimenting with specialized deployment and adaptation as the product develops.

This version uses hosted Gemma to keep the initial setup manageable. It does not yet implement local inference or fine-tuning. Moving inference to infrastructure I control would require additional engineering, but Gemma’s open weights make that a practical direction to investigate.

The application’s review and validation logic is also explicit in the code. I can change how uncertain fields are handled, add checks, and improve the workflow as users reveal where it breaks down.

My goal is a small, useful tool: spend less time copying receipt details, retain the original evidence, and keep the user involved in deciding what is correct.

Prize Categories

Best Use of Gemma — Gemma powers the receipt-to-structured-expense extraction at the center of ReceiptPilot.