I Fine-Tuned Gemma 4 to Read Receipts. #Hactoberfest26

Kazuto — Sun, 25 Oct 2026 04:04:32 +0000

The project my team and I worked on during the MLH Hacktoberfest Open-Source AI Hack Day on October 24, 2026 was ReceiptLens.

We were a team of first-year engineering students, and we decided to take on one of the advanced challenges: fine-tuning Google's Gemma 4 model for receipt information extraction.

What I Built

ReceiptLens is an AI-powered receipt extraction and expense analysis application.

The idea is straightforward. Instead of manually entering the information printed on a receipt, you upload an image and let the model extract it.

Our original task focused on four fields: merchant name, date, address, and total amount.

We used Google's Gemma 4 E2B vision-language model, fine-tuned with LoRA using Unsloth, to convert receipt images into structured JSON.

But as we continued building, we started thinking about what happens after the information is extracted.

An extracted total isn't particularly useful if it just sits inside a JSON object. And what happens when the model makes a mistake?

That led us to extend ReceiptLens beyond the original challenge.

We added support for extracting two more optional fields — the receipt or transaction number and merchant phone number — along with editable results, CSV/JSON exports, and a spending-analysis dashboard.

The expanded application is designed around this workflow:

Upload → Extract → Review → Correct → Save → Analyze

The idea is to turn a receipt extraction experiment into the foundation of a practical expense-management tool.

What does the output look like?

Our expanded data structure contains six fields:

{ "company": "EXAMPLE STORE", "date": "2026-10-24", "address": "12 MAIN STREET", "total": "245.50", "receipt_number": "TXN-10482", "phone_number": "012-3456789" }

This is an illustrative example of the expanded schema, not an actual model prediction.

There's an important distinction here.

Our fine-tuned model was trained and evaluated on the original four fields. The additional receipt and phone numbers are extracted through a separate prompted image-inference pass. Since the SROIE dataset doesn't provide labels for those additional fields, we haven't measured their extraction accuracy.

That means users still need to verify the output — which is exactly why we added manual correction.

How we built it?

I wanted ReceiptLens to be more than an API wrapper, so we organized it around an actual machine learning workflow.

The dataset

We used the ICDAR 2019 SROIE dataset, available on Hugging Face. It contains scanned receipts along with annotated information for company, date, address, and total.

The dataset version we used provides 626 training receipts and 361 test receipts. For our initial experiment, we configured 160 training examples, 20 validation examples, and a fixed 30-receipt evaluation sample.

Demo

KAZUTO0 / ReceiptLens

ReceiptLens directly conditions a vision-language model on both the receipt image and a text instruction. It learns to generate a structured response instead of first requiring a separate OCR engine.

ReceiptLens — Receipt Information Extraction with Gemma 4

Turning receipt images into structured data with an open-weight vision-language model and LoRA fine-tuning.

ReceiptLens is a Python-based experiment and interactive demo that extracts merchant name, date, address, total amount, receipt/transaction number, and contact number from scanned receipts. It fine-tunes Google Gemma 4 E2B using Unsloth + LoRA, then compares the adapted model with the original model on the same receipt images.

Built for HacktoberFest Hack Day — Open-Source AI, Theme 3B: Receipt Information Extraction.

Experimental project, not production-ready OCR. Results below are from a 30-receipt evaluation shared from a Colab run. On a 30-receipt test sample, complete extraction accuracy improved from 20% to 30%, with improvements in merchant, date, and total extraction.

Tech Stack: Python, Gemma 4 E2B, Unsloth, LoRA, Hugging Face, PyTorch, Google Colab, and Gradio

Key Features:

AI-powered receipt information extraction
Fine-tuned multimodal vision-language model
Editable extracted…

View on GitHub

Partner Technologies

Gemma 4 E2B is an open-weight vision-language model. It can process an image alongside a text instruction, which makes it suitable for extracting information directly from receipt photographs.

Full fine-tuning would have been expensive and difficult with our available computing resources. So we used LoRA — Low-Rank Adaptation. Rather than updating every weight in the original model, LoRA trains small additional adapter weights.

We used Unsloth to manage the vision fine-tuning workflow, with 4-bit model loading to reduce memory requirements.

Our project configuration started with LoRA rank 8, a learning rate of 0.0002, and 40 training steps.

We also kept preprocessing and evaluation separate from the training code so we could compare the original and fine-tuned models under consistent conditions.

Hugging Face provided the receipt dataset and access to the open-weight model ecosystem.

Google Colab and PyTorch provided the environment for GPU-based inference and training

And Gradio helped us build a usable Python interface without needing a separate frontend framework.

Hackathon Experience

This was a particularly interesting experience because we were all first-year engineering students, and most of us were still relatively new to practical AI development.

We chose an advanced problem despite that.

Of course, things didn't work perfectly from beginning to end.

There were dependency warnings, GPU memory considerations, and the occasional frustrating Colab error.

At one point, Colab reported that CUDA wasn't available. At another, our application couldn't find the expected project directory.

It was a reminder that building an AI application involves much more than getting a model to respond.

You have to understand your environment, manage dependencies, keep track of files and checkpoints, test the actual output, and make sure everything still works when you demonstrate it.

I also learned to be more careful about how we describe model performance.

An improvement from 20% to 30% sounds impressive as a relative percentage. But when you say that it means going from six correct receipts to nine, it becomes easier to understand both the improvement and how much work remains.

GitHub: https://github.com/KAZUTO0/ReceiptLens
Event: MLH Hacktoberfest Open-Source AI Hack Day, October 24, 2026
Team: PARADOX
Name: Sameer Kumar Karn

DEV Community: Kazuto