FridgeFlow AI: Never Stand Staring at an Open Refrigerator Door Again

What I Built

We've all done it: standing in the kitchen after a long day, holding the refrigerator door wide open for three minutes, letting all the cold air escape while blankly wondering:

“Do we have enough milk for tomorrow? Are those tomatoes still good? Do I need eggs, or did I buy some on Tuesday?”
Then you get to the grocery store, can't remember, buy another carton of oat milk anyway, and come home to find two already sitting in the door shelf.

FridgeFlow AI is a 100% private, local-first grocery intelligence application that lets you be genuinely lazy about managing food.
Instead of typing spreadsheets or doing manual inventory:

1. Snap a photo of your fridge, freezer, or pantry.
2. A locally running Gemma 3 (4B) vision model via Ollama detects your groceries, estimates quantities, and flags uncertain items.
3. You tap Confirm once.
4. Deterministic restock logic automatically builds your supermarket shopping list grouped by aisle. Who gets to be a little lazier? Anyone who cooks, anyone who shares a kitchen, and anyone who hates standing in grocery store aisles trying to remember what is in their crisper drawer.Key Interactive Features: Staged AI Detection Review: Visual distinction between high-confidence items and occluded warnings (needs_review). Never invents phantom groceries.

Smart Reconciliation: Rescanning your fridge updates quantities and merges existing items rather than creating duplicates.
Store Shopping Mode: High-contrast, large touch-target checklist mode built for walking supermarket aisles on your phone.
Recipe Ingredient Matcher: Compare any dinner recipe against your fridge; missing items queue to your shopping list in 1 click.
Code

The complete source code is released under the permissive MIT License:

omshree59 / Hacto-FridgeAI

🥦 FridgeFlow AI 🥛

Local-First AI Fridge Scanner & Smart Grocery Intelligence

"Your fridge. Decoded."
Snap a photo. Know what you have. Shop only what you need.

Key Features • Architecture • Quickstart • Supported AI Models • API Reference • Testing • Privacy & Security

---

📖 Overview

FridgeFlow AI is a complete, production-grade, privacy-first web application designed to eliminate open-door kitchen guesswork and unnecessary grocery spending.

Instead of relying on costly, privacy-invasive cloud APIs (like OpenAI or Google Cloud Vision), FridgeFlow AI executes 100% locally on your own computer using Ollama and vision-capable open models like Gemma 3 or LLaVA.

The Core Experience

1. Snap or Upload: Take a quick picture of your refrigerator shelves, freezer drawers, or pantry.
2. Local Vision Ingestion: Your image is sanitized, corrected for EXIF orientation, and parsed by a local AI model in seconds.
3. Editable Review: Detected items are classified with…

View on GitHub

Frontend: Next.js 16 (App Router), React 19, TypeScript, Tailwind CSS, Lucide Icons.
Backend: Python 3.11+, FastAPI, SQLAlchemy, SQLite, Pillow, HTTPX.
Local AI Runtime: Ollama with Gemma 3 4B Vision (gemma3:4b).
Test Suite: 22 automated tests (17 pytest backend + 5 Vitest frontend).
How I Built It

1. Zero-Cloud Local Computer Vision

Instead of sending private images to paid cloud APIs, FridgeFlow AI binds directly to Ollama at http://localhost:11434.

Image Pipeline: Safe image decoding with Pillow, automated EXIF orientation transposition (ImageOps.exif_transpose), and Lanczos downsampling (max 1280px) to optimize model memory without sacrificing label clarity.
Structured Pydantic Contract: The model produces strict JSON matching AIScanResult. If an item is obscured behind a jar, it marks quantity_estimate: null and sets needs_review: true—strictly avoiding hallucinations.

1. Deterministic Inventory Logic & Atomic Reconciliation

AI models should detect, not dictate. Stock evaluation is deterministic:

quantity <= 0 → out_of_stock
quantity <= threshold → low_stock
unknown quantity → unknown (never assumed as zero)

When a rescan occurs, the reconciliation engine matches normalized stems (tomatoes → tomato, while preserving distinctions like oat milk ≠ milk). It proposes updates and only modifies SQLite when confirmed.

1. Aisle-Grouped Shopping Engine

Confirmed low-stock items are automatically grouped into standard supermarket aisles: Produce, Dairy & Eggs, Meat & Seafood, Pantry & Grains, Frozen, and Beverages. Lists export to .csv or formatted .txt anytime.

Why Does Open Innovation Matter?

Open innovation is what made this project possible:

Privacy In the Home: Photographs of your personal living space and food habits should not become training data for closed advertising platforms. Running open-weight models like Gemma 3 locally guarantees zero telemetry.
Zero API Cost & Rate Limits: Cloud vision APIs charge per image and throttle requests. With open-source runtimes like Ollama, scanning your fridge 10 times a day is completely free forever.
Auditability: When tracking food stock and planning meals, you need inspectable algorithms. Because the prompt engineering, JSON validation, and database reconciliation code are open source, there are no proprietary black boxes.

My Agent Session

This entire application—from initial backend schema design to Next.js UI implementation, Pillow security validation, and automated test suites—was built and documented through an interactive pair-programming trajectory with Antigravity and preserved via DevRelay:

FridgeFlow AI: Local-First AI Fridge Scanner & Grocery List Architecture

You

Build a Complete Local-First AI Fridge Scanner & Grocery List Application named FridgeFlow AI using Next.js, FastAPI, SQLite, Pillow, and Ollama with local vision models (Gemma 3). Zero external APIs, privacy-first.

Agent

Implementation Plan for FridgeFlow AI

1. Backend Core & Database:
   • Setup FastAPI application with Pydantic v2 schemas and SQLAlchemy SQLite models.
   • Database entities: InventoryItem, GroceryListItem, ScanRecord, ScanDetection, RecipeIngredient.
   • Pillow image preprocessing pipeline (EXIF transpose, RGB sanitization, Lanczos downsampling).
2. Local Vision Engine (Ollama):
   • HTTPX integration with Ollama API (/api/generate with base64 image encoding).
   • Structured Pydantic validation for detected ingredients, confidence levels, and uncertainty flags.
3. Inventory & Grocery Services:
   • Deterministic low-stock logic (low stock, out of stock, unknown quantities).
   • Atomic inventory reconciliation to prevent duplicate items when rescanning.
   • Aisle-grouped grocery checklist with Shopping Mode and CSV/TXT exports.
   • Recipe ingredient matcher comparing meal requirements with current stock.
4. Frontend UI/UX:
   • Next.js 16 App Router with React 19, TypeScript, and Tailwind CSS.
   • Custom design system: Forest Green, Sage, Cream, Citrus accents with Light/Dark mode.
   • Dedicated Scanner, Review screen, Inventory manager, Grocery list, and History.
5. Verification & Delivery:
   • Unit and integration tests for backend (pytest) and frontend (Vitest).
   • GitHub repository sync and comprehensive README.

Show more

▸ write_file backend/app/core/config.py

Input

backend/app/core/config.py

Output

Created FastAPI settings with Ollama base URL, timeout, and storage directories.

▸ write_file backend/app/models/all_models.py

Input

backend/app/models/all_models.py

Output

Created SQLAlchemy models for inventory items, scans, detections, and grocery lists.

▸ write_file backend/app/services/ollama_service.py

Input

backend/app/services/ollama_service.py

Output

Implemented Ollama local vision integration with JSON validation and fallback recovery.

▸ write_file backend/app/services/image_service.py

Input

backend/app/services/image_service.py

Output

Implemented Pillow image verification, EXIF correction, and safe disk storage.

▸ write_file backend/app/services/inventory_service.py

Input

backend/app/services/inventory_service.py

Output

Implemented deterministic stock status computation and reconciliation proposals.

▸ write_file backend/app/services/grocery_service.py

Input

backend/app/services/grocery_service.py

Output

Implemented aisle grouping, low-stock restock generation, and CSV/text exports.

▸ write_file backend/app/services/recipe_service.py

Input

backend/app/services/recipe_service.py

Output

Implemented ingredient normalizer and shortage calculator for planned meals.

▸ run_command pytest backend/tests -v

Input

$ pytest backend/tests -v

Output

17 passed in 0.64s: image validation, JSON parsing, stock status, reconciliation, recipe match.

▸ write_file frontend/app/scan/page.tsx

Input

frontend/app/scan/page.tsx

Output

Created dedicated scanner with camera capture, staged progress, and editable detection review.

▸ write_file frontend/app/inventory/page.tsx

Input

frontend/app/inventory/page.tsx

Output

Created kitchen inventory interface with category pills, debounced search, and steppers.

▸ write_file frontend/app/grocery-list/page.tsx

Input

frontend/app/grocery-list/page.tsx

Output

Created smart grocery list with aisle grouping, low-stock generator, and touch Shopping Mode.

▸ run_command npm --prefix frontend test

Input

$ npm --prefix frontend test

Output

5 passed in 195ms: API parameter encoding, aisle categorization, and checklist progress.

▸ run_command npm --prefix frontend run build

Input

$ npm --prefix frontend run build

Output

Next.js 16 production build succeeded; all 11 static pages generated cleanly.

You

Write a good readme file

Agent

Generated comprehensive 20KB README.md covering architectural diagrams, hardware recommendations (RAM/VRAM), local Ollama setup with Gemma 3 / LLaVA / Moondream, step-by-step setup guides for Windows/macOS/Linux, Docker Compose with host-gateway mapping, complete REST API docs, and MIT License.

▸ write_file README.md

Input

README.md

Output

Created production-grade README.md and MIT LICENSE file.

You

https://github.com/omshree59/Hacto-FridgeAI this is the github repo

Agent

Removed nested gitlink in frontend directory, staged all 40 frontend source files, and committed and pushed full application and docs directly to https://github.com/omshree59/Hacto-FridgeAI.

▸ run_command git push origin main

Input

$ git push origin main

Output

4cc43e5..735bdbc main -> main successfully pushed.

6 of 6 messages

(Direct Session Link: FridgeFlow AI Session on DEV
)

The Lazy Payoff

Before FridgeFlow AI:

Staring blankly into the fridge for 5 minutes before leaving home.
Texting roommates or partners: "Do we have butter? Can you check?"
Buying duplicate produce that ends up rotting in the back of the shelf.
Wandering back and forth across supermarket aisles trying to remember what was missing.

With FridgeFlow AI:

One 3-second camera snap.
Tap Confirm Inventory → Tap Generate from Low Stock.
Walk through the store in Shopping Mode with an aisle-ordered checklist.
Estimated weekly time saved: ~45 minutes of kitchen rummaging and aisle backtracking.

Prize Categories
🌟 Primary Category: Lazy Innovation Track

(Building an open-source tool that automates tedious everyday tasks and lets humans be productively lazy).

🛡️ Secondary Category: Open Innovation Track

(Leveraging open-weight models via Ollama to protect household data sovereignty and eliminate cloud vendor lock-in).