MenuMind: From a restaurant's PDF menu to a bilingual QR menu with Gemma 3

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

MenuMind is a tool for small restaurants that turns their existing PDF menu into a digital, bilingual (Spanish/English) menu that customers open by scanning a QR code.

I'm a web development student in Spain, and where I live a lot of small bars and restaurants still run on printed menus. That works fine until a tourist sits down, or prices change, or a dish comes off the menu. Then someone has to redo the document, get it translated, and keep two or three versions in sync. Big chains have tools and agencies for that. The family-run place on the corner usually doesn't.

So the idea behind MenuMind is simple: the restaurant uploads the menu it already has, and gets back something its customers can read on their phone, in their language.

The flow looks like this:

1. Upload the menu as a PDF.
2. Extract: each page is analysed by Gemma 3 4B, which identifies dishes, categories and prices.
3. Review: the owner sees every extracted dish in an editable table and can fix names, categories or prices, or delete anything that shouldn't be there.
4. Translate: once the Spanish version is right, Gemma translates dish names and categories into English.
5. Publish: the menu is saved as the restaurant's own public page, with a Spanish/English toggle. The page also shows a QR code pointing to it, meant to be printed and placed on the tables, so customers can open the menu from their phones.

The owner's side of the app is in Spanish on purpose: the first users I have in mind are bars and restaurants around me, and the tool should speak their language. The English is for their customers.

The review step is deliberate. AI does the boring part, the typing, but the restaurant has the final word on what its customers read. A wrong price on a menu is a real problem, so I didn't want the model to be the last line of defence.

"Why not just put the PDF behind a QR code?" I asked myself this halfway through the build, and it's a fair question: plenty of bars already do exactly that. But a PDF behind a QR code is still the same document. It's still only in Spanish, so a tourist still can't read it, and on a phone it means pinching and zooming around a page designed for paper.

MenuMind isn't about showing the PDF. The PDF is just the way in. The AI's job is to save the owner from retyping sixty dishes by hand, which is the real reason small places never get around to a proper digital menu. Once the dishes are structured data (a name, a category and a price) instead of a document, the menu can be translated, read comfortably on a phone, and, as the product grows, updated or extended to more languages while the QR codes on the tables stay the same.

Who would pay for it? Small restaurants, cafés and bars, especially in areas that get tourists and need their menu in more than one language. The natural business model would be a small monthly subscription per venue, cheaper than paying for a translation every time the menu changes. Right now, though, my focus is building and validating the product, not selling it yet.

Demo

MenuMind currently runs entirely on my own machine (frontend, API, database and the model), so instead of a public link, here's a short video of the full flow: uploading a real menu PDF, reviewing the extracted dishes, translating them, opening the public menu and scanning its QR code from an Android phone on the same network. The interface you'll see is in Spanish, so turn on captions (CC) for an English walkthrough of each step.

Watch the demo on YouTube

I briefly deployed the frontend, but took it down: without the local API and model behind it, it couldn't show the real experience, and I'd rather show something that actually works.

Code

itsnazaretdev / menuMind

MenuMind

Turn a restaurant's PDF menu into a bilingual (Spanish/English) digital menu that customers open by scanning a QR code, using Gemma 3 4B running locally through Ollama.

MenuMind is built for small restaurants, cafés and bars that still rely on printed menus and need to offer them in more than one language.

The PDF is only the way in: Gemma reads it so the owner doesn't have to retype every dish. What customers get is not the PDF but a structured menu that can be translated and read comfortably on a phone.

The owner-facing interface is in Spanish, because it's designed first for restaurants in Spain. The public menu is bilingual for their customers.

Built for the Hacktoberfest Open-Source AI Challenge, Week 2: Side Hustle. The project currently runs locally; there is no public deployment.

How it works

1. Upload the menu as a PDF.
2. Extract: each…

View on GitHub

The project is a pnpm workspaces monorepo:

• apps/api: Fastify backend. PDF processing, calls to Gemma through Ollama, translation, and the menu endpoints.
• apps/web: Next.js frontend. The upload and review screen, plus the public menu page with the language toggle and QR code.
• packages/database: Prisma schema and client, backed by SQLite (restaurants, menus and menu items with their English translations).

The README has the setup steps if you want to run it yourself: pull gemma3:4b with Ollama, install with pnpm, create the database and start both apps with pnpm dev.

How I Built It

The model: Gemma 3 4B (gemma3:4b), an open-weight model from Google, running locally through Ollama. I picked it because it's multimodal, so it can look at a menu page, and small enough to run on my laptop. The same model does both jobs in MenuMind: reading the menu and translating it.

Reading the menu. Restaurant menus are hard to parse: two columns, dish names that wrap across lines, prices floating on the right, section titles in decorative fonts, an allergen table at the end. Plain text extraction loses most of that structure. So for each page, the API sends Gemma two things together:

1. The page rendered as an image (with pdfjs-dist and @napi-rs/canvas), so the model can see the layout.
2. The page's text layer, extracted with pdfjs-dist and rebuilt into lines by grouping text fragments by their position on the page.

The prompt asks Gemma to use the image to understand where each dish starts and ends, and the text to get the words right. It also has to classify the page (menu, allergens or other), join dish names that span several lines, assign each dish to the last section title above it, and use null instead of guessing when a price isn't clear. Responses come back as JSON (Ollama's format: "json"), with temperature 0 and a fixed seed so results are as repeatable as possible, and the API validates the shape before using it.

Translating. After the owner confirms the Spanish version, the dishes go back to Gemma in small batches of three, with instructions to translate dish names and categories naturally, keep proper and place names, and never touch the prices. Small batches turned out to matter: with bigger ones, a 4B model would sometimes skip items or leave a name half in Spanish.

The rest of the stack:

• Fastify for the API, with @fastify/multipart for uploads (PDF only, size-limited, and checked for a real PDF header).
• Prisma + SQLite to store restaurants, menus and items. Each item keeps its Spanish name and category plus their English versions, and only confirmed menus are served publicly.
• Next.js, React, TypeScript and Tailwind CSS for the interface.
• qrcode.react to generate the QR code on the public menu page.

What it doesn't do yet. Being honest about the gaps, the two that matter most come straight from the question above:

• Photos as input. Many small bars don't have a PDF at all, just a printed menu or a chalkboard. A photo taken with a phone is the more realistic way in, and since Gemma already reads each page as an image, it's the first thing I want to add.
• Editing after publishing. Right now, changing a price means uploading the menu again. Editing dishes directly on the saved menu is what would make "update it without redoing documents" fully true.
• A proper printable QR. Right now the QR code appears at the bottom of the public menu page; a "download / print QR" button would make it easier to put on the tables.

Beyond that, MenuMind works best with PDFs that have selectable text, translation is Spanish to English only, the owner interface is Spanish only (internationalizing it would open the tool to restaurants outside Spain), and everything runs locally, so a restaurant can't use it yet without someone running it for them.

Why Does Open Innovation Matter?

For a project like this, an open-weight model changed what was realistic for me to build.

Experimenting cost me nothing but time. Getting extraction right took a lot of iterations: rewriting prompts, re-running the same menu again and again, trying different batch sizes for translation. With a local model, that loop has no per-request bill and no rate limit, so I could iterate as much as the problem needed. For a student, that's the difference between trying an idea and not trying it.

A small model is enough for a focused job. MenuMind doesn't need a giant general-purpose model. It needs one that can read a menu page and translate dish names well. A 4B multimodal model running through Ollama does that on ordinary hardware, which makes the long-term cost of the product something I can reason about, instead of a variable tied to someone else's pricing page.

The restaurant's data stays where it's processed. Menus aren't secret, but they are a business's working documents. In MenuMind, the PDF, the extracted dishes and the translations are processed by the model running on the same machine as the app, not sent to a third-party AI API.

I can see and control the whole pipeline. Every piece, from pdf.js to Prisma to the model itself, is open. When extraction failed on a page, I could look at exactly what text and image Gemma received, change how lines were rebuilt, and try again. Nothing in the chain was a black box I had to work around.

To be fair, open isn't automatically free or better: running models still needs hardware, a bigger hosted model might extract some menus more accurately, and a 4B model needs careful prompting and validation. But open models made it possible for one student to build the whole thing, end to end, and own every part of it.

Prize Categories

• Gemma: MenuMind uses Gemma 3 4B, run locally with Ollama, for both menu extraction (vision + text) and translation.