Dinner Wasn't Missing Ingredients. It Was Missing Coordination: Building New Dish with Local Open-Source AI

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

The idea arrived during one of those completely ordinary dinner moments.

I opened the fridge and found tomatoes, some cooked rice, half an onion, and several other ingredients that were all technically food but did not yet feel like dinner. A one of my friend had spinach and chickpeas and the other had carrots and yogurt.

Individually, none of us had an exciting plan. Together, we almost had a meal.

That thought became New Dish: a local-first community kitchen that turns ingredients from one to three households into one real, safe recipe.

It works for one person too. You can type what you have, photograph an ingredient list, or take a picture of your fridge. New Dish proposes ingredients, asks you to confirm them, applies the group's allergies and preferences, and matches everything against a documented recipe.

The people I imagine paying for this are roommates, shared apartments, hostels, small community kitchens, and sustainability groups trying to reduce food waste. A paid version could add a shared weekly pantry, expiry reminders, shopping lists, and private household groups. The core version stays useful without an account or an API bill.

How the idea changed

My first version was called CrowdChef. It imagined nearby households discovering one another and pooling leftovers.

It sounded fun, but it was too large. Location, strangers, trust, scheduling, and food safety all arrived before anyone had cooked a single thing.

So I reduced the idea to a small, real moment:

1. One person should be able to use it.
2. A few known neighbors or roommates can join.
3. Everyone's allergies must be treated as group-wide boundaries.
4. The result has to be an actual edible recipe, not an AI improvisation.
5. Nutrition must come from a source, not from a model guessing confidently.

That smaller idea was much better. The cartoon interface and the animated pepper guide, Peper, came later because dinner planning should feel friendly—not like filling out tax software.

The full New Dish workspace: kitchens on the left, a verified recipe on the right.

Demo

🚀 Live demo: New Dish

The quickest way to try it:

1. Choose one, two, or three kitchens.
2. Rename the neighbor kitchens if you want.
3. Add ingredients manually or scan a photo.
4. Select allergies and cooking preferences.
5. Create the dish and review the source recipe, missing ingredients, instructions, and per-serving nutrition.
6. Ask Peper to read the recipe aloud.

The app is responsive, so the same flow works on a phone:

Code

The complete project is open on GitHub:

nikisha77 / New-Dish

New Dish

New Dish is a cheerful, local-first recipe matcher for one person or a group. Kitchens can type ingredients or confirm suggestions from an ingredient-list/fridge photo, choose group-wide dietary boundaries, and get a documented edible recipe with source nutrition per serving.

Local AI

• Verified recipes: UniTools Recipes v1 (CC BY-SA 4.0), including quantities, detailed method, timings, licensed photos, and source-provided calories/protein/fat/carbs per serving.
• OCR: Tesseract.js with local image preprocessing.
• Fridge suggestions: Xenova/detr-resnet-50 through Transformers.js. Detections always require confirmation.
• Optional wording helper: HuggingFaceTB/SmolLM2-360M-Instruct, falling back to onnx-community/Qwen2.5-0.5B-Instruct.
• Voice: onnx-community/Kokoro-82M-v1.0-ONNX through Kokoro.js.
• Models run in browser workers. Model files are downloaded once and cached by the browser.
• The language model never creates recipe facts or nutrition. If no safe source recipe matches, the app says so instead of fabricating one.

Run locally

python3 -m http.server 4173 --directory dist

Then open http://127.0.0.1:4173.

or try here - https://nikisha77.github.io/New-Dish/#recipe

View on GitHub

Repository: github.com/nikisha77/New-Dish

There is no backend and no secret API key. The application is static HTML, CSS, JavaScript, Web Workers, open models, and an open recipe dataset.

How I Built It

The architecture

New Dish deliberately separates recognition, recipe truth, and presentation.

Fridge photo / ingredient label
              ↓
   Local OCR + object detection
              ↓
      Human confirmation
              ↓
 Allergies + dietary preferences
              ↓
   Verified open recipe matching
              ↓
 Recipe, licensed photo, instructions
 and sourced nutrition per serving
              ↓
       Peper's local voice

1. Reading a fridge without uploading it

For printed ingredient lists, I use Tesseract.js in the browser. Before OCR, the image is resized and processed using grayscale, contrast enhancement, and a second high-contrast pass when the first result is weak.

For fridge photographs, a DETR ResNet-50 object-detection model runs through Hugging Face Transformers.js in a Web Worker. The model never receives an uploaded URL from a server; it works with the image selected in the browser.

The most important part is what happens after inference: the app does not automatically trust it.

My first fridge test was humbling. The recognizer found one correct word—apple—surrounded by a parade of meaningless fragments. The pipeline had technically succeeded, but the result was useless.

I fixed that by:

• matching OCR output against a vocabulary built from the recipe dataset;
• rejecting random text fragments;
• deduplicating singular and plural ingredient names;
• requiring higher confidence for fridge detections;
• preselecting only reliable suggestions; and
• requiring the person using the app to confirm every ingredient.

That failure changed the product. A confident model is not the same thing as a correct model.

2. Refusing to let a language model invent dinner

The early prototype asked a small language model to create a recipe. That was exciting until I asked a more important question: Would I actually cook and serve this?

A plausible sentence is not evidence that a recipe exists. The same problem applies to nutrition. A model can produce wonderfully formatted calories and macros that have no relationship to the actual meal.

So the final architecture uses UniTools Recipes v1, an open CC BY-SA 4.0 collection of documented recipes. Each record includes ingredient quantities, scaling behavior, detailed steps, timings, a licensed image, and source-provided calories, protein, carbohydrates, and fat per serving.

The browser matches confirmed pantry ingredients against that data, removes recipes conflicting with selected allergies, applies preferences such as vegan, vegetarian, gluten-free, dairy-free, high-protein, easy, and cooking-time limits, then scales the ingredient quantities for the requested number of people.

If no safe recipe matches, New Dish says so. It does not manufacture one to avoid an empty screen.

3. Open models with graceful fallbacks

The optional local language helper uses SmolLM2 360M Instruct as its primary open-weight model and Qwen2.5 0.5B Instruct as the fallback. Both run through Transformers.js, using WebGPU when available and WASM otherwise.

They are intentionally not responsible for recipe facts or nutrition. Their role is limited to presentation help, where a wording error is inconvenient rather than unsafe.

Peper's natural voice uses the open Kokoro 82M model locally. If that model cannot load, the app falls back to the browser's device voice. Peper's face animates while speaking, which is a small detail but makes the interface feel alive.

4. Safety and privacy in the interface

Allergy exclusions apply to the complete group. The current list includes peanuts, tree nuts, dairy, eggs, gluten, soy, fish, shellfish, sesame, mustard, celery, lupin, and sulphites.

Typed ingredients that match a selected allergen are removed from recipe matching, and recipe records are checked too. The interface still reminds people to verify packaging and cross-contact because software cannot inspect a kitchen.

Neighbor names are chosen by the user and saved only in browser storage. Images and ingredients stay on the device. No account is required.

5. Shipping it

New Dish is a buildless static application hosted on Github pages. That keeps deployment simple.

Why Does Open Innovation Matter?

Open innovation is not decoration in this project. It changes whether the idea is practical.

With closed APIs, every photo scan, recipe request, and spoken instruction could become a metered request. That would make a small household tool expensive before it had its first paying customer. It would also require sending photographs of people's kitchens and food habits to servers they did not choose.

Using open models locally gave me four important freedoms:

• Privacy: fridge photographs and ingredient lists stay on the device.
• Low operating cost: using the app does not create a growing inference bill.
• Replaceability: SmolLM2, Qwen, DETR, Tesseract, and Kokoro can be updated or exchanged without redesigning the product around one vendor.
• Honesty: model output can be constrained to what models do well, while recipe and nutrition facts come from inspectable open data.

The most valuable freedom is the ability to disagree with the model. I can put confidence thresholds around it, add a food vocabulary, require human confirmation, or remove it entirely from a safety-sensitive step. A closed black-box response would give me fewer places to intervene.

Open did not simply make New Dish cheaper. It made the product's privacy and trust model possible.

My First User

My first user was me, standing in front of a fridge and expecting the camera feature to understand what I could already see.

It did not work beautifully on the first attempt. It returned one correct apple and a lot of nonsense. That was actually the most useful feedback I could have received because it exposed a dangerous assumption: I was treating “the model returned something” as “the model understood the image.”

After adding food-vocabulary validation and confidence thresholds, I tested the exact failed output again. Only apple survived.

No one has paid for New Dish yet. I would rather say that honestly than invent traction for a submission. But the product already received its first useful requests: support a single person, show verified nutrition per serving, let users set their neighbors' names, and never fabricate a recipe. Each request changed the app.

The next real test is putting it in front of roommates or a shared household for a week and measuring whether it helps them rescue at least one dinner they would otherwise have abandoned.

---

Open-source building blocks

• Transformers.js for in-browser model inference
• SmolLM2-360M-Instruct
• Qwen2.5-0.5B-Instruct ONNX
• DETR ResNet-50 for Transformers.js
• Tesseract.js
• Kokoro 82M ONNX
• UniTools Recipes v1, CC BY-SA 4.0

Thanks for reading. If your fridge has ever contained plenty of ingredients but somehow no dinner, New Dish is for you.