I built Simmer: an open-source AI agnet that turns cooking reels into recipes you can actually cook and ask questions about that recipe in real time

Simmer is an open-source AI recipe assistant. Share a cooking Reel, Short, or recipe link — or paste a recipe you already have — and it turns the mess into a clean recipe with ingredients, amounts, times, and ordered steps. Then it reads the steps to you and answers questions while your hands are busy.

The 20-second version

• Problem: I save cooking reels every week and cook almost none of them. The recipe is split across a caption, a 45-second video, and a comment thread.
• Product: Simmer turns that mess into a structured recipe, then becomes a hands-free cooking assistant.
• Core AI: Gemma 4, open weights, for extraction, chat, and translation.
• Inputs: Instagram, YouTube Shorts, recipe links, or pasted text.
• Output: ingredients, amounts, times, tags, ordered steps, 10 languages, voice mode, installable PWA.
• Traction: one paid early user. Small amount, real signal.

What I Built

Every week I save cooking reels I never cook.

The recipe is usually not in one place. The ingredients are in the caption. The method is in the audio. The quantities are in a comment. The video is 45 seconds of fast cuts and no measurements. By the time I’m hungry, rewriting all of that into something usable feels like homework.

Simmer is the step between saving and cooking.

You add a public link or paste text. Simmer reads the page, the caption, the metadata, and — if the text is useless — the audio. Then Gemma 4 turns that into a structured recipe: ingredients with amounts, ordered steps, times, servings, tags, and a title. Your original text is kept alongside it, so you can always check what the source actually said.

Then the app changes jobs. It becomes a cook-buddy:

• It reads the steps out loud.
• You can ask “how much salt?” without touching the screen.
• It switches between 10 languages.
• It installs as a PWA and opens from the share sheet.

The people I built this for are not professional chefs. They are people who save more recipes than they cook, people cooking in a second language, and anyone who has stood in a kitchen with wet hands trying to scroll back to step four.

The bet is simple: the input layer for recipes is broken. People already pay for recipe managers and meal planners. Simmer fixes the part before that — getting the recipe out of the feed and into a format you can cook from.

Demo

Live demo: simmer-live-demo-link

Video demo: Watch the 45-second demo

Images demo:

Code

Repo: github.com/Ayush-Barai/simmer — MIT licensed.

Ayush-Barai / simmer

Simmer

Share a cooking reel → get a clean recipe → cook hands-free with an open-AI buddy.

Built for the Hacktoberfest Open-Source AI Challenge (Week 2: Side Hustle).

What it does

• Link → recipe: share a Reel/Short/TikTok or paste a link. Simmer reads the description first, then transcribes the audio if the text has no recipe; Gemma extracts ingredients, amounts, timings and ordered steps. You can also paste a recipe you already have — Simmer keeps your original wording and saves a clean, structured version.
• Hands-free cooking: a live voice loop (ElevenLabs Conversational AI) with a browser speech fallback, so you can ask questions mid-step without touching the phone.
• 10 languages: read, chat and translate in English, Spanish, French, German, Italian Portuguese, Hindi, Japanese, Chinese and Arabic. Translations are saved with the recipe so switching language is instant.
• Private cookbook: Clerk auth, per-user recipes, thumbnails and saved chat history.
• …

View on GitHub

git clone https://github.com/Ayush-Barai/simmer.git
cd simmer
npm install
cp .env.example .env
npm run dev

Add your keys to .env — the README covers Gemma, MongoDB Atlas, Clerk, ElevenLabs, and Sentry.

The shape of the app:

link / pasted text
        │
        ▼
  page text + metadata
        │
        ├─ social caption / Instagram embed
        ├─ yt-dlp metadata + subtitles
        └─ ElevenLabs Scribe (audio fallback)
        │
        ▼
  Gemma 4 → structured recipe JSON
        │
        ▼
  MongoDB Atlas → PWA → cook mode / voice

How I Built It

1. Ingestion is a cascade, not a scraper

The hardest part is not the AI. It is getting clean text out of a link that was never designed to give it to you.

server/ingest.js tries the sources in order: page text and metadata, the social caption, the Instagram embed page, then yt-dlp for metadata and subtitles. If the description is empty or useless, server/transcript.js downloads the audio and server/stt.js sends it to ElevenLabs Scribe.

Each source fails differently. Instagram blocks one path but exposes another. A YouTube Short may have no captions. A recipe blog buries the ingredients under three popups. The cascade is the product.

2. Gemma 4 turns text into structure

server/gemma.js asks Gemma for strict JSON: title, ingredients, steps, times, servings, tags. It uses gemma-4-26b-a4b-it and gemma-4-31b-it, with a fallback list and a JSON repair path.

One bug cost me an afternoon: Gemma’s thinking output was being mixed into the response, which made valid recipes look empty. Filtering thought parts in both streaming and non-streaming responses fixed it. The model IDs are config, not hardcoded, so swapping models is an env-var change.

3. Translations are precomputed, not generated on click

Simmer supports 10 languages: English, Spanish, French, German, Italian, Portuguese, Hindi, Japanese, Chinese, and Arabic.

The first version translated on demand. It worked, but it felt slow and fragile: every language switch was another model call. Now server/localize.js prefetches the other nine languages in the background after a recipe is saved. Translations are stored in MongoDB with the recipe.

The result: switching languages after the first pass returns stored data in milliseconds. In my test, reopening a recipe and changing language made zero translation requests. The model still does the work — just not while you are waiting.

4. Voice and PWA

Cook mode uses ElevenLabs Conversational AI for the live agent, with a browser speech fallback. Sessions are scoped to a recipe and language. The same Scribe integration that rescues silent captions also makes the app work for people who would rather talk than type.

The frontend is vanilla JavaScript, CSS variables, and a service worker. No framework, no build step. The PWA has a share target, an offline shell, light and dark themes, and maskable icons.

5. The boring parts that make it real

• Clerk for auth, with every query owner-scoped.
• MongoDB Atlas for recipes, stored translations, and agent sessions.
• Sentry spans around ingest, extraction, translation, chat, and agent calls.
• Docker with Python, ffmpeg, and yt-dlp, plus a render.yaml blueprint.
• Tests for the parser, ownership checks, and API behavior.

What broke, and what I changed

• Instagram is hostile to scrapers. I stopped pretending one path would work and built the cascade.
• Captions lie. Some have no amounts; some have no method. Scribe handles the audio fallback.
• Latency is a feature. I removed fake progress and made the status text honest.
• Mobile needed to be first-class. Share target, large tap targets, voice, install prompt.
• Translations should not be a loading spinner. Precompute them, then get out of the way.

Why Does Open Innovation Matter?

Open innovation matters here because the economics of a small product like this are brutal if every useful action is a markup on a closed API.

Gemma is open-weight. That means:

• Model choice is a config line, not a rewrite. If a smaller model gets good enough, I can swap it and cut costs.
• A niche product can exist. A tool for home cooks in 10 languages does not need venture scale to be useful. Open weights make the unit economics possible.
• Privacy and self-hosting are real options. The Docker image can run with your own keys and your own database. Your recipes are not locked into my account.
• The long tail gets served. Regional dishes, regional languages, and weird family recipes are exactly the cases closed models ignore.
• The glue is open source too. yt-dlp, ffmpeg, Express, and the PWA platform do a lot of the unglamorous work.

I still use hosted services — ElevenLabs, Atlas, Clerk, Sentry — and I am honest about that. But the intelligence at the center is open-weight and replaceable. That is the difference between building a product and renting one.

My First User

I gave an early build to a close friend who cooks at home almost every day. I’ll keep his name private. He is exactly the person I built this for: saves reels, cooks from his phone, hates reading captions.

He used it on the links he already had saved. Then he asked to keep using it.

Would he pay? He already did. That is the only validation I trust right now.

I am not calling Simmer a startup. It is one paid pilot, a lot of late nights, and a clear next step. That is enough to keep going.

FAQ

What is Simmer?

An open-source AI recipe assistant that turns cooking links, reels, and pasted text into structured recipes, then helps you cook them hands-free.

Does Simmer work with Instagram Reels?

Yes, for public links. Instagram is the messiest source, so Simmer has caption, embed, metadata, and audio fallbacks rather than a single scraper.

Does it work with YouTube Shorts and TikTok?

YouTube is a tested path. TikTok depends on the region and whether yt-dlp can reach it, so it is not the primary path.

How long does it take?

About 1–3 minutes for a public link. Audio transcription can take longer. Translations are prepared in the background, so switching languages is instant after the first pass.

Is it really open source?

Yes. MIT licensed. The repo includes the server, PWA, Dockerfile, and setup docs.

Can I self-host it?

Yes. Run the Docker image with your own Gemma, MongoDB, Clerk, and ElevenLabs keys. The README has the details.

Why 10 languages?

Because recipe videos travel across languages, but the useful instructions usually do not. Translations are stored per recipe, so the app stays fast.

Prize Categories

• Best Use of Gemma — Gemma 4 is the extraction, chat, and translation engine. It is open-weight and swappable with an environment variable.
• Best Use of Render — Simmer ships as a Docker service with a render.yaml blueprint.
• Best Use of ElevenLabs — Scribe for audio fallback, and the conversational agent for hands-free cook mode.
• Best Use of MongoDB Atlas — recipes, owner-scoped data, stored translations, and agent sessions.
• Best Use of Sentry Agent Tracing — spans around ingest, extraction, translation, chat, and agent calls.

Final thought

Simmer is the first thing I have built that someone paid for before it was finished. That is the side-hustle test I care about.

If you cook from reels, try it on the link that usually defeats you. If it breaks, send it to me. That is the next bug I want.