CrumbCraft AI: An Offline, Voice-First Agent for My Baker Friend Niki 🎂🤖

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built CrumbCraft AI, a hands-free, voice-activated baking and cake decorating assistant designed specifically for my friend Niki.

Niki runs a boutique home bakery specializing in custom, multi-tiered cakes. Her process is creative and precise, but working in her basement prep kitchen presents unique real-world challenges:

• Flour-Covered Hands: When she's piping Swiss Meringue buttercream or sculpting fondant, touching a tablet or phone to look up recipe scalings or substitution math washes 10+ minutes down the drain in hand-sanitizing.

• Strict Allergen Safety: A math miscalculation when scaling a gluten-free or vegan recipe for a sensitive client can ruin an event—or pose real health risks.

• Kitchen Dead Zones: Her basement prep kitchen has unreliable Wi-Fi. Cloud-only voice assistants fail right when she needs an immediate unit conversion mid-bake.

CrumbCraft AI solves this by running as a 100% offline, hands-free voice loop on her kitchen laptop. Niki simply speaks her question out loud, and CrumbCraft processes her speech, calculates exact recipe scalings, verifies allergen rules, and speaks the answer back—all with zero internet connection.

Demo
Here is a recording of CrumbCraft AI running locally with zero Wi-Fi connection, listening through the microphone and speaking answers back via native audio:
{https://drive.google.com/file/d/1QmiBeJ2JV8rOoBrSpMsIlhqiAUh3gDhj/view?usp=share_link}

Code
You can check out the open-source repository and build setup below:
{https://github.com/priyeshdabre07/crumbcraft-ai}

Core Tech Stack

• Agent Framework: Mastra (@mastra/core)
• Local LLM Engine: Ollama running Google's open-weight gemma4:e4b
• Local Speech Recognition (STT): OpenAI Whisper (small.en via whisper-node / whisper.cpp)
• Local Audio Recording & TTS: SoX (rec) & macOS Native Speech Engine (say)
• Local Memory Store: Embedded JSON Recipe Database

How I Built It

CrumbCraft AI is designed around a fully local, privacy-first pipeline where open-source components handle every step of the voice agent loop:
+-----------------------------------------------------------------------+
| KITCHEN WORKSPACE |
| |
| 🎙️ Microphones Capture Speech ---> 🔊 Speakers Output Audio |
+----------------------------------+------------------------------------+
|
v
+-----------------------------------------------------------------------+
| CRUMBCRAFT LOCAL PIPELINE |
| |
| 1. Speech-to-Text: OpenAI Whisper (small.en running locally) |
| │ |
| v |
| 2. Agent Orchestration: Mastra Framework |
| ├── LLM Reasoning: Google Gemma 4 (gemma4:e4b via Ollama) |
| ├── Deterministic Tools: frostingScalerTool & allergenCheckerTool |
| └── Embedded Memory: local_recipes.json Database |
| │ |
| v |
| 3. Text-to-Speech: Native macOS 'say' Engine / Local Audio Synthesis |
+-----------------------------------------------------------------------+

1. Deterministic Baking Tools with Mastra:

We defined specialized tools within Mastra so the agent doesn't hallucinate baking math:
TypeScript

import { createTool } from '@mastra/core';
import { z } from 'zod';
import { searchLocalRecipes } from '../db/localDb';

export const frostingScalerTool = createTool({
  id: 'frosting-scaler',
  description: 'Calculates total frosting volume and key ingredient weights based on cake tier diameters.',
  inputSchema: z.object({
    tierSizesInInches: z.array(z.number()).describe('Array of tier diameters in inches, e.g. [6, 8, 10]'),
    frostingType: z.string().describe('Type of frosting, e.g., Swiss Meringue Buttercream'),
  }),
  outputSchema: z.object({
    totalCupsNeeded: z.number(),
    butterGrams: z.number(),
    sugarGrams: z.number(),
  }),
  execute: async ({ context }) => {
    const { tierSizesInInches } = context;
    const totalSurfaceArea = tierSizesInInches.reduce((acc, d) => acc + Math.PI * Math.pow(d / 2, 2), 0);
    const totalCupsNeeded = Math.round((totalSurfaceArea * 0.15) * 10) / 10;

 return {
   totalCupsNeeded,
   butterGrams: Math.round(totalCupsNeeded * 115),
   sugarGrams: Math.round(totalCupsNeeded * 120),
    };
  },
});

2. Local Open Weights with Gemma 4:

We targeted gemma4:e4b running locally via Ollama. It strikes the perfect balance for Apple Silicon (M4 with 16GB Unified Memory): delivering native function-calling capabilities, zero network latency, and low memory usage (~6.6 GB RAM) so the machine stays cool and responsive.

3. Local Voice Pipeline:

Input: Audio is captured directly from the microphone using sox and transcribed locally via Whisper (small.en).
Output: Speech responses are generated instantly through macOS's native TTS engine (say), eliminating cloud per-character API fees.

Why Does Open Innovation Matter?

Open-source AI was not just a convenience for this project—it was an absolute requirement:

• Kitchen Reliability (Zero Internet Dependency): Niki's basement prep room is an internet dead zone. Cloud AI models (like ChatGPT or proprietary voice APIs) freeze or fail completely when connection drops. Open-weight models like Gemma 4 run 100% locally on device, guaranteeing that her voice assistant works every single time.
• Data Privacy for Client Relationships: Niki stores client allergy histories, family dietary preferences, and proprietary recipe ratios. Keeping execution entirely local ensures zero client data or custom recipes are uploaded to external training servers.
• Zero Operating Cost: Running open-source models via Ollama and local speech engines means CrumbCraft AI costs $0.00/month to run. An artisan baker shouldn't have to worry about token billing while mixing batter.
• Custom Tooling Control: Proprietary black-box endpoints can hallucinate substitution advice. Open frameworks like Mastra allow us to inspect, trace, and strictly enforce deterministic rule checking for allergen safety.

Prize Categories
Best Use of Gemma: Used gemma4:e4b locally via Ollama as the core reasoning engine for unit conversions, allergen substitutions, and tool calls.
Best Use of Mastra: Built the entire agent workflow, tool execution layer, and structured response pipeline using the Mastra TypeScript framework.