This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
https://voice-order-taker-jvh7zuvem9z6qbfuqfcvpm.streamlit.app/
What I Built
Voice Order Taker is a local app that turns WhatsApp voice notes into a clean order sheet.
In Pakistan, a lot of home businesses run on voice notes. A baker, a tiffin cook or a small clothing seller gets messages like "do kilo chocolate cake aur teen dozen samosay kal shaam tak, house 12 Jinnah Road", in Urdu, Roman Urdu, Pashto or a mix with English. The seller replays each note, then writes the order down by hand. With 20 or more orders a week, that is hours of work, and one misheard quantity means a wrong delivery.
With my app, the seller uploads the voice notes and the app:
- Transcribes each note with Whisper.
- Extracts customer, item, quantity, unit, delivery date, address and notes with a Gemma model.
- Shows an editable order table, a CSV download, and a daily prep list that adds up how many of each item are due on each date.
The person who would pay for this is a home seller handling a steady stream of orders. They don't need a big business tool. They need something that saves an hour a day and doesn't send their customers' addresses to a stranger's server.
Demo
Code
Voice Order Taker
https://voice-order-taker-jvh7zuvem9z6qbfuqfcvpm.streamlit.app/
Turn WhatsApp voice notes in Urdu, Roman Urdu, Pashto or English into a clean order sheet and a daily prep list. Everything runs on your own computer. No cloud, no API key, no per-order cost.
Built for the Hacktoberfest Open-Source AI Challenge (Week 2: Side Hustle).
The problem
Home bakers, tiffin services and small clothing sellers take most of their orders as WhatsApp voice notes. Someone says "do kilo chocolate cake aur teen dozen samosay kal shaam tak, house 12 Jinnah Road", and the seller has to replay it and write it down by hand. With 20 or more orders a week, this takes hours and mistakes happen.
What it does
- You upload one or more voice notes (
.ogg,.opus,.m4a,.mp3,.wav). - Whisper transcribes each note locally.
- A Gemma model running in Ollama extracts structured orders: customer, item, quantity…
How I Built It
Everything runs on a normal laptop with no cloud service.
-
faster-whisper (open-source Whisper) transcribes audio on the CPU. It reads WhatsApp
.opus,.oggand.m4afiles directly. - Gemma, running locally through Ollama, reads the transcript and returns structured orders.
- Streamlit is the interface, with a custom dark navy and teal theme and a CSS-only 3D robot mascot (no images or external libraries, so it works offline too).
- pandas handles the editable table, the CSV export and the daily prep list.
The most important part was making a small model reliable. A free-form prompt gave me bad results, so I forced the output into a fixed JSON schema and added a worked example inside the prompt:
class OrderLine(BaseModel):
customer: str = ""
item: str = ""
quantity: float = 1
unit: str = ""
delivery_date: str = ""
address: str = ""
notes: str = ""
class OrderList(BaseModel):
orders: list[OrderLine]
r = ollama.chat(
model=model,
format=OrderList.model_json_schema(),
options={"temperature": 0, "num_ctx": 2048},
messages=[{"role": "system", "content": system},
{"role": "user", "content": transcript}],
)
The prompt also converts words like "kal" (tomorrow) and "parson" (day after tomorrow) into real dates, and I give Whisper a short list of dish names as an initial_prompt so it recognises local food words.
What went wrong along the way. I built this on a laptop with a nearly full C: drive and limited RAM, which turned out to be good testing for a "runs on cheap hardware" project:
- The first model download failed because the disk was full, so I moved the model and cache folders to another drive.
- A 6.6 GB Gemma model ran out of memory while Whisper was also loaded, so I switched to a smaller Gemma and reduced the context size.
- A newer audio library version broke Whisper's decoder, so I pinned
av<17. - My first extraction attempt with the smallest model returned a quantity with no item name. The schema and the worked example fixed most of that.
[Add your real numbers here, for example: "I tested X voice notes. N came out fully correct, M needed one edit, and K failed. Average time per note was about Y seconds on CPU."]
Why Does Open Innovation Matter?
For this kind of customer, open models are not a nice extra. They are what makes the product possible.
- Privacy. A voice note contains a customer's name, address and what they bought. With a closed API, all of that goes to a third-party server. Here it never leaves the seller's laptop.
- Cost. Transcription and extraction cost nothing per order. A home baker can't justify per-minute and per-token fees, and a small flat monthly price still leaves a margin.
- Offline use. Power and internet outages are common where I live. The app keeps working because nothing depends on a connection after the one-time model download.
- Control. I could change the prompt, give Whisper local dish names, shrink the context, and swap one Gemma size for another when my laptop couldn't handle the bigger one. With a closed API, I would have been stuck with whatever the provider supports and charges.
The trade-off is honest too: a small local model is less accurate than a large cloud model on messy mixed-language speech, so the app always shows an editable table and the seller confirms the result before acting on it.
My Agent Session
[Add your DevRelay or Entire session link here, or delete this section.]
My First User
[Who they were (for example, a home baker, a relative who runs a food business), what voice notes they tried, what they said, and whether they would pay. Even one real quote helps. If you did not get a user, delete this section.]
Prize Categories
Best Use of Gemma: the extraction step runs on Gemma locally through Ollama, so all order data stays on the user's machine.
Top comments (1)
Underrated point. Most people skip straight to the flashy part.