This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
YapIt is a private voice notebook. You tap the mic, talk the way you normally would, and it turns what you said into a structured note you can actually use later. Nothing leaves your computer.
"Ravi owes me 1.5 lakh for the laptop, he'll pay next Friday."
→ Payment · Ravi · owes you · ₹1,50,000 · due Fri 30 Oct"Team review with Sunita and Tarun next Wednesday at 2 pm."
→ Event · Team review · Wed 28 Oct · 2 PM · Sunita, Tarun
It sorts every note into one of five types (payment, promise, event, info, general), fills in the details, double-checks them, and shows you a draft to confirm. It also:
- turns away things that aren't notes (a mic test, "play some music");
- asks you to re-record unclear audio;
- warns you about notes that don't make sense, like a meeting at 25 PM.
The problem it solves: quick thoughts get lost, or end up as messy text you never read again. Apps that understand your notes usually send everything to a cloud server, and many notes are private: money, plans, other people's details.
Who would pay: anyone who captures things on the move.
- Students: deadlines, and who borrowed their notes.
- Freelancers: client promises, and who owes them.
- Small business owners: orders, payments, appointments.
- Everyone: splitting bills, birthdays, reminders.
YapIt runs on your own machine with zero per-note cost, so it can be sold as a cheap one-time purchase. There's no server bill eating the margin.
Demo
Code
gouravk001
/
YapIt
AI Note taking application
YapIt
A private voice notebook: say it, and YapIt turns it into an organised note, entirely on your own computer.
Speak naturally and YapIt figures out what kind of note it is and fills in the details:
| You say | YapIt saves |
|---|---|
| "Ravi owes me 1.5 lakh for the laptop, he'll pay next Friday" | Payment: Ravi · owes you · ₹1,50,000 · due Fri 30 Oct |
| "Physics assignment is due on the 25th, I told Meera I'd share my notes" | Promise: Meera · share notes · due Sun 25 Oct |
| "Team review with Sunita and Tarun next Wednesday at 2 pm" | Event: Team review · Wed 28 Oct · 2 PM · Sunita, Tarun |
| "The office Wi-Fi password is brew2026" | Info: Office Wi-Fi password · brew2026 |
You check the draft and save it. Speech recognition and the language model run locally on open-weight models, so nothing leaves your machine and there's…
Setup is in the README. With Ollama installed, ./dev.sh starts everything.
How I Built It
Everything runs locally on my laptop: an RTX 4060 Laptop GPU with 8 GB of VRAM.
- Gemma 3 4B (open weights, run through Ollama) understands the notes.
-
faster-whisper (
small.en) turns speech into text. - FastAPI + SQLite with full-text search store the notes; React + Vite + Tailwind for the app.
The model size wasn't arbitrary: an 8 GB GPU has to hold Whisper and the language model at the same time, so about 4B parameters is the sweet spot. Bigger models get much slower.
A single "turn this into JSON" prompt on a 4B model is fragile, so YapIt runs a short pipeline where each step does one job, and every LLM reply is forced into a JSON schema:
- Transcribe with Whisper, with voice activity detection turned on.
- Classify: is this a note at all, and which type?
- Extract the fields. The schema comes from a YAML type definition, so a new note type needs no code.
- Resolve dates, money and names in code, not in the model. "Next Friday", "end of the month" and "1.5 lakh" are computed exactly.
- Refine: a second pass re-reads the transcript, fixes mistakes, and writes a one-line summary.
- Plausibility check: cheap rules first, then one short question to the model.
The UI streams each stage live (Understanding → Filling in details → Tidying up → Double-checking), so the ~2.7 s wait never feels stuck.
What broke, and how I fixed it
Some of the Problems I encountered:
Whisper invented text from silence. When a recording had no real speech, just silence, a beep or background noise, Whisper didn't return nothing. It returned confident nonsense. Turning on voice activity detection, rejecting known junk phrases and low-confidence transcripts, and refusing transcripts that are too short fixed it. YapIt now says "Let's try that again" instead of saving garbage.
Gemma kept getting who owes whom backwards. "I have to give Ram 200 rupees" was saved as Ram owes me. Two changes helped most:
- the model first restates the note in plain words ("so I will give Ram ₹200");
- it chooses between plain sentences, "I will give the money" and "I will receive the money", instead of codes like
i_owe_them.
On my hardest phrasings, that went from 4/10 to 8/10 correct.
Small models copy your examples. After I added example summaries to the prompt, "Team meeting tomorrow" came back as "Team lunch with Sara, Thursday", lifted straight from an example. Examples are now explicitly marked "style only".
"30th of February" became 30 October. My date parser ignored month names entirely. It now understands them, and impossible dates are left blank and flagged rather than guessed.
Warning about unrealistic notes took three attempts:
- asking inside an existing step: the model never raised anything;
- a multiple-choice "what's wrong?" prompt: 31 false alarms on 40 ordinary notes;
- a second "verify" pass: it rejected every warning, real ones included.
What worked was rules first (dates decades away, absurd amounts, impossible dates), then one focused yes/no question only when the rules find nothing. Result: all 6 unreasonable test notes caught, and 1 false alarm in 39 ordinary ones.
Tried and postponed: auto-correcting mishearings ("welding" → "wedding"). The model fixed about half, but it also "fixed" "buy welding rods" into "wedding rods". A wrong correction is worse than none.
Picking the model: a benchmark, not vibes
Because the models are open, I could test them properly. I built a benchmark that runs 119 held-out test notes through YapIt's real pipeline: tricky payment directions, Indian amounts, relative dates, non-notes and deliberately unreasonable notes. Adding a model to compare is one line in a YAML file: it downloads the model if needed, tests it, and updates the comparison.
Results on the same laptop (8 GB VRAM):
| Gemma 3 4B | Qwen 3.5 4B | Llama 3.2 3B | |
|---|---|---|---|
| Note type correct | 95.8% | 97.5% | 85.7% |
| Field accuracy | 94.6% | 96.2% | 81.5% |
| Who owes whom | 81.0% | 95.2% | 59.5% |
| Amounts | 97.5% | 100% | 70.0% |
| Dates | 100% | 90.9% | 95.5% |
| Non-notes rejected | 100% | 93.8% | 68.8% |
| Time per note | 2.67 s | 3.39 s | 1.57 s |
| GPU memory | 3.9 GB | 3.9 GB | 2.6 GB |
What the numbers say:
- Qwen 3.5 is the most accurate, but it "helps" in ways that hurt. It sometimes replaces your date words with its own wrong guess ("the day after tomorrow" → "Tuesday"), and it's slower.
- Gemma 3 is perfect on dates and on rejecting non-notes, and faster. It's the default for now.
- Llama 3.2 is the fastest but too weak on money and directions.
- Small differences are noise: the same model moved 1–2 points between runs. The direction gap (95% vs 81%) is real.
Honest limitations
- English only, for now. That was mainly a time constraint. A Hinglish version was the original idea, and it's the next big step.
- Mistakes still happen (81% on who-owes-whom with Gemma), which is why nothing is saved until you confirm the draft.
- The test set is synthetic, hand-written to cover hard cases, not real users' recordings yet.
- A GPU is recommended; it works on CPU, just slower.
Why Does Open Innovation Matter?
For YapIt, open models aren't a nice-to-have. They're what make the product possible:
- Privacy: notes are personal, and payment notes involve other people's money. With local open models, nothing ever leaves the laptop. A closed API would mean sending every note to someone else's server.
- A side hustle that actually makes money: zero per-note cost means a one-time price works. With a paid API, every note a user records costs me money, forever.
- Works offline: after the first start, no internet is needed at all.
- I could measure and choose. Swapping Gemma for Qwen or Llama is one config line, and my benchmark shows exactly what each choice costs in accuracy, speed and memory. With a closed API you get whatever version they ship.
- I can fine-tune it next. The weakest spot, who owes whom, is exactly the narrow kind of task where a small fine-tuned open model can beat a general one.
My First User
My first user was my brother. He used YapIt every day for a while, to take notes in his classes and to capture day-to-day things: plans, events, and promises he'd made to people.
What went wrong. He found real problems, and they shaped the app more than anything else. His most memorable complaint:
"It listens to whatever I say and doesn't care whether it's a note or not."
He was right. Early YapIt turned everything into a note: a half-finished sentence, a random question, someone talking in the background. That feedback led directly to two features:
- The "is this actually a note?" check: if what you said isn't a note (a mic test, a question meant for an assistant, background chatter), YapIt says so instead of saving it.
- The sense check: notes that can't be right, like a date that doesn't exist or a meeting at 25 PM, now come with a warning before you save them.
What worked for him. At the end of each day he could quickly see what was coming up, his upcoming events and the promises he'd made, without scrolling through raw text.
What he asked for next: reminders, meaning an alarm or notification when an event or promise is due. I couldn't build it before the deadline, so it's the first thing on my list.
Prize Categories
- Best Use of Gemma: Gemma 3 4B runs locally through Ollama as YapIt's brain, and was benchmarked against Qwen 3.5 and Llama 3.2.
Top comments (1)
Really impressive build, Gourav! I love that everything runs locally to protect privacy. The pipeline approach you used to keep the 4B model accurate instead of relying on a single prompt is super smart.