Tired of Writing Notes in Class?I Built an App for That.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

I have been lowkey falling behind my lectures. I start each semester with a burning motivation to pay attention in class. BUT writing notes is just too damn tiresome, especially when I also need to pay attention to what my professors are saying. Because let's be honest, once they go off on a tangent, I have no clue what the hell they just said.

AND for some reason, none of them ever record their lectures despite it being compulsory for them 🫩.

HOWEVER, for International Sloth Day, I decided to just make an app that takes my phone recordings and makes notes for me, so I don't have to do the hard work 😄.

And that's how LazyNotes came about.

The idea is simple: record a lecture using your phone, upload it, and get organized, topic-wise study notes with headings, bullet points, and explanations.

You can also provide the topics covered in class and an optional syllabus to help the model understand the context.

The goal is to spend less time frantically copying everything down and more time actually understanding what's being taught.

Demo

And here's what the app actually looks like!

The demo shows me uploading a real lecture recording, providing the topics covered, and getting organized study notes.

Note: The processing portion has been cut for the demo. Actual generation time depends on the recording length and hardware.

Code

LazyNotes is built around open-source technologies and open-weight AI models.

GitHub: https://github.com/Zemksh/lazynotes

The project includes the mobile application, backend processing pipeline, transcription, note generation, and validation and recovery systems.

How I Built It

I went with a mobile app and a Python backend.

The workflow is straightforward:

Record → Upload → Transcribe → Generate Notes → Study

Here's the stack:

Component	Technology
Mobile application	React Native, Expo, TypeScript
Backend	Python, FastAPI
Background processing	Celery, Redis
Transcription	faster-whisper
Speech recognition model	Distil-Whisper Small English
Note generation	Gemma 3:4B
Local model inference	Ollama
Validation	Pydantic
Transcription diagnostics	Python-based heuristics
Speech activity detection	Silero VAD

But What About Noisy Classrooms?

One problem I had to consider was that classroom recordings aren't exactly studio-quality audio.

There's always something going on. Students talking in the background, chairs scraping against the floor, awkward silences, and sometimes a professor who seems determined to speak from the opposite end of the room.

So I didn't want the transcription pipeline to blindly treat every second of audio as speech.

I'm using faster-whisper with Silero Voice Activity Detection (VAD).

Silero VAD helps identify which parts of the recording contain speech and which parts are likely silence or non-speech. This allows Whisper to focus its transcription on detected speech regions instead of processing the entire recording indiscriminately.

However, VAD isn't the same as noise cancellation. If someone is talking loudly in the background, or the professor's voice is drowned out, transcription can still make mistakes.

That's where the second part of my approach comes in.

Whisper provides numerical diagnostics such as average token log probability and compression ratio. I use these with lightweight Python heuristics to flag potentially unreliable transcription segments.

For example, unusually low transcription scores or suspiciously repetitive text can indicate that a segment might need attention.

It's not a perfect system, and these metrics don't directly tell us whether every word is correct. But they provide a useful way to identify suspicious output without making another expensive language-model call.

I also wanted this reliability analysis to stay separate from the generated notes. Nobody wants their revision material cluttered with confidence percentages and technical warnings.

The Part That Took Longer Than Expected

Getting a model to generate notes wasn't particularly difficult.

Getting it to generate useful notes, reliably, in a reasonable amount of time was a different story.

My initial approach used Gemma 3:4B to generate notes from chunks of the transcript and then repeatedly combine those chunks using more model calls.

It worked in theory. In practice, it was painfully slow on my CPU.

I also ran into problems with incomplete JSON responses, validation failures, and repeated explanations across transcript chunks.

So I started simplifying the pipeline.

Instead of asking Gemma to repeatedly merge its own output, I moved consolidation into Python. I added chunk-level checkpoints so completed work could be reused after failures, and tightened validation to catch malformed output.

I also realized that the notes themselves needed work.

The initial results were too paragraph-heavy and repetitive. They read more like essays than something I'd actually use to revise before an exam.

I changed the generation format to focus on headings, concise bullet points, and important explanations.

That was a useful lesson: generating more text doesn't necessarily mean generating better notes.

Keeping the Notes Faithful

I also wanted to avoid blindly trusting everything the models produced.

The transcript is treated as the primary source of what was actually taught. The topics and syllabus are only there to help with terminology and context, not to introduce information that wasn't covered.

The generated notes are checked against structural requirements and valid transcript references using Pydantic and Python.

These checks help catch malformed outputs and invalid references, but they don't guarantee that every explanation is factually correct or that every important point was captured.

That's still an area I want to improve.

Why Does Open Innovation Matter?

The biggest reason I chose open-weight models was control.

I wanted to understand what was happening at each stage of the pipeline rather than sending a recording to an API and treating the response as a black box.

With Whisper and Gemma, I can experiment with model configurations, inspect performance, change the generation process, and run inference locally.

That also means the core AI processing can run without sending lecture recordings to a commercial AI inference API.

Of course, there's a trade-off.

Running Gemma 3:4B on my laptop's CPU is considerably slower than using a powerful cloud GPU. I've spent a surprising amount of time watching token-generation logs.

But being able to inspect and optimize the system has taught me a lot about inference performance, structured outputs, and building around the limitations of smaller models.

For me, that's one of the most valuable parts of building with open AI technologies.

The Lazy Payoff

I tested LazyNotes using actual classroom recordings, including lectures of approximately 10 and 50 minutes.

One of the biggest improvements came from removing unnecessary model calls and making the generated notes more useful for studying.

In a recent 10-minute lecture test, transcription took approximately one minute, while note generation took around four to five minutes in a successful run.

That's still slower than I'd like, but it's progress from the much longer runs and failures I encountered earlier.

More importantly, the output is moving closer to what I originally wanted: notes I can open later and actually revise from.

There's still plenty to improve, especially generation speed, formatting, and reliability.

But LazyNotes has already taught me something useful about building AI applications:

The hard part isn't always getting AI to do something. It's getting it to do exactly what you need, without making everything unnecessarily complicated.

And if that means I get to spend a little less time writing notes in class, I'll happily take it.

My Agent Session

I used a Codex along with Opencode to build and debug LazyNotes,
particularly the Gemma inference pipeline and
checkpoint recovery system.

[View my development session]

LazyNotes: optimizing Gemma notes, debugging JSON validation, and preserving checkpoints

You

I want to optimize Gemma 3:4B note generation by eliminating unnecessary work, without sacrificing the quality or completeness of the notes.

Agent

The tests pass so far. The updated audit estimates about 18% less prompt text on the 240-segment synthetic case, with the same four calls, and about 7% less output JSON from removing the title alone.

Very short fixtures show essentially no prompt savings: the overlap-ownership instructions offset the removed metadata. These are character-based estimates, not measured Gemma token counts or speedups.

You

I approve implementing the first round of generation optimizations.

Agent

Implemented the first round without changing inference settings or the frontend.

• Gemma receives IDs/text only and returns sections without a title.
• Python restores the supplied title.
• Context filtering retains matching terminology and definitions, excluding zero-match passages.
• Overlap is context-only; validation rejects sections citing no new segment.
• Checkpoints include the wire payload, ownership, prompt, and generation schema.

You

I want to change the generated notes from paragraph-heavy explanations to structured, student-style bullet notes.

Agent

Implemented bullet-first study notes.

• Schema: bullets remain strings, with optional {"text": "...", "children": ["..."]} groups for one nested level. Older paragraphs remain supported.
• Generation: bullets are the default; explanations, examples, qualifications, and equations remain required. Removed bullet-count and per-bullet length caps.
• Consolidation: identical bullets merge under matching topics/subtopics, preserving unique children and source references. Paraphrases are not merged.
• UI: indented nested bullets, readable spacing, and parent/child editing.
• Recovery: checkpointing and internal reliability checks remain; prompt/schema changes invalidate old drafts.

Show more

You

the notes generation did not work . check the logs and fix the errors

Agent

The logs show Gemma’s first chunk failed schema validation twice—not a transcription timeout. Unfortunately, the old logs didn’t retain the exact invalid field.

• Log failing fields safely, without lecture content.
• Give the existing retry specific validation feedback.
• Invalidate affected checkpoint fingerprints.

You

error again . it kept preparing notes for like 15 mins . check logs and fix

Agent

Found a schema mismatch: Gemma was allowed to produce empty sections or blank bullets, which Python then rejected. The expensive retry repeated that failure.

Fixed:
- Generation now requires nonempty bullets.
- Logs distinguish missing content from blank bullets.
- Older saved notes remain compatible; checkpoints invalidate correctly.

64 tests passed, 1 skipped; lint passed. No inference was run, so real-model verification remains pending.

You

I'm retrying the existing lecture after deploying the blank-bullet/schema fix, but generation has already taken 5–6 minutes.

Agent

1. 2 of 2 chunks completed; none remain.

2. The first request passed validation. I independently checked both saved drafts against Pydantic, their source segments, and current-chunk ownership.

3. The regex warning is real. Ollama ignored the unsupported pattern, so whitespace-only strings remained possible despite minLength: 1. Python still rejects them—but that did not happen in this run.

4. The time was spent generating two chunks. Your reported first request took roughly three minutes; the second took another few minutes. There isn’t a comparable successful paragraph-generation timing available to establish whether bullets themselves slowed it down. Earlier failures stopped at chunk 1.

Smallest proposed fix: anchor the pattern as ^[\s\S]*\S[\s\S]*$. Test blank, whitespace-only, multiline, and nested bullets offline first; mocked responses alone cannot prove Ollama’s grammar converter accepts it.

Show more

12 of 12 messages

Prize Categories

Best Use of Gemma

I'm entering Best Use of Gemma because Gemma 3:4B is at the core of LazyNotes' note-generation pipeline.

Rather than using a hosted LLM API, I run Gemma locally through Ollama to convert classroom transcripts into structured study notes.

A significant part of the project involved making Gemma work reliably on ordinary CPU hardware: managing long transcripts, controlling output length, validating structured JSON, handling failures, and reducing unnecessary model calls.

The ability to run and experiment with an open-weight model is what made this project possible in the way I wanted to build it.

Overall Winner — Slow and Steady

I'm also submitting LazyNotes for consideration for the Overall Winner prize.

I think LazyNotes fits the spirit of the challenge pretty well. The whole idea came from wanting to do less of something tedious so I could focus on something more important: actually understanding my lectures.

What started as a simple recording-to-notes app turned into a much bigger engineering challenge than I expected. I spent a surprising amount of time figuring out how to make small, locally running AI models produce useful notes without taking forever or failing halfway through.

It's still a work in progress, but that's also what made building it interesting.

At the end of the day, LazyNotes is my attempt at making one small part of university life a little less exhausting.

Less writing. More listening.