I Built a Private, Offline Lecture-to-Notes Tool with Whisper and Gemma 3

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Lecture2Notes is a local-first command-line tool that turns a lecture recording into study material. Give it an MP3, M4A or MP4 and it produces:

• structured study notes (overview, definitions, formulas, algorithms, examples, common mistakes, quick-revision summary and revision questions)
• a timestamped transcript
• a validated list of assignments, quizzes, exams and deadlines, each with the timestamp and the evidence from the recording

It saves the hour of re-listening and note-taking that usually follows a lecture, and it helps anyone who has ever missed an announcement like "the lab report is due next Friday" because they were writing something else down.

The part I care most about is trust. An AI note-taker that makes up a deadline is worse than none, so:

• the lecturer's wording is kept as-is ("next Friday")
• calendar dates stay null unless you supply the real lecture date
• every action item needs evidence from the transcript or it is dropped
• if any step fails, the notes carry a visible INCOMPLETE banner instead of pretending everything worked

Everything runs on your own machine: no cloud transcription, no API key, no uploading private recordings.

Demo

Example of what the action-item output looks like:

{
  "type": "assignment",
  "description": "Submit the lab report on binary search trees",
  "deadline_text": "next Friday",
  "deadline_date": null,
  "source_timestamp": "00:01:20",
  "evidence": "The lecturer says the lab report must be submitted next Friday.",
  "confidence": "high",
  "needs_verification": true
}

Code

shauryasri14 / Lecture2notes

Lecture2Notes

Turn a lecture recording into study notes, revision questions and a list of assignments/deadlines, entirely on your own computer.

Lecture2Notes transcribes a recording with faster-whisper, then uses a local LLM served by Ollama (default: gemma3:4b) to write structured notes. No cloud service, no API key, no upload of your recordings.

---

Contents

1. What it does
2. How it works
3. Requirements
4. Setup
5. Quick start
6. Command-line options
7. Output files
8. Trying it without audio
9. Demo guide
10. Configuration (.env)
11. Troubleshooting
12. Performance expectations
13. Privacy
14. Project structure and tests
15. Known limitations

---

What it does

• Transcribes MP3, WAV, M4A, MP4 and other audio/video files, keeping real segment timestamps.
• Splits the transcript into overlapping chunks and summarizes each one.
• Merges the summaries into one study document: overview, definitions, formulas, algorithms, examples, common mistakes, quick-revision summary and revision questions.
• Finds assignments, quizzes, exams and deadlines in the transcript and saves them as validated JSON with the timestamp and…

View on GitHub

git clone https://github.com/shauryasri14/Lecture2Notes
cd Lecture2Notes
python -m venv .venv && source .venv/bin/activate   # Windows: .\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
ollama pull gemma3:4b
python lecture2notes.py lecture.mp3 --whisper-model base --language en

The README covers setup for Windows, Linux and macOS, plus a troubleshooting table.

How I Built It

Open-source stack

• faster-whisper (Whisper speech recognition) for local transcription with real segment timestamps, running on CPU by default
• Gemma 3 4B (an open-weight model) served by Ollama for summaries, merging and extraction
• Plain Python, requests, watchdog for optional folder watching, and pytest

Pipeline

recording → faster-whisper → timestamped segments → chunker (1200 words, 100 overlap)
  → per-chunk summaries → hierarchical merge → notes.md
  → per-chunk action-item extraction (JSON, validated, de-duplicated) → action_items.json

Design decisions:

• Chunking is deterministic and tested separately from the models, so overlap and forward progress are guaranteed even when one segment is huge.
• Hierarchical merging keeps very long lectures from overflowing the model's context window.
• Action items come from the transcript chunks, not from the summary, and the model's JSON is validated before use. Malformed JSON gets a bounded retry, then fails safely instead of producing made-up items.
• Bounded retries and timeouts for every Ollama request, and a failed chunk is recorded rather than hidden.
• Resilience on real machines: while testing on a Windows laptop I hit a PyAV version mismatch and a missing CUDA library. The tool now falls back to FFmpeg decoding and to CPU automatically.

The test suite (53 tests) mocks Whisper and Ollama, so it needs no models. I also ran the real HTTP client and CLI against a fake local Ollama server. Those tests check the logic and wiring, not the quality of notes from real audio. [Add your own honest result from running it on a real lecture clip, and what you'd improve.]

Why Does Open Innovation Matter?

Lecture recordings are private: they can contain classmates' voices, a professor's unpublished material and personal details. With open models, all of it stays on the laptop. Whisper and Gemma run locally through Ollama, so there is no per-minute fee, no usage limit and no data leaving the machine. That matters especially for students on limited budgets and unreliable internet. A closed API would have meant uploading recordings and paying per hour of audio.

Open components also let me inspect and control behavior. I could set the temperature, choose the model size to match the hardware, and swap in a different Ollama model with one flag. And when the tool broke, the open libraries let me find the actual cause (a metadata_errors argument PyAV didn't accept) and work around it instead of waiting on a vendor.

Prize Categories

Best Use of Gemma: Lecture2Notes runs Gemma 3 4B locally through Ollama for every language-model step: per-chunk lecture summaries, hierarchical merging into one study document, and structured extraction of assignments and deadlines. Everything runs on a CPU-only laptop with no cloud API, and the model is switchable with one flag (--model).

Why Gemma 3 4B? I wanted something a student could actually run: it fits on an ordinary laptop with no GPU, through Ollama, with one ollama pull gemma3:4b. A small model needs guardrails, so the pipeline works around it. Transcripts are split into ~1,200-word chunks so each request stays small, prompts tell the model to use only what is in the transcript, and the deadline extraction returns JSON that is validated before anything is used. Items without evidence are dropped, and malformed JSON gets a limited number of retries and then fails safely.

Honest limits. A 4B model can miss or misread an announcement, especially in a noisy recording or a long lecture, and speech-recognition mistakes pass straight through to its output. That is why every action item carries a timestamp and evidence quote, and why calendar dates stay null unless you supply the real lecture date. If you have more RAM or a GPU, --model lets you swap in a larger Gemma through Ollama with no code changes.