DEV Community: Aman Kumar

LectureAI: I Let Whisper and Gemma Attend Class for Me

Aman Kumar — Sun, 25 Oct 2026 13:48:27 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

LectureAI turns a lecture recording into everything you need to revise it. Upload an MP3, WAV or M4A in Hindi, English or Hinglish, and it gives you:

a timestamped transcript
a summary and a topic timeline
likely exam topics, each linked to the moment it was taught
structured notes
flashcards with a revision queue
assignments and deadlines the teacher mentions, collected on a calendar for you to confirm or dismiss

It saves the hour you would spend replaying a recording, pausing and typing notes. It also catches the "submit this by Friday" line you missed while writing something else. The lazy student gets to actually listen in class.

By default, everything runs on my own computer. The dashboard shows an "All local services ready" badge, and no audio or transcript is sent to the cloud. If you choose the optional ElevenLabs engine, that lecture's audio is uploaded for transcription.

Demo

🎥 Video demo: https://www.youtube.com/watch?v=dIBrWXM4P5E

Dashboard: courses, lectures, flashcards due and open deadlines at a glance.

Lecture view: summary, topic timeline and likely exam topics sit next to an audio player, so every topic is one click away from the audio.

Assignments: deadlines detected from lectures appear on a calendar with Open, Needs review, Done and Dismissed tabs. The AI proposes and you decide.

Code

https://github.com/AMAN240310/Lectur-to-everything

How I Built It

LectureAI is built on open-source models running locally, with an optional cloud engine:

Whisper transcribes the audio, including Hindi and Hinglish, with timestamps.
Gemma, served through Ollama, reads the transcript and produces the summary, topic timeline, exam topics, notes, flashcards and deadline candidates.
ElevenLabs Scribe is an optional cloud transcription engine. Whisper runs locally by default, and you can switch a lecture to ElevenLabs when you want faster or more accurate transcripts and are fine with the audio being uploaded.

The pipeline is: upload → Whisper (or ElevenLabs) transcript → Gemma structured extraction → notes, flashcards, deadlines → dashboard.

A few design choices that mattered:

Timestamps everywhere. Every topic and exam topic links back to the audio, so you can check the AI against what was actually said.
Human in the loop for deadlines. Detected assignments go into a "Needs review" tab instead of silently landing on your calendar, because a wrong deadline is worse than a missing one.
Reprocess and processing log. Any lecture can be re-run, and the log shows what happened at each step, so weak output is fixable instead of a dead end.

Why Does Open Innovation Matter?

A lecture recording is private: your teacher's words, your classmates' questions, your own voice. With a closed API, every recording is uploaded, billed per minute, and useless when the Wi-Fi drops. Open-weight models changed that:

Privacy: by default, nothing leaves the laptop.
Cost: a whole semester of lectures costs nothing beyond electricity, which matters for students.
Language: Whisper handles Hindi and Hinglish, which many note-taking tools handle poorly.
Control: I can swap model sizes, tune prompts and re-run processing as often as I like without worrying about a bill.

Prize Categories

Best Use of Gemma: Gemma, served locally through Ollama, is the brain of LectureAI. It turns a raw transcript into the summary, topic timeline, likely exam topics, notes, flashcards and deadline candidates, all on a laptop with no internet connection and no API bill.

Best Use of ElevenLabs: LectureAI can transcribe lectures with ElevenLabs Scribe as an alternative to local Whisper. The default path stays fully local, and ElevenLabs is an opt-in engine for people who want cloud-grade transcription.

The Lazy Payoff

In the demo, a 7-minute lecture became a full summary, a topic timeline, seven likely exam topics and flashcards with no typing on my side. Across four lectures, LectureAI generated 41 flashcards ready to revise.

InvoicePilot: a private AI invoice sorter for freelancers that runs offline with Gemma 4

Aman Kumar — Sun, 18 Oct 2026 19:38:36 +0000

This is my submission for the Hacktoberfest Open-Source AI Challenge: Week 2 (Side Hustle).

The problem

Every freelancer I know has the same quarterly ritual: dig through email attachments, download folders and photos of paper receipts, then retype everything into a spreadsheet.

The "AI receipt scanners" that promise to fix this share two problems. They upload your client's invoices to someone else's API, which can break an NDA, and they add yet another monthly subscription.

What I built

InvoicePilot is a local-first invoice and receipt assistant. Drop in a PDF, PNG or JPG and it:

Reads the text, using the PDF text layer or Tesseract OCR for scans and photos
Sends it to Gemma 4 E2B running locally through Ollama, to extract vendor, dates, line items, tax and total
Validates the result and flags anything suspicious
Shows the original document next to the extracted fields so you can review it
Exports approved records to CSV and charts spending by month, category and vendor

Everything runs on localhost. No document text, price or client name leaves the machine.

Demo video:

Why open models matter here

An invoice is one of the most sensitive documents a freelancer holds: client names, rates, bank details, tax IDs. Open weights change what's possible:

Privacy: the model runs on my laptop, so none of that data goes to a server I don't control.
Cost: there is no per-document fee and no subscription, which matters for a tool whose users watch every rupee or dollar.
Swappable: the model is one line in .env (OLLAMA_MODEL). I run Gemma 4 E2B by default, and I can try Llama 3.2 or Qwen without touching any code.
Offline: after the one-time model download, it works with no internet connection.

A small model like Gemma 4 E2B is what makes this realistic on an ordinary laptop (mine has 16 GB RAM and an RTX 3050).

Small models make mistakes, so I don't trust them blindly

Small on-device models can invent a zero tax line when a tax breakdown exists, confuse MM/DD with DD/MM dates, or return totals that don't add up. I treat the model's output as untrusted input:

Schema validation (Zod) rejects malformed output
One bounded repair attempt sends the exact validation error back to the model. If that fails, the record goes to human review instead of crashing
A deterministic math check: the backend verifies subtotal + tax - discount = total itself, in integer minor units, and warns on any mismatch
A human approves everything: nothing reaches the CSV export until you click approve

Who would pay

Freelancers, consultants and small business owners who handle client documents under confidentiality. The open core stays free (MIT). A paid one-time desktop build with extras such as separate client books is the way I'd make money from it.

Try it

Code (MIT): https://github.com/AMAN240310/invoicepilot

Stack: React + Vite, Express, SQLite, Tesseract.js, Ollama with Gemma 4 E2B.

Setup: ollama pull gemma4:e2b-it-qat, copy .env.example to server/.env, then run the server and the client.

I Tested a Local Gemma Plant Guide. It Got the Sunflower and Missed the Lily

Aman Kumar — Sat, 10 Oct 2026 14:29:56 +0000

I wanted an excuse to look at plants instead of my phone. Wander is a plant guide where the phone's job is small: take one photo, get one answer, ask one follow-up, and put the phone away.

A Gemma model running on my own laptop names the plant and explains what it saw. Then I decide whether the answer is good enough to keep.

It started with a mistake

My first test photo was a sunflower, and Gemma got it right. My second was a pink flower with six spotted petals, and Gemma told me, confidently, that it was a Fuchsia. It was a lily.

That shaped the app. A small local model is useful, but it can be wrong in a convincing voice. So Wander keeps the human in charge:

Nothing counts until you confirm it, and you can edit the name first.
Every answer shows its doubt: a confidence score, the features it used, and look-alikes it may be confusing.
You can cross-examine it. The Ask tab sends the same photo back and lets you challenge the answer, for example "Why not a lily?"
The model is told never to call a plant safe, and every result carries a warning not to eat or handle plants based on an AI guess.

What it does

Find: photograph a plant and get a name, confidence, a short summary and visible features.
Map: each confirmed find appears where you took the photo, or where you drop a pin.
Collection: your species, catalogued like a herbarium, with every sighting.
Badges: milestones like first find, five species and an early-morning sighting. These are plain rules in code, not AI.

Why open-weight mattered

Identification and follow-up questions run through Ollama on my own laptop (RTX 3050, 16 GB RAM) using gemma4:e2b-it-qat, a 4.3 GB download. The photos are processed on my machine, there's no cost per photo, and the identification prompt, safety rules and chat behavior are all text I can change in my own code.

It isn't fully offline. The photos and the AI model stay on my laptop, but the map background loads from OpenStreetMap, and sighting records are stored in MongoDB, which has to be reachable when the app runs.

How it's built

React and Tailwind on the front, an Express server for uploads, MongoDB and badges, and a small FastAPI service that makes Gemma answer in strict JSON. My phone connects to the laptop over the local network. Most of the code was written with AI coding assistants. I directed the design, ran every service and tested it on photos of plants.

What's next

A proper walk. So far I've tested Wander on photos of plants on my laptop. The next step is taking it outside and counting how often it's right.
Offline map tiles. The map background needs internet today, so I want to cache map tiles for the area I'll walk in.
A specialist plant model for naming. Gemma sometimes names a flower wrongly with high confidence. I'd like to test a dedicated plant classifier for the naming step and keep Gemma for the explanations and questions.

GitHub: https://github.com/AMAN240310/Wander

Prize category: Best Use of Gemma

I built a hands-free slide remote for a friend who can never find the right page

Aman Kumar — Sun, 04 Oct 2026 14:44:08 +0000

I Built a Presentation Remote That Uses Hand Gestures and Voice

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend.

What I Built

My friend Dinesh Prajapath presents frequently. His decks are often large PDFs, and halfway through a talk he can lose track of where something is:

"The pricing part was around page 12... no, 14?"

Scrolling through a PDF in front of an audience is stressful. Traditional clickers make sequential navigation easy, but jumping directly to a specific slide usually still requires knowing its position.

So I built a presentation remote controlled using a laptop webcam and microphone.

The goal was simple:

Let the presenter navigate a presentation without touching the laptop.

What It Does

Hand Gestures

The webcam tracks the presenter's hand using MediaPipe.

Swipe right/left → change slides
Open palm → blank the presentation screen
Fist → jump to the first slide

No special hardware or physical clicker is required.

Voice Navigation

The presenter can hold a key and say something like:

"Go to the pricing slide."

Instead of requiring the presenter to remember the page number, the system searches the content of the presentation and finds the most relevant slide.

It can also handle commands such as:

"page 7"
"first slide"
"last slide"
"go to pricing"
"show weather GPT"

Why Open Models Mattered

Everything runs locally on the laptop:

MediaPipe → hand tracking
Whisper → speech-to-text
Gemma via Ollama → semantic understanding and slide selection

This was important for several reasons.

Privacy

The presentation, webcam feed, and microphone audio are processed locally and are not sent to a cloud service.

That matters when presenting slides that are confidential or haven't been published yet.

Cost

There is no per-request API cost.

Once the required models are downloaded, there is no cloud inference bill for each presentation.

Offline

After the initial model downloads, the system can operate without an internet connection.

That's useful in classrooms, seminar halls, and event venues where Wi-Fi can be unreliable.

Control

I could change the models, prompts, matching logic, and decision-making process myself.

This ended up being one of the biggest advantages of using an open/local approach.

Prize Category

I'm entering the Best Use of Gemma category because Gemma runs locally through Ollama and is used for semantic slide selection when deterministic matching is insufficient.

Instead of sending the presentation content to a cloud API, Gemma helps the system understand ambiguous voice commands and identify the most relevant slide locally.

Gemma is not responsible for controlling the presentation directly. It acts as the final semantic decision-maker when deterministic matching cannot confidently choose a slide.

The Problem With My First Approach

My first version was much simpler.

I extracted the text from every slide, gave all of it to Gemma 2B, and asked it to return the page number matching the user's request.

It didn't work reliably.

For example:

"intelligent record system" → page 14 instead of page 5
"WeatherGPT" → page 15 instead of page 18

One problem was that some slides contained numbers such as:

04 Land Records

Those numbers were section labels printed on the slides, not the actual PDF page numbers.

The small language model could confuse those numbers with the real slide positions.

So instead of making the LLM responsible for everything, I changed the architecture.

The Hybrid Approach

The final system uses multiple levels of matching.

1. Direct Commands

Simple commands don't need an LLM.

For example:

page 7
first slide
last slide
next slide
previous slide

These are handled directly by the application.

This makes them faster and more reliable.

2. Keyword + Fuzzy Matching

For content-based requests, the system first searches the extracted slide text.

This handles obvious matches without involving the LLM.

It also helps with small differences caused by speech recognition.

For example:

Weather GPT

can still match:

WeatherGPT

3. Gemma as a Final Decision Maker

If the keyword matching produces several plausible slides, Gemma receives the remaining candidates and decides which one best matches the request.

The normal path limits this to the top five candidates.

If keyword matching cannot produce useful candidates, the system can fall back to semantic matching against the slide titles.

Gemma can also return:

none

when none of the candidates are relevant.

In that case, the presentation does nothing instead of jumping to a potentially incorrect slide.

That conservative behavior is intentional.

How Voice Recognition Works

Whisper runs locally, so the microphone audio does not need to be uploaded to a speech API.

I also give Whisper the presentation's slide titles as vocabulary hints.

For example, if the deck contains terms such as:

Land Records
WeatherGPT
Intelligent Record System
Digital Governance

those terms can be provided as context to improve recognition.

I also filter out Whisper segments that are likely to be silence or low-confidence/noise.

This helps prevent silence or background noise from accidentally becoming a navigation command.

How It Works

Gesture Pipeline

Webcam
   ↓
MediaPipe Hand Landmarks
   ↓
Gesture Rules
   ↓
Keyboard / Presentation Action

Voice Pipeline

Microphone
   ↓
Whisper
   ↓
Command Detection
   ↓
Keyword + Fuzzy Matching
   ↓
Gemma / Ollama when needed
   ↓
Target Slide

Overall Architecture

                     ┌─────────────────────┐
                     │     Presentation    │
                     │       PDF/PPTX      │
                     └──────────┬──────────┘
                                │
                           Extract Text
                                │
                                ▼
                     ┌─────────────────────┐
                     │    Slide Indexing   │
                     └─────────────────────┘


 Webcam ──→ MediaPipe ──→ Gesture Rules ──→ Slide Action

 Microphone ──→ Whisper ──→ Keyword/Fuzzy Search
                              │
                              ▼
                       Candidate Slides
                              │
                              ▼
                         Gemma / Ollama
                              │
                              ▼
                         Target Slide

What I Learned

The biggest lesson from this project was that using an LLM for everything isn't necessarily the best architecture.

My first instinct was:

User Request
     ↓
    LLM
     ↓
Slide Number

But the system became much more reliable when I divided the problem into smaller parts:

Simple request
     ↓
Deterministic logic

Clear content match
     ↓
Keyword + fuzzy matching

Ambiguous request
     ↓
Small candidate set
     ↓
LLM decision

This reduced the amount of reasoning the model had to perform and also made failures safer.

The LLM became a decision-making component rather than the entire navigation system.

Why I Chose This Design

A presentation remote has an unusual requirement:

Being wrong is worse than doing nothing.

If the system fails to find a slide, the presenter can try again.

But if it confidently jumps to the wrong slide during a live presentation, it creates confusion.

So I designed the system to prefer:

No action

over:

Probably the right slide

when confidence is low.

Limitations

The project is still a prototype, so there are several limitations.

Gesture Detection

The gesture thresholds are tuned for a particular webcam setup.

Lighting, camera position, hand visibility, and background can affect detection accuracy.

Speech Recognition

Whisper can sometimes misrecognise accented or noisy speech.

A larger model such as small can improve recognition, but requires more computational resources.

Text-Based Navigation

Voice navigation depends on text extracted from the presentation.

Image-only or scanned PDFs may not contain useful extractable text.

Slides with very little text are also harder to locate using voice commands.

Slide Synchronization

The remote maintains its own slide position.

If the presenter manually changes slides using the mouse or keyboard, the internal slide counter can potentially drift from the actual presentation.

Local Model Resources

Larger language models can improve semantic matching, but they require more RAM and CPU/GPU resources.

The project therefore involves a trade-off between:

Model Size
    ↕
Accuracy
    ↕
Local Hardware Requirements

What I'd Improve Next

There are several directions I would like to explore:

Better automatic gesture calibration
More robust hand tracking across different lighting conditions
OCR support for image-based/scanned slides
Better synchronization with browser-based PDF viewers
More advanced semantic slide indexing
Support for multiple presentation applications
Confidence scores and visual feedback before changing slides
More natural voice commands such as:
- "Go back to the slide about pricing"
- "Show the architecture section"
- "Find the slide mentioning MongoDB"

Demo

Code

The complete source code is available on GitHub:

View the project on GitHub

The project includes the gesture recognition, voice navigation, PDF/PPTX slide extraction, fuzzy matching, and local Gemma/Ollama integration.

Final Thoughts

I started this project because a friend kept running into a very practical problem while presenting.

What looked like a simple "control the slides with gestures and voice" project turned into a useful lesson in system design.

The most important improvement wasn't choosing a bigger model.

It was reducing what the model had to solve.

Instead of asking an LLM to understand the entire presentation and return a page number, I combined deterministic rules, fuzzy search, and an LLM only where it actually added value.

That made the system faster, more predictable, and safer to use during a live presentation.

And most importantly, Dinesh no longer has to remember:

"Was pricing on page 12 or 14?"