AirSketch

Lenin Goud — Thu, 22 Oct 2026 19:32:10 +0000

We Built an AI-Powered Whiteboard in 24 Hours: Hand Gestures, Speech Recognition, and AI-Generated Notes

What if you could write on a whiteboard without touching it, explain your ideas out loud, and automatically turn everything into organized notes?

That's the idea behind our AI-powered smart whiteboard, a project we worked on at HackGB 2026, hosted at the University of Wisconsin–Green Bay on October 17–18.

We wanted to explore how computer vision, speech recognition, and generative AI could work together to create a more interactive learning experience.

And we had just one hackathon weekend to bring it all together!

The Inspiration

Think about a typical classroom.

A professor explains a complicated concept while writing on a whiteboard. Students try to listen, understand, and take notes simultaneously.

Sometimes, important explanations get missed because everyone is busy copying what's on the board.

We started with a simple question:

What if a whiteboard could understand both what you're drawing and what you're saying?

Instead of treating drawing and speaking as separate activities, we imagined a system that combines them into a single learning experience.

Our goal was to build a whiteboard that could:

Recognize hand gestures through a webcam.
Allow users to draw without touching a screen.
Capture spoken explanations through speech recognition.
Transform transcriptions into structured, AI-generated notes.
Make interactive learning more accessible and engaging.

What We Built

Our project brings together three core capabilities.

1. Gesture-Controlled Whiteboard

Using computer vision, we created a virtual drawing interface controlled by hand movements.

The webcam captures the user's hand, and the gesture-recognition pipeline tracks hand landmarks to interpret movement.

Instead of using a physical marker or touchscreen, users can interact with a digital canvas through gestures.

The interesting challenge wasn't simply detecting a hand. It was translating continuously changing hand positions into meaningful drawing actions.

We had to think about coordinate mapping, gesture states, and how to prevent unintended drawing.

2. Speech Recognition

Drawing is only part of teaching. The explanation behind a diagram is often just as important as the diagram itself.

We incorporated speech recognition to capture spoken explanations and convert them into text.

This creates a textual record of the ideas being discussed, reducing the need to manually transcribe everything.

3. AI-Generated Notes

Raw speech transcripts aren't always easy to study.

They can contain repetition, incomplete sentences, and conversational language.

Our idea was to use generative AI to turn these transcripts into clearer, structured notes.

For example, imagine someone explaining machine learning while drawing a neural network.

The system can use the spoken explanation to produce organized notes covering concepts such as input layers, hidden layers, activation functions, and model training.

This is where the project becomes more than a virtual whiteboard: it becomes a potential AI-powered learning assistant.

How It Works

At a high level, the system contains two main processing pipelines.

Gesture pipeline:

Webcam → Hand Detection → Landmark Tracking → Gesture Interpretation → Digital Canvas

Speech pipeline:

Microphone → Speech Recognition → Transcript → AI Processing → Structured Notes

These pipelines handle different types of information but contribute to the same learning experience.

The Computer Vision Pipeline

The gesture-controlled whiteboard uses a sequence of operations:

Capture frames from the webcam.
Detect the user's hand.
Extract hand landmark positions.
Interpret the positions as gestures.
Map hand coordinates to the digital canvas.
Update the drawing based on the detected gesture.

One important engineering consideration is the difference between camera coordinates and screen coordinates.

The camera captures images at one resolution, while the drawing canvas may have different dimensions.

To make the interaction feel natural, the system must correctly translate movement between these coordinate systems.

Another challenge is maintaining stable drawing behavior. Even small variations in landmark detection can make a line appear shaky.

These details matter because a gesture-controlled application must feel responsive, not just technically functional.

The Speech and AI Pipeline

The speech-processing side follows a different workflow.

First, spoken audio is converted into text.

Then, the transcript is provided to an AI model with instructions to organize the content into readable notes.

A useful note-generation pipeline should preserve the original meaning, identify important concepts, and avoid introducing unsupported information.

The goal isn't simply to summarize everything. It's to produce notes that someone can actually use when reviewing a lesson.

Building as a Team

One of the most valuable parts of HackGB was collaborating under a tight deadline.

We divided the project into separate responsibilities so development could happen in parallel.

I focused on the gesture-controlled whiteboard and computer vision functionality, while my teammate worked on speech transcription and AI-generated notes.

This separation allowed us to work independently before integrating the components.

It also taught us an important software engineering lesson:

Building individual features is only half the challenge. Making them work together is where system design becomes important.

We had to consider how different components would communicate, how users would interact with the combined system, and how to keep the demonstration understandable.

Challenges and Lessons Learned

1. Real-Time Computer Vision Is More Than Model Accuracy

A system can detect hand landmarks correctly and still provide a frustrating user experience.

Responsiveness, stability, and gesture interpretation are equally important.

We learned to think about the entire interaction pipeline rather than focusing only on detection.

2. Integration Needs Planning

When multiple people build different components, consistent interfaces and clear responsibilities become essential.

Even in a short hackathon, thinking about component boundaries early can save valuable development time.

3. AI Output Needs Structure

Converting speech to text is useful, but turning that text into educational notes requires additional design decisions.

The AI needs to identify the most important information without losing context.

This reinforced the importance of prompt design and output quality when building AI applications.

4. A Good Demo Tells a Story

Hackathons aren't only about how much code you write.

You also need to explain the problem, demonstrate the solution, and communicate why the technology matters.

We learned that a focused, understandable demonstration can be more compelling than a long list of features.

Why This Matters

We built this project with education in mind, but the underlying idea has applications beyond classrooms.

Imagine using a similar system for:

Remote education: Instructors explaining concepts with interactive drawings.
Technical interviews: Candidates drawing system architectures while describing their reasoning.
Team brainstorming: Teams capturing spoken ideas alongside diagrams.
Accessibility: Exploring alternative ways to interact with digital interfaces.
Personal learning: Students organizing explanations into review materials.

These are future possibilities rather than features we've fully implemented, but they show how the concept could evolve.

What's Next?

A hackathon prototype is only the beginning.

Some improvements we'd like to explore include:

Better gesture stability and smoother drawing.
More intuitive gesture-based tools and controls.
Synchronizing spoken explanations with drawing activity.
Saving and replaying whiteboard sessions.
Exporting notes into shareable documents.
Adding AI assistance that understands both diagrams and spoken explanations.
Testing the system with real students and educators.

One especially interesting direction is multimodal understanding.

Rather than processing drawings and speech separately, a future system could reason about both together.

For example, when an instructor points to a specific part of a diagram and says, "This layer transforms the input," the AI could connect that explanation to the corresponding visual element.

That would move the project closer to a genuinely context-aware learning assistant.

My Biggest Takeaway

HackGB 2026 reminded me why I enjoy building AI applications.

Learning about computer vision, machine learning, and generative AI is exciting, but applying those concepts to a real problem is a completely different experience.

Working under a deadline pushed us to prioritize, experiment, debug, collaborate, and communicate our ideas.

And that's what made the experience memorable.

The most exciting part of AI isn't just what a model can do. It's what we can build when we connect different technologies to solve real problems.

Thank You!

A big thank you to the HackGB organizers, mentors, sponsors, volunteers, and everyone who helped make the event possible. Hackathons are a great reminder that you don't need months to start building something meaningful.

DEV Community: Lenin Goud