Building an ASL Learning App That Uses Computer Vision to Evaluate Your Signs in Real Time

QuickSign — Tue, 21 Jul 2026 12:17:50 +0000

We built QuickSign, a gamified American Sign Language (ASL) learning app that uses the user's webcam to evaluate whether they actually performed a sign correctly. Instead of only asking users to recognize signs, QuickSign analyzes handshape, location, movement, and palm orientation and provides feedback on what went wrong.

Live demo: https://aslgame.vercel.app

Why we started building this

Learning a new language usually involves a lot of repetition.

You learn a word, practice it, get feedback, and try again.

But learning sign language has an interesting problem.

A lot of digital learning tools can show you how a sign looks. They can ask you to identify a sign from a video or choose the correct answer from multiple options.

But there is a difference between:

"I know what this sign looks like."

and:

"I can actually produce this sign correctly."

We wanted to build something focused on the second problem.

So my co-founder Muhammad Saad and I started building QuickSign, a gamified ASL learning platform where users can learn signs and then perform them in front of their own webcam.

The application analyzes the user's movements and provides feedback on their performance.

The goal isn't just to tell someone that they got a sign wrong.

The goal is to help them understand why.

What QuickSign does

The basic experience looks something like this:

Learn a sign
↓
Practice the sign
↓
Turn on webcam
↓
Perform the sign
↓
Extract hand + pose landmarks
↓
Analyze handshape, location, movement, orientation
↓
Check the sign against its requirements
↓
Provide feedback

For example, instead of simply returning:

❌ Incorrect

the system can evaluate individual components of the sign.

Conceptually, the feedback might look like:

Handshape ✓
Location ✓
Palm orientation ✓
Movement ✗

This gives the learner something actionable to work on.

The technology behind it

QuickSign is built as a React and TypeScript web application.

Our current stack includes:

React 19
TypeScript
Vite
Tailwind CSS
Zustand
Supabase
Vercel
MediaPipe Tasks API
TensorFlow.js
WebRTC
PostHog

The webcam-based recognition happens directly in the browser.

We use MediaPipe hand and pose landmark detection to extract the information needed for recognition.

The important part is that the recognition pipeline does not simply take one frame and ask:

"Does this look like the sign?"

Instead, we treat a sign as a collection of parameters.

For our recognition architecture, these include:

Handshape
Location
Movement
Palm orientation
Non-manual markers where applicable

The exact parameters required depend on the sign being evaluated.

The first big lesson: a single frame isn't enough

This was probably one of the most important problems we encountered.

We had a bug with our COFFEE sign recognition.

The sign requires a specific movement involving the hands.

Our original implementation checked whether the user's hands were approximately in the right position and whether the handshapes looked correct.

The problem?

Someone could hold two static fists in roughly the right position and pass the check.

They weren't actually performing the sign.

They were just holding a pose that happened to look similar to one frame of the sign.

This taught us an important lesson:

If a sign requires movement, you cannot reliably verify it from a single frame.

So we changed our approach.

Instead of only looking at the current frame, movement-based signs are evaluated using a rolling window of landmark history.

Conceptually:

Frame 1 ─┐
Frame 2 │
Frame 3 │
Frame 4 ├──→ Movement analysis
Frame 5 │
Frame 6 │
Frame 7 ─┘

The system can now analyze how the hands moved over time rather than only where they are right now.

This allows us to reason about things like:

Linear movement
Circular movement
Repeated movement
Movement direction
Movement magnitude

This was a major architectural change for us.

It also changed how we thought about computer vision problems in general.

A static image and a sequence of movements are fundamentally different problems.

Our recognition architecture

The recognition engine is designed around sign definitions.

Instead of hardcoding every sign into separate verification logic, we define the requirements for each sign as structured data.

Conceptually, a sign might specify:

Sign
├── Handshape requirements
├── Spatial relationship
├── Movement requirements
├── Palm orientation
└── Non-manual markers

The generic verification engine then evaluates the required parameters.

A sign only passes if every required parameter meets its own threshold.

We deliberately avoided simply averaging everything into one score.

For example, imagine:

Handshape: 95%
Location: 92%
Palm orientation: 90%
Movement: 30%

An average score might still look reasonably high.

But if movement is essential to the sign, that shouldn't be enough.

The user didn't perform the sign correctly.

So our approach is closer to:

Required parameter 1 → PASS
Required parameter 2 → PASS
Required parameter 3 → PASS
Required parameter 4 → FAIL

Overall → FAIL

This makes the system more strict, but also makes the feedback more meaningful.

Where machine learning fits in

We also experimented with machine learning for sign recognition.

Our current system uses a small Bi-GRU classifier built with TensorFlow.js.

However, we made a deliberate architectural decision:

The ML model is not allowed to grant a successful sign on its own.

Instead, it acts as a veto or disambiguation layer.

Conceptually:

Rule Engine
│
├── FAIL ───────→ FAIL
│
└── PASS
│
↓
Bi-GRU Model
│
┌────┴────┐
│ │
VETO ACCEPT
│ │
↓ ↓
FAIL PASS

This means the machine learning model can catch potential false positives, but it cannot turn an obviously incorrect performance into a successful one.

We chose this approach because the rule engine gives us something very valuable:

Interpretability.

If a user fails a sign, we can potentially explain which component caused the failure.

A black-box classifier might simply say:

"The model predicts this isn't COFFEE."

Our system can instead reason about the individual requirements.

That makes the feedback much more useful for learning.

The second big lesson: passing automated tests doesn't mean the product works

Another challenge appeared when we started testing our recognition system with real people.

Some signs passed our automated tests perfectly.

Then a real human tried them.

And they failed.

At first, this was confusing.

We had automated fixtures.

We had regression tests.

We had confusor tests designed to catch false positives.

Everything looked good.

But our tests were based too heavily on idealized or synthetic movements.

Real people don't move like perfect mathematical models.

People have different:

Arm lengths
Camera positions
Sitting distances
Movement speeds
Hand sizes
Execution styles

The result was a classic problem:

The system worked in our test environment but didn't work reliably in the real world.

We had to start calibrating our thresholds against actual human performances.

This was a big shift in our development process.

We learned that for computer vision applications, automated tests are necessary, but they aren't sufficient.

You need both:

Automated tests
+
Real-world calibration
+
Actual human testing

All three reveal different classes of problems.

Making recognition work at different camera distances

Another problem we had to solve was scale.

A user's hands might occupy a large part of the camera frame.

Or they might sit farther away.

If we used raw pixel distances for spatial relationships, the same sign could look completely different depending on how close the user was to the webcam.

So instead of relying on absolute pixel measurements, our spatial thresholds are normalized using the user's shoulder width.

Conceptually:

Raw pixel distance
↓
Normalize relative to body scale
↓
Compare spatial relationship

This makes the recognition system less dependent on how close the user is sitting to the camera.

It's a small implementation detail, but it makes a big difference in real-world usability.

Building the actual learning experience

We didn't want QuickSign to feel like a computer vision demo.

The goal was to make it feel like a game.

We've built multiple learning systems around the recognition engine.

Currently, QuickSign includes:

Lessons
Practice mode
Story mode
Speed challenges
Streaks
Badges
Ranks
A shop
Spaced repetition
Multiplayer rooms
PvP duels

The app currently has three scenarios:

☕ Coffee Shop
🏥 Hospital
🏫 Classroom

Across these scenarios, the current system contains 51 signs.

The idea is to teach signs in context rather than presenting the learner with a completely disconnected list of vocabulary.

Multiplayer was another interesting challenge

We also wanted users to be able to practice together.

Instead of building a dedicated game server for every multiplayer interaction, we use:

WebRTC for peer-to-peer communication
Supabase Realtime for signaling

This allows us to build multiplayer functionality without requiring a traditional multiplayer game backend.

There are still plenty of challenges with this architecture, especially around connection reliability and NAT traversal, but it has been an interesting part of the project.

Privacy was important to us

Because QuickSign uses a webcam, privacy was something we wanted to think about from the beginning.

The actual sign recognition pipeline runs client-side in the browser.

The webcam video and landmark data used for recognition are not streamed to a server for the recognition process.

This means the recognition itself can happen locally:

User webcam
↓
Browser
↓
MediaPipe
↓
Landmarks
↓
Recognition engine
↓
Feedback

No recognition server is required for the core experience.

For a product that asks users to point a camera at themselves, we believe this is an important architectural decision.

We're still learning about ASL itself

There's also an important limitation we want to be transparent about.

Neither of us is Deaf.

We're developers who are building technology around ASL education, and we know that this comes with a responsibility to approach the subject carefully.

ASL isn't simply "English represented with your hands."

It's a real language with its own linguistic structure, grammar, and cultural context.

We're still learning, and we don't want to pretend that building a technically impressive recognition system automatically makes us experts in ASL.

Our goal is to keep improving the product, listen to feedback, and work toward making the learning experience more accurate, respectful, and useful.

What we're working on next

QuickSign is currently live and we're preparing for a broader public launch.

Our immediate focus is:

Improving recognition accuracy with real users
Expanding the number of signs
Improving calibration
Making feedback more useful
Continuing to test the recognition engine against real-world performances
Growing the learning content
Getting feedback from ASL learners and people with relevant expertise

The biggest metric we're watching isn't how many people visit the website.

It's whether someone:

Completes their first lesson.
Finds the experience useful.
Comes back and continues learning.

For us, that's a much better indication that we're actually solving a problem.

Try QuickSign

If you're interested in computer vision, machine learning, language learning, or just want to see what we've built, we'd love for you to try it.

👉 https://aslgame.vercel.app

We're especially interested in hearing from developers who have built applications involving:

Computer vision
Real-time ML
Browser-based AI
Human motion tracking
EdTech
Language learning

One of the biggest lessons we've learned so far is that getting a computer vision system to work in a controlled demo is one thing.

Getting it to work reliably for real humans, with different cameras, different environments, and different ways of performing the same task is a completely different challenge.

If you've worked on something similar, I'd love to hear what you learned.

What was the biggest gap you encountered between "it works in the demo" and "it actually works for users"?

DEV Community: QuickSign

Building an ASL Learning App That Uses Computer Vision to Evaluate Your Signs in Real Time