Unmute: reading silent lips for ICU patients who can't make a sound (offline, on a laptop)

Pankaj Ashok Magadum — Sun, 15 Nov 2026 17:27:52 +0000

After a tracheostomy or throat surgery, an ICU patient can be awake, alert, and completely silent. A tube in the neck means no air reaches the voice box. But their lips still move. So they mouth words at the nurse, and the nurse guesses. "Water? Pain? Your family?" When the guess is wrong, the patient tries again, gets tired, and gives up.

We spent HackUMass building Unmute: an ordinary laptop on the bed table that reads those silent lips, says the sentence out loud, and gets the nurse when it matters. No setup, no internet, no special hardware.

What we Built
The patient silently mouths a sentence at the laptop camera. Unmute shows the 3 most likely of 20 bedside sentences as big cards ("I need water", "I am in pain", "Call my family", "I cannot breathe"...). The patient confirms one with whatever they can still move, and the laptop speaks it.

No calibration for a new patient. An open-source lip reader plus other people's recordings works from minute one.
Confirm with a touch, a nod, one switch, or a long blink. The nurse sets it up with one question: "What can this patient do?" There's also a hands-free mode: recording starts when the lips move and stops when they rest.
Urgent sentences don't wait. When the lip reader is sure the patient said "I cannot breathe", a 2-second countdown starts. Then the screen turns red, an alarm sounds, and the nurse's phone buzzes, unless the patient stops it. A misread alarm is cancelled with one tap, and the nurse is told.
Always a way out. A draggable touch button opens every sentence as a big button. Every correction is saved as a sample of the patient's own lips, so the mistake teaches it.
The video never leaves the laptop. The server listens on 127.0.0.1 only. With Wi-Fi off, everything still works except the phone push.
How it works
camera
-> MediaPipe face tracking, in the browser (~108 numbers per frame + a small mouth crop)
-> local FastAPI server (127.0.0.1 only)
-> DTW matcher over everyone's recordings
-> Auto-AVSR lip reader on the laptop's GPU (PyTorch, Apple MPS)
-> blend -> top 5 -> Gemini reorders using bedside context (optional, text only)
-> 3 cards -> touch / nod / switch / blink
-> spoken aloud + Nurse page + ntfy push to the nurse's phone (urgent only)

A matcher with no training. Every recording becomes a template: a sequence of lip shapes, centred on the mouth, head tilt removed, scaled by eye distance. A new recording is compared with every template by dynamic time warping, so mouthing fast or slow doesn't matter. Every sentence the patient confirms becomes a new template, so it learns their lips (swollen, taped, lopsided) just from being used.
A real lip reader, asked the right question. Auto-AVSR (Imperial College London, Apache 2.0) is a 250M-parameter model trained on 3,291 hours of video. It runs offline on the laptop's GPU. The trick that made it work for us: don't ask it "what did they say?" Ask it "how likely is each of these 20 sentences?" Scoring a fixed list (decoder + CTC) was far more reliable than free-text transcription. We blend the two:

log P(phrase) = w · log P_lipreader + (1 − w) · log P_matcher (w = 0.25, measured)

Gemini as a careful second opinion. Gemini gets the top 5 candidates, the time of day, the last 5 requests, and an optional care note ("post-surgery, nil by mouth"). It may reorder them to break near-ties. It may not invent a sentence. In testing, it once dropped "I need water" because water had "just been given". So the safety rule lives in code, not in the prompt:

must = []
for c in [candidates[0]] + [c for c in candidates[:3] if c["phrase"] in URGENT]:
if c["phrase"] not in must:
must.append(c["phrase"])
for m in must:
if m in out:
continue
for i in range(len(out) - 1, -1, -1): # replace Gemini's lowest pick that isn't protected
if out[i] not in must:
out[i] = m
break
The lip reader's top match, and any urgent sentence it put in its top 3, can never be removed. If Gemini takes more than 3 seconds, or there's no network, Unmute uses the lip-reading order and shows an "offline mode" badge.

Demo

Code:

2004Hrishikesh / unmute

Unmute

Unmute helps hospital patients who can't make a sound, such as ICU patients with a breathing tube or a tracheostomy, talk again. The patient silently mouths a sentence at a laptop camera. Unmute reads their lips and shows the 3 most likely sentences. The patient nods to confirm one and the laptop says it out loud. Urgent requests ("I cannot breathe") turn the screen red and buzz the nurse's phone.

The one tool that exists understands 40 fixed phrases. Patients have more to say than that.

Unmute is a communication aid. It is not a medical device and it does not diagnose anything.

How it works

 Browser (web/)                                         Python backend (server/, FastAPI)
 ┌──────────────────────────────────────┐               ┌───────────────────────────────────────────┐
 │ webcam → MediaPipe Face Landmarker   │   frames of   │ /api/predict                              │
 │   per frame: 28 mouth blendshapes +  │   numbers     │   z-normalise with this user's stats      │
 │   40 lip points (centred, de-rolled, │

…

View on GitHub

The numbers (measured, not guessed)
We tested on people the system had never seen, with no calibration: 240 silent recordings of the 20 sentences. Chance is 5% for the first card and 15% for the 3 cards.

Brand-new patient, no calibration First card In the 3 cards
Matcher alone (other people's recordings) 37.9% 65.0%
Lip reader alone, zero-shot 66.7% 83.3%
Lip reader + matcher (what ships) 71.3% 87.1%
After the patient has used it + lip reader 78.3% 93.8%
The automatic alarm fired on about 6 in 10 urgent sentences, and wrongly on 1 of 180 others. Caveats, said out loud: only 2 people in that test, and the blend weight was picked on those same recordings.

Partner Technologies
Google Gemini API. Gemini is the bedside-context layer: it turns "the lips look like water or cold" into the right first card when the care note and the last few requests make one more likely. Things that mattered in practice:

Structured output, then distrust it. Gemini answers in JSON, and every choice is checked against the candidate list before anything reaches the screen.
Free-tier reality. Free keys can get as few as 20 requests a day per model. The server lists the Flash models the key can use (this costs no quota) and prefers the fast "lite" ones: about 0.5–1.3 s per call, against 1.3–7 s for the full models. A model that returns 429, 503, or 404 rests (until its quota resets, 60 s, or a week), and the next one is tried within the same 3-second budget.
Optional by design. The core path, lips to cards to speech, never waits on the network. A test captures everything sent to Gemini and checks that it holds no picture, no face numbers, and no name.
Hackathon Experience
Our git log tells the story better than we can:

Saturday 13:32: the whole pipeline worked end to end: calibrate, recognise, Gemini, nod, alert. Accuracy: 19% first card, with chance at 10%.
15:15: centre each recording on its own mean, add lip velocity, trim still frames. 19% → 40%. Then a hubness correction (it stops short phrases like "Yes" from soaking up wrong answers) took it to 47.5%.
17:22: the question that changed the design: "A sick patient can't sit through calibration." We made a shared starter model over everyone's recordings, and learning from every confirmed sentence.
20:21: we trained our own small neural network on volunteers' recordings. With 2 people it beat the matcher. With 4, measured leave-one-person-out, it didn't, so our training script switched it off by itself. It hurt, and it was the right call.
20:38: Auto-AVSR running on the laptop GPU. Nobody could tell us if a model trained on people talking would read silent mouthing. So we measured it: 83.3% in the 3 cards, zero-shot.
21:17 → 23:02: our own rule for showing the lip reader's free text was wrong. It would have put a wrong sentence on card 1 in 124 of 240 recordings, because its reading is usually a near miss ("called my family"). The new rule hides near misses: 4 of 240.
Sunday 02:23: urgent sentences raise the alarm by themselves when the lip reader is sure.
What we'd tell another team:

Test on people your model has never seen. Testing on the same person flattered every number we had.
Ask a big model to choose, not to generate. "Which of these 20?" beat "what did they say?" by a mile.
Put the LLM where it's safe, and write the safety rule in code, not in the prompt.
Accessibility is the last step. For a weak patient, picking a card is harder than the AI is for us.
Measure the thing that proves you wrong. Two of our proudest moments were numbers that killed our own ideas.
What's next
Test with speech-language pathologists and real tracheostomy patients, record more volunteers, add sentences the care team can set for each patient, and connect to hospital nurse-call systems.

Unmute is a communication aid. It is not a medical device, and it does not diagnose anything.

DEV Community: Pankaj Ashok Magadum

Unmute: reading silent lips for ICU patients who can't make a sound (offline, on a laptop)

2004Hrishikesh / unmute

Unmute

How it works