This is a submission for the Presage SmartSpectra SDK Challenge
What I Built
A stroke has taken their voice and their hands. They're thirsty. They know exactly what they want to say. All they can do is blink.
Hear Me is an open-source desktop communication aid for people who have great difficulty speaking or using their hands, for example with ALS or after a severe stroke. It reads the live face landmarks from Presage SmartSpectra through an ordinary laptop webcam and turns a deliberately held eye closure into a spoken phrase, in English or Hindi.
People with ALS, or after a severe stroke, can often still move their eyes after they lose their voice and their hands. Dedicated eye-gaze communication systems help, but they are expensive: Ontario's 2025 Assistive Devices Program price list shows an approved purchase price of CA$20,083 for one integrated eye-gaze system, a Tobii Dynavox I-13 (source). Most homes already have a laptop. I wanted to see how far that could go. Hear Me is not a replacement for a clinical AAC assessment or dedicated assistive technology.
How someone talks with Hear Me
- A highlight moves down the rows of the board on its own (every 1.6 s by default; the caregiver can slow it down).
- When the row they want is lit, the person closes their eyes for about a second. The highlight then moves through that row's phrases, ending on a Back step.
- When the phrase they want is lit, they close their eyes again. Hear Me says it aloud as soon as the eyes open.
- Made a mistake? For 3 seconds after a phrase, one more long blink says "Sorry, not that" and puts the highlight back on it.
Call for help is the first phrase in the first row, so it takes two long blinks. Four long blinks within 5 seconds sound an alarm from anywhere on the board, until a caregiver presses "I'm here". A letter board with word suggestions covers anything that isn't on the board.
Demo
The face in the video is a free stock clip from Pexels, edited so the person holds a blink, and played into the real Presage SDK frame by frame in real time (I don't show my own face publicly), so the LIVE values are the SDK's own output on each frame. The app's privacy view draws the eyes instead of showing the camera picture, and the live Presage values stay on screen in the "LIVE · Presage SmartSpectra" panel, so you can see the signal is real. Every selection in the video comes from the app's own blink detection.
Code
npm install, npm start, then paste your Presage API key on the first screen. The start screen shows the three steps (sit in front of the camera, calibrate for 20 s, close your eyes for a second to choose). The Space bar also works as a select, so you can try the board straight away. If the key is wrong or out of credits, Hear Me says so and stops, instead of retrying forever.
How It Responds to Human Signals
-
face.landmarks→ the long-blink switch. My first version used the SDK'sface.blinkingflag, and nothing got selected when I tested it on myself. Recording my own face showed why: the flag marks the moment of a blink, not how long the eyes stay shut. So Hear Me measures eye openness from the six landmark points around each eye (the eye aspect ratio), against each person's own open-eye level. A closure of at least 0.5 s and at most 5 s selects; shorter ones are natural blinks, longer ones are resting eyes. The tracker sometimes stops sending eye landmarks while the eyes are fully shut, so a short gap counts as closed only if the eyes were already closing. If the face stays out of view, the board pauses and says what to fix. -
face.talking→ for people who can still say a few words, the board pauses while the SDK detects talking and for 1.5 s after, which reduces accidental selections mid-sentence. -
face.expression→ a live second switch: the SmartSpectra surprise expression score, held above the threshold for 0.4 s, selects like a long blink. The gesture is raising the eyebrows, for people whose blinks are weak, and the score is on screen the whole time as "eyebrows %" in the LIVE panel. - Validation hints ("Face the camera", "Increase light on face") are shown to the caregiver, so setup doesn't need any knowledge of the SDK.
Everyone blinks differently, so a 20-second calibration records normal blinks, then three long ones, and sets the threshold between them, never below 500 ms (in my own recording a slow natural blink while looking down lasted about 450 ms). Switch-access devices use the same idea of a minimum hold; PRC-Saltillo's scanning settings, for example, describe a half-second "acceptance time" (source).
What I learned integrating the SDK in Electron: my laptop lists a virtual camera first, which can't hold 30 fps, so Hear Me picks the real one. And at 1280x720 the renderer couldn't keep the 25 fps the SDK needs on my CPU; 640x360 holds 30 fps.
For the people around the bed
- "I heard you". A caregiver presses one button, and the person sees "✓ Heard" next to what they said. Guidance for AAC communication partners says to acknowledge every message (AssistiveWare); this makes the acknowledgement visible.
- Undo works like de-selecting on a switch-scanning device. Some of those give one second to press the switch again (PRC-Saltillo); a deliberate blink takes longer than a button press, so Hear Me gives 3 seconds. Call for help is never undone.
- The caregiver's own phrases ("Call Mummy"), English and Hindi, and the privacy view.
Testing
The repo ships one 22.6-second recording of eye landmarks (numbers only, no video), so anyone can rerun the benchmark, or run the same script on their own recordings.
node tools/benchmark.mjs runs the long-blink switch on recorded eye landmarks:
| clip | what it contains | natural blinks | long blinks | false selects | missed |
|---|---|---|---|---|---|
| eyes-1 | natural blinks, one long closure | 9 | 1 | 0 | 0 |
The logic (scanning, switches, eye measurement, calibration, undo, alarm) has 50 tests (npm test). End-to-end checks replay recorded clips through the real SDK in a hidden window: natural blinks ignored and only the long one selected, the board pausing with no face, the caregiver flows, and a first-time setup from pasting the key to choosing a phrase.
If the face or the eyes can't be seen, the board pauses instead of guessing. The alarm is a sound from the laptop, so it needs the volume up and is not an emergency call system.
Prize Categories
Entering both: Best Thoughtful Real-World Use and Most Creative or Delightful Use.
Best Thoughtful Real-World Use. A board that speaks is only half of a conversation. The parts I spent most time on are the other half: undoing a wrong phrase without waiting for the whole board, letting the person see that their message was heard, a way to call for help from anywhere, and never scanning blind when the face slips out of view. Consent is a real button: the camera starts only when someone presses it, no frames are saved, SDK telemetry is off, and one button deletes the saved key and settings.
Most Creative or Delightful Use. Hear Me doesn't use SmartSpectra for vitals at all. It reads the face for intent: the same landmarks become a switch, raised eyebrows become a second one, and the board quietly waits while someone is talking. The small moment I like most is the "✓ Heard" mark, when the person on the bed sees that what they said actually landed.
Consent and Data Handling
- Consent. The camera turns on only after Turn on the camera and start is pressed, and turns off with Stop camera. Hear Me asks for the camera only, no microphone.
- Processing. Hear Me does not record or upload camera frames. SmartSpectra processes the camera stream on the computer; Presage's SDK documentation says vitals are computed on-device and its telemetry sends no raw video.
-
Network. The SDK contacts Presage to validate the API key and count measurement credits. SDK telemetry is off (
enableTelemetry: false). Hear Me itself sends nothing: no analytics, no AI service. - Stored, and for how long. Only the pasted API key and the settings (including the caregiver's own phrases), on that computer, until you delete them. The phrases said in a session are shown on screen and not saved.
- Deletion. One button on the start screen, Delete my saved key and settings from this computer, removes all of it.
- Sharing. Hear Me doesn't share camera images or what the person says with anyone. The only data that leaves the computer is what the SDK itself sends to Presage, such as the key check and credit counting.
- Hear Me is a communication aid, not a medical device. It does not diagnose, treat or monitor anything.
If you have cared for someone who couldn't speak, which phrase would you put on the board first?
Find me: GitHub · YouTube · LinkedIn · Portfolio · Hashnode · Peerlist · npm · LeetCode · Email




Top comments (7)
What an interesting project! Looking forward to checking out the implementation and exploring its features.
Motivational
Impressive
This has definitely caught my attention! Excited to give it a try. Excellent work
Super impressive! The creativity and ambition reflected in this submission make it an interesting project to explore.
This looks like a valuable contribution to the hackathon. Appreciate the creativity and commitment demonstrated through this submission.
Really fascinating idea! Can't wait to explore its capabilities and see it in action.