This is a submission for the Presage SmartSpectra SDK Challenge
What I Built
You're following a paper box tutorial. The instructor says to pinch two corners while you tuck in a flap.
By the time you look back up, they've turned the box around and moved on. Now you have to let go of the fold to scrub back.
I built cueback for that moment. Look down to work and the video pauses. Look up and it replays the step from the beginning, so you hear the whole instruction again. The app comes with a short paper box lesson. You can also open a local MP4 or WebM and give it chapter times for your own tutorial.
Demo
Showcase video
Video placeholder for the live camera demo.
I'll show the same fold twice, first with the player controls and then with SmartSpectra on.
You'll be able to see me look down to fold, the player pause, and the same step start again when I look up.
Try it yourself
Extract the Windows x64 ZIP and open cueback.exe. Try simulation lets you test the pause and rewind with the camera off.
For live replay, open Set up camera, enter your own Presage key, and capture your screen and work poses. Windows download
Code
The source and setup instructions are on GitHub. From the checkout, run pnpm install --frozen-lockfile and pnpm start. Source and setup
How It Responds to Human Signals
cueback gives the webcam stream to the SmartSpectra Node SDK over Electron's camera bridge and requests live FACE_LANDMARKS. It measures where your nose sits relative to the line between the outer corners of your eyes, scaled by the distance between them.
Calibration samples that measurement while you face the screen and while you look down at your work. Camera integration ยท Pose detection
A brief glance won't pause the video. When you hold the work pose, cueback pauses and saves the last step you were watching. Holding the screen pose again rewinds to that step's chapter marker.
Saving the step matters if the tutorial crosses a chapter boundary while you're looking down. If cueback loses your face in this mode, it holds playback until you press Play.
There's also a calibrated Long blink option that replays the current step when you reopen your eyes. Playback logic
Prize Categories
Best Thoughtful Real-World Use. This is for people following tutorials with their hands full. Paper folding makes the problem easy to see, and the same interruption happens when you're holding a part in place during a repair. Captions, keyboard controls, and manual replay are available with the camera off.
Replay check lets you compare how many times you touch the controls while doing the same step in manual and camera modes.
Most Creative or Delightful Use. I started with the fold because looking away is part of following the instruction.
I wanted that movement to cue the whole step again. The chapter marker lets the player bring back the sentence you need, even if you never saw how much of it you missed.
Consent and Data Handling
The camera is off when you open cueback. Set up camera explains what it uses, and capture starts when you select Turn on camera.
Camera mode is for people aged 18 and older. The app shows a local preview while the camera is on.
SmartSpectra processes the camera frames and face landmarks on your device. cueback doesn't save or share them. Turning off the camera or quitting releases it and clears the tracking data.
Replay check counts and notes stay in memory and are cleared when you reset the check, switch lessons, or quit.
A key entered in the dialog is used for that session. A key saved in a local .env file stays there until you remove it, and Git ignores the file.
Thank you so much for reading it till here!
Top comments (0)