What I Built
Pip is a tiny digital companion that lives on your signals.
Smile, and Pip smiles back. Look surprised, and Pip startles. Look sad, and Pip droops and leans in to keep you company. Underneath the cuteness, Pip is wired directly into three live SmartSpectra signal families at once:
- Its heart — and its whole round body — beat at your measured pulse.
- Its belly rises and falls with your breathing trace.
- It blinks when you blink.
And then there's Calm Together, my favourite part: a tiny shared ritual. Tick the toggle, breathe slowly and steadily, and Pip's eyelids droop stage by stage until it falls asleep in your heartbeat. Breathe fast again — or startle it with a surprised face — and Pip wakes up. You calm the pet by calming yourself: a mindful micro-break for people who live at a desk, disguised as a pet.
Pip is deliberately a toy and a mirror. It is not a medical device: no diagnosis, no treatment advice, no health decisions — just play and gentle self-awareness.
Demo
What you're seeing: the demo uses Pip's recorded check-in mode. I consent and click Play a recorded check-in — no camera involved. The SDK measures a real 62-second face video I recorded earlier on my phone (it plays in the corner so you can see exactly what Pip sees, and the session is labelled RECORDED CHECK-IN REPLAY throughout — it is never presented as live). The SDK's own hints appear, the pulse locks on, and Pip starts living on that heartbeat: watch the readouts settle (pulse, breathing ≈ 15 breaths/min, expression) until the badge flips to RECORDED CHECK-IN — COMPLETE. Measured from that very recording by the SDK: pulse readings centred around ≈ 87 bpm, breathing dipping to ≈ 9 breaths/min in the calm, eyes-closed segment. The live camera mode is real too — during development Pip mirrored my expressions live from a phone camera — but my PC has no webcam and its tethered-phone camera link couldn't hold the steady frame rate the vitals need, which is exactly why the check-in mode exists.
No webcam or API key handy? The app ships with a clearly labelled demo mode (simulated signals, with a "puppeteer" panel to steer the imaginary person) so anyone can try the full experience in seconds. The video above uses real SDK-measured signals only.
Code
https://github.com/firmanyuansyach/pip-pulse-pet
A small, dependency-light Node app:
-
server.mjs— serves the page and runs the SmartSpectra session. Browser frames arrive over a local WebSocket and are fed to@smartspectra/node-sdkthrough its documented custom-input path (useCustomInput()/sendFrame()), requesting the breathing, cardio, and face metric bundles. If a bundle is rejected at start, the server steps down a graceful ladder (all three → pulse + breathing → breathing) and tells the UI exactly which signals are live. -
video-runner.mjs— the recorded check-in worker. The SDK'swaitUntilComplete()blocks the calling thread for the whole file read, so the file mode runs in a child process: it feeds a recorded video to the SDK withuseFile(), using the same metric bundles and the samedecodeMetrics→lib/normalize.mjspipeline, and streams the normalized frames back to the server as JSON lines. Only the frame source differs — the pet can't tell the two modes apart. -
lib/normalize.mjs— turnsdecodeMetricsoutput into the small JSON shape the pet logic consumes. -
public/pet.js— Pip's behaviour state machine: expression mirroring with anti-flicker debounce, heartbeat rendering at the measured BPM, the breathing belly (driven by the SDK's breathing trace), blink mirroring, and the Calm Together sleep logic. -
public/app.js+index.html— the consent gate, camera capture (native-aspect frames at ~30 fps), the recorded check-in player, live readouts (pulse, breathing rate, expression, HRV once available), and the SDK's ownvalidationStatushints surfaced as friendly camera guidance.
Setup is npm install, put your key in .env, npm start, open http://localhost:3000. There's also a headless smoke test (npm run smoke) that verifies the server, the no-key error path, and the complete demo-mode flow end to end.
How It Responds to Human Signals
| Live signal (SmartSpectra) | Pip's response |
|---|---|
| Expression: happiness | Smiles back, cheeks bloom |
| Expression: surprise | Eyes widen, brows jump — can startle a sleeping Pip awake |
| Expression: sadness | Droops, leans in, stays close |
| Blinking | Blinks when you blink |
| Pulse rate | Heart + body thump at your BPM; glow pulses with each beat |
| Breathing trace (upper/lower) | Belly rises and falls with your chest |
| Breathing rate | Calm Together: sustained ≤ ~11 breaths/min → eyelids droop → sleep; ≥ ~14 → wakes |
| HRV (RMSSD) | Shown in the readout panel after ~60 s of pulse, labelled informational only |
Two details I'm proud of: expression changes need ~half a second of persistence before Pip reacts, so it never flickers between faces; and numbers are only trusted in the UI when the SDK marks them stable — until then Pip honestly says it's still "finding your pulse."
Prize Categories
I'm submitting Pip for:
- Most Creative or Delightful Use — a pet whose heartbeat, breathing, facial expressions, and sleep are all yours. The Calm Together loop turns raw signals into a tiny emotional ritual: the only way to put Pip to sleep is to slow your own breathing down.
- Best Thoughtful Real-World Use — desk workers rarely take real breaks. Pip is a consent-first, privacy-by-default, 60-second mindful micro-break that needs no wearable, no account, and no data collection — and it is honest about what it is: wellness play, never a medical claim.
Consent and Data Handling
Pip was designed so the honest answer to "what do you keep?" is nothing.
- Consent before capture. The camera cannot start until the user reads a plain-language notice on the consent gate and clicks I agree. There is no covert monitoring of any kind: the camera is only ever on while the page says it is, and pressing Stop (or closing the tab) ends capture immediately and releases the camera.
- Storage. Pip records no video, takes no screenshots, and writes no measurements to disk. There is no account, no database, and no history. Session state exists only in memory while you use it — which also means there is nothing to delete afterwards.
- Where processing happens. Frames are processed in memory by the SmartSpectra SDK's native runtime on the user's own machine. The SDK contacts Presage to authenticate the API key and validate the subscription. Pip's own server (also local) adds no uploads of its own.
-
SDK telemetry — disclosed, and off by default in Pip. Per Presage's official SDK Telemetry & Privacy documentation, the SmartSpectra SDK's own telemetry is enabled by default. When enabled, it sends Presage (a) aggregate measurement-session summaries — SDK build/version, platform/OS/device model, session shape and outcome, performance and frame-rate statistics, counts of validation states, coarse error codes, and a wall-clock timestamp — and (b) full device-readiness diagnostic reports, which include device information, requested metrics, camera/input details, timing/resource checks, and free-text diagnostic messages that Presage notes can include paths or error text and should not be treated as guaranteed free of identifying information. Presage states that telemetry never attaches raw video frames or measured physiological values; that reports travel over the SDK's authenticated, TLS-pinned channel and can be correlated with the SDK account (i.e., they are not anonymous); that size caps apply (256 KiB for summaries, 128 KiB for readiness reports); and that no automatic expiration is currently configured for the stored reports. Because of that, Pip sets
enableTelemetry: false— with Pip as shipped, the SDK sends no session summaries and no readiness reports. An operator can opt in withSMARTSPECTRA_TELEMETRY=true, and the app's consent screen discloses the current setting before the user agrees to anything. -
API key. The key lives only in a local
.envfile (gitignored, never committed) and is used solely inside the SDK's own authenticated requests. - AI services. Pip uses no LLM or third-party AI service. The only models involved are the SDK's on-device ones; everything else is a few hundred lines of pet logic.
- Wellness, not medicine. SmartSpectra metrics are for general wellness and informational purposes only and are not cleared for medical diagnosis or treatment. Pip says the same thing in its UI and footer, and never turns a number into advice.
Top comments (0)