DEV Community

Cover image for ClipPulse: Contactless Biometric OBS Overlay & Deterministic VOD Auto-Clipper
Devansh
Devansh

Posted on Edited on

ClipPulse: Contactless Biometric OBS Overlay & Deterministic VOD Auto-Clipper

Presage SmartSpectra SDK Challenge Submission

This is a submission for the Presage SmartSpectra SDK Challenge

Transforming streamer pulse and breathing signals into transparent 60fps OBS HUDs and automated highlight reels — zero wearables, zero facecam required, zero video persistence.

Demo

studio-preview

export-dialog-preview

gate-preview

Inspiration

Every gamer, speedrunner, and VTuber knows the feeling: your heart hammers in your chest during a 1v4 clutch, a grueling soulslike boss encounter, or a world-record pace run. That physiological reaction is what makes live streaming electrifying.

Yet sharing that excitement with an audience today is broken:

  • Wearable friction: Streamers must wear uncomfortable chest straps or smartwatches that disconnect via Bluetooth or run out of battery mid-broadcast.
  • Privacy exposure: Faceless creators and VTubers are forced to choose between revealing their real facial features or sacrificing biometric audience engagement entirely.
  • Post-stream editing fatigue: After an 8-hour broadcast, finding the peak reaction moments means scrubbing through hundreds of gigabytes of VOD footage for hours.

We asked: What if an everyday webcam could measure pulse and breathing cadence completely contactlessly, project a transparent 60fps HUD onto OBS, and automatically cut highlights into industry-standard editing timelines the moment peak reactions occur?

That vision is ClipPulse.

What It Does

ClipPulse is a local-first Electron broadcast companion powered by the Presage SmartSpectra SDK:

  1. Contactless Biometric Capture: Uses remote photoplethysmography (rPPG) to track pulse rate, pulse delta, and breathing cadence directly from optical camera reflections. No wearables, chest straps, or smartwatches required.
  2. Transparent 60fps OBS Overlay: Serves a dynamic, hardware-accelerated glass HUD at http://127.0.0.1:4789/overlay via local loopback. Displays real-time pulse waveforms, breathing rings, confidence indicators, and animated AUTO-CLIPPED toast alerts.
  3. Deterministic Hype Detector: Replaces probabilistic cloud AI with a pure mathematical signal pipeline:
    • 120-second rolling baseline that adapts to the creator's natural resting state.
    • Dynamic outlier rejection (max(6 BPM, 2.5σ)) so temporary adrenaline surges never contaminate the baseline.
    • Composite 0–100 Hype Score combining pulse delta (up to 62 pts), rate-of-rise slope (up to 18 pts), breathing cadence deviation (up to 12 pts), and sharp-inhale irregularity (up to 8 pts).
    • Continuous 2.5-second sustain requirement and 30-second cooldown gate to eliminate false triggers.
  4. Instant OBS Studio Integration: Connects over OBS WebSocket v5 to fire SaveReplayBuffer and CreateRecordChapter the instant a peak moment is captured.
  5. Universal Video Editor Conformance:
    • CMX 3600 EDL: Generates frame-accurate Edit Decision Lists that import directly into DaVinci Resolve and Adobe Premiere Pro, automatically laying cut clips end-to-end on the record timeline.
    • YouTube & Twitch Chapters: Outputs timestamped chapter lists starting at 00:00.
    • Session Telemetry JSON: Emits structured session analytics adhering to clippulse.session.v1.
    • Conformed Highlight Video Reel: Bundles a ready-to-watch 30-second highlight reel (clippulse-highlights.mp4) cut and assembled by FFmpeg from peak moments.
  6. Strict Privacy Architecture:
    • Faceless preview enabled by default (heavy 42px silhouette blur) with a 5-second unblur alignment hold.
    • Zero raw video persistence — frames remain in ephemeral volatile memory and are never written to disk or sent across the network.
    • Instant hardware KILL CAMERA switch that immediately drops camera media tracks, revokes consent, zeroes telemetry buffers, and returns to the gate screen.
  7. Complete SIMULATOR Mode: Built-in 10 Hz synthetic signal generator with interactive triggers (Boss-Fight Spike +18 BPM, Gasp / Sharp Inhale, Lean Away / Validation Lost, and Skip Warmup), allowing anyone to demo the entire pipeline without a camera or API key.

How It Responds to Human Signals

ClipPulse interacts with real-time physiological signals captured via the Presage SmartSpectra SDK through a three-stage mathematical pipeline:

respond flow

1. Ingestion & Optical Tracking

  • Standard webcam reflections are captured via @smartspectra/node-sdk inside the renderer process.
  • The SDK isolates micro-vascular pulse signals and thoracic/diaphragmatic respiratory patterns at 10 Hz.
  • Monotonic per-stream timestamps are preserved, and telemetry forwarding is disabled (enableTelemetry: false).

2. Deterministic Mathematical Scoring

Instead of relying on opaque cloud-based models, ClipPulse uses a transparent, explainable mathematical detection system to analyze real-time telemetry.

- Adaptive Baseline (120-Second Rolling Window): Continuously tracks resting heart rate and respiratory cadence using a rolling two-minute data buffer.

- Dynamic Outlier Filtering: Any reading exceeding the maximum of 6 BPM or 2.5 standard deviations is evaluated for highlight detection but excluded from baseline updates. This prevents extended boss fights or intense gameplay from artificially increasing resting metrics.

- Composite Hype Score (0 to 100): Calculates an overall excitement score using five weighted signals:

  • Pulse Change: 0 to 62 points
  • Heart Rate Acceleration: 0 to 18 points
  • Breathing Pattern Change: 0 to 12 points
  • Sharp Inhalation Detection: 0 to 8 points
  • Facial Expression Recognition: 0 to 12 points

- 2.5-Second Continuous Threshold: The combined score must remain at or above 70 for at least 2.5 continuous seconds before a highlight is confirmed. This filters out temporary spikes caused by talking, swallowing, or other brief movements.

- 30-Second Cooldown: Enforces a 30-second waiting period between automatic clip triggers to prevent redundant recordings during extended high-action sequences.

3. Immediate Broadcast and Editing Triggers

Once a high-intensity moment is confirmed, ClipPulse automatically initiates the following actions:

1. OBS HUD Overlay: The local overlay at http://127.0.0.1:4789/overlay displays real-time visual feedback. The 60 FPS waveform responds with high-contrast amplitude animations, the breathing cadence indicator expands, and an animated AUTO-CLIPPED notification appears on the stream.

2. OBS WebSocket Version 5 Integration: Sends the Save Replay Buffer command to immediately save the previous 30 seconds of high-quality broadcast footage to disk. It also inserts a recording chapter marker into the recording timeline.

3. Automatic Clip Generation: Calculates the optimal clip duration using 20 seconds before the detected trigger and 10 seconds after it. The resulting clip information is automatically added to:

  • CMX 3600 Edit Decision List
  • YouTube video chapters
  • Session telemetry JSON exports

This creates a fully automated highlight workflow that detects exciting gameplay moments, provides immediate broadcast feedback, and organizes recorded footage for post-stream editing.

What the YouTuber Walks Away With:

Output File YouTube Format What It Achieves
clippulse-chapters.txt YouTube Description Automatic chapter scrub bar on YouTube with biometric tags.
clippulse-highlights.mp4 YouTube Short / Montage Instant, zero-editing highlight video ready for upload.
clippulse-highlights.edl Premiere / DaVinci Timeline Auto-slices the 3-hour raw stream down to just the clutch moments.
clippulse-session.json Title & Thumbnail Data Provides exact peak BPM numbers for packaging and click-through rate.

What I Learned

  • How contactless remote photoplethysmography (rPPG) extracts micro-vascular blood volume pulse waveforms from standard webcam optical sensors.
  • How CMX 3600 Edit Decision Lists structure source-in, source-out, record-in, and record-out timecodes for professional post-production workflows.
  • How to tune high-framerate HTML5 2D canvas smoothing algorithms to render responsive 60fps telemetry overlays inside OBS Browser Sources.

What's Next for ClipPulse

  • Twitch & YouTube Chat Interactivity: Twitch Channel Points integration enabling viewers to wager points on whether the streamer's pulse will spike above threshold during high-stakes boss attempts.
  • Direct NLE Plugins: A DaVinci Resolve Python Studio script and Adobe Premiere Pro CEP extension to import ClipPulse highlights in a single click directly inside the editing timeline.
  • Multi-Creator Squad HUD: Composite overlay for esports teams and collaborative streams, displaying synchronized pulse deltas across multiple players in real time.

Prize Categories

Best Thoughtful Real-World Use

ClipPulse addresses three fundamental real-world barriers facing creators:

  1. Zero Wearable Friction: Replaces intrusive smartwatches and chest straps with optical sensing that never runs out of battery or disconnects during broadcasts.
  2. Universal Post-Production Conformance: Outputs CMX 3600 EDLs that conform in DaVinci Resolve and Adobe Premiere Pro in 3 clicks, cutting multi-hour broadcasts into an assembled highlight reel without human scrubbing.
  3. Privacy by Design for Faceless Creators: Built specifically for VTubers and anonymous creators who refuse to stream with a facecam. Default 42px silhouette blur, local-only loopback sockets, zero video persistence on disk, and instant hardware killswitch ensure zero facial features or raw footage ever escape the creator's machine.

Most Creative or Delightful Use

  • Transforms invisible physiological reactions into high-energy live entertainment through a hardware-accelerated 60fps glass OBS HUD.
  • Features complete zero-hardware SIMULATOR Mode with interactive triggers (Boss-Fight Spike, Gasp / Sharp Inhale, Lean Away / Tracking Lost) allowing judges to demo the entire live biometric broadcast loop without a camera or API key.

Top comments (1)

Collapse
 
devanshdubey profile image
Devansh •

I would love your feedback on it :)