Most AI today is blind to human context. Models classify images, transcribe speech, and generate text—but they don’t perceive. They miss the silent cues: hesitation in voice, micro-expressions, posture shifts. That’s the gap I’m hacking on as a solo founder. At EmoPulse (emo.city), we’re building a real-time perception layer that fuses multimodal biometrics—audio prosody, facial dynamics, galvanic skin response—to infer cognitive and emotional states beneath surface behavior. This isn’t sentiment analysis on text. This is low-latency signal processing meeting transformer-based sequence modeling to close the loop between human expression and machine awareness.
The stack starts at the edge. On-device preprocessing (in C++ with LLVM-compiled kernels) reduces raw video and audio streams into privacy-preserving embeddings before any data leaves the device. We use MediaPipe for facial landmarks and a custom CNN-RNN hybrid to extract temporal affective features—think eyebrow raises over 200ms windows, not static frames. Audio goes through a learned filter bank (think learnable Mel-spectrogram layers in PyTorch) trained end-to-end on paralinguistic tasks. These streams are fused via cross-modal attention in a lightweight transformer (4 layers, 256-dim), optimized using TorchScript and quantized for <50ms inference on mid-tier smartphones. The output? A low-dimensional state vector—focus, confusion, engagement—that apps can react to in real time.
We’re not building another emotion API. We’re building the perception infrastructure for AI to finally sense human context—without compromising privacy or latency. But here’s the hard part: ground truth. How do you label "cognitive load" at scale? We’re experimenting with implicit signals (mouse dynamics, speech pause frequency) as proxy labels, but it’s messy. If you’re working on sensor fusion, on-device ML, or subjective state modeling—how are you validating what you can’t directly observe?
Top comments (0)