DEV Community

ujja
ujja Subscriber

Posted on Edited on AI-assisted

Beyond the Words: Exploring the Experience of Stuttering with AI and SmartSpectra

Presage SmartSpectra SDK Challenge Submission

This is a submission for the Presage SmartSpectra SDK Challenge

What I Built

I built Fluency Lab, a multimodal speech exploration experience designed primarily for people who stutter.

I stutter myself, so I've spent quite a bit of time looking for tools that could make speaking a little easier. I've experimented with offline speech recognition, smaller language models, and running AI models directly on devices.

I've written about some of those experiments before:

Each experiment taught me something, but I kept coming back to a slightly different question:

What else could we understand about the experience of speaking, beyond simply transcribing words?

Most speech tools focus on what someone said. That's useful, but it doesn't necessarily tell us much about what was happening during the speaking experience itself.

For someone who stutters, two conversations can look very similar in a transcript but feel completely different.

That's where Fluency Lab comes in.

The prototype brings together different kinds of observations during a speaking session, with important limits:

  • Speech: participants can record their own repetitions, prolongations, pauses, possible blocks, jaw tension/locking, or other moments. These are annotations, not sensor measurements. Android also offers separate, optional experimental acoustic summaries; they are not transcription, speaking-rate analysis, or stutter detection. SmartSpectra's talking-activity output is not stutter detection either.
  • Facial observations: the integration requests SmartSpectra blink, talking activity, landmarks, and expression-model scores. A local Mac session received blink and talking events. These outputs do not measure jaw locking, gaze, emotion, or facial EMG; jaw tension or locking can be added as a self-reported note.

  • Physiological signals: the integration requests supported pulse, HRV, breathing, and EDA Proxy outputs. The app records and displays outputs it receives, subject to SDK support and account entitlement.

SmartSpectra is the core technical integration

SmartSpectra is not a decorative mention or a source of mock values in Fluency Lab. The camera-derived observations displayed by the app come from SmartSpectra SDK outputs received by its platform adapter. The Android path uses Presage's native SDK; the macOS Electron path uses @smartspectra/node-sdk. Both configure the SDK to request its supported cardio, breathing, face, and EDA Proxy bundles, then map callbacks/metric packets into local observations with their SDK method, timestamp, value/unit, and stable flag. SDK telemetry is disabled; the app does not request AI Insights or add a speech classifier on top.

In a local Mac session, Fluency Lab received SmartSpectra blink and talking-activity events and showed the SDK's camera guidance. Talking activity is not stutter detection, and the app keeps it separate from a person's own speech notes.

The recap shows the SmartSpectra outputs received in a session. Each saved observation keeps its source, method, timestamp, value, unit, and SDK stable flag. The app does not turn that flag into a general quality score. For pulse, the recap can compare stable samples during and outside completed talking intervals; it needs at least three samples in each group and shows the counts. This is a simple comparison, not a causal or medical conclusion.

The Trends page summarizes SDK-stable pulse, HRV, and breathing-rate samples. It calculates a median for each session, then a median across comparable sessions. Sessions must match the scenario, personal context, SDK method, and unit; at least three sessions are needed. Missing values are left out, not invented. These are personal reference points—not medical baselines, predictions, or significance tests.

These are personal, health-related observations, so I am not publishing a dataset of exact personal session readings. The screenshots illustrate the experience; anyone who wants to explore their own measurements can run Fluency Lab locally. The app stores received observations and derived audio summaries on that device, and does not save raw camera video or microphone audio.

Imagine practising an everyday situation, such as ordering a coffee.

Ordering Coffee

You might notice a repetition in your speech at a particular moment. At around the same time, your pulse or breathing measurements might change.

That doesn't mean one caused the other. It doesn't automatically mean you were anxious either. But it gives you something interesting to explore.

The goal isn't to give someone a fluency score or tell them how they should speak. It's to help people explore their own experiences.

Speaking scenarios

The experience is designed around familiar situations rather than clinical exercises.

Some examples include:

  • Ordering coffee or food.
  • Having a conversation with a friend.
  • Meeting someone new.
  • Participating in a team meeting.
  • Attending a job interview.
  • Giving a presentation.

Looking beyond individual sessions

Across repeated sessions, Fluency Lab compares only SDK-stable scalar samples from matching scenarios, personal contexts, SDK methods, and units. It uses a median for each session and a median of those session medians, with at least three completed sessions. The result is a personal reference point, not a clinical baseline or prediction; missing values are not filled in.

Each session can differ with the situation, surroundings, and how someone feels. Rather than compare people against a universal fluency score, Fluency Lab lets each person explore their own notes and available measurements over time—for example, across several coffee-ordering practices.

Demo

To try the prototype, follow the local run instructions below.

Running locally

  1. Clone the project and open its root directory. Install Node.js 24 (nvm use selects the version in .nvmrc; Node.js 22.13+ is the declared minimum), then install dependencies with npm install.
  2. Add a development PRESAGE_API_KEY to an untracked .env file, or configure Android OAuth as described in Presage's Android setup guide. Do not commit credentials. SmartSpectra authentication and entitlement are required for live SDK processing.
  3. Choose a target:
    • Android: install Android Studio/Android SDK 37+ and Java 21+, connect an Android device (recommended for camera/signal verification), then run npm run android. This builds/launches the native development app; Expo Go does not include the required native modules.
    • macOS desktop: on an Apple Silicon Mac, run npm run desktop. This starts the Expo web renderer inside Electron and connects the Node SDK; allow camera access when macOS prompts.
    • Browser preview: run npm run web. This is UI preview only, not a SmartSpectra session.
  4. Run npm run typecheck and npm test to check the TypeScript project and app tests.

The session timeline and Trends page show observations received during your own run. Exact readings stay on the device; see the privacy note below.

Code

Fluency Lab on GitHub

The repository includes:

  • Project setup and installation instructions.
  • Architecture overview.
  • SmartSpectra SDK integration details.
  • Environment variable documentation.
  • Testing instructions.
  • Privacy and data-handling information.
  • Known limitations and future work.

Current platforms

Fluency Lab is currently a local-development prototype, not a released or hosted app:

  • Android mobile: the primary native target. It uses the Android SmartSpectra module and requires camera consent, Android camera permission, and valid Presage SDK authentication and account entitlement.
  • Desktop: the app targets Electron on Apple Silicon macOS, using Presage's Node SDK and macOS camera permission. A local Mac session received talking-activity and blink events. Presage's Node SDK also documents native packages for Linux x64/ARM64 and Windows x64; this guide covers the Mac app.
  • Web browser: available as a preview of the app UI only. It does not connect to SmartSpectra; use the Electron app for Mac camera processing.
  • iOS: not implemented.

The app's current value is a supportive scenario-based session, participant-entered speech observations, and local views of measurements actually received from SmartSpectra. It does not automatically detect stuttering, jaw locking, speaking rate, or emotion.

The intention is to keep the implementation understandable and reproducible, rather than introduce a lot of infrastructure that isn't needed for the experience.

How It Responds to Human Signals

This is the part I'm particularly interested in exploring with Presage SmartSpectra.

Instead of treating speech as just text, Fluency Lab looks at how different observations can be brought together during a speaking session.

Physiological signals

The application requests SmartSpectra cardio, breathing, face, and EDA Proxy bundles, subject to API-key entitlement and platform availability. The Trends page currently summarizes only scalar pulse, HRV, and breathing-rate measurements that the SDK marks stable.

Potential observations include:

  • Pulse or heart rate.
  • Heart rate variability.
  • Breathing rate.
  • Breathing waveform.

Fluency Lab keeps the SDK's stable flag as-is and does not present it as a general quality or health score. It records and displays measurements the SDK actually sends; requesting a metric does not create a reading.

Facial observations

The integration requests SmartSpectra blink and talking activity, face landmarks, and expression-model scores. Blink and talking events appeared in a local Mac session. The app does not use these outputs to measure jaw movement, jaw locking, gaze, emotion, or facial EMG. People can add jaw tension or locking as a self-reported note, which the recap labels separately from SDK measurements.

I don't want to turn a facial output into an assumption about someone's emotional state. Expression-model scores are classifications, not a report of how someone feels.

Speech observations

Speech annotations are separate from SmartSpectra measurements. Participants can annotate repetitions, prolongations, pauses, possible blocks, jaw tension/locking, and other moments. Those labels are self-reported, not detected. On Android, a separate optional audio module processes short 16 kHz mono frames in memory and stores only derived acoustic activity, average microphone level, rough pitch estimates, and low-energy intervals. It does not save raw audio or transcripts, calculate speaking rate/rhythm, or detect stuttering. The installed SmartSpectra SDK exposes talking activity but does not expose a stutter, repetition, prolongation, block, or jaw-lock metric.

SmartSpectra provides physiological and facial observations; it does not independently detect stuttering. No custom acoustic classifier or fabricated speech statistic is presented as an SDK result.

Bringing everything together

The recap can compare pulse samples with talking-state intervals using their SDK timestamps. This is a descriptive, within-session comparison—not a stutter measure, causal analysis, or comparison between devices.

For example, during a coffee-ordering scenario, someone might notice:

  • A repetition in their speech.
  • A pause before continuing.
  • A change in pulse measurements.
  • A variation in breathing measurements.

The comparison needs at least three SDK-stable pulse samples during talking intervals and three outside them. If there are too few, the app says there is not enough data.

The application should make it clear what was measured, what was inferred and what remains uncertain.

I also want to be careful about signal quality. Physiological measurements aren't always reliable in every situation, and speaking itself can affect the quality of some measurements. Missing or unreliable data should be shown as such rather than treated as a meaningful result.

Technical integration

The repository includes Presage's Android SDK 3.4.0 bridge and a macOS Electron integration using @smartspectra/node-sdk 3.5.x. The Android app uses its native camera view; the Mac path uses the official Node SDK in Electron. Both request cardio, breathing, face, and EDA Proxy metric bundles, disable SDK telemetry, map received SDK outputs to local observation records, and stop processing when the user stops or the app lifecycle requires it. The Electron desktop path is not a regular browser feature: Chrome/Web preview is preview-only.

The SmartSpectra adapter writes received measurements to local SQLite as observations; it does not persist camera frames or raw video. SDK outputs retain their source/method, timestamp and clock metadata, value/unit, and stable flag. Self-reported speech notes are stored separately as participant annotations, so they cannot be mistaken for sensor detections. Trends and the pulse/talking comparison read those stored SDK observations; the separate local audio summaries are experimental acoustic estimates, not speech-event statistics.

The repository also includes an Elixir/Phoenix backend foundation for validating observation ingestion and exploring time-based correlation. It is not connected to the app: real sessions remain in local SQLite and are not uploaded. To exercise the Elixir path without using personal data, cd backend && mix demo runs a fixed synthetic fixture through ingestion and correlation inside a transaction that is rolled back. The HTTP router still returns 404, so this is a local development demo—not a sync service.

The Android app now includes the optional local audio-capture module. Its configuration requests RECORD_AUDIO; microphone access follows separate in-app consent and Android permission, and audio processing stops when the user stops or the app moves to the background. The Android native build and merged manifest still need validation after enabling this module.

Camera and audio processing require separate in-app consent; Android also requests the corresponding operating-system permissions separately. Short audio frames are processed in memory, and only derived summaries are stored locally; raw audio and camera video are not saved. Development uses an environment-provided key, not production credential storage. SmartSpectra still needs authentication and network access. The app disables SDK telemetry and does not request AI Insights; those settings do not disable SDK authentication or other required communications.

Integration lessons: issues, fixes and what helped

Most of the work was connecting the SDK to Expo and each platform, rather than building another signal-processing system.

  • Keep preview and SDK processing distinct. The browser preview is for the interface. Android uses the native SDK module; the Mac Electron app uses Presage's Node SDK.
  • Keep optional capabilities explicit. The Android audio module is now enabled as a separate opt-in capability. Its in-app consent and operating-system microphone permission are independent of camera consent and permission.
  • Requesting a field does not guarantee a sample. The Android bridge can parse optional fields that the SDK may not provide. Presage's metric guides list breathing amplitude, apnea detection, respiratory line length, baseline, and inhale/exhale ratio as unavailable. Fluency Lab stores received SDK outputs only; it does not fill missing values. The SDK does not provide stutter, speech-rate, or jaw-lock detection.

The most useful references were Presage's Android SDK overview, Android metrics guide, Node SDK API reference, Node metrics guide, device-readiness guide, measurement-quality guide, Electron quickstart, and Expo's local-module/autolinking guidance. The SDK docs explained platform setup, supported metrics, quality flags, and Electron's main/preload/renderer design. Gradle's module list and manifest report confirmed which Android modules and permissions made it into the build.

A possible direction for future research

In the future, it may be worth researching how speech, facial movement, and physiological signals relate to neurological health. For example, Parkinson's can affect speech and movement. Fluency Lab does not detect or predict Parkinson's or any other condition; any such research would need clinical oversight. The current focus is helping people who stutter explore their own speaking experiences.

Prize Categories

I'm entering both categories.

Most Creative or Delightful Use

Many speech tools focus on words and transcripts. I wanted to explore how speech notes, facial observations, and physiological measurements could help someone reflect on their own speaking experience. The warm, storybook-style scenarios are meant to make practice feel welcoming, not like a test.

Best Thoughtful Real-World Use

Stuttering affects people differently. Fluency Lab pairs self-reported speech notes with available facial and physiological signals so people can explore familiar situations and their own patterns—not chase a universal score. A session is still useful when a signal is missing.

Consent and Data Handling

Because this application works with camera and physiological measurements, privacy needs to be part of the design from the beginning.

The app asks for separate session-specific camera and audio consent; Android also requests each operating-system permission separately. People can stop processing at any time. Fluency Lab does not save raw microphone audio or camera video. Short audio frames are processed in memory; derived summaries, received SDK observations, and self-reported notes are stored in local SQLite. The app has no session-upload route. Exact readings are personal health-related data, so I am not publishing a dataset of them here. Readers can run the app locally to collect and explore their own session data.

SmartSpectra still needs network access for authentication and SDK processing. SDK telemetry is disabled, and the app does not request AI Insights; neither setting means that all SDK network communication is disabled. Presage's documentation describes camera processing on-device and explains that Insights send a prompt and analysis for processing, including by a third-party AI provider.

Before release, the privacy notice should explain camera use, measurements and network flows, local storage and backups, retention and deletion, SDK telemetry, and any third-party processing. Fluency Lab is for personal exploration, not a medical device; it does not diagnose, predict, or recommend treatment.

Conclusion and what's next

I know from my own stutter that a transcript can capture the words without capturing the work it took to say them. Fluency Lab began with a simple hope: make room for that fuller experience, without grading or trying to fix the person.

SmartSpectra gives the project another lens: camera-derived observations to explore alongside a person's own notes. There is much more to explore. Next, I want to go deeper into multimodal patterns across everyday situations and over time, while shaping the experience with people who stutter. The goal is to learn what feels genuinely helpful, not to turn someone's life into a score.

These observations are personal health-related data, so I won't publish exact readings. If you'd like to explore your own, run Fluency Lab locally; your session observations are stored on your device.

Top comments (0)