This is a submission for the Presage SmartSpectra SDK Challenge
What I Built
I built Fluency Lab, a multimodal speech exploration experience designed primarily for people who stutter.
I stutter myself, so I've spent quite a bit of time looking for tools that could make speaking a little easier. I've experimented with offline speech recognition, smaller language models, and running AI models directly on devices.
I've written about some of those experiments before:
- Offline AI on Mobile: Tackling Whisper, LLMs, and Size Roadblocks
- My Adventures with Client-Side AI Models
- Has Anyone Actually Shipped a Free Offline Mobile LLM?
Each experiment taught me something, but I kept coming back to a slightly different question:
What else could we understand about the experience of speaking, beyond simply transcribing words?
Most speech tools focus on what someone said. That's useful, but it doesn't necessarily tell us much about what was happening during the speaking experience itself.
For someone who stutters, two conversations can look very similar in a transcript but feel completely different.
That's where Fluency Lab comes in.
The prototype brings together different kinds of observations during a speaking session, with important limits:
- Speech: after a session, participants can optionally reflect on a speaking moment—whether it felt expected, spontaneous, or uncertain—and add a note. The reflection is self-reported and is not assigned an event time. Android and the Mac desktop app also offer optional experimental acoustic summaries; they are not transcription, word or speaking-rate counts, or stutter detection. SmartSpectra's talking-activity output is not stutter detection either.
Facial observations: participants can choose supported SmartSpectra signals such as blink, talking activity, landmarks, and expression-model scores. A local Mac session received blink and talking events. These outputs do not measure jaw locking, gaze, emotion, or facial EMG; jaw tension or locking can be added as a self-reported note.
Physiological signals: the integration requests supported pulse, HRV, breathing, and EDA Proxy outputs. The app saves outputs it receives and presents them in the post-session recap, subject to SDK support and account entitlement; raw readings are not streamed in the active session view.
SmartSpectra is the core technical integration
SmartSpectra is not a decorative mention or a source of mock values in Fluency Lab. Camera-derived observations come from SmartSpectra SDK outputs received by the platform adapter. The Android path uses Presage's native SDK; the macOS Electron path uses @smartspectra/node-sdk. In the Camera choice, participants can select individual supported signals. Each adapter requests the selected metrics and maps returned outputs into local observations with their SDK method, timestamp, value/unit, stable flag, and confidence when supplied. These observations are saved for the post-session recap rather than shown as a live signal feed. SDK telemetry is disabled; the app does not request AI Insights or add a speech classifier on top.
In a local Mac session, Fluency Lab received SmartSpectra blink and talking-activity events and showed the SDK's camera guidance. Talking activity is not stutter detection, and the app keeps it separate from a person's own speech notes.
With Camera enabled, a live preview appears during setup so participants can frame their face and upper chest. The preview hides when guided questions begin.
The post-session recap gives a compact overview of received SmartSpectra signal categories and keeps the detailed observation timeline optional and collapsed by default. Each saved observation keeps its source, method, timestamp, value, unit, and SDK flags. The stable flag and confidence have metric-specific meanings; neither is an independent guarantee of accuracy. Fluency Lab does not compare pulse samples with talking intervals: a sample timestamp does not describe the full rolling analysis window.
The Trends page summarizes SDK-stable pulse, HRV, and breathing-rate samples only when their documented analysis windows and elapsed times can be verified: 12 seconds for pulse, 30 seconds for breathing, and 60 seconds for HRV. It calculates a median for each session, then a median across comparable sessions. Sessions must match the scenario, personal context, SDK method, and unit; at least three sessions are needed. Confidence is shown separately, and missing values are left out, not invented. These are personal reference points—not medical baselines, predictions, accuracy scores, or significance tests.
These are personal, health-related observations, so I am not publishing a dataset of exact personal session readings. The screenshots illustrate the experience; anyone who wants to explore their own measurements can run Fluency Lab locally. The app stores received observations and derived audio summaries on that device, and does not save raw camera video or microphone audio.
Imagine practising an everyday situation, such as ordering a coffee.
After the session, you might reflect on a repetition or another speaking moment and separately review the measurements the SDK returned. Pulse and breathing values can summarize rolling windows rather than the exact instant of that moment, so nearby timestamps do not show that one caused the other.
That doesn't mean one caused the other. It doesn't automatically mean you were anxious either. But it gives you something interesting to explore.
The goal isn't to give someone a fluency score or tell them how they should speak. It's to help people explore their own experiences.
Speaking scenarios
The experience is designed around familiar situations rather than clinical exercises.
Some examples include:
- Ordering coffee or food.
- Having a conversation with a friend.
- Meeting someone new.
- Participating in a team meeting.
- Attending a job interview.
- Giving a presentation.
Each scenario has a bank of 100 distinct prompts. A session uses a stable, session-specific selection of five questions, one minute per question, within a five-minute active-practice limit. Participants can move on or finish early, pause, and resume the same session; pausing stops the active clock and capture while preserving prompt progress and saved observations. Camera setup is optional and happens before the questions, so the preview is not shown during guided practice. Audio summaries are also optional and can be enabled independently.
Looking beyond individual sessions
Across repeated sessions, Fluency Lab compares only SDK-stable scalar samples with a complete, verifiable analysis window, from matching scenarios, personal contexts, SDK methods, and units. It uses a median for each session and a median of those session medians, with at least three completed sessions. The result is a personal reference point, not a clinical baseline or prediction; missing values are not filled in.
Each session can differ with the situation, surroundings, and how someone feels. Rather than compare people against a universal fluency score, Fluency Lab lets each person explore their own notes and available measurements over time—for example, across several coffee-ordering practices.
Demo
To try the prototype, follow the local run instructions below.
Running locally
- Clone the project and open its root directory. Install Node.js 24 (
nvm useselects the version in.nvmrc; Node.js 22.13+ is the declared minimum), then install dependencies withnpm install. - Add a development
PRESAGE_API_KEYto an untracked.envfile, or configure Android OAuth as described in Presage's Android setup guide. Do not commit credentials. SmartSpectra authentication and entitlement are required for live SDK processing. - Choose a target:
-
Android: install Android Studio/Android SDK 37+ and Java 21+, connect an Android device (recommended for camera/signal verification), then run
npm run android. This builds/launches the native development app; Expo Go does not include the required native modules. -
macOS desktop: on an Apple Silicon Mac, run
npm run desktop. This starts the Expo web renderer inside Electron and connects the Node SDK; allow camera access when macOS prompts. -
Browser preview: run
npm run web. This is UI preview only, not a SmartSpectra session.
-
Android: install Android Studio/Android SDK 37+ and Java 21+, connect an Android device (recommended for camera/signal verification), then run
- Run
npm run typecheckandnpm testto check the TypeScript project and app tests.
After you finish, the recap shows a compact overview; its detailed timeline is optional and collapsed by default. Trends is also available for comparable sessions. Exact readings stay on the device; see the privacy note below.
Code
The repository includes:
- Project setup and installation instructions.
- Architecture overview.
- SmartSpectra SDK integration details.
- Environment variable documentation.
- Testing instructions.
- Privacy and data-handling information.
- Known limitations and future work.
Current platforms
Fluency Lab is currently a local-development prototype, not a released or hosted app:
- Android mobile: the primary native target. It uses the Android SmartSpectra module and native audio capture. Camera signals and audio are optional, with separate in-app choices and Android camera/microphone permissions when enabled; live SmartSpectra processing also requires valid authentication and account entitlement.
- Desktop: the app targets Electron on Apple Silicon macOS, using Presage's Node SDK and macOS camera permission. A local Mac session received talking-activity and blink events. Presage's Node SDK also documents native packages for Linux x64/ARM64 and Windows x64; this guide covers the Mac app.
- Web browser: available as a preview of the app UI only. It does not connect to SmartSpectra; use the Electron app for Mac camera processing.
- iOS: not implemented.
The app's current value is a supportive scenario-based session, optional retrospective self-reflection, and local views of measurements actually received from SmartSpectra. It does not automatically detect stuttering, jaw locking, speaking rate, or emotion.
The intention is to keep the implementation understandable and reproducible, rather than introduce a lot of infrastructure that isn't needed for the experience.
How It Responds to Human Signals
This is the part I'm particularly interested in exploring with Presage SmartSpectra.
Instead of treating speech as just text, Fluency Lab looks at how different observations can be brought together during a speaking session.
Physiological signals
Participants can select individual supported SmartSpectra cardio, breathing, face, and EDA Proxy signals, subject to API-key entitlement and platform availability. The Trends page summarizes only supported scalar pulse, HRV, and breathing-rate measurements that the SDK marks stable, and only after their documented analysis windows are complete and elapsed time is verifiable: 12 seconds for pulse, 30 seconds for breathing, and 60 seconds for HRV.
Potential observations include:
- Pulse or heart rate.
- Heart rate variability.
- Breathing rate.
- Breathing waveform.
Fluency Lab keeps the SDK's stable flag and confidence as-is and does not present either as a general quality or health score. It saves measurements the SDK actually sends for the post-session recap; requesting a metric does not create a reading. Motion and speech can affect measurements, and confidence is not an accuracy guarantee.
Facial observations
The integration requests SmartSpectra blink and talking activity, face landmarks, and expression-model scores. Blink and talking events appeared in a local Mac session. The app does not use these outputs to measure jaw movement, jaw locking, gaze, emotion, or facial EMG. People can add jaw tension or locking as a self-reported note, which the recap labels separately from SDK measurements.
I don't want to turn a facial output into an assumption about someone's emotional state. Expression-model scores are classifications, not a report of how someone feels.
Speech observations
Speech reflection is separate from SmartSpectra measurements. After a session, participants can optionally note whether a speaking moment felt expected, spontaneous, or uncertain. This is self-reported, not detected, and is not assigned an event time. On Android, a separate optional audio module processes short 16 kHz mono frames in memory and stores only derived acoustic activity, average microphone level, rough pitch estimates, and low-energy intervals. It does not save raw audio or transcripts, calculate speaking rate/rhythm, or detect stuttering. The installed SmartSpectra SDK exposes talking activity but does not expose a stutter, repetition, prolongation, block, or jaw-lock metric.
SmartSpectra provides physiological and facial observations; it does not independently detect stuttering. No custom acoustic classifier or fabricated speech statistic is presented as an SDK result.
Bringing everything together
The recap does not compare pulse samples with talking-state intervals. SDK timestamps identify sample times, while rate and waveform measurements summarize rolling analysis windows; they cannot establish an exact speaking-event time or causal relationship. The optional reflection remains a separate, untimed self-report.
For example, after a coffee-ordering scenario, someone might reflect on:
- A repetition in their speech.
- A pause before continuing.
- Whether a speaking moment felt expected or spontaneous.
They can review available pulse and breathing summaries separately, without treating nearby timestamps as proof that the readings correspond to a particular speech moment.
The application should make it clear what was measured, what was inferred and what remains uncertain.
I also want to be careful about signal quality. Physiological measurements aren't always reliable in every situation, and speaking itself can affect the quality of some measurements. Missing or unreliable data should be shown as such rather than treated as a meaningful result.
Technical integration
The repository includes Presage's Android SDK 3.4.0 bridge and a macOS Electron integration using @smartspectra/node-sdk 3.5.x. The Android app uses its native camera view; the Mac path uses the official Node SDK in Electron. Both request only the camera signals selected by the participant, disable SDK telemetry, map selected outputs to local observation records, and stop processing when the user stops or the app lifecycle requires it. The Electron desktop path is not a regular browser feature: Chrome/Web preview is preview-only.
The SmartSpectra adapter writes received measurements to local SQLite as observations; it does not persist camera frames or raw video. SDK outputs retain their source/method, timestamp and clock metadata, value/unit, stable flag, and confidence when supplied. Signal values are saved during capture but shown in the recap rather than streamed as a live feed. Optional retrospective reflections are stored separately as untimed user annotations, so they cannot be mistaken for sensor detections. Trends read eligible SDK observations only after their analysis windows can be verified; separate local audio summaries are experimental acoustic estimates, not speech-event statistics.
The repository also includes an Elixir/Phoenix backend foundation for validating observation ingestion and exploring time-based correlation. It is not connected to the app: real sessions remain in local SQLite and are not uploaded. To exercise the Elixir path without using personal data, cd backend && mix demo runs a fixed synthetic fixture through ingestion and correlation inside a transaction that is rolled back. The HTTP router still returns 404, so this is a local development demo—not a sync service.
The Android app includes a native optional local audio-capture module. The Mac desktop app uses a separate Electron-only Web Audio adapter; ordinary browser sessions never request microphone access. One concise Camera & audio choice sheet lets participants opt into camera signals and audio independently, and select individual camera signals. The app stores separate consent records for those capabilities; Android and macOS can also request operating-system camera and microphone permissions separately. Capture stops when the user stops or pauses, or when the app is backgrounded. Short audio frames are processed in memory; only acoustic-activity ratios, average microphone levels, rough-pitch estimates, and low-energy intervals are stored locally. Raw audio files, transcripts, and camera video are not saved. Physical-device permission prompts, signal delivery, and pause/resume behavior still need verification on Android and Mac.
Development uses an environment-provided key, not production credential storage. SmartSpectra still needs authentication and network access. The app disables SDK telemetry and does not request AI Insights; those settings do not disable SDK authentication or other required communications.
Integration lessons: issues, fixes and what helped
Most of the work was connecting the SDK to Expo and each platform, rather than building another signal-processing system.
- Keep preview and SDK processing distinct. The browser preview is for the interface. Android uses the native SDK module; the Mac Electron app uses Presage's Node SDK.
- Keep optional capabilities explicit. Camera signals and audio summaries are separate opt-ins in one concise choice sheet, with individual camera-signal checkboxes. Android native audio and Mac Electron audio also use their respective operating-system microphone permissions; standard browsers do not capture microphone audio.
- Requesting a field does not guarantee a sample. The Android bridge can parse optional fields that the SDK may not provide. Presage's metric guides list breathing amplitude, apnea detection, respiratory line length, baseline, and inhale/exhale ratio as unavailable. Fluency Lab stores received SDK outputs only; it does not fill missing values. The SDK does not provide stutter, speech-rate, or jaw-lock detection.
The most useful references were Presage's Android SDK overview, Android metrics guide, Node SDK API reference, Node metrics guide, device-readiness guide, measurement-quality guide, Electron quickstart, and Expo's local-module/autolinking guidance. The SDK docs explained platform setup, supported metrics, quality flags, and Electron's main/preload/renderer design. Gradle's module list and manifest report confirmed which Android modules and permissions made it into the build.
A possible direction for future research
In the future, it may be worth researching how speech, facial movement, and physiological signals relate to neurological health. For example, Parkinson's can affect speech and movement. Fluency Lab does not detect or predict Parkinson's or any other condition; any such research would need clinical oversight. The current focus is helping people who stutter explore their own speaking experiences.
Prize Categories
I'm entering both categories.
Most Creative or Delightful Use
Many speech tools focus on words and transcripts. I wanted to explore how a person's own post-session reflections, facial observations, and physiological measurements could help them reflect on their speaking experience. The warm, storybook-style scenarios are meant to make practice feel welcoming, not like a test.
Best Thoughtful Real-World Use
Stuttering affects people differently. Fluency Lab keeps optional retrospective reflection separate from available facial and physiological signals so people can explore familiar situations and their own patterns—not chase a universal score. A session is still useful when a signal is missing.
Consent and Data Handling
Because this application works with camera and physiological measurements, privacy needs to be part of the design from the beginning.
The app presents camera and audio as independent opt-ins in one session-specific choice sheet; camera signals can be selected individually. Android and macOS also request the corresponding operating-system permissions separately. People can stop capture or revoke either choice at any time; already saved observations remain in that session's local database. Fluency Lab does not save raw microphone audio or camera video. Short audio frames are processed in memory; derived acoustic summaries, selected SDK observations, and self-reported notes are stored in local SQLite. The app has no session-upload route. SmartSpectra still uses Presage SDK authentication and network services; disabled telemetry and unrequested AI Insights do not mean the SDK makes no network communication. Exact readings are personal health-related data, so I am not publishing a dataset of them here. Readers can run the app locally to collect and explore their own session data.
SmartSpectra still needs network access for authentication and SDK processing. SDK telemetry is disabled, and the app does not request AI Insights; neither setting means that all SDK network communication is disabled. Presage's documentation describes camera processing on-device and explains that Insights send a prompt and analysis for processing, including by a third-party AI provider.
Before release, the privacy notice should explain camera use, measurements and network flows, local storage and backups, retention and deletion, SDK telemetry, and any third-party processing. Fluency Lab is for personal exploration, not a medical device; it does not diagnose, predict, or recommend treatment.
Conclusion and what's next
I know from my own stutter that a transcript can capture the words without capturing the work it took to say them. Fluency Lab began with a simple hope: make room for that fuller experience, without grading or trying to fix the person.
SmartSpectra gives the project another lens: camera-derived observations to explore alongside a person's own notes. There is much more to explore. Next, I want to go deeper into multimodal patterns across everyday situations and over time, while shaping the experience with people who stutter. The goal is to learn what feels genuinely helpful, not to turn someone's life into a score.
These observations are personal health-related data, so I won't publish exact readings. If you'd like to explore your own, run Fluency Lab locally; your session observations are stored on your device.












Top comments (3)
Keeping your own notes and the SDK readings in separate records, so a note can never pass for a detection, is the right call, and so is refusing to fill gaps.
One thing in the measurement-quality guide you linked changes how I'd read the talking comparison. It defines
stableas the face and landmark tracking being reliable frame to frame, which isn't the same as the reading being right. It also says to avoid talking during capture, that speech moves the chest in ways that don't reflect breathing, and that motion can give readings with "deceptively high confidence". So a stable sample taken mid-sentence can still be off, and a gap between talking and not talking could come from the camera rather than from you.Timing matters too. The guide gives pulse a 12-second analysis window, breathing 30 and HRV 60, so a sample stamped inside a short talking turn can be built partly from the seconds around it. Counting a sample only when its whole window sits on one side would cost samples, but your at-least-three rule already handles that. The guide also says to keep the 0 to 100 confidence value with every reading, and the post doesn't mention it.
All of that makes the quiet stretches look like your strongest data, like waiting for your turn in the coffee queue. For you, does the hard part start before you speak? If it does, that could be the comparison to build first.
(The GitHub link 404s for me, in case the repo is still private.)
Thanks for such a thoughtful read and for digging into the measurement details. The point about stable readings not necessarily meaning accurate readings is particularly important, and definitely something worth exploring further.
On your question, I think it varies depending on the situation. If it's something I'm anticipating, like a presentation I have to give, I can get blocks right from the beginning because my body is already in fight-or-flight mode before I even start speaking.
But if it's something completely spontaneous, like someone suddenly asking me for directions, I sometimes don't stutter at all. I guess my mind hasn't had time to anticipate the situation, and I haven't fallen into that usual cycle of fear and overthinking.
So I find the idea of looking at what happens before speaking particularly interesting. The anticipation itself might be a significant part of the experience, and it probably varies quite a bit from one situation to another.
Also, thanks for flagging the GitHub link! I'll check what's happening with it.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.