On-Device Video Captions with SpeechAnalyzer on iOS 26: What Worked and What Didn't

We build FoxTake, an iPhone app for recording talking-head videos with a teleprompter. After you record, it cuts retakes and burns captions into the video. Version 2.0 targets iOS 26, so captions use Apple's new SpeechAnalyzer / SpeechTranscriber APIs and are transcribed on the phone.

This post covers what we learned getting from a recorded .mov to captions that look right on TikTok, Reels and Shorts. Some of it is in the docs. Most of it we found by testing.

Why on-device

Our users mostly record short clips: one take of 30 seconds to 3 minutes. For that size, on-device transcription is quick, works without a network connection, and costs nothing per minute. We tested the iOS 26 transcriber against Whisper small on a 46.8-second outdoor Mandarin clip.

• Speed. On an iPhone with an A14 chip, SpeechTranscriber finished in 0.86 s, about 54× real time.
• Accuracy. Whisper small made one word error and dropped two filler particles. The on-device transcriber got all three right.

That is one clip, not a benchmark, but it was enough for us to make on-device the default.

Setting up the transcriber

The module setup is small:

let transcriber = SpeechTranscriber(
    locale: locale,
    transcriptionOptions: [],
    reportingOptions: [],
    attributeOptions: [.audioTimeRange, .transcriptionConfidence]
)

.audioTimeRange is the important one. Every run in the result's AttributedString carries its start and end time. That gives you word-level timing (character-level for Chinese and Japanese), which is what captions need.

Things we tried that did not help:

• Passing the script as contextualStrings. Our users have a script, so biasing the recognizer toward it seemed obvious. In our tests the output was identical with and without it, so we dropped it.
• DictationTranscriber for punctuation. It returns punctuated text, which looks attractive for captions. On the same clip it was slower (2.74 s vs 0.86 s) and made more mistakes: it dropped words and swapped one for a similar-sounding word. We take punctuation from the user's script instead (more on that below).
• The old SFSpeechRecognizer with on-device forced. For our Chinese test it returned empty text.

Language assets: online once, then offline

Recognition is offline, but the language model may need a one-time download. Check before you start, and show progress:

let module = makeTranscriber(locale)
if await AssetInventory.status(forModules: [module]) != .installed,
   let request = try await AssetInventory.assetInstallationRequest(supporting: [module]) {
    let observation = request.progress.observe(\.fractionCompleted) { p, _ in
        progress(p.fractionCompleted)
    }
    defer { observation.invalidate() }
    try await request.downloadAndInstall()
}

Use SpeechTranscriber.supportedLocale(equivalentTo:) to map the user's language to a supported locale. Don't compare identifiers by hand: zh-Hant-TW vs zh-TW will catch you.

Feeding a recorded file, not the microphone

Most examples stream microphone audio. We transcribe a finished recording, so we read the audio track with AVAssetReader and push buffers into the analyzer ourselves:

1. Ask for the format the analyzer wants: SpeechAnalyzer.bestAvailableAudioFormat(compatibleWith: [transcriber]).
2. Decode the audio track to mono Float32 PCM at that sample rate.
3. If the analyzer's format still differs, for example Int16, convert each buffer with AVAudioConverter.
4. Yield each buffer into an AsyncStream<AnalyzerInput> that you passed to analyzer.start(inputSequence:).
5. When the reader is done, call cont.finish() and then try await analyzer.finalizeAndFinishThroughEndOfInput().

Start collecting transcriber.results in a separate Task before you feed any audio, or you can miss results. Each result is roughly one phrase. We keep them as segments of (text, start, end, confidence) and cache them next to the recording, so re-rendering a video never transcribes again.

Timing: phrase starts come in early

The raw start time of each phrase was consistently early, by 0.49 s on average in our clip. Only 2 of 9 phrases were within 0.2 s. A caption that appears half a second before the speaker opens their mouth looks wrong.

The cause: the per-word time ranges are back to back, with no gaps. After a pause, the first word absorbs the pause into its own range. In one case a single character was stamped 6.12–6.90 s, but the speaker started at 6.79 s.

The fix is to distrust long first words. If a word lasts more than 0.35 s, search inside its range for where the audio energy actually rises, and start the caption there. Even a crude rule of start = end − 0.2 s brought the average error to 0.13 s, with 8 of 9 phrases within 0.2 s. If your captions feel "ahead", check this before anything else.

Tokenizing for captions: don't lose the %

Caption lines are built from tokens: per character for Chinese, Japanese and Korean, per word for everything else. Two bugs we shipped and then fixed:

• Symbols attached to numbers were dropped. Our first tokenizer kept only letters and digits, so 5% became 5 and $20 became 20. The fix is a small allow-list of symbols that belong to a word (% ° $ € £ ¥ + & @ # /). We also keep ., , and : when they sit between two digits, so 3.5, 1,000 and 10:30 survive.
• Traditional vs Simplified Chinese. A user's script may be in Traditional characters while the recognizer outputs Simplified, or the other way round. For comparison we normalize each character with StringTransform("Hant-Hans") (cached, since it is called per character). For display we keep the script's form, so a Traditional script gets Traditional captions.

Punctuation from the script also drives line breaks. A full stop always ends a caption. A comma is a preferred place to break. In Chinese scripts people often use a space instead of a comma, so we treat a space between CJK characters as a soft break too.

Rendering: fonts, line breaks, safe areas

We draw captions with Core Text into a CIImage and composite them over the frames during export. Three details made a visible difference:

Pick the font by language. PingFang SC is a fine default for Chinese. For Japanese, though, it uses Chinese glyph shapes, which Japanese viewers notice right away. We switch to Hiragino Sans for ja, Apple SD Gothic Neo for ko, and PingFang TC or HK for Traditional Chinese. Everything else uses the system font.

Break lines at word boundaries, even without spaces. For Latin text you break at spaces. Chinese and Japanese have no spaces, so we use NLTokenizer(unit: .word) with the dominant language from NLLanguageRecognizer to find word boundaries. Without it you get lines split in the middle of a word. If a line doesn't fit, we try every candidate break and pick the one that makes the two lines most equal in width. If two lines still don't fit, we shrink the font instead of adding a third line.

Don't use kCTStrokeWidth for outlines. It strokes along the glyph contours, and where strokes cross you get visible artifacts. We render the text once, dilate it with CIMorphologyMaximum, turn the result black with CIColorMatrix, and put the original text on top. The result is a clean, even outline.

Place captions per platform. Our first version put captions 22% from the bottom, right where TikTok, Reels and Shorts overlay their own captions and buttons. Now each export target has a preset. The default sits about 60% from the top, inside a horizontal safe area of 120–888 px on a 1080×1920 frame, which is the intersection of the three platforms.

What we'd tell someone starting today

• Request .audioTimeRange and build everything on the timed runs.
• Measure on your own recordings before adding features: contextualStrings and DictationTranscriber looked useful and weren't, for us.
• Check phrase start times against the audio. They may be early.
• Test captions with real numbers, currencies and mixed scripts, not just plain sentences.
• Design for the platform UI, not for the empty video frame.

If you want to see the result, FoxTake is on the App Store for iPhone with iOS 26. There is a guide on reviewing and correcting captions on our site. Questions about SpeechAnalyzer are welcome in the comments. We're still learning this API too.

Comments

[Odd Eggs Studio]

Interesting that contextualStrings changed nothing even with the exact script, I'd have bet on that one. Have you tried any Vietnamese clips yet, and does supportedLocale(equivalentTo:) map vi-VN at all?