SFSpeechRecognizer gives you a flag called requiresOnDeviceRecognition. Set it
to true and the audio is transcribed by a model that already lives on the phone.
Set it to false and the audio is streamed to Apple's speech service and the text
comes back.
That flag is the source of nearly every misleading privacy claim in the transcription
category, because it is a request, not a guarantee, and because the failure mode
when the request cannot be honoured is an empty transcript rather than an error you
can show a user.
This is a write-up of what the API actually does, what a real implementation ends up
looking like, and what you cannot truthfully promise.
The problem
You want to transcribe a recording. The requirements are ordinary: it should work
offline where possible, it should work on files longer than a minute, it should
produce word-level timestamps so you can seek to a phrase, and it should not
silently hand the user an empty string.
Why the obvious approach fails
"Just set requiresOnDeviceRecognition = true and we're private"
Three things break this.
Supported is not installed. supportsOnDeviceRecognition tells you whether a
recognizer can run locally. The on-device model for a language arrives when the
system fetches it, which in practice means after that language has been in use and
the phone has had time on Wi-Fi. A freshly reset phone can report "supported" and
still have nothing local to run.
Coverage varies by language. Apple ships on-device recognition for a subset of
the languages the framework supports overall. Widely used ones have local models on
current hardware; smaller languages and some regional variants are server-only. The
list changes between iOS releases and no app controls it.
Local models are less tolerant. Quiet, distant, accented or noisy audio that the
server model handles will sometimes come back empty from the local one — not an
error, just nothing.
So a strict on-device-only implementation is honest and frequently useless. Users do
not report "the on-device model lacks coverage for your dialect". They report "the
app is broken".
"Just hand the whole file to the recognizer"
SFSpeechURLRecognitionRequest on a long file is where you learn about the
framework's practical limits. Long recognitions become unreliable: the task can run
long, drop most of the audio, or never deliver an isFinal result at all. There is
no documented ceiling you can code against, and the failure is not an exception.
"The completion handler will tell me when it's done"
It will, until it doesn't. A recognition task that hangs simply never calls back.
Without your own timeout, one bad chunk hangs the entire transcription forever.
What actually works
1. Probe capability, but treat it as a hint
AsyncFunction("isAvailableAsync") { (localeIdentifier: String?) -> Bool in
guard let recognizer = makeRecognizer(localeIdentifier) else { return false }
if #available(iOS 13.0, *) { return recognizer.supportsOnDeviceRecognition }
return false
}
And build the recognizer defensively, because locale tags do not always match the
way callers expect (zh-Hans-CN is a classic):
private func makeRecognizer(_ localeIdentifier: String?) -> SFSpeechRecognizer? {
// Try the requested language; fall back to the device default rather than
// returning nil for a tag that simply didn't match.
if let localeIdentifier, !localeIdentifier.isEmpty,
let r = SFSpeechRecognizer(locale: Locale(identifier: localeIdentifier)) {
return r
}
return SFSpeechRecognizer()
}
2. Chunk the audio, with overlap
Short recognitions are reliable; long ones are not. So the file is exported into
~50-second segments with a few seconds of overlap so a sentence straddling a
boundary is not lost, and each segment is recognised independently. Results
accumulate, and progress is reported per chunk so a long file shows movement:
private func transcribeStreaming(url: URL, recognizer: SFSpeechRecognizer,
onDevice: Bool, recId: String) async throws
-> (String, [[String: Any]]) {
let asset = AVURLAsset(url: url)
let total = CMTimeGetSeconds(asset.duration)
guard total.isFinite && total > 0 else { return ("", []) }
let chunk = 50.0
let overlap = 4.0 // so a sentence cut at a boundary isn't lost
let step = max(1.0, chunk - overlap)
var allSegs: [[String: Any]] = []
var acceptedUntil = -1.0 // global end time already accepted, for dedupe
var start = 0.0
while start < total - 0.05 {
let end = min(start + chunk, total)
let segURL = try await exportSegment(asset: asset, start: start, end: end)
defer { try? FileManager.default.removeItem(at: segURL) }
let (piece, segs) = try await recognizeChunk(url: segURL, recognizer: recognizer,
onDevice: onDevice)
if segs.isEmpty {
// recognised text but no word timings (some recognizers): only accept it if
// it advances past what we already have, so the overlap can't duplicate
if !piece.isEmpty && end > acceptedUntil {
allSegs.append(["s": piece, "t": max(start, acceptedUntil < 0 ? start : acceptedUntil), "d": 0.0])
acceptedUntil = end
}
} else {
for s in segs {
let gStart = s.1 + start
let gEnd = gStart + s.2
if gStart >= acceptedUntil - 0.15 { // drop words already taken from the overlap
allSegs.append(["s": s.0, "t": gStart, "d": s.2])
acceptedUntil = max(acceptedUntil, gEnd)
}
}
}
let progress = min(1.0, end / total)
let final = end >= total - 0.05
self.sendEvent("onTranscription", ["id": recId, "text": joinSegments(allSegs),
"isFinal": final, "progress": progress])
if final { break }
start += step
}
return (joinSegments(allSegs), allSegs)
}
Overlap buys reliability and costs deduplication. The dedupe is by global end time
with a small tolerance, using the word-level timestamps the framework gives you in
bestTranscription.segments.
Chunking also means each segment must be exported to a temporary file, which costs
CPU and disk churn on top of the recognition itself.
3. Attempt on-device first, fall back explicitly
This is the design decision that has to be documented rather than buried:
// Recognise one chunk: local first (offline / privacy), and if the local pass is
// empty or throws, retry against the server recognizer (when online).
// Throwing = a real failure; an empty string = that chunk genuinely had no speech.
private func recognizeChunk(url: URL, recognizer: SFSpeechRecognizer, onDevice: Bool)
async throws -> (String, [(String, Double, Double)]) {
if onDevice {
if let r = try? await recognizeOnce(url: url, recognizer: recognizer, forceOnDevice: true),
!r.0.isEmpty {
return r
}
// local empty/failed -> server fallback (more accurate online; offline it just
// returns empty again)
if let r = try? await recognizeOnce(url: url, recognizer: recognizer, forceOnDevice: false) {
return r
}
return ("", [])
}
if let r = try? await recognizeOnce(url: url, recognizer: recognizer, forceOnDevice: false) {
return r
}
return ("", [])
}
The on-device path is requested first, every time. When it returns nothing or errors,
the same audio is retried without the on-device flag — which means that audio
goes to Apple's speech service if the device is online. That is a real, deliberate
trade: the alternative is handing the user an empty transcript.
What you must not do is call this "fully on-device". It is not. It is on-device
preferred, with a first-party server fallback. Apple's server path is not a
third-party company, but "first-party" is not the same as "local", and those two get
conflated constantly in app descriptions.
The honest user-facing test is Airplane Mode: with no network, the fallback has
nowhere to go, so either the local model handles it or you get nothing — and either
way nothing was transmitted.
4. Own your timeout
private func recognizeOnce(url: URL, recognizer: SFSpeechRecognizer,
forceOnDevice: Bool) async throws
-> (String, [(String, Double, Double)]) {
try await withCheckedThrowingContinuation { continuation in
let request = SFSpeechURLRecognitionRequest(url: url)
request.shouldReportPartialResults = false
request.requiresOnDeviceRecognition = forceOnDevice
if #available(iOS 16.0, *) { request.addsPunctuation = true }
var didResume = false
var task: SFSpeechRecognitionTask?
let lock = NSLock()
func finish(_ r: Result<(String, [(String, Double, Double)]), Error>) {
lock.lock(); defer { lock.unlock() }
guard !didResume else { return }
didResume = true
continuation.resume(with: r)
}
// If a chunk never reports isFinal (the server fallback occasionally stalls),
// fail it after 60s so the layer above can fall back or skip, instead of the
// whole transcription hanging forever.
DispatchQueue.global().asyncAfter(deadline: .now() + 60) {
lock.lock(); let done = didResume; lock.unlock()
if !done {
task?.cancel()
finish(.failure(LocalTranscriberException("Recognition timed out.")))
}
}
task = recognizer.recognitionTask(with: request) { result, error in
if let result, result.isFinal {
let segs = result.bestTranscription.segments.map { ($0.substring, $0.timestamp, $0.duration) }
finish(.success((result.bestTranscription.formattedString, segs)))
return
}
if let error { finish(.failure(error)); return }
}
}
}
Two details that are easy to get wrong: a continuation must be resumed exactly once,
so the resume is guarded by a flag under a lock (the timeout and the callback race);
and the real error is thrown rather than swallowed, so the caller — not this function
— decides whether to fall back.
5. Recording is a separate reliability problem
Transcription quality is downstream of capture, and capture has its own silent
failure: an interruption. A phone call takes the microphone. If you do nothing, you
lose the rest of the session.
// Interrupted (incoming call / another app / headphones unplugged):
// .began -> pause, keep the file (don't finalize), wait to resume
// .ended -> reactivate the session and record() into the SAME file;
// if resuming fails, finalize so the captured part isn't lost
if type == .began {
if let recorder = self.recorder {
recorder.pause()
self.stopVadTimer()
self.interruptedRecording = true
self.sendEvent("onRecordingPaused", [:])
}
return
}
guard self.interruptedRecording, let recorder = self.recorder else { return }
self.interruptedRecording = false
try? AVAudioSession.sharedInstance().setActive(true)
if recorder.record() {
self.sendEvent("onRecordingResumed", [:])
} else {
recorder.updateMeters()
let duration = recorder.currentTime
let url = recorder.url
recorder.stop()
self.recorder = nil
self.endInterruptionObserving()
self.sendEvent("onRecordingInterrupted", ["uri": url.lastPathComponent, "duration": duration])
}
The recording format itself is mono AAC in .m4a, with sample rate and encoder
quality as the only user-facing knobs:
let settings: [String: Any] = [
AVFormatIDKey: Int(kAudioFormatMPEG4AAC),
AVSampleRateKey: sampleRate, // 22050 / 44100 / 48000
AVNumberOfChannelsKey: 1,
AVEncoderAudioQualityKey: encQuality.rawValue
]
Worth stating plainly because it is widely believed otherwise: a higher sample
rate does not make iOS use the on-device model. That choice depends on language
coverage and whether the local model is installed, not on how the audio was encoded.
Recording at 48kHz buys you nothing in recogniser selection.
Costs and boundaries
Being precise about what this design does and does not give you:
- It is not "audio never leaves the device." On-device recognition is requested first. When it is unavailable or returns nothing, the implementation can fall back to Apple's server-based recognition, and on that path the audio itself is what gets sent — not a summary, not just text. Airplane Mode is the only way a user can be certain a particular transcription stayed local.
- You can choose strictness instead, and some apps should: request on-device only and report failure. The cost is empty transcripts for every language and every clip the local model cannot handle. That is a product decision, not a technical one, and it should be stated either way.
- Empty is ambiguous. An empty chunk means "no speech here" or "recognition failed quietly". The code above distinguishes them by convention (throw = failure, empty = genuinely silent), but the framework does not hand you that distinction.
- Accuracy is unmeasured. Both the local and the server model are guessing under uncertainty; they differ in size, not in kind. Do not publish accuracy percentages you have not measured.
- Chunking has costs: temporary file exports, overlap deduplication logic, and word timings that are only as good as what each chunk returned. Some recognizers return text with no segments at all, which is why the accumulator has a branch for it.
- Speaker labels and summaries are not in this box. Diarization is not something the on-device path provides, and any summarisation means calling a model somewhere — so the text leaves even if the audio did not.
- The capability check is per-locale and per-device, and it changes with iOS releases and with what the phone has downloaded. Cache it at your peril.
This is the transcription pipeline used by Voice Studio, an iOS recorder that
attempts the on-device path first and documents the fallback rather than claiming it
does not exist.
Top comments (0)