[AI in Practice] Gemini 3.5 Transcribe: Real-time Transcription and Speaker Diarization in a macOS Meeting Translation App

Previously

I have a macOS App I use myself, gemini-live-translate-macos. It uses ScreenCaptureKit to directly capture audio from a specified App, eliminating the need for virtual sound cards like BlackHole. It then sends the audio to the Gemini Live API for real-time translation, outputting Traditional Chinese subtitles while playing Chinese audio. I've written two posts about the development process: the first one was about building it from scratch using AGY CLI, and the second one was about using Claude Code to take it from "functional" to "user-friendly."

The starting point for this new addition was simple: I saw a document for "Real-time Transcription" added to the Live API. Since I was already connected to the Live API, I thought adding a pure transcription mode would just be a matter of changing a few parameters.

However, after checking the documentation, I realized that Google released two models with very similar names but very different capabilities at once. The specific feature I actually wanted (speaker diarization) wasn't available at all on the model I originally thought it was.

---

Two Models with Names Differing by Only Two Words

Let's lay out the differences first; this is the part I spent the most time figuring out:

	gemini-3.5-transcribe-live	gemini-3.5-transcribe
API Used	Live API (WebSocket streaming)	Interactions API (Standard HTTP request)
Usage Scenario	Transcribe while speaking	Upload the whole file after recording
Speaker Diarization	Not supported	Up to 8 speakers
Word-level Timestamps	Not supported	Supported
Audio Length	10 minutes per session	1 hour (30 mins with diarization)
Smart Mode	SMART available	smart is mutually exclusive with diarization
Interim Subtitles	Has interimInputTranscription	Not applicable

The official documentation on the Live page's limitations section is very blunt:

Speaker diarization is not supported in live streaming sessions. For speaker diarization, use the non-streaming Audio transcription endpoint.

So, "seeing who is saying what in real-time" is currently impossible. For speaker diarization, you must record it and send the whole thing after the meeting. This limitation determined my entire subsequent architecture.

The Real-time One: Interim is the Key

The setup shape is different from the original translation model. responseModalities must be TEXT, and transcription parameters are placed under setup.inputAudioTranscription:

{
  "setup": {
    "model": "models/gemini-3.5-transcribe-live",
    "generationConfig": { "responseModalities": ["TEXT"] },
    "inputAudioTranscription": {
      "languageCodes": [],
      "mode": "SMART"
    },
    "realtimeInputConfig": {
      "automaticActivityDetection": { "disabled": false }
    }
  }
}

Leaving languageCodes empty enables automatic language detection; filling it with BCP-47 codes like ["en-US"] gives it a language preference. mode has two values: VERBATIM keeps everything word-for-word, while SMART removes filler words like "uh" and "um" and automatically adds formatting. For meeting minutes, I chose SMART, which is much cleaner to read.

There is also a customVocabulary where you can stuff technical terms, with a limit of 1000 items, though the documentation suggests staying under 100 for best results. I didn't implement this yet; I'll wait until I encounter names that are constantly misheard.

The response side adds a field that the original translation model didn't have:

• interimInputTranscription: Tentative results while speaking, which will be overwritten by subsequent content.
• inputTranscription: The finalized text when the speaker pauses or the turn ends.

This distinction affects how the UI is written, which I'll discuss later.

The Batch One: Uses a Completely Different API

This is the easiest place to trip up. gemini-3.5-transcribe doesn't use generateContent; it uses the Interactions API:

POST https://generativelanguage.googleapis.com/v1beta/interactions

{
  "model": "gemini-3.5-transcribe",
  "input": [
    { "type": "audio", "uri": "YOUR_FILE_URI", "mime_type": "audio/wav" }
  ],
  "generation_config": {
    "transcription_config": {
      "language_codes": [],
      "mode": {
        "type": "verbatim",
        "diarization_mode": "speaker",
        "timestamp_granularities": ["word"]
      }
    }
  }
}

diarization_mode: "speaker" is the switch for speaker diarization, but it can only be paired with verbatim. In other words: if you want speaker diarization, you have to give up the benefits of SMART mode's filler word removal; you can't have both.

I didn't include timestamp_granularities at first because it wasn't in the short example in the documentation. Without it, the entire feature silently fails—a process I'll describe in the final section.

Audio must first be uploaded to the Files API to get a file URI, which is then included in the request. The official documentation doesn't provide a base64 embedding example. The response looks like this:

{
  "steps": [{
    "type": "model_output",
    "content": [{
      "type": "text",
      "text": "Hello world",
      "annotations": [
        { "type": "word_info", "text": "Hello", "speaker": "spk_1",
          "start_offset": "0.100s", "end_offset": "0.450s" }
      ]
    }]
  }]
}

Speaker labels are in the word-level annotations as IDs like spk_1, spk_2. The model doesn't know who is who. The documentation also notes a maximum of 8 speakers, and "attribution for 3 or more people is experimental."

---

Phase 1: Adding Pure Transcription Mode to the App

This part was simpler than I expected because the App was already parsing inputTranscription and outputTranscription (the translation model returns both for bilingual subtitles). The real change was in the mode branching.

The original connection layer determined it like this:

let isTranslateModel = modelName.contains("live-translate")

A boolean split into two paths. Now needing three, I switched to an enum:

enum LiveMode {
    case translate // gemini-*-live-translate-*: output translated audio + bilingual subtitles
    case transcribe // gemini-*-transcribe-live: text only
    case general // other Live models: rely on systemInstruction for interpretation

    static func from(modelName: String) -> LiveMode {
        if modelName.contains("transcribe") { return .transcribe }
        if modelName.contains("live-translate") { return .translate }
        return .general
    }
}

I also took the opportunity to extract the setup config generation and server response parsing from GeminiLiveConnection into pure functions. That file was getting a bit bloated; extracting these reduced it by over sixty lines, and the extracted parts can be tested directly.

A Preventive Trade-off: Duplicate Words

When parsing responses, I made a decision worth noting. Besides inputTranscription, the parts in modelTurn might also carry the same text. If both are collected, the same sentence will appear twice in the subtitles and export file.

To be honest, I haven't actually seen this happen, but while reading the documentation and old code, I noticed both paths would lead to didReceiveOutputTranscription. I blocked it to avoid the risk:

// Content for pure transcription mode is already provided by inputTranscription.
// Receiving text from modelTurn again would cause the same sentence to appear twice.
if mode != .transcribe,
   let text = part["text"] as? String, !text.isEmpty {
    events.append(.outputTranscription(text))
}

The downside of this preventive defense is that if it never actually happens, this line is dead code that no one understands. So I wrote it as a test ("transcribe mode ignores modelTurn text"), ensuring the behavior is locked in and the intent is documented.

How to Run Tests in a Project Without a Testing Framework

This project doesn't use an Xcode project file; it uses a build_app.sh that calls swiftc to compile all .swift files into a .app. No SwiftPM means no swift test and no XCTest.

My solution was to use the same trick again: since the extracted parts are pure functions, I wrote a main.swift as an assertion runner. I compile it with those pure functions into an executable and use the exit code to determine success or failure.

swiftc -sdk "$SDK_PATH" -target "${ARCH}-apple-macos13.0" \
  -o "${BUILD_DIR}/run_tests" \
  LiveSetupConfig.swift TranscriptFormatter.swift \
  WAVRecorder.swift AudioChunker.swift \
  GeminiTranscribeService.swift GeminiSummaryService.swift \
  Tests/main.swift

The assertion function itself is less than ten lines:

func checkEqual<T: Equatable>(_ actual: T?, _ expected: T, _ name: String) {
    if actual == expected {
        passedCount += 1
    } else {
        failures.append("\(name) (Actual: \(String(describing: actual)), Expected: \(expected))")
    }
}

This obviously can't compete with a real testing framework—no setup/teardown, no parallel execution, and it won't tell you which line failed. But it runs, it prevents regressions, and it doesn't require converting the whole project to SwiftPM just for testing. This phase ended with 37 assertions; after adding speaker diarization, it grew to 98.

---

Phase 2: Speaker Diarization Requires More Changes Than Expected

"Adding speaker diarization" sounds like calling one more API, but it actually involves three more tasks: saving audio to a file, uploading it, and then connecting to a completely different API. And because batch transcription might take several minutes, it can't block the original stop process.

The final data flow looks like this:

graph TD
    A[ScreenCaptureKit captures PCM] --> B[Gemini Live API<br/>Real-time Subtitles]
    A --> C[WAVRecorder<br/>Synchronous writing]

    D[Press Stop] --> E[Phase 1: Immediate Output]
    E --> E1[Transcript md]
    E --> E2[AI Meeting Summary]
    E --> E3[Meeting Record Webpage]

    D --> F[Phase 2: Background Execution]
    F --> F1[Collect file → Split if > 30 mins]
    F1 --> F2[Upload to Files API]
    F2 --> F3[Interactions API Batch Transcription]
    F3 --> F4[Delete cloud copy]
    F4 --> F5[Write diarized.md]
    F5 --> F6[Rerun summary with speaker info + infer names]
    F6 --> F7[Regenerate webpage]
    F7 --> F8[Delete local recording]

Splitting it into two phases was intentional. Every step in Phase 2 could fail (upload timeout, quota exhausted, response format mismatch). If it were hooked into the middle of the original process, a failure would mean losing even the basic transcript and summary. Now, Phase 1 remains untouched, and Phase 2 is just appended; if it fails, you just lose one output.

The recording format was also conveniently available: the audio sent to the Live API is already resampled to 16kHz mono 16-bit PCM. Writing that same data to disk results in valid WAV content—just add a 44-byte header, no re-encoding required.

The same defensive mindset was applied to parsing: parseDiarized assumes the response structure based on documentation, but if the actual format differs, I left a fallback—if word annotations aren't found, it falls back to the entire output_text so the transcript isn't completely empty. This fallback actually came in handy, though not in the way I expected, as I'll explain in the last section.

---

Major Pitfalls and Solutions

Pitfall 1: Chunking cannot be "fill 30 minutes and leave a remainder"

The audio limit for speaker diarization is 30 minutes, but meetings often last over an hour, so chunking is necessary.

My first idea was the most intuitive: fill 30 minutes for one segment and leave the rest as the last segment. While writing tests, I realized this approach had two holes.

A 60-minute and 5-second meeting would be split into 30 mins, 30 mins, and 5 seconds. Sending that 5-second tail for transcription is meaningless and costs an extra upload and API call. What if I merge the tail into the previous segment? Then that segment becomes 30 mins and 5 seconds, exceeding the API limit, and the whole segment gets rejected.

Cause & Solution: Switch to even splitting. Calculate how many segments are needed (round up), then distribute the total length evenly:

let count = (usable + maxChunkBytes - 1) / maxChunkBytes
guard count > 1 else { return [0..<usable] }

var ranges: [Range<Int>] = []
var start = 0
for index in 1...count {
    var end = usable * index / count
    end -= end % blockAlign // Align to 16-bit sample boundary
    if index == count { end = usable }
    ranges.append(start..<end)
    start = end
}

60 mins and 5 seconds becomes two segments of 30 mins and 2.5 seconds? No, that would exceed the limit. Actually, it's ceil(3605 / 1800) = 3, split into three segments of 20 minutes each. There will never be a tiny remainder segment, and it will never exceed the limit.

The end -= end % blockAlign is also necessary. 16-bit mono uses 2 bytes per sample. If you cut on an odd byte, the entire subsequent audio stream's bytes will be shifted by one, resulting in noise when played.

Pitfall 2: spk_1 in Segment 2 is not the same as spk_1 in Segment 1

A problem I thought of only after chunking. Each segment is an independent API call. The model doesn't know what happened in the previous segment, so spk_1 in Segment 2 has no relation to spk_1 in Segment 1; they could be different people.

If you just concatenate the three segments, the reader will naturally assume spk_1 is the same person throughout. This is worse than having no speaker labels: it gives a false sense of certainty.

Cause & Solution: Whenever chunking occurs, append the segment number to the ID to create a namespace:

static func qualifiedLabel(chunkIndex: Int, chunkCount: Int, speaker: String) -> String {
    guard chunkCount > 1 else { return speaker }
    return "Seg\(chunkIndex + 1)-\(speaker)"
}

Insert an explanatory line at the segment boundaries during export:

---

> Segment 2 (Speaker IDs are not continuous with the previous segment)

These IDs are also used for the text sent to the AI for name inference. By sharing the same vocabulary, the model has a chance to link the same person across segments—if someone is called "Evan" in both Segment 1 and Segment 3, it can map them individually rather than being forced to assume the IDs are identical.

Pitfall 3: Joining words results in "H e l l o W o r l d"

The batch API returns word-level annotations, which you have to join into sentences yourself. For English, it's intuitive: join with spaces.

The problem is Chinese. Gemini's Chinese tokens joined with spaces look like "Ni hao shi jie wo men jin tian" (Hello world we today), looking like a word segmentation exercise.

Cause & Solution: When joining characters, check the properties of the characters on both sides. If either side is CJK (Chinese, Japanese, Korean), don't add a space; also, don't add a space before punctuation:

private static func needsSpace(after previous: Character, before next: Character) -> Bool {
    if isCJK(previous) || isCJK(next) { return false }
    if next.isPunctuation { return false }
    return true
}

isCJK checks Unicode blocks, covering CJK Unified Ideographs, Kana, Hangul, and full-width characters.

The punctuation rule was added later. Originally I only blocked CJK, but testing ["Hello", ",", "world"] revealed it became Hello , world. You'd never notice this without writing tests because it doesn't "break"—it's just a bit ugly.

Pitfall 4: Two asynchronous processes racing to write the same summary

Phase 1 generates an AI meeting summary after stopping. Phase 2 generates another one after getting the diarized transcript (this time with speaker info, so the "Assignee" field in action items can be filled).

Normally, Phase 2 is definitely slower—it has to upload over 100MB of audio and wait for transcription. But "normally slower" isn't a guarantee. If the Phase 1 summary API happens to hang and retry, while the Phase 2 audio is only one minute long and finishes quickly, the order will reverse. The late-returning old summary from Phase 1 would overwrite the diarized result.

Cause & Solution: Phase 2 waits for Phase 1 to finish before writing:

// Wait for Phase 1 summary to land, otherwise it might return after us and overwrite the diarized result
await minutesTask?.value
guard !Task.isCancelled else { return }

A one-line fix, but you have to first realize that "these two things actually have no guaranteed order." This kind of race almost never appears in testing; it only happens on a day with particularly bad network, leaving the user with a confusingly reverted meeting record.

Pitfall 5: During the TDD RED phase, the test itself crashed

While writing tests for the WAV header, I followed TDD rules: write the test first, create a stub returning an empty Data(), and run it to see it fail. Instead of failing, the entire test program crashed:

Swift/arm64e-apple-macos.swiftinterface:41299: Fatal error:
UnsafeRawBufferPointer.load out of bounds
Trace/BPT trap: 5

My test was trying to read the 24th byte to check if the sample rate was 16000, but the stub returned empty Data, causing an out-of-bounds read.

Cause & Solution: Zero-pad before reading:

let produced = WAVRecorder.header(dataByteCount: 64000)
checkEqual(produced.count, 44, "WAV header is 44 bytes")

// Pad with zeros if length is insufficient, so subsequent fields report failure instead of crashing the test
let header = produced + Data(repeating: 0, count: max(0, 44 - produced.count))

This is a small thing, but it highlighted something I usually ignore: the test program itself must be resilient to the object under test being completely broken. If a test ends in a crash during the RED phase instead of reporting a failure, you only know "something broke," not which of the twelve fields were wrong. After padding with zeros, a single run lists all twelve expected values, allowing me to implement them by following a list rather than guessing.

Pitfall 6: Scroll anchor fails in the new mode

There's a structural difference between real-time transcription and translation modes: translation mode accumulates words into the "current sentence" and pushes to history only when punctuation is reached; transcription mode's inputTranscription is a finalized sentence that goes straight to history.

The original auto-scroll was written like this:

proxy.scrollTo("currentLine", anchor: .bottom)

The ID currentLine was attached to the display block for the "current sentence." In transcription mode, the sentence goes to history as soon as it's finalized, and that block immediately disappears—the scroll target no longer exists, so the screen stays put.

Cause & Solution: Use a bottom anchor that always exists:

Color.clear
    .frame(height: 1)
    .id("bottomAnchor")

This bug had no error message or crash; the "new feature just wouldn't scroll," and it only became apparent when there was enough content to exceed the screen. I found it by reading the view's conditional branches, not by running it.

Pitfall 7: Letting AI guess names—the key is stopping it from hallucinating

Once you have spk_1 and spk_2, the natural desire is to replace the IDs with real names. This is actually feasible—meetings often contain clues like "Evan, how's the progress on your end?" or "I'm Sarah, in charge of frontend." If you give the tagged transcript to the model, it can make the connection.

But this is also where hallucinations are most likely. Models are happy to "infer" a name from tone, job content, or speaking frequency and present it with the same confidence as a fact in the meeting minutes.

Cause & Solution: Three methods combined.

First, the schema explicitly allows null and requires evidence:

properties["speakers"] = [
    "type": "ARRAY",
    "items": [
        "type": "OBJECT",
        "properties": [
            "label": ["type": "STRING"],
            // Must return null if no clues are found, rather than forcing a name
            "name": ["type": "STRING", "nullable": true],
            "evidence": ["type": "STRING", "nullable": true]
        ],
        "required": ["label"]
    ]
]

Second, the prompt sets strict rules:

Only fill in the name if there is a clear address, roll call, or self-introduction in the transcript or meeting notes. Explain in 'evidence' which sentence led to this conclusion. Do not speculate on names based on tone, job content, or speaking frequency. If no clear clues are found, both name and evidence must return null.

Third, remove the confidence score. I originally designed a confidence field but later removed it. The reason is that a model's self-assessment of confidence is inherently unreliable, and evidence already fully serves this role: if there's evidence, the inference succeeded; if not, it didn't. An extra confidence score just makes people think it's more credible than it is—"It says 0.7 here, so it's probably 70% accurate"—but that 0.7 doesn't come from any real probability distribution.

The output looks like this, with evidence for successful inferences and honest admissions for failures:

## Participants
- **Evan** (spk_1) — Evidence: Addressed as "Evan, how's the progress on your end?" in Segment 3
- spk_2 — Insufficient clues in transcript and notes to identify name

Keeping the ID in parentheses is also intentional. Seeing **Evan (spk_1)** tells you it was inferred; if it's wrong, you can verify it yourself. If it just said **Evan**, it would look like an absolute fact.

Another source of clues: I also send the meeting notes the user wrote on the spot. Notes often already contain a list of participants, which significantly increases the success rate. But a safeguard is needed here—the prompt must clearly state "Notes can only be used to map speaker names; do not treat note content as something someone said in the summary or action items," otherwise your own memos might turn into someone else's speech.

---

Privacy and Cleanup: Don't leave behind what you don't want to keep

Batch transcription inherently creates two more pieces of data than real-time streaming: the local recording file and the copy uploaded to Google. Both must have a clear disposal plan.

• Recording files are only created if the option is checked. If speaker diarization isn't enabled, no file is written, and nothing stays on the disk.
• Local recordings are deleted upon successful transcription. A one-hour meeting is about 115MB, which adds up quickly.
• Files are kept if transcription fails. This is an intentional exception: if it fails, the recording is kept, and the status bar tells you the path so you can retry or handle it yourself. Deleting it then would be actual data loss.
• Cloud copies are deleted immediately after use. Files API files expire automatically after 48 hours, but "it will disappear eventually" and "I'm sure it's gone now" are two different things.

// Delete the cloud copy immediately after use, don't wait 48 hours for auto-expiration
await GeminiFilesUploader.delete(apiKey: apiKey, name: uploaded.name)

---

Results and Benefits

The numbers for both phases combined:

	Pure Transcription Mode	Speaker Diarization	Post-testing Fixes
New Files	2	4	0
Total Assertions	37	98	108
Commit	b0e12c5	486ba8f	686899a, 1cc41c3

GeminiLiveConnection.swift, the file that originally did everything, was reduced by over sixty lines after setup generation and response parsing were extracted. Those two pure function modules are now guarded by over 20 tests. This was an unexpected benefit: the decoupling done to make things testable was itself a refactoring I had put off for a long time.

First, confirm which model has the feature you want. I initially assumed "Real-time Transcription" would have speaker diarization—they're both transcription models, right? The only difference should be real-time vs. batch. I only found out otherwise after checking the docs, and the difference isn't just a parameter switch; it's architectural: for diarization, you must record, upload, use a different API, accept a 30-minute limit, and give up SMART mode. If I hadn't checked first, I would have hit a whole new subsystem while expecting to just "change a few parameters."

Limitations often dictate the architecture. Almost every design decision this time was forced by limitations: the 30-minute limit forced chunking, chunking forced ID namespaces, the slowness of batching forced two-phase export, and the mutual exclusivity of smart and diarization forced me to choose between a clean transcript and speaker labels. Checking limitations before designing is much easier than designing and then hitting a wall.

The real-time transcription half has been tested in the field. The screenshot above is the result of feeding it a Japanese video; gemini-3.5-transcribe-live automatically detected Japanese and output a Japanese transcript directly without translation, and the meeting record webpage was generated as usual after stopping. Leaving languageCodes empty for auto-detection actually works.

The speaker diarization half had issues during field testing, and in a way I didn't expect: the file was generated, the transcript was correct, and the program reported no errors, but there was no speaker separation. This part deserves its own section.

---

Postscript: Documentation examples might give you an empty result without errors

After posting, I ran speaker diarization on a real conversation. meeting-2026-08-28-11-54-diarized.md was generated, the content was complete, not a word was missing, but there were no speaker labels from beginning to end—just one continuous block of text.

No error messages, the status bar showed success, and the file had everything it should. This kind of failure is the hardest to debug because it looks like success.

Create a Minimal Reproducible Example First

The first problem was that I had no evidence—the App didn't save the raw response, and the recording of that meeting was automatically deleted because "transcription succeeded." To re-run it, I'd have to start another meeting with no guarantee of reproduction.

So instead of guessing what broke, I first found a way to get a raw response. macOS's built-in say command can use different voices, so I used it to synthesize a two-person conversation:

say -v Alex -o a1.aiff "Hi Samantha, did you finish the quarterly report yesterday?"
say -v Samantha -o a2.aiff "Yes Alex, I sent it to the whole team this morning."
say -v Alex -o a3.aiff "Sure, I will look at the budget section this afternoon."

for f in a1 a2 a3; do afconvert -f WAVE -d LEI16@16000 -c 1 $f.aiff $f.wav; done

The three segments joined together are 17 seconds long with two speakers, in the exact same format as the App's recordings (16kHz mono 16-bit). Then I used curl to go through the entire upload and transcription process, dumping the full JSON.

This step took less than five minutes, but it turned "starting another meeting, running for ten minutes, and not being sure of reproduction" into "changing one field, running for ten seconds, and seeing the difference immediately." A minimal reproducible example is worth the time, especially when the original reproduction path is expensive.

The dumped response looked like this:

{
  "steps": [{
    "content": [{ "text": "Hi Samantha, did you finish...", "type": "text" }],
    "type": "model_output"
  }]
}

811 bytes, complete transcript, zero annotations. So the problem wasn't my parsing; it was the request.

Root Cause 1: Word-level annotations must be explicitly enabled

Using the same audio and the same uploaded file, I changed just one field in the request:

Request Content	Objects with speaker in response
Only diarization_mode: "speaker"	0
Removed language_codes: []	0
Added timestamp_granularities: ["word"]	41

Speaker IDs are attached to word_info annotations, and word-level annotations must be explicitly requested in the request to be returned. No request, no annotations; no annotations, no speakers.

I didn't include this field originally because I copied the shortest Python example from the documentation—that example only had type and diarization_mode. The full REST example elsewhere in the documentation actually has timestamp_granularities, but by then I already "knew" how to write it and didn't look back.

The most frustrating part is that it doesn't report an error. The API returns 200, gives you the full transcript, and the status is completed. If it had returned an error like "You requested speaker diarization but didn't enable word annotations," I would have fixed it in five minutes.

Root Cause 2: The actual label format differs from the documentation

After adding the field, annotations appeared, but they looked different from what I expected:

{"text":"Hi,","start_offset":"0.100s","end_offset":"0.500s","speaker":"spk:0","type":"word_info"}

spk:0. A colon, and starting from 0. The documentation consistently uses spk_1, spk_2.

My display logic was written like this:

if let range = speaker.range(of: "spk_"), let number = Int(speaker[range.upperBound...]) {
    return "Speaker \(number)"
}
return speaker // ← Fallback prints the raw ID if no match

spk:0 doesn't match spk_, so it fell straight into the fallback, displaying **spk:0**: on the screen.

My tests completely missed this bug for a simple reason: the test data was written according to the documentation. The documentation was wrong, so the tests were wrong, and the green light told me everything was fine. This is the main thing I want to record: for external API tests, you are actually testing "my understanding of this API," not the API itself. If your understanding is wrong, the test will faithfully protect that error.

I didn't fix it by changing spk_ to spk:; that would just be betting in a different direction. Instead, I stopped parsing the number in the ID and used the order in which the speaker first appeared in that segment:

/// The actual format of the ID is determined by the API (actual returns spk:0, spk:1, while official docs say spk_1),
/// so we don't parse the number in the ID. Instead, we number them based on their first appearance in this segment.
static func speakerOrder(in chunk: [DiarizedSegment]) -> [String: Int] {
    var order: [String: Int] = [:]
    for segment in chunk where !segment.speaker.isEmpty {
        if order[segment.speaker] == nil {
            order[segment.speaker] = order.count + 1
        }
    }
    return order
}

It won't break even if the format changes again because it doesn't look at the format at all.

My own fallback made failure look like success

I wrote this earlier and was quite proud of it:

parseDiarized assumes the response structure based on documentation, but if the actual format differs, I left a fallback—if word annotations aren't found, it falls back to the entire output_text so the transcript isn't completely empty.

This fallback did work, and the effect was as expected: the user got a complete transcript with nothing lost.

But it also hid the failure. If I hadn't had that fallback, -diarized.md would have been empty or not generated at all, and I would have known immediately that something was wrong. With it, I got a file that looked perfectly normal, just missing the feature I wanted—the very feature that was the sole reason for enabling it.

I haven't fully figured out the balance here. Graceful degradation isn't wrong; the mistake is not speaking up after degrading. My current approach is to keep the fallback but change the status bar message when it's triggered, explicitly stating "Speaker info not obtained this time" instead of the usual "Diarized transcript saved."

After the Fix

Running the full parsing with a real response, both segments spoken by Alex correctly returned to Speaker 1:

**Speaker 1**: Hi, Samantha. Did you finish the quarterly report yesterday?
**Speaker 2**: Yes, Alex. I sent it to the whole team this morning. Could you review the budget section?
**Speaker 1**: Sure. I will look at the budget section this afternoon and get back to you.

The quality of the speaker diarization itself is good. Tests increased from 98 to 106, with the extra eight being regression tests for these two root causes—this time, the test data wasn't copied from the docs but clipped from real responses.

Three more details learned from field testing:

• The REST endpoint only accepts snake_case; sending generationConfig is explicitly rejected with Unknown parameter 'generationConfig'. Did you mean 'generation_config'?. This is a great error message, a hundred times more useful than the silent failure above.
• Annotations include start_index and end_index, mapping directly to positions in content.text. Using these to split strings is much more accurate than my heuristic for joining CJK/English words, meaning the entire joinWords could be removed. I haven't done this yet.
• The state of uploaded files is immediately ACTIVE; 17 seconds of audio didn't go through a PROCESSING phase. My polling logic seems redundant for short audio, but I don't know if it's needed for long audio, so I'll keep it for now.

This article thus has two endings, and I've decided to keep both. The first part said "Speaker diarization hasn't been verified yet; I'll update after I run it," and then I actually ran it, and it broke. If I had edited the first part and only kept the fixed version, this would have been a smooth "I did X, and it worked" post—but what actually happened was "I did X according to the docs, it failed silently, and it took me half an hour to find out why." The latter is much more useful to readers.

The code is at kkdai/gemini-live-translate-macos. The two official documents are Live transcription and Audio transcription. I recommend reading the limitations section on the speaker diarization page thoroughly before starting.