Introduction
During a televised debate, a false claim travels faster than its correction. Newsrooms that analyze live need two things at once: what was said, and which speaker said it. Amazon Transcribe answers the first question well. Identifying the speaker by voice is harder than it looks, and it is the subject of this post.
Live processing imposes constraints that batch processing ignores. Amazon Transcribe finalizes a speech segment about every five seconds, while an automated analysis runs several AI agents for two to five minutes. Naming the author of each segment has to fit in a budget of a few hundred milliseconds. Speaker turns sometimes last under a second.
Three failures are possible, and they do not cost the same. Not knowing who speaks is inconvenient. Answering too late is useless. Attributing a sentence to the wrong person is worse than attributing nothing.
Attributing a claim to someone
A verified claim without an author loses most of its journalistic value. “Unemployment fell by three points” carries different weight depending on whether a candidate, a minister, or an invited economist says it. The journalist needs to attach each claim to a person. They also need to follow that person from one broadcast to the next, to measure how consistent their statements are.
Amazon Transcribe provides diarization: the service segments the stream and labels speaker turns. That information is useful, and it is not sufficient for this use case.
Three limits of diarization alone
Diarization assigns local labels. It tells you that two passages come from two different people within one stream. It does not tell you who they are. Nothing makes a given speaker keep the same label for the whole stream either, since a label can be reassigned or split mid-session. And those labels do not survive the end of the stream, because they are renumbered on the next one.
The three limits compound. A participant becomes “speaker 1” on Monday and “speaker 3” on Thursday, with nothing linking the two. Comparing what one person said across several shows then takes manual reconciliation, which is what live processing rules out.
The first approach: inferring names from text
The first version of the identification named speakers without analyzing their voices. An AI agent “Context” on Bedrock AgentCore Runtime kept every segment of the session in Amazon Bedrock AgentCore Memory. It exploited a regularity of talk shows: at the opening, each participant introduces themselves or is introduced by the host.
From those sentences, the “Context” agent linked the anonymous diarization label to a name, then propagated that link across the rest of the broadcast. The mechanism is still readable in that sub-agent prompt. It records facts of the form “speaker 0 in session S is the first candidate”, then tries to relate any new label to the facts already stored.
That approach worked often, and that is the problem. The name came from a language model inference over text, not from a measurement on the signal. Nothing made the same conclusion reproducible from one run to the next, and no value quantified the confidence.
Failure was also silent. A missing introduction, an ambiguous phrasing, or a guest arriving mid-show, and the agent produced a plausible name rather than no name. A journalist cannot publish an attribution on that basis, because they cannot know which one to verify. The decision needed to be reproducible, to carry a score, and to be computed on the voice itself.
Solution overview
A voice fingerprint answers all those limits. It describes the timbre as a vector, independent of the show where it was computed. It is the principle of a fingerprint, applied to the speech signal. Two nearby vectors designate the same voice, across shows and across weeks. The platform compares each finalized speech segment against a registry of known voices, and attaches either a name, a stable anonymous identifier, or an explicit abstention.
That choice moves the problem rather than removing it. It introduces persistent biometric data, with the obligations that come with it, and a new risk: attributing a sentence to the wrong person. For a newsroom, that error is worse than no attribution, and the two rules in the next section come from it.
Two design rules before any optimization
The system prefers silence to error. When the resemblance to a known voice is insufficient or ambiguous, it assigns “no name”. It produces a stable anonymous label instead, such as “voice 12”, which stays the same for the whole show and for later shows. The journalist sees that the same person is speaking, without being told who. An operator can name that voice afterwards, and the history already produced becomes named.
Recognition never blocks verification. If identification fails, exceeds its time budget, or could not start, the segment still goes to the AI agents. It then carries an identity marked as degraded, along with the reason. A pipeline whose optional link can interrupt the main link is not usable live.
The architecture
The diagram that follows traces one audio stream from the sender through the streaming server to the registry and the verification agents. Read it left to right: the sender feeds a WebSocket connection, the server splits the audio between transcription and identification, and the identity joins the segment before dispatch.
Figure 1. Speaker identification architecture, from audio capture to the fingerprint registry.
An FFmpeg process decodes the source and emits signed 16-bit little-endian pulse-code modulation (PCM) audio, 16 kHz mono. It is pushed over a WebSocket connection in 100 millisecond frames.
The streaming server is a container running on AWS Fargate, in an Amazon Elastic Container Service (Amazon ECS) cluster. That container does four things. It relays the audio to Amazon Transcribe Streaming and receives partial then finalized results, with their native time boundaries. It keeps recent audio in a 40 second rolling buffer, one per session. It runs two Open Neural Network Exchange (ONNX) models on CPU, and it compares the resulting fingerprint to the registry of known voices.
The buffer holds 40 seconds for a reason. That covers the 30 second cap observed on finalized results, plus the two second widening described in the section on going from signal to identity, plus delivery lag. The cost is bounded: 40 seconds of 16 kHz 16-bit mono audio is 1.28 MB per session.
Amazon Transcribe Streaming is used, and it is the only transcription engine in the system. It produces the text, the partial results displayed to the journalist immediately, the segmentation of speaker turns, and their time boundaries. What the system does not use is the diarization the same service offers alongside transcription.
The platform needs to follow a speaker from one broadcast to the next and to attach a name. Diarization labels, whether from Amazon Transcribe or from any other model, are anonymous identifiers local to a single inference. They distinguish speakers within one stream but do not name them and do not persist across streams. The fingerprint registry solves both: it carries names, and it survives sessions.
Two models run inside the container, and neither provides identity. Segmentation uses pyannote segmentation-3.0 under the MIT license. It signals where there is voice, where the speaker changes, and where two people speak at once. Those are segmentation signals: they tell the fingerprint extractor where to cut, not who speaks. The pyannote speaker classes are just as anonymous and local as the Transcribe spk_N labels, and the system reads none of them. The fingerprint itself uses WeSpeaker ResNet34-LM under the Apache 2.0 license, which produces a 256 dimension vector. Identity comes from comparing that vector against the registry, and from nowhere else.
Both artifacts are published to a versioned Amazon Simple Storage Service (Amazon S3) bucket, and the model version is recorded with every registry entry. A fingerprint computed by one model is comparable only to fingerprints from the same model, and that constraint has to stay verifiable months later.
Running inference inside the container avoids an extra network hop on a time-constrained path. Inference is CPU-bound, so it runs in a dedicated thread pool, never on the event loop that serves the WebSocket connections.
The registry lives in Amazon DynamoDB, in an infrastructure stack separate from the server. That separation is deliberate: biometric data has its own lifecycle, independent of redeployments of the compute that produces it. An AWS Lambda function handles enrolment of reference voices.
There is no synchronization, and that is the design
One question follows from the architecture. The ONNX models run inside the container, and Amazon Transcribe is a remote service. The identity has to land exactly on the boundaries of the text. How are the two clocks kept in step?
They are not, because there are not two streams to align. There is one stream of PCM audio, and both consumers count the same samples. A single call site pushes each frame to the transcription queue and to the rolling buffer, in that order, from one coroutine.
The two clocks are in fact the same clock. The rolling buffer is addressed in bytes, not in time. Its position in milliseconds is the number of bytes received divided by 32, because 16 kHz at two bytes per sample gives 32 bytes per millisecond. Its origin advances only when the buffer trims, so it is a sample counter rather than a wall clock.
Amazon Transcribe returns start and end times in seconds since the beginning of its own stream. That stream was fed by those same bytes. Byte N therefore sits at the same millisecond on both sides. The audio is the clock, nothing needs reconciling, and no correlation between concurrent tasks is required.
In practice, a finalized result arrives with boundaries such as 78787 to 85500 milliseconds. The server cuts its own buffer at those indices and runs the ONNX models on those bytes. The remote service never returns audio.
The principle is worth stating once. The remote automatic speech recognition (ASR) service is used as a function from audio to text plus time coordinates. The reference audio stays local, always.
From signal to identity
The next diagram shows the decision path a single finalized segment travels. It runs from the Transcribe boundaries through the quality tiers to one of three outcomes: a name, an anonymous group, or a degraded identity.
Figure 2. From audio signal to speaker identity, including every abstention path.
Each segment finalized by Amazon Transcribe triggers the same sequence, and the two models play distinct roles. Amazon Transcribe supplies the segment time boundaries, which trigger the processing and frame the window cut from the buffer.
The pyannote segmentation gives voice activity at millisecond resolution, which measures the speech actually usable in the window rather than its raw duration. It gives the instants where the speaker changes, and the passages where two people speak at once. The Transcribe boundaries say when a sentence starts and ends, but not where the silence sits inside it, or where the voice changes.
A short sentence supplies little material, so the system widens the window by up to two seconds before and after. Two rules bound that widening. It never crosses a speaker change boundary that the segmentation detected, otherwise the fingerprint would mix two voices. And it uses only audio already in the buffer, never waiting for audio still to come, which would hold up the verification.
The usable speech duration then determines what the system allows itself to do. Below 30 seconds it extracts no fingerprint and returns a degraded identity.
Between 30 and 75 seconds it extracts a low quality fingerprint. That fingerprint can attach the segment to an already known voice, but it never creates a new voice and never enriches an existing one.
Above 75 seconds the fingerprint is high quality and does both.
This asymmetry protects the registry. A fingerprint computed on a speech fragment is noisy. Using it to label one segment costs at most one wrong label, visible with its score. Using it to create or update a registry entry makes that noise permanent, and every later comparison inherits it. The system therefore lets a weak fingerprint read the registry, never write to it.
Two distinct thresholds govern what follows. Above 0.60 cosine similarity, the segment attaches to the nearest voice and inherits its stable identifier. Above 0.70, and only if that voice already carries a name, the identity is named. Between the two, the segment joins the group without carrying a name.
A third safeguard applies. If the two highest-scoring candidates are separated by less than 0.05, no name is assigned. Two voices too close to tell apart produce an abstention, not a bet. Every branch converges on sending the segment to the verification agents, including the degraded branches.
Concurrency: two invariants to hold
Several Fargate tasks can process sessions in parallel. Two properties of the registry have to hold under that concurrency, and neither comes from a naive write.
- one new voice must never create two entries. A lock held in a DynamoDB item serializes the critical section that compares then creates. The lock is acquired by conditional write, and carries a time to live that releases it if its holder disappears. The holder rereads the registry before deciding, and therefore sees any entry created in the meantime.
- two simultaneous enrichments of the same voice must both be applied. The aggregate fingerprint of a voice is therefore not a vector rewritten on every contribution, which would lose one write in two. Each contribution is an immutable item on a distinct sort key, and the aggregate fingerprint is a derived value recomputed from the full set of contributions. Two concurrent writes land on two different keys and cannot overwrite each other.
That recomputation happens off the critical path. It cost 95 to 153 milliseconds per segment when it ran synchronously, for a value that is only a cache of contributions already made durable.
Keeping up with live: a session round robin
The dispatch design follows from one piece of arithmetic, which the next diagram lays out alongside the pool it produces.
Figure 3. Session round robin across Amazon Bedrock AgentCore session identifiers.
Amazon Transcribe finalizes a segment about every five seconds, while a full verification runs multiple sub-agents and takes minutes. Processing segments one after another therefore opens a backlog that never closes.
The first version of the POC framed that imbalance with a semaphore: one segment at a time, the others dropped. It lost about 80 percent of the stream. A debate verified at 20 percent has no editorial value.
Amazon Bedrock AgentCore Runtime offers the lever that solves this. It isolates execution by session identifier, so two calls carrying two distinct identifiers run in parallel, in separate environments. Having K identifiers is therefore enough to obtain K concurrent verifications, with no containers to provision or manage.
The server builds that pool at the start of each broadcast. The K identifiers are derived deterministically from the broadcast identifier, which keeps them stable across reconnections, and AgentCore requires each one to be at least 33 characters long. Segments are then distributed across the pool in round robin. The sizing follows from the arithmetic above: the segment rate multiplied by the worst case latency.
The call is made without waiting for the response, and that is the second point holding the whole thing together. The streaming server does not need the verdict. The agent publishes it to AWS AppSync thanks the hook feature of Strands Agent SDK, and the journalist interface receives it through a GraphQL subscription. The server drops the segment on a thread pool and moves to the next one.
This inversion removes backpressure instead of managing it. There is no semaphore, no reading of the response stream, and no segment dropped for lack of room. The server sends 12 segments per minute whatever a verification costs.
The pool is prewarmed. When the broadcast opens, the server sends K warmup calls in parallel, so the first real segments meet environments that are already active rather than a cold start.
Two properties of the system make giving up session affinity acceptable. The AI sub-agents are stateless from one segment to the next. And the facts accumulated about speakers live in the long-term memory of Amazon Bedrock AgentCore Memory, whose namespace is partitioned by broadcast identifier rather than by session identifier. The executions therefore read and write the same set of facts, while short-term memory stays per session.
Two tradeoffs come with this choice. Verdicts arrive out of order, because verifications started in sequence do not finish in the same sequence, so the interface reorders them on their original timestamp. And the shared facts are eventually consistent: a fact written by one execution can take a few seconds to become visible to another. That is acceptable here because a speaker identity stays stable during a show.
One implementation detail is worth calling out, because it connects this section to the concurrency invariants. The container uses two distinct thread pools. One is sized on K for the calls to the agents, which wait on the network. The other is limited to four threads for model inference, which consumes CPU. Mixing them would place voice identification behind 24 in-flight network calls, and would blow its time budget on every segment.
Security and GDPR
A voice fingerprint vector used to uniquely identify a person is biometric data. GDPR places it in the special categories, whose processing is prohibited unless an explicit exception applies. What follows describes technical measures, not a legal qualification. Security on AWS rests on a shared responsibility model. AWS is responsible for security of the cloud, and provides tools such as encryption and fine-grained access management. The customer remains responsible for security in the cloud: configuration, access control, and the compliance of their own processing. AWS helps customers work toward their compliance goals, and no AWS service by itself makes a GDPR processing activity compliant.
Raw audio is never retained. The buffer holds 40 seconds of signal, is trimmed continuously, and is released in full when the session ends or the connection drops. Only the 256 dimension vector crosses the boundary into durable storage. That is data minimization applied as close as possible: the system keeps what it needs to compare, and nothing more.
The registry is encrypted at rest with an AWS managed key, and point-in-time recovery is enabled. Access is restricted by AWS Identity and Access Management policies to the Fargate task role and the enrolment function alone. No other component of the platform reads that data.
Every entry carries its traceability : provenance, indicating whether an operator enrolled the voice or the system built it automatically, creation date, last update date, and model version. That traceability serves accountability under Article 5, and makes it possible to invalidate a batch of entries if the model changes.
Deletion is a tested code path , not an intention. A voice can be deleted by identifier or by person name, which erases the aggregate entry, all of its contributions, and the references to its enrolment clips. After deletion, the system no longer produces the corresponding name or identifier.
What this prototype does not handle deserves to be stated as plainly. It does not manage consent collection or its traceability. It applies no automatic retention policy, so a voice stays in the registry until explicit deletion. It produces no record of processing activities. Those elements are required for a production deployment and sit outside the technical scope described here.
Measured results, and what is not achieved
On an extract of a televised debate used as a demonstration dataset, the system processed 24 speaker turns and told the two speakers apart, with no wrong attribution observed. On a measurement set of 11 annotated segments, 8 are named correctly and 3 stay anonymous because those voices are absent from the registry, which is the expected behavior. These samples are small, and they validate behavior rather than establish general performance.
Measurement on Fargate gives about 470 milliseconds on average and 991 milliseconds at the 95th percentile.
The breakdown explains why, and it is the most useful lesson of this work. The time splits into three parts of comparable weight: inference of the two models, signal preparation ahead of the models, and round trips to DynamoDB. There is no isolated hot spot. The fingerprint comparison itself, which you might suspect first, costs a tenth of a millisecond, because the registry holds a few hundred entries and the vectors are normalized.
I removed what could be removed without a tradeoff. The aggregate fingerprint recomputation moved off the critical path, and the read cache is no longer invalidated in full after each write. That gained about 25 percent on the average, and it is where the cheap wins stopped.
Conclusion
A newsroom that analyzes live can now attach each verified claim to a named speaker, and follow that speaker from one broadcast to the next. That is the benefit the voice fingerprint delivers over the diarization labels the platform started from, and the measurement in this post shows where the two diverge.
Recognizing a voice in a real-time stream asks less for algorithmic sophistication than for rigor on edge cases. The decisions that mattered most are not the choice of models. They are three tradeoffs: preferring abstention to error, preventing identification from blocking processing, and forbidding low quality fingerprints from writing to the registry.
The sensitivity of the data imposes a framework from the design stage. Keeping only the vector, encrypting it, restricting access, tracing provenance, and making deletion effective are measures to build in from the start, not to add afterwards.
To go further, read the Amazon Transcribe Developer Guide, and the Amazon DynamoDB Developer Guide on the conditional writes that make the concurrency invariants holdable. The Amazon Bedrock AgentCore documentation covers the session isolation the round robin relies on.
If you work on an adjacent problem, the first question to settle is not technical: what does your system do when it is not sure?
Top comments (0)