A real-time voice companion can behave correctly while its test suite reports failure.
The usual culprit is not that “the LLM knows better.” It is that the test encoded an assumption the product never promised: an exact sentence, a fixed callback order, or playback beginning within an arbitrary sleep interval.
That creates an uncomfortable engineering tension. You do not want to dismiss failures as harmless AI variability, but you also do not want brittle tests blocking valid behavior.
A better approach is to test the conversation protocol deterministically and evaluate the conversation content separately. For a Tencent RTC conversational AI application, that means verifying facts such as:
- User speech invalidates the previous turn.
- An interrupted response cannot resume later.
- Stale ASR, LLM, or playback callbacks cannot take control of the session.
- A timeout produces visible recovery instead of indefinite silence.
- Every asynchronous result remains correlated with its originating turn and request.
We will build that test oracle in TypeScript, reproduce the dangerous races, and then map it onto the live voice pipeline.
Start with the disputed failure
Consider this test:
expect(answer).toBe("Your meeting begins at 3 PM.");
The model returns:
The meeting starts at 3 PM.
The assertion failed, but that alone does not establish a product defect. The test mixed three different questions:
- Protocol: Did the response belong to the current user turn?
- Policy: Was the assistant permitted to answer?
- Content: Was the answer sufficiently correct and useful?
Only the last question involves linguistic variability. The first two should remain deterministic.
The human decision is therefore not “trust the model or trust the test.” It is: which product contract was this test supposed to enforce?
Draw the asynchronous boundary
A production voice companion is not one AI call. Keep these responsibilities separate:
Microphone / RTC media
↓
Speech recognition
↓
Application turn coordinator
↓
LLM
↓
Speech synthesis
↓
RTC playback
The application coordinator owns the current turn. Neither a late model response nor a delayed playback callback gets to decide that it is still relevant.
Tencent RTC documents Conversational AI as a real-time voice interaction scenario that can connect users with multiple LLM providers. Its LLM configuration guidance also covers OpenAI-compatible models and request identifiers used for routing and observability:
The event and effect names below belong to our application. They are not Tencent RTC SDK method names.
Create the reproducible project
mkdir voice-protocol-oracle
cd voice-protocol-oracle
npm init -y
npm install --save-dev typescript vitest tsx @types/node
npm pkg set type=module
npm pkg set scripts.test="vitest run"
mkdir src
Create src/session.ts:
export type Phase =
| "idle"
| "userSpeaking"
| "transcribing"
| "thinking"
| "preparingAudio"
| "speaking"
| "recovery"
| "offline";
export type State = {
phase: Phase;
turn: number;
requestId?: string;
};
export type Event =
| { type: "USER_STARTED" }
| { type: "USER_ENDED" }
| {
type: "TRANSCRIPT_READY";
turn: number;
requestId: string;
text: string;
}
| { type: "TRANSCRIPT_FAILED"; turn: number }
| {
type: "MODEL_DONE";
turn: number;
requestId: string;
text: string;
}
| { type: "MODEL_TIMEOUT"; turn: number; requestId: string }
| { type: "PLAYBACK_STARTED"; turn: number; requestId: string }
| { type: "PLAYBACK_ENDED"; turn: number; requestId: string }
| { type: "PLAYBACK_FAILED"; turn: number; requestId: string }
| { type: "DISCONNECTED" };
export type Effect =
| { type: "REQUEST_TRANSCRIPT"; turn: number }
| {
type: "REQUEST_MODEL";
turn: number;
requestId: string;
text: string;
}
| {
type: "SPEAK";
turn: number;
requestId: string;
text: string;
}
| { type: "CANCEL_AUDIO"; turn: number; requestId: string }
| { type: "NOTICE"; message: string }
| { type: "LOG_STALE"; source: string; turn: number; requestId?: string };
export const initialState: State = {
phase: "idle",
turn: 0,
};
function isCurrent(
state: State,
turn: number,
requestId?: string,
): boolean {
return (
state.turn === turn &&
(requestId === undefined || state.requestId === requestId)
);
}
function stale(
state: State,
source: string,
turn: number,
requestId?: string,
): [State, Effect[]] {
return [
state,
[{ type: "LOG_STALE", source, turn, requestId }],
];
}
export function reduce(
state: State,
event: Event,
): [State, Effect[]] {
switch (event.type) {
case "USER_STARTED": {
if (state.phase === "userSpeaking") return [state, []];
const effects: Effect[] = [];
if (
(state.phase === "preparingAudio" || state.phase === "speaking") &&
state.requestId
) {
effects.push({
type: "CANCEL_AUDIO",
turn: state.turn,
requestId: state.requestId,
});
}
return [
{
phase: "userSpeaking",
turn: state.turn + 1,
},
effects,
];
}
case "USER_ENDED": {
if (state.phase !== "userSpeaking") return [state, []];
return [
{ ...state, phase: "transcribing" },
[{ type: "REQUEST_TRANSCRIPT", turn: state.turn }],
];
}
case "TRANSCRIPT_READY": {
if (
state.phase !== "transcribing" ||
!isCurrent(state, event.turn)
) {
return stale(state, "transcript", event.turn, event.requestId);
}
if (!event.text.trim()) {
return [
{ ...state, phase: "recovery" },
[{ type: "NOTICE", message: "I did not catch that. Please try again." }],
];
}
return [
{
...state,
phase: "thinking",
requestId: event.requestId,
},
[
{
type: "REQUEST_MODEL",
turn: event.turn,
requestId: event.requestId,
text: event.text,
},
],
];
}
case "TRANSCRIPT_FAILED": {
if (state.phase !== "transcribing" || !isCurrent(state, event.turn)) {
return stale(state, "transcript-error", event.turn);
}
return [
{ ...state, phase: "recovery" },
[{ type: "NOTICE", message: "Speech recognition is unavailable. Try again." }],
];
}
case "MODEL_DONE": {
if (
state.phase !== "thinking" ||
!isCurrent(state, event.turn, event.requestId)
) {
return stale(state, "model", event.turn, event.requestId);
}
if (!event.text.trim()) {
return [
{ ...state, phase: "recovery" },
[{ type: "NOTICE", message: "No answer was produced. Please retry." }],
];
}
return [
{ ...state, phase: "preparingAudio" },
[
{
type: "SPEAK",
turn: event.turn,
requestId: event.requestId,
text: event.text,
},
],
];
}
case "MODEL_TIMEOUT": {
if (
state.phase !== "thinking" ||
!isCurrent(state, event.turn, event.requestId)
) {
return stale(state, "model-timeout", event.turn, event.requestId);
}
return [
{ ...state, phase: "recovery" },
[{ type: "NOTICE", message: "The response took too long. Please try again." }],
];
}
case "PLAYBACK_STARTED": {
if (
state.phase === "preparingAudio" &&
isCurrent(state, event.turn, event.requestId)
) {
return [{ ...state, phase: "speaking" }, []];
}
// Playback may have crossed an interruption or cancellation boundary.
// Stop it instead of accepting its late callback.
return [
state,
[
{
type: "CANCEL_AUDIO",
turn: event.turn,
requestId: event.requestId,
},
{
type: "LOG_STALE",
source: "playback-start",
turn: event.turn,
requestId: event.requestId,
},
],
];
}
case "PLAYBACK_ENDED": {
if (
state.phase !== "speaking" ||
!isCurrent(state, event.turn, event.requestId)
) {
return stale(state, "playback-end", event.turn, event.requestId);
}
return [{ phase: "idle", turn: state.turn }, []];
}
case "PLAYBACK_FAILED": {
if (
!["preparingAudio", "speaking"].includes(state.phase) ||
!isCurrent(state, event.turn, event.requestId)
) {
return stale(state, "playback-error", event.turn, event.requestId);
}
return [
{ ...state, phase: "recovery" },
[{ type: "NOTICE", message: "Audio playback failed. A text answer can be shown." }],
];
}
case "DISCONNECTED": {
const effects: Effect[] = [];
if (
(state.phase === "preparingAudio" || state.phase === "speaking") &&
state.requestId
) {
effects.push({
type: "CANCEL_AUDIO",
turn: state.turn,
requestId: state.requestId,
});
}
effects.push({ type: "NOTICE", message: "Connection lost. Reconnect to continue." });
return [{ ...state, phase: "offline" }, effects];
}
}
}
There are two important design choices here.
First, turn changes as soon as new user speech begins. That immediately invalidates work from the previous turn.
Second, a playback callback is not automatically trusted. If audio starts after its turn was interrupted, the coordinator issues another cancellation rather than moving back into speaking.
Make the races executable
Create src/session.test.ts:
import { describe, expect, it } from "vitest";
import {
type Effect,
type Event,
initialState,
reduce,
} from "./session.js";
function scenario() {
let state = initialState;
return {
step(event: Event): Effect[] {
const [next, effects] = reduce(state, event);
state = next;
return effects;
},
state() {
return state;
},
};
}
describe("voice turn protocol", () => {
it("rejects an LLM result from an interrupted turn", () => {
const s = scenario();
s.step({ type: "USER_STARTED" });
s.step({ type: "USER_ENDED" });
s.step({
type: "TRANSCRIPT_READY",
turn: 1,
requestId: "req-1",
text: "Tell me a story",
});
// The user begins a replacement turn before req-1 completes.
s.step({ type: "USER_STARTED" });
s.step({ type: "USER_ENDED" });
s.step({
type: "TRANSCRIPT_READY",
turn: 2,
requestId: "req-2",
text: "Actually, stop",
});
const effects = s.step({
type: "MODEL_DONE",
turn: 1,
requestId: "req-1",
text: "Once upon a time...",
});
expect(effects.some((effect) => effect.type === "SPEAK")).toBe(false);
expect(s.state()).toMatchObject({
phase: "thinking",
turn: 2,
requestId: "req-2",
});
});
it("cancels playback that starts after a barge-in", () => {
const s = scenario();
s.step({ type: "USER_STARTED" });
s.step({ type: "USER_ENDED" });
s.step({
type: "TRANSCRIPT_READY",
turn: 1,
requestId: "req-1",
text: "Say hello",
});
s.step({
type: "MODEL_DONE",
turn: 1,
requestId: "req-1",
text: "Hello!",
});
const interruption = s.step({ type: "USER_STARTED" });
expect(interruption).toContainEqual({
type: "CANCEL_AUDIO",
turn: 1,
requestId: "req-1",
});
const lateStart = s.step({
type: "PLAYBACK_STARTED",
turn: 1,
requestId: "req-1",
});
expect(lateStart).toContainEqual({
type: "CANCEL_AUDIO",
turn: 1,
requestId: "req-1",
});
expect(s.state().phase).toBe("userSpeaking");
});
it("does not speak a response that arrives after timeout", () => {
const s = scenario();
s.step({ type: "USER_STARTED" });
s.step({ type: "USER_ENDED" });
s.step({
type: "TRANSCRIPT_READY",
turn: 1,
requestId: "req-1",
text: "What should we do today?",
});
s.step({
type: "MODEL_TIMEOUT",
turn: 1,
requestId: "req-1",
});
const lateResult = s.step({
type: "MODEL_DONE",
turn: 1,
requestId: "req-1",
text: "Let us go for a walk.",
});
expect(lateResult.some((effect) => effect.type === "SPEAK")).toBe(false);
expect(s.state().phase).toBe("recovery");
});
it.each([
"The meeting starts at 3 PM.",
"Your meeting begins at 3 PM.",
])("accepts a protocol-valid wording: %s", (answer) => {
const s = scenario();
s.step({ type: "USER_STARTED" });
s.step({ type: "USER_ENDED" });
s.step({
type: "TRANSCRIPT_READY",
turn: 1,
requestId: "req-1",
text: "When is my meeting?",
});
const effects = s.step({
type: "MODEL_DONE",
turn: 1,
requestId: "req-1",
text: answer,
});
expect(effects).toContainEqual({
type: "SPEAK",
turn: 1,
requestId: "req-1",
text: answer,
});
});
});
Run the suite:
npm test
These tests do not claim that both meeting answers are factually correct. They establish that wording variation is not a turn-protocol failure. Factual correctness needs its own evidence and oracle.
Use three test oracles, not one giant assertion
A practical test strategy separates concerns:
| Layer | Suitable oracle | Example failure |
|---|---|---|
| Turn protocol | Deterministic state and effect assertions | Interrupted turn begins playback |
| Safety and policy | Rules, schemas, allowlists, and human-approved cases | Companion gives disallowed advice |
| Open-ended quality | Rubrics, reference facts, sampled human review | Answer is confusing despite being allowed |
An LLM-based evaluator can assist the third layer, but it should not silently become the authority for the first two. If an evaluator says a response is acceptable while the response belongs to a stale turn, the application must still reject it.
Likewise, when a test and an LLM disagree, inspect the contract before changing either one:
- Is the expected behavior explicitly documented?
- Is the assertion about protocol, policy, or prose?
- Would multiple outputs satisfy the user’s intent?
- Could the test pass for the wrong reason?
- Does accepting the output weaken a safety or control boundary?
This turns “the model was right” from a rhetorical conclusion into a reviewable engineering decision.
Connect the effects to the live voice path
Keep vendor and network operations behind ports:
export interface VoicePorts {
requestTranscript(turn: number): Promise<void>;
requestModel(input: {
turn: number;
requestId: string;
text: string;
}): Promise<void>;
speak(input: {
turn: number;
requestId: string;
text: string;
}): Promise<void>;
cancelAudio(input: {
turn: number;
requestId: string;
}): Promise<void>;
showNotice(message: string): void;
}
Your effect dispatcher translates application effects into the configured speech recognition, LLM, synthesis, and RTC operations. The corresponding callbacks are translated back into reducer events.
For the LLM path, preserve the same application request identifier through the configured integration where supported. The Tencent RTC LLM configuration documentation is the implementation reference for connecting an OpenAI-compatible model or an agent platform such as Dify or Coze.
Do not let the adapter mutate session state directly. Its job is to report outcomes such as MODEL_DONE, MODEL_TIMEOUT, or PLAYBACK_FAILED. The reducer decides whether those outcomes are still current.
Latency needs phase measurements
A single “response latency” number cannot tell you where a turn stalled. Record timestamps around boundaries such as:
type TurnTiming = {
turn: number;
requestId: string;
userEndedAt?: number;
transcriptReadyAt?: number;
modelRequestedAt?: number;
modelCompletedAt?: number;
playbackRequestedAt?: number;
playbackStartedAt?: number;
};
From those fields, derive separate measurements for:
- End of user speech to transcript readiness
- Model request to model completion
- Speech preparation request to audible playback
- Total end-of-speech to first playback
Choose timeout budgets from measurements in your own supported environments. Do not hide a slow phase behind automatic retries: a retry can produce a second, even later answer after the user has moved on.
The user-visible fallback should also match the failed phase. “I didn’t catch that” is appropriate for transcription failure but misleading when the model timed out.
Failure drills for a real companion
Unit tests prove reducer behavior. Staging drills prove that adapters preserve it.
Delayed ASR callback
Start one utterance, begin another, and release the first transcript afterward.
Expected: The old transcript is logged as stale and never reaches the LLM.
LLM completion after recovery is shown
Delay model completion beyond the application deadline.
Expected: Recovery remains visible. The late answer is not synthesized.
Playback starts after cancellation
Introduce a delay between requesting synthesis and receiving the playback-start callback. Interrupt during that gap.
Expected: The late audio is cancelled immediately. State remains with the user’s new turn.
Playback fails after valid model output
Make speech output unavailable while retaining the model result.
Expected: The interface exposes a clear fallback, such as displaying the text. It must not pretend the answer was spoken.
Connection loss during assistant speech
Disconnect the RTC path while audio is active.
Expected: Local playback is stopped, the session becomes visibly offline, and reconnection requires an explicit application policy.
Duplicate callback delivery
Deliver MODEL_DONE or PLAYBACK_ENDED twice.
Expected: The duplicate cannot trigger a second spoken answer or advance a later turn.
Preserve human control in the interface
A social or companion experience should make its AI nature clear and provide visible controls for muting, interrupting, ending the session, and managing any retained transcript data. Moderation and privacy decisions belong to the application design; fluent output is not evidence that those decisions are safe.
For character dialogue, AI companions, voice rooms, and related use cases, Tencent RTC also describes the broader Social Entertainment solution. Treat that page as scenario context, while using the product documentation for implementation details.
Release verification checklist
Before shipping, verify that:
- [ ] Every turn receives an application-owned monotonic turn number.
- [ ] Every model request has a correlation identifier.
- [ ] A new user utterance invalidates previous asynchronous work.
- [ ] Stale model output cannot reach synthesis.
- [ ] Late playback is actively cancelled, not merely ignored in state.
- [ ] Transcription, model, and playback failures have distinct recovery messages.
- [ ] Timeouts leave the session in a defined state.
- [ ] Duplicate callbacks are harmless.
- [ ] Protocol tests do not require one exact natural-language sentence.
- [ ] Policy tests remain deterministic where possible.
- [ ] Open-ended quality judgments have a documented rubric and human review path.
- [ ] Users can interrupt, mute, and leave without asking the AI for permission.
The useful reframing is simple: a failed test is evidence of disagreement, not automatic proof that either the implementation or the model is correct. In a live voice application, the strongest tests are the ones that defend turn ownership and human control while leaving room for valid language variation.
Relationship disclosure: This article was produced in connection with Tencent RTC. Official Tencent RTC documentation was used as the implementation reference.
Discussion
Which assertion in your current AI test suite is really a protocol guarantee—and which one is only an accidental preference for a particular wording?
Top comments (0)