DEV Community: Jingyi

OpenAI Didn’t Publish GPT-Live’s Latency. So We Measured It.

Jingyi — Fri, 17 Jul 2026 04:51:47 +0000

On a real phone, GPT-Live was 500 ms slower to yield when interrupted, ignored background speech that fooled the older modes, and gave up 314 ms to packet loss that cost the previous generation 2.4 seconds.

Agora Media Lab · July 10, 2026
Methods and limits at the end.

OpenAI launched GPT-Live on July 8 promising AI powered voice conversations that feel natural; it listens while it speaks, waits while you think, and hands harder questions to a background model without leaving the conversation. What the launch didn’t include is any end-to-end number for what “natural” means on a real device.

These just so happen to be the numbers our lab’s engineers measure as part of their daily tests. We ran the latest ChatGPT app through our full protocol, from testing on real devices, to repeatable speech injected through an artificial mouth (actual lab equipment, not a metaphor), controlled network impairment, and every number read off of dual-track waveform recordings. We tested three generations of ChatGPT’s voice interface:

GPT-Live-1, the new full-duplex mode. It listens while it talks, like a phone call.
Advanced Voice Mode, the previous generation. A single speech-to-speech model, but strictly turn-based: it waits for you to finish before it responds.
Standard Voice Mode, the original pipeline: transcribe your speech, generate a text reply, read it aloud. Three hand-offs per turn.

To our knowledge, this is the first public controlled measurement of GPT-Live as users experience it: model, client, network, serving, and playback together.

Three results stood out. A fourth kept coming up in the lab but resisted measurement; it’s at the end.

500 ms to think: slower to stop, harder to fool

When you deliberately interrupt while the AI is speaking, GPT-Live takes the longest to go silent. It spends that time deciding whether you meant it:

Measured alone, that’s a 498 ms regression against Advanced, the turn-based predecessor, and getting talked over for an extra half second is the kind of thing you notice. But stop latency isn’t the only thing users experience. We also played sounds that should not count as interruptions:

The cough row is one event versus two, statistically a tie. The background-speech row is not: GPT-Live rejected all thirty probes; Advanced stopped for twenty; Standard stopped for every one. (In the same noise, all three modes understood the primary speaker, 30/30. The generations don’t differ in hearing you through noise; they differ in staying on course while speaking through it.) The old systems treat any sound as an interruption: fast, consistent, and frequently wrong. GPT-Live waits for enough evidence to tell a genuine barge-in from an acoustic event that happens to overlap its speech. (The 498 ms is an end-to-end number: decision policy, network round trip, playback buffer, and client stop behavior all contribute.)

The new policy also fails in a way the old reflexes never could. In a noisy room where nobody was addressing ChatGPT, GPT-Live spontaneously answered background voices in 4 of 30 ten-second windows; the older modes stayed silent. That’s a speaker-attribution problem: the system detects speech correctly but misjudges who it’s for. OpenAI’s help docs acknowledge the boundary.

The trade-off: GPT-Live takes longer to yield, it fell for 1 of our 60 false-interruption probes where Advanced fell for 22, and picks up a new risk of joining conversations that weren’t addressed to it.

GPT-Live’s real gain is 5× less jitter, not 5× more speed

Response latency, from the last frame of user speech to the first audible frame back:

You can write off Standard Voice Mode: the original transcribe-reply-speak pipeline is a museum piece at five seconds per turn, two product generations behind industry norms. The live comparison is GPT-Live against Advanced Voice Mode, and at the median GPT-Live wins by just 205 ms, hardly a generational leap. The tail is where the generation changed. Advanced’s P90 is 2,318 ms, nearly double its own median: one response in ten leaves you hanging past two seconds, and you never know which. GPT-Live’s P90 sits 104 ms above its median. Call it turn-taking jitter, borrowing the transport term for what users feel: GPT-Live barely moved the median, but it collapsed the variance around it. The spread shrank nearly fivefold (σ: 489 → 104 ms).

Your brain interprets that consistency as “natural.” Human turns transition in roughly 200 ms (Stivers et al., PNAS 2009); a response that lands on the same beat every time reads as thoughtful, while one that oscillates between 1.3 and 2.3 seconds reads as stalling. GPT-Live is nowhere near human speed, but it made the distance predictable.

“First audible frame” hides something, though: it includes acknowledgments. When a question needs search or reasoning, GPT-Live says “let me check that” while GPT-5.5 works in the background, so its first sound arrives early even when its answer doesn’t. The hard latency didn’t disappear; GPT-Live redistributed it into the conversation. Voice agents need three clocks: acknowledgment onset, substantive-answer onset, and answer completion. This round measures the first; the next will instrument all three.

10% packet loss breaks the old models and barely touches GPT-Live

We degraded the network to find out how much damage it takes to erase a generation of model progress. Turns out 10% packet loss does it with room to spare: GPT-Live’s P90 under loss (1,836 ms) still beat the P90 that turn-based Advanced managed on a clean network (2,318 ms). The degraded new model outperforms the healthy old one.

The medians say the same thing. Under 10% uplink loss, GPT-Live gives up 314 ms while Advanced gives up 2,448 ms, nearly tripling to within sight of Standard’s clean-network baseline. Swap the impairment to the downlink and add 100 ms of delay, and the ranking doesn’t move.

From outside the stack we can’t separate the three plausible contributors: an incremental architecture that consumes audio continuously, so a lost packet costs context rather than stalling a pipeline; transport and client differencesin codec, loss concealment, and jitter buffering across modes; and trainingon degraded audio. Assigning percentages would require internal telemetry, and we won’t pretend otherwise. For anyone building on these models the split matters less than the sum: the resilience ships with the product, whatever produced it.

The hidden requirement: full-duplex is a network contract

In a text chatbot, jitter is an inconvenience. In a full-duplex system, timing ismeaning: a pause is input, so network delay can masquerade as user hesitation; barge-in is a round trip, which is why stop latency is a systems number, not a model number; backchannels have a deadline, so predictableround-trip time matters as much as low; and loss concealment now serves two listeners, because audio smoothed for human ears may not preserve what a model needs.

A full-duplex model can’t deliver a natural experience alone: model, serving, transport, and client have to agree on time. OpenAI tunes all four because it owns the stack (see the infrastructure work it published this spring). Most teams own none of it; they consume a model API over the public internet and inherit every trade-off as their own problem. That gap is the layer Agora builds: a global real-time network in place of the public internet’s default paths, loss-resilient transport tuned for model consumption, and the client-side barge-in and AI-VAD behavior that decides how fast an agent shuts up when you talk over it. Disclosure: this is our business, so discount our emphasis accordingly. It’s also why we knew which probes to build; we’ve spent years tuning exactly these trade-offs.

What we felt but haven’t measured yet

Our test engineers kept coming back to one behavior rather than any single number: GPT-Live answers quickly, works on something else in the background, and never leaves dead air while doing both. The older modes went silent in exactly that gap. That’s the distance the three-clock framework above is meant to quantify: sounding responsive versus being done.

Three more consistent observations: GPT-Live handled our backchannels without losing its thread, where older modes treated a murmured “mhmm” as a new turn and changed topic. Its emotional register shifted naturally with context. And it showed a working sense of elapsed time in “remind me in 40 seconds” probes, though it occasionally missed; older modes fail these outright. All are candidates for the next protocol, alongside speaker attribution, code-switching, and long-session drift.

Putting it all together, on July 8 the bar was reset for every voice agent: roughly 1.1 s to acknowledge, 1.4 s to yield when interrupted, zero false stops under background speech, +314 ms under 10% packet loss. Those are the numbers users will now feel in anything they compare to it. (The GPT-Live API itself is still waitlist-only; the current Realtime API default, gpt-realtime-2.1, is a different model that we didn’t test.)

Methods and limits

Setup. iPhone 13, ChatGPT v1.2026.183 (App Store build), Plus account with GPT-Live-1 active. All speech in English. Tested July 9, 2026.

**Timing. **Prerecorded utterances played through an artificial mouth, so every trial hears identical input. Both sides recorded on separate tracks; every number read off the waveform, none from a stopwatch. n=30 per condition. Response latency: last frame of user speech to first audible frame of output. Stop latency: first frame of the barge-in to the last audible frame from the AI. Deliberate barge-ins succeeded 30/30 in every mode, which is why we report how fast each yields, not whether.

**Limits. **At n=30, don’t over-read single-digit differences, and treat P90s as descriptive. One device, one location, one account, launch week. One impairment level, and the downlink test mixed loss with delay. Treat this as a snapshot, not a census.

Next. We’ll run the same protocol on other realtime voice models as they ship.

The full-duplex era has started, and now we can measure it.

Originally published on the [Agora blog](https://www.agora.io/en/blog/openai-didnt-publish-gpt-lives-latency-so-we-measured-it/)

Introducing the Agora CLI

Jingyi — Fri, 17 Jul 2026 04:45:08 +0000

A command-line interface for developers and AI coding agents building with Agora.

AI coding agents can create apps, but the initial setup still requires human intervention: creating a project, retrieving project credentials, creating the env file without exposing secrets to the agent context, and choosing the right quickstart so it doesn't generate a bunch of boilerplate code. Developers hit the same loop. You open the console, copy values, switch back to the terminal, create the env file, then search for the quickstart link so the agent has a good example to start with.

Today we're releasing the Agora CLI to move that setup path into the terminal. Developers, scripts, and agents can use the CLI to write environment files, scaffold starter repos, and check Agora environment readiness without leaving the workflow they're already in. Most CLI tools print human-readable output, which leaves coding agents parsing loose terminal text. The Agora CLI supports both humans and agents, with a JSON output mode that returns structured responses an agent can parse and act on. We built the CLI for our own agentic development workflows. Now we're releasing it so Agora developers can give their agents the same ability to provision projects, write env files, and keep setup inside the terminal.

The shortest path from install to runnable project is three commands:

curl -fsSL https://agoraio.github.io/cli/install.sh | sh
agora login
agora init my-python-demo --template python

agora init creates or binds an Agora project, clones the selected quickstart, writes the env file that quickstart expects, records the repo-local binding in .agora/project.json, and prints the next steps. Templates ship for Next.js, Python, and Go.

The Command Surface

The CLI separates account-level actions from local project setup:

agora
├── init        # create or bind a project, clone a quickstart, write env
├── project     # projects, features, credentials, readiness
├── quickstart  # starter repos and template-aware env files
├── auth        # login, logout, session status
├── doctor      # install and project diagnostics
└── mcp         # run the CLI as a local MCP server

Use init when you want the whole onboarding path in one command. Drop into project and quickstart when you want to inspect or control each step:

agora project create my-agent-demo --feature rtc --feature convoai
agora quickstart create my-python-demo --template python --project my-agent-demo
agora quickstart env write my-python-demo --project my-agent-demo

The CLI also covers the daily checks developers reach for:

agora -v         # show the installed version
agora help       # learn commands and flags
agora whoami     # check the current login
agora doctor     # check the local install and project setup
agora upgrade    # upgrade the CLI

The CLI keeps the quickstart, credentials, and project binding aligned. It writes the env file the selected template expects, preserves values you've already added, and remembers which Agora project belongs to the repo so future commands don't drift into the wrong setup. Under the hood, the CLI stores repo-local project metadata in .agora/project.json. When you run another command from that repo, the CLI uses that binding before falling back to a global project selection.

Built for Agents, Useful for Humans

Agora Skills gives an AI coding assistant the context to understand Agora's products and integration patterns. The CLI gives that assistant a way to act.

That matters because setup work is full of external dependencies: project configurations, env file setup, feature readiness, quickstart bindings, and local diagnostics. A coding agent can write source code from documentation, but it can't complete a real setup unless it can also ask the platform what exists and change what needs to change.

For agent and automation workflows, use --json to return structured data instead of human-formatted terminal output:

agora doctor --json
agora auth status --json
agora project env --json
agora introspect --json

In JSON mode, commands return fields a script or agent can inspect directly, such as auth state, project metadata, env values, diagnostics, and command capabilities.

agora introspect --json exposes the command tree in a machine-readable form, including which commands are safe for headless runs. agora mcp serve lets MCP clients call the CLI through local tools instead of scraping terminal text.

The console remains the visual control plane. The CLI gives developers, scripts, and agents a reliable path through the same setup work from the terminal.

Install

curl -fsSL https://agoraio.github.io/cli/install.sh | sh
On Windows, install with PowerShell:

irm https://agoraio.github.io/cli/install.ps1 | iex
The installer configures PATH and shell completion by default.

After installing, agora doctor checks the local CLI setup, including the resolved binary, version, PATH, network access, auth state, and project readiness.

Get Started

After you install the CLI, and run agora login, you can start from one of the official quickstarts:

agora init my-python-demo --template python
agora init my-nextjs-demo --template nextjs
agora init my-go-demo --template go

Full CLI documentation is available at Agora CLI Docs.

This is the first release. If a command is missing or a setup path still forces you back into manual project work, open an issue. That feedback should shape the next version.

Originally published on the Agora blog

Agora Agents SDK: Build Voice Agents in Minutes

Jingyi — Fri, 17 Jul 2026 04:25:54 +0000

Introducing Agora Agents SDK, a typed SDK for building voice agents on Agora's Conversational AI Engine in Python, Node.js, and Go.

Voice agents have moved past demos. Teams are putting them in support flows, onboarding, scheduling, field operations, and internal tools where a delayed response or missed interruption feels broken.

Today we're releasing Agora Agents SDK: typed builders, built-in auth, token generation, and session management on top of the existing REST API. It ships for Python, Node.js / TypeScript, and Go.

The REST API remains fully supported. The SDK sits on top of it.

What the SDK Handles

You configure the agent in typed code instead of assembling payloads by hand.

Typed pipeline builders. Build voice agents in Python, TypeScript and Go. Compose your voice agent configuration with speech-to-text using .with_stt(), the LLM using .with_llm(), and text-to-speech using .with_tts(). For real-time multimodal models, use the MLLM builder path .with_mllm() for providers such as OpenAI Realtime and Gemini Live.
Dynamic token generation. Pass your Agora App ID and App Certificate once. The SDK generates fresh auth tokens for each agent request, so developers don't have to build or manage token logic themselves. When you expose agent management endpoints for web or mobile apps, the SDK can also help configure the token server your clients use to join Agora channels.
Session lifecycle. Start, stop, query, and recover sessions through SDK objects instead of keeping raw agent IDs, status values, and retry logic scattered through your backend.
Python, Node.js / TypeScript, and Go support. The SDKs use the same agent model, with APIs shaped for each language instead of one generic wrapper.

Install the SDK

Use your preferred package manager to install the SDK:

pip install agora-agents
npm install agora-agents
go get github.com/AgoraIO/agora-agents-go

Quick Start

This is a basic Python voice-agent pipeline: STT -> LLM -> TTS.

import time

from agora_agent import Agora, Agent, Area
from agora_agent.utils import expires_in_hours
from agora_agent.agentkit import DeepgramSTT, MiniMaxTTS, OpenAI

client = Agora(
    area=Area.US,
    app_id=APP_ID,
    app_certificate=APP_CERTIFICATE,
)

agent = (
    Agent(client=client, turn_detection={"language": "en-US"})
    .with_stt(
        DeepgramSTT(
            model="nova-3",
            language="en",
        )
    )
    .with_llm(
        OpenAI(
            model="gpt-4o-mini",
            system_messages=[{"role": "system", "content": AGENT_PROMPT}],
            greeting_message=GREETING,
            failure_message="Please wait a moment.",
            max_history=50,
            params={
                "max_tokens": 1024,
                "temperature": 0.7,
                "top_p": 0.95,
            },
        )
    )
    .with_tts(
        MiniMaxTTS(
            model="speech_2_6_turbo",
            voice_id="English_captivating_female1",
        )
    )
)

session = agent.create_session(
    channel=f"demo-channel-{int(time.time())}",
    agent_uid="123456",
    remote_uids=["*"],
    name=f"conversation-{int(time.time())}",
    idle_timeout=30,
    expires_in=expires_in_hours(1),
    debug=False,
)

return session.start()

That code starts an agent and joins it to an Agora RTC channel. The builder keeps the agent configuration readable: turn detection, providers, model behavior, channel, session name, timeout, and expiry all stay in one place. The same setup can carry model parameters, history limits, and debug settings when you need them.

If You Already Use the REST API

Nothing breaks. The REST API remains the foundation of the platform, and it is still a good fit for teams that need direct HTTP control or use a backend language without an SDK.

For Python, Node.js, and Go services, the SDK removes the code most teams don't want to own:

manual token signing
large config objects assembled across files
repeated start/stop boilerplate
one-off retry and timeout handling around session startup

You can migrate one boundary at a time. Keep the rest of your backend as it is, replace the raw setup and lifecycle calls with the SDK, and compare the code side by side.

Your Models, Agora's Real-Time Infrastructure

The SDK doesn't lock you into one model stack. Bring your own STT, LLM, and TTS providers, or use Agora-managed models when you want fewer vendor credentials in your app. You keep control of the model choices and prompts that define the agent. Agora runs the real-time media path, interruption behavior, and channel lifecycle users feel the moment they start talking.

We built Agora Agents SDK so the first working agent doesn't require a pile of glue code. The SDK keeps the setup close to the way you think about the agent: models, prompts, voice, channel, and session behavior.

Source and examples for Python, Node.js/Typescript and Go are on Docs:
https://docs.agora.io/en/ai/build/start-stop-agent#install-the-sdk

Build an agent, put it in front of a real conversation, and tell us where the SDK still makes you think about plumbing. That feedback shapes the next version.

‍Original link:https://www.agora.io/en/blog/agora-agents-sdk-build-voice-agents-in-minutes/

Original author:Hermes Frangoudis

While Learning LangChain, I Ended Up With This Realtime Voice Pattern

Jingyi — Fri, 26 Jun 2026 11:46:09 +0000

I've recently started learning LangChain while exploring different ways to build voice agents.

As someone who's still new to LangChain, I tried a few different approaches before landing on one that felt surprisingly clean.

I don't know whether this is already a common pattern in the LangChain ecosystem, but it worked well enough that I thought it was worth sharing. Hopefully it can also spark some discussion with people who've been building voice agents for longer.

The idea

The basic idea is to separate responsibilities instead of letting one framework handle everything.

In this pattern:

LangChain stays responsible for tool selection, tool execution, and response composition.
A separate realtime runtime handles RTC / RTM, speech input/output, and session lifecycle.
The two communicate through an OpenAI-compatible endpoint.

What surprised me

When I first started experimenting, I assumed adding realtime voice would require restructuring most of the application.

Instead, it felt more like adding another interaction layer.

LangChain continues doing what it already does well—tool orchestration and workflow management—while the realtime runtime focuses on voice-specific concerns.

That separation made the overall architecture feel much simpler.

The pattern

To better understand the idea, I put together a small recipe that demonstrates this integration pattern.

The recipe consists of three pieces:

Python exposes an OpenAI-compatible endpoint and manages the agent lifecycle.
Next.js handles the client-side realtime interaction.
LangChain remains server-side as the orchestration and tool layer.

It's not intended to be a complete application or a production-ready project.

It's simply a pattern that helped me understand how these pieces can fit together.

A quick note

For transparency, I work at Agora, so I naturally used Agora as the realtime runtime while putting this recipe together.

That said, I'm much more interested in learning how other people are building voice agents with LangChain.

If you're using a different stack or have taken a completely different approach, I'd love to hear about it.

The recipe

If you're curious, here's the recipe:

https://github.com/bluemotional/recipe-agent-langchain

Since I'm still learning LangChain myself, I'd really appreciate any feedback or suggestions.

What's next?

If people find this pattern useful, I'd like to keep expanding it with more recipes, for example:

RAG
Internal tools
Docs Copilot
MCP integrations

I'm also curious to see what other patterns the community has found useful.

Project Spotlight: An Open-Source AI Sports Commentator That Watches the Match Live

Jingyi — Fri, 26 Jun 2026 01:35:29 +0000

I recently came across an open-source project called WorldCupVoice and thought it was worth sharing.

Quick note first: I’m not the creator.

The project is by zicojiao.

The demo makes the idea click immediately: this is not a post-game recap system. It watches a live match feed and speaks over the action in real time.

Project links:

What makes this project interesting is that it is not a “generate a match summary later” system.

It is a live pipeline:

a match feed comes in through RTMP
Agora converts it into a real-time RTC stream
the backend samples frames from the live video
OpenAI generates grounded play-by-play commentary
TTS turns that into speech
the generated audio is published back into the same RTC room

What I found especially interesting from reading the README and code is that the project is clearly trying to solve a harder problem than image captioning.

The prompts appear to push the model toward a live football commentator role by constraining it to:

comment on visible action rather than narrate loosely
avoid inventing off-screen goals or fouls
use player names only when visual evidence is strong enough
return NO_CALL when the frame is not readable enough

Another thing I liked was that the project seems to take operational reality seriously.

It includes explicit Start AI / Stop AI controls, viewer heartbeat, and a hard session TTL, which makes a lot of sense for a live AI system where cost can keep growing even when nobody is watching.

The World Cup framing is fun, but the accessibility angle might be even more compelling.

A system like this could potentially add more field-level spatial detail for blind and low-vision fans than traditional commentary often does.

Again, I’m not the author, so if I’m misreading any implementation detail, happy to be corrected. But as an open-source example of a real-time multimodal media pipeline, this was one of the more interesting projects I’ve come across recently.

Full credit to zicojiao for the project itself. I’m only sharing it because I found it worth discussing.