DEV Community: Stephen568hub

How AI Agents Are Empowering Voice Social Games

Stephen568hub — Thu, 09 Apr 2026 07:18:08 +0000

Voice social games such as Werewolf, Avalon, and party voice rooms have always been built around real-time interaction. The excitement comes from live discussion, strategic reasoning, and the emotional tension between players. However, one long-standing challenge has limited user retention for years: many users leave before the game can even start because there are not enough players in the room.

This is exactly where AI agents are beginning to reshape the experience.

Why Voice Social Games Need AI Agents

Traditional voice games rely heavily on human concurrency. A room often requires six or more players before the session can begin, which makes off-peak usage particularly challenging. If one or two users are missing, the entire room stalls, and this “failed room start” moment often becomes a major churn point.

AI agents solve this by instantly filling the missing roles. Instead of waiting for enough players to join, a single real player can immediately start a match against multiple AI participants. This transforms the room from a concurrency-dependent experience into an always-available one, which significantly improves room start rates and keeps users engaged.

For social products, this directly impacts retention and session completion.

How AI Enhances the Gameplay Experience

AI in voice games is not only about filling empty seats. It can actively improve the overall gameplay experience.

For example, an AI agent can act as a host, explain the rules, manage speaking order, control voting rounds, and keep the pace of the room smooth. In social deduction scenarios, AI can also participate as real players, analyzing the conversation context, inferring hidden roles, and making strategic decisions based on what other users are saying.

This creates a much more dynamic experience than traditional scripted NPCs.

In addition, if a player drops out in the middle of a session, the AI can immediately replace that role and preserve the continuity of the game, preventing the frustration that often leads to user drop-off.

Why Real-Time Performance Is Critical

For voice social games, timing is part of the product experience.

If an AI player responds three seconds after being asked a question, the room immediately feels artificial. The rhythm of the conversation breaks, and users quickly lose immersion.

This is why the full pipeline must be optimized for low latency.

The typical workflow includes speech recognition, reasoning, response generation, and voice playback. All of this must happen in near real time to maintain the natural pace of live conversation.

How ZEGOCLOUD Supports AI Voice Games

For voice-based social games, latency is not just a technical metric. It directly affects gameplay.

In scenarios like Werewolf or Avalon, if the AI responds a few seconds late, the conversation flow immediately feels unnatural. This is why AI for voice games requires more than a standard LLM API call.

Low-Latency Real-Time Interaction

ZEGOCLOUD AI Agent combines LLM + ASR/TTS + RTC into one real-time pipeline, allowing AI players and hosts to respond in under one second.

This helps AI conversations feel much closer to real human pacing, which is essential for live discussion and deduction gameplay.

Multi-Agent Room Architecture

Voice games often require multiple roles at the same time.

With ZEGOCLOUD, developers can support scenarios such as:

one player + multiple AI teammates
multiple users + AI opponents
AI DM / game host
AI replacing dropped users

This makes it possible to start rooms instantly, even with only one real player.

Better Audio Quality for Live Rooms

Real-world voice rooms are often noisy.

To improve clarity, ZEGOCLOUD provides built-in capabilities such as:

AI noise suppression
echo cancellation
voice activity detection

This ensures both players and AI agents can clearly hear each other.

Flexible AI Role Design

Different AI roles can be configured with different personalities, strategies, and speaking styles.

For example, one AI player can behave aggressively while another follows a more conservative deduction style. This helps make each game session feel less repetitive and more dynamic.

Final Thoughts

AI agents are becoming a practical way to improve retention, reduce operational costs, and make voice social games playable at any time.

With solutions like ZEGOCLOUD AI Agent, developers can move beyond human-only room dependency and build experiences where AI and human players interact naturally in real time.

This may well define the next generation of social voice gaming.

What is an RTC SDK? A Developer-Friendly Guide

Stephen568hub — Fri, 27 Mar 2026 06:29:46 +0000

Real-time communication has become a core part of many modern applications, from video meetings and online classes to voice chat, telehealth, and interactive live streaming. For developers, building these experiences is not just about sending audio and video over the internet. It also involves handling latency, media processing, network adaptation, and cross-platform delivery in a reliable way.

This is where an RTC SDK becomes important. It gives developers a faster way to build real-time communication features without having to create the entire media stack from scratch. In this guide, we’ll explain what an RTC SDK is, how RTC differs from traditional streaming, and which core capabilities matter most when evaluating a real-time communication solution.

What Is RTC?

RTC stands for Real-Time Communication, a category of technology designed for scenarios where audio, video, or data must be transmitted with extremely low delay. The key idea is not simply that media is being delivered over the internet, but that it arrives fast enough for users to interact naturally. In a real-time conversation, even a small increase in latency can make turn-taking feel awkward, interrupt the flow of discussion, or reduce the sense of immediacy that users expect.

This is what separates RTC from traditional streaming. A standard video platform can tolerate buffering because the viewer is mostly consuming content passively, but RTC applications are interactive by nature. In products such as video conferencing, online education, voice chat, telehealth, customer support, or live collaboration, the system has to respond almost instantly to keep the experience usable.

Scenario Typical Latency Requirement
Video conferencing < 300 ms
Interactive online education < 400 ms
In-game voice chat < 200 ms
Remote surgery or industrial control < 50 ms

Because of these strict latency requirements, RTC systems must be designed very differently from traditional media delivery systems. They need to optimize every stage of the communication pipeline, from media capture and encoding to transmission, decoding, and playback, so that the interaction still feels live even under changing network conditions.

What Is an SDK?

An SDK, or Software Development Kit, is a packaged set of tools that helps developers build specific features without starting from zero. It usually includes code libraries, APIs, documentation, sample projects, and debugging tools that simplify implementation. Rather than building every low-level function themselves, developers can use the SDK as a foundation and focus more on product logic, interface design, and user experience.

When applied to real-time communication, this becomes especially valuable because RTC is much harder to build than it first appears. A simple video call feature is not just about capturing a camera feed and sending it over the network. It also requires codec support, packet handling, weak-network adaptation, device compatibility, audio processing, synchronization logic, and room management. An RTC SDK packages these layers into a more accessible form, which is why it has become the standard approach for teams building communication products.

What Is an RTC SDK?

Put simply, an RTC SDK is a development toolkit that packages real-time audio and video communication capabilities into reusable components. It allows developers to integrate features such as voice calls, video meetings, live broadcasting, and interactive rooms without having to build the media infrastructure themselves. In practice, this means developers can call SDK APIs to capture media, publish streams, receive remote audio and video, manage rooms, and handle network changes through a structured framework rather than through custom-built low-level systems.

A modern RTC SDK usually includes a full chain of capabilities rather than a single isolated feature. That chain often starts with audio and video capture, continues through encoding and real-time transmission, and ends with decoding, rendering, and playback on the receiving side. Around that core pipeline, it also includes supporting capabilities such as signaling, echo cancellation, noise suppression, adaptive bitrate control, and optional features like beauty effects or virtual backgrounds. Because all of these modules need to work together in real time, the real value of an RTC SDK is not just that it provides APIs, but that it provides an already-optimized communication architecture.

Core Capabilities of an RTC SDK

A complete RTC SDK supports the full lifecycle of a real-time session. That means it does much more than send media from one device to another. It must capture audio and video, process and compress that media, transmit it under unstable network conditions, recover from packet loss, decode it efficiently on the receiving side, and render it in a way that feels smooth to the user. Understanding these layers helps developers evaluate SDK quality more clearly and choose the right solution for their product.

1. Audio Capture and Encoding

Audio handling begins at the microphone, where physical sound waves are converted into digital signals that software can process. At this stage, the system captures raw PCM audio data, which is accurate but far too large for efficient real-time delivery. Because RTC applications need both low delay and efficient bandwidth use, this raw audio must go through pre-processing and compression before it can be transmitted over the network.

The quality of this stage depends heavily on several core parameters, including sample rate, bit depth, and channel count. These settings influence both fidelity and bandwidth consumption. For example, mono audio is usually enough for voice communication, while stereo may be more suitable for music or entertainment scenarios. Once the input is captured, the SDK typically applies processing such as gain control or noise handling before encoding it into a format that is smaller and easier to send in real time.

Among audio codecs, Opus is widely considered the default choice for RTC because it balances compression efficiency, low latency, and good quality across different network conditions. Other codecs such as AAC, G.711, and G.722 are also used in specific scenarios, but Opus is especially common in modern communication products because it adapts well to both speech and mixed audio workloads.

Codec Characteristics Common Use Cases
Opus Low latency, high compression efficiency, open source Voice calls, video meetings
AAC Strong audio quality, broad compatibility Live streaming, and recording
G.711 Very low latency, low compression efficiency Telephony, legacy systems
G.722 Better quality than G.711 HD voice calls

What matters most here is not just the codec itself, but how efficiently the SDK handles the full audio path. If capture is unstable, if pre-processing is weak, or if encoding introduces too much delay, the user will hear the impact immediately. That is why audio handling remains one of the most important layers in RTC system design.

2. Video Capture and Encoding

Video follows a similar pattern, but the technical challenge is even greater because raw video data is much larger than raw audio. A camera can output high-resolution frame data continuously, and without compression the amount of data quickly becomes impossible to send in real time. This is why video capture must be closely tied to efficient encoding, especially in products that need to support mobile devices, unstable networks, or multiple participants at once.

Before encoding, the SDK may also apply video pre-processing steps such as rotation correction, mirroring, beauty filters, or background treatment. Once that is done, the video encoder compresses the frames by removing both spatial and temporal redundancy. In simple terms, it avoids resending information that is either already repeated within the same frame or has barely changed from the previous frame. This is what makes real-time video feasible on consumer networks.

Different codecs serve different priorities. H.264 remains the most common choice because it offers strong compatibility across devices and browsers, while codecs like H.265, VP9, and AV1 improve compression efficiency but may face trade-offs in hardware support, licensing, or deployment maturity.

Codec Compression Efficiency Hardware Support Licensing Typical Use Cases
H.264 (AVC) Medium Broad platform support Patent licensed General RTC use, best compatibility
H.265 (HEVC) High Newer devices Higher licensing cost 4K, bandwidth-sensitive scenarios
VP8 Medium Partial support Open source Default in many WebRTC cases
VP9 High Partial support Open source Chrome-based environments
AV1 Very high Growing support Open source Next-generation ultra-efficient video

Beyond codec selection, the SDK also has to manage bitrate, frame rate, keyframe intervals, and encoding latency in a coordinated way. These settings directly affect how smooth the video feels, how quickly it recovers after packet loss, and how well it adapts to network fluctuations. In other words, video quality is not controlled by one parameter alone. It is the result of multiple encoding decisions working together under real-time constraints.

3. Network Transmission

Once media is encoded, it must be transmitted in a way that keeps latency low while preserving enough stability for real-time interaction. This is one of the biggest technical differences between RTC and traditional media delivery. In standard streaming, buffering can hide network variability. In RTC, buffering must stay minimal, so the transport layer has to react much faster to changing bandwidth, delay, and packet loss.

For this reason, RTC systems usually rely on transport methods built on UDP, because UDP avoids the overhead of connection-oriented retransmission at the transport layer. Protocols such as RTP, RTCP, SRTP in WebRTC, and increasingly QUIC are designed or adapted to support media delivery where timeliness is more important than perfect packet delivery. By contrast, protocols such as RTMP, which depend on TCP, are reliable but generally too slow for two-way real-time conversation.

Protocol Base Layer Latency Reliability Use Cases
UDP Connectionless Very low No guaranteed delivery Core transport for RTC
RTP / RTCP Over UDP Very low Feedback through RTCP Standard audio/video transport
WebRTC (SRTP) Over UDP Low Encrypted transport Browser-based RTC
QUIC Over UDP Low More reliable behavior Emerging transport option
RTMP Over TCP High, usually 1 to 3 s Reliable Live streaming ingest, not ideal for calls

The system architecture also matters. In a simple one-to-one scenario, peer-to-peer delivery may achieve very low latency, but it becomes harder to maintain under NAT restrictions, firewall limits, and larger room sizes. That is why most commercial RTC platforms rely on server-assisted architectures such as SFU and, in some cases, MCU. An SFU forwards streams to participants without mixing them, which keeps delay lower and works well for interactive communication. An MCU mixes streams on the server side, which can be useful in specific broadcast or composite scenarios. The point is that transmission is not just about sending packets. It is about choosing an architecture that can scale while still protecting real-time performance.

4. Weak Network Handling

Real-world networks are unstable, which means an RTC SDK cannot assume that packets will always arrive on time, in order, or without loss. This is where weak-network handling becomes one of the most important parts of the RTC stack. In many cases, the difference between a good call experience and a frustrating one comes down to how well the SDK adapts when the network becomes unpredictable.

Several mechanisms work together here. Forward Error Correction (FEC) adds redundant information so some lost packets can be recovered without retransmission. Negative Acknowledgement (NACK) lets the receiver request missing packets when recovery is still possible within acceptable timing. Jitter buffers smooth out uneven packet arrival times so playback remains more stable. Bandwidth estimation continuously monitors current network capacity and adjusts bitrate, resolution, and frame rate before congestion becomes severe. Each mechanism solves a different part of the problem, and strong RTC performance depends on how intelligently they are combined.

For developers, the important point is that weak-network adaptation is rarely visible when it works well. Users simply notice that the call remains usable even as the connection worsens. Video may become less sharp, or the frame rate may temporarily drop, but the interaction continues. That graceful degradation is one of the clearest indicators of a mature RTC SDK.

5. Audio and Video Decoding and Rendering

On the receiving side, the media pipeline reverses. Incoming packets must be reordered, unpacked, decoded, and then rendered through the appropriate device interfaces. This stage may sound straightforward compared with encoding or transport, but it plays a major role in perceived quality because any delay, sync issue, or rendering inefficiency becomes immediately visible or audible to the user.

Video decoding can be handled through hardware acceleration or software-based methods, depending on platform support and device capability. Hardware decoding is usually preferred because it reduces power consumption and improves performance, especially on mobile devices. Once decoded, video frames still need to be converted and rendered through the platform’s graphics pipeline, whether that is SurfaceView on Android, Metal on iOS, or WebGL in a browser environment.

Audio playback follows a similar pattern. The received stream is decoded back into PCM audio and then passed through output handling before being played through the device's speaker or headphones. In group communication scenarios, multiple audio streams may need to be mixed locally, and levels may need to be balanced so the output remains clear and comfortable. This is why decoding and rendering are not just final display steps. They are part of the real-time experience itself.

6. Echo Cancellation, Noise Suppression, and Gain Control

Even if the network is stable and the codecs are efficient, a real-time call can still feel poor if the audio contains echo, background noise, or inconsistent volume levels. That is why audio enhancement remains a core part of RTC engineering. In most SDKs, this is handled through the so-called 3A stack, which includes AEC for echo cancellation, ANS for noise suppression, and AGC for automatic gain control.

These functions work together rather than independently. Echo cancellation removes the remote participant’s voice that may be picked up again by the local microphone. Noise suppression reduces unwanted background sounds such as keyboards, fans, traffic, or office chatter. Gain control keeps voice levels within a more consistent range so users do not sound too quiet one moment and too loud the next. Because these issues often happen simultaneously in real-world environments, the SDK must coordinate all three layers carefully to avoid making the audio sound unnatural.

From a developer perspective, this matters because users judge call quality largely through what they hear. A technically working call can still feel broken if speech is hard to understand. Strong audio processing therefore does more than clean up sound. It directly improves usability, comfort, and trust in the product.

7. Signaling and Room Management

Real-time communication is not only about media streams. It also requires a control layer that coordinates who joins a room, who publishes a stream, who starts receiving one, and how user state changes are synchronized. This control layer is known as signaling, and it is separate from the actual audio and video transport.

That separation is important because signaling and media have different priorities. Signaling messages are relatively small and infrequent, but they must be delivered reliably. Media packets are much larger and far more frequent, and for them, low latency is usually more important than guaranteed delivery. By handling these two planes separately, RTC systems can preserve both responsiveness and reliability where each matters most.

In practice, signaling supports actions such as room login, user join and leave events, stream publish and unpublish notifications, connection state updates, and custom in-room messages. Without signaling, participants would have no way to coordinate the session even if the media engine itself were working perfectly. This is why a complete RTC SDK usually provides both media APIs and room-level signaling capabilities as part of the same workflow.

8. Beauty Effects and Virtual Backgrounds

In many products, especially social apps, live streaming platforms, online classes, and creator tools, real-time communication is not limited to pure transmission. Visual enhancement also becomes part of the user experience. That is where optional capabilities such as beauty effects, background blur, and virtual backgrounds come in.

These features typically run in the video pre-processing stage before the final encoded stream is sent. The system first analyzes the incoming camera frames, then applies face detection, landmark tracking, or person segmentation, and finally produces an adjusted output frame for encoding. Because all of this must happen in real time, the challenge is not only visual quality but also processing efficiency. If the enhancement pipeline is too heavy, it can increase latency, raise device temperature, or reduce frame stability.

For developers, the value of these features depends on the product category. In some business scenarios, they may be optional. In social and creator-facing scenarios, they may strongly affect user engagement. Either way, their presence in an RTC SDK reflects how the market has evolved from basic communication toward more polished and interactive real-time experiences.

Why Developers Use an RTC SDK Instead of Building One In-House

At a high level, the appeal of an RTC SDK is simple: it removes years of infrastructure work from the product roadmap. Building a real-time communication system internally means solving low-latency transport, media synchronization, codec integration, device compatibility, network adaptation, room coordination, and quality optimization across many platforms. Each of these areas is difficult on its own, and together they form a highly specialized engineering problem.

An RTC SDK reduces that burden by packaging proven real-time capabilities into a reusable framework. That allows teams to spend less time on transport and media internals, and more time on the features that make their product different. For most companies, this is not just a technical convenience. It is also a practical decision about speed, stability, and long-term maintenance cost.

Build Faster With ZEGOCLOUD RTC SDK

For teams building products that rely on real-time voice or video, choosing the right SDK can make a major difference in both development speed and product quality. A strong RTC SDK should not only provide basic calling APIs, but also deliver stable performance across devices, good weak-network resilience, and a workflow that fits how developers actually build applications.

ZEGOCLOUD RTC SDK is designed to help developers integrate real-time communication across platforms with fast onboarding and production-ready capabilities. It supports audio calls, video calls, interactive live streaming, and other real-time scenarios through a developer-friendly integration path, allowing teams to launch communication features more efficiently without building the full stack themselves.

Final Thoughts

RTC SDKs are now part of the core infrastructure behind modern interactive products. They make it possible to build communication experiences that feel immediate, responsive, and scalable without having to recreate the underlying media system from scratch. As use cases such as AI voice interaction, remote collaboration, immersive live experiences, and real-time service platforms continue to expand, the role of RTC technology will only become more important.

For developers, understanding how an RTC SDK works is useful not only when evaluating vendors but also when designing better product architecture. The more clearly you understand its media pipeline, network behavior, and control layers, the easier it becomes to choose the right communication foundation for your application.

RTC in Law: How Real-Time Communication Can Enhance Legal Services

Stephen568hub — Wed, 18 Mar 2026 06:37:51 +0000

As legal services continue to move online, communication is becoming a core part of digital legal infrastructure. Law firms, legal aid platforms, compliance teams, and legal tech startups all need faster and more flexible ways to connect lawyers, clients, experts, and other participants.

This is where RTC, or real-time communication, becomes valuable. With low-latency audio, video, messaging, and collaboration features, RTC can help legal platforms deliver more responsive, accessible, and efficient services. It reduces the need for travel, shortens response time, and makes it easier to support clients across different locations.

For legal services, communication is not just about exchanging information. It is also about trust, clarity, timing, and privacy. A strong RTC experience can improve all of these areas when it is designed well.

In this article, I’ll look at how RTC fits into legal services, the benefits it can bring, the challenges that come with legal use cases, and what developers should consider when choosing an RTC solution for legal applications.

Why RTC Matters in Legal Services

Legal work depends heavily on live communication. Lawyers need to understand client concerns quickly, explain complex issues clearly, review evidence with others, and often coordinate with multiple people under time pressure.

Traditional channels such as email or standard phone calls are still useful, but they are often not enough for situations that require immediate feedback, visual explanation, or real-time collaboration. RTC helps fill that gap by making live interaction part of the platform itself.

That matters in legal scenarios for several reasons. First, a live conversation usually gives more context than text alone. Lawyers can catch tone, pauses, hesitation, and other nonverbal details that may shape how a situation is understood. Second, legal communication is often urgent. Delays can slow down case progress, increase risk, or create frustration for clients. Third, many legal workflows involve more than one participant, so real-time coordination becomes especially important.

In short, RTC is not just a communication feature for legal products. In many cases, it becomes part of the service model.

Common Use Cases of RTC in Legal Applications

RTC can support a wide range of legal scenarios, from client communication to internal operations.

Remote legal consultation

One of the most obvious uses of RTC is remote consultation. Lawyers can meet clients through audio or video calls without requiring them to visit an office in person. This is useful for routine consultations, urgent legal questions, and follow-up discussions.

It also helps make legal support more accessible for people in rural areas, clients with mobility limitations, or users who simply need a faster and more convenient way to get professional advice.

Witness interviews and statement collection

Legal professionals often need to speak with witnesses or involved parties before formal proceedings begin. RTC can support remote interviews and early fact gathering while preserving the value of live conversation.

Compared with text or email, video communication makes it easier to capture reactions and clarify details immediately.

Virtual hearings, mediation, and arbitration

As more legal and quasi-legal processes move online, RTC can support structured interactions between lawyers, clients, judges, arbitrators, mediators, and other participants. Features such as screen sharing, participant management, and session recording can help remote sessions run more smoothly.

Internal legal team collaboration

Legal teams often need to align on case strategy, review documents together, or consult external experts. RTC allows lawyers to communicate with colleagues instantly, no matter where they are located. This improves coordination and helps teams make decisions more quickly.

Cross-border legal communication

Law firms may need to work with overseas clients, experts, translators, or asset-related contacts. RTC makes these interactions easier to manage without the delay and cost of travel. For firms handling international matters, this can significantly improve responsiveness and efficiency.

How RTC Improves the Quality of Legal Services

The value of RTC in legal services is not limited to convenience. It can directly improve service quality in several ways.

Faster communication and case progress

Legal issues are often time-sensitive. Clients may need immediate guidance after receiving a notice, teams may need to resolve a question before a filing deadline, or a lawyer may need to review details with a witness quickly. RTC reduces waiting time and allows communication to happen in the moment.

This can help move cases forward faster and reduce unnecessary process delays.

Better accessibility

Many people do not access legal services as easily as they should. Distance, disability, scheduling conflicts, and cost can all become barriers. RTC makes it possible to bring legal support to more people by removing the requirement for physical presence in many scenarios.

This is especially important for legal aid, immigration support, family law, and other services where accessibility can affect outcomes.

More natural client communication

Legal matters can be stressful, personal, and difficult to understand. A live audio or video conversation usually feels more human than text alone. Lawyers can explain issues more clearly, answer follow-up questions immediately, and create a stronger sense of support.

That can improve both client understanding and client satisfaction.

Stronger collaboration

Cases often involve several stakeholders. A lawyer may need to talk with a client, a co-counsel, a compliance officer, a witness, or an outside expert. RTC allows these interactions to happen in one live environment, which reduces fragmentation and makes collaboration more efficient.

Lower operating costs

RTC can reduce travel expenses, meeting logistics, office coordination, and other operational costs. This benefits both legal service providers and clients. In some cases, it can also make legal services more affordable and scalable.

Better service continuity

When services depend too heavily on physical meetings, continuity becomes harder to maintain across locations and time zones. RTC helps legal teams deliver support more consistently, even when participants are distributed.

Challenges of Using RTC in Legal Scenarios

While RTC offers clear benefits, legal use cases also come with stricter requirements than many other industries.

Security and privacy

This is the most important concern. Legal communication often involves highly sensitive information, including contracts, financial records, personal details, case strategy, and confidential discussions. If a platform does not provide strong security protections, the risks are serious.

For legal services, encryption, access control, secure transmission, and privacy compliance are essential rather than optional.

Technical issues and network instability

A poor connection during an important consultation or witness interview can disrupt communication and damage confidence in the platform. Lag, frozen video, audio distortion, and unexpected disconnections all affect the quality of the experience.

In legal use cases, reliability matters because the conversation itself may be critical.

User readiness and technical literacy

Not every client is comfortable using online communication tools. Some users may struggle to join a session, enable their microphone, upload documents, or understand how a digital legal meeting works. If the product is too complicated, adoption becomes more difficult.

That means legal applications need a simple interface, clear onboarding, and strong usability.

Building trust remotely

Face-to-face legal meetings often help establish trust through body language, eye contact, and the overall seriousness of the setting. Remote communication can still work well, but it requires thoughtful design. Good video quality, clear audio, professional interface design, and smooth interaction all contribute to trust.

Sharing and presenting evidence

In a physical setting, showing documents or evidence is straightforward. In an online setting, the process becomes more complex. Legal teams may need secure file sharing, screen presentation, annotation, permission control, and recording to make digital evidence handling more effective.

What Developers Should Look for in an RTC Solution for Legal Apps

If you are building a legal application, choosing the right RTC infrastructure is a strategic decision. The wrong choice can lead to security gaps, poor user experience, and scalability problems later.

Security and compliance

This should be the top priority. Look for strong encryption, secure data transmission, access controls, and support for privacy-focused architecture. Depending on your market, compliance readiness may also matter, including GDPR and other regulatory requirements.

Reliability

Legal communication should not break down because one participant is using a weaker network. A good RTC solution should have strong weak-network performance, low latency, jitter handling, and reconnect stability.

Easy integration

Developers usually want an SDK that is practical to implement and works across different products and platforms. It helps if the RTC solution supports Web, iOS, Android, desktop, and other environments without adding too much complexity.

Built-in collaboration capabilities

Many legal applications need more than audio and video. Features such as screen sharing, instant messaging, file transfer, session recording, whiteboard interaction, and real-time event callbacks can reduce development effort and make the product more useful.

Scalability

A legal product may start with one-on-one consultations and later expand into multi-party conferencing, expert collaboration, or broader online legal workflows. The RTC stack should support that growth without forcing a major rebuild.

Cost structure

Developers and product teams should also evaluate pricing carefully. Some platforms offer free testing, flexible plans, or usage-based billing, which can help during product validation and early rollout.

Building Legal Service Platforms with ZEGOCLOUD RTC

For developers building legal service applications, ZEGOCLOUD RTC can support many of the real-time communication needs this industry requires.

It supports low-latency audio and video interaction, which helps lawyer-client communication feel more natural and immediate. It also supports instant messaging, which can be useful for sharing text, images, and legal files during a live consultation.

For legal scenarios where records matter, recording and encrypted cloud storage can help preserve communication data for later review when needed. Multi-platform support across Web, iOS, Android, Windows, Linux, and mini programs also makes it easier to build legal products for different audiences.

Network conditions are another important factor in legal use cases. Communication may happen in offices, homes, or mobile environments where connection quality is not always ideal. With weak-network optimization, packet loss handling, and low-latency transmission, RTC performance can remain more stable even under less-than-perfect conditions.

For teams that want to move quickly, SDK-based integration also helps shorten the time needed to launch real-time communication features.

Final Thoughts

RTC is changing how legal services are delivered. It helps law firms and legal platforms offer faster communication, better accessibility, stronger collaboration, and a more flexible service experience.

At the same time, legal communication is not a casual use case. It demands a high standard for privacy, reliability, and trust. That means developers need to think beyond basic video calling and focus on building communication systems that truly fit legal workflows.

When implemented well, RTC can do much more than add convenience. It can improve the quality of legal services in a meaningful and practical way.

How to Build an AI Voice Assistant on Android with Flutter

Stephen568hub — Fri, 13 Mar 2026 07:28:29 +0000

Voice interfaces represent the next evolution in mobile human-computer interaction. Users increasingly expect natural, hands-free communication with their applications rather than traditional touch input. For developers seeking to build an AI voice assistant for Android using Flutter, ZEGOCLOUD provides a comprehensive cross-platform solution.

This technical guide presents a systematic approach to integrating voice AI capabilities into Flutter applications. Through ZEGOCLOUD's Conversational AI platform, developers can implement automatic speech recognition, large language model processing, and neural text-to-speech within a unified Dart SDK architecture.

ZEGOCLOUD Platform Overview

ZEGOCLOUD delivers real-time communication services through cloud-based infrastructure. The Conversational AI product consolidates three distinct AI services:

Service	Technology	Output
ASR	DeepSpeech-based recognition	Transcribed text
LLM	Large language model inference	Contextual responses
TTS	Neural voice synthesis	Natural audio output

The platform abstracts complexity associated with managing separate ASR, LLM, and TTS providers while maintaining sub-second response latency suitable for conversational applications.

System Architecture

Component Interaction

sequenceDiagram
    participant C as Flutter Client
    participant B as Backend Server
    participant Z as ZEGO Cloud
    participant A as AI Services

    C->>B: Request auth token
    B-->>C: Return token

    C->>Z: Join RTC room + publish stream
    C->>B: Create AI agent request
    B->>Z: Register agent with LLM/TTS
    B->>Z: Instantiate agent
    Z-->>B: Agent stream identifier
    B-->>C: Agent stream ID

    C->>Z: Subscribe to agent audio

    Note over C,Z: Conversation loop active
    C->>Z: User audio frames
    Z->>A: ASR transcription
    A->>Z: Text payload
    Z->>A: LLM inference request
    A->>Z: Response text
    Z->>A: TTS synthesis request
    A->>Z: Audio payload
    Z-->>C: AI voice + subtitle data

    C->>B: Terminate agent
    B->>Z: Delete agent instance

Data Flow Summary

Phase	Action	Endpoint
Authentication	Token request	`/api/zego/token`
Session Init	Room login + publish	ZEGO RTC
Agent Creation	AI agent registration	`/api/zego/start`
Media Playback	Stream subscription	ZEGO RTC
Conversation	Bidirectional audio	ZEGO Cloud
Termination	Agent cleanup	`/api/zego/stop`

Understanding the Server-Client Separation

Building a flutter ai voice assistant requires a two-tier architecture rather than direct client-to-service communication. This design decision stems from fundamental security and operational requirements.

Architectural Roles:

Layer	Primary Function	Credentials Stored
Backend Server	Authentication, agent lifecycle, API signing	Full access (server secret, LLM keys)
Flutter Client	Audio I/O, stream management, UI	None (receives short-lived tokens only)
ZEGO Cloud	ASR, LLM, TTS processing	N/A (external service)

Rationale for Separation:

Credential Security — Server secrets (ZEGO_SERVER_SECRET, LLM API keys) remain in your controlled infrastructure. Mobile applications can be decompiled, making them unsuitable for permanent credential storage.
Token Lifecycle Management — Clients authenticate using time-limited tokens (default 1 hour). This approach provides revocable access without exposing long-term credentials.
Dynamic AI Configuration — System prompts, model selections, and voice profiles are managed server-side. Changes deploy instantly without requiring client app updates through app stores.
Operational Control — Centralized logging, rate limiting, usage analytics, and abuse detection operate at the server layer, providing a single control point for production monitoring.

When you build voice ai assistant for android applications using Flutter, this architecture ensures security, maintainability, and operational visibility across both Android and iOS platforms.

Implementation Guide

Phase 1: Backend Infrastructure

Deploy a Next.js server to handle authentication and agent lifecycle management.

Environment Setup:

# .env.local configuration
NEXT_PUBLIC_ZEGO_APP_ID=<your_app_id>
ZEGO_SERVER_SECRET=<32_character_secret>

# AI Agent parameters
ZEGO_AGENT_ID=voice_assistant_01
SYSTEM_PROMPT="You are a helpful voice assistant. Provide concise, spoken-friendly responses."

# External AI services
LLM_URL=https://api.provider.com/v1/chat/completions
LLM_API_KEY=<your_api_key>
LLM_MODEL=gpt-4

# TTS configuration
TTS_VENDOR=ByteDance
TTS_VOICE_TYPE=zh_female_wanwanxiaohe_moon_bigtts

Token Generation Handler:

// app/api/zego/token/route.ts
import { NextRequest, NextResponse } from 'next/server';
import crypto from 'crypto';

const generateToken = (appId: number, userId: string, secret: string, ttl: number): string => {
  const payload = {
    app_id: appId,
    user_id: userId,
    nonce: Math.floor(Math.random() * 2147483647),
    ctime: Math.floor(Date.now() / 1000),
    expire: Math.floor(Date.now() / 1000) + ttl,
    payload: ''
  };

  const encrypted = crypto
    .createCipheriv('aes-256-gcm', secret, crypto.randomBytes(12))
    .update(JSON.stringify(payload), 'utf8');

  return '04' + Buffer.concat([
    Buffer.alloc(8).writeBigInt64BE(BigInt(payload.expire)),
    Buffer.from([0, 12]),
    encrypted.final(),
    Buffer.from([1])
  ]).toString('base64');
};

export async function POST(req: NextRequest) {
  const { userId } = await req.json();
  const token = generateToken(
    parseInt(process.env.NEXT_PUBLIC_ZEGO_APP_ID!),
    userId,
    process.env.ZEGO_SERVER_SECRET!,
    3600
  );
  return NextResponse.json({ token });
}

Agent Management API:

// app/api/zego/start/route.ts
import { sendZegoRequest } from '@/lib/zego-utils';

export async function POST(req: Request) {
  const { roomId, userId, userStreamId } = await req.json();

  const agentData = await sendZegoRequest('CreateAIAgent', {
    agentId: process.env.ZEGO_AGENT_ID,
    roomId,
    userId,
    userStreamId,
    llmConfig: {
      model: process.env.LLM_MODEL,
      systemPrompt: process.env.SYSTEM_PROMPT
    },
    ttsConfig: {
      vendor: process.env.TTS_VENDOR,
      voiceType: process.env.TTS_VOICE_TYPE
    }
  });

  return Response.json({
    agentInstanceId: agentData.instanceId,
    agentStreamId: agentData.streamId
  });
}

Phase 2: Flutter Client Implementation

With the backend infrastructure complete, we now build the Flutter client. This phase covers project configuration, network communication, ZEGOCLOUD engine initialization, and UI integration for both Android and iOS platforms.

Step 2.1: Project Configuration

Create a new Flutter project and add the required dependencies to your pubspec.yaml:

# pubspec.yaml
name: ai_voice_assistant
description: Cross-platform AI voice assistant using ZEGOCLOUD

environment:
  sdk: ^3.8.1

dependencies:
  flutter:
    sdk: flutter

  # ZEGO Express SDK
  zego_express_engine: ^3.22.0

  # HTTP client for API calls
  http: ^1.2.0

  # Permission handling
  permission_handler: ^11.3.0

  # State management (optional)
  provider: ^6.1.1

After updating dependencies, run flutter pub get to fetch packages.

Step 2.2: Platform Permissions

Configure permissions for both Android and iOS platforms:

<!-- android/app/src/main/AndroidManifest.xml -->
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

<!-- ios/Runner/Info.plist -->
<key>NSMicrophoneUsageDescription</key>
<string>This app requires microphone access for voice assistant functionality</string>

Note: Without these permissions, the app will not be able to capture audio for voice interaction.

Step 2.3: Application Configuration

Create a centralized configuration class to store app-wide constants and utility functions:

// lib/config/app_config.dart
class AppConfig {
  static const int appId = 1234567890;
  static const String serverUrl = 'https://your-deployment.vercel.app';

  static String generateSessionId() {
    return 's${DateTime.now().millisecondsSinceEpoch % 1000000}';
  }

  static String generateUserId(String sessionId) {
    return 'u$sessionId';
  }

  static String getUserStreamId(String sessionId) {
    return '${sessionId}_user';
  }
}

Important: For Android emulator testing with a local backend, use http://10.0.2.2:3000 instead of localhost to access your development server.

Step 2.4: Network Service Layer

The API service handles all HTTP communication with your backend server. It manages three core operations: authentication, agent creation, and agent cleanup.

// lib/services/api_service.dart
import 'dart:convert';
import 'package:http/http.dart' as http;
import '../config/app_config.dart';

class ApiService {
  static final _client = http.Client();

  static Future<String?> authenticate(String userId) async {
    try {
      final response = await _client.post(
        Uri.parse('${AppConfig.serverUrl}/api/zego/token'),
        headers: {'Content-Type': 'application/json'},
        body: jsonEncode({'userId': userId}),
      );

      if (response.statusCode == 200) {
        final data = jsonDecode(response.body);
        if (data['code'] == 0 && data['data'] != null) {
          return data['data']['token'] as String?;
        }
      }
      return null;
    } catch (e) {
      debugPrint('[ApiService] Authentication error: $e');
      return null;
    }
  }

  static Future<String?> createAgent({
    required String roomId,
    required String userId,
    required String userStreamId,
  }) async {
    try {
      final response = await _client.post(
        Uri.parse('${AppConfig.serverUrl}/api/zego/start'),
        headers: {'Content-Type': 'application/json'},
        body: jsonEncode({
          'roomId': roomId,
          'userId': userId,
          'userStreamId': userStreamId,
        }),
      );

      if (response.statusCode == 200) {
        final data = jsonDecode(response.body);
        if (data['code'] == 0 && data['data'] != null) {
          return data['data']['agentStreamId'] as String?;
        }
      }
      return null;
    } catch (e) {
      debugPrint('[ApiService] Create agent error: $e');
      return null;
    }
  }

  static Future<bool> destroyAgent(String agentInstanceId) async {
    try {
      final response = await _client.post(
        Uri.parse('${AppConfig.serverUrl}/api/zego/stop'),
        headers: {'Content-Type': 'application/json'},
        body: jsonEncode({'agentInstanceId': agentInstanceId}),
      );
      return response.statusCode == 200;
    } catch (e) {
      debugPrint('[ApiService] Destroy agent error: $e');
      return false;
    }
  }
}

Implementation notes:

All methods are static for simple access without instantiation
Error handling returns null or false on failure — calling code should handle these cases
The http.Client is reused across requests for efficiency
Debug print statements help with development debugging

Step 2.5: RTC Engine Manager

The ZegoEngineManager class encapsulates all ZEGOCLOUD Express SDK operations, providing a clean interface for the rest of your application. It uses a singleton pattern to ensure only one engine instance exists:

// lib/services/zego_engine_manager.dart
import 'package:zego_express_engine/zego_express_engine.dart';
import '../config/app_config.dart';

class ZegoEngineManager {
  static final ZegoEngineManager _instance = ZegoEngineManager._internal();
  factory ZegoEngineManager() => _instance;
  ZegoEngineManager._internal();

  Function(String, ZegoRoomStateChangedReason, int)? onRoomStateChanged;
  bool _isInitialized = false;

  Future<void> initialize() async {
    if (_isInitialized) return;

    // Configure engine
    final config = ZegoEngineConfig(advancedConfig: {
      'set_audio_volume_ducking_mode': '1',
      'enable_rnd_volume_adaptive': 'true',
    });
    await ZegoExpressEngine.setEngineConfig(config);

    // Create engine with profile
    final profile = ZegoEngineProfile(
      AppConfig.appId,
      ZegoScenario.HighQualityChatroom,
      enablePlatformView: false,
    );
    await ZegoExpressEngine.createEngineWithProfile(profile);

    // Setup event handlers
    _setupEventHandlers();

    // Audio optimization for voice interaction
    await _configureAudioSettings();

    _isInitialized = true;
  }

  void _setupEventHandlers() {
    ZegoExpressEngine.onRoomStateChanged = (roomID, reason, errorCode, _) {
      onRoomStateChanged?.call(roomID, reason, errorCode);
    };
  }

  Future<void> _configureAudioSettings() async {
    final engine = ZegoExpressEngine.instance;
    await engine.enableAGC(true);      // Gain control
    await engine.enableANS(true);      // Noise suppression
    await engine.setANSMode(ZegoANSMode.Medium);
    await engine.enableAEC(true);      // Echo cancellation
    await engine.setAECMode(ZegoAECMode.AIBalanced);
  }

  Future<int> joinRoom(String roomId, String userId, String token) async {
    final user = ZegoUser(userId, userId);
    final config = ZegoRoomConfig(0, true, token);
    final result = await ZegoExpressEngine.instance.loginRoom(
      roomId, user, config: config,
    );
    return result.errorCode;
  }

  Future<void> publishAudio(String streamId) async {
    await ZegoExpressEngine.instance.muteMicrophone(false);
    await ZegoExpressEngine.instance.startPublishingStream(streamId);
  }

  Future<void> playAudio(String streamId) async {
    await ZegoExpressEngine.instance.startPlayingStream(streamId);
  }

  Future<void> leaveRoom(String roomId) async {
    await ZegoExpressEngine.instance.stopPublishingStream();
    await ZegoExpressEngine.instance.logoutRoom(roomId);
  }

  Future<void> destroy() async {
    if (!_isInitialized) return;
    await ZegoExpressEngine.destroyEngine();
    _isInitialized = false;
  }
}

Audio optimization explanation:

Setting	Purpose	Recommended Value
AGC	Automatic Gain Control — normalizes volume levels	Enabled
ANS	Automatic Noise Suppression — reduces background noise	Medium mode
AEC	Acoustic Echo Cancellation — prevents echo feedback	AI Balanced mode

Step 2.6: Subtitle Integration

ZEGOCLOUD provides official subtitle components for Flutter that handle ASR and LLM message parsing. These components are part of the AI Agent SDK.

Download required files from ZEGO Flutter Subtitle Guide:

lib/audio/subtitles/model.dart
lib/audio/subtitles/view.dart
lib/audio/subtitles/message_protocol.dart
lib/audio/subtitles/message_dispatcher.dart

Add these files to your project structure, then create a handler wrapper:

// lib/services/subtitle_handler.dart
import '../audio/subtitles/model.dart';
import '../audio/subtitles/message_protocol.dart';
import '../audio/subtitles/message_dispatcher.dart';

class SubtitleHandler implements ZegoSubtitlesEventHandler {
  final ZegoSubtitlesViewModel model;

  SubtitleHandler(this.model) {
    ZegoSubtitlesMessageDispatcher().registerEventHandler(this);
  }

  @override
  void onRecvAsrChatMsg(ZegoSubtitlesMessageProtocol message) {
    model.handleRecvAsrMessage(message);
  }

  @override
  void onRecvLLMChatMsg(ZegoSubtitlesMessageProtocol message) {
    model.handleRecvLLMMessage(message);
  }

  void onExperimentalAPI(String content) {
    ZegoSubtitlesMessageDispatcher.handleExpressExperimentalAPIContent(content);
  }

  void dispose() {
    ZegoSubtitlesMessageDispatcher().unregisterEventHandler(this);
  }
}

How subtitles work:

The onRecvExperimentalAPI callback receives raw room channel messages
ZegoSubtitlesMessageDispatcher parses and routes messages based on type
SubtitleHandler forwards ASR/LLM messages to the ZegoSubtitlesViewModel
The ZegoSubtitlesView widget displays messages with proper formatting

Step 2.7: UI Integration

Finally, create the main voice assistant page that ties all components together:

// lib/ui/voice_assistant_page.dart
import 'package:flutter/material.dart';
import 'package:zego_express_engine/zego_express_engine.dart';
import '../services/zego_engine_manager.dart';
import '../services/api_service.dart';
import '../services/subtitle_handler.dart';
import '../audio/subtitles/model.dart';
import '../audio/subtitles/view.dart';
import '../config/app_config.dart';

class VoiceAssistantPage extends StatefulWidget {
  const VoiceAssistantPage({super.key});

  @override
  State<VoiceAssistantPage> createState() => _VoiceAssistantPageState();
}

class _VoiceAssistantPageState extends State<VoiceAssistantPage> {
  final _engineManager = ZegoEngineManager();
  late final SubtitleHandler _subtitleHandler;
  late final ZegoSubtitlesViewModel _subtitleModel;

  String? _sessionId;
  String? _userId;
  String? _agentInstanceId;
  bool _isInSession = false;
  bool _isLoading = false;

  @override
  void initState() {
    super.initState();
    _subtitleModel = ZegoSubtitlesViewModel();
    _subtitleHandler = SubtitleHandler(_subtitleModel);
    _initializeEngine();
  }

  Future<void> _initializeEngine() async {
    await _engineManager.initialize();
    _engineManager.onRoomStateChanged = _onRoomStateChanged;
    ZegoExpressEngine.onRecvExperimentalAPI = _subtitleHandler.onExperimentalAPI;
  }

  void _onRoomStateChanged(String roomId, ZegoRoomStateChangedReason reason, int errorCode) {
    if (reason == ZegoRoomStateChangedReason.Logined && !_isInSession) {
      setState(() => _isInSession = true);
    } else if (reason == ZegoRoomStateChangedReason.Logout ||
               reason == ZegoRoomStateChangedReason.KickOut) {
      setState(() => _isInSession = false);
    }
  }

  Future<void> _startSession() async {
    if (_isLoading) return;

    setState(() { _isLoading = true; });

    try {
      _sessionId = AppConfig.generateSessionId();
      _userId = AppConfig.generateUserId(_sessionId!);
      final userStreamId = AppConfig.getUserStreamId(_sessionId!);

      // Authenticate
      final token = await ApiService.authenticate(_userId!);
      if (token == null) { _showError('Authentication failed'); return; }

      // Join room
      final loginCode = await _engineManager.joinRoom(_sessionId!, _userId!, token);
      if (loginCode != 0) { _showError('Room join failed: $loginCode'); return; }

      // Publish audio
      await _engineManager.publishAudio(userStreamId);

      // Create agent
      final agentStreamId = await ApiService.createAgent(
        roomId: _sessionId!,
        userId: _userId!,
        userStreamId: userStreamId,
      );
      if (agentStreamId == null) { _showError('Agent creation failed'); return; }

      _agentInstanceId = agentStreamId;
      await _engineManager.playAudio(agentStreamId);

      setState(() { _isInSession = true; });
    } catch (e) {
      _showError('Error: ${e.toString()}');
    } finally {
      setState(() { _isLoading = false; });
    }
  }

  Future<void> _endSession() async {
    if (_isLoading) return;

    setState(() { _isLoading = true; });

    try {
      if (_agentInstanceId != null) {
        await ApiService.destroyAgent(_agentInstanceId!);
      }
      if (_sessionId != null) {
        await _engineManager.leaveRoom(_sessionId!);
      }

      _sessionId = null;
      _userId = null;
      _agentInstanceId = null;
      setState(() { _isInSession = false; });
    } catch (e) {
      _showError('Error: ${e.toString()}');
    } finally {
      setState(() { _isLoading = false; });
    }
  }

  void _showError(String message) {
    if (!mounted) return;
    ScaffoldMessenger.of(context).showSnackBar(
      SnackBar(content: Text(message), backgroundColor: Colors.red),
    );
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: SafeArea(
        child: Column(
          children: [
            // Control panel
            _buildControlPanel(),
            const Divider(height: 1),
            // Subtitle display
            Expanded(child: ZegoSubtitlesView(model: _subtitleModel)),
          ],
        ),
      ),
    );
  }

  Widget _buildControlPanel() {
    return Container(
      padding: const EdgeInsets.symmetric(vertical: 40, horizontal: 20),
      color: Colors.grey[100],
      child: Column(
        children: [
          const Text('AI Voice Assistant',
              style: TextStyle(fontSize: 24, fontWeight: FontWeight.bold)),
          const SizedBox(height: 20),
          // Status indicator
          Row(
            mainAxisAlignment: MainAxisAlignment.center,
            children: [
              Container(
                width: 12, height: 12,
                decoration: BoxDecoration(
                  shape: BoxShape.circle,
                  color: _isInSession ? Colors.green : Colors.grey,
                ),
              ),
              const SizedBox(width: 8),
              Text(_isInSession ? 'Connected' : 'Disconnected',
                  style: TextStyle(
                    fontSize: 16,
                    color: _isInSession ? Colors.green : Colors.grey,
                  )),
            ],
          ),
          const SizedBox(height: 20),
          // Call button
          SizedBox(
            width: 120, height: 120,
            child: ElevatedButton(
              onPressed: _isLoading ? null : (_isInSession ? _endSession : _startSession),
              style: ElevatedButton.styleFrom(
                shape: const CircleBorder(),
                backgroundColor: _isInSession ? Colors.red : Colors.green,
                foregroundColor: Colors.white,
              ),
              child: _isLoading
                  ? const CircularProgressIndicator(color: Colors.white)
                  : Icon(_isInSession ? Icons.call_end : Icons.call, size: 48),
            ),
          ),
        ],
      ),
    );
  }

  @override
  void dispose() {
    _subtitleHandler.dispose();
    _engineManager.destroy();
    super.dispose();
  }
}

Session flow summary:

Step	Action	Component
1	Generate session ID	`AppConfig`
2	Request auth token	`ApiService.authenticate()`
3	Join RTC room	`ZegoEngineManager.joinRoom()`
4	Publish user audio	`ZegoEngineManager.publishAudio()`
5	Create AI agent	`ApiService.createAgent()`
6	Play AI audio	`ZegoEngineManager.playAudio()`
7	End session	`destroyAgent()` + `leaveRoom()`

Conclusion

This guide demonstrated how to build an AI voice assistant for Android using Flutter and ZEGOCLOUD's integrated platform. The architecture separates concerns between backend authentication, agent orchestration, and client-side media handling. Key implementation points include proper token management, RTC room coordination, and subtitle message parsing.

Developers can extend this foundation with custom wake words, multi-language support, or domain-specific conversation flows. The same architectural pattern applies across customer service, education, accessibility, and IoT control scenarios requiring natural voice interaction.

For production deployment, consider implementing connection retry logic, offline fallbacks, and comprehensive error handling based on your specific use case requirements. The Flutter approach additionally provides iOS deployment with minimal code changes.

How to Build an AI Voice Assistant on Android

Stephen568hub — Thu, 12 Mar 2026 10:47:11 +0000

Voice interfaces represent the next evolution in mobile human-computer interaction. Users increasingly expect natural, hands-free communication with their applications rather than traditional touch input. For developers seeking to build an AI voice assistant for Android, ZEGOCLOUD(https://www.zegocloud.com/) provides a comprehensive solution.

This technical guide presents a systematic approach to integrating voice AI capabilities into Android applications. Through ZEGOCLOUD's Conversational AI platform（https://www.zegocloud.com/solutions/conversational-ai）, developers can implement automatic speech recognition, large language model processing, and neural text-to-speech within a unified SDK architecture.

ZEGOCLOUD Platform Overview

ZEGOCLOUD delivers real-time communication services through cloud-based infrastructure. The Conversational AI product consolidates three distinct AI services:

Service	Technology	Output
ASR	DeepSpeech-based recognition	Transcribed text
LLM	Large language model inference	Contextual responses
TTS	Neural voice synthesis	Natural audio output

The platform abstracts complexity associated with managing separate ASR, LLM, and TTS providers while maintaining sub-second response latency suitable for conversational applications.

System Architecture

Component Interaction

sequenceDiagram
    participant C as Android Client
    participant B as Backend Server
    participant Z as ZEGO Cloud
    participant A as AI Services

    C->>B: Request auth token
    B-->>C: Return token

    C->>Z: Join RTC room + publish stream
    C->>B: Create AI agent request
    B->>Z: Register agent with LLM/TTS
    B->>Z: Instantiate agent
    Z-->>B: Agent stream identifier
    B-->>C: Agent stream ID

    C->>Z: Subscribe to agent audio

    Note over C,Z: Conversation loop active
    C->>Z: User audio frames
    Z->>A: ASR transcription
    A->>Z: Text payload
    Z->>A: LLM inference request
    A->>Z: Response text
    Z->>A: TTS synthesis request
    A->>Z: Audio payload
    Z-->>C: AI voice + subtitle data

    C->>B: Terminate agent
    B->>Z: Delete agent instance

Data Flow Summary

Phase	Action	Endpoint
Authentication	Token request	`/api/zego/token`
Session Init	Room login + publish	ZEGO RTC
Agent Creation	AI agent registration	`/api/zego/start`
Media Playback	Stream subscription	ZEGO RTC
Conversation	Bidirectional audio	ZEGO Cloud
Termination	Agent cleanup	`/api/zego/stop`

Understanding the Server-Client Separation

Building an android ai voice assistant requires a two-tier architecture rather than direct client-to-service communication. This design decision stems from fundamental security and operational requirements.

Architectural Roles:

Layer	Primary Function	Credentials Stored
Backend Server	Authentication, agent lifecycle, API signing	Full access (server secret, LLM keys)
Android Client	Audio I/O, stream management, UI	None (receives short-lived tokens only)
ZEGO Cloud	ASR, LLM, TTS processing	N/A (external service)

Rationale for Separation:

Credential Security — Server secrets (ZEGO_SERVER_SECRET, LLM API keys) remain in your controlled infrastructure. Mobile applications can be decompiled, making them unsuitable for permanent credential storage.
Token Lifecycle Management — Clients authenticate using time-limited tokens (default 1 hour). This approach provides revocable access without exposing long-term credentials.
Dynamic AI Configuration — System prompts, model selections, and voice profiles are managed server-side. Changes deploy instantly without requiring client app updates through app stores.
Operational Control — Centralized logging, rate limiting, usage analytics, and abuse detection operate at the server layer, providing a single control point for production monitoring.

When you build voice ai assistant for android applications, this architecture ensures security, maintainability, and operational visibility in production environments.

Implementation Guide

Phase 1: Backend Infrastructure

Deploy a Next.js server to handle authentication and agent lifecycle management.

Environment Setup:

# .env.local configuration
NEXT_PUBLIC_ZEGO_APP_ID=<your_app_id>
ZEGO_SERVER_SECRET=<32_character_secret>

# AI Agent parameters
ZEGO_AGENT_ID=voice_assistant_01
SYSTEM_PROMPT="You are a helpful voice assistant. Provide concise, spoken-friendly responses."

# External AI services
LLM_URL=https://api.provider.com/v1/chat/completions
LLM_API_KEY=<your_api_key>
LLM_MODEL=gpt-4

# TTS configuration
TTS_VENDOR=ByteDance
TTS_VOICE_TYPE=zh_female_wanwanxiaohe_moon_bigtts

Token Generation Handler:

// app/api/zego/token/route.ts
import { NextRequest, NextResponse } from 'next/server';
import crypto from 'crypto';

const generateToken = (appId: number, userId: string, secret: string, ttl: number): string => {
  const payload = {
    app_id: appId,
    user_id: userId,
    nonce: Math.floor(Math.random() * 2147483647),
    ctime: Math.floor(Date.now() / 1000),
    expire: Math.floor(Date.now() / 1000) + ttl,
    payload: ''
  };

  const encrypted = crypto
    .createCipheriv('aes-256-gcm', secret, crypto.randomBytes(12))
    .update(JSON.stringify(payload), 'utf8');

  return '04' + Buffer.concat([
    Buffer.alloc(8).writeBigInt64BE(BigInt(payload.expire)),
    Buffer.from([0, 12]),
    encrypted.final(),
    Buffer.from([1])
  ]).toString('base64');
};

export async function POST(req: NextRequest) {
  const { userId } = await req.json();
  const token = generateToken(
    parseInt(process.env.NEXT_PUBLIC_ZEGO_APP_ID!),
    userId,
    process.env.ZEGO_SERVER_SECRET!,
    3600
  );
  return NextResponse.json({ token });
}

Agent Management API:

// app/api/zego/start/route.ts
import { sendZegoRequest } from '@/lib/zego-utils';

export async function POST(req: Request) {
  const { roomId, userId, userStreamId } = await req.json();

  const agentData = await sendZegoRequest('CreateAIAgent', {
    agentId: process.env.ZEGO_AGENT_ID,
    roomId,
    userId,
    userStreamId,
    llmConfig: {
      model: process.env.LLM_MODEL,
      systemPrompt: process.env.SYSTEM_PROMPT
    },
    ttsConfig: {
      vendor: process.env.TTS_VENDOR,
      voiceType: process.env.TTS_VOICE_TYPE
    }
  });

  return Response.json({
    agentInstanceId: agentData.instanceId,
    agentStreamId: agentData.streamId
  });
}

Phase 2: Android Client Implementation

With the backend infrastructure in place, we now turn to the Android client. This phase involves configuring your Android project, setting up network communication, initializing the ZEGOCLOUD Express Engine, and integrating the UI components.

Step 2.1: Project Configuration

The ZEGOCLOUD Express SDK is distributed as a JAR file with native .so libraries. You need to manually integrate these files rather than using a Maven dependency.

Setup instructions:

Download the AI Agent-optimized SDK from ZEGO Download Page
Extract and copy files to your project:
- ZegoExpressEngine.jar → app/libs/
- libZegoExpressEngine.so (arm64) → app/libs/arm64-v8a/
- libZegoExpressEngine.so (v7a) → app/libs/armeabi-v7a/

Then configure your build script:

// app/build.gradle.kts
android {
    defaultConfig {
        minSdk = 24
        ndk {
            abiFilters += listOf("armeabi-v7a", "arm64-v8a")
        }
    }
    sourceSets {
        getByName("main") {
            jniLibs.srcDirs("libs")
        }
    }
}

dependencies {
    implementation(files("libs/ZegoExpressEngine.jar"))
    implementation("com.squareup.okhttp3:okhttp:4.12.0")
    implementation("com.google.code.gson:gson:2.10.1")
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.7.3")
}

Note: The AI Agent-optimized SDK is required for receiving subtitle messages via onRecvExperimentalAPI. The standard Maven version does not support this feature.

Step 2.2: Application Configuration

Create a centralized configuration object to store app-wide constants and utility functions:

// config/AppConfig.kt
object AppConfig {
    const val APP_ID: Long = 1234567890L
    const val SERVER_URL = "https://your-deployment.vercel.app"

    fun generateSessionId(): String =
        "s${System.currentTimeMillis() % 1000000}"
}

Important: For Android emulator testing, use http://10.0.2.2:3000 instead of localhost to access your local development server.

Step 2.3: Network Service Layer

The API service handles all HTTP communication with your backend server. It manages three core operations: authentication, agent creation, and agent cleanup.

// api/ApiService.kt
object ApiService {
    private val httpClient = OkHttpClient()
    private val jsonSerializer = Gson()

    suspend fun authenticate(userId: String): String? {
        val request = Request.Builder()
            .url("${AppConfig.SERVER_URL}/api/zego/token")
            .post(
                jsonSerializer.toJson(mapOf("userId" to userId))
                    .toRequestBody("application/json".toMediaType())
            )
            .build()

        return try {
            val response = httpClient.newCall(request).execute()
            val json = JSONObject(response.body?.string() ?: "")
            json.optJSONObject("data")?.optString("token")
        } catch (e: Exception) { null }
    }

    suspend fun createAgent(roomId: String, userId: String, streamId: String): String? {
        val request = Request.Builder()
            .url("${AppConfig.SERVER_URL}/api/zego/start")
            .post(
                jsonSerializer.toJson(mapOf(
                    "roomId" to roomId,
                    "userId" to userId,
                    "userStreamId" to streamId
                )).toRequestBody("application/json".toMediaType())
            )
            .build()

        return try {
            val response = httpClient.newCall(request).execute()
            val json = JSONObject(response.body?.string() ?: "")
            json.optJSONObject("data")?.optString("agentStreamId")
        } catch (e: Exception) { null }
    }

    suspend fun destroyAgent(agentInstanceId: String): Boolean {
        val request = Request.Builder()
            .url("${AppConfig.SERVER_URL}/api/zego/stop")
            .post(
                jsonSerializer.toJson(mapOf("agentInstanceId" to agentInstanceId))
                    .toRequestBody("application/json".toMediaType())
            )
            .build()

        return try {
            httpClient.newCall(request).execute().isSuccessful
        } catch (e: Exception) { false }
    }
}

Implementation notes:

All functions are suspend functions, designed to be called from Kotlin coroutines
Error handling returns null or false on failure — the calling code should handle these cases
The OkHttpClient is configured with default timeouts (10s connect, 10s read, 10s write)

Step 2.4: RTC Engine Initialization

The ZegoEngineManager class encapsulates all ZEGOCLOUD Express SDK operations, providing a clean interface for the rest of your application.

// rtc/ZegoEngineManager.kt
class ZegoEngineManager(private val context: Context) {
    private var engine: ZegoExpressEngine? = null

    fun initialize() {
        val profile = ZegoEngineProfile().apply {
            appID = AppConfig.APP_ID
            scenario = ZegoScenario.HIGH_QUALITY_CHATROOM
            application = context.applicationContext
        }
        engine = ZegoExpressEngine.createEngine(profile, null)

        // Audio optimization for voice interaction
        engine?.apply {
            enableAGC(true)      // Gain control
            enableANS(true)      // Noise suppression
            enableAEC(true)      // Echo cancellation
            setANSMode(ZegoANSMode.MEDIUM)
        }
    }

    fun joinRoom(roomId: String, userId: String, token: String): Int {
        val user = ZegoUser(userId)
        val config = ZegoRoomConfig().apply { this.token = token }
        var resultCode = -1

        CountDownLatch(1).apply {
            engine?.loginRoom(roomId, user, config) { code, _ ->
                resultCode = code
                countDown()
            }
            await(5000, TimeUnit.MILLISECONDS)
        }
        return resultCode
    }

    fun publishAudio(streamId: String) {
        engine?.muteMicrophone(false)
        engine?.startPublishingStream(streamId)
    }

    fun playAudio(streamId: String) {
        engine?.startPlayingStream(streamId)
    }

    fun leaveRoom(roomId: String) {
        engine?.stopPublishingStream()
        engine?.logoutRoom(roomId)
    }

    fun destroy() {
        ZegoExpressEngine.destroyEngine()
    }
}

Audio optimization explanation:

Setting	Purpose	Recommended Value
AGC	Automatic Gain Control — normalizes volume levels	Enabled
ANS	Automatic Noise Suppression — reduces background noise	Medium mode
AEC	Acoustic Echo Cancellation — prevents echo feedback	AI Balanced mode

Step 2.5: Subtitle Integration

ZEGOCLOUD provides official subtitle components that handle ASR and LLM message parsing. These components are implemented in Java and need to be integrated into your Kotlin codebase.

Download required files:

AudioChatMessageParser.java
AIChatListView.java

Available from ZEGO Subtitle Guide.

// subtitle/SubtitleHandler.kt
class SubtitleHandler(private val view: AIChatListView) {
    private val messageParser = AudioChatMessageParser()

    init {
        messageParser.setAudioChatMessageListListener(
            object : AudioChatMessageParser.AudioChatMessageListListener {
                override fun onMessageListUpdated(messages: MutableList<AudioChatMessage>) {
                    view.post { view.onMessageListUpdated(messages) }
                }

                override fun onAudioChatStateUpdate(status: AudioChatAgentStatusMessage) {
                    // Handle agent state changes if needed
                }
            }
        )
    }

    fun onExperimentalAPI(content: String) {
        try {
            val json = JSONObject(content)
            if (json.getString("method") == "liveroom.room.on_recive_room_channel_message") {
                val msgContent = json.getJSONObject("params").getString("msg_content")
                messageParser.parseAudioChatMessage(msgContent)
            }
        } catch (e: Exception) { e.printStackTrace() }
    }
}

How subtitles work:

The onRecvExperimentalAPI callback receives raw room channel messages
Messages are parsed by AudioChatMessageParser to extract ASR (user speech) and LLM (AI response) text
Parsed messages are displayed in the AIChatListView component
The listener ensures UI updates run on the main thread via view.post()

Step 2.6: Activity Integration

Finally, tie all components together in your main Activity:

// ui/VoiceAssistantActivity.kt
class VoiceAssistantActivity : AppCompatActivity() {
    private lateinit var engineManager: ZegoEngineManager
    private lateinit var subtitleHandler: SubtitleHandler

    private var sessionId: String? = null
    private var userId: String? = null
    private var agentInstanceId: String? = null

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(ActivityVoiceAssistantBinding.inflate(layoutInflater))

        engineManager = ZegoEngineManager(application)
        engineManager.initialize()

        subtitleHandler = SubtitleHandler(binding.subtitleView)
        setupExperimentalAPIListener()

        binding.actionButton.setOnClickListener {
            if (sessionId == null) startSession() else endSession()
        }
    }

    private fun setupExperimentalAPIListener() {
        ZegoExpressEngine.onRecvExperimentalAPI = { content ->
            subtitleHandler.onExperimentalAPI(content)
        }
    }

    private fun startSession() {
        lifecycleScope.launch {
            sessionId = AppConfig.generateSessionId()
            userId = "u${sessionId}"
            val userStreamId = "${sessionId}_user"

            val token = ApiService.authenticate(userId!!)
            if (token == null) { showError("Authentication failed"); return@launch }

            val loginCode = engineManager.joinRoom(sessionId!!, userId!!, token)
            if (loginCode != 0) { showError("Room join failed"); return@launch }

            engineManager.publishAudio(userStreamId)

            val agentStreamId = ApiService.createAgent(sessionId!!, userId!!, userStreamId)
            if (agentStreamId == null) { showError("Agent creation failed"); return@launch }

            agentInstanceId = agentStreamId
            engineManager.playAudio(agentStreamId)

            updateUI(true)
        }
    }

    private fun endSession() {
        lifecycleScope.launch {
            agentInstanceId?.let { ApiService.destroyAgent(it) }
            sessionId?.let { engineManager.leaveRoom(it) }

            sessionId = null
            userId = null
            agentInstanceId = null
            updateUI(false)
        }
    }

    override fun onDestroy() {
        super.onDestroy()
        engineManager.destroy()
    }
}

Session flow summary:

Step	Action	Component
1	Generate session ID	`AppConfig`
2	Request auth token	`ApiService.authenticate()`
3	Join RTC room	`ZegoEngineManager.joinRoom()`
4	Publish user audio	`ZegoEngineManager.publishAudio()`
5	Create AI agent	`ApiService.createAgent()`
6	Play AI audio	`ZegoEngineManager.playAudio()`
7	End session	`destroyAgent()` + `leaveRoom()`

Conclusion

This guide demonstrated how to build an AI voice assistant for Android using ZEGOCLOUD's integrated platform. The architecture separates concerns between backend authentication, agent orchestration, and client-side media handling. Key implementation points include proper token management, RTC room coordination, and subtitle message parsing.

For production deployment, consider implementing connection retry logic, offline fallbacks, and comprehensive error handling based on your specific use case requirements.

How to Build an AI Chatbot on Android with Flutter

Stephen568hub — Fri, 06 Mar 2026 10:19:14 +0000

Flutter has revolutionized cross-platform mobile development. With a single codebase, you can deploy to both Android and iOS—perfect for startups and enterprises alike. If you're looking to build an AI chatbot for Android using Flutter, this guide is for you.

This tutorial demonstrates how to create a fully functional Android AI chatbot using Flutter and ZEGOCLOUD's Conversational AI platform. You'll implement speech recognition, language processing, and text-to-speech—all through a clean Dart SDK. By the end, your Flutter app will deliver real-time voice conversations that work seamlessly across platforms.

Let's begin.

Quick Links

Resource	Repository
Backend Server + Web Client	github.com/ZEGOCLOUD/blog-aiagent-server-and-web
Flutter Client	github.com/ZEGOCLOUD/blog-aiagent-flutter

Prerequisites

Before you begin, ensure you have:

Flutter SDK (latest stable version)
Android Studio or VS Code with Flutter extensions
ZEGOCLOUD Account — Create free account
Backend deployment — Next.js on Vercel (covered in this guide)

Architecture Overview

System Components

ZEGOCLOUD's Conversational AI handles three core cloud services:

Service	Function
ASR (Automatic Speech Recognition)	Voice to text conversion
LLM (Large Language Model)	Response generation
TTS (Text-to-Speech)	Natural speech synthesis

System Flow

sequenceDiagram
    participant User as Flutter Client
    participant Server as Backend Server
    participant ZEGO as ZEGO Cloud
    participant LLM as LLM Service
    participant TTS as TTS Service

    User->>Server: 1. Request Token
    Server-->>User: 2. Return Token

    User->>ZEGO: 3. Login Room & Publish Audio Stream
    User->>Server: 4. Request to Start AI Agent
    Server->>ZEGO: 5. Register Agent (LLM/TTS config)
    Server->>ZEGO: 6. Create Agent Instance
    ZEGO-->>Server: 7. Return Agent Instance ID
    Server-->>User: 8. Return Agent Stream ID

    User->>ZEGO: 9. Play Agent's Audio Stream

    loop Conversation
        User->>ZEGO: User speaks (audio stream)
        ZEGO->>ZEGO: ASR: Speech to Text
        ZEGO->>LLM: Send text to LLM
        LLM-->>ZEGO: LLM response
        ZEGO->>TTS: Text to Speech
        TTS-->>ZEGO: Audio data
        ZEGO-->>User: AI voice + Subtitles
    end

    User->>Server: 10. Request to Stop AI Agent
    Server->>ZEGO: 11. Delete Agent Instance

Why Do We Need Both Server and Client?

When developing an AI chatbot for Android applications with Flutter, developers often ask: Why is a backend server necessary? This architecture decision is fundamental to building secure, production-ready applications.

The Three-Tier Architecture:

Tier	Responsibility	Trust Level
Backend Server	Token generation, AI agent lifecycle, API signing	High (your infrastructure)
Flutter Client	Audio I/O, stream management, UI rendering	Low (runs on user devices)
ZEGO Cloud	ASR, LLM, TTS processing	External (third-party service)

Security Considerations:

Credential Protection — Sensitive values like ZEGO_SERVER_SECRET and LLM API keys must remain on your server. Mobile apps can be reverse-engineered, exposing any embedded secrets.
Short-Lived Tokens — Instead of permanent credentials, clients request time-limited tokens (typically 1 hour). This containment strategy limits exposure if a token is intercepted.
Server-Side AI Configuration — System prompts, model selections, and voice settings live on your server. This enables dynamic AI behavior updates without requiring app store releases.
Monitoring & Controls — Your backend provides a single point for logging, rate limiting, abuse detection, and usage analytics.

Understanding this separation is critical when you build an AI chatbot on Android for production environments.

Step 1: Set Up the Backend

Your backend manages authentication and the AI agent lifecycle.

1.1 Environment Configuration

Create .env.local in your server root:

# ZEGO Credentials
NEXT_PUBLIC_ZEGO_APP_ID=your_app_id
ZEGO_SERVER_SECRET=your_32_char_secret

# AI Agent Settings
ZEGO_AGENT_ID=aiAgent1
ZEGO_AGENT_NAME=AI Assistant

# AI Personality
SYSTEM_PROMPT="You are my best friend whom I can talk to about anything. You're warm, understanding, and always there for me."

# LLM Configuration
LLM_URL=https://your-llm-provider.com/api/chat/completions
LLM_API_KEY=your_llm_api_key
LLM_MODEL=your_model_name

# TTS Configuration
TTS_VENDOR=ByteDance
TTS_APP_ID=zego_test
TTS_TOKEN=zego_test
TTS_CLUSTER=volcano_tts
TTS_VOICE_TYPE=zh_female_wanwanxiaohe_moon_bigtts

Variable	Purpose	Source
`NEXT_PUBLIC_ZEGO_APP_ID`	Application ID	ZEGOCLOUD Console
`ZEGO_SERVER_SECRET`	32-char secret	ZEGOCLOUD Console
`SYSTEM_PROMPT`	AI behavior	Customize per use case
`LLM_*`	Language model	Your LLM provider
`TTS_*`	Voice settings	Test or production TTS

1.2 Token Generation

// app/api/zego/token/route.ts
import { NextRequest, NextResponse } from 'next/server';
import crypto from 'crypto';

function generateToken(appId: number, userId: string, secret: string,
                       effectiveTimeInSeconds: number): string {
  const tokenInfo = {
    app_id: appId,
    user_id: userId,
    nonce: Math.floor(Math.random() * 2147483647),
    ctime: Math.floor(Date.now() / 1000),
    expire: Math.floor(Date.now() / 1000) + effectiveTimeInSeconds,
    payload: ''
  };

  const plainText = JSON.stringify(tokenInfo);
  const nonce = crypto.randomBytes(12);
  const cipher = crypto.createCipheriv('aes-256-gcm', secret, nonce);
  const encrypted = Buffer.concat([cipher.update(plainText, 'utf8'),
                                   cipher.final(), cipher.getAuthTag()]);

  const buf = Buffer.concat([
    Buffer.alloc(8).writeBigInt64BE(BigInt(tokenInfo.expire), 0),
    Buffer.from([0, 12]), nonce,
    Buffer.from([encrypted.length >> 8, encrypted.length & 0xff]), encrypted,
    Buffer.from([1])
  ]);
  return '04' + buf.toString('base64');
}

export async function POST(request: NextRequest) {
  const { userId } = await request.json();
  const token = generateToken(
    parseInt(process.env.NEXT_PUBLIC_ZEGO_APP_ID!),
    userId,
    process.env.ZEGO_SERVER_SECRET!,
    3600
  );
  return NextResponse.json({ token });
}

1.3 Deploy to Vercel

Push to GitHub
Import in Vercel
Add environment variables
Deploy

Server will be at https://your-project.vercel.app.

Step 2: Create the Flutter Application

Now we'll create an AI chatbot in Android using Flutter and Dart.

2.1 Create Flutter Project

flutter create aiagent_demo
cd aiagent_demo

2.2 Add Dependencies

Update pubspec.yaml:

dependencies:
  flutter:
    sdk: flutter

  # ZEGO Express SDK
  zego_express_engine: ^3.22.0

  # HTTP client
  http: ^1.2.0

  # Permission handling
  permission_handler: ^11.3.0

Run:

flutter pub get

2.3 Configure Permissions

Android (android/app/src/main/AndroidManifest.xml):

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

iOS (ios/Runner/Info.plist):

<key>NSMicrophoneUsageDescription</key>
<string>This app requires microphone access for voice chat</string>

2.4 App Configuration

// lib/config/app_config.dart
class AppConfig {
  // Must match backend NEXT_PUBLIC_ZEGO_APP_ID
  static const int appID = 1234567890;

  // Your Vercel deployment URL
  static const String serverURL = 'https://your-project.vercel.app';

  // Generate unique room ID
  static String generateRoomId() {
    final random = DateTime.now().millisecondsSinceEpoch % 100000;
    return 'room$random';
  }

  // Generate unique user ID
  static String generateUserId() {
    final random = DateTime.now().millisecondsSinceEpoch % 100000;
    return 'user$random';
  }

  // Generate stream IDs
  static String getAgentStreamId(String roomId) => 'agent_stream_$roomId';
  static String getUserStreamId(String roomId) => 'user_stream_$roomId';
}

2.5 API Service Layer

// lib/services/api_service.dart
import 'dart: convert';
import 'package:http/http.dart' as http;
import '../config/app_config.dart';

class ApiService {
  /// Get authentication token
  static Future<String?> getToken(String userId) async {
    try {
      final response = await http.post(
        Uri.parse('${AppConfig.serverURL}/api/zego/token'),
        headers: {'Content-Type': 'application/json'},
        body: jsonEncode({'userId': userId}),
      );

      if (response.statusCode == 200) {
        final data = jsonDecode(response.body);
        if (data['code'] == 0 && data['data'] != null) {
          return data['data']['token'] as String?;
        }
      }
      return null;
    } catch (e) {
      print('[ApiService] Token error: $e');
      return null;
    }
  }

  /// Start AI agent instance
  static Future<String?> startAgent({
    required String roomId,
    required String userId,
    required String userStreamId,
  }) async {
    try {
      final response = await http.post(
        Uri.parse('${AppConfig.serverURL}/api/zego/start'),
        headers: {'Content-Type': 'application/json'},
        body: jsonEncode({
          'roomId': roomId,
          'userId': userId,
          'userStreamId': userStreamId,
        }),
      );

      if (response.statusCode == 200) {
        final data = jsonDecode(response.body);
        if (data['code'] == 0 && data['data'] != null) {
          return data['data']['agentStreamId'] as String?;
        }
      }
      return null;
    } catch (e) {
      print('[ApiService] Start agent error: $e');
      return null;
    }
  }

  /// Stop AI agent
  static Future<bool> stopAgent(String roomId) async {
    try {
      final response = await http.post(
        Uri.parse('${AppConfig.serverURL}/api/zego/stop'),
        headers: {'Content-Type': 'application/json'},
        body: jsonEncode({'roomId': roomId}),
      );

      return response.statusCode == 200;
    } catch (e) {
      print('[ApiService] Stop agent error: $e');
      return false;
    }
  }
}

2.6 ZEGO Engine Manager

// lib/services/zego_express_manager.dart
import 'package:zego_express_engine/zego_express_engine.dart';
import '../config/app_config.dart';

class ZegoExpressManager {
  static final ZegoExpressManager _instance = ZegoExpressManager._internal();
  factory ZegoExpressManager() => _instance;
  ZegoExpressManager._internal();

  Function(String, ZegoRoomStateChangedReason, int)? onRoomStateChanged;
  bool _isInitialized = false;

  /// Initialize ZEGO Express Engine
  Future<void> initEngine() async {
    if (_isInitialized) return;

    // Configure engine
    final engineConfig = ZegoEngineConfig(advancedConfig: {
      'set_audio_volume_ducking_mode': '1',
      'enable_rnd_volume_adaptive': 'true',
    });
    await ZegoExpressEngine.setEngineConfig(engineConfig);

    // Create engine with profile
    final profile = ZegoEngineProfile(
      AppConfig.appID,
      ZegoScenario.HighQualityChatroom,
      enablePlatformView: false,
    );
    await ZegoExpressEngine.createEngineWithProfile(profile);

    // Setup event handlers
    _setupEventHandlers();

    // Configure audio (3A processing)
    await _configureAudioSettings();

    _isInitialized = true;
  }

  void _setupEventHandlers() {
    ZegoExpressEngine.onRoomStateChanged = (roomID, reason, errorCode, _) {
      onRoomStateChanged?.call(roomID, reason, errorCode);
    };
  }

  Future<void> _configureAudioSettings() async {
    final engine = ZegoExpressEngine.instance;
    await engine.enableAGC(true);  // Automatic Gain Control
    await engine.enableAEC(true);  // Echo Cancellation
    await engine.setAECMode(ZegoAECMode.AIBalanced);
    await engine.enableANS(true);  // Noise Suppression
    await engine.setANSMode(ZegoANSMode.Medium);
  }

  /// Login to room
  Future<int> loginRoom(String roomId, String userId, String token) async {
    final user = ZegoUser(userId, userId);
    final config = ZegoRoomConfig(0, true, token);
    final result = await ZegoExpressEngine.instance.loginRoom(
      roomId, user, config: config,
    );
    return result.errorCode;
  }

  /// Start publishing audio
  Future<void> startPublishing(String streamId) async {
    await ZegoExpressEngine.instance.muteMicrophone(false);
    await ZegoExpressEngine.instance.startPublishingStream(streamId);
  }

  /// Play remote stream
  Future<void> startPlaying(String streamId) async {
    await ZegoExpressEngine.instance.startPlayingStream(streamId);
  }

  /// Stop publishing
  Future<void> stopPublishing() async {
    await ZegoExpressEngine.instance.stopPublishingStream();
  }

  /// Stop playing
  Future<void> stopPlaying(String streamId) async {
    await ZegoExpressEngine.instance.stopPlayingStream(streamId);
  }

  /// Logout from room
  Future<void> logoutRoom(String roomId) async {
    await ZegoExpressEngine.instance.logoutRoom(roomId);
  }

  /// Destroy engine
  Future<void> destroyEngine() async {
    if (!_isInitialized) return;
    await ZegoExpressEngine.destroyEngine();
    _isInitialized = false;
  }
}

2.7 ViewModel (Provider Pattern)

// lib/viewmodels/chat_viewmodel.dart
import 'package:flutter/foundation.dart';
import '../services/api_service.dart';
import '../services/zego_express_manager.dart';

class ChatViewModel extends ChangeNotifier {
  final ZegoExpressManager _zegoManager = ZegoExpressManager();

  bool _isConnected = false;
  bool _isLoading = false;
  String? _errorMessage;
  String? _currentRoomId;
  String? _currentUserId;

  bool get isConnected => _isConnected;
  bool get isLoading => _isLoading;
  String? get errorMessage => _errorMessage;

  String get statusText {
    if (_isLoading) return 'Processing...';
    return _isConnected ? 'Connected' : 'Not connected';
  }

  ChatViewModel() {
    _setupCallbacks();
  }

  void _setupCallbacks() {
    _zegoManager.onRoomStateChanged = (roomId, reason, errorCode) {
      if (reason == ZegoRoomStateChangedReason.Logined) {
        _isConnected = true;
        notifyListeners();
      } else if (reason == ZegoRoomStateChangedReason.Logout ||
                 reason == ZegoRoomStateChangedReason.KickOut) {
        _isConnected = false;
        notifyListeners();
      }
    };
  }

  /// Start AI conversation
  Future<void> startCall() async {
    if (_isLoading) return;

    _isLoading = true;
    _errorMessage = null;
    notifyListeners();

    try {
      _currentRoomId = AppConfig.generateRoomId();
      _currentUserId = AppConfig.generateUserId();
      final userStreamId = AppConfig.getUserStreamId(_currentRoomId!);

      // Initialize
      await _zegoManager.initEngine();

      // Get token
      final token = await ApiService.getToken(_currentUserId!);
      if (token == null) throw Exception('Failed to get token');

      // Login room
      final loginResult = await _zegoManager.loginRoom(
        _currentRoomId!, _currentUserId!, token,
      );
      if (loginResult != 0) throw Exception('Login failed: $loginResult');

      // Publish stream
      await _zegoManager.startPublishing(userStreamId);

      // Start AI agent
      final agentStreamId = await ApiService.startAgent(
        roomId: _currentRoomId!,
        userId: _currentUserId!,
        userStreamId: userStreamId,
      );
      if (agentStreamId == null) throw Exception('Failed to start AI agent');

      // Play agent stream
      await _zegoManager.startPlaying(agentStreamId);

      _isConnected = true;
    } catch (e) {
      _errorMessage = e.toString();
      await _cleanup();
    } finally {
      _isLoading = false;
      notifyListeners();
    }
  }

  /// End call
  Future<void> endCall() async {
    if (_isLoading) return;

    _isLoading = true;
    notifyListeners();

    await _cleanup();

    _isLoading = false;
    notifyListeners();
  }

  Future<void> _cleanup() async {
    if (_currentRoomId != null) {
      await ApiService.stopAgent(_currentRoomId!);
      await _zegoManager.stopPlaying(
        AppConfig.getAgentStreamId(_currentRoomId!));
      await _zegoManager.stopPublishing();
      await _zegoManager.logoutRoom(_currentRoomId!);
    }
    _isConnected = false;
    _currentRoomId = null;
    _currentUserId = null;
  }

  @override
  void dispose() {
    _zegoManager.destroyEngine();
    super.dispose();
  }
}

2.8 Subtitle Display

Create subtitle models and views using ZEGO's official components. Download from ZEGO Flutter Subtitle Guide.

Key files:

lib/audio/subtitles/model.dart — Message data model
lib/audio/subtitles/view.dart — UI display widget
lib/audio/subtitles/message_protocol.dart — Message parsing
lib/audio/subtitles/message_dispatcher.dart — Event routing

2.9 Main UI (ChatPage)

// lib/widgets/chat_page.dart
import 'package:flutter/material.dart';
import 'package:zego_express_engine/zego_express_engine.dart';
import '../viewmodels/chat_viewmodel.dart';
import '../audio/subtitles/view.dart';
import '../audio/subtitles/model.dart';
import '../audio/subtitles/message_dispatcher.dart';

class ChatPage extends StatefulWidget {
  const ChatPage({super.key});

  @override
  State<ChatPage> createState() => _ChatPageState();
}

class _ChatPageState extends State<ChatPage> implements ZegoSubtitlesEventHandler {
  final ChatViewModel _viewModel = ChatViewModel();
  late ZegoSubtitlesViewModel _subtitlesModel;

  @override
  void initState() {
    super.initState();
    _subtitlesModel = ZegoSubtitlesViewModel();
    ZegoExpressEngine.onRecvExperimentalAPI = _onRecvExperimentalAPI;
    ZegoSubtitlesMessageDispatcher().registerEventHandler(this);
  }

  void _onRecvExperimentalAPI(String content) {
    ZegoSubtitlesMessageDispatcher.handleExpressExperimentalAPIContent(content);
  }

  @override
  void onRecvAsrChatMsg(ZegoSubtitlesMessageProtocol message) {
    _subtitlesModel.handleRecvAsrMessage(message);
  }

  @override
  void onRecvLLMChatMsg(ZegoSubtitlesMessageProtocol message) {
    _subtitlesModel.handleRecvLLMMessage(message);
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: SafeArea(
        child: ListenableBuilder(
          listenable: _viewModel,
          builder: (context, _) {
            return Column(
              children: [
                _buildControlPanel(),
                const Divider(height: 1),
                Expanded(child: ZegoSubtitlesView(model: _subtitlesModel)),
              ],
            );
          },
        ),
      ),
    );
  }

  Widget _buildControlPanel() {
    return Container(
      padding: const EdgeInsets.symmetric(vertical: 40, horizontal: 20),
      color: Colors.grey[100],
      child: Column(
        children: [
          const Text('ZEGO AI Agent',
              style: TextStyle(fontSize: 24, fontWeight: FontWeight.bold)),
          const SizedBox(height: 20),
          // Status indicator
          Row(
            mainAxisAlignment: MainAxisAlignment.center,
            children: [
              Container(
                width: 12, height: 12,
                decoration: BoxDecoration(
                  shape: BoxShape.circle,
                  color: _viewModel.isConnected ? Colors.green : Colors.grey,
                ),
              ),
              const SizedBox(width: 8),
              Text(_viewModel.statusText,
                  style: TextStyle(
                    fontSize: 16,
                    color: _viewModel.isConnected ? Colors.green : Colors.grey,
                  )),
            ],
          ),
          const SizedBox(height: 20),
          // Call button
          SizedBox(
            width: 120, height: 120,
            child: ElevatedButton(
              onPressed: _viewModel.isLoading
                  ? null
                  : (_viewModel.isConnected ? _viewModel.endCall : _viewModel.startCall),
              style: ElevatedButton.styleFrom(
                shape: const CircleBorder(),
                backgroundColor: _viewModel.isConnected ? Colors.red : Colors.green,
                foregroundColor: Colors.white,
              ),
              child: _viewModel.isLoading
                  ? const CircularProgressIndicator(color: Colors.white)
                  : Icon(_viewModel.isConnected ? Icons.call_end : Icons.call, size: 48),
            ),
          ),
        ],
      ),
    );
  }
}

2.10 App Entry Point

// lib/main.dart
import 'package:flutter/material.dart';
import 'package:permission_handler/permission_handler.dart';
import 'widgets/chat_page.dart';

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'AI Agent Demo',
      debugShowCheckedModeBanner: false,
      theme: ThemeData(
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.blue),
        useMaterial3: true,
      ),
      home: const PermissionWrapper(),
    );
  }
}

// Permission handling wrapper
class PermissionWrapper extends StatefulWidget {
  const PermissionWrapper({super.key});

  @override
  State<PermissionWrapper> createState() => _PermissionWrapperState();
}

class _PermissionWrapperState extends State<PermissionWrapper> {
  bool _permissionGranted = false;
  bool _permissionChecked = false;

  @override
  void initState() {
    super.initState();
    _checkPermission();
  }

  Future<void> _checkPermission() async {
    final status = await Permission.microphone.status;
    if (status.isGranted) {
      setState(() {
        _permissionGranted = true;
        _permissionChecked = true;
      });
    } else {
      final result = await Permission.microphone.request();
      setState(() {
        _permissionGranted = result.isGranted;
        _permissionChecked = true;
      });
    }
  }

  @override
  Widget build(BuildContext context) {
    if (!_permissionChecked) {
      return const Scaffold(
        body: Center(child: CircularProgressIndicator()),
      );
    }

    if (!_permissionGranted) {
      return Scaffold(
        body: Center(
          child: Column(
            mainAxisAlignment: MainAxisAlignment.center,
            children: [
              const Icon(Icons.mic_off, size: 64, color: Colors.grey),
              const SizedBox(height: 16),
              const Text('Microphone permission required',
                  style: TextStyle(fontSize: 18)),
              const SizedBox(height: 16),
              ElevatedButton(
                onPressed: () async {
                  await openAppSettings();
                  _checkPermission();
                },
                child: const Text('Open Settings'),
              ),
            ],
          ),
        ),
      );
    }

    return const ChatPage();
  }
}

Run the Application

Android:

flutter run

iOS:

flutter run -d ios

Conclusion

You've now built a cross-platform AI chatbot for Android and iOS using Flutter. Here's what you've accomplished:

Secure token-based authentication
Real-time voice streaming with ZEGOCLOUD
AI-powered responses with natural TTS
Live subtitle display for accessibility
Single codebase for Android and iOS

Next steps:

Customize SYSTEM_PROMPT for your brand voice
Add visual avatars for enhanced UX
Test on physical devices for optimal audio

The same architecture works for customer service bots, language tutors, AI companions, and more.

Ready to deploy? Start with ZEGOCLOUD and launch your Flutter AI chatbot today.

Questions? Visit the ZEGOCLOUD community for support and inspiration.

youtube link: https://www.youtube.com/shorts/8bwF3l-trEU

How to Build an AI Chatbot on Android

Stephen568hub — Wed, 04 Mar 2026 10:04:23 +0000

Voice-first AI applications are reshaping mobile experiences. Users expect more than text input—they want natural, spoken conversations with their apps. If you're looking to build an AI chatbot for Android, you've come to the right place.

This tutorial walks you through creating a fully functional Android AI chatbot from scratch. We'll use ZEGOCLOUD's Conversational AI platform to handle speech recognition, language processing, and text-to-speech—all through a clean, integrated SDK. By the end, your Android app will have real-time voice conversation capabilities that rival the big tech apps.

Let's get started.

Quick Links

Resource	Repository
Backend Server + Web Client	github.com/ZEGOCLOUD/blog-aiagent-server-and-web
Android Client	github.com/ZEGOCLOUD/blog-aiagent-android

Prerequisites

Before diving in, ensure you have:

Android Studio (latest stable version)
Kotlin programming experience
ZEGOCLOUD Account — Create free account
Backend deployment — We'll use Next.js on Vercel (one-click deploy)

Architecture Overview

Understanding the Components

ZEGOCLOUD's Conversational AI runs as a cloud service handling three core functions:

Service	Function
ASR (Automatic Speech Recognition)	Converts voice to text
LLM (Large Language Model)	Processes and generates responses
TTS (Text-to-Speech)	Converts text back to natural speech

System Flow

sequenceDiagram
    participant User as Android Client
    participant Server as Backend Server
    participant ZEGO as ZEGO Cloud
    participant LLM as LLM Service
    participant TTS as TTS Service

    User->>Server: 1. Request Token
    Server-->>User: 2. Return Token

    User->>ZEGO: 3. Login Room & Publish Audio Stream
    User->>Server: 4. Request to Start AI Agent
    Server->>ZEGO: 5. Register Agent (LLM/TTS config)
    Server->>ZEGO: 6. Create Agent Instance
    ZEGO-->>Server: 7. Return Agent Instance ID
    Server-->>User: 8. Return Agent Stream ID

    User->>ZEGO: 9. Play Agent's Audio Stream

    loop Conversation
        User->>ZEGO: User speaks (audio stream)
        ZEGO->>ZEGO: ASR: Speech to Text
        ZEGO->>LLM: Send text to LLM
        LLM-->>ZEGO: LLM response
        ZEGO->>TTS: Text to Speech
        TTS-->>ZEGO: Audio data
        ZEGO-->>User: AI voice + Subtitles
    end

    User->>Server: 10. Request to Stop AI Agent
    Server->>ZEGO: 11. Delete Agent Instance

Why Do We Need Both Server and Client?

A common question when building an AI chatbot on Android applications: why can't the client connect directly to ZEGOCLOUD? The answer lies in security and architectural design.

Component	Role	Security Boundary
Backend Server	Token generation, AI agent management, API signing	Trusted environment (your control)
Android Client	Audio capture, stream publishing, UI display	Untrusted environment (user devices)
ZEGO Cloud	ASR, LLM, TTS processing	External service

Key reasons for this architecture:

Secret Protection — Your ZEGO_SERVER_SECRET and LLM API keys never leave your server. If these credentials were embedded in the Android app, attackers could decompile it and steal them.
Token-Based Authentication — Clients request short-lived tokens (typically 1 hour) instead of using permanent credentials. This limits the damage if a token is compromised.
Centralized AI Configuration — System prompts, LLM settings, and TTS voices are configured on your server, not in the client. This allows you to update AI behavior without app updates.
Audit & Rate Limiting — Your server can log all AI agent requests, implement rate limiting, and monitor usage patterns.

This separation ensures your build an AI chatbot for Android deployment remains secure and maintainable in production.

Step 1: Set Up the Backend

Your backend handles authentication and AI agent management. We use Next.js for simplicity.

1.1 Environment Configuration

Create a .env.local file in your server root:

# ZEGO Credentials (from Console: https://console.zegocloud.com/)
NEXT_PUBLIC_ZEGO_APP_ID=your_app_id
ZEGO_SERVER_SECRET=your_32_char_secret

# AI Agent Settings
ZEGO_AGENT_ID=aiAgent1
ZEGO_AGENT_NAME=AI Assistant

# Define AI Personality
SYSTEM_PROMPT="You are my best friend who I can talk to about anything. You're warm, understanding, and always there for me. Respond naturally like a close friend would."

# LLM Provider (OpenAI, Doubao, Claude, etc.)
LLM_URL=https://your-llm-provider.com/api/chat/completions
LLM_API_KEY=your_llm_api_key
LLM_MODEL=your_model_name

# Text-to-Speech Configuration
TTS_VENDOR=ByteDance
TTS_APP_ID=zego_test
TTS_TOKEN=zego_test
TTS_CLUSTER=volcano_tts
TTS_VOICE_TYPE=zh_female_wanwanxiaohe_moon_bigtts

Variable	Purpose	Where to Get
`NEXT_PUBLIC_ZEGO_APP_ID`	Application identifier	ZEGOCLOUD Console → Project Settings
`ZEGO_SERVER_SECRET`	32-character secret key	ZEGOCLOUD Console → Project Settings
`SYSTEM_PROMPT`	AI behavior definition	Customize for your use case
`LLM_*`	Language model config	From your LLM provider
`TTS_*`	Voice synthesis settings	Use test values or your TTS service

1.2 Token Generation API

// app/api/zego/token/route.ts
import { NextRequest, NextResponse } from 'next/server';
import crypto from 'crypto';

function generateToken(appId: number, userId: string, secret: string,
                       effectiveTimeInSeconds: number): string {
  const tokenInfo = {
    app_id: appId,
    user_id: userId,
    nonce: Math.floor(Math.random() * 2147483647),
    ctime: Math.floor(Date.now() / 1000),
    expire: Math.floor(Date.now() / 1000) + effectiveTimeInSeconds,
    payload: ''
  };

  const plainText = JSON.stringify(tokenInfo);
  const nonce = crypto.randomBytes(12);
  const cipher = crypto.createCipheriv('aes-256-gcm', secret, nonce);
  const encrypted = Buffer.concat([cipher.update(plainText, 'utf8'),
                                   cipher.final(), cipher.getAuthTag()]);

  const buf = Buffer.concat([
    Buffer.alloc(8).writeBigInt64BE(BigInt(tokenInfo.expire), 0) || Buffer.alloc(8),
    Buffer.from([0, 12]), nonce,
    Buffer.from([encrypted.length >> 8, encrypted.length & 0xff]), encrypted,
    Buffer.from([1]) // GCM mode
  ]);
  return '04' + buf.toString('base64');
}

export async function POST(request: NextRequest) {
  const { userId } = await request.json();
  const token = generateToken(
    parseInt(process.env.NEXT_PUBLIC_ZEGO_APP_ID!),
    userId,
    process.env.ZEGO_SERVER_SECRET!,
    3600
  );
  return NextResponse.json({ token });
}

1.3 AI Agent Signature Utility

// app/api/zego/utils.ts
import crypto from 'crypto';

export function generateSignature(appId: number, signatureNonce: string,
                                  serverSecret: string, timestamp: number): string {
  const str = appId.toString() + signatureNonce + serverSecret + timestamp.toString();
  return crypto.createHash('md5').update(str).digest('hex');
}

export async function sendZegoRequest<T>(action: string, body: object): Promise<T> {
  const appId = parseInt(process.env.NEXT_PUBLIC_ZEGO_APP_ID!);
  const serverSecret = process.env.ZEGO_SERVER_SECRET!;
  const signatureNonce = crypto.randomBytes(8).toString('hex');
  const timestamp = Math.floor(Date.now() / 1000);
  const signature = generateSignature(appId, signatureNonce, serverSecret, timestamp);

  const url = new URL('https://aigc-aiagent-api.zegotech.cn');
  url.searchParams.set('Action', action);
  url.searchParams.set('AppId', appId.toString());
  url.searchParams.set('SignatureNonce', signatureNonce);
  url.searchParams.set('Timestamp', timestamp.toString());
  url.searchParams.set('Signature', signature);
  url.searchParams.set('SignatureVersion', '2.0');

  const response = await fetch(url.toString(), {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(body)
  });

  const result = await response.json();
  return result.Data as T;
}

1.4 Deploy to Vercel

Push code to GitHub
Import repository in Vercel
Configure environment variables
Click Deploy

Your server will be live at https://your-project.vercel.app.

Step 2: Build the Android Client

Now we'll create an AI chatbot in Android using Kotlin and the ZEGOCLOUD SDK.

2.1 Create New Project

Start a new Android project:

Language: Kotlin
Minimum SDK: API 24

2.2 Add Dependencies

Important: Use the AI Agent-optimized ZEGO SDK. The standard Maven version won't support subtitle callbacks (onRecvExperimentalAPI).

Download from: ZEGO AI Agent SDK

Integration steps:

Extract and copy files:
- ZegoExpressEngine.jar → app/libs/
- libZegoExpressEngine.so (arm64) → app/libs/arm64-v8a/
- libZegoExpressEngine.so (v7a) → app/libs/armeabi-v7a/
Update app/build.gradle:

android {
    defaultConfig {
        ndk {
            abiFilters 'armeabi-v7a', 'arm64-v8a'
        }
    }

    sourceSets {
        main {
            jniLibs.srcDirs = ['libs']
        }
    }
}

dependencies {
    // ZEGO Express SDK (AI Agent version)
    implementation files('libs/ZegoExpressEngine.jar')

    // Networking & JSON
    implementation 'com.squareup.okhttp3:okhttp:4.12.0'
    implementation 'com.google.code.gson:gson:2.10.1'

    // Coroutines
    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-android:1.7.3'
}

2.3 Add Permissions

In AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

2.4 Configuration Class

Centralize your app settings:

// config/AppConfig.kt
object AppConfig {
    // Must match NEXT_PUBLIC_ZEGO_APP_ID in backend
    const val APP_ID: Long = 1234567890L

    // Your Vercel deployment URL
    const val SERVER_URL = "https://your-project.vercel.app"

    // Helper functions for unique IDs
    fun generateUserId(): String = "user${System.currentTimeMillis() % 100000}"
    fun generateRoomId(): String = "room${System.currentTimeMillis() % 100000}"
}

Note: Use http://10.0.2.2:3000 for emulator testing to access localhost.

2.5 API Service Layer

Handle all backend communication:

// api/ApiService.kt
object ApiService {
    private val client = OkHttpClient.Builder()
        .connectTimeout(30, TimeUnit.SECONDS)
        .readTimeout(30, TimeUnit.SECONDS)
        .build()
    private val gson = Gson()
    private val JSON = "application/json; charset=utf-8".toMediaType()

    // Get authentication token
    suspend fun getToken(userId: String): TokenResponse = withContext(Dispatchers.IO) {
        val body = mapOf("userId" to userId)
        val request = Request.Builder()
            .url("${AppConfig.SERVER_URL}/api/zego/token")
            .post(gson.toJson(body).toRequestBody(JSON))
            .build()

        val response = client.newCall(request).execute()
        val responseBody = response.body?.string()
        gson.fromJson(responseBody, TokenResponse::class.java)
    }

    // Start AI agent instance
    suspend fun startAgent(roomId: String, userId: String, userStreamId: String): AgentResponse =
        withContext(Dispatchers.IO) {
            val body = mapOf(
                "roomId" to roomId,
                "userId" to userId,
                "userStreamId" to userStreamId
            )
            val request = Request.Builder()
                .url("${AppConfig.SERVER_URL}/api/zego/start")
                .post(gson.toJson(body).toRequestBody(JSON))
                .build()

            val response = client.newCall(request).execute()
            val responseBody = response.body?.string()
            gson.fromJson(responseBody, AgentResponse::class.java)
        }

    // Stop AI agent
    suspend fun stopAgent(agentInstanceId: String): StopResponse = withContext(Dispatchers.IO) {
        val body = mapOf("agentInstanceId" to agentInstanceId)
        val request = Request.Builder()
            .url("${AppConfig.SERVER_URL}/api/zego/stop")
            .post(gson.toJson(body).toRequestBody(JSON))
            .build()

        val response = client.newCall(request).execute()
        val responseBody = response.body?.string()
        gson.fromJson(responseBody, StopResponse::class.java)
    }
}

// Response data classes
data class TokenResponse(val code: Int?, val data: TokenData?, val message: String?)
data class TokenData(val token: String?)

data class AgentResponse(val code: Int?, val data: AgentData?, val message: String?)
data class AgentData(val agentInstanceId: String?, val agentUserId: String?, val agentStreamId: String?)

data class StopResponse(val code: Int?, val message: String?)

2.6 Initialize ZEGO Engine

Set up audio-optimized RTC:

class ZegoExpressManager(private val application: Application) {
    private var engine: ZegoExpressEngine? = null

    fun initEngine() {
        val profile = ZegoEngineProfile().apply {
            appID = AppConfig.APP_ID
            scenario = ZegoScenario.HIGH_QUALITY_CHATROOM
            application = this@ZegoExpressManager.application
        }

        engine = ZegoExpressEngine.createEngine(profile, eventHandler)

        // Audio optimization
        engine?.apply {
            enableAGC(true)  // Automatic Gain Control
            enableANS(true)  // Noise Suppression
            setANSMode(ZegoANSMode.MEDIUM)
        }
    }
}

2.7 Room & Stream Management

// Login to RTC room
fun loginRoom(roomId: String, userId: String, token: String, callback: (Int) -> Unit) {
    val user = ZegoUser(userId)
    val config = ZegoRoomConfig().apply { this.token = token }
    engine?.loginRoom(roomId, user, config) { errorCode, _ ->
        callback(errorCode)
    }
}

// Publish user's audio stream
fun startPublishing(streamId: String) {
    engine?.startPublishingStream(streamId)
}

// Play AI agent's audio stream
fun startPlaying(streamId: String) {
    engine?.startPlayingStream(streamId)
}

2.8 Subtitle Display

ZEGO provides official components for subtitle parsing:

// Create parser instance
private val audioChatMessageParser = AudioChatMessageParser()

// Set up listener
audioChatMessageParser.setAudioChatMessageListListener(object : AudioChatMessageParser.AudioChatMessageListListener {
    override fun onMessageListUpdated(messagesList: MutableList<AudioChatMessage>) {
        runOnUiThread {
            binding.messageList.onMessageListUpdated(messagesList)
        }
    }

    override fun onAudioChatStateUpdate(statusMessage: AudioChatAgentStatusMessage) {
        // Handle status updates
    }
})

// Parse incoming messages
zegoManager.onRecvExperimentalAPI = { content ->
    try {
        val json = JSONObject(content)
        if (json.getString("method") == "liveroom.room.on_recive_room_channel_message") {
            val msgContent = json.getJSONObject("params").getString("msg_content")
            audioChatMessageParser.parseAudioChatMessage(msgContent)
        }
    } catch (e: Exception) { e.printStackTrace() }
}

Download AudioChatMessageParser.java and AIChatListView.java from ZEGO Subtitle Guide.

2.9 Layout Design

Simple two-section UI:

<ConstraintLayout>
    <!-- Top: Status + Controls -->
    <LinearLayout android:id="@+id/topSection">
        <TextView android:id="@+id/tvStatus" />
        <Button android:id="@+id/btnCall" android:text="Start Call" />
    </LinearLayout>

    <!-- Bottom: Subtitles -->
    <com.zegocloud.aiagent.subtitle.AIChatListView
        android:id="@+id/messageList" />
</ConstraintLayout>

2.10 Complete Call Flow (ViewModel)

Orchestrate the entire conversation flow:

class MainViewModel(private val zegoManager: ZegoExpressManager) : ViewModel() {

    fun startCall() {
        viewModelScope.launch {
            _isLoading.value = true
            try {
                val roomId = AppConfig.generateRoomId()
                val userId = AppConfig.generateUserId()
                val userStreamId = "${roomId}_${userId}_main"

                zegoManager.initEngine()

                // 1. Get token
                val token = ApiService.getToken(userId)
                    ?: throw Exception("Failed to get token")

                // 2. Login room
                val loginResult = zegoManager.loginRoom(roomId, userId, token)
                if (loginResult != 0) throw Exception("Login failed")

                // 3. Publish stream
                zegoManager.startPublishing(userStreamId)

                // 4. Start AI agent
                val agentStreamId = ApiService.startAgent(roomId, userId, userStreamId)
                    ?: throw Exception("Failed to start agent")

                // 5. Play agent stream
                zegoManager.startPlaying(agentStreamId)

                _isConnected.value = true
                _currentRoomId = roomId
            } catch (e: Exception) {
                _error.value = e.message
            } finally {
                _isLoading.value = false
            }
        }
    }

    fun endCall() {
        viewModelScope.launch {
            _currentRoomId?.let { roomId ->
                ApiService.stopAgent(roomId)
                zegoManager.logoutRoom(roomId)
            }
            _isConnected.value = false
            _currentRoomId = null
        }
    }
}

Step	Action	Method
1-2	Authentication	`ApiService.getToken()`
3	Connect & Publish	`loginRoom()` + `startPublishing()`
4-8	Launch AI	`ApiService.startAgent()`
9	Receive Response	`startPlaying()`
10-11	Cleanup	`ApiService.stopAgent()`

Conclusion

You now have a production-ready AI chatbot for Android with full voice conversation capabilities. Here's what you've built:

Secure token-based authentication
Real-time voice streaming via ZEGOCLOUD
AI-powered responses with natural TTS
Live subtitle display for accessibility

Next steps to consider:

Customize the SYSTEM_PROMPT to match your brand voice
Add visual avatars for enhanced engagement
Test on physical devices for optimal audio performance

The same architecture scales to customer service bots, language tutors, AI companions, and more. The foundation is solid—now it's time to innovate.

Ready to ship? Start with ZEGOCLOUD and deploy your Android AI chatbot today.

Questions or feedback? Reach out to the ZEGOCLOUD community for support and inspiration.

Top 6 Tips for Implementing Real-Time Chat in Gaming Apps

Stephen568hub — Fri, 13 Feb 2026 07:19:16 +0000

The gaming industry has become one of the most competitive digital markets worldwide. Players today expect more than engaging mechanics and high-quality graphics. Social interaction has become a core element of player retention and long-term community growth. In modern multiplayer environments, in-app communication is no longer a secondary feature. It directly shapes how players collaborate, compete, and build relationships inside the game world.

If you are building or scaling a gaming application, integrating a real-time chat solution can significantly enhance user engagement and business performance. Below are six practical considerations that can help your game stand out.

1. Strengthen Player Interaction and Team Coordination

At its core, multiplayer gaming depends on effective collaboration. When players can communicate clearly, they coordinate strategies more efficiently and feel more connected to their team. While text messaging remains important for planning and structured discussion, real-time messaging systems allow multiple players to exchange information instantly across devices without noticeable delay.

Modern chat platforms also provide synchronized message history, read status tracking, and push notifications. These features ensure that players remain connected even when switching devices or temporarily going offline. By reducing communication friction, you create a smoother gameplay experience and stronger team dynamics.

2. Introduce Real-Time Voice for Faster Decision-Making

Text communication works well for strategy, but during active gameplay, voice interaction is often more efficient. Real-time voice channels allow players to react instantly to threats, coordinate movements, and share updates without interrupting gameplay to type.

A flexible communication system should support multiple modes, including text chat, voice messages, and live voice rooms. It should also allow clear permission management, such as team-only channels, squad chat, or private conversations. Seamless switching between communication methods enhances immersion and supports different gameplay scenarios. In competitive environments where timing is critical, stable voice interaction can directly influence outcomes.

3. Enable Global Connectivity

Multiplayer games thrive when they bring together players from different regions. Expanding communication capabilities across borders increases the potential user base and strengthens community effects. However, global connectivity requires a stable infrastructure that can handle high concurrency and maintain consistent performance.

Players located in different countries should experience similar latency and connection stability. When cross-region communication feels smooth, the game naturally becomes more attractive to a broader audience and benefits from organic growth driven by community participation.

4. Ensure Cross-Platform Compatibility

Today’s players access games through various devices and operating systems. Communication features must function reliably across Android, iOS, and web platforms while maintaining a consistent user experience.

This involves managing microphone permissions correctly, optimizing audio encoding to reduce device resource consumption, and ensuring interface layouts adapt to different screen sizes. A well-designed SDK simplifies integration across platforms and reduces development overhead. Thorough testing under different hardware conditions and network environments is essential to maintain stable performance.

5. Optimize for Low Latency and Reliable Message Delivery

In competitive gaming scenarios, latency can undermine the entire experience. Delayed voice responses or dropped messages frustrate players and can affect fairness during gameplay. Efficient transmission protocols such as WebSocket enable near real-time message delivery, while acknowledgment mechanisms and automatic retry strategies prevent message loss during unstable connections.

Content distribution networks and intelligent routing strategies further reduce transmission delays. For example, solutions such as ZEGOCLOUD’s real-time messaging and audio SDK provide low-latency communication supported by global network scheduling, helping developers deliver stable performance across more than 200 countries and regions. Reliable communication infrastructure builds trust and supports large-scale multiplayer interaction.

6. Implement Content Moderation and Safety Controls

Healthy gaming communities require structured moderation systems. Real-time filtering can automatically detect and block inappropriate language, while AI-assisted review mechanisms reduce manual moderation workload. Clear reporting tools empower players to flag violations quickly.

Additional safeguards, such as message frequency limits and permission-based voice access, help prevent abuse. Establishing these mechanisms protects both players and the platform’s reputation, contributing to sustainable community growth.

Conclusion

In-game communication has evolved from a supporting feature to a core component of multiplayer design. Real-time chat, voice interaction, global connectivity, and moderation systems directly influence engagement and retention. For developers, the challenge is not only integrating messaging functionality but ensuring scalability, low latency, cross-platform compatibility, and infrastructure stability.

Whether you choose to build your own communication architecture or adopt platforms such as ZEGOCLOUD to accelerate integration, the objective remains consistent: provide players with communication tools that feel immediate, reliable, and seamless. In a competitive gaming market, high-quality real-time interaction can determine whether a game builds a lasting community or fades after initial downloads.

How to Build an LGBTQ Dating App

Stephen568hub — Tue, 13 Jan 2026 07:08:13 +0000

Building a dating app for the LGBTQ community requires more than standard matching logic. It demands thoughtful design, inclusive identity support, and strong safety mechanisms. Users need spaces where they can express themselves comfortably, without fear of harassment or misrepresentation.

This article explores what makes LGBTQ dating apps different, why they matter, and how developers can approach building inclusive, real-world products with long-term value.

What Is an LGBTQ Dating App?

An LGBTQ dating app is a mobile or web platform designed to help lesbian, gay, bisexual, transgender, queer, and non-binary users connect for dating, relationships, friendships, or casual interactions.

Unlike mainstream dating apps, these platforms are built with LGBTQ identities in mind from the beginning. This includes inclusive language, flexible profile structures, safety-focused features, and community guidelines that address real risks such as harassment or unwanted outing.

Most LGBTQ dating apps allow users to select gender identities and pronouns, define sexual orientation and relationship intent, control profile visibility, and match with others based on identity compatibility rather than assumptions. Many also go beyond dating by offering groups, events, or community spaces that support social connection.

Why LGBTQ Dating Apps Matter?

For many LGBTQ users, dating apps are not just convenient tools. They are often the primary way to meet partners safely, especially in regions where offline spaces are limited or unsafe. Studies consistently show that LGBTQ adults rely on dating apps at higher rates than straight users, and many long-term relationships now begin online.

From a product perspective, LGBTQ-focused platforms represent a growing yet still underserved segment. While mainstream apps have added inclusive options, many users prefer platforms where inclusivity is a core design principle rather than an afterthought. This creates opportunities for products that prioritize identity-first matching, trust, and long-term engagement.

LGBTQ Dating Apps vs Traditional Dating Apps

Although both types of apps may share similar mechanics, their design priorities differ significantly. LGBTQ dating apps typically support multiple gender identities and pronouns by default, use matching logic designed for queer and non-binary users, and place stronger emphasis on privacy and moderation. Traditional dating apps often assume heterosexual pairing and rely on broader safety tools that may not address LGBTQ-specific risks.

Because of this, LGBTQ users tend to rely more heavily on specialized platforms to meet partners and build connections.

Core Features of an Inclusive LGBTQ Dating App

An inclusive LGBTQ dating app is built on a combination of thoughtful identity design, strong safety measures, and reliable communication features.

1. Inclusive profiles and matching

Users should be able to choose from a wide range of gender identities and sexual orientations without being forced into rigid categories. These options should remain flexible and easy to update over time.

2.Safety, privacy, and moderation

Strong blocking and reporting tools, profile verification, location controls, and clear moderation policies help build trust and reduce harmful interactions. Safety features should be designed with LGBTQ-specific risks in mind.

3.Community and social features

Many successful LGBTQ apps include group chats, interest channels, or local event listings. These features support social connection beyond dating and help users feel less isolated.

4. Secure communication

Messaging should be encrypted and include options such as image blurring, spam protection, and simple account deletion. If voice or video calls are offered, stability and privacy controls become even more important.

How to Build an LGBTQ Dating App

Building an LGBTQ dating app requires more than basic matching logic. It involves thoughtful design, inclusive identity support, and strong safety practices to create a space where users feel respected and in control.

Step 1: Define your niche and values

Start by deciding who the app serves. Pick a clear focus, such as LGBTQ users in a specific region, a trans-first space, or a relationship-style community.
Then write your core principles. Safety, consent, and inclusion should be non-negotiable. These principles should guide product decisions, feature design, and moderation rules.

Step 2: Design identity-first profiles

Build profiles that let users describe themselves accurately. Support gender identity, pronouns, sexual orientation, and relationship intent.
Avoid assumptions in labels and defaults. Keep sensitive fields optional. Give users the option to skip, edit, or update identity details at any time.

Step 3: Map the core user flows

Keep key flows respectful and simple. Plan the full journey: onboarding, discovery, matching, messaging, and reporting.
Users should always feel in control. Make privacy settings easy to find. Add clear actions like block, report, and hide profile. Reduce friction in each step.

Step 4: Build secure real-time communication

Messaging is the core of most dating apps. Start with a secure 1-on-1 chat. Add spam protection, rate limits, and optional media controls such as image blur.
If you support voice or video calls, treat them as “safety-first” features. Focus on low-latency, reliable connections, and call controls. Users should be able to end calls fast, block instantly, and report abuse during or after a session.

Step 5: Add moderation and compliance from day one

Do not treat moderation as an add-on. Define community rules early and enforce them consistently.
Follow data privacy requirements and age restrictions. Explain content rules clearly. Provide a simple appeal process. Publish transparency practices when possible. Long-term trust comes from consistent enforcement.

Choosing the Right Real-Time Technology for an LGBTQ Dating App

When building an LGBTQ dating app, real-time communication is a core part of the product experience. Features like text chat, voice calls, video calls, and live streaming shape how users connect, build trust, and feel safe on the platform.

Instead of developing real-time infrastructure from scratch, many teams rely on third-party communication SDKs to reduce complexity and technical risk. Solutions like ZEGOCLOUD support in-app chat, voice and video calls, and live streaming, while handling challenges such as scalability, low latency, and network stability across regions.

Using a mature real-time SDK helps teams avoid maintaining signaling systems, media transmission, and performance tuning on their own. This allows developers to focus more on product-specific concerns such as safety controls, moderation workflows, privacy settings, and inclusive user experience design, all of which are especially important when building dating apps for LGBTQ communities.

Final Thoughts

LGBTQ dating apps succeed when they prioritize inclusivity, safety, and trust over short-term growth metrics. For developers, this means designing with empathy, listening to user feedback, and treating identity as a core system feature rather than an optional setting.

When built responsibly, these platforms create safer digital spaces and more meaningful connections.

Integrating Screen Sharing into a React Native Video Call App

Stephen568hub — Thu, 08 Jan 2026 03:26:29 +0000

Screen sharing is a feature users expect in modern video call apps. Whether for remote work, online classes, or live demos, the experience should feel seamless and should not interrupt the ongoing call.

In this post, I’ll explain how to integrate screen sharing into a React Native video call app, focusing on the architecture and design decisions rather than low-level implementation. The full code and setup details are available in the GitHub repository and the original blog post.

How Does Screen Sharing Work in a React Native Video Call App?

At a high level, screen sharing works by capturing the device screen and sending it as a live video stream to other participants in the call.

Instead of replacing the camera feed, screen sharing is treated as an additional video source. This allows the app to maintain the normal video call experience while temporarily adding screen content as another stream.

How Can Camera Video and Screen Sharing Run at the Same Time?

The key is to separate responsibilities:

The camera stream continues to run normally
The screen stream is published independently

By keeping these two streams separate, users can start or stop screen sharing without affecting their camera or audio. This avoids common issues such as camera freezes or call reconnects.

Why Use a Separate Channel for Screen Sharing?

Using a dedicated auxiliary channel for screen sharing makes the system more reliable and easier to manage.

This approach allows:

Independent control of camera and screen streams
Cleaner stream lifecycle management
Better scalability for group calls
A user experience similar to professional meeting apps

From both a technical and product perspective, this separation reduces complexity.

What Are the Android Requirements for Screen Sharing in React Native?

On Android, screen sharing relies on system-level screen capture permissions.

Key considerations include:

Handling runtime permission requests
Responding to the system screen capture consent dialog
Managing foreground services on newer Android versions

Once permission is granted, the screen content can be captured and published as a real-time video stream.

What Are the iOS Requirements for Screen Sharing in React Native?

iOS places stricter limits on screen capture.

Cross-app screen sharing requires a Broadcast Upload Extension built on ReplayKit. This extension runs in a separate process and sends screen data back to the main app.

Important points to consider:
The extension has a strict memory limit
Screen capture is handled by the system, not the app
Setup is more complex than Android, but very stable once configured

How Does the Screen Sharing Flow Work from Start to End?

The typical flow looks like this:

A user joins a video call room
The camera stream starts on the main channel
Screen sharing is initiated on the auxiliary channel
Other users detect and subscribe to the screen stream
Screen sharing stops without interrupting the call

This flow keeps both the media pipeline and the user experience predictable.

Where Can I Find the Full Implementation?

To keep this post focused on concepts and architecture, detailed code examples are not included here.

Documentation: Share the screen in React Native

Step-by-step tutorial: How to Integrate Screen Sharing in React Native Video Chat

These resources cover SDK setup, permissions, and platform-specific configuration in detail.

When Should You Add Screen Sharing to a Video Call App?

Screen sharing is especially useful for:

Remote collaboration tools
Online education platforms
Live product demos
Technical support or walkthroughs

If your app already supports video calls, adding screen sharing as a parallel stream is often the most flexible approach.

Final Thoughts

To successfully integrate screen sharing into a React Native video call, it’s best to treat screen capture as a parallel media stream rather than a replacement for the camera.

By keeping video calls on the main channel and screen sharing on an auxiliary channel, you can deliver a smooth and professional user experience while keeping the system architecture clean and maintainable.

Building a Real-Time AI Voice Assistant with WebRTC

Stephen568hub — Wed, 07 Jan 2026 08:38:27 +0000

Voice-first interfaces are becoming a core interaction pattern for modern apps. Instead of typing, users can speak naturally and receive instant voice responses. This post walks through how to build an AI voice assistant that runs entirely inside a WebRTC room.

The assistant supports real-time speech recognition, AI-generated responses, and natural voice playback, while exposing clear interaction states like listening, thinking, and speaking.

How AI Voice Assistant Works

The architecture is based on a WebRTC room with an embedded AI agent:

The user joins a WebRTC room and streams microphone audio
Audio is processed in real time by an AI agent (ASR → LLM → TTS)
The agent responds with both live speech and text
The UI reflects conversation states for better feedback

All audio streaming is handled via WebRTC, so responses feel immediate and conversational.

Key Features of AI Voice Assistant

Real-time voice input using WebRTC
Live speech-to-text with partial and final transcripts
AI-generated responses with natural voice playback
Conversation state indicators (listening, thinking, speaking)
Optional text input alongside voice
Session-based conversations inside isolated rooms

Tech Stack

WebRTC for real-time audio streaming
ZEGOCLOUD Conversational AI for ASR, LLM, and TTS
React + TypeScript for the frontend
Node.js for session and agent management

All heavy audio and AI processing runs on the server side, keeping the client lightweight.

Why Use WebRTC for Voice AI?

WebRTC provides low-latency, bi-directional audio streaming, which is critical for voice assistants. Running the AI agent inside the same real-time room avoids delays caused by traditional request-response audio pipelines.

This pattern works well for:

Voice assistants
AI companions
Accessibility tools
Hands-free or screenless interfaces

Source Code

GitHub repository with full source code and setup instructions:
👉 AI voice assistant source code on GitHub

Tutorial

Full step-by-step guide covering architecture, backend setup, WebRTC integration, and UI state handling:
👉 How to Create an AI Voice Assistant

If you're exploring how to combine WebRTC with conversational AI for real-time voice experiences, this project provides a practical reference implementation.

Common Mistakes When Building an AI Assistant

Stephen568hub — Fri, 26 Dec 2025 08:44:31 +0000

Building an AI assistant looks easy at first. Connect a language model, add a chat UI, and you are done.
In reality, many assistants fail once users start talking to them in real time.

After building a real-time AI assistant with voice and text, I ran into several problems that are easy to overlook. This article shares the most common mistakes and how to avoid them.

Mistake 1: Treating Voice as Just “Audio Input”

Many AI assistants still handle voice as recorded audio files instead of live streams. This forces the system to wait until the user finishes speaking before processing anything, which introduces unnecessary delay and breaks conversational flow. Voice interactions should feel continuous, but batch processing makes them feel slow and transactional.

How to avoid it

Treat voice as a continuous audio stream, not recorded clips
Start speech recognition while the user is still speaking
Stream responses back as soon as they are available

Mistake 2: Ignoring Interruptions

Some assistants continue speaking even when users start talking, leading to overlapping audio or frozen states. This behavior feels unnatural and immediately reminds users that they are talking to a machine. Real conversations rely on smooth turn-taking and quick interruption handling.

How to avoid it

Monitor incoming audio during AI speech playback
Stop or pause speech output immediately when user speech is detected
Resume listening without clearing conversation context

Mistake 3: Mixing Business Logic with Media Logic

Combining session management, AI reasoning, audio streaming, and UI logic in a single backend service often leads to complex and fragile systems. As features grow, these systems become difficult to debug and scale.

How to avoid it

Keep the backend focused on authentication and session control
Delegate audio streaming and message delivery to a real-time layer
Isolate AI logic from transport and media handling

Mistake 4: Overloading the Assistant with Memory

Feeding long conversation histories into the model increases latency and cost, and can even reduce response quality. More memory does not always mean better understanding, especially in real-time scenarios.

How to avoid it

Use a sliding window of recent messages
Summarize older context instead of passing full histories
Keep memory lightweight and task-focused

Mistake 5: No Visual Feedback for Users

When users interact with a voice assistant, they need reassurance that the system is working. Without feedback, users are left guessing whether the assistant is listening, thinking, or responding, which quickly causes frustration.

How to avoid it

Show clear states like listening, thinking, and speaking
Display live transcription when possible
Keep controls simple and responsive
Visual cues build trust and reduce frustration

Mistake 6: Focusing Only on the AI Model

Teams often prioritize prompt tuning and model selection while overlooking real-time performance. However, users usually care more about speed, responsiveness, and smooth turn-taking than slightly smarter answers.

How to avoid it

Measure speech-to-speech latency, not just text response time
Test interactions in noisy or unstable environments
Treat real-time performance as a core feature, not an optimization