How to Build Low-latency AI Video Avatars for Real-time Interactions

AI avatars are changing the face of content creation and how appealing videos can be created for various purposes, whether it’s training video content, internal team communication, or customer support. Advanced AI tools and video generators now make it possible to create studio-quality, customized videos featuring avatars that can speak multiple languages. An AI video generator enables you to create videos with human-like digital avatars that can connect with a global audience.

In this guide, you'll explore how to build a low-latency AI video avatar using Simli's API and WebRTC. By the end, you’ll be able to create and upload AI-generated videos that sync perfectly with real-time speech, ideal for training, interactive simulations, and presentations. You’ll also be able to generate stunning AI avatars and animated videos in minutes, reaching audiences at scale with human-like, multilingual content.

Prerequisites

To follow through this tutorial, you'll need:

• A basic understanding of Javascript and Next.js
• A Simli account. To get started, create a free Simli account.
• OpenAI account (paid version).

The complete source code is available on GitHub.

With these essentials in place, let's now explore the core elements driving low-latency AI avatars and why low latency is required to provide a fluid, realistic user experience.

Understanding the Key Components for Low-Latency AI Avatars

To demonstrate the importance of low latency, imagine a virtual assistant that takes several seconds to respond to a user’s question. Frustrating, right? Another illustration is an AI instructor in a training video whose voice and lip movements lag behind the audio in AI videos. These irregularities affect the effectiveness of AI interactions, making low latency an absolute necessity.

Latency determines the amount of interaction and realistic experience with AI-powered avatars. In scenarios like training sessions, live presentations, or customer support, an avatar needs to respond as naturally as a human would. A latency of more than 200 milliseconds makes the responses appear delayed and unnatural for any human interaction. Low latency, therefore, gives way to flawless and real-time engagement, thereby leading to a lot more satisfaction and a much better user experience.

Let's look closer at the core technologies and tools behind low-latency AI video avatars and see how each adds to creating an ideal, real-time interaction.

Core Technologies for Building Low-Latency AI Video Avatars

To accomplish low-latency setup, you’ll use the following technologies and format:

• Simli's API: Simli's API allows you to create lip-synced AI avatars that convert audio into video in real time. This API provides the foundation for transforming audio into synchronized visual avatars quickly.

• WebRTC: This handles the media stream capture and adds to the peer connection.

• PCM and Audio Downsampling: To minimize latency in audio transmission, pulse code modulation (PCM) is used for audio data, which WebRTC can handle efficiently. PCM digitizes audio signals by sampling analog waveforms and converting them into uncompressed binary data. Downsampling to 16kHz PCM ensures seamless compatibility with WebRTC and the Simli client, supporting real-time, synchronized audio playback.

• OpenAI API: Powers real-time language understanding and response generation, enabling the avatar to engage in dynamic, intelligent interactions.

Having covered the technologies and tools, let's dive into how you can set up your API environment. For the following steps, you will be configuring access to both Simli and OpenAI so you can have smooth integration and interaction within your application.

Set Up Your API Environment

You'll need API keys for both Simli and OpenAI to set up the API environment. To get a Simli API key, go to Simli's official website, click "Get API Access" and create an account.

After logging in, you should see your developer dashboard with your API keys as shown below.

Copy and keep the API key somewhere safe because you will need it to authenticate requests within your application.

With your API key ready, let’s move on to customizing the appearance of your AI avatar. Navigate to the ‘Available Faces’ section in Simli’s API documentation to find listed Avatar face IDs as seen below:

You can also create your own avatar by clicking on the “Create Avatar” Icon on your dashboard. Once you follow the steps highlighted on the page, you’ll be able to create your own Avatar.

To get the ID for each face, copy the random text after the name. For example, the ID for Jenna will be tmp9i8bbq7c. If you created your own Avatar, an ID would also be given alongside. Now that you have the face ID and the Simli API key, let’s create a Next.js app.

Create the Project

Create a Next.js app by running this command in your terminal:

npx create-next-app@latest simli-demo

Answer the following prompts:

What is your project named? my-app
Would you like to use TypeScript? No / Yes
Would you like to use ESLint? No / Yes
Would you like to use Tailwind CSS? No / Yes
Would you like your code inside a `src/` directory? No / Yes
Would you like to use App Router? (recommended) No / Yes
Would you like to use Turbopack for `next dev`?  No / Yes
Would you like to customize the import alias (`@/*` by default)? No / Yes
What import alias would you like configured? @/*

Typescript and Tailwind are essential for this tutorial, as they simplify styling and provide type safety, making it easier to create a responsive and maintainable avatar interface. Once that is done, install the dependencies or packages for this project. They include:

• simli-client: This will be used for handling facial animations or avatar interactions.
• openai: This is the main OpenAI SDK and will be used for accessing the AI service.
• openai/realtime-api-beta:This is OpenAI's real-time language API that will be used for voice conversations.
• Ws: WebSocket client and server implementation for real-time communication.

Open up your terminal once more and run the command below, it will install all the dependencies needed for the project.

npm install simli-client openai github:openai/openai-realtime-api-beta ws

Add Environment Variables
Create a .env file and copy your Simli API key and OpenAI key. Paste it in as shown below:

NEXT_PUBLIC_SIMLI_API_KEY="SIMLI-API-KEY"
NEXT_PUBLIC_OPENAI_API_KEY="OPENAI-API-KEY"

Create the VideoBox Components

You'll need to create a component to handle the video feed display and the audio playing. Inside the app folder, create a folder called components, and inside it, create a file called VideoBox.tsx and add the following code:

// app/components/VideoBox.tsx

    // Render a styled video and audio container with auto-play functionality.

    export default function VideoBox(props: any) {
       return (
           <div className="aspect-video flex rounded-sm overflow-hidden items-center h-[350px] w-[350px] justify-center bg-simligray">
               <video ref={props.video} autoPlay playsInline></video>
               <audio ref={props.audio} autoPlay ></audio>
           </div>
       );
    }

The code defines a VideoBox component that displays a video and audio player within a styled container. This component expects two props: video and audio, which are references passed to the <video> and <audio> elements. Both elements have autoPlay enabled, starting playback immediately, and playsInline is set on the video to prevent it from going fullscreen on mobile devices. These ref attributes (props.video and props.audio) allow the primary component to control playback, pause, and other media interactions.

Build the SimliOpenAIPushToTalk Component

Inside the app folder, create a file called SimliOpenAIPushToTalk.tsx. This file will contain much of the core functionality. Start by adding the following code:

// app/SimliOpenAIPushToTalk.tsx

    // import the necessary packages
    ...

    import React, { useCallback, useEffect, useRef, useState } from "react";
    import { RealtimeClient } from "@openai/realtime-api-beta";
    import { SimliClient } from "simli-client";
    import VideoBox from "./Components/VideoBox";
    import IconExit from "@/media/IconExit";
    interface SimliOpenAIPushToTalkProps {
      simli_faceid: string;
      openai_voice: "echo" | "alloy" | "shimmer";
      initialPrompt: string;
      onStart: () => void;
      onClose: () => void;
    }
    const simliClient = new SimliClient();
    const SimliOpenAIPushToTalk: React.FC<SimliOpenAIPushToTalkProps> = ({
      simli_faceid,
      openai_voice,
      initialPrompt,
      onStart,
      onClose,
    }) => {
      const [isLoading, setIsLoading] = useState(false);
      const [isAvatarVisible, setIsAvatarVisible] = useState(false);
      const [error, setError] = useState("");
      const [isRecording, setIsRecording] = useState(false);
      const [userMessage, setUserMessage] = useState("...");
      const [isButtonDisabled, setIsButtonDisabled] = useState(false);
      const videoRef = useRef<HTMLVideoElement>(null);
      const audioRef = useRef<HTMLAudioElement>(null);
      const openAIClientRef = useRef<RealtimeClient | null>(null);
      const audioContextRef = useRef<AudioContext | null>(null);
      const streamRef = useRef<MediaStream | null>(null);
      const processorRef = useRef<ScriptProcessorNode | null>(null);
      const audioChunkQueueRef = useRef<Int16Array[]>([]);
      const isProcessingChunkRef = useRef(false);
    ...

In the above codes, the necessary packages and hooks are imported, a prop interface SimliOpenAIPushToTalkProps is defined for component configuration, and an instance of SimliClient is created to manage multimedia interactions. The SimliOpenAIPushToTalk component requires a specific set of props, which are defined in the SimliOpenAIPushToTalkProps interface.

References are also set up for video and audio elements, the OpenAI client, audio context, media stream, and audio processor, allowing direct control over multimedia resources within the component.

Next, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // initializing the SimliClient
    ...

    const initializeSimliClient = useCallback(() => {
        if (videoRef.current && audioRef.current) {
          const SimliConfig = {
            apiKey: process.env.NEXT_PUBLIC_SIMLI_API_KEY,
            faceID: simli_faceid,
            handleSilence: true,
            videoRef: videoRef,
            audioRef: audioRef,
          };
          simliClient.Initialize(SimliConfig as any);
          console.log("Simli Client initialized");
        }
      }, [simli_faceid]);
      const initializeOpenAIClient = useCallback(async () => {
        try {
          console.log("Initializing OpenAI client...");
          openAIClientRef.current = new RealtimeClient({
            apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY,
            dangerouslyAllowAPIKeyInBrowser: true,
          });
          await openAIClientRef.current.updateSession({
            instructions: initialPrompt,
            voice: openai_voice,
            turn_detection: { type: "server_vad" },
            input_audio_transcription: { model: "whisper-1" },
          });
          openAIClientRef.current.on(
            "conversation.updated",
            handleConversationUpdate
          );
          openAIClientRef.current.on(
            "input_audio_buffer.speech_stopped",
            handleSpeechStopped
          );
          await openAIClientRef.current.connect();
          console.log("OpenAI Client connected successfully");
          setIsAvatarVisible(true);
        } catch (error: any) {
          console.error("Error initializing OpenAI client:", error);
          setError(`Failed to initialize OpenAI client: ${error.message}`);
        }
      }, [initialPrompt]);
      const handleConversationUpdate = useCallback((event: any) => {
        console.log("Conversation updated:", event);
        const { item, delta } = event;
        if (item.type === "message" && item.role === "assistant") {
          console.log("Assistant message detected");
          if (delta && delta.audio) {
            const downsampledAudio = downsampleAudio(delta.audio, 24000, 16000);
            audioChunkQueueRef.current.push(downsampledAudio);
            if (!isProcessingChunkRef.current) {
              processNextAudioChunk();
            }
          }
        } else if (item.type === "message" && item.role === "user") {
          setUserMessage(item.content[0].transcript);
        }
      }, []);
    ...

The code above does the following:

• Initializes the SimliClient with configuration parameters, including simli_faceid for Simli's face recognition and video references, and manages audio-visual synchronization with the AI avatar using the API key and face ID. With this, you'll have a real-time lip-sync between the avatar's visuals and audio input.
• Sets up and connects the OpenAI client—registers event listeners for updates in conversation and speech detection. The initializeOpenAIClient function sets up the connection to the OpenAI client, using the provided API key and settings, and registers event listeners to handle conversation updates and detect when speech stops.
• Handles real-time updates from the conversation, processing both assistant messages and user inputs. For audio responses from the assistant, the code downsamples and adds chunks to a queue.

Directly after handleConversationUpdate, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // process audio chunks and send to Simli
    ...

    const processNextAudioChunk = useCallback(() => {
        if (
          audioChunkQueueRef.current.length > 0 &&
          !isProcessingChunkRef.current
        ) {
          isProcessingChunkRef.current = true;
          const audioChunk = audioChunkQueueRef.current.shift();
          if (audioChunk) {
            const chunkDurationMs = (audioChunk.length / 16000) * 1000; 

            simliClient?.sendAudioData(audioChunk as any);
            console.log(
              "Sent audio chunk to Simli:",
              chunkDurationMs,
              "Duration:",
              chunkDurationMs.toFixed(2),
              "ms"
            );
            isProcessingChunkRef.current = false;
            processNextAudioChunk();
          }
        }
      }, []);

      const handleSpeechStopped = useCallback((event: any) => {
        console.log("Speech stopped event received", event);
      }, []);

      const downsampleAudio = (
        audioData: Int16Array,
        inputSampleRate: number,
        outputSampleRate: number
      ): Int16Array => {
        if (inputSampleRate === outputSampleRate) {
          return audioData;
        }
        const ratio = inputSampleRate / outputSampleRate;
        const newLength = Math.round(audioData.length / ratio);
        const result = new Int16Array(newLength);
        for (let i = 0; i < newLength; i++) {
          const index = Math.round(i * ratio);
          result[i] = audioData[index];
        }
        return result;
      };
    ...

Here:

• The processNextAudioChunk manages the processing of audio chunks from the assistant, sending them immediately to Simli.
• The downsampleAudio converts audio data to a 16kHz PCM format. PCM captures audio in an uncompressed, raw format, ideal for maintaining low latency. The PCM audio is then sent to SimliClient without encoding delays.

After the downsampleAudio function, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...

    // Manages audio recording setup and cleanup for real-time processing.
    ...

    const startRecording = useCallback(async () => {
        if (!audioContextRef.current) {
          audioContextRef.current = new AudioContext({ sampleRate: 24000 });
        }
        try {
          console.log("Starting audio recording...");
          streamRef.current = await navigator.mediaDevices.getUserMedia({
            audio: true,
          });
          const source = audioContextRef.current.createMediaStreamSource(
            streamRef.current
          );
          processorRef.current = audioContextRef.current.createScriptProcessor(
            2048,
            1,
            1
          );
          processorRef.current.onaudioprocess = (e) => {
            const inputData = e.inputBuffer.getChannelData(0);
            const audioData = new Int16Array(inputData.length);
            for (let i = 0; i < inputData.length; i++) {
              audioData[i] = Math.max(
                -32768,
                Math.min(32767, Math.floor(inputData[i] * 32768))
              );
            }
            openAIClientRef.current?.appendInputAudio(audioData);
          };
          source.connect(processorRef.current);
          processorRef.current.connect(audioContextRef.current.destination);
          setIsRecording(true);
          console.log("Audio recording started");
        } catch (err) {
          console.error("Error accessing microphone:", err);
          setError("Error accessing microphone. Please check your permissions.");
        }
      }, []);

      const stopRecording = useCallback(() => {
        if (processorRef.current) {
          processorRef.current.disconnect();
          processorRef.current = null;
        }
        if (streamRef.current) {
          streamRef.current.getTracks().forEach((track) => track.stop());
          streamRef.current = null;
        }
        setIsRecording(false);
        console.log("Audio recording stopped");
      }, []);
    ...

The startRecording function accesses the microphone in real time using getUserMedia, captures audio with AudioContext, processes it as 16-bit PCM, and sends it to the OpenAI client. The stopRecording function stops the stream and releases resources. This setup uses AudioContext for real-time audio, while WebRTC configuration is managed by the Simli client.

Next to add is the stopRecording function, paste this code below:

// app/SimliOpenAIPushToTalk.tsx
...
// Control interaction, push-to-talk, and audio visualization.

...

const handleStart = useCallback(async () => {
    setIsLoading(true);
    setError("");
    onStart();
    try {
      await simliClient?.start();
      await initializeOpenAIClient();
    } catch (error: any) {
      console.error("Error starting interaction:", error);
      setError(`Error starting interaction: ${error.message}`);
    } finally {
      setIsAvatarVisible(true);
      setIsLoading(false);
    }
  }, [initializeOpenAIClient, onStart]);
  const handleStop = useCallback(() => {
    console.log("Stopping interaction...");
    setIsLoading(false);
    setError("");
    setIsAvatarVisible(false);
    simliClient?.close();
    openAIClientRef.current?.disconnect();
    if (audioContextRef.current) {
      audioContextRef.current.close();
    }
    stopRecording();
    onClose();
    console.log("Interaction stopped");

    window.location.reload();
  }, [stopRecording]);

  const handlePushToTalkStart = useCallback(() => {
    if (!isButtonDisabled) {
      startRecording();

      simliClient?.ClearBuffer();
      openAIClientRef.current?.cancelResponse("");
    }
  }, [startRecording, isButtonDisabled]);
  const handlePushToTalkEnd = useCallback(() => {
    setTimeout(() => {
      stopRecording();
    }, 500);
  }, [stopRecording]);

  const AudioVisualizer = () => {
    const [volume, setVolume] = useState(0);
    useEffect(() => {
      const interval = setInterval(() => {
        setVolume(Math.random() * 100);
      }, 100);
      return () => clearInterval(interval);
    }, []);
    return (
      <div className="flex items-end justify-center space-x-1 h-5">
        {[...Array(5)].map((_, i) => (
          <div
            key={i}
            className="w-2 bg-black transition-all duration-300 ease-in-out"
            style={{
              height: `${Math.min(100, volume + Math.random() * 20)}%`,
            }}
          />
        ))}
      </div>
    );
  };

The code above does the following:

• Initializes the Simli and OpenAI clients through the handleStart function, and resets the error state. It also sets isLoading to true while the clients start.
• Stops recording, disconnects clients through the handleStop function and resets the component state.
• Manages push-to-talk functionality through the Push-to-Talk functions starting and stopping recording when the user interacts with the button.
• Displays an animated audio visualizer that gives feedback on microphone volume through the audio visualizer.

Following the audioVisualizer function, add the code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // Initialize Simli client and render video and push-to-talk UI.

    useEffect(() => {
        initializeSimliClient();
        if (simliClient) {
          simliClient?.on("connected", () => {
            console.log("SimliClient connected");
            const audioData = new Uint8Array(6000).fill(0);
            simliClient?.sendAudioData(audioData);
            console.log("Sent initial audio data");
          });
        }
        return () => {
          try {
            simliClient?.close();
            openAIClientRef.current?.disconnect();
            if (audioContextRef.current) {
              audioContextRef.current.close();
            }
          } catch {}
        };
      }, [initializeSimliClient]);
      return (
        <>
          <div className="transition-all duration-300 ">
            <VideoBox video={videoRef} audio={audioRef} />
          </div>
          <div className="flex flex-col items-center">
            {!isAvatarVisible ? (
              <button
                onClick={handleStart}
                disabled={isLoading}
                className="w-full h-[52px] mt-4 disabled:bg-gray-600 disabled:text-white disabled:hover:rounded-full bg-green-500 text-white py-3 px-6 rounded-full transition-all duration-300 hover:text-black hover:bg-white hover:rounded flex justify-center items-center"
              >
                {isLoading ? (
                  <span>Loading...</span>
                ) : (
                  <span className="font-abc-repro-mono font-bold w-[164px]">
                    Test Interaction
                  </span>
                )}
              </button>
            ) : (
              <>
                <div className="flex items-center gap-4 w-full">
                  <button
                    onMouseDown={handlePushToTalkStart}
                    onTouchStart={handlePushToTalkStart}
                    onMouseUp={handlePushToTalkEnd}
                    onTouchEnd={handlePushToTalkEnd}
                    onMouseLeave={handlePushToTalkEnd}
                    disabled={isButtonDisabled}
                    className={`mt-4 text-white flex-grow bg-green-500 hover:rounded hover:bg-opacity-70 h-[52px] px-6 rounded-full transition-all duration-300 ${
                      isRecording ? "bg-gray-900 rounded hover:bg-opacity-100" : ""
                    }`}
                  >
                    <span className="font-abc-repro-mono font-bold w-[164px]">
                      {isRecording ? "Release to Stop" : "Push & hold to talk"}
                    </span>
                  </button>
                  <button
                    onClick={handleStop}
                    className=" group w-[52px] h-[52px] flex items-center mt-4 bg-red text-white justify-center rounded-[100px] backdrop-blur transition-all duration-300 hover:bg-white hover:text-black hover:rounded-sm"
                  >
                    <IconExit className="group-hover:invert-0 group-hover:brightness-0 transition-all duration-300" />
                  </button>
                </div>
              </>
            )}
          </div>
        </>
      );
    };
    export default SimliOpenAIPushToTalk;

In the code above:

• The useEffect hook runs when the page mounts to initialize simliClient and sets up cleanup operations to close simliClient and audioContextRef when the component unmounts.
• Render the UI to:
  • Display the VideoBox component, which receives video and audio props for controlling media playback, along with controls to start interaction or initiate push-to-talk.
  • A button triggers handleStart or handleStop, while the push-to-talk button manages recording on mouse down and up.

Now that the core components and UI elements are set up, let’s move on to integrating these into the main demo component. Head over to page.tsx and replace its contents with the following code to complete the setup.

Set Up Demo Component
Navigate to page.tsx and replace it with the following code:

    // app/page.tsx

    // render the AI avatar with push-to-talk functionality
    "use client";
    import React, { useEffect, useState } from "react";
    import SimliOpenAIPushToTalk from "./SimliOpenAIPushToTalk";
    interface avatarSettings {
      name: string;
      openai_voice: "echo" | "alloy" | "shimmer";
      simli_faceid: string;
      initialPrompt: string;
    }
    const avatar: avatarSettings = {
      name: "Frank",
      openai_voice: "echo",
      simli_faceid: "5514e24d-6086-46a3-ace4-6a7264e5cb7c",
      initialPrompt:
        "You are a helpful AI assistant named Frank. You are friendly and concise in your responses. Your task is to help users with any questions they might have. Your answers are short and to the point, don't give long answers be brief and straightforward.",
    };
    const Demo: React.FC = () => {
      const [interactionMode, setInteractionMode] = useState<
        "pushToTalk" | undefined
      >(undefined);
      useEffect(() => {
        const storedInteractionMode = localStorage.getItem("interactionMode");
        if (storedInteractionMode === "pushToTalk") {
          setInteractionMode("pushToTalk");
        }
      }, []);
      const onStart = () => {
        console.log("Setting setshowDottedface to false...");
      };
      const onClose = () => {
        console.log("Setting setshowDottedface to true...");
      };
      return (
        <div className="bg-black min-h-screen flex flex-col items-center font-abc-repro font-normal text-sm text-white p-8">
          <div className="flex flex-col items-center gap-6 bg-effect15White p-6 pb-[40px] rounded-xl w-full">
            <div>
              <SimliOpenAIPushToTalk
                openai_voice={avatar.openai_voice}
                simli_faceid={avatar.simli_faceid}
                initialPrompt={avatar.initialPrompt}
                onStart={onStart}
                onClose={onClose}
              />
            </div>
          </div>
        </div>
      );
    };
    export default Demo;

In the above code, you created a component that renders the AI avatar with push-to-talk functionality. It also initializes settings for the avatar. You can also customize your avatar here by changing it to any face you want for interaction. Simli supports a range of custom avatar faces that you can apply to various use cases.

Set Up Layout
The last thing you’ll need to do is edit the layout, in the layout.tsx, paste this code below:

 // app/layout.tsx

    // edit the layout
    import type { Metadata } from "next";
    import { Inter } from "next/font/google";
    import { abcRepro, abcReproMono } from './fonts/fonts';
    import "./globals.css";

    const inter = Inter({ subsets: ["latin"] });
    export const metadata: Metadata = {
     title: "Simli App",
     description: "create-simli-app (OpenAI)",
    };
    export default function RootLayout({
     children,
    }: Readonly<{
     children: React.ReactNode;
    }>) {
     return (
       <html lang="en" className={`${abcReproMono.variable} ${abcRepro.variable}`}>
         <body className={inter.className}>{children}</body>
       </html>
     );
    }

Test the application by running the following command on your terminal:

npm run dev

Check the video demonstration of the project to see it in action. You can also explore the Simli repository to build avatars and work on more projects using Simli API.

Conclusion

In this project, you have implemented a low-latency AI video avatar for realistic and dynamic interactions. Simli's API with WebRTC allows developers to easily deploy and manage latency-free, perfectly synchronized audio-visual avatars that are very well-suited for a wide range of applications, such as training simulations, interactive presentations, customer support, and real-time virtual assistants.

This setup opens the door to engaging, human-like interactions across various settings.

Explore the documentation to learn more about the features and tools Simli offers.