DEV Community: Tiioluwani

Navigating Search Solutions: A Comprehensive Comparison Guide to Meilisearch, Algolia, and ElasticSearch

Tiioluwani — Sat, 21 Dec 2024 12:37:49 +0000

Introduction

As more information is available at our fingertips, one faces the challenge of filtering through large datasets to obtain relevant information within the shortest time possible. This necessity has resulted in the widespread use of search solutions. They are essential for up-to-date applications because they allow you to find the required information in a large dataset.

Well-developed and well-functioning search solution features help improve user satisfaction, increase activity, and ultimately increase business and organizational efficiency by finding the ‘hidden’ correlation in large datasets.

This article will compare Meilisearch, Algolia, and ElasticSearch for developers, tech enthusiasts, and companies seeking to explore various search solutions. We'll discuss each of them indicating its distinctive features, benefits, and applications, which would be helpful for decision-making when choosing the most suitable search solution for certain project needs and limitations of the software environment

Key Considerations When Choosing a Search Solution

When choosing the type of search solution for an application there are several factors that one must put in mind. Some of these factors include:

Scalability and Performance: An ideal search solution should meet the requirement of serving a large volume of data and large numbers of traffic. It should have quick query response time and should be scalable, that is, it should be able to respond to queries out of large traffic and should be able to handle search queries with various structural queries.
Ease of Integration: It should be easy to fit into your application environment and offer a basic and user-friendly API that makes it easy to add the search solution to any application. A suitable solution should also have good documentation and support materials to assist these developers in getting started.
Indexing and Querying: The search solution has to contribute to the development of effective indexing and query solutions and needs to be capable of handling various types of data that can be textual, numerical, or geographical data types. The solution should also contain extensions to basic search and sort functions with features such as fuzzy search, geo-search and filters, etc.
Relevance and Customization: There should be a relevance algorithm delivered with a search solution that can be adapted to your precise requirements. The solution should also contain diverse options that define the look and functionality of the search results interface to offer you a well-incorporated and easy-to-use environment.
Cost and Licensing: The search solution must be affordable and it must incorporate a licensing model that is going to suit your needs. Hardware, software, and maintenance costs must also be considered, and the licensing model should be manageable as needs change.

Unveiling the Contenders

Here we take a look at the peculiar features and capabilities of these search solutions for your application. In this section, we will delve into the details of three prominent search solutions: Meilisearch, Algolia, and ElasticSearch.

Meilisearch

Image Source

Meilisearch is a lightweight, developer-first, API-first search engine that is designed to be easily integrated into an existing application. Its principal components are high speed, easy configuration, and good documentation.

Strengths
Below are some of the strengths of Milliesearch:

Fast Performance: Meilisearch is characterized by incredibly fast search speed, so it can be appropriate for applications with high response rates.
Simple Setup: Meilisearch does not require much configuration which implies that developers can easily implement it.
Excellent Documentation: Meilisearch has a comprehensive knowledge base which is relatively easy to navigate and which contains a wealth of information for developers.
Open-Source: Meilisearch is an open-source tool, which makes it free to use and can also be self-hosted, which gives users more control over the underlying architecture.

Limitations
Despite its many benefits, Meilisearch also has some limitations. Some limitations of Millesearch include:

Lacks Advanced Features: Despite the fact that Meilisearch has many features that it can implement, it may not be the best suited for data sets of large size. There could be other additional features useful in some cases of massive data that can be missing in Meilisearch.
Limited Scalability: Compared to ElasticSearch, Meilisearch might not be as highly scalable, and it might become an issue with an extremely large amount of users or data.
Limited Community Support: Meillisearch has a smaller community of developers when compared to ElasticSearch. This could be a disadvantage as there could be fewer resources available in the solution's community.
Limited Geo-Search Capabilities: While it is also possible to implement certain basic forms of geo-search capabilities (the ability of a search solution to perform a search based on the location of the user), it is not as specifically developed as Algolia’s geo-search capabilities.

Algolia

Image Source

Algolia is a blazing-fast search engine that has rich API support and geo-search features. Its fundamental capabilities include high performance, rich extensibility, and robust tools for developers.

Strengths
Below are some of the strengths of Algolia:

Superior Speed: It has a nearly instant search result which can be a plus when you are dealing with applications that are sensitive to delay.
Extensive Customizations: Algolia also has many features that allows for flexibility to configure the search based on specific requirements.
Powerful Developer Tools: Among the tools that Algolia provides to developers, there are analytics and monitoring tools that help developers improve the search process and solve the problems they may face.
Geo-Search Capabilities: Geo-search functionality is available in Algolia, and this can be very useful for developers as they can search for results depending on location.

Limitations
As valuable as it may be, Algolia has its weaknesses that potential users should take into account. Below are some limitations of Aloglia:

Primarily a Hosted Solution: Algolia is predominantly a hosted platform which can be a potential limitation where users need a higher level of control over its architecture.
Higher Costs: Since Algolia is primarily hosted, there are additional costs that clients are required to meet. This may be a crucial point for some developers and businesses.
Steeper Learning Curve: Compared to Meilisearch Algolia has a steeper learning curve that makes the application of the solution challenging for the developers.
Limited Open-Source Options: Due to it being a hosted service, there are limited open-source options. This may reduce the possibilities for customization compared to other open-source platforms similar to ElasticSearch.

ElasticSearch

Image Source

ElasticSearch is a popular scalable open-source search solution used today due to its ability to provide strong, powerful queries. Some of its key capabilities involve high scalability, a very large and, indeed, highly active base of users, and numerous features.

Strengths
Below are key strengths ElasticSearch possesses:

Flexibility: ElasticSearch is indeed very flexible in that, developers can customize the solution to fit their needs.
Vast Community Support: With ElasticSearch having a large developer community, there is a lot of support and information available for the solution.
Extensive Features: ElasticSearch has many features such as query time complexities, real-time indexing, and distributed indexing.
Highly Scalable: ElasticSearch is highly scalable and thus can be used in an application requiring thousands of users or massive data.

Limitations
Like any other solution, ElasticSearch has its drawbacks as well. Below are some limitations of ElasticSearch

Steeper Learning Curve: ElasticSearch has a complicated learning curve compared to Meilisearch and Algolia, which may discourage developers from using the solution.
Requires More Operational Overhead: ElasticSearch requires a higher level of configuration and management compared to setting up and running Meilisearch or Algolia, which could present challenges in maintaining the solution.
Limited Geo-Search Capabilities: While ElasticSearch offers basic geo-search capabilities, it may not be as powerful as the geo-search capabilities offered by Algolia.
Higher Resources Requirement: Compared to Meilisearch and Algolia, ElasticSearch requires more resources, which can become a problem when it comes to running the solution on small or old hardware.

Feature Comparison Table

Feature	Meilisearch	Algolia	ElasticSearch
Deployment	Self-Hosted (Binary)	Hosted (SaaS)	Self-hosted (Java)
Pricing	Free, Open-source	Paid	Free, Open-source
Scalability	Good	Excellent	Excellent
Performance	Fast	Very Fast	Fast
Ease of Use	Easy	Easy	Moderate
Features	Basic	Extensive	Extensive
Customization	Limited	Extensive	Extensive
Geo-Search	Yes	Yes	Yes
Full-Text Search	Yes	Yes	Yes
Real-Time Indexing	Yes	Yes	Yes
Community Support	Growing	Strong	Strong
Best Use Case	Small to medium-sized applications	Large-scale applications, high traffic	Large-scale applications, complex data
Data Ingestion	Simple	Simple	Complex
Security	Good	Strong	Strong

For a deeper view, let’s examine the key factors to consider when choosing a search solution: We’ll also explain how those factors connect to Meilisearch, Algolia, and ElasticSearch.

Scalability

Meilisearch: Meilisearch is built with horizontal scalability in mind, meaning that you can add more nodes to your cluster if necessary. This makes it suitable for use in applications that are expected to register high growth in the future.
Algolia: Horizontal scalability is another feature of the Algolia solution, which is optimal for large traffic loads. Features such as automatic sharding and replication ensure that the database offers its best performance and is always reliable.
ElasticSearch: ElasticSearch is highly scalable and thus can work with very large sets of data and big loads of work. It has distributional indexing and search capabilities, thus making it appropriate for use in large applications.

Performance

Meilisearch: As a fast indexing and query search system, Meilisearch is famous for its speed in handling searches. There is evidence that it fares well in benchmarks, particularly when working with smaller to midsize datasets. Its performance is attributed to its ability to use efficient data structures and indexing algorithms.
Algolia: Algolia is also praised for the high speed of search and the platform is built for large-scale applications with extensive traffic. It is designed for high-speed performance and has wide usage for applications where searches should take almost no time.
ElasticSearch: ElasticSearch has been made fairly efficient for most applications as well as offering multiple potent functionalities. However, it is not as fast as Algolia in every case, although it can perform outstandingly in certain situations if it is finely tuned to suit specific environments. It depends on different factors, including the characteristics of the data and the queries, as well as the available computing power.

Ease of Integration

Meilisearch: Meilisearch is easy to install and integrates well into applications due to the API and well-documented documentation. It provides client libraries in several programming languages, so novice programmers can also work with it.
Algolia: Algolia has a welcoming API that comes with relevant documentation for developers to work with. It supports many languages and it can also integrate with other software frameworks.
ElasticSearch: ElasticSearch can be harder to implement compared to other systems, particularly for those new to distributed systems. However, it has a strong API and a vast number of users who can turn to it for solutions in case of any issue.

Indexing

Meilisearch: It allows for efficient indexing and applies to different types of data. It also allows real-time indexing, which means it can be used where data is frequently changing.
Algolia: It allows for indexing and can also update the index in real-time. It can handle complex volumes of data and data structure that varies in type.
ElasticSearch: It is primarily acknowledged for its indexing capabilities and supports many data types in its framework. One benefit is that it indexes the content of files in real-time and there are some features included as well like dynamic mapping.

Querying

Meilisearch: It has basic search query syntax and multiple forms of filtering. They also include additional characteristics like the option of the fuzzy search (a sort of search that allows for searching with makes use of spelling differences) and geo-search.
Algolia: It offers great opportunities regarding filtering functionality, and the query language is rather rich. It also provides more advanced features for the search of information including the options of fuzzy search, geo search, and tolerance of typos.
ElasticSearch: It provides certain levels of query string elasticity and supports the most possible filtering. It offers advanced features like fuzzy search, grouping, geo search, and much more.

Relevance
Meilisearch: Among the features provided within Meilisearch is the relevance algorithm which you can fine-tune to obtain the desired outcome for your specific circumstances.
Algolia: Algolia offers a relevance function and some options like synonyms or stop words for the further improvement of searching.
ElasticSearch: ElasticSearch has a strong relevance algorithm and many customization options. It also supports query time boosting and relevance scoring.

Customization and User Experience
Meilisearch: Meilisearch is quite limited in how it allows users to customize the appearance of the search results interface and the interactions that occur within that space. However, it can be integrated with the user interface of your existing application.
Algolia: Algolia offers more flexibility regarding the templates and display of search results and their usage. It can also be implemented alongside your existing application user interface.
ElasticSearch: The users can also interact with ElasticSearch in multiple ways; there are many possibilities as to how the search results are presented to the users. It can also be associated with the user interface of the existing application that you have.

Choosing the Right Solution

It is advisable to consider certain factors that may be vital when choosing a search solution for your project. Depending on the data size, the budget, the technical skills needed, and preferences in terms of features, users can select different search solutions. Here are guidelines based on different project situations.

1. Data Size

Small Projects: Meilisearch is a perfect option for applications with less data as it provides clear and fast solutions. They are easy to implement and are not very demanding in terms of costs and resources and as such are recommendable for use in small businesses or during the initial stages of business.
Medium to Large Projects: If you are dealing with a medium to a large set of data, Algolia is faster and has more features to work with if the project has some level of complexity. As it provides strong API support and extensive choices of customization, it is good for e-commerce sites or sites with large content.
Enterprise-Level Applications: When it comes to big data processing with high availability and extensibility and when complex queries are likely to be needed, it is better to use ElasticSearch. It can handle large quantities of data and offers a rich set of analytical tools, which will be useful for large organizations requiring processing large amounts of information.

2. Budget
Cost-Conscious Solution: In cases where one needs to work within a certain budget, then this is where Meilisearch comes in handy. The platform is highly performant but lacks the requisites of self-hosting that other services require. This enables developers to incorporate Search as an additional feature with minimal extra expense.
Mid-Range Budget: Algolia can be considered a paid solution that is pricier, though it has more features compared to the free solution. It is ideal for businesses that are ready to spend resources, which will lead to a more advanced search option that improves customer experience.
Unlimited Budget: For organizations that have large budgets, ElasticSearch can be implemented with extra resources for maintenance as well as fine-tuning. This is because flexibility may enable several extensions that can justify the investment.

3. Technical Expertise
Limited Technical Skills: If the team does not have deep technical experience, Meilisearch is easy to manage and implement, which is suitable for devs with different levels of experience. Its comprehensive documentation makes it even easier to implement.
Moderate Technical Skills: Algolia offers powerful functionality while remaining easy to use. Though it is not very intuitive, getting into it requires a little technical know-how but its great tools accompanied by comprehensive documentation make it quite usable by mid-level developers.
High Technical Skills: ElasticSearch is suitable for teams with advanced technical knowledge since it requires additional technical guidance for installation and maintenance. Its use is best suited for organizations that have the technical expertise to understand and operate search technologies because of its complexity and abundance of features.

4. Desired Features

Basic Search Functionality: For applications that only need basic search functionalities on projects, Meilisearch provides built-in features like indexing and real-time search, making it a practical choice.
Advanced Features: For specific features such as geo-search, faceted search, and extensive customizations, Algolia is a strong contender because of the abundance of features it offers developers.
Complex Querying and Analytics: ElasticSearch is the most powerful choice for applications that need to provide complex search skills, work in real-time, and process unstructured data. It has a comprehensive array of features that allow it to meet extended search needs and assist with data processing.

Conclusion

The choice between Meilisearch, Algolia, and Elasticsearch boils down to your specific project needs. Meilisearch is most suitable for smaller projects with limited resources, offering simplicity and performance. Algolia's speed and customization options are perfect for complex search requirements. Elasticsearch is recommended for large-scale applications needing advanced querying and data processing capabilities.

Consider factors like the amount of data involved, budget constraints, and desired features when making your decision. An effective search solution can greatly improve user experience and have a direct impact on broad business goals. Selecting the appropriate tool increases the chances of innovation, and distinguishes your application from others in the market.

How to Build Low-latency AI Video Avatars for Real-time Interactions

Tiioluwani — Wed, 27 Nov 2024 09:48:45 +0000

AI avatars are changing the face of content creation and how appealing videos can be created for various purposes, whether it’s training video content, internal team communication, or customer support. Advanced AI tools and video generators now make it possible to create studio-quality, customized videos featuring avatars that can speak multiple languages. An AI video generator enables you to create videos with human-like digital avatars that can connect with a global audience.

In this guide, you'll explore how to build a low-latency AI video avatar using Simli's API and WebRTC. By the end, you’ll be able to create and upload AI-generated videos that sync perfectly with real-time speech, ideal for training, interactive simulations, and presentations. You’ll also be able to generate stunning AI avatars and animated videos in minutes, reaching audiences at scale with human-like, multilingual content.

Prerequisites

To follow through this tutorial, you'll need:

A basic understanding of Javascript and Next.js
A Simli account. To get started, create a free Simli account.
OpenAI account (paid version).

The complete source code is available on GitHub.

With these essentials in place, let's now explore the core elements driving low-latency AI avatars and why low latency is required to provide a fluid, realistic user experience.

Understanding the Key Components for Low-Latency AI Avatars

To demonstrate the importance of low latency, imagine a virtual assistant that takes several seconds to respond to a user’s question. Frustrating, right? Another illustration is an AI instructor in a training video whose voice and lip movements lag behind the audio in AI videos. These irregularities affect the effectiveness of AI interactions, making low latency an absolute necessity.

Latency determines the amount of interaction and realistic experience with AI-powered avatars. In scenarios like training sessions, live presentations, or customer support, an avatar needs to respond as naturally as a human would. A latency of more than 200 milliseconds makes the responses appear delayed and unnatural for any human interaction. Low latency, therefore, gives way to flawless and real-time engagement, thereby leading to a lot more satisfaction and a much better user experience.

Let's look closer at the core technologies and tools behind low-latency AI video avatars and see how each adds to creating an ideal, real-time interaction.

Core Technologies for Building Low-Latency AI Video Avatars

To accomplish low-latency setup, you’ll use the following technologies and format:

Simli's API: Simli's API allows you to create lip-synced AI avatars that convert audio into video in real time. This API provides the foundation for transforming audio into synchronized visual avatars quickly.
WebRTC: This handles the media stream capture and adds to the peer connection.
PCM and Audio Downsampling: To minimize latency in audio transmission, pulse code modulation (PCM) is used for audio data, which WebRTC can handle efficiently. PCM digitizes audio signals by sampling analog waveforms and converting them into uncompressed binary data. Downsampling to 16kHz PCM ensures seamless compatibility with WebRTC and the Simli client, supporting real-time, synchronized audio playback.
OpenAI API: Powers real-time language understanding and response generation, enabling the avatar to engage in dynamic, intelligent interactions.

Having covered the technologies and tools, let's dive into how you can set up your API environment. For the following steps, you will be configuring access to both Simli and OpenAI so you can have smooth integration and interaction within your application.

Set Up Your API Environment

You'll need API keys for both Simli and OpenAI to set up the API environment. To get a Simli API key, go to Simli's official website, click "Get API Access" and create an account.

After logging in, you should see your developer dashboard with your API keys as shown below.

Copy and keep the API key somewhere safe because you will need it to authenticate requests within your application.

With your API key ready, let’s move on to customizing the appearance of your AI avatar. Navigate to the ‘Available Faces’ section in Simli’s API documentation to find listed Avatar face IDs as seen below:

You can also create your own avatar by clicking on the “Create Avatar” Icon on your dashboard. Once you follow the steps highlighted on the page, you’ll be able to create your own Avatar.

To get the ID for each face, copy the random text after the name. For example, the ID for Jenna will be tmp9i8bbq7c. If you created your own Avatar, an ID would also be given alongside. Now that you have the face ID and the Simli API key, let’s create a Next.js app.

Create the Project

Create a Next.js app by running this command in your terminal:

npx create-next-app@latest simli-demo

Answer the following prompts:

What is your project named? my-app
Would you like to use TypeScript? No / Yes
Would you like to use ESLint? No / Yes
Would you like to use Tailwind CSS? No / Yes
Would you like your code inside a `src/` directory? No / Yes
Would you like to use App Router? (recommended) No / Yes
Would you like to use Turbopack for `next dev`?  No / Yes
Would you like to customize the import alias (`@/*` by default)? No / Yes
What import alias would you like configured? @/*

Typescript and Tailwind are essential for this tutorial, as they simplify styling and provide type safety, making it easier to create a responsive and maintainable avatar interface. Once that is done, install the dependencies or packages for this project. They include:

simli-client: This will be used for handling facial animations or avatar interactions.
openai: This is the main OpenAI SDK and will be used for accessing the AI service.
openai/realtime-api-beta: This is OpenAI's real-time language API that will be used for voice conversations.
Ws: WebSocket client and server implementation for real-time communication.

Open up your terminal once more and run the command below, it will install all the dependencies needed for the project.

npm install simli-client openai github:openai/openai-realtime-api-beta ws

Add Environment Variables
Create a .env file and copy your Simli API key and OpenAI key. Paste it in as shown below:

NEXT_PUBLIC_SIMLI_API_KEY="SIMLI-API-KEY"
NEXT_PUBLIC_OPENAI_API_KEY="OPENAI-API-KEY"

Create the VideoBox Components

You'll need to create a component to handle the video feed display and the audio playing. Inside the app folder, create a folder called components, and inside it, create a file called VideoBox.tsx and add the following code:

// app/components/VideoBox.tsx

    // Render a styled video and audio container with auto-play functionality.

    export default function VideoBox(props: any) {
       return (
           <div className="aspect-video flex rounded-sm overflow-hidden items-center h-[350px] w-[350px] justify-center bg-simligray">
               <video ref={props.video} autoPlay playsInline></video>
               <audio ref={props.audio} autoPlay ></audio>
           </div>
       );
    }

The code defines a VideoBox component that displays a video and audio player within a styled container. This component expects two props: video and audio, which are references passed to the <video> and <audio> elements. Both elements have autoPlay enabled, starting playback immediately, and playsInline is set on the video to prevent it from going fullscreen on mobile devices. These ref attributes (props.video and props.audio) allow the primary component to control playback, pause, and other media interactions.

Build the SimliOpenAIPushToTalk Component

Inside the app folder, create a file called SimliOpenAIPushToTalk.tsx. This file will contain much of the core functionality. Start by adding the following code:

// app/SimliOpenAIPushToTalk.tsx

    // import the necessary packages
    ...

    import React, { useCallback, useEffect, useRef, useState } from "react";
    import { RealtimeClient } from "@openai/realtime-api-beta";
    import { SimliClient } from "simli-client";
    import VideoBox from "./Components/VideoBox";
    import IconExit from "@/media/IconExit";
    interface SimliOpenAIPushToTalkProps {
      simli_faceid: string;
      openai_voice: "echo" | "alloy" | "shimmer";
      initialPrompt: string;
      onStart: () => void;
      onClose: () => void;
    }
    const simliClient = new SimliClient();
    const SimliOpenAIPushToTalk: React.FC<SimliOpenAIPushToTalkProps> = ({
      simli_faceid,
      openai_voice,
      initialPrompt,
      onStart,
      onClose,
    }) => {
      const [isLoading, setIsLoading] = useState(false);
      const [isAvatarVisible, setIsAvatarVisible] = useState(false);
      const [error, setError] = useState("");
      const [isRecording, setIsRecording] = useState(false);
      const [userMessage, setUserMessage] = useState("...");
      const [isButtonDisabled, setIsButtonDisabled] = useState(false);
      const videoRef = useRef<HTMLVideoElement>(null);
      const audioRef = useRef<HTMLAudioElement>(null);
      const openAIClientRef = useRef<RealtimeClient | null>(null);
      const audioContextRef = useRef<AudioContext | null>(null);
      const streamRef = useRef<MediaStream | null>(null);
      const processorRef = useRef<ScriptProcessorNode | null>(null);
      const audioChunkQueueRef = useRef<Int16Array[]>([]);
      const isProcessingChunkRef = useRef(false);
    ...

In the above codes, the necessary packages and hooks are imported, a prop interface SimliOpenAIPushToTalkProps is defined for component configuration, and an instance of SimliClient is created to manage multimedia interactions. The SimliOpenAIPushToTalk component requires a specific set of props, which are defined in the SimliOpenAIPushToTalkProps interface.

References are also set up for video and audio elements, the OpenAI client, audio context, media stream, and audio processor, allowing direct control over multimedia resources within the component.

Next, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // initializing the SimliClient
    ...

    const initializeSimliClient = useCallback(() => {
        if (videoRef.current && audioRef.current) {
          const SimliConfig = {
            apiKey: process.env.NEXT_PUBLIC_SIMLI_API_KEY,
            faceID: simli_faceid,
            handleSilence: true,
            videoRef: videoRef,
            audioRef: audioRef,
          };
          simliClient.Initialize(SimliConfig as any);
          console.log("Simli Client initialized");
        }
      }, [simli_faceid]);
      const initializeOpenAIClient = useCallback(async () => {
        try {
          console.log("Initializing OpenAI client...");
          openAIClientRef.current = new RealtimeClient({
            apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY,
            dangerouslyAllowAPIKeyInBrowser: true,
          });
          await openAIClientRef.current.updateSession({
            instructions: initialPrompt,
            voice: openai_voice,
            turn_detection: { type: "server_vad" },
            input_audio_transcription: { model: "whisper-1" },
          });
          openAIClientRef.current.on(
            "conversation.updated",
            handleConversationUpdate
          );
          openAIClientRef.current.on(
            "input_audio_buffer.speech_stopped",
            handleSpeechStopped
          );
          await openAIClientRef.current.connect();
          console.log("OpenAI Client connected successfully");
          setIsAvatarVisible(true);
        } catch (error: any) {
          console.error("Error initializing OpenAI client:", error);
          setError(`Failed to initialize OpenAI client: ${error.message}`);
        }
      }, [initialPrompt]);
      const handleConversationUpdate = useCallback((event: any) => {
        console.log("Conversation updated:", event);
        const { item, delta } = event;
        if (item.type === "message" && item.role === "assistant") {
          console.log("Assistant message detected");
          if (delta && delta.audio) {
            const downsampledAudio = downsampleAudio(delta.audio, 24000, 16000);
            audioChunkQueueRef.current.push(downsampledAudio);
            if (!isProcessingChunkRef.current) {
              processNextAudioChunk();
            }
          }
        } else if (item.type === "message" && item.role === "user") {
          setUserMessage(item.content[0].transcript);
        }
      }, []);
    ...

The code above does the following:

Initializes the SimliClient with configuration parameters, including simli_faceid for Simli's face recognition and video references, and manages audio-visual synchronization with the AI avatar using the API key and face ID. With this, you'll have a real-time lip-sync between the avatar's visuals and audio input.
Sets up and connects the OpenAI client—registers event listeners for updates in conversation and speech detection. The initializeOpenAIClient function sets up the connection to the OpenAI client, using the provided API key and settings, and registers event listeners to handle conversation updates and detect when speech stops.
Handles real-time updates from the conversation, processing both assistant messages and user inputs. For audio responses from the assistant, the code downsamples and adds chunks to a queue.

Directly after handleConversationUpdate, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // process audio chunks and send to Simli
    ...

    const processNextAudioChunk = useCallback(() => {
        if (
          audioChunkQueueRef.current.length > 0 &&
          !isProcessingChunkRef.current
        ) {
          isProcessingChunkRef.current = true;
          const audioChunk = audioChunkQueueRef.current.shift();
          if (audioChunk) {
            const chunkDurationMs = (audioChunk.length / 16000) * 1000; 

            simliClient?.sendAudioData(audioChunk as any);
            console.log(
              "Sent audio chunk to Simli:",
              chunkDurationMs,
              "Duration:",
              chunkDurationMs.toFixed(2),
              "ms"
            );
            isProcessingChunkRef.current = false;
            processNextAudioChunk();
          }
        }
      }, []);

      const handleSpeechStopped = useCallback((event: any) => {
        console.log("Speech stopped event received", event);
      }, []);

      const downsampleAudio = (
        audioData: Int16Array,
        inputSampleRate: number,
        outputSampleRate: number
      ): Int16Array => {
        if (inputSampleRate === outputSampleRate) {
          return audioData;
        }
        const ratio = inputSampleRate / outputSampleRate;
        const newLength = Math.round(audioData.length / ratio);
        const result = new Int16Array(newLength);
        for (let i = 0; i < newLength; i++) {
          const index = Math.round(i * ratio);
          result[i] = audioData[index];
        }
        return result;
      };
    ...

Here:

The processNextAudioChunk manages the processing of audio chunks from the assistant, sending them immediately to Simli.
The downsampleAudio converts audio data to a 16kHz PCM format. PCM captures audio in an uncompressed, raw format, ideal for maintaining low latency. The PCM audio is then sent to SimliClient without encoding delays.

After the downsampleAudio function, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...

    // Manages audio recording setup and cleanup for real-time processing.
    ...

    const startRecording = useCallback(async () => {
        if (!audioContextRef.current) {
          audioContextRef.current = new AudioContext({ sampleRate: 24000 });
        }
        try {
          console.log("Starting audio recording...");
          streamRef.current = await navigator.mediaDevices.getUserMedia({
            audio: true,
          });
          const source = audioContextRef.current.createMediaStreamSource(
            streamRef.current
          );
          processorRef.current = audioContextRef.current.createScriptProcessor(
            2048,
            1,
            1
          );
          processorRef.current.onaudioprocess = (e) => {
            const inputData = e.inputBuffer.getChannelData(0);
            const audioData = new Int16Array(inputData.length);
            for (let i = 0; i < inputData.length; i++) {
              audioData[i] = Math.max(
                -32768,
                Math.min(32767, Math.floor(inputData[i] * 32768))
              );
            }
            openAIClientRef.current?.appendInputAudio(audioData);
          };
          source.connect(processorRef.current);
          processorRef.current.connect(audioContextRef.current.destination);
          setIsRecording(true);
          console.log("Audio recording started");
        } catch (err) {
          console.error("Error accessing microphone:", err);
          setError("Error accessing microphone. Please check your permissions.");
        }
      }, []);

      const stopRecording = useCallback(() => {
        if (processorRef.current) {
          processorRef.current.disconnect();
          processorRef.current = null;
        }
        if (streamRef.current) {
          streamRef.current.getTracks().forEach((track) => track.stop());
          streamRef.current = null;
        }
        setIsRecording(false);
        console.log("Audio recording stopped");
      }, []);
    ...

The startRecording function accesses the microphone in real time using getUserMedia, captures audio with AudioContext, processes it as 16-bit PCM, and sends it to the OpenAI client. The stopRecording function stops the stream and releases resources. This setup uses AudioContext for real-time audio, while WebRTC configuration is managed by the Simli client.

Next to add is the stopRecording function, paste this code below:


// app/SimliOpenAIPushToTalk.tsx
...
// Control interaction, push-to-talk, and audio visualization.

...

const handleStart = useCallback(async () => {
    setIsLoading(true);
    setError("");
    onStart();
    try {
      await simliClient?.start();
      await initializeOpenAIClient();
    } catch (error: any) {
      console.error("Error starting interaction:", error);
      setError(`Error starting interaction: ${error.message}`);
    } finally {
      setIsAvatarVisible(true);
      setIsLoading(false);
    }
  }, [initializeOpenAIClient, onStart]);
  const handleStop = useCallback(() => {
    console.log("Stopping interaction...");
    setIsLoading(false);
    setError("");
    setIsAvatarVisible(false);
    simliClient?.close();
    openAIClientRef.current?.disconnect();
    if (audioContextRef.current) {
      audioContextRef.current.close();
    }
    stopRecording();
    onClose();
    console.log("Interaction stopped");

    window.location.reload();
  }, [stopRecording]);

  const handlePushToTalkStart = useCallback(() => {
    if (!isButtonDisabled) {
      startRecording();

      simliClient?.ClearBuffer();
      openAIClientRef.current?.cancelResponse("");
    }
  }, [startRecording, isButtonDisabled]);
  const handlePushToTalkEnd = useCallback(() => {
    setTimeout(() => {
      stopRecording();
    }, 500);
  }, [stopRecording]);

  const AudioVisualizer = () => {
    const [volume, setVolume] = useState(0);
    useEffect(() => {
      const interval = setInterval(() => {
        setVolume(Math.random() * 100);
      }, 100);
      return () => clearInterval(interval);
    }, []);
    return (
      <div className="flex items-end justify-center space-x-1 h-5">
        {[...Array(5)].map((_, i) => (
          <div
            key={i}
            className="w-2 bg-black transition-all duration-300 ease-in-out"
            style={{
              height: `${Math.min(100, volume + Math.random() * 20)}%`,
            }}
          />
        ))}
      </div>
    );
  };

The code above does the following:

Initializes the Simli and OpenAI clients through the handleStart function, and resets the error state. It also sets isLoading to true while the clients start.
Stops recording, disconnects clients through the handleStop function and resets the component state.
Manages push-to-talk functionality through the Push-to-Talk functions starting and stopping recording when the user interacts with the button.
Displays an animated audio visualizer that gives feedback on microphone volume through the audio visualizer.

Following the audioVisualizer function, add the code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // Initialize Simli client and render video and push-to-talk UI.

    useEffect(() => {
        initializeSimliClient();
        if (simliClient) {
          simliClient?.on("connected", () => {
            console.log("SimliClient connected");
            const audioData = new Uint8Array(6000).fill(0);
            simliClient?.sendAudioData(audioData);
            console.log("Sent initial audio data");
          });
        }
        return () => {
          try {
            simliClient?.close();
            openAIClientRef.current?.disconnect();
            if (audioContextRef.current) {
              audioContextRef.current.close();
            }
          } catch {}
        };
      }, [initializeSimliClient]);
      return (
        <>
          <div className="transition-all duration-300 ">
            <VideoBox video={videoRef} audio={audioRef} />
          </div>
          <div className="flex flex-col items-center">
            {!isAvatarVisible ? (
              <button
                onClick={handleStart}
                disabled={isLoading}
                className="w-full h-[52px] mt-4 disabled:bg-gray-600 disabled:text-white disabled:hover:rounded-full bg-green-500 text-white py-3 px-6 rounded-full transition-all duration-300 hover:text-black hover:bg-white hover:rounded flex justify-center items-center"
              >
                {isLoading ? (
                  <span>Loading...</span>
                ) : (
                  <span className="font-abc-repro-mono font-bold w-[164px]">
                    Test Interaction
                  </span>
                )}
              </button>
            ) : (
              <>
                <div className="flex items-center gap-4 w-full">
                  <button
                    onMouseDown={handlePushToTalkStart}
                    onTouchStart={handlePushToTalkStart}
                    onMouseUp={handlePushToTalkEnd}
                    onTouchEnd={handlePushToTalkEnd}
                    onMouseLeave={handlePushToTalkEnd}
                    disabled={isButtonDisabled}
                    className={`mt-4 text-white flex-grow bg-green-500 hover:rounded hover:bg-opacity-70 h-[52px] px-6 rounded-full transition-all duration-300 ${
                      isRecording ? "bg-gray-900 rounded hover:bg-opacity-100" : ""
                    }`}
                  >
                    <span className="font-abc-repro-mono font-bold w-[164px]">
                      {isRecording ? "Release to Stop" : "Push & hold to talk"}
                    </span>
                  </button>
                  <button
                    onClick={handleStop}
                    className=" group w-[52px] h-[52px] flex items-center mt-4 bg-red text-white justify-center rounded-[100px] backdrop-blur transition-all duration-300 hover:bg-white hover:text-black hover:rounded-sm"
                  >
                    <IconExit className="group-hover:invert-0 group-hover:brightness-0 transition-all duration-300" />
                  </button>
                </div>
              </>
            )}
          </div>
        </>
      );
    };
    export default SimliOpenAIPushToTalk;

In the code above:

The useEffect hook runs when the page mounts to initialize simliClient and sets up cleanup operations to close simliClient and audioContextRef when the component unmounts.
Render the UI to:
- Display the VideoBox component, which receives video and audio props for controlling media playback, along with controls to start interaction or initiate push-to-talk.
- A button triggers handleStart or handleStop, while the push-to-talk button manages recording on mouse down and up.

Now that the core components and UI elements are set up, let’s move on to integrating these into the main demo component. Head over to page.tsx and replace its contents with the following code to complete the setup.

Set Up Demo Component
Navigate to page.tsx and replace it with the following code:

    // app/page.tsx

    // render the AI avatar with push-to-talk functionality
    "use client";
    import React, { useEffect, useState } from "react";
    import SimliOpenAIPushToTalk from "./SimliOpenAIPushToTalk";
    interface avatarSettings {
      name: string;
      openai_voice: "echo" | "alloy" | "shimmer";
      simli_faceid: string;
      initialPrompt: string;
    }
    const avatar: avatarSettings = {
      name: "Frank",
      openai_voice: "echo",
      simli_faceid: "5514e24d-6086-46a3-ace4-6a7264e5cb7c",
      initialPrompt:
        "You are a helpful AI assistant named Frank. You are friendly and concise in your responses. Your task is to help users with any questions they might have. Your answers are short and to the point, don't give long answers be brief and straightforward.",
    };
    const Demo: React.FC = () => {
      const [interactionMode, setInteractionMode] = useState<
        "pushToTalk" | undefined
      >(undefined);
      useEffect(() => {
        const storedInteractionMode = localStorage.getItem("interactionMode");
        if (storedInteractionMode === "pushToTalk") {
          setInteractionMode("pushToTalk");
        }
      }, []);
      const onStart = () => {
        console.log("Setting setshowDottedface to false...");
      };
      const onClose = () => {
        console.log("Setting setshowDottedface to true...");
      };
      return (
        <div className="bg-black min-h-screen flex flex-col items-center font-abc-repro font-normal text-sm text-white p-8">
          <div className="flex flex-col items-center gap-6 bg-effect15White p-6 pb-[40px] rounded-xl w-full">
            <div>
              <SimliOpenAIPushToTalk
                openai_voice={avatar.openai_voice}
                simli_faceid={avatar.simli_faceid}
                initialPrompt={avatar.initialPrompt}
                onStart={onStart}
                onClose={onClose}
              />
            </div>
          </div>
        </div>
      );
    };
    export default Demo;

In the above code, you created a component that renders the AI avatar with push-to-talk functionality. It also initializes settings for the avatar. You can also customize your avatar here by changing it to any face you want for interaction. Simli supports a range of custom avatar faces that you can apply to various use cases.

Set Up Layout
The last thing you’ll need to do is edit the layout, in the layout.tsx, paste this code below:

 // app/layout.tsx

    // edit the layout
    import type { Metadata } from "next";
    import { Inter } from "next/font/google";
    import { abcRepro, abcReproMono } from './fonts/fonts';
    import "./globals.css";

    const inter = Inter({ subsets: ["latin"] });
    export const metadata: Metadata = {
     title: "Simli App",
     description: "create-simli-app (OpenAI)",
    };
    export default function RootLayout({
     children,
    }: Readonly<{
     children: React.ReactNode;
    }>) {
     return (
       <html lang="en" className={`${abcReproMono.variable} ${abcRepro.variable}`}>
         <body className={inter.className}>{children}</body>
       </html>
     );
    }

Test the application by running the following command on your terminal:

npm run dev

Check the video demonstration of the project to see it in action. You can also explore the Simli repository to build avatars and work on more projects using Simli API.

Conclusion

In this project, you have implemented a low-latency AI video avatar for realistic and dynamic interactions. Simli's API with WebRTC allows developers to easily deploy and manage latency-free, perfectly synchronized audio-visual avatars that are very well-suited for a wide range of applications, such as training simulations, interactive presentations, customer support, and real-time virtual assistants.

This setup opens the door to engaging, human-like interactions across various settings.

Explore the documentation to learn more about the features and tools Simli offers.

Creating A Virtual Assistant Using Python

Tiioluwani — Sat, 03 Dec 2022 05:23:19 +0000

Are you attempting to obtain a Python project that will familiarize you with particular Python libraries? A project that could undoubtedly help you with that is a virtual assistant.

The Python libraries used in this project can help you improve your Python programming skills. Working on this project is also enjoyable because having a virtual assistant is incredible.read more

Understanding Python Functions

Tiioluwani — Tue, 02 Aug 2022 20:37:41 +0000

As you progress in Python programming, you will undoubtedly want to use some functions to perform specific programming tasks, so understanding what they are is critical.

In this article, we'll look at these functions, learning what they are and when and how to use them.To read more

How To Enable "running Scripts" On Your System

Tiioluwani — Sat, 16 Jul 2022 21:31:48 +0000

Have you ever been in a situation whereby you're unable to perform specific tasks on your source code editor? For instance, in trying to activate a virtual environment on your editor, you come across an error that says "the command cannot be loaded on your system because running scripts is disabled on your system". We'll take a look at how to solve this in this article.To know more

Basic Built-in Python Functions To Know As A Beginner

Tiioluwani — Tue, 14 Jun 2022 19:53:32 +0000

In python, getting familiar with built-in functions helps a lot while programming. They help in breaking big tasks into smaller chunks and increase code’s clarity. Also, big programs are broken down into smaller units thus making it easier to read, understand and debug if the need arises. In this tutorial, we’ll be taking a look at some built-in functions in python programming.To continue reading

SETTING UP A VIRTUAL ENVIRONMENT

Tiioluwani — Mon, 13 Jun 2022 11:41:37 +0000

It is a fact that as we engage in various kinds of projects in python, we'll come across dependencies. These dependencies are a general phrase in software engineering that refers to when one piece of software depends on another or, simply put, a set of other people's code that we can install and utilize instead of having to rewrite it ourselves. When we come across these dependencies, having a virtual environment then comes in handy.To continue reading

DEV Community: Tiioluwani

Navigating Search Solutions: A Comprehensive Comparison Guide to Meilisearch, Algolia, and ElasticSearch

Introduction

Key Considerations When Choosing a Search Solution

Unveiling the Contenders

Meilisearch

Algolia

ElasticSearch

Feature Comparison Table

Scalability

Performance

Ease of Integration

Indexing

Querying

Relevance

Customization and User Experience

Choosing the Right Solution

1. Data Size

2. Budget

3. Technical Expertise

4. Desired Features

Conclusion

How to Build Low-latency AI Video Avatars for Real-time Interactions

Prerequisites

Understanding the Key Components for Low-Latency AI Avatars

Core Technologies for Building Low-Latency AI Video Avatars

Set Up Your API Environment

Create the Project

Create the VideoBox Components

Build the SimliOpenAIPushToTalk Component

Conclusion

Creating A Virtual Assistant Using Python

Understanding Python Functions

How To Enable "running Scripts" On Your System

Basic Built-in Python Functions To Know As A Beginner

SETTING UP A VIRTUAL ENVIRONMENT