DEV Community

Cover image for How to Build an AI Companion That Listens, Thinks, Speaks, and Reacts in Real Time
Praneeth Kawya Thathsara
Praneeth Kawya Thathsara

Posted on

How to Build an AI Companion That Listens, Thinks, Speaks, and Reacts in Real Time

#ai #aicompanion #app #development

AI assistants are becoming more capable every year.

They can understand speech, generate natural responses, remember context, analyze user intent, and communicate using realistic voices.

But there is still one major problem:

Most AI products have no visible personality.

A user speaks.

The system processes the request.

A voice answers.

But visually, there may be nothing more than a waveform, glowing orb, loading spinner, or static avatar.

For many products, that is a missed opportunity.

A well-designed interactive mascot can become the visual personality layer between the AI system and the user.

Instead of simply showing that an AI is “loading,” a character can visibly:

  • listen
  • think
  • speak
  • react
  • celebrate
  • show empathy
  • recover from errors
  • fall asleep when inactive
  • follow the user's cursor or touch position

This article explains how I approach building these systems using Rive, State Machines, runtime inputs, lip sync, and AI application events.

If you are building an AI assistant, AI tutor, voice companion, SaaS copilot, or conversational application, this architecture can be used to turn a static mascot into a real product interface.

The Important Shift: Stop Thinking About Animation Clips

The first mistake is treating an AI character as a collection of animations.

For example:

idle.riv
thinking.riv
talking.riv
happy.riv
error.riv
Enter fullscreen mode Exit fullscreen mode

That can work for simple projects, but it quickly becomes difficult to maintain.

A better approach is to think of the mascot as a small visual system controlled by the application.

The application already knows what the AI is doing.

For example:

User is speaking
AI is processing
AI is responding
Connection failed
Answer was correct
User has been inactive
Enter fullscreen mode Exit fullscreen mode

The animation system should receive these events and visually represent them.

This means the mascot becomes part of the application logic rather than an animation playing on top of the UI.

This is also the approach we focus on at Mascot Engine, where interactive Rive mascots are prepared around real product events, AI responses, runtime inputs, and developer handoff.

Mascot Engine:
https://mascotengine.com

A Simple AI Companion Architecture

A production-ready AI character does not necessarily need dozens of separate controls.

You can build a surprisingly flexible system using a few well-planned inputs.

For example:

activity
emotion
viseme
audioLevel
lookX
lookY
blink
Enter fullscreen mode Exit fullscreen mode

Let's break these down.

1. Activity State

The first input controls what the AI is currently doing.

For example:

activity = 0  // Idle
activity = 1  // Listening
activity = 2  // Thinking
activity = 3  // Speaking
activity = 4  // Connecting
activity = 5  // Error
activity = 6  // Sleeping
Enter fullscreen mode Exit fullscreen mode

Now the application can control the character using its actual internal state.

Example flow:

User starts speaking
        ↓
activity = 1

User finishes speaking
        ↓
activity = 2

LLM begins generating the response
        ↓
activity = 3

Response finishes
        ↓
activity = 0
Enter fullscreen mode Exit fullscreen mode

The mascot is no longer guessing what should happen.

It is simply visualizing the state of the application.

2. Emotion Should Be Independent

Activity and emotion should usually be separated.

Why?

Because an AI can be speaking while happy, concerned, empathetic, surprised, or neutral.

For example:

emotion = 0  // Neutral
emotion = 1  // Happy
emotion = 2  // Empathetic
emotion = 3  // Concerned
emotion = 4  // Surprised
Enter fullscreen mode Exit fullscreen mode

Now we can combine states.

activity = speaking
emotion = happy
Enter fullscreen mode Exit fullscreen mode

or:

activity = speaking
emotion = concerned
Enter fullscreen mode Exit fullscreen mode

This dramatically reduces the number of animations that need to be created.

Instead of manually creating:

happy-speaking
sad-speaking
concerned-speaking
surprised-speaking
happy-listening
concerned-listening
...
Enter fullscreen mode Exit fullscreen mode

the system can layer reusable behaviors.

This modular approach is one reason State Machines work particularly well for interactive characters.

Rive has also demonstrated this idea in real AI-character use cases. For Duolingo's AI-powered Video Call experience, modular animation and State Machine blending were used to help Lily listen, process, and react dynamically rather than relying on a huge library of pre-baked reactions.

3. Listening State

Listening should communicate:

“I am paying attention to you.”

It does not need to be dramatic.

Subtle behaviors often work better.

For example:

  • slightly leaning forward
  • focusing the eyes
  • reducing random idle movement
  • occasional blinking
  • subtle head movement
  • reacting slightly to microphone activity

The important point is that the user should immediately understand that the system is receiving input.

4. Thinking State

Thinking is one of the most important AI states because latency becomes much easier to tolerate when the interface acknowledges it.

Instead of freezing the character, the mascot can:

  • look upward
  • glance sideways
  • move its eyes slowly
  • change posture
  • perform a subtle thinking loop

The goal is not to fake intelligence.

The goal is to clearly communicate:

“Your request was received and the system is processing it.”

5. Speaking State

Speaking should not simply mean playing a looping mouth animation.

There are multiple levels of implementation.

Level 1 — Basic talking animation 

isTalking = true
Enter fullscreen mode Exit fullscreen mode

A mouth animation loops while speech is active.

Simple and lightweight.

Level 2 — Audio-driven mouth movement

The application sends something like:

audioLevel = 0.0 → 1.0
Enter fullscreen mode Exit fullscreen mode

The mouth responds to audio amplitude.

For example:

0.0 = closed
0.3 = slightly open
0.7 = open
1.0 = fully open
Enter fullscreen mode Exit fullscreen mode

This is much more convincing than a random speaking loop.

Level 3 — Viseme Lip Sync

For more advanced AI voice products, the application can provide phoneme or viseme information.

For example:

viseme = 0
viseme = 1
viseme = 2
...
viseme = 9
Enter fullscreen mode Exit fullscreen mode

Each value represents a mouth shape.

The result can feel significantly more natural.

Recent Rive character examples also demonstrate layered State Machine systems with visemes, expressions, triggers, and speech-driven lip sync inside a single interactive character file.

At Mascot Engine, this is one of the systems we can prepare for AI and TTS-driven products:

https://mascotengine.com

6. Interactive Eye Tracking

Another relatively small feature can make a huge difference:

lookX = -1 → 1
lookY = -1 → 1
Enter fullscreen mode Exit fullscreen mode

These values can be connected to:

  • cursor position
  • touch position
  • camera direction
  • product UI events
  • predefined conversational behaviors

Example:

lookX = -1
Enter fullscreen mode Exit fullscreen mode

Character looks left.

lookX = 1
Enter fullscreen mode Exit fullscreen mode

Character looks right.

lookX = 0
lookY = 0
Enter fullscreen mode Exit fullscreen mode

Character returns to the center.

A subtle gaze system can make the character feel much more aware of the interface around it.

Mascot Engine uses similar runtime-ready controls for interactive gaze, AI states, product reactions, and other app-driven behaviors.

7. Success, Failure, and Product Events

Do not make the mascot respond only to the AI conversation.

It can also respond to product events.

For example:

answerCorrect
answerIncorrect
goalCompleted
paymentSuccessful
streakReached
levelUp
connectionLost
retry
Enter fullscreen mode Exit fullscreen mode

This becomes especially useful for:

  • AI education apps
  • habit trackers
  • health applications
  • fintech products
  • productivity apps
  • onboarding experiences
  • gamified SaaS products

Imagine an AI learning app.

Student answers correctly
       ↓
success trigger
       ↓
Mascot celebrates
Enter fullscreen mode Exit fullscreen mode

The character is now reinforcing the product experience.

8. What About Inactivity?

AI companions can also respond when nothing is happening.

For example:

0–5 minutes
Idle

5–10 minutes
Bored / relaxed

10+ minutes
Sleeping
Enter fullscreen mode Exit fullscreen mode

When the user returns:

wake trigger
      ↓
wake animation
      ↓
idle
Enter fullscreen mode Exit fullscreen mode

These small details make a character feel much more persistent.

9. Build One State Machine, Not 30 Disconnected Animations

A simplified architecture might look like:

                 ┌─────────────┐
                 │    IDLE     │
                 └──────┬──────┘
                        │
         ┌──────────────┼──────────────┐
         ↓              ↓              ↓
     LISTENING       THINKING        ERROR
         │              │
         └──────────┬───┘
                    ↓
                 SPEAKING
                    │
                    ↓
                   IDLE
Enter fullscreen mode Exit fullscreen mode

Then emotion, blinking, gaze, and mouth behavior can operate as additional layers.

This creates a much cleaner system than manually triggering unrelated clips.

10. Example Runtime Contract

Before handing the animation to developers, I like to define a simple contract.

For example:

Artboard:
AI_Companion

State Machine:
CompanionController
Enter fullscreen mode Exit fullscreen mode

Inputs:

activity
Number
0–6

emotion
Number
0–4

viseme
Number
0–9

audioLevel
Number
0–1

lookX
Number
-1–1

lookY
Number
-1–1

blink
Trigger
Enter fullscreen mode Exit fullscreen mode

Then document exactly what every input means.

This sounds simple, but it prevents a huge number of integration problems.

11. Example AI Event Mapping

The development team can then map existing application events to the character.

Conceptually:

onUserSpeechStarted(() => {
  activity = 1;
});

onUserSpeechEnded(() => {
  activity = 2;
});

onAIResponseStarted(() => {
  activity = 3;
});

onAIResponseFinished(() => {
  activity = 0;
});

onConnectionError(() => {
  activity = 5;
});
Enter fullscreen mode Exit fullscreen mode

The exact implementation depends on your framework and AI architecture.

But the animation contract stays predictable.

That is the important part.

12. Where the AI Fits

The mascot itself does not need to contain the AI.

Think of it as three separate layers:

AI / LLM
     ↓
Application Logic
     ↓
Rive Character
Enter fullscreen mode Exit fullscreen mode

The LLM decides what to say.

Your application manages the conversation.

Rive visualizes what is happening.

This separation makes the system easier to maintain.

13. What About AI-Generated Emotion?

You can take this one step further.

Suppose the AI returns structured metadata alongside its response.

Conceptually:

{
  "response": "You did really well today.",
  "emotion": "happy"
}
Enter fullscreen mode Exit fullscreen mode

Your application maps:

happy → emotion = 1
Enter fullscreen mode Exit fullscreen mode

Then the mascot speaks using the correct facial expression.

You could use the same idea for:

neutral
happy
empathetic
concerned
surprised
Enter fullscreen mode Exit fullscreen mode

The important rule is to keep the animation interface simple.

Do not expose dozens of animation controls to the LLM if five stable states will achieve the same result.

14. Where This Architecture Works Well

I especially like this approach for:

AI Tutors

The character can listen, think, explain answers, celebrate correct responses, and react to mistakes.

AI Companions

A persistent character can visually communicate emotion and conversation state.

Voice Assistants

Listening, processing, speaking, interruption, and connection states become visible.

SaaS Copilots

A mascot can guide onboarding and respond to product events.

Kids' AI Products

Visual communication can make abstract system states easier to understand.

Health and Wellness Apps

A calm companion can communicate support and progress without overwhelming the UI.

Productivity Apps

The mascot can react to goals, focus sessions, completion, streaks, and inactivity.

15. Why Rive Works Well for This

There are several reasons I use Rive for these projects.

Runtime control

The application can control inputs while the character is running.

State Machines

Complex transitions can be kept inside the animation system.

Reusable rigs

The same character foundation can support many behaviors.

Interactive animation

The mascot can react to product data rather than simply playing videos.

Web and app workflows

The same animation concept can be prepared for modern product environments.

Rive's own marketplace increasingly includes production-oriented mascot examples built around State Machines, expressions, interactive controls, cursor tracking, data binding, and AI-like listening/thinking states.

16. The Developer Handoff Matters as Much as the Animation

A beautiful character that developers cannot integrate easily is not production-ready.

A good delivery should clearly define:

.riv source
artboard names
State Machine names
input names
input types
number ranges
event mappings
default values
fallback behavior
responsive behavior
Enter fullscreen mode Exit fullscreen mode

Depending on the project, it can also include:

PNG stills
editable source
Flutter guidance
React Native guidance
Web guidance
test implementation
Enter fullscreen mode Exit fullscreen mode

At Mascot Engine, the focus is specifically on this kind of production-ready Rive mascot system rather than only delivering isolated animation loops.

17. A Practical Minimum Viable AI Companion

You do not need 50 animations for version one.

A strong MVP could contain:

Idle
Listening
Thinking
Speaking
Happy
Concerned
Error
Sleeping
Blink
Enter fullscreen mode Exit fullscreen mode

plus:

audioLevel
lookX
lookY
Enter fullscreen mode Exit fullscreen mode

That is already enough to create a surprisingly expressive AI interface.

You can always expand it later.

18. The Bigger Idea

The interesting opportunity is not simply adding animation to AI.

It is giving the AI system a consistent visual behavior layer.

The AI already has:

  • intelligence
  • memory
  • voice
  • context

The mascot provides:

  • presence
  • expression
  • feedback
  • personality
  • visual state communication

That is why I believe interactive characters will become increasingly useful inside AI products.

The goal is not to put a cartoon character everywhere.

The goal is to use a character where human-like visual feedback improves the product experience.

Need an AI Companion or Interactive Rive Mascot?

I run Mascot Engine, a studio focused on production-ready interactive Rive characters for:

  • AI assistants
  • AI companions
  • AI tutors
  • SaaS products
  • mobile apps
  • web applications
  • conversational interfaces

Projects can include:

  • character preparation
  • Rive rigging
  • idle / listening / thinking / speaking states
  • interactive State Machines
  • emotion systems
  • eye tracking
  • audio-driven mouth animation
  • viseme lip sync
  • product-event reactions
  • editable Rive files
  • developer handoff
  • Flutter, React Native, and web preparation

You can see live interactive mascot examples and technical details here:

👉 https://mascotengine.com

If you are building an AI product and already have a mascot, UI prototype, character sketch, or even only an idea, you can send the product brief and required behaviors to Mascot Engine to plan the animation architecture before development.

Do not ask:

“What animation should the mascot play?”

Instead ask:

“What does the application know right now, and how should the character visually communicate that state?”

That small change turns a mascot from decoration into part of the product.

Rive #AI #Animation #WebDevelopment #AppDevelopment #GenerativeAI #ArtificialIntelligence #SaaS #Flutter #ReactNative #UX #UIUX #CharacterAnimation #AIAssistant #AICompanion

Top comments (0)