Praneeth Kawya Thathsara

Posted on Sep 9 Originally published at mascotengine.com

How to Build an Interactive AI Assistant Avatar for Web, Flutter and React Native

#rive #flutter #reactnative #ai

An AI assistant avatar is not just a looping video. It is a visible interface to an asynchronous system: microphone capture, model responses, speech playback, interruptions, and failures.

If the character keeps talking after the user presses Stop, the animation is communicating the wrong product state. If it celebrates before an action succeeds, it creates the wrong expectation. A convincing avatar starts with reliable behavior, then adds personality.

For founders and developers building a web product, Flutter app, React Native experience, or SaaS assistant, the useful question is: what contract connects the character to the application?

This guide proposes a practical architecture using Rive State Machines, six named controls, and a shared runtime asset. The state names and numeric ranges below are design recommendations, not built-in Rive conventions. You can adapt them to your product without coupling the animation to one AI provider.

Why Rive fits live product states

A video has a predetermined sequence. A live assistant needs to change direction when something happens: the user interrupts, audio stalls, a request fails, or the interface loses focus.

Rive State Machines let designers connect animation states through conditional transitions. Layers can separate behaviors such as body motion and facial response. That makes it possible to author reusable reactions instead of exporting a video for every combination. See Rive’s State Machine overview.

The important boundary is that your application owns the conversation and business logic. It decides whether a request succeeded and whether audio is actually playing. Rive presents that decision through motion; it should not become your networking or conversation controller.

Treat “lightweight” as a measurable target, not a guarantee. Budget the .riv asset, runtime download, external assets, memory, and frame time together. A simple vector character can still become expensive when the scene accumulates unnecessary complexity. Rive’s runtime introduction also cautions that feature availability depends on the runtime and renderer.

Start with seven essential states

Before rigging a character, write down what each state means and which application event ends it.

State	Product meaning	Suggested visual behavior
`idle`	Ready for the next interaction	Quiet breathing, occasional blink
`listening`	Microphone capture is active	Attentive posture, restrained input response
`thinking`	Waiting for an answer or preparing playback	Small recurring motion that tolerates unknown duration
`speaking`	Assistant audio is playing	Mouth motion with controlled facial expression
`success`	The requested operation has completed successfully	Brief acknowledgement, then return to the appropriate state
`error/offline`	The interaction cannot continue normally	Calm concern, accompanied by a useful text message
`sleeping`	The assistant is deliberately inactive	Resting pose with minimal movement

These meanings matter more than the animation style. Microphone permission alone does not mean listening. Receiving the first text token does not mean speaking. A completed answer does not prove that a requested database update succeeded.

Define transition priority explicitly. An error or cancellation should interrupt speaking; a success gesture should not delay a new listening session. Use interruptible transitions where responsiveness matters, and reserve longer transitions for moments that cannot block interaction.

For an initial implementation, allow one primary activity at a time. Keep emotion and gaze separate. That avoids multiplying states into combinations such as “happy-speaking-looking-left” and makes the design easier to maintain.

Build a clean input architecture

There is an important terminology update: Rive recommends Data Binding and View Models for new projects. Legacy State Machine Inputs still appear in older integrations, but they are no longer the preferred starting point. View Model properties can drive transitions and other bound elements. Read the official Data Binding overview.

Here, “inputs” means the public controls your application supplies. Implement them as exported View Model properties for a new system, with a documented adapter if you must support an older file.

For example, name the artboard Assistant, the State Machine AssistantSM, and the View Model AssistantVM. The following deliberately uses numeric properties to keep the example contract simple; enum properties are another option when supported by your selected runtime versions.

Control	Proposed values	Responsibility
`activity`	Integer 0–6	0 idle, 1 listening, 2 thinking, 3 speaking, 4 success, 5 error/offline, 6 sleeping
`emotion`	Integer 0–3	0 neutral, 1 warm, 2 pleased, 3 concerned
`viseme`	Integer 0–7	Project-specific mouth shape; 0 is rest
`audioLevel`	Number 0–1	Smoothed level of the active audio source
`lookX`	Number -1–1	Horizontal gaze; positive means character’s screen-right
`lookY`	Number -1–1	Vertical gaze; positive means down on screen

Validate values before writing them. Reject invalid activity IDs, clamp continuous controls, and map unknown mouth shapes to rest. Document whether numeric choices are discrete selectors or blend values: interpolating between activity IDs should never accidentally select an unrelated state.

Give each property one owner. The conversation controller owns activity; the speech controller owns mouth timing; the pointer or attention controller owns gaze. Otherwise, an idle timer can overwrite a speaking update while audio is still playing.

flowchart LR
    A[AI event] --> B[Application logic]
    B --> C[Rive State Machine input]
    C --> D[Character response]

For a new file, the diagram’s input step is a View Model property update that drives a State Machine transition. A speech-start event becomes activity = 3; playback completion resets the mouth and selects the next activity.

Use a conversation or utterance ID to discard late events. After cancellation, a delayed completion callback from the previous utterance must not reset a new listening session. Store the latest complete control snapshot so a newly loaded character can immediately catch up with the application.

Need an interactive Rive character for your product?

Mascot Engine creates app mascots, AI companions, State Machines, lip sync, and developer-ready Rive systems for Web, Flutter, and React Native.

View live work and request an estimate at Mascot Engine.

Send your project brief on WhatsApp.

Choose the right speaking system

There are two useful levels of mouth animation. Choose based on the speech data your application can actually provide.

Audio-driven speaking

For a simple assistant, derive a normalized amplitude envelope from the assistant’s output audio. Smooth abrupt changes and map that value to jaw opening or a speaking blend.

This communicates activity without requiring phoneme timing. It is not phonetic lip sync: loudness cannot tell you whether the speaker is forming an “M” or an “O.” Avoid choosing random mouth shapes and describing the result as accurate synchronization.

Use output audio for speaking and microphone audio only for listening feedback. Gate mouth movement on actual playback. During silence, buffering, cancellation, or playback failure, return the mouth to rest rather than leaving the last open pose visible.

Specify which animation owns mouth opening. If audioLevel and viseme both drive the same deformation independently, they can conflict. A sensible contract uses one speaking mode at a time, or explicitly defines amplitude as a small modulation of a selected mouth pose.

Viseme lip sync

A viseme is a visible mouth pose associated with speech sounds. Several sounds can share a pose, so the character usually needs fewer visual shapes than there are phonemes.

Prepare a small mouth library with rest, closed lips, wide, rounded, and other shapes suited to the design. Then document the mapping from the speech system’s timing labels to your viseme IDs. The proposed 0–7 range is only an example; choose the actual set with your animator.

Your speech or alignment system must provide the timing data. This architecture does not assume that loading a .riv file automatically extracts visemes from arbitrary audio.

Schedule changes against the audio playback position, not the arrival time of network chunks. For streamed speech, account for each chunk’s offset and any buffering. Briefly blend compatible shapes where useful, while preserving important closures. Test pauses, rapid speech, interruption, and the end of every utterance.

If timing data is unavailable, fall back to amplitude-driven speaking or a restrained speaking loop. Keep the same activity contract so the rest of the product remains predictable.

Add gaze without making it distracting

Convert a pointer position or UI attention target into normalized character coordinates. Clamp lookX and lookY, ease toward the target, and return to center when the target disappears.

Gaze should support attention: toward the microphone during listening, toward a relevant result after completion, or toward the user during speech. It does not require camera-based eye tracking.

On touch devices, use a tap target or product event instead of assuming a continuously moving cursor exists. Test the gaze after resizing and changing character placement; screen coordinates and artboard coordinates are not interchangeable.

Prepare one .riv system for three platforms

The reusable unit is the exported asset plus its behavioral contract. The host integrations remain platform-specific. Choose runtime versions and renderers first, then validate the same exported file on every target. Do not infer mobile compatibility solely from the editor preview.

Web

Load the file, choose the artboard and State Machine, bind the default View Model instance, and obtain property references only after loading completes.

For example, inside a Web runtime load callback, an adapter can cache a property and update it:

// Requires a loaded Rive instance with autoBind: true,
// an exported default View Model instance, and a number property.
const activity = riveInstance.viewModelInstance?.number("activity");
if (!activity) throw new Error("Missing avatar activity property");

function showSpeaking() {
  activity.value = 3;
}

This illustrates the Web Data Binding API, not a complete voice assistant. Your application still manages audio, readiness, errors, and cancellation.

Handle canvas sizing, asset-load failure, and cleanup when the component unmounts. Rive’s Web parameters reference documents artboard selection, State Machine selection, automatic binding, and lifecycle APIs.

Flutter

Use the current Rive Flutter integration appropriate to your project. Rive documents file loaders, RiveWidgetBuilder, and RiveWidgetController in its Flutter runtime guide; its Flutter Data Binding guide explains accessing and binding View Model instances.

Create and retain the loader and controller at the correct widget lifecycle boundary. Follow the selected API’s ownership and disposal rules. Keep audio-level updates localized instead of rebuilding the entire screen for each sample.

Flutter’s performance guidance recommends avoiding costly repeated work in build() and limiting rebuild scope. Profile the actual screen with the avatar, scrolling, and audio active together, especially on the lowest device tier you support.

React Native

Rive’s current React Native setup guide uses @rive-app/react-native and requires Nitro Modules. Check the documented React Native, native OS, and build-tool requirements before integrating it into an existing app.

Use that runtime’s file loading, view, and View Model APIs to implement the same six controls. Avoid mixing snippets from the legacy package with current APIs; Rive provides a migration guide.

Connect foreground/background changes to your playback policy using React Native’s AppState API. On resume, synchronize from current application state instead of blindly replaying an old speaking animation.

Across platforms, show textual status and preserve usable controls even if the character cannot load. Offer reduced motion and avoid conveying errors through expression alone. React Native’s AccessibilityInfo API exposes reduced-motion preferences and change notifications.

Developer handoff checklist

Ask for a package that another developer can integrate without reverse-engineering the editor file.

Editable .riv source requirement, clarified: request editable Rive project access or a .rev backup and the runtime-ready .riv. Rive distinguishes its editable backup export from its runtime export; a runtime .riv alone is not the complete editable source.
Artboard name: exact spelling, dimensions, intended fit, and alignment.
State Machine name: exact exported identifier and initial behavior.
Named inputs: property names, types, View Model name, default instance, and any legacy adapter mapping.
Valid ranges: discrete values, continuous bounds, viseme mapping, and coordinate conventions.
Default values: initial activity, neutral emotion, closed mouth, silent level, and centered gaze.
Transition rules: priorities, interruptions, gesture completion, timeouts, and recovery behavior.
Runtime test notes: tested package versions, renderers, devices, load behavior, and known limitations.
Fallback state: safe idle or static image on asset failure, with errors explained in the surrounding UI.

Before acceptance, replay a small event sequence: load, listen, think, speak, interrupt, fail, retry, and background/resume. Include missing visemes and a slow network. Confirm that cancellation stops both audio and mouth movement, and that stale events cannot change the next interaction.

When to hire a specialist

Bring in a specialist when the work spans character design, rigging, layered State Machines, mouth-pose design, and runtime integration. The difficult part is often making those pieces agree under interruption, not producing an attractive idle animation.

Mascot Engine’s interactive character services cover Rive mascots, AI characters, rigging, State Machines, viseme lip sync, gaze controls, Data Binding/ViewModels, runtime-ready files, and developer documentation for Web, Flutter, and React Native.

For a useful estimate, send your target platforms, existing artwork, required states, speech provider or sample timing data, expected character size, deadline, and budget range. Mention whether your developers need integration support or only documented assets. Those details help define a realistic scope before animation begins.

By Praneeth Kawya Thathsara, founder of Mascot Engine.