Building Malo: what makes a small robot feel present?

Emmanuel Letremble — Sun, 04 Oct 2026 22:04:17 +0000

A personal Reachy Mini project exploring conversation, attention, and expressive behavior.

I had planned to spend the weekend working on my Anthropic certifications. Then my Reachy Mini arrived on Friday, and the study plan acquired a head, two antennas, and a rather different set of priorities.

I originally started studying machine learning and AI because I wanted to give robots a spark of life. My career took me toward other AI applications, but I kept coming back to the same idea: building something that could share a physical space with people and respond to them at any time, without being too intrusive, yet still being present when it matters.

This isn't my first attempt to make that idea tangible (several of them still haunt my house), but this is the first one that seems to be working well enough to explore questions such as: How can humans and robots share a space naturally? How can a robot become part of our everyday environment without demanding our constant attention, interrupting conversations, or being too discreet? And how can interactions with AI feel natural, expressive, and genuinely enjoyable?

My new laboratory for exploring these questions is Reachy Mini, an open-source robotics platform with integrated motor control, sensors, camera, audio, networking, and software APIs, all managed by a Raspberry Pi.

To begin with, I turned Reachy Mini into Malo, a small robot you can easily speak with, that can turn toward a speaker, recognize previously enrolled voices, remember useful information about them, search for information on the internet, express itself through movement, and check if we are spending too much time on the phone or even speak in beeps with other machines.

The phone reminder has a very practical inspiration: my kid spending too long on its phone. Apparently, my childhood dream of robotics now includes outsourcing “Could you put that phone down?”

Here is Malo in action. The demo is in French, with English subtitles, and I recorded it roughly 24 hours after receiving the package.

While it mainly showcases the speech-to-speech capabilities of Gemini 3.8 Live, it also provides good examples of how the LLM interacts with the robot’s body and, indirectly, with its environment (Philips Hue, ggwave devices...). Overall, it gives a good sense of just how close today’s technology is getting to natural, embodied conversation and how fun and easy this can be.

Provided with the Reachy Mini

The Reachy Mini SDK and daemon included with the robot provide the motors, kinematics, camera, microphone and speaker access, GStreamer media pipelines, the SDK's YuNet face detector and Robot API.

By the way, thanks to Pollen Robotics for making Reachy Mini such an approachable platform to experiment with. Having the SDK, daemon and hardware environment ready to use makes it possible to focus on building behaviors and ideas instead of fighting the robot.

On top of that foundation, Malo adds local speech and vision models, speaker-aware behavior, persistent memory, device messaging and external integrations. The following sections explain how these pieces work together.

Added in this project

Offline wake-up: sherpa-onnx Zipformer with a faster-whisper Whisper Tiny fallback, using ONNX Runtime and CTranslate2 for local inference;
Speaker recognition: sherpa-onnx Silero VAD and CAM++;
Speaker-aware gaze: custom gaze logic combining the Reachy Mini SDK's YuNet face detector with the ReSpeaker microphone direction;
Phone awareness: local D-FINE detection on camera images;
Real-time conversation and orchestration: Gemini Live, Python/asyncio and the Reachy Mini SDK/Robot API;
Long-term memory: Karpathy's LLM Wiki pattern, Markdown journal and wiki pages, a compact index and remote text-model consolidation;
Settings: FastAPI for the settings server and ggwave for device-to-robot transfer;
Integrations: the Philips Hue local API (CLIP v2) through the Hue Bridge and optional web-search providers.

The design challenge is bringing these capabilities together into one coherent, responsive interaction within the limits of a small onboard computer. And to do that, we need to use very small models...

Local models and behavior

sherpa-onnx's French streaming Zipformer model in int8 for offline wake-phrase detection, with Whisper Tiny (~39M parameters) as a fallback;
sherpa-onnx's Silero VAD model and 3D-Speaker CAM++ (~7.2M parameters) for identifying enrolled speakers;
YuNet face detection, combined with ReSpeaker microphone direction, to follow the person who is talking;
D-FINE nano (4M parameters) to detect a phone in someone's hand.

These are compact, specialized models rather than one large local brain. The largest local model bundle is the deployed Zipformer wake-word model: its encoder, decoder and joiner files occupy about 123 MB on disk and contain approximately 73.8M stored tensor values, mostly quantized. This complete local workload fits comfortably in 4 GB, but the board is not suited to running a local LLM or several large vision models continuously; the models are executed at modest rates and the larger conversational model remains remote.

Speaker enrollment and identification also run locally. A person reads for about 25 seconds; Silero VAD isolates the speech, then CAM++ turns roughly 3-second samples into embeddings whose average becomes that person's voiceprint. During a conversation, each sufficiently long phrase is compared with the enrolled voiceprints using cosine similarity. After about two seconds of speech, Malo starts an early identification check in the background, in parallel with the ongoing conversation (this does not delay Gemini's response). If it finds a sufficiently confident match, that provisional identity can be sent to Gemini before the phrase ends; because it uses only partial audio, it may be corrected later. When the phrase ends, a second check uses more audio to confirm or correct the result and improve reliability. The recognized name is sent to Gemini, added to the transcript and used to route the journal entry to the right memory page; recordings and voiceprints stay on the robot.

YuNet is the face detector model accessed through the Reachy Mini SDK and runs locally in Malo; the daemon/API provides the camera stream, while the app runs the detection and its speaker-aware gaze logic. It provides each face's position and height. To help it find the person who is speaking, we use the ReSpeaker microphone array, which is hardware rather than a model and estimates the rough direction of the voice. If no face is there, the head turns toward the sound; while Malo speaks, it ignores the microphones so it does not chase its own voice.

All of this runs on Malo's Raspberry Pi: local models, audio, vision, safety checks and orchestration. The remaining challenge is keeping the system responsive as these tasks run together, while preventing unsafe motion and self-triggering. When Malo sleeps, the Gemini session and camera stop too, so the expensive parts are not running unnecessarily.

Responsiveness is a whole-system property

Conversation makes timing immediately noticeable. While Malo listens and speaks, it may also need to pay attention to someone, move, or respond to a request involving its surroundings.

An onboard computer has a limited processing budget. The practical challenge is keeping the overall interaction responsive as these activities overlap. Having a capability available matters less if using it makes everything else feel sluggish.

Malo combines onboard processing with cloud services. Local perception and behavior coordination run on the robot, while conversation currently relies on Gemini Live over the internet.

I designed the voice layer to be replaceable, so Malo can evolve beyond a single provider. That leaves room for other speech-to-speech services or a combination of speech recognition (STT), a conversational model, and speech synthesis (TTS). The design goal is to preserve Malo's behaviors and continuity as those choices evolve. Languages, voice quality, and response timing will depend on the selected solution.

A conceptual view of the design: Gemini Live powers the current demo; dashed cards represent options to explore.

This balance brought me back to a familiar engineering question: where does each capability best serve the experience, given the constraints of the device?

Memory changes the next conversation

A companion feels different when it can carry something useful from one conversation into the next. Malo can retain information between sessions and recognize voices that have been enrolled beforehand.

That continuity opens up more personal interactions, but it also raises the importance of getting context right. A remembered detail is useful when it is relevant and associated with the right person. Repeating an old detail at the wrong moment or to the wrong person can be more distracting than forgetting it.

For this project, memory is about supporting that sense of continuity. It is one more part of the robot's behavior to judge through actual conversations.

The actual memory system design is inspired by Andrej Karpathy's LLM Wiki idea and the discussion of applying the same principle to code: keep raw material separate, consolidate it into structured linked pages, and load a compact index instead of making the model reread everything. This gives Malo persistent memory between live sessions: a new Gemini Live session starts without the previous session's context. Because the durable information is stored as Markdown pages and a compact index on the robot, rather than inside a particular conversational model, we can change that model later without losing it. A remote text model consolidates the memory while Gemini Live handles real-time speech-to-speech and function calling.

Attention is a robotics problem

For a robot sharing a room with people, attention has a physical direction. Facing someone helps make it clear who the robot is engaging with.

But a room is ambiguous. There may be several people, a face may be out of view, or a voice may be too brief to identify confidently. Recognizing who is speaking and knowing where to look are related challenges, but they are not interchangeable.

With Malo, I worked on speaker-aware behavior and gaze because they make a tangible difference to the conversation. The goal is for its attention to feel understandable to the person in front of it.

I also tried including Malo in a conversation with four enrolled users, and the results were surprisingly good. It was able to identify the different speakers, respond using their names and information it remembered about them, and keep up with the overall conversation. The main weakness was turn-taking: Malo tended to answer whenever it could rather than waiting for a natural opening or for its turn to speak.

That also means treating uncertainty as part of the interaction. An occasional missed cue is one thing; confidently addressing the wrong person is another. For me, this is an important lesson: a perception capability is only useful if the resulting behavior makes sense when the signal is imperfect.

One of my next priorities is helping Malo distinguish speech directed at it from conversations between people in the room. Recognizing a speaker does not tell the robot whether it is being addressed. I want to work on that distinction so it can respond when invited, let people talk without interjecting, and handle ambiguous situations appropriately. This is planned work: deciding when to participate is part of making a robot a considerate presence.

Movement and interaction sounds give the conversation another dimension

Reachy Mini has a compact body, yet a head movement or a change in antenna position can make an exchange feel surprisingly expressive.

Working with that body means thinking about range of motion, posture, and timing. Looking toward a person and performing a gesture both involve the same physical robot. Those intentions need to coexist coherently, within the platform's movement limits.

The question I kept returning to was: what does this movement communicate? A small, well-timed gesture can contribute more than a large movement that has little connection to the conversation.

There is room for play, too. Malo has pretend flying gestures and a robot-like chirping voice. The chirps can actually carry short messages to and from a compatible device, which makes them both a playful expression and an interaction of their own.

ggwave is an acoustic modem: it encodes short text messages as audible tones that a nearby device can send to Malo, and Malo can chirp its replies back. These are not decorative beeps; they carry real commands, text and responses. I also use it for short messages about Malo's current feelings or state: a compatible ggwave receiver can decode the tones back into readable text, so you can read what the robot is expressing. I chose it partly because the modem bursts sound like R2-D2 while transmitting "useful" information.

Interacting with the outside world

Another way for a robot to express itself is through tools that connect it to the world around it. So I decided to connect Malo to my colorful Hue setup and let it play with my lighting system at will.

Philips Hue is integrated locally through the bridge's CLIP v2 API over HTTPS. Malo can discover the bridge, pair with it using its button, store the application key on the robot and read or control rooms, zones, lamps and scenes: on/off, brightness, colors, white temperature and transitions. The cloud is used only for optional bridge discovery; lighting commands stay on the local network.

What the prototype has taught me

Malo is still an early prototype. One moment in the demo captures that well: it presented PostgreSQL 17 as the latest version. Having web search available did not guarantee that its answer was current.

It’s no surprise, since this project inherits many of the same problems as LLMs. Still, I found it a useful reminder that we need to evaluate the complete behavior rather than blindly trust it just because it has a pretty face. A fluent answer, a successful detection, or an available tool only tells part of the story. What matters is what happens during the interaction, including when something goes wrong.

For web search, Tavily is the default provider (Google Search as an alternative). But if its key is missing, refused or out of quota, the app falls back to DuckDuckGo/Bing through the ddgs search package.

The same applies to the phone reminder I built with my kid's screen time in mind. Noticing a phone staying in view is one capability; making a reminder that feels playful rather than repetitive or intrusive is a question of interaction design.

These are the details I want to keep improving: how reliably Malo responds, when it takes the initiative, and whether its behavior remains easy to understand.

Next: experimenting with agentic architectures and loops

I also plan to test different agentic architectures and loops: ways for Malo to work toward a goal by observing, choosing an action, checking the result, and deciding what to do next. I want to explore useful multi-step behaviors that can adapt when the situation changes or a person interrupts.

The interesting question is which approaches fit a physical companion with limited onboard resources. I want to compare how they affect responsiveness, use of tools, and the coherence of the interaction. These are experiments planned for future iterations, alongside the work on recognizing when someone is addressing Malo.

Final word

For now, I have a small robot that talks, turns toward people, remembers things, and can help deliver the family phone-break reminder. Whether my kid finds the message more convincing with antennas remains to be seen.

But what strikes me most is not only what Malo can do, but how accessible it was to build. Existing AI APIs and models can be connected to a well-designed robotic body and software stack with surprisingly little integration work. In roughly 10 to 20 hours of work, I brought together a remarkably complete system on Malo (with guided coding agents helping me accelerate selected parts of the implementation).

The result feels far beyond what that amount of work would suggest; with what I know now, I could probably build it faster. The work of the teams behind these models, APIs, libraries and robot platforms (let's not forget we are indeed standing on the shoulders of giants, so, many thanks to them) makes it possible for an individual to turn the dream of bringing robots to life into a working prototype.

What would make a small companion robot genuinely integrated in your daily life?

DEV Community: Emmanuel Letremble