DEV Community: Paul Ruiz

Coding a Robot’s Personality: HRI with the Open Duck Mini, Animations, Vibe Coding, and Gemini Live

Paul Ruiz — Sat, 13 Jun 2026 20:00:05 +0000

I believe the next big leap in AI will be in the robotics space, taking the reasoning power of Large Language Models (LLMs) and putting it into physical bodies. While focusing on the utility of a robot, like having it perform tasks or chores, is a compelling area of development, we often overlook the interaction between the human user and the machine itself. Rather than focusing on what the robot can do, we should also look at how a robot's movements and actions can trigger specific expectations and social norms for the person interacting with it.

Recently I’ve been experimenting with the Open Duck Mini, and while finally getting the robot to move its head, walk, and generally just do something is an amazing feeling all on its own, I wanted to take some time and explore the topics of human-robot interaction (HRI) and character development. To do this, I built a custom animation tool that helps define animations that the robot can run through, allowing the robot to move in ways that humans naturally recognize as conveying a message, such as nodding or shaking its head.

With some animations defined, I paired them with the Gemini Live API, which serves as an excellent tool for HRI by allowing people to speak directly to the robot and trigger these gestures so the machine can interact with the world in an appropriate, recognizable way. All of the code for this project, including the animation tool client, Flask API server, and playback scripts, is available on GitHub.

Note: It’s worth noting that I recorded this with all durations set to 300ms, which is why it moves quickly and then directly into the next keyframe interpolation. You can slow down animations by extending the duration of the steps along with changing the easing functions, which I’ll cover more later in this article.

For most of the middleware that I'll discuss here today, I was able to generate the code using Gemini and AI Studio to get what I wanted done quickly, helping me bridge the gap between hardware control and high-level AI reasoning without getting bogged down in the setup. This let me go from defining the idea over a few days of casual thinking, generally while I was gardening or doing something around the house, to spending less than a day creating the end project. That said, I closely orchestrated what was generated and reviewed the code to avoid unexpected hiccups.

Character First: The Disney Approach to HRI

We can learn a lot from the animators and imagineers at Disney, who I highly admire for all of the cool work they do with character creation and human-robot interaction (if you’re unfamiliar with it, definitely look up their concept of Living Characters and what they’ve done in that space; it’s kind of chaotic, but I love it). Often the characters that Disney is building are already defined in a movie or TV show, but sometimes they’re brand new and need to be fleshed out for how they interact in a given situation, what physical quirks they have, how they speak, if there is an accent or specific dialect that’s appropriate, and a whole lot of other things that need to be considered. They establish the character’s personality and the specific ways the character expresses emotions like happiness, curiosity, or frustration, and then they engineer the physical robot to embody that character.

While a robot does not strictly need a personality to function, defining these traits early in the design phase helps ease the interaction between the machine and the people around it. By establishing what personality the robot should project as it is being built, we can ensure its physical behaviors align with human social expectations. Disney's work, such as their autonomous walking Olaf robot, which they’ve released a bit on their process of getting a proper heel to toe gait, shows how combining artistic animation principles with robotics can create machines that feel alive and responsive, rather than merely mechanical.

Along with their work in robotics, I’ve had a copy of The Illusion of Life on my shelf for years, as it’s an amazing reference for general animation (I’ve been drawing as a hobby for a few years), and something I used regularly when working as an Android developer years ago. Some of the concepts, such as the various ways to express reality in cartoons, and especially the concepts around variations in movement speed that are needed when interpolating between key frames, were really useful for this project.

As this open source robot platform is based on a Star Wars droid (which, as a member of the 501st and Rebel Legion costuming charity groups, I knew I “needed” to make one), the personality can be focused on cute beeps and whirls instead of words, and with the way the two legs hold itself being similar to a duck (hence the robot name), I took some inspiration from the overly exaggerated movements of early Donald Duck animations.

Middleware: Why We Build Our Own

Designing physical gestures requires a direct link between the animator's intent and the robot's hardware. Because public animation tools for custom-built hardware are practically non-existent, I needed to develop a custom pipeline to manually position the duck, record those motor configurations as keyframes, and compile them into structured action plans that could execute in sync with audio files that enrich my robot’s response personality. To achieve this quickly, I used AI Studio to generate a custom Flask API server for interfacing with the robot and a companion applet for changing settings on the key frames.

Designing Gestures with the Animation Tool

The workflow centers around a companion applet running on a laptop, which communicates over the network with the Flask server hosted on the Raspberry Pi Zero 2W inside the robot. This applet allows me to read the live motor positions as I physically pose the robot (focused on motor IDs 30, 31, 32, and 33, which are the neck and head motors on the Open Duck Mini), decide what other emotive features should be used, such as eye and projector lights and antenna orientations, and save the entire sequence as a JSON action script that can later be read by my robot’s python code.

The reason this is using SSH instead of a direct USB connection to the robot is two fold: the Raspberry Pi Zero 2W has one USB port that I expanded with a small hub to get a USB microphone added (I could have used an i2s microphone, but I wanted options), but cable management is kind of a nightmare, so I wanted less attached to my robot. That USB microphone also requires that the robot be in a USB host orientation, rather than peripheral, so instead of messing up my working configuration, I just went for an all software approach. I’m also able to take the robot away from home and work on it because I configured the Pi to connect to my home network, and if it doesn’t find that network, it turns on its own hotspot that I can connect to for creating a connection from my laptop.

Here is an example of an action script generated by the tool with the pose in a neutral position, though each frame becomes another node in the keyframes array:

{  
  "keyframes": [  
    {  
      "id": "mpzph7m3gpw97yqw2c",  
      "durationMs": 300,  
      "pauseMs": 200,  
      "motors": {  
        "30": 0.5905826033359718,  
        "31": 0.3988350048502669,  
        "32": -0.18867963690993372,  
        "33": -0.15186409800067846  
      },  
      "lightsOn": false,  
      "projectorOn": false,  
      "interpolation": "bezier",  
      "antennas": {  
        "left": "center",  
        "right": "center"  
      }  
    },  
    ...
  ],  
  "globalSound": "beep1.wav"  
}

The script defines keyframes with target motor positions, lights, and antenna angles. It also specifies a globalSound (like beep1.wav) to play alongside the movement.

The Flask API Server

The API server running on the Raspberry Pi serves as the bridge between the high-level animation design and the physical hardware. It exposes endpoints to read the current motor positions during the recording phase and to execute the playbacks of the saved JSON files for testing (there’s a separate Python code snippet for directly playing back an animation during regular robot use).

Again, this code was generated, but I planned out what it should look like and reviewed it for quality, so let’s check it out in smaller chunks.

1. Initialization and Setup

First, we set up the Flask application, enable Cross-Origin Resource Sharing (CORS) so the laptop applet can communicate with it, and initialize the hardware interface (HWI) for the motors that we’ll be reading. We also define the GPIO pins for the LEDs, projector/flashlight, speaker for audio playback, and small servos that control the duck's antennas.

from flask import Flask, request, jsonify  
from flask_cors import CORS  
from mini_bdx_runtime.duck_config import DuckConfig  
from mini_bdx_runtime.rustypot_position_hwi import HWI  
from gpiozero import LED, AngularServo  
import time


app = Flask(__name__)  
CORS(app)


config = DuckConfig()  
hwi = HWI(config)  
motor_ids = [30, 31, 32, 33]


SPEAKER_INDEX = 0


led1 = LED(23)  
led2 = LED(24)  
projector = LED(25)  
servo_left = AngularServo(12, min_angle=-90, max_angle=90)  
servo_right = AngularServo(13, min_angle=-90, max_angle=90)

Note on Imports: You’ll notice we are importing from mini_bdx_runtime. This is the core runtime package from the original BDX robot project. It wraps the low-level motor communication (using rustypot to talk to the Dynamixel Protocol 2.0 used by the Feetech servos), saving us from writing raw serial byte packets ourselves.

2. Hardware Helper and Dampening Functions

Next, we define helper functions to control the physical components, a mathematical helper for Bezier interpolation, and hardware dampening functions.

While Bezier interpolation in software handles the mathematical path, we also dynamically adjust the motors' internal hardware limits (acceleration and velocity profiles) using the rustypot library. This allows us to create effects like "viscous" (sluggish, heavy) or "clamped" (snappy but smoothed) movements by writing directly to the motor registers.

def set_lights(on):  
    if on:  
        led1.on()  
        led2.on()  
    else:  
        led1.off()  
        led2.off()


def set_projector(on):  
    if on:  
        projector.on()  
    else:  
        projector.off()


def get_antenna_angle(pos_str):  
    if pos_str == 'back': return -90  
    if pos_str == 'forward': return 90  
    return 0


def set_antennas(positions):  
    servo_left.angle = get_antenna_angle(positions.get('left', 'center'))  
    servo_right.angle = get_antenna_angle(positions.get('right', 'center'))


ADDR_PROFILE_ACCEL = 108
ADDR_PROFILE_VELOC = 112


def set_hardware_dampening(accel, veloc):
    try:
        for mid in motor_ids:
            if hasattr(hwi.io, 'write_data'):
                hwi.io.write_data(mid, ADDR_PROFILE_ACCEL, accel, 4)
                hwi.io.write_data(mid, ADDR_PROFILE_VELOC, veloc, 4)
    except Exception as e:
        pass


def apply_interpolation_dampening(interp_type):
    if interp_type == 'linear':
        set_hardware_dampening(0, 400)
    elif interp_type == 'bezier_viscous':
        set_hardware_dampening(10, 200)
    elif interp_type == 'bezier_clamped':
        set_hardware_dampening(40, 500)
    else: # bezier
        set_hardware_dampening(30, 400)


def bezier_interpolate(t, type_str): 
   if type_str == 'linear':
       return t
   if type_str == 'bezier':
       return t * t * (3.0 - 2.0 * t)                  # Standard smoothstep (cubic Hermite)
   if type_str == 'bezier_viscous':
       return t * t * t * (t * (t * 6.0 - 15.0) + 10.0) # Quintic ease-in-out (smoother start/end)
   if type_str == 'bezier_clamped':
       return 1.0 - (1.0 - t) ** 3                      # Cubic ease-out (fast start, slow end)
   return t

3. Reading Motor Positions

To record keyframes in the applet, we need to read the current state of the robot. The /read endpoint queries the hardware interface and returns the current positions of the four head motors.

@app.route('/read', methods=['GET'])  
def read_pos():  
    try:  
        positions = hwi.io.read_present_position(motor_ids)  
        res = {str(mid): pos for mid, pos in zip(motor_ids, positions)}  
        return jsonify(res)  
    except Exception as e:  
        return jsonify({"error": str(e)}), 500

4. The Playback Loop (Gestures & Audio)

The /play endpoint is where the magic happens. It receives the JSON action script, triggers the audio playback (assuming there’s an associated audio file) asynchronously (using sounddevice), and then steps through the keyframes, interpolating motor positions at a strict 30Hz update rate to prevent jitter.

@app.route('/play', methods=['POST'])  
def play_macro():  
    data = request.json  
    keyframes = data.get('keyframes', [])  
    global_sound = data.get('globalSound', '')


    # Handle Audio Playback Asynchronously
    if global_sound:  
        try:  
            import sounddevice as sd  
            import soundfile as sf  
            import numpy as np  
            import os


            sound_path = f"assets/{global_sound}"  
            if os.path.exists(sound_path):  
                audio_data, fs = sf.read(sound_path, dtype='float32')  
                if len(audio_data.shape) == 1:  
                    audio_data = audio_data.reshape(-1, 1)  
                    audio_data = np.tile(audio_data, (1, 2))


                audio_data = audio_data * 2.0  
                sd.play(audio_data, fs, device=SPEAKER_INDEX)  
            else:  
                print(f"Sound file not found: {sound_path}")  
        except Exception as e:  
            print(f"Failed to play sound: {e}")


    # Motor Sequencing Loop
    try:  
        start_positions = hwi.io.read_present_position(motor_ids)  
        current_positions = {str(mid): pos for mid, pos in zip(motor_ids, start_positions)}  
    except:  
        current_positions = {str(mid): 0 for mid in motor_ids}


    try:  
        hwi.io.enable_torque(motor_ids)  
    except:  
        pass


    for frame in keyframes:  
        set_lights(frame.get('lightsOn', False))  
        set_projector(frame.get('projectorOn', False))  
        set_antennas(frame.get('antennas', {}))


        dur_sec = frame.get('durationMs', 1000) / 1000.0  
        steps = max(1, int(dur_sec * 30)) # 30Hz update rate  
        interp = frame.get('interpolation', 'linear')  
        apply_interpolation_dampening(interp)  
        target_motors = frame.get('motors', {})


        start_frame_pos = current_positions.copy()
        start_time = time.time()  
        for step in range(1, steps + 1):  
            t = step / float(steps)  
            eased_t = bezier_interpolate(t, interp)


            step_targets = []  
            for mid in motor_ids:  
                s_val = start_frame_pos.get(str(mid), current_positions.get(str(mid), 0))  
                end_val = target_motors.get(str(mid), target_motors.get(mid, s_val))  
                val = s_val + (end_val - s_val) * eased_t  
                step_targets.append(float(val))  
                current_positions[str(mid)] = val


            try:  
                hwi.io.write_goal_position(motor_ids, step_targets)  
            except Exception as e:  
                print(f"HW error on step {step}: {e}")


            # Strict timing to prevent jitter and stalls  
            target_time = start_time + step * (dur_sec / steps)  
            now = time.time()  
            if target_time > now:  
                time.sleep(target_time - now)


        pause_sec = frame.get('pauseMs', 0) / 1000.0  
        if pause_sec > 0:  
            time.sleep(pause_sec)


    # Reset hardware to default state after animation
    set_lights(False)  
    set_projector(False)  
    set_antennas({'left': 'center', 'right': 'center'})  
    try:  
        hwi.io.disable_torque(motor_ids)  
    except:  
        pass


    return jsonify({"success": True})


if __name__ == '__main__':  
    app.run(host='0.0.0.0', port=5000)

Hardware Dampening vs. Software Timing

You might wonder: if we are already calculating Bezier curves in Python, why do we need to write to the motor's hardware registers?

In robotics, the best results come from combining Software Timing and Hardware Dampening:

Software Timing (Duration & Easing): The Python loop computes precise, time-synchronized positions (e.g., at 30Hz) and sends them to the motors. This ensures that all joints (like neck and head) arrive at their targets at the exact same millisecond, keeping the motion coordinated. It also allows for complex easing curves (like overshoot or anticipation) that hardware cannot calculate on its own.
Hardware Dampening (Smoothing & Physics): The Feetech STS3215 motors have internal PID controllers and motion profiles. By writing to Profile Acceleration (Address 108) and Profile Velocity (Address 112) using the rustypot library, we tune how the hardware physically reacts. This acts as a physical low-pass filter, rounding off any micro-jitters caused by USB-to-Serial (how the Raspberry Pi Zero connects to the motors) communication latency and preventing the motors from buzzing or jerking.

For example, bezier_viscous caps the hardware acceleration and velocity, forcing the motor to "drag" behind the software commands, creating a heavy, fluid, "underwater" feel. Conversely, bezier_clamped allows aggressive tracking for sharp movements but maintains enough acceleration limit to protect the gears from grinding. With these different interpolators in play, you will need to be considerate of how long you specify the motors to take to reach their position, or provide enough of a post-keyframe wait period for things to catch up so that animations aren’t cut short during playback.

Real-Time Interaction with the Gemini Live API

To transform these static gestures into responsive behaviors, I integrated the Gemini Live API into a python script running on the robot. The Live API is particularly well-suited for human-robot interaction because it supports low-latency, low-overhead bidirectional streaming of audio, allowing the user to speak naturally to the machine.

Instead of configuring the model to return voice responses, I structured the interaction so that the robot communicates entirely through physical actions. By utilizing system instructions and function calling, I instructed Gemini to act as a silent physical agent that translates the user's speech into gesture triggers.
The model is configured with the following system instruction:

"You are a physical robot. Do not speak. Respond ONLY using tools. Use trigger_action with 'yes' to agree, 'no' to disagree, or 'beep1' to acknowledge neutrally."

This setup ensures that when the user speaks, Gemini evaluates the input and selects the most appropriate gesture to trigger, reinforcing social norms, like nodding to show agreement, without relying on synthetic speech. In a more robust project, we could add dozens of animations for different emotions and moods to truly give the robot a specific personality.

Let's look at how the Live API integration script is constructed.

1. Imports, Hardware, Audio, and Dampening Setup

We start by importing the necessary libraries (including the google-genai SDK), configuring the audio parameters, and setting up the hardware dampening registers and helper functions. The script needs to handle both microphone input and speaker output, as well as managing physical motor dampening on the fly.

import asyncio  
import os  
import sys  
import json  
import time  
import pyaudio  
import numpy as np  
import soundfile as sf  
from google import genai  
from google.genai import types


from mini_bdx_runtime.duck_config import DuckConfig  
from mini_bdx_runtime.rustypot_position_hwi import HWI  
from gpiozero import LED, AngularServo


config_hw = DuckConfig()  
hwi = HWI(config_hw)  
motor_ids = [30, 31, 32, 33]  
MIC_INDEX = 1  
SPEAKER_INDEX = 0


led1, led2, projector = LED(23), LED(24), LED(25)  
servo_left = AngularServo(12, min_angle=-90, max_angle=90)  
servo_right = AngularServo(13, min_angle=-90, max_angle=90)


FORMAT = pyaudio.paInt16  
CHANNELS = 1  
HW_SAMPLE_RATE = 48000  
API_SEND_RATE = 16000  
CHUNK_SIZE = 1024  
MIC_FACTOR = 3


pya = pyaudio.PyAudio()  
audio_queue_mic = asyncio.Queue(maxsize=5)  
audio_queue_output = asyncio.Queue()


# Register addresses for DYNAMIXEL Protocol 2.0 Dampening (used by Feetech STS3215)
ADDR_PROFILE_ACCEL = 108
ADDR_PROFILE_VELOC = 112


def set_hardware_dampening(accel, veloc):
    """Writes to motor registers to handle hardware-level smoothing."""
    try:
        for mid in motor_ids:
            if hasattr(hwi.io, 'write_data'):
                hwi.io.write_data(mid, ADDR_PROFILE_ACCEL, accel, 4)
                hwi.io.write_data(mid, ADDR_PROFILE_VELOC, veloc, 4)
    except Exception:
        pass


def apply_interpolation_dampening(interp_type):
    """Sets different hardware snappiness based on the JSON interpolation."""
    if interp_type == 'linear':
        set_hardware_dampening(0, 400)
    elif interp_type == 'bezier_viscous':
        set_hardware_dampening(10, 200)
    elif interp_type == 'bezier_clamped':
        set_hardware_dampening(40, 500)
    else: # standard bezier
        set_hardware_dampening(30, 400)


def bezier_interpolate(t, type_str):  
    if type_str == 'linear':
        return t
    if type_str == 'bezier':
        return t * t * (3.0 - 2.0 * t)
    if type_str == 'bezier_viscous':
        return t * t * t * (t * (t * 6.0 - 15.0) + 10.0)
    if type_str == 'bezier_clamped':
        return 1.0 - (1.0 - t) ** 3
    return t

2. Live API Configuration

Here we initialize the Gemini client and define the LiveConnectConfig object. We pass our system instruction and define the trigger_action tool, which tells the Gemini Live API what actions it is able to perform.

api_key = os.environ.get("GEMINI_API_KEY")  
client = genai.Client(api_key=api_key, http_options={'api_version': 'v1alpha'})


live_config = types.LiveConnectConfig(  
    response_modalities=["AUDIO"],  
    system_instruction=types.Content(parts=[types.Part(text=(  
        "You are a physical robot. Do not speak. Respond ONLY using tools. "  
        "Use trigger_action with 'yes' to agree, 'no' to disagree, or 'beep1' to acknowledge neutrally."  
    ))]),  
    tools=[types.Tool(function_declarations=[  
        types.FunctionDeclaration(  
            name="trigger_action",  
            description="Triggers a robot animation and sound.",  
            parameters=types.Schema(  
                type="OBJECT",  
                properties={"action_name": types.Schema(type="STRING", description="Action name like 'yes' or 'beep1'")},  
                required=["action_name"]  
            )  
        )  
    ])]  
)

Note: You'll need to install the dependency via pip install -q -U google-genai and export your API key with export GEMINI_API_KEY="your_key". You can find more information on Gemini API keys here.

3. Audio Input Handling

These two asynchronous tasks handle capturing audio from the physical microphone and sending it to the active Gemini Live session. Since the API expects 16kHz audio, but my mic runs at 48kHz, I downsample the audio in listen_audio by slicing the numpy array. As this is specific to my own hardware modification, your own project may need some modifications here.

async def listen_audio():  
    stream = await asyncio.to_thread(  
        pya.open, format=FORMAT, channels=CHANNELS, rate=HW_SAMPLE_RATE,  
        input=True, input_device_index=MIC_INDEX, frames_per_buffer=CHUNK_SIZE * MIC_FACTOR  
    )  
    while True:  
        data = await asyncio.to_thread(stream.read, CHUNK_SIZE * MIC_FACTOR, exception_on_overflow=False)  
        audio_array = np.frombuffer(data, dtype=np.int16)  
        resampled = audio_array[::MIC_FACTOR].tobytes()  
        await audio_queue_mic.put({"data": resampled, "mime_type": "audio/pcm"})


async def send_realtime(session):  
    while True:  
        msg = await audio_queue_mic.get()  
        await session.send_realtime_input(audio=msg)

If you are running this on a Pi and get PyAudio errors, it is likely because MIC_INDEX or SPEAKER_INDEX doesn't match your hardware. For example, my USB microphone and speaker index swapped at some point when running two pieces of hardware, so I had to go back and update my code values. You can find your specific indexes by running a quick Python script on the Pi:
import pyaudio
p = pyaudio.PyAudio()
for i in range(p.get_device_count()):
    print(f"Index {i}: {p.get_device_info_by_index(i)['name']}")
python
Just run that, look for your USB Mic and Speaker names, and update the index numbers in the setup config

4. Receiving Responses and Dispatching Actions

The receive_and_trigger task listens for responses from Gemini. When the model decides to trigger an action, it sends a tool call. The script intercepts this, spawns a background thread to play the animation JSON that was generated via the animation tool, and sends a success response back to Gemini.

async def receive_and_trigger(session):  
    while True:  
        turn = session.receive()  
        async for response in turn:  
            if response.tool_call:  
                for call in response.tool_call.function_calls:  
                    if call.name == "trigger_action":  
                        action = call.args.get("action_name")  
                        asyncio.create_task(asyncio.to_thread(play_animation_task, action))


                    await session.send_tool_response(  
                        function_responses=[types.FunctionResponse(  
                            id=call.id, name=call.name, response={"status": "ok"}  
                        )]  
                    )  
            sc = response.server_content  
            if sc and sc.input_transcription:  
                print(f"You: {sc.input_transcription.text}")

5. Playing Animations and Audio Output

When an action is triggered, play_animation_task loads the corresponding JSON file, handles the motor sequencing, and loads the audio file. Because the Live API connection is active, we use a shared speaker queue (audio_queue_output) and a unified play_audio task to write the bytes to the speaker hardware.

Because this is a longer bit of code, let’s break it down into its key components.

A. Loading the Action Plan

First, the function locates and loads the JSON file that defines the animation. If the file is missing, it exits early to prevent crashes.

def play_animation_task(action_name):  
    """Parses JSON, triggers motors, and queues audio for the shared speaker task."""  
    try:  
        json_path = f"assets/{action_name}.json"  
        if not os.path.exists(json_path):  
            print(f"File not found: {json_path}")  
            return


        with open(json_path, 'r') as f:  
            data = json.load(f)

B. Audio Processing and Queuing

If the animation has an associated sound, we load the WAV file. Because our hardware might require a different sample rate than the file, we perform real-time resampling. We also apply digital normalization and a 3x gain boost so the duck can be heard over the motor noise before converting the audio to PCM bytes and pushing it to the output queue.

        global_sound = data.get('globalSound')  
        if global_sound:  
            sound_path = f"assets/{global_sound}"  
            if os.path.exists(sound_path):  
                # Read file as float32 for high-quality gain math  
                audio_data, samplerate = sf.read(sound_path, dtype='float32')  
                if len(audio_data.shape) > 1: audio_data = audio_data[:, 0]


                # Resample to hardware 48kHz  
                if samplerate != HW_SAMPLE_RATE:  
                    duration = len(audio_data) / samplerate  
                    audio_data = np.interp(  
                        np.linspace(0, len(audio_data), int(duration * HW_SAMPLE_RATE)),  
                        np.arange(len(audio_data)),  
                        audio_data  
                    )


                # Digital Normalization + Volume Boost  
                max_val = np.max(np.abs(audio_data))  
                if max_val > 0: audio_data = (audio_data / max_val)  
                audio_data = audio_data * 3.0 # Strong 3x Gain


                final_pcm = (audio_data * 32767).clip(-32768, 32767).astype(np.int16)  
                audio_queue_output.put_nowait(final_pcm.tobytes())  
                print(f"ROBOT ACTION: {action_name} (Audio: {global_sound})")

C. Motor Initialization

Before we can move the motors, we need to enable torque and read their current positions so we know where to start the interpolation.

        keyframes = data.get('keyframes', [])  
        try:  
            hwi.io.enable_torque(motor_ids)  
            start_pos = hwi.io.read_present_position(motor_ids)  
            current_pos = {str(mid): pos for mid, pos in zip(motor_ids, start_pos)}  
        except: current_pos = {str(mid): 0 for mid in motor_ids}

D. The Sequencing Loop

This is the core loop. For each keyframe, we set the peripheral states (lights, projector, antennas) and calculate the intermediate motor positions using Bezier interpolation. The loop runs at a target rate, sleeping as necessary to maintain strict timing and prevent jitter.

        for frame in keyframes:  
            set_lights(frame.get('lightsOn', False))  
            set_projector(frame.get('projectorOn', False))  
            set_antennas(frame.get('antennas', {}))  

            interp = frame.get('interpolation', 'linear')
            apply_interpolation_dampening(interp)

            dur = frame.get('durationMs', 1000) / 1000.0  
            steps = max(1, int(dur * 30))  
            target_motors = frame.get('motors', {})  
            start_frame_pos = current_pos.copy()  
            start_t = time.time()  
            for step in range(1, steps + 1):  
                t = step / float(steps)  
                eased_t = bezier_interpolate(t, interp)  
                step_targets = []  
                for mid in motor_ids:  
                    s_val = start_frame_pos.get(str(mid), current_pos.get(str(mid), 0))  
                    e_val = target_motors.get(str(mid), s_val)  
                    val = s_val + (e_val - s_val) * eased_t  
                    step_targets.append(float(val))  
                    current_pos[str(mid)] = val
                try: hwi.io.write_goal_position(motor_ids, step_targets)  
                except: pass  
                target_time = start_t + step * (dur / steps)  
                now = time.time()  
                if target_time > now: time.sleep(target_time - now)  
            if frame.get('pauseMs', 0) > 0: time.sleep(frame['pauseMs'] / 1000.0)

E. Cleanup and Reset

Once the animation completes, we turn off the lights and projector, center the antennas, and disable torque on the motors to prevent overheating. This is where you could change these defaults if your own application required it.

        # Reset components after animation
        set_lights(False); set_projector(False); set_antennas({'left': 'center', 'right': 'center'})  
        try: hwi.io.disable_torque(motor_ids)  
        except: pass  
    except Exception as e:  
        print(f"Action Error: {e}")

6. The Unified Speaker Task and Main Loop

Finally, the play_audio task continuously monitors audio_queue_output and writes any outgoing sound bytes to the speaker hardware. The run function connects the session and orchestrates all the concurrent tasks.

async def play_audio():  
    """Unified speaker task for all robot sounds."""  
    stream = await asyncio.to_thread(  
        pya.open, format=FORMAT, channels=CHANNELS, rate=HW_SAMPLE_RATE,  
        output=True, output_device_index=SPEAKER_INDEX  
    )  
    while True:  
        bytestream = await audio_queue_output.get()  
        await asyncio.to_thread(stream.write, bytestream)


async def run():  
    try:  
        async with client.aio.live.connect(  
            model="gemini-3.1-flash-live-preview", config=live_config  
        ) as live_session:  
            print("Robot Ready. Monitoring mic...")  
            async with asyncio.TaskGroup() as tg:  
                tg.create_task(send_realtime(live_session))  
                tg.create_task(listen_audio())  
                tg.create_task(receive_and_trigger(live_session))  
                tg.create_task(play_audio())  
    except Exception as e: print(f"Error: {e}")  
    finally: pya.terminate()


if __name__ == "__main__":  
    try: asyncio.run(run())  
    except KeyboardInterrupt: pass

Future Explorations

While the Open Duck Mini is a small-scale robot, the engineering challenges involved in this project, such as latency management, audio resampling, real-time synchronization, and hardware-software co-design, are ideal for learning about the challenges faced in large-scale robotics development.

Using generative AI to develop the necessary middleware allowed me to build a complete HRI pipeline in a fraction of the time it would have taken traditionally. By combining flexible models like Gemini with custom physical gestures, we can start building robots that don't just execute commands, but participate in social spaces in a recognizable way.

One thing I would like to do at some point is start modifying this robot to be more robust. I want to put neopixel lights in for the eyes to support various colors based on the robot’s ‘mood’, upgrade from the Raspberry Pi Zero 2w to a Raspberry Pi 5 or Jetson Orin Nano, potentially replace the motors with 12v motors (including the necessary power upgrades). At that point though I may as well build a new BDX droid, which my wallet wouldn’t appreciate, but it’s so cool, so we’ll see :)

I'm looking forward to expanding this setup, and I hope this encourages other makers to explore the intersection of LLMs and physical embodiment. To this end, the next tools to explore would be the Gemini Robotics-ER model for perception and task orchestration, as well as the MuJoCo simulation tool.

Teaching a Robot to Play a Toddler Game: VLAs, Gemini 3 Flash, and First Orchard

Paul Ruiz — Sat, 21 Feb 2026 20:34:35 +0000

As we think about the future of AI, we often land on robotics, or "Embodied AI", as the next logical frontier. It is the quest to take the reasoning capabilities of Large Language Models (LLMs) and make them physically useful in the real world.

That said, the robotics field is vast. It spans everything from mechanical and electrical engineering to software control systems, computer vision, and the complex math required just to handle basic movement.

While I come from a general IoT hobbyist background, I am still getting my bearings in the robotics space. To learn, I decided to dive into a project where I could run head-first into these challenges and work through them in real-time. For this experiment, I am using an SOARM101 robot arm to play the toddler board game First Orchard.

This project brings together two powerful AI components:

A newly trained vision-language-action (VLA) model to control the robot's physical movements.
Gemini 3 Flash to act as the "brain" tracking the game state and rules.

In this post, I will walk you through exactly what this project is, the hurdles I faced, and how I built it.

The Game: First Orchard

First Orchard is a simple, cooperative game designed for two-year-olds. It consists of four circular tree tiles, each holding wooden fruit pieces, a path made of tiles, a crow game piece, and a six-sided die.

The die faces include:

Four Colors: Matching the four fruit types (Red, Blue, Green, Yellow).
The Basket: Allows the player to remove any fruit piece from the board.
The Crow: Moves the crow game piece one step forward along the path.

The Objective: Roll the die and harvest all the fruit pieces before the crow reaches the end of the path.

The Setup

For this project, I used an overhead webcam to provide a "God's eye view" of the board and a wrist camera mounted on the SOARM101 for close-up manipulation.

To keep the environment consistent for the VLA model, I taped down the path tiles, the tree tiles, and a small box used as the "harvest" area where the robot drops removed pieces. While the robot can reach every tree tile, the crow path was intentionally placed out of reach due to table space. For this iteration of the project, the crow is moved manually by a human player when its face is rolled.

Data Collection: The Grind

Data collection was, by far, the most tedious part of this project. After spending hours on it, I have a newfound respect for the teams that collect hundreds of thousands of hours of footage to train foundation models.

I originally considered two approaches:

Task Decomposition: Breaking the movement down into smaller chunks (reaching, gripping, lifting) and solving them via individual policies or inverse kinematics.
End-to-End Policy: Training a single policy that handles the entire sequence—moving to the fruit, gripping it, lifting, moving to the box, dropping it, and returning to the home position.

Since my goal was to push the boundaries of VLA capabilities in a home environment, I opted for the end-to-end flow. I defined four specific tasks to record:

"Place the red fruit in the box"
"Place the blue fruit in the box"
"Place the green fruit in the box"
"Place the yellow fruit in the box"

I aimed for 100 episodes per task, totaling 400 episodes. To get through it, I put on a TV show and settled into the task of teleoperating the arm using a leader/follower robot combination.

Here is the command I used to configure the cameras, robot IDs, and dataset settings:

 lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=follower_arm \
    --robot.cameras="{ wrist_cam_left: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}, overhead_cam: {type: opencv, index_or_path: 4, width: 640, height: 480, fps: 30}}"  \
    --teleop.type=so101_leader \
    --robot.port=/dev/ttyACM1 \
    --teleop.id=leader_arm \
    --display_data=true \
    --dataset.repo_id=paultr/first-orchard \
    --dataset.num_episodes=25 \
    --dataset.single_task="Place the blue fruit in the box"

I recorded the data in smaller batches, using the --resume=true flag to continue adding to the dataset. This was a lifesaver, allowing me to take breaks or recover if a session failed. When I finally finished, I had a library of short video clips capturing every nuance of the harvest.

You can explore the full dataset on HuggingFace here, or watch a set of episodes in action below:

VLA Training: Trials, Errors, and Hardware Gore

With the data collected, the next step was training a model to handle the actual pick-and-place operation. This was the part of the project I was most curious about, so I ran multiple iterations to see how different models and parameters would behave in this specific environment.

Iteration 1: The Quick Test

I started with a very fast (~2 hour) refinement of SmolVLA. I used a small batch size of 8 and only 20,000 steps. I also renamed my camera inputs to camera1 and camera2 to match the SmolVLA convention.

!lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --policy.repo_id=paultr/tmp_smolvla \
  --dataset.repo_id=paultr/first-orchard \
  --batch_size=8 \
  --steps=20000 \
  --output_dir=outputs/train/tmp_smolvla \
  --job_name=my_smolvla_training \
  --policy.device=cuda \
  --rename_map='{"observation.images.overhead_cam": "observation.images.camera1", "observation.images.wrist_cam_left": "observation.images.camera2"}'

The results were mixed. Some episodes worked amazingly well given the short training time:

Others were off-target, "twitchy," or resulted in the robot hovering indefinitely:

Iteration 2: Pushing the Boundary with SmolVLA

While I was convinced that the model could work, I was curious to see if more resources would yield a smoother result. For the next step, I trained SmolVLA for 100,000 steps with a much larger batch size (64). I ran this on an A-100 high-memory Colab session because, as truth would have it, I appreciate being able to spend Google’s money to see just how far I can take a project. You can find the colab I used here.

!lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=paultr/first-orchard \
  --batch_size=64 \
  --steps=100000 \
  --save_freq=10000 \
  --output_dir=/content/drive/MyDrive/outputs/first-orchard-smolvla \
  --job_name=smolvla_first_orchard \
  --policy.device=cuda \
  --policy.push_to_hub=true \
  --policy.repo_id=paultr/first-orchard-smolvla \
  --rename_map='{"observation.images.overhead_cam": "observation.images.camera1", "observation.images.wrist_cam_left": "observation.images.camera2"}'

The quality for this new model was significantly better. It still wasn't perfect, but it was reliable enough to play a real game.

Iteration 3: The Pi0 Disaster

Still curious about alternative architectures, I decided to try training a Pi0-Fast model. I spent about five days training on an A-100 for 200,000 steps. Since Colab sessions shut down after about 24 hours on the Pro plan, I had to link my Google Drive to save and resume checkpoints periodically. You can find the Colab that I used here.

!lerobot-train \
  --dataset.repo_id=paultr/first-orchard \
  --policy.type=pi0_fast \
  --policy.pretrained_path=lerobot/pi0fast-base \
  --output_dir=/content/drive/MyDrive/outputs/first-orchard_pi0_fast \
  --policy.repo_id=paultr/first-orchard_pi_fast \
  --steps=200000 \
  --batch_size=16 \
  --policy.chunk_size=32 \
  --policy.n_action_steps=8 \
  --policy.device=cuda \
  --save_freq=5000

When the model was finally done, I had a real bad time.

During testing, the arm folded in on itself and snapped the 3D-printed camera mount. Luckily, I had extras printed, so it was a quick swap, but no amount of code changes or camera adjustments that I could figure out were able to get this model to behave.

On the plus side, I learned a lot about LeRobot’s optional training flags while trying to debug the carnage. Ultimately, I decided to finish the project with the SmolVLA model, but I'll revisit other models at some point.

Setting up the Python App

Now that the training is out of the way, we can get into the actual code! We’ll start by setting up the base project structure. This handles the fundamental connection to the SOARM101 and establishes a "Home" pose. Even though it's simple, there’s always a little thrill the first time you run a script and the robot moves to a pose on its own.

import sys
import time
import random
import torch
import numpy as np
from PIL import Image
from pydantic import BaseModel, Field

from lerobot.cameras.opencv.configuration_opencv import OpenCVCameraConfig
from lerobot.datasets.utils import hw_to_dataset_features
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
from lerobot.policies.utils import build_inference_frame, make_robot_action
from lerobot.robots.so_follower import SO101Follower, SO101FollowerConfig

FOLLOWER_PORT = "/dev/ttyACM0"
FOLLOWER_ID = "follower_arm"
ROBOT_TYPE = "so101_follower"

# Pre-defined motor positions to bring the arm to a safe starting state
HOME_VALUES = {
    'shoulder_pan.pos': -3.7974683544303787, 'shoulder_lift.pos': -98.51155503329416,
    'elbow_flex.pos': 98.1199641897941, 'wrist_flex.pos': 76.095278604849,
    'wrist_roll.pos': -51.585014409221905, 'gripper.pos': 0.990099009901
}

device = None
robot = None

def home():
    if robot:
        robot.send_action(HOME_VALUES)
        time.sleep(1)

def init():
    global robot
    print("Connecting to Robot...")
    # In the final version, camera_config will be defined here
    robot_cfg = SO101FollowerConfig(port=FOLLOWER_PORT, id=FOLLOWER_ID, cameras=camera_config)
    robot = SO101Follower(robot_cfg)
    robot.connect()
    home()

def main():
    init()
    if robot:
        robot.disconnect()

if __name__ == "__main__":
    main()

Integrating the VLA

With the foundation laid, we need to update the script to load the VLA model that we worked so hard to train. First, we’ll add some configuration constants to the top of the file, including the model ID and camera settings.

Note the MAX_STEPS value: 900 steps at 30 FPS gives the robot roughly 30 seconds to complete a pick-and-place task before it "gives up."

MAX_STEPS = 900 
TOP_CAM_INDEX = 6
TOP_CAM_WIDTH = 640
TOP_CAM_HEIGHT = 480
TOP_CAM_FPS = 30

WRIST_CAM_INDEX = 4
WRIST_CAM_WIDTH = 640
WRIST_CAM_HEIGHT = 480
WRIST_CAM_FPS = 30

MODEL_ID = "paultr/first-orchard-smolvla-run2"
TASK_TEMPLATE = "Place the {} fruit in the box"

model = None
preprocess = None
postprocess = None
dataset_features = None

Next, let's refine the init() function. This version loads the model into GPU memory and sets up the pre- and post-processors that bridge the gap between raw camera pixels and the model's action outputs.

def init():
    global device, robot, model, preprocess, postprocess, dataset_features
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    model = SmolVLAPolicy.from_pretrained(MODEL_ID)
    model.to(device)

    preprocess, postprocess = make_pre_post_processors(
        model.config,
        MODEL_ID,
        preprocessor_overrides={"device_processor": {"device": str(device)}},
    )

    camera_config = {
        "camera1": OpenCVCameraConfig(
            index_or_path=TOP_CAM_INDEX, width=TOP_CAM_WIDTH, height=TOP_CAM_HEIGHT, fps=TOP_CAM_FPS),
        "camera2": OpenCVCameraConfig(
            index_or_path=WRIST_CAM_INDEX, width=WRIST_CAM_WIDTH, height=WRIST_CAM_HEIGHT, fps=WRIST_CAM_FPS),
    }

    print("Connecting to Robot...")
    robot_cfg = SO101FollowerConfig(port=FOLLOWER_PORT, id=FOLLOWER_ID, cameras=camera_config)
    robot = SO101Follower(robot_cfg)
    robot.connect()

    home()

    # Map hardware features to dataset features for the VLA
    action_features = hw_to_dataset_features(robot.action_features, "action")
    obs_features = hw_to_dataset_features(robot.observation_features, "observation")
    dataset_features = {**action_features, **obs_features}

Executing the Action

Finally, we add run_task(). This is the core VLA inference loop. It takes a color (red, blue, green, or yellow), formats it into the task template, and asks the model to generate robot actions step-by-step until the harvest is complete.

def run_task(color):
    for i in range(MAX_STEPS):
        obs = robot.get_observation()

        # Build the frame for the model to "see" and "read" the task
        obs_frame = build_inference_frame(
            observation=obs,
            ds_features=dataset_features,
            device=device,
            task=TASK_TEMPLATE.format(color),
            robot_type=ROBOT_TYPE
        )

        processed_obs = preprocess(obs_frame)
        action = model.select_action(processed_obs)
        action = postprocess(action)

        # Convert model output back into physical motor commands
        robot_action = make_robot_action(action, dataset_features)
        robot.send_action(robot_action)

Building a Game Loop with Gemini

Now that the foundational infrastructure is in place, it’s time to actually play the game. First Orchard turns are driven by a speciality die, so we’ll start by simulating that in code.

def roll_dice():
    outcomes = {1: 'crow', 2: 'blue', 3: 'red', 4: 'green', 5: 'yellow', 6: 'basket'}
    res = outcomes[random.randint(1, 6)]
    print(f"Dice Roll: {res}")
    return res

Next, we update the main() function to react to these outcomes. If we roll a color (2-5), we call our VLA task. If we roll a crow or a basket, we’ll need some specialized logic.

def main():
    init()
    try:
        while True:
            home()

            result = roll_dice()

            if result == 'crow':
                move_crow()
            elif result == 'basket':
                basket_rolled()
            else:
                run_task(result)
    except KeyboardInterrupt:
        print("\nStopping game...")
    finally:
        if robot:
            robot.disconnect()

Adding the "Brain": Gemini 3 Flash

For the crow and basket options to work, the robot needs to do some higher-level reasoning. For example, if the crow moves to the final tile, the game is over. If a basket is rolled, the robot needs to look at the board and decide which fruit color is the most "urgent" to harvest.

To handle this, I brought in Gemini 3 Flash.

from google import genai
from google.genai import types

GEMINI_MODEL_NAME = "gemini-3-flash-preview"
GENAI_CLIENT = genai.Client()

Note: You'll need to install the dependency via pip install -q -U google-genai and export your API key with export GEMINI_API_KEY="your_key". This is the step I almost always forget, so double-check your environment variables before running!

Defining the Rules

To get accurate results, I provided Gemini with a detailed set of system instructions and prompts. This helps the model distinguish between the wooden fruit pieces, the circular tree tiles, and the robot arm itself.

SYSTEM_INSTRUCTIONS = "You will receive images from a top-down perspective of a simple game called First Orchard. There are four circular tiles that can each have between 0 and 4 game pieces shaped like a fruit with the colors green, blue, red, and yellow. All fruit of the same color will be grouped together on their own circle tile. There is also a line of tiles representing a path with a crow game piece on one of the tiles at all times. The end of that path is represented by the tile with a rounded end. There will be a blue robot arm on the edge of the image that should not be relevant to the tasks you will be asked to perform. There is also a box beside that robot arm that may be empty or may contain fruit game pieces. Fruit game pieces in the box are considered out of play and should be ignored during counting operations."

BASKET_PROMPT = "Locate the game pieces shaped like red, blue, green, and yellow fruit. Ignore any that are in the box besides the robot. Count how many pieces are in play per group. You should return the 'color' of whichever grouping has the most pieces. Select at random if there is a tie for the most pieces in a group."

Structured Output & Agentic Vision

To make the response easy to handle in Python, I’m using Gemini's Structured Output feature. Instead of parsing messy strings, Gemini will return a clean object based on a Pydantic model.

class BasketColor(BaseModel):
    color: str = Field(description="Color of the group with the most pieces in play.")

Now we can implement basket_rolled(). You’ll notice two key settings here:

Thinking Level: For simple perception, I keep this at low to save on latency.
ToolCodeExecution: This enables Agentic Vision. Gemini can actually "zoom in" and crop the image to get a better look at the tiles before it makes a decision. This is perfect for correctly counting small wooden fruits.

def basket_rolled():
    image = get_image_from_robot()

    response = GENAI_CLIENT.models.generate_content(
        model=GEMINI_MODEL_NAME,
        contents=[BASKET_PROMPT, image],
        config=types.GenerateContentConfig(
            system_instruction=SYSTEM_INSTRUCTIONS,
            response_mime_type="application/json",
            response_json_schema=BasketColor.model_json_schema(),
            thinking_config=types.ThinkingConfig(include_thoughts=False, thinking_level="low"),
            tools=[types.Tool(code_execution=types.ToolCodeExecution())]
        )
    )

    basket_data = BasketColor.model_validate_json(response.text)
    run_task(basket_data.color)

Connecting the Eyes

To send an image to Gemini, we need a way to grab a frame from the robot's cameras. Since the VLA is already using them, we can just pull the latest observation from the camera1 (overhead) feed and convert it into a format Gemini understands.

def get_image_from_robot():
    obs = robot.get_observation()
    pixel_values = obs["camera1"]

    if torch.is_tensor(pixel_values):
        img_np = pixel_values.to(torch.uint8).cpu().numpy()
    else:
        img_np = pixel_values.astype(np.uint8)

    # Ensure the dimensions are correct for PIL (H, W, C)
    if img_np.ndim == 3:
        if img_np.shape[0] == 3:
            img_np = np.transpose(img_np, (1, 2, 0))

    return Image.fromarray(img_np)

Finally, we handle the crow. Because the crow's path was out of reach for my specific robot setup, I opted for a "human-in-the-loop" approach.

def move_crow():
    print("ACTION REQUIRED: Move the crow one step forward manually!")
    time.sleep(10)

End Game Conditions

At this point, the robot can move fruit and handle the die, but it doesn't actually know when the game is over. If the trees are empty or the crow reaches the final tile, the loop would just keep running. To fix this, we need Gemini to act as the referee after every turn.

First, let's define a new prompt and a Pydantic model to handle the game's win/loss state.

GAME_END_PROMPT = "Analyze the image to determine the current state of the game. Look at the path tiles: if the crow piece is on the final tile (rounded edge with 'HABA'), the game is lost (result: 'crow'). Look at the four circular fruit tiles: if all fruit pieces have been removed from these tiles, ignoring any in the box, the game is won (result: 'empty'). If neither of these conditions is met, the game continues (result: 'continue'). Return the corresponding string result. Zoom in and crop on the path tiles and circular tiles to get a clearer view if needed."

class GameState(BaseModel):
    status: str = Field(description="The current state of the game: 'crow', 'empty', or 'continue'.")

Now, we can implement check_game_state(). This function asks Gemini to look at the overhead view and tell us if we should keep playing or stop.

def check_game_state():
    print("Checking game state...")
    image = get_image_from_robot()

    response = GENAI_CLIENT.models.generate_content(
        model=GEMINI_MODEL_NAME,
        contents=[GAME_END_PROMPT, image],
        config=types.GenerateContentConfig(
            system_instruction=SYSTEM_INSTRUCTIONS,
            response_mime_type="application/json",
            response_json_schema=GameState.model_json_schema(),
            thinking_config=types.ThinkingConfig(include_thoughts=False, thinking_level="low"),
            tools=[types.Tool(code_execution=types.ToolCodeExecution())]
        )
    )

    game_data = GameState.model_validate_json(response.text)

    if game_data.status == 'crow':
        print("GAME OVER: The crow has reached the orchard!")
        sys.exit(0)
    elif game_data.status == 'empty':
        print("GAME OVER: You win! All fruits are harvested.")
        sys.exit(0)

Optimizing the Game Flow

Calling Gemini every single turn is effective, but we can be a bit smarter about it. For instance, if the robot rolls "Red" but we already know the red tree is empty, it’s a wasted turn—and a wasted API call if we ask Gemini to check the whole board.

To optimize this, I introduced a should_check_game_state flag and a set called EMPTY_FRUITS to cache the state of the board locally.

should_check_game_state = True
EMPTY_FRUITS = set()

# We update run_task and move_crow to set should_check_game_state = True 
# whenever a physical change happens on the board.

Next, I added check_fruit_exists(). This function specifically checks if a rolled color is still available on the board. If Gemini sees that a tree is empty, we add that color to our EMPTY_FRUITS set so we can skip the check next time that color is rolled.

FRUIT_CHECK_PROMPT = "Look at the four circular tiles. Is there at least one {} fruit piece remaining on its designated tile? Ignore pieces inside the box. Return True if at least one is present, otherwise return False. Crop and zoom in on the individual circular tiles to help ignore the box and correctly identify the presence of fruit game pieces."

class FruitExists(BaseModel):
    exists: bool = Field(description="True if the specified fruit color is still on the circular tiles, False otherwise.")

def check_fruit_exists(color):
    if color in EMPTY_FRUITS:
        print(f"Skipping: {color} fruit is already gone.")
        return False

    image = get_image_from_robot()
    print(f"Gemini: Checking if {color} fruit is available...")

    response = GENAI_CLIENT.models.generate_content(
        model=GEMINI_MODEL_NAME,
        contents=[FRUIT_CHECK_PROMPT.format(color), image],
        config=types.GenerateContentConfig(
            system_instruction=SYSTEM_INSTRUCTIONS,
            response_mime_type="application/json",
            response_json_schema=FruitExists.model_json_schema(),
            tools=[types.Tool(code_execution=types.ToolCodeExecution())]
        )
    )

    fruit_data = FruitExists.model_validate_json(response.text)
    if not fruit_data.exists:
        EMPTY_FRUITS.add(color)

    return fruit_data.exists

The Final Loop

Finally, we update the main() loop to tie all these pieces together. The robot now homes itself, verifies the game isn't over, rolls the die, and uses Gemini to confirm the move is valid before handing control over to the VLA for the physical harvest.

def main():
    init()
    try:
        while True:
            home()
            check_game_state()

            result = roll_dice()

            if result == 'crow':
                move_crow()
            elif result == 'basket':
                basket_rolled()
            else:
                if check_fruit_exists(result):
                    run_task(result)
                else:
                    continue
    except KeyboardInterrupt:
        print("\nStopping game...")
    finally:
        if robot:
            robot.disconnect()

if __name__ == "__main__":
    main()

Future Explorations: Beyond "Done"

While I’m very happy with how this project turned out, I’m a big believer in the saying, "Perfection is the enemy of done." In robotics, you could spend a lifetime tweaking a single movement, so I’ve decided to treat this as a successful baseline and keep a "wishlist" of things I’d love to explore in future iterations.

Dynamic Termination: Currently, the VLA runs for a fixed number of steps. I really want to update the loop so the robot stops the moment the task is complete. I have some ideas about using the Gemini Live API to provide real-time visual feedback to the robot.
Exploring VLA Architectures: I want to experiment with additional VLA models and action techniques to see how they compare. While SmolVLA was a great starting point, I’m curious to revisit the Pi0-Fast model or explore ACT (Action Chunking with Transformers). Documentation for these in the LeRobot ecosystem is still relatively sparse, and I think getting more successful implementations out there would be a general win for the robotics community.
Local State Tracking: The game relies heavily on Gemini to understand the board state. If the VLA is working successfully and the robot isn't crashing (admittedly big "ifs"!), I’d like to modify the code to track the fruit count locally. This would allow me to only trigger a Gemini "referee" call when a count hits zero or the crow is moved, making the game much faster.

Final Thoughts

At the end of the day, this was an experiment with toys, but the problems I ran into like data scarcity, hardware failures, and model inference quirks, are the same ones facing "real" robotics engineers every day. There isn't anything drastic I’d change about this project as I think it served its purpose as a learning project very well.

I’m looking forward to taking what I learned here and applying it to my next project, so hopefully I catch you there too.

Thanks for following

Video Understanding with Gemini 3.0 Flash for Robotics

Paul Ruiz — Thu, 05 Feb 2026 15:01:36 +0000

In the robotics field, as with nearly every other corner of tech, the landscape is shifting rapidly as we integrate AI into our workflows and systems. For this post, I’ve put together a few demos to explore Gemini's multimodal capabilities for video understanding. We’ll look at how these features can be applied to robotics-specific use cases, as well as how they can be used for general learning augmentation.

If you're not familiar with Gemini yet, here's a quick start for running a basic "Hello World" example. Be sure to set your GOOGLE_GEMINI_API variable so that your API key can be found (I forget this step pretty much every time :)).

With that, let's jump in.

Analyzing a local file: Video to Action

In this first example, I have a video file saved locally that shows a bi-arm Aloha robot doing various tasks on a desk (if you'd like to use the same video to follow along, you can find it here).

If I wanted to make this sequence repeatable with a Vision-Language-Action (VLA) model, I’d first need to break the video down into subtasks. I wrote a small program that reviews the video and returns a structured list of actions, identifying the "actor" and the specific task for each segment.

from google import genai
from google.genai import types

import time
import json
import pandas as pd
import plotly.express as px
from datetime import timedelta


client = genai.Client()

myfile = client.files.upload(file="desk_organization.mp4")

while myfile.state == "PROCESSING":
  print(".", end="")
  time.sleep(1)
  myfile = client.files.get(name=myfile.name)

if myfile.state.name == "FAILED":
  raise ValueError(myfile.state.name)

print("Processed")


prompt = """
Review this video and break the actions into a structured JSON list. 
Each object in the list must have:
- "actor": The entity performing the action (e.g., 'Left Robot Arm').
- "action": A short description of the task.
- "start_s": Start time in total seconds (integer).
- "end_s": End time in total seconds (integer).

Output ONLY the raw JSON list.
"""

response = client.models.generate_content(
    model="gemini-3-flash-preview", 
    contents=[myfile, prompt],
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        thinking_config=types.ThinkingConfig(thinking_budget=-1)
    ),
)

print(response.text)

Let's break this into smaller parts. First, I use Gemini's Files API to upload the video. The client.files.upload command used here will block the script until the file has been uploaded, and then the begins processing the file for use. So we don't accidentally try to access the file before it's ready, which will cause an error, I also have a loop that checks the status of the file before letting the program continue.

myfile = client.files.upload(file="desk_organization.mp4")

while myfile.state == "PROCESSING":
  print(".", end="")
  time.sleep(1)
  myfile = client.files.get(name=myfile.name)

if myfile.state.name == "FAILED":
  raise ValueError(myfile.state.name)

Next I have a very specific prompt about how I want the data to be returned to me so that I can get the actor, action, and starting and ending time for that action. I also use the response_mime_type flag to specify that I only want JSON data returned to me.

prompt = """
Review this video and break the actions into a structured JSON list. 
Each object in the list must have:
- "actor": The entity performing the action (e.g., 'Left Robot Arm').
- "action": A short description of the task.
- "start_s": Start time in total seconds (integer).
- "end_s": End time in total seconds (integer).

Output ONLY the raw JSON list.
"""

response = client.models.generate_content(
    model="gemini-3-flash-preview", 
    contents=[myfile, prompt],
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        thinking_config=types.ThinkingConfig(thinking_budget=-1)
    ),
)

At this point we can get the JSON data outlining the actions that the robots have taken, which can then be used to prompt a VLA to repeat the tasks.

[
  {
    "actor": "Left Robot Arm",
    "action": "pick up the green marker",
    "start_s": 0,
    "end_s": 3
  },
  {
    "actor": "Left Robot Arm",
    "action": "place the green marker in the wooden bowl",
    "start_s": 3,
    "end_s": 6
  },
  {
    "actor": "Left Robot Arm",
    "action": "pick up the blue pen",
    "start_s": 13,
    "end_s": 16
  },
  {
    "actor": "Left Robot Arm",
    "action": "place the blue pen in the pencil holder",
    "start_s": 18,
    "end_s": 22
  },
  {
    "actor": "Right Robot Arm",
    "action": "pick up the red pen",
    "start_s": 22,
    "end_s": 25
  },
  {
    "actor": "Right Robot Arm",
    "action": "place the red pen in the pencil holder",
    "start_s": 25,
    "end_s": 28
  }
]

Structured data is great for code, but for humans, visuals are better. I asked Gemini to write a script to turn that JSON into a Gantt chart using Plotly. This makes it easy to see the task orchestration and timestamps at a glance.

data = json.loads(response.text)
df = pd.DataFrame(data)

base_time = pd.to_datetime("2025-01-01")
df['start_dt'] = df['start_s'].apply(lambda x: base_time + timedelta(seconds=x))
df['end_dt'] = df['end_s'].apply(lambda x: base_time + timedelta(seconds=x))

dynamic_height = 150 + (len(df['actor'].unique()) * 60)

fig = px.timeline(
    df, 
    x_start="start_dt", 
    x_end="end_dt", 
    y="actor", 
    color="actor",
    text="action",
    template="plotly_white",
    height=dynamic_height
)

fig.update_layout(
    title_text="Video Orchestration",
    title_x=0.5,
    showlegend=False,
    margin=dict(l=10, r=10, t=40, b=30),
    xaxis_title=None,
    yaxis_title=None,
    font=dict(size=11)
)

fig.layout.xaxis.update({
    'tickformat': '%M:%S',
    'fixedrange': True
})

fig.update_yaxes(autorange="reversed", fixedrange=True)
fig.update_traces(
    textposition='inside', 
    insidetextanchor='middle',
    marker_line_width=1,
    marker_line_color="white",
    width=0.6
)

fig.show()

This gives us a clean, visual breakdown of the robot's performance:

Understanding videos from YouTube

For my next experiment, I took some inspiration from this paper on Mobility VLA. I wanted to try using a long YouTube video of a tour (specifically one from the former Egyptian Museum in Cairo because it was one of my favorites and I generally just loved Egypt the few times I've been), and then ask questions about things seen in that video.

Since I could potentially do this on a robot with its own mobility stack where I could return to a location based on a time stamp, I figured I'd also ask for a timestamp for when the video shows the largest item on the tour.

Luckily, the code for this is really straightforward. You just need to provide a YouTube link as a file_data object and send in your prompt.

from google import genai
from google.genai import types

client = genai.Client()
response = client.models.generate_content(
    model='models/gemini-3-flash-preview',
    contents=types.Content(
        parts=[
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=EdCReWs6-wI')
            ),
            types.Part(text='Please summarize the different things seen in this video and provide a timestamp for the location where the largest object is seen.')
        ]
    ),
)

print(response.text)

This will give you a response like:

The Egyptian Museum, also known as the Museum of Ancient Egyptian Antiquities, is a renowned institution in Cairo, Egypt. It is home to a vast and priceless collection of ancient Egyptian artifacts, including the world-famous treasures of Tutankhamun. The video displays various things like:

* **Museum exterior:** The video begins with an exterior shot of the museum at night.
* **Sarcophagi:** There are many sarcophagi of different sizes and materials, including stone, granite, and wood.
* **Statues:** The museum houses a wide range of statues representing pharaohs, gods, goddesses, and everyday people.
* **Wooden boats:** Ancient Egyptian wooden boats used for burial rituals are on display.
* **Display cases:** Many of the museum's smaller artifacts, such as jewelry, amulets, and pottery, are shown in display cases.

The largest object is seen at [05:54](https://www.youtube.com/watch?v=EdCReWs6-wI&t=352).

OK, so one downside: this isn't really useful if I want to use the information in my robot code, as I'd have to do extra work to extract the timestamp. but it turns out there's a more organized way to do this using a Gemini capability called structured outputs.

To get the data back in a way I can use it, I'll still request a JSON mime type, but I'll also create some objects representing the data I want:

class ItemSeen(BaseModel):
    object: str = Field(description="Object seen in the video")
    description: str = Field(description="Description of the object seen in the video")

class Navigation(BaseModel):
    itemsSeen: List[ItemSeen]
    timestamp: int = Field(description="Timestamp where the largest item is seen")

Then I'll use those objects as a schema for the data by adding a config object to my generate_content call.

response = client.models.generate_content(
    model='models/gemini-3-flash-preview',
    contents=types.Content(
        parts=[
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=EdCReWs6-wI')
            ),
            types.Part(text='Please summarize the different things seen in this video and provide a timestamp for the location where the largest object is seen.')
        ]
    ),
    config={
        "response_mime_type": "application/json",
        "response_json_schema": Navigation.model_json_schema(),
    }
)

At this point I can extract the data returned into the model object and use it in my app. The output will look like this:

itemsSeen = [
ItemSeen(object='Pharaohs and Queens statues', description='Statues depicting different Pharaohs and Queens, carved from various materials such as stone and wood, showcasing traditional poses and royal regalia.'),
ItemSeen(object='Sarcophagi and mummy cases', description='Ornate containers used to hold mummies, including stone sarcophagi and wooden mummy cases adorned with intricate hieroglyphs and religious imagery.'),
ItemSeen(object='Animal-headed deities', description='Statues of gods and goddesses represented with animal heads, like the jackal-headed Anubis, falcon-headed Horus, and lioness-headed Sekhmet.'),
ItemSeen(object='Pyramidions', description='Small pyramid-shaped stones, often made of basalt or granite, that once capped the tops of pyramids or obelisks, inscribed with prayers and scenes.'),
ItemSeen(object='Ancient wooden boat', description='A well-preserved funerary boat made of wood, reconstructed to show how these vessels were used for symbolic journeys in the afterlife.'),
ItemSeen(object='Reliefs and stelae', description='Stone slabs and wall segments featuring carved or painted scenes and inscriptions, documenting the lives, achievements, and religious beliefs of the ancient Egyptians.'),
ItemSeen(object='Display cases with artifacts', description='Glass-enclosed cases containing smaller items such as jewelry, figurines, tools, and household objects, providing a glimpse into daily life and craftsmanship.')
]
timestamp = 361

Querying multiple YouTube videos

So now that we know how to query against a YouTube video, let's take this a step further. With an API key that's tied to a paid account, I can run Gemini over 10 different YouTube videos with one call. This could be great for analyzing multiple camera feeds to check for task success, but since I don't have that data on hand, I'm going to focus on the fact that this field requires a lot of constant learning and I need all of the help I can get.

For this example I'll load six robotics lectures from Stanford University (I have no affiliation with them or this content, I just really like and value free educational content) and ask Gemini to create some concise notes, as well as use Google Search to look up and give me a recommended reading list to support my robotics learning journey.

from google import genai
from google.genai import types

client = genai.Client()

google_search_tool = types.Tool(
    google_search=types.GoogleSearch()
)

response = client.models.generate_content(
    model='models/gemini-3-flash-preview',
    contents=types.Content(
        parts=[
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=o5bW3C5OD6U'),
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=PYh9k4cy25w'),
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=RKFRO_G4YkA'),
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=v18Jo2ILXZ8'),
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=5uWtpDON7Vs'),
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=05SuBLowwKM'),
            ),
            types.Part(text='This is a set of six lectures for a robotics course. Please write concise notes for each video in markdown, then create a list of research papers and books that would be relevant to each course. Check reviews for books that you provide and mention why they would be worth reading to learn this material in depth.')
        ]
    ),
    config=types.GenerateContentConfig(
        tools=[google_search_tool],
    ),
)

print(response.text)

Now if we run this we get... an error:

google.genai.errors.ClientError: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Please use fewer than 10800 images in your request to this model', 'status': 'INVALID_ARGUMENT'}}

What's happening here is that Gemini 3.0-Flash's context window supports a maximum of 10,800 images. At 1 frame per second, that's three hours of video, and the six long lectures exceed this limit.

To get around this, we can play around with some video settings since these are lectures that don't have a ton changing with each frame and aren't graphics-intensive. For starters, let's add a media_resolution property to the config object to convert the video into a low resolution video when processing. You can find more about this property in the official documentation here.

We can also lower the frame rate that gets processed on each video, so in this case we'll only look at one frame every ten seconds rather than one frame per second.

In the end we land on a simple script that looks like this:

from google import genai
from google.genai import types

client = genai.Client()

google_search_tool = types.Tool(
    google_search=types.GoogleSearch()
)


response = client.models.generate_content(
    model='models/gemini-3-flash-preview',
    contents=types.Content(
        parts=[
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=o5bW3C5OD6U'),
                video_metadata=types.VideoMetadata(fps=0.1),
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=PYh9k4cy25w'),
                video_metadata=types.VideoMetadata(fps=0.1)
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=RKFRO_G4YkA'),
                video_metadata=types.VideoMetadata(fps=0.1)
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=v18Jo2ILXZ8'),
                video_metadata=types.VideoMetadata(fps=0.1)
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=5uWtpDON7Vs'),
                video_metadata=types.VideoMetadata(fps=0.1)
            ),
            types.Part(
                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=05SuBLowwKM'),
                video_metadata=types.VideoMetadata(fps=0.1)
            ),
            types.Part(text='This is a set of six lectures for a robotics course. Please write concise notes for each video in markdown, then create a list of research papers and books that would be relevant to each course. Check reviews for books that you provide and mention why they would be worth reading to learn this material in depth.')
        ]
    ),
    config=types.GenerateContentConfig(
        tools=[google_search_tool],
        media_resolution=types.MediaResolution.MEDIA_RESOLUTION_LOW
    ),
)

print(response.text)

which in turn gives us the following output:

Here are concise markdown notes for the six lectures featured in this robotics seminar, followed by curated lists of research papers and books to deepen your understanding of each specific domain.

---

# Lecture 1: Autonomous Navigation in Complex Outdoor Environments
**Speaker:** Jing Liang (Stanford/UMD)

### Concise Notes
*   **Problem Definition:** Moving beyond obstacle avoidance to "traversability analysis"—understanding which surfaces (grass, gravel, etc.) a robot can actually handle.
*   **VLM Integration:** Utilizing Vision-Language Models (VLMs) to translate natural language goals and visual cues into viable path Candidates.
*   **Gaussian Splats for Mapping:** Implementing 3D Gaussian Splatting to estimate not just geometry, but semantic material types and physical properties (friction, hardness, density).
*   **Companion Robotics:** Applying these navigation stacks to older adults ("longevity robots") to assist with outdoor exercise and health monitoring.
*   **Dataset:** Introduction of the Global Navigation Dataset (GND) covering 10 campuses with multi-modal sensor data.

### Relevant Resources
**Research Papers:**
*   *MaPNav: Trajectory Generator with Traversability Coverage for Outdoor Navigation* (Liang et al., 2024).
*   *SplatFlow: Traversability-Aware Gaussian Splatting for Outdoor Robot Navigation* (Chopra et al., 2024).

**Books:**
*   **"Probabilistic Robotics" by Sebastian Thrun, Wolfram Burgard, and Dieter Fox.**
    *   *Review:* Regarded as the "bible" of modern navigation. It is essential for understanding the SLAM and state estimation fundamentals that Jing Liang’s complex environment navigation is built upon.

---

# Lecture 2: From Digital Humans to Safe Humanoids
**Speaker:** Yao Feng (Stanford)

### Concise Notes
*   **GentleHumanoid Framework:** Focuses on safe physical contact. Instead of just following a path, the robot must regulate interaction forces.
*   **Force Modeling:** Differentiates between *Resistive Contact* (robot hitting an object) and *Guiding Contact* (human pulling the robot’s hand).
*   **Tunable Force:** Implements a safety threshold (e.g., limiting force to 5N or 15N) to ensure the robot "gives way" during a hug or assistance task.
*   **Grounded Reasoning:** Uses "ChatPose" and "ChatHuman" to allow robots to predict human intent and next-frame poses from visual/textual data.

### Relevant Resources
**Research Papers:**
*   *GentleHumanoid: Learning Upper-body Compliance for Contact-rich Human and Object Interaction* (Lu et al., 2024).
*   *ChatPose: Chatting about 3D Human Pose* (Feng et al., 2024).

**Books:**
*   **"Humanoid Robotics" by Shuuji Kajita et al.**
    *   *Review:* A comprehensive guide to the kinematics and dynamics of two-legged machines. It helps bridge the gap between Feng’s "digital humans" and the physical constraints of a humanoid.

---

# Lecture 3: Resilient Autonomy in Extreme Environments
**Speaker:** Sebastian Scherer (Carnegie Mellon University)

### Concise Notes
*   **Definition of Resilience:** The ability to maintain performance in "degraded" conditions (smoke, dust, total darkness, or GPS-denied areas).
*   **MapAnything:** A unified feed-forward model that performs 3D reconstruction and metric-scale estimation from simple monocular video.
*   **Multi-Modal Sensors:** Leveraging thermal cameras (AnyThermal) and Doppler radar to navigate when visual cameras fail due to glare or darkness.
*   **Triage Challenge:** Application of these robots as "pre-first responders" to locate and assess casualties in disaster zones autonomously.

### Relevant Resources
**Research Papers:**
*   *MapAnything: Unified 3D Reconstruction from any Visual Input* (Scherer Lab, 2024).
*   *AnyThermal: A Single Backbone for Multiple Thermal Perception Tasks* (Li et al., 2024).


**Books:**
*   **"Autonomous Mobile Robots" by Roland Siegwart and Illah Nourbakhsh.**
    *   *Review:* Excellent for learning about the trade-offs between different sensor modalities (Lidar vs. Vision vs. Thermal), which is the core of Scherer's "resilience" strategy.

---4.  **Lecture 4: Robot Motion Learning with Physics-Based PDE Priors**
**Speaker:** Abdul H. Qureshi (Purdue University)

### Concise Notes
*   **Neural Time Fields (NTFields):** Using neural networks to solve the *Eikonal Partial Differential Equation (PDE)* for motion planning.
*   **TD-Learning for Motion:** Borrowing Temporal Difference learning from RL to regularize gradients between consecutive points in a path.
*   **Scalability:** This approach allows for planning in extremely high-dimensional spaces (up to 15-DOF for quadrupeds and arms) much faster than traditional sampling-based planners.
*   **Unknown Environments:** The robot builds a "Time Field" map in real-time, treating navigation as following the gradient of arrival time.

### Relevant Resources
**Research Papers:**
*   *Physics-Informed Neural Time Fields for Motion Planning* (Ni et al., 2023).
*   *Domain Decomposition for Large Scale Neural Motion Planning* (Liu et al., 2024).

**Books:**
*   **"Planning Algorithms" by Steven M. LaValle.**
    *   *Review:* This is the definitive text on how robots find paths. Reading this is necessary to understand why Qureshi’s use of PDEs is such a radical and efficient departure from traditional RRT* or PRM methods.

---

# Lecture 5: Learning to Control Large Teams of Robots
**Speaker:** Eduardo Montijano (University of Zaragoza)

### Concise Notes
*   **Distributed Policies:** Moving away from a "central brain" to local policies where each agent makes decisions based on its immediate neighbors.
*   **Self-Attention Swarms:** Utilizing Transformer-like attention mechanisms so robots can handle a varying number of neighbors (scaling from 3 to 3,000 robots).
*   **Port-Hamiltonian Systems:** Integrating energy-based physics equations into the neural network to ensure the learned behavior is physically stable and explainable.
*   **Gen-Swarms:** Applying Generative AI (Diffusion models) to "draw" complex shapes (like a dragon) for drone shows, then using local controllers to achieve those shapes.

### Relevant Resources
**Research Papers:**
*   *LEMURS: Learning Distributed Multi-Robot Interactions* (Sebastián et al., 2023).
*   *Gen-Swarms: Generative AI for Swarm Robotics* (Pueyo et al., 2024).

**Books:**
*   **"Graph Theoretic Methods in Multiagent Networks" by Mehran Mesbahi and Magnus Egerstedt.**
    *   *Review:* Essential for understanding how local connectivity influences global swarm behavior. It provides the mathematical logic for the "neighborhood" approach discussed by Montijano.

---

# Lecture 6: Next Generation Dexterous Manipulation
**Speaker:** Monroe Kennedy III (Stanford)

### Concise Notes
*   **The Manipulation Gap:** While robots can walk and flip, they still struggle with small, soft, or articulated objects (like tying shoelaces).
*   **Optical Tactile Sensors (DenseTact):** Using internal cameras within soft "fingertips" to sense 4-axis stress fields (normal, shear, and torsion).
*   **J-PARSE Algorithm:** Resolving "kinematic singularities" (when a robot arm gets stuck at full extension) through a safety Jacobian projection.
*   **Cross-Modality Learning:** Combining vision and touch (Touch-GS) to create 3D Gaussian Splats of objects that are otherwise invisible to cameras (transparent or highly reflective surfaces).

### Relevant Resources
**Research Papers:**
*   *DenseTact 2.0: High-Resolution Tactile Sensing for Robot Manipulation* (Do et al., 2023).
*   *Touch-GS: Visual-Tactile Supervised 3D Gaussian Splatting* (Swann et al., 2024).

**Books:**
*   **"Mechanics of Manipulation" by Matthew T. Mason.**
    *   *Review:* This book focuses on the physics of pushing, grasping, and friction. It is vital for understanding the "force closure" and "form closure" concepts Kennedy uses to move beyond simple suction-cup grippers.

The output gives us concise notes for all lectures and curated reading lists. Having recently read one of the recommended books (Introduction to Autonomous Mobile Robotics by Siegwart and Nourbakhsh), I can say that I think the recommendations seem solid.

Conclusion

This was a look into a newer capability for Gemini that you can try today with the gemini-3.0-flash-preview model. If you found it interesting or have other things you'd want to know about, definitely leave a comment.