DEV Community: Kevin Lu

Voice-first PC Builder Agent Built on Gemini Live API

Kevin Lu — Mon, 16 Mar 2026 23:10:20 +0000

This post was written for my submission to the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

Inspiration

I've built PCs before and know the pain firsthand from squinting at tiny connectors, pausing YouTube every 10 seconds with greasy fingers, and second-guessing whether you're about to break your build. But the real breaking point was trying to help a friend build their first PC remotely over a video call. I'm watching a shaky camera feed, trying to describe which cable goes where, and they can't find the front panel headers or find the parts I'm referencing unless I pull up pictures. I realized what they needed wasn't me on a video call, it was an AI that could see what they see, talk them through it hands-free, and not lose patience on the 15th, "which one is the 8-pin?"

What it does

BuildBuddy is a hands-free, voice-first AI assistant that guides users through a PC build in real time. You talk to it, it talks back, no typing, no pausing, no touching your phone with thermal-paste fingers. Point your camera at a part or connector you don't recognize, and it identifies it visually. A built-in parts reference shows labeled diagrams for tricky connectors like PSU cables. It tracks your build progress step-by-step and logs every action with timestamps and camera snapshots to a shareable timeline so a friend, mentor, or forum can review exactly what happened and verify the AI's guidance. Because AI can be wrong, and accountability matters.

How I built it

Gemini Live API: for real-time bidirectional audio streaming, the user speaks, the AI responds with voice, all over a persistent WebSocket connection
Google ADK (Agent Development Kit): for agent orchestration, tool definitions, and session management
Custom tool calls: for build progress tracking (update_part_status), connector image references (get_connector_image, show_user_part), enabling the AI to trigger UI updates mid-conversation
Cloud Run: for deployment with session affinity to maintain WebSocket connections
Cloud Build: for automated CI/CD, pushes to the repo trigger a Docker image build and deployment to Cloud Run without manual steps
Firestore: for logging every build event with timestamps, part status, and notes
Google Cloud Storage: for storing camera snapshots at each build step
Vanilla HTML/CSS/JS frontend: for no frameworks, mobile-first, designed for one-handed use with a phone propped up next to your build

The ToolContext state persistence across conversation turns is what makes it a stateful agent vs. a stateless function caller.

Here's what that looks like:

# app/tools/update_part_status.py lines 23-60
def update_part_status(
    tool_context: ToolContext,
    part_id: str,
    status: Literal["NOT_STARTED", "IN_PROGRESS", "DONE", "BLOCKED"],
    notes: str = "",
) -> dict:
    build = tool_context.state.get("build_progress", {})
    build[part_id]["status"] = status
    build[part_id]["notes"] = notes
    tool_context.state["last_updated_part"] = part_id
    tool_context.state["build_progress"] = build
    ...

Since ADK's after_tool_callback only receives tool_context, we store last_updated_part as a state breadcrumb so the Firestore logger knows which delta to snapshot without storing the full build every time.

Challenges I ran into

Real-time audio was the hardest part of the entire project. The PCM microphone recorder would flood the WebSocket with audio buffer data, causing crashes, and Nvidia Broadcast made it worse with suspected buffer timing issues. Muting the mic after the speech and eventually bypassing Nvidia Broadcast almost resolved the symptoms, but it took real debugging time to isolate. On top of that, camera frames are constantly sent during tool call execution, which would interrupt the audio data flow, causing the WebSocket to lose connection entirely. The Gemini Live API and ADK expect very specific WebSocket data timing, so if tool calls and camera frames collide with the audio stream, everything falls apart. The fix required async waits and blocking mechanisms to prevent simultaneous data floods.

# Tool call websocket fix
# app/main.py ~lines 190-310
frames_allowed = asyncio.Event()
frames_allowed.set()

# downstream_task: lock when tool starts
if event.get_function_calls():
    frames_allowed.clear()  # LOCK realtime input

# unlock after turn completes + cooldown
if getattr(event, "turn_complete", False):
    await asyncio.sleep(1.5) # important in case of consec toolcalls
    frames_allowed.set()

# upstream_task: gate audio and camera behind the lock
if not frames_allowed.is_set():
    continue  # drop frame/audio while tool is running

We always send the camera feed at 1fps to Gemini, not at the raw capture rate from the frontend, and always send the most recent frame rather than queuing every frame.

Client sends every captured frame to the WebSocket
Server buffers only the latest in latest_image_blob, overwriting constantly
frame_injection_worker forwards to Gemini at a 1s interval, but only if the frame changed (tracked by bytes identity, not object identity)

# Camera feed
# app/main.py
async def frame_injection_worker():
    last_sent_data = None
    while True:
        await frames_allowed.wait() # blocked during tool calls
        await asyncio.sleep(1) # hard limit on 1fps in the backend
        if (latest_image_blob is not None
                and latest_image_blob.data is not last_sent_data):
            live_request_queue.send_realtime(latest_image_blob)
            last_sent_data = latest_image_blob.data

Audio buffering fix

let pcmBuffer = [];
let pcmBufferBytes = 0;

const flushTimer = setInterval(flushBuffer, SEND_INTERVAL_MS); // ~50ms

audioRecorderNode.port.onmessage = (event) => {
    const pcmData = convertFloat32ToPCM(event.data);
    pcmBuffer.push(pcmData); // accumulate
    pcmBufferBytes += pcmData.byteLength;
};

function flushBuffer() {
    if (pcmBufferBytes === 0) {
            return;
    }

    // accumulate
    const merged = new Uint8Array(pcmBufferBytes);
    let offset = 0;
    for (const chunk of pcmBuffer) {
        merged.set(new Uint8Array(chunk), offset);
        offset += chunk.byteLength;
    }

    // reset 
    pcmBuffer = [];
    pcmBufferBytes = 0;

    audioRecorderHandler(merged.buffer);
}

These fixes together, the async gate for camera feed and audio, and the PCM buffer stabilized the WebSocket connection.

Cloud Run deployment was its own adventure. The app failed immediately with a single concurrent instance because the frontend must query JavaScript and static files while the WebSocket maintains the connection. With one instance, we'd get rate-limited on our own requests. The solution was setting a minimum of one warm instance and a max of two, plus enabling session affinity to keep WebSocket connections stable. GCS bucket configuration for public image access was also non-obvious. Disabling the "prevent public access" setting isn't enough, and you need to explicitly grant the allUsers principal a Storage Object Viewer role. I would have preferred per-object public access, but the bucket's uniform access policy doesn't allow mixed permissions.

Finally, ADK documentation didn't always match the actual behavior of the current version, which meant a lot of trial-and-error to figure out how things actually worked versus how they were documented.

Accomplishments that we're proud of

The thing I'm most proud of is that it actually works end-to-end as a real-time voice agent. You can have a natural conversation with your hands full while building a PC, and the AI genuinely helps. The shareable build timeline turned out to be a surprisingly compelling feature. It reframes AI assistance from "trust the black box" to "here's a reviewable record of everything that happened."

The ADK before/after callbacks power the whole thing:

# app/tools/update_part_status.py
def before_tool_modifier(tool, args, tool_context):
    if tool.name == "update_part_status":
        b64, mime = get_latest_image()
        tool_context.state["pending_blob"] = {"data": b64, ...}

def after_tool_report_log(tool, args, tool_context, tool_response):
    if tool.name == "update_part_status":
        snapshot = _build_snapshot(tool, tool_context)

        # Phase 1: insert with gcs_url="PENDING"
        doc_id = _log_to_firestore(snapshot) 
        blob_url = _upload_blob_to_gcs(tool_context)

        # Phase 2: update with image url from gcs
        _update_firestore_record(doc_id, blob_url)

What I learned

This was my deep dive into Gemini's Live API and real-time multimodal streaming. The biggest takeaway is that voice-first interfaces have fundamentally different UX constraints than text-based ones, latency matters more than token count, interruption handling is critical, and you can't show a wall of text to someone whose hands are inside a PC case. I also learned that tool calls in a streaming audio context need careful state management to prevent race conditions between camera frames, audio chunks, and tool executions happening simultaneously. On the deployment side, Cloud Run's session affinity is non-negotiable for WebSocket applications, something that's obvious in hindsight but costs real debugging time.

What's next for BuildBuddy

The current version is scoped as a focused single-user experience, but for production:

Multi-user sessions: right now there's no session isolation. Adding proper authentication and per-session Firestore collections would let multiple people use BuildBuddy simultaneously, each with their own build timeline
AR connector overlay: use the camera feed to overlay labels directly on the motherboard showing exactly where each cable connects
Build templates: pre-loaded guides for popular builds so the AI has part-specific knowledge from the start, with compatibility checking to warn about mismatches before you start building
Motherboard manual RAG: use embeddings to index motherboard manuals so the AI can reference exact pin layouts, header locations, and BIOS settings specific to the user's board since the motherboard manual is the real source of truth for every build
Community review: let experienced builders comment on shared timelines to catch AI mistakes, suggest better cable routing, or help troubleshoot blocked steps
Multi-language support: the voice assistant should support more than just English
Noise robustness: the hackathon demo runs in a quiet space, but a real user has fans, tools, and background noise. Proper noise cancellation and wake-word detection would be essential
Production security: the hackathon version uses broad permissions and a public GCS bucket for convenience. In production, I'd use presigned URLs with expiration for snapshot access, lock down Firestore rules per user, and apply least-privilege IAM roles for each service account across Cloud Run, Firestore, and GCS

Devpost Submission | Demo Video

I Built a Bathroom Occupancy Notifier with Home Assistant

Kevin Lu — Wed, 04 Mar 2026 03:47:03 +0000

As a backend dev who runs his own network, NAS, and home lab, I kept getting interrupted mid-game and during meetings to check if the bathroom was free. Walking over to check is the whole problem, especially during moments when I can't afford to take time to step away. So I built a notifier in Home Assistant.

Now, anyone in the house opens the Home Assistant app, sees the live occupancy status, and taps one button to get pinged the moment the bathroom is free. Whoever is inside also gets alerted when someone is waiting, which was the original motivation.

What You Need

A door contact sensor paired to Home Assistant (Zigbee, Z-Wave, or WiFi, anything that exposes a binary_sensor)
input_boolean.guest_bathroom_queue created in Settings → Device & Services → Helpers → Toggle
Home Assistant Companion app installed on family devices for push notifications
A separate guest VLAN if you want the auto-login flow (optional)

How It Works

A door contact sensor (binary_sensor.contact_sensor_bathroom_door_intrusion) drives everything. "off" = door closed = occupied. "on" = door open = available.

Note: This can vary depending on the sensor model. Please verify and adjust the provided YAML accordingly.

One input_boolean.guest_bathroom_queue tracks whether anyone is waiting. I deliberately kept this as a shared boolean rather than tracking individual users. A per-user queue would expose who's in the bathroom and who's waiting to use it. Beyond privacy, it doesn't matter who is waiting, as long as someone is. This keeps state management much simpler.

Three automations handle the core logic.

1. Someone joins the queue

Triggers 3 seconds after guest_bathroom_queue flips on. A 3-second debounce filters out accidental taps.

alias: Notify users when guest is waiting for bathroom
triggers:
  - trigger: state
    entity_id: input_boolean.guest_bathroom_queue
    from: "off"
    to: "on"
    for:
      seconds: 3
actions:
  - action: notify.family_devices
    data:
      title: Bathroom
      message: someone is waiting for the bathroom
mode: single

2. Bathroom becomes free while someone is waiting

Fires when the door opens and guest_bathroom_queue is still on. Turns off the queue, then notifies.

alias: Notify Guest When Bathroom Free
triggers:
  - trigger: state
    entity_id: binary_sensor.contact_sensor_bathroom_door_intrusion
    from: "off"
    to: "on"
conditions:
  - condition: state
    entity_id: input_boolean.guest_bathroom_queue
    state: "on"
actions:
  - action: input_boolean.turn_off
    target:
      entity_id: input_boolean.guest_bathroom_queue
  - action: notify.family_devices
    data:
      title: Bathroom
      message: Bathroom is free 🚻
mode: single

3. Someone cancels their request

Triggers when guest_bathroom_queue flips back off while the door is still closed, meaning they cancelled manually, not because the bathroom freed up.

alias: Cancel Bathroom Request
triggers:
  - trigger: state
    entity_id: input_boolean.guest_bathroom_queue
    from: "on"
    to: "off"
conditions:
  - condition: state
    entity_id: binary_sensor.contact_sensor_bathroom_door_intrusion
    state: "off"
actions:
  - action: notify.family_devices
    data:
      title: request cancelled
      message: user canceled their bathroom request
mode: single

The Dashboard Card

The card has two states driven by binary_sensor.contact_sensor_bathroom_door_intrusion.

Door open → green icons, shows time since it opened.

Door closed → door icons, shows time since it closed, plus a queue button.

I used conditional cards to handle the two states rather than a custom component. It's more YAML, but everything stays in the dashboard config.

The button toggles input_boolean.guest_bathroom_queue. When the queue is active, the icon turns orange, indicating someone is waiting. Tapping it again cancels the request.

Note: The card_mod blocks are optional. It just handles button sizing and color. If you don't have the card-mod HACS frontend integration installed, remove those blocks, and everything still works.

card:
  type: vertical-stack
  cards:
    - type: conditional
      conditions:
        - condition: state
          entity: binary_sensor.contact_sensor_bathroom_door_intrusion
          state: "on"
      card:
        type: tile
        entity: binary_sensor.contact_sensor_bathroom_door_intrusion
        name: ✅ Bathroom is Available
        icon: mdi:door-open
        color: green
        hide_state: false
        state_content:
          - last_changed
        vertical: true
        tap_action:
          action: none
        icon_tap_action:
          action: none
        features_position: bottom
    - type: conditional
      conditions:
        - condition: state
          entity: binary_sensor.contact_sensor_bathroom_door_intrusion
          state: "off"
      card:
        type: vertical-stack
        cards:
          - type: tile
            entity: binary_sensor.contact_sensor_bathroom_door_intrusion
            name: 🚪 Bathroom is Occupied
            icon: mdi:door-closed
            color: red
            vertical: true
            hide_state: false
            tap_action:
              action: none
            icon_tap_action:
              action: none
            state_content:
              - last_changed
          - type: conditional
            conditions:
              - condition: state
                entity: input_boolean.guest_bathroom_queue
                state: "off"
            card:
              type: button
              entity: input_boolean.guest_bathroom_queue
              icon: mdi:account-outline
              name: Wait for Bathroom
              tap_action:
                action: toggle
              card_mod:
                style: |
                  ha-card {
                    margin: 0 auto;
                    width: 200px;
                    text-align: center;
                  }
          - type: conditional
            conditions:
              - condition: state
                entity: input_boolean.guest_bathroom_queue
                state: "on"
            card:
              type: vertical-stack
              cards:
                - type: button
                  entity: input_boolean.guest_bathroom_queue
                  icon: mdi:human-male
                  name: Waiting for Bathroom
                  tap_action:
                    action: toggle
                  card_mod:
                    style: |
                      ha-card {
                        margin: 0 auto;
                        width: 200px;
                        text-align: center;
                        color: orange;
                      }
                - type: markdown
                  content: >
                    <div style="text-align:center; font-size:16px; color:
                    orange;">
                      🕒 You're in the queue! We'll notify you when it's your turn.

                    </div>

Notifying All Devices Without Repeating Yourself

You can call individual device services directly in automations:

actions:
  - action: notify.mobile_app_mom_big_tablet
    data:
      message: Bathroom is free 🚻
  - action: notify.mobile_app_kevin_f6
    data:
      message: Bathroom is free 🚻

That works. The problem is every automation that needs to notify everyone repeats the same device list. Add a new phone, edit every automation.

Instead, define a notify group once in configuration.yaml:

notify:
  - platform: group
    name: family_devices
    services:
      - service: mobile_app_device_1
      - service: mobile_app_device_2
      - service: mobile_app_device_3
      - service: mobile_app_device_4

Now every automation calls notify.family_devices. Add or remove a device in one place.

Triggered when someone joins the queue

Triggered when the bathroom becomes free

Guest Access via Trusted Networks

Guests connect to a dedicated VLAN (192.168.10.0/24). A guest WiFi password alone isn't enough here. Anyone with the password could browse to Home Assistant and see the full dashboard. The VLAN lets me scope exactly what they can access at the network level, and Home Assistant auto-logs them in as a guest user when they're on that subnet, bypassing the password prompt.

In configuration.yaml:

homeassistant:
  auth_providers:
    - type: trusted_networks
      trusted_networks:
        - 192.168.10.0/24
      allow_bypass_login: true
      trusted_users:
        192.168.10.0/24: <your_guest_user_id>
    - type: homeassistant

Navigate to http://<your-ha-ip>:8123/config/users and click the guest user. The ID is shown explicitly in the first row.

Guests who don't have the Home Assistant app get a QR code to the bathroom dashboard. They land on the card, see live status, and can join the queue, but they can't access anything else.

The guest dashboard itself is restricted to two user IDs via the visible field, so the isolation is enforced at the Home Assistant layer.

How I Found a CSS Bug on Etsy's Engineering Blog

Kevin Lu — Tue, 24 Feb 2026 00:04:00 +0000

As a backend dev, I was browsing Etsy's Code as Craft blog and noticed the category dropdown was cutting off the last option. Everything displayed as expected except the CodeMosaic option, which appeared below the dropdown border.

I opened DevTools to inspect the element. The HTML was fine, CodeMosaic was inside the same parent <div> as everything else. But the inspect element highlight showed it was rendering outside the container's boundaries.

Next, I checked the computed styles on the dropdown container (.wt-menu__body). It has a rendered height of 478px. The shared stylesheet (settings-overlay.css) sets overflow: hidden and a max-height:

max-height: calc(var(--clg-dimension-pal-size-1000,60px)*8);

60px × 8 = 480px. Each menu item is 51.2px tall, so 10 items require 512px, which is more than the container allows. Bumping the multiplier to 9 (60 × 9 = 540px) is enough to fit all 10.

Normally, overflow: hidden would clip the extra items. But the blog page's own stylesheet (blog_home_page.css) has a more specific selector, .cac-header .wt-menu div.wt-menu__body, that sets overflow back to visible. So CodeMosaic renders below the dropdown border instead of being hidden.

Changed the max-height in DevTools, and CodeMosaic snapped into place. A fix would be to increase the multiplier. A more durable fix would be either removing the max-height constraint so the dropdown sizes to its content, or fixing the specificity override in blog_home_page.css so the container can use overflow-y: auto and scroll when it needs to.

Update 02/26: Just checked and Etsy has since fixed this issue :)