DEV Community: Bizzi Cole87

I Built a Voice-First AI Photo & Document Editor with the Gemini Live API— Here's How

Bizzi Cole87 — Sun, 15 Mar 2026 22:04:07 +0000

This article was created for the purposes of entering the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

There's a version of photo editing where you don't touch a single slider. You click the part of the image you want to change, say what you want out loud, and watch it happen. That's Say Edit.

Over the past few weeks I built Say Edit — a voice-first AI workspace that lets you edit images and navigate documents entirely by speaking. It's powered by the Gemini Live API, Gemini image generation, and deployed on Google Cloud Run. This article is the behind-the-scenes of how I built it, what broke, and what surprised me.

The Core Idea

Most AI tools make you type. You open a chat window, describe what you want, wait for a response, copy it somewhere, repeat. I wanted to eliminate every one of those steps for two use cases I found genuinely painful:

Editing a photo — you know exactly what you want to change, but you have to hunt through menus, masks, and sliders to get there.
Reading a dense document — you have a question, but finding the exact passage means scrolling through 80 pages.

The answer to both is the same: a persistent voice session that listens continuously, understands your intent, and acts immediately.

The Voice Loop — Gemini Live API

The foundation of Say Edit is a bi-directional WebSocket with the Gemini Live API (gemini-2.5-flash-native-audio-preview-12-2025). This is not a push-to-talk button. The model listens the entire time you're in the workspace.

Here's what the audio pipeline looks like on the browser side:

const source = audioCtx.createMediaStreamSource(stream);
const processor = audioCtx.createScriptProcessor(4096, 1, 1);
source.connect(processor);
processor.connect(audioCtx.destination);

processor.onaudioprocess = (e) => {
  const float32 = e.inputBuffer.getChannelData(0);
  // Float32 → Int16 → base64
  const int16 = new Int16Array(float32.length);
  for (let i = 0; i < float32.length; i++) {
    int16[i] = Math.max(-32768, Math.min(32767, float32[i] * 32768));
  }
  session.sendRealtimeInput({
    media: { data: btoa(String.fromCharCode(...new Uint8Array(int16.buffer))), mimeType: 'audio/pcm;rate=16000' }
  });
};

16kHz PCM, streamed continuously. The model responds with audio at 24kHz, which I queue through AudioBufferSourceNode for gapless playback.

The part that changed how I think about this kind of app: interruption. When you speak while the model is responding, the Live API sends serverContent.interrupted. I clear the audio queue instantly and the model re-listens. No waiting. That single behavior makes the whole thing feel like talking to a person rather than waiting for a request.

Image Editing — Hotspot + Voice + Gemini Image Generation

The image editor works in three steps:

Click anywhere on the image. The frontend records the pixel coordinates (x, y) as a "hotspot" and renders a crosshair.
Speak your edit. The Gemini Live session calls the edit_image_region tool with your spoken instruction and the hotspot coordinates.
Watch the edit apply. The current image is sent as base64 to gemini-2.0-flash-exp-image-generation alongside the prompt and coordinates. The result replaces the current frame in the history stack.

// Tool definition registered with the Live session
{
  name: 'edit_image_region',
  description: 'Edit a specific region of the current image.',
  parameters: {
    type: Type.OBJECT,
    properties: {
      edit_prompt: { type: Type.STRING },
      hotspot_x: { type: Type.NUMBER },
      hotspot_y: { type: Type.NUMBER },
    },
    required: ['edit_prompt', 'hotspot_x', 'hotspot_y'],
  }
}

The edit call to Gemini image generation:

const response = await ai.models.generateContent({
  model: 'gemini-3.1-flash-image-preview',
  contents: [
    { inlineData: { mimeType: file.type, data: base64 } },
    { text: `Edit: "${editPrompt}". Focus around pixel (x: ${x}, y: ${y}). Return ONLY the edited image.` }
  ],
  config: { responseModalities: ['IMAGE', 'TEXT'] }
});

Every edit is non-destructive — pushed onto a history stack with full undo/redo and a hold-to-compare button that fades back to the original.

The Stale Closure Problem

This one burned me for a while. The Live API's onmessage callback is registered once at session start. Any React state it closes over is immediately stale — meaning history[historyIndex] would always point to the original image, no matter how many edits had been applied.

The fix was to maintain refs that mirror the state and always read from those inside the callback:

const historyRef = useRef<File[]>([initialFile]);
const historyIndexRef = useRef<number>(0);
historyRef.current = history;       // kept in sync on every render
historyIndexRef.current = historyIndex;

// Inside the async tool handler — always gets the live value
const imageFile = historyRef.current[historyIndexRef.current];

This is now a fixture of how I build anything on top of the Live API.

Document Navigation — Spatial Search + Live Highlights

The document workspace is a different beast. When you ask a question, the AI shouldn't just answer — it should show you exactly where in the document the answer lives.

The pipeline:

1. Ingestion (NestJS backend on Google Cloud Run)

When you upload a PDF, the backend runs it through pdfjs-dist, extracts text items with their transform matrices, groups words into lines by Y-coordinate proximity, and merges lines into sentence-level chunks. Every chunk stores a tight bounding box [x, y, width, height] alongside the text.

One gotcha: PDF coordinates use bottom-left origin, but the React PDF viewer uses top-left. Every Y coordinate gets flipped during ingestion:

y = pageHeight - transform[5] - (height || 10)

Each chunk is then embedded with gemini-embedding-001 and stored in a Supabase pgvector index with an HNSW index for fast cosine similarity search.

2. Search during the Live session

When you ask a question, the Live session calls search_document. The frontend hits the backend's /query endpoint, which embeds your query and runs a cosine similarity search against the document's chunks. The top results come back with page numbers and bounding boxes.

3. Spatial highlight

The model then calls focus_document_section with the page and an array of bounding box coordinates. The frontend jumps the PDF viewer to the right page and renders yellow highlight overlays at the exact pixel positions of the relevant sentences:

// Convert stored PDF-point bboxes to percentage overlays
const highlight = {
  pageIndex,
  left:   (bx / pageWidthPx) * 100,
  top:    (by / pageHeightPx) * 100,
  width:  (bw / pageWidthPx) * 100,
  height: (bh / pageHeightPx) * 100,
};

You hear the answer and see it highlighted on the page at the same moment.

The Compose Studio

Beyond single-image editing, Say Edit has a Compose Studio that merges two photos by voice:

"Dress the person from A in the outfit from B."
"Put the product on the background from B."

Both images are encoded as base64 and sent to Gemini image generation together:

const response = await ai.models.generateContent({
  model: 'gemini-3.1-flash-image-preview',
  contents: [
    { inlineData: { mimeType: imgA.type, data: base64A } },
    { inlineData: { mimeType: imgB.type, data: base64B } },
    { text: `Composition request: "${prompt}". Return ONLY the composed image.` }
  ],
  config: { responseModalities: ['IMAGE', 'TEXT'] }
});

The result lands in the same history stack as regular edits — so you can compose two images together and then keep refining the result by voice.

Infrastructure — Google Cloud Run

Both the frontend (React + Vite) and the backend (NestJS) are containerized and deployed to Google Cloud Run. The Dockerfile for the frontend bakes the Vite environment variables at build time using ARG injection:

ARG VITE_GEMINI_API_KEY
ARG VITE_BACKEND_URL
RUN echo "VITE_GEMINI_API_KEY=$VITE_GEMINI_API_KEY" > .env
RUN echo "VITE_BACKEND_URL=$VITE_BACKEND_URL" >> .env
RUN npx vite build

Deploy with:

gcloud run deploy say-edit \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --set-build-env-vars "VITE_GEMINI_API_KEY=...,VITE_BACKEND_URL=..."

What I Learned

The Live API's interruption model is the whole game. Every other voice interface I've used makes you wait. The ability to interrupt mid-sentence — backed by a proper server-side signal rather than a client-side hack — is what makes Say Edit feel like a real conversation.

Tool design is UX design. The get_current_hotspot tool exists purely so users can say "edit this" without re-stating coordinates. Getting the tool schema right — what the model calls, when, and with what arguments — determines the quality of the interaction more than any UI element.

Refs are mandatory in Live API callbacks. Any async callback registered at session start captures stale React state. The ref-mirror pattern is non-negotiable.

PDF extraction is harder than it looks. pdfjs-dist gives you transform matrices, not sentences. The grouping and chunking pipeline is the most underestimated part of the whole system.

Try It

Live demo: https://glyph-client-229364276486.us-central1.run.app/
Client repo: https://github.com/greatsage-raphael/say_edit
Server repo: https://github.com/greatsage-raphael/say_edit_server

Built with the Gemini Live API, NaNo Banana, Google Cloud Run, and Supabase.

This article was created for the purposes of entering the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

Business Portfolio Generator

Bizzi Cole87 — Sun, 26 Jan 2025 22:15:12 +0000

Business Portfolio Generator 💼🌐
This is a submission for the Agent.ai Challenge: Assembly of Agents
What I Built
An automated portfolio website generator combining Vercel's prompt generation with Code Genie's development capabilities. Users input basic business details to receive a professional website instantly.

Demo
Try it here: https://agent.ai/agent/portfolio

Process Flow:

User provides business name, industry, and description
Vercel agent generates optimized prompts
Code Genie transforms prompts into website code
Complete portfolio website delivered in HTML

Agent.ai Experience:
The development highlighted agent collaboration's potential for business automation. Combining prompt engineering with code generation created a streamlined website creation process.
Key Achievements:

Seamless integration between prompt and code generation
Automated business-specific customization
Rapid portfolio deployment

This project demonstrates how agent assembly can simplify complex business tasks through AI collaboration.

Career Launcher: Improve your linked-in today

Bizzi Cole87 — Sun, 26 Jan 2025 20:02:55 +0000

Professional Development Hub 🚀👔

This is a submission for the Agent.ai Challenge: Assembly of Agents

What I Built

I created a powerful professional development agent that leverages multiple specialized agents to generate a comprehensive career enhancement package. By combining CareerGPS, Personal Portfolio, and LinkedIn Post Generator, this agent transforms a simple LinkedIn profile ID into a holistic career growth toolkit.

Key features:

Personalized career advice generation
Automated personal portfolio website creation
Tailored LinkedIn post drafting
One-click professional branding solution

The agent addresses a critical need: helping professionals efficiently showcase their skills, receive targeted career guidance, and maintain a strong online presence.

Demo

You can experience the Professional Development Hub agent here: https://agent.ai/agent/careerLauncher

Workflow Breakdown:

User enters LinkedIn Profile ID
Select professional development themes
CareerGPS generates personalized career advice
Personal Portfolio agent creates a website
LinkedIn Post Generator crafts an engaging post
Outputs are generated in multiple formats

Agent.ai Experience

Building this agent was an exploration of AI's collaborative potential. The most exciting aspect was watching different specialized agents seamlessly work together to create a comprehensive solution.

Delightful moments:

Seeing how different agents complement each other
Witnessing the transformation of a simple profile ID into multi-format professional content
Realizing the power of agent assembly

Challenges included:

Ensuring smooth communication between agents
Maintaining consistency across different generated outputs

The key breakthrough was understanding that specialized agents, when strategically combined, can create solutions far more powerful than individual agents.

Conclusion

The Professional Development Hub demonstrates how agent assembly can simplify complex professional branding and career development processes. It's not just a tool, but a career acceleration platform powered by AI collaboration.

Thank you for the opportunity to showcase the potential of assembled agents! 🙏

Podcaster: Your Daily News podcast.

Bizzi Cole87 — Sat, 25 Jan 2025 10:50:18 +0000

Daily News Podcast 🎙️📰

This is a submission for the Agent.ai Challenge: Productivity-Pro Agent

What I Built

I created an AI-powered Daily News Podcast agent that automatically generates a concise, up-to-date audio podcast on any global topic within minutes. The agent transforms the complex world of current events into an easily digestible audio experience, making staying informed effortless and convenient.

Key features of the Daily News Podcast agent include:

Dynamic topic selection based on user input
Real-time news aggregation from Google News
AI-powered script generation using the latest news data
Automatic audio podcast creation

Why did I build this? In our fast-paced world, staying informed is crucial, but finding time to read extensive news articles can be challenging. This agent provides a quick, accessible way to consume current events during commutes, workouts, or any spare moment.

Demo

You can experience the Daily News Podcast agent here: https://agent.ai/agent/dailynewspodcast

Workflow Breakdown:

User selects or inputs a topic (default: "Rapidly changing global alliances")
Agent retrieves latest Google News results for the topic
GPT-4o generates a professional podcast script from the news data
Audio is generated from the script
Podcast is output in an automated format

Agent.ai Experience

Building this agent on the Agent.ai platform was an exciting journey of creativity and technological exploration. The platform's intuitive interface made complex automation feel surprisingly straightforward.

The most delightful moments were:

Seeing the agent seamlessly pull real-time news
Watching an AI transform raw news data into a coherent podcast script
Hearing the generated audio podcast come to life

Challenges included fine-tuning the script generation to maintain a natural, engaging tone and ensuring the audio output sounds conversational.

The key to success was breaking down the podcast generation into clear, modular steps:

Input gathering
News retrieval
Script generation
Audio creation
Output formatting

Conclusion

The Daily News Podcast agent represents a small but meaningful step towards making information consumption more accessible, personalized, and convenient. It showcases how AI can transform how we interact with news and stay informed.

Thank you for the opportunity to participate in this challenge! 🙏

check out my new app. Leave a like if you love it

Bizzi Cole87 — Tue, 21 Jan 2025 14:43:43 +0000

Habits: A Modern Habit Tracking Application

Bizzi Cole87 ・ Jan 16

#devchallenge #githubchallenge #webdev #ai

Habits: A Modern Habit Tracking Application

Bizzi Cole87 — Thu, 16 Jan 2025 10:00:05 +0000

This is a submission for the GitHub Copilot Challenge: New Beginnings

What I Built

I created "Habits" a minimalist yet powerful habit tracking application that helps users build and maintain positive habits throughout the year. The application features a clean, intuitive interface that allows users to:

-Track multiple habits across a 52-week timeline
-Break down habits into actionable subtasks with progress tracking
-Celebrate completion with a delightful confetti animation
-Customize habits with different colors for visual organization
-Mark daily completions with an elegant checkbox system
-Navigate through weeks effortlessly
-Add and remove habits dynamically
-Store progress persistently using Supabase

The project uses React with TypeScript, integrating modern UI components from shadcn/ui, and features responsive design principles for a seamless experience across devices.

Demo

Repo

Check out the source code on GitHub: https://github.com/greatsage-raphael/habits

How GitHub Copilot Made It All Possible

In this project, GitHub Copilot was more than just a tool. It was like having a pair programmer with me throughout the day. Here's how Copilot made a difference:

Code Writing & Suggestions

Copilot was instrumental in accelerating development by suggesting smart, concise code snippets. Some key areas where it excelled:

Type definitions for Habit and HabitLog interfaces
State management setup with useState hooks
Database integration with Supabase
Complex UI grid layouts with Tailwind CSS

Debugging Assistance

Copilot helped identify and resolve several potential issues:

Suggested proper error handling in database operations
Helped implement proper TypeScript types
Provided solutions for state management edge cases
Assisted with component lifecycle management

Faster Development

Thanks to Copilot's suggestions, I was able to develop the website quickly and efficiently. Some notable time-savings came from:

Auto-completing repetitive UI patterns
Generating consistent styling patterns
Suggesting optimal data structures
Creating efficient database queries

Inline Suggestions ⚡

Copilot provided valuable inline suggestions for:

Error handling patterns in database operations
Consistent variable naming conventions
Type definitions and interfaces
UI component structure

Model Switching 🔄

I leveraged different AI models for specific tasks:

Code Completion: GitHub Copilot
Documentation: Claude
Debugging: GPT-4

Common Prompts Used 🎯

# Function implementation
/implement habit toggle logic
/suggest state management pattern
/optimize database queries

# UI Components
/create responsive grid layout
/implement week navigation
/design habit row component

Code Edits ✏️

Copilot helped with several important code improvements:

Refactored habit management functions
Added proper TypeScript types
Improved error handling
Enhanced component documentation
Optimized database queries

GitHub Models

The project extensively used GitHub Copilot's code completion and suggestions. The model was particularly helpful in:

Generating TypeScript interfaces
Suggesting optimal React hooks usage
Creating efficient database queries
Implementing proper error handling
Structuring components effectively

Technical Implementation

The application is built using:

Next.js with TypeScript for the frontend
Supabase for data persistence
shadcn/ui for UI components
Tailwind CSS for styling
Clerk for user authentication

Key features include:

Real-time habit tracking
Persistent storage
Responsive design
User authentication
Custom color coding for habits

Conclusion

Building New Beginnings with GitHub Copilot demonstrated the power of AI-assisted development. The tool not only accelerated the development process but also helped maintain high code quality and consistency throughout the project.

The habit tracker serves its purpose of helping users build positive habits while showcasing modern web development practices. The combination of TypeScript, React, and Supabase, aided by Copilot's suggestions, resulted in a robust and maintainable application.

Future enhancements could include:

Data visualization for habit streaks
Mobile applications
Social features for accountability
Custom reminder systems
Progress sharing capabilities

This project showcases how AI tools like GitHub Copilot can significantly enhance developer productivity while maintaining code quality and best practices.

Evolution By Sound

Bizzi Cole87 — Wed, 15 Jan 2025 09:46:53 +0000

Evolution by Sound: Transitions and Transformations

This project, evolution by sound, is a dynamic web application that allows users to upload audio files and experience their music in a new visual dimension through an interactive and visually captivating audio spectrum rendered with Three.js. It combines creative coding, modern web development, and AI assistance to deliver an engaging user experience.

What I Built

Evolution by Sound leverages Three.js and React to create a dynamic audio visualization application. Users can upload an audio file, and the app analyzes the audio's frequency data to drive a real-time, interactive visual experience. Customization options for colors, intensity, and animation speed further enhance the user’s connection to the music.

Inspiration

Abiogenesis: The origin of life from non-living matter.

Some studies suggest that acoustic forces may have played a role in prebiotic chemistry and the organization of materials that could have led to the first protocells. For example:

Acoustic waves can create small droplets and concentrate molecules, potentially helping to form membrane-like structures.
Sound waves can drive chemical reactions through cavitation (the formation and collapse of bubbles), which creates localized areas of high temperature and pressure.

This inspired the concept of Evolution by Sound, where the initially inactive, slow-moving patterns transform into an interactive audio spectrum visualization reminiscent of abiogenesis through sound. Once a user uploads a song, the visualizations evolve dynamically, symbolizing life emerging through the transformative power of sound.

Features

Interactive Audio Spectrum Visualization:
- A real-time visual representation of the uploaded audio, using Three.js shaders.
- Frequency data is analyzed and mapped to create an immersive visual experience.
User Customization:
- RGB sliders allow users to adjust the color palette.
- Intensity and speed controls provide further customization for the visualization.
Audio Upload and Playback:
- Users can upload an audio file, which is processed via an API to provide playback and analysis.
- A built-in audio player allows users to control playback seamlessly.
Clean and Responsive UI:
- Components like Button and Slider make interactions intuitive.
- The app is optimized for responsiveness and usability across devices.

Demo

Live app: Evolution by Sound

Screenshots/GIFs

Repo

GitHub Repository: https://github.com/greatsage-raphael/evolutionBySound

Copilot Experience

This was my first time using Three.js, and GitHub Copilot played a pivotal role in developing this project. From conceptualization to execution, Copilot assisted in:

Component Development:
- Copilot generated much of the foundational functionality for both the Cell and InteractiveMusicWave components based on my prompts.
- By providing clear context and intent, I could rely on Copilot’s autocomplete to implement Three.js logic, which would have otherwise required significant manual learning.
Iterative Improvements:
- Copilot offered suggestions for optimizing shader logic and connecting audio analysis to the visuals.
- Edits and adjustments to prompts allowed me to refine functionality without needing to switch contexts frequently.
Learning and Experimentation:
- Copilot served as a guide while exploring Three.js, offering concise examples and snippets I could adapt for my use case.
- I used the chat feature and model switcher to refine more complex interactions, such as connecting audio data to shader uniforms.

GitHub Models

I utilized GitHub Copilot’s autocomplete and context-aware coding suggestions throughout the development process. While this submission didn’t directly involve prototyping LLM capabilities, it’s evident that Copilot’s integration into the workflow streamlined my progress and enhanced productivity.

Conclusion

Working on Evolution by Sound has been an enriching experience that showcased the creative possibilities of combining AI, music, and visuals. GitHub Copilot significantly lowered the barrier to entry for using Three.js and enabled me to focus on creativity rather than being bogged down by the technical details.

This project highlights how AI tools can empower developers to explore new domains, even with limited prior knowledge. I look forward to expanding this app further, perhaps by adding support for live audio streams or multiplayer audio-visual experiences.

This project was developed independently.*