SAM3 + Gemma 4 on CUDA: Chaining Candidate Segmentation Masks with Local Multimodal Reasoning

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Ask any freelance video editor, VFX artist, e-commerce product photographer, or boutique AI data-labeling contractor what eats up the most unpaid hours in their week, and they will give you the same answer: isolating specific objects across messy images and video timelines.

When a client hands an editor raw footage and asks to "track and highlight only the specific target items in the scene," or an e-commerce seller needs to isolate products from cluttered tabletop photos alongside structured catalog metadata, standard tools fall apart:

• Generic background removers blindly strip out everything in the foreground with zero semantic awareness.
• Manual frame-by-frame rotoscoping in editing suites takes hours of tedious keyframing.
• Cloud-based AI video masking tools lock freelancers into expensive monthly subscriptions, charge steep per-minute rendering credits, and require uploading gigabytes of confidential client footage to third-party servers.

To turn this tedious bottleneck into a zero-marginal-cost service, I built SAM3-Gemma4-CUDA—an interactive, GPU-accelerated computer vision and multimodal reasoning studio. By leveraging SAM3 to generate robust candidate segmentation masks and using Gemma 4 to intelligently filter and reason about these proposals based on user prompts, the tool delivers highly accurate, context-aware visual grounding right on a local machine.

Wrapped in a custom-styled "Steel Blue" Gradio interface, the application provides three core commercial workflows that freelancers, creators, and small studios would gladly pay for:

1. Image Detection with AI Filtering: Utilizes SAM3 to generate candidate regions from a text prompt. Gemma 4 then analyzes these proposals and filters them to match the user's exact request—providing a clean visual output alongside a natural-language explanation and structured JSON breakdown of its selections.
2. Automated Video Segmentation: Processes raw video files (.mp4) using SAM3's video tracking model. Users simply enter a text prompt to automatically segment and track target objects across frames, rendering either pure colored mask overlays or comprehensive bounding boxes and contours in seconds rather than hours.
3. Interactive Click Segmentation: Features an interactive canvas for pixel-perfect client revisions where users click directly on target objects, and the SAM3 tracker instantly generates and overlays precise segmentation masks based on cumulative point data.

Demo

🔗 Live Demo / Deployed Space: [Will update shortly]

🎥 Video Walkthrough / Screenshots: [Will update shortly]

Code

UMANG-SH941 / SAM3---Gemma-4---Segmentation

🔗 Direct Repository Link: https://github.com/UMANG-SH941/SAM3---Gemma-4---Segmentation

Repository Structure

text
├── examples/
│   ├── 1.jpg
│   ├── 1V.mp4
│   ├── 2.jpg
│   └── 3.jpg
├── app.py
├── LICENSE.txt
├── pre-requirements.txt
├── README.md
└── requirements.txt