How I Built OmniSence -A Multimodal AI That Streams Text, Images & Audio Together

Balaraj R — Mon, 16 Mar 2026 18:21:55 +0000

Disclosure: I created this piece of content for the purposes of
entering the Gemini Live Agent Challenge hackathon on Devpost.

GeminiLiveAgentChallenge

The Problem That Kept Me Up at Night

Every AI tool I've used thinks in documents, not experiences.

You get text here. An image there. Maybe audio if you switch tabs
and use a different tool entirely. But a real creative director doesn't
hand you a Word document — they paint a scene with words, sketches, and
emotion simultaneously.

That gap is what I built OmniSence to close.

What OmniSence Does

OmniSence is a Creative Director AI that takes a single idea — spoken
or typed — and streams text, images, and audio together in real-time
as one cohesive, interleaved experience.

You speak: "A girl who discovers she can paint the future."

OmniSence responds with:

📝 Narrative prose streaming word by word
🖼️ Watercolor illustrations appearing inline mid-sentence
🔊 Studio-quality narration reading the story back to you

All at once. All live. No switching tabs.

The Core Technical Innovation: Orchestrated Interleaved Streaming

This was the hardest problem to solve.

Gemini doesn't natively emit image bytes mid-text stream. So I designed
a pattern I call Orchestrated Interleaved Streaming using Google ADK:

User Prompt
    ↓
Google ADK Agent
    ↓
Gemini 3.1 Flash streams text with [IMAGE_DIRECTIVE: ...] markers
    ↓ (on marker detection)
Imagen 4 (async) ←→ Cloud TTS (async)
    ↓                    ↓
GCS Upload           GCS Upload  
    ↓                    ↓
SSE: {type:"image"}  SSE: {type:"audio"}
    ↓
React frontend renders everything inline, live

The key insight: image and audio generation run in parallel while
text continues streaming. The user never waits for one to finish before
the next begins.

The perceived latency formula:

$$T_{total} = T_{text} + \max(T_{imagen}, T_{tts}) - T_{overlap}$$

In practice, this means a full illustrated, narrated story appears in
under 30 seconds from a single voice prompt.

Building the ADK Agent

The Google ADK (Agent Development Kit) was the backbone of the entire
project. Instead of chaining API calls manually, I defined the agent
with 5 real async tools:

root_agent = Agent(
    name="omnisence",
    model="gemini-3.1-flash",
    description="OmniSence — Elite Creative Director AI",
    instruction=SYSTEM_PROMPT,
    tools=[
        generate_scene_image,   # Imagen 4 → GCS
        narrate_text,           # Cloud TTS → GCS  
        search_creative_references,  # Google Search grounding
        save_session_asset,     # GCS persistence
        get_style_constraints,  # Creative mode framework
    ],
)

What surprised me: the agent naturally decides when to generate
images vs. keep writing — creating organic pacing without me hardcoding
any rules. That emergent creative judgment was something I didn't expect.

Grounding: Preventing Hallucinations

For educational mode, I integrated Google Search grounding with a
single SDK addition:

if mode == "educational":
    generation_config["tools"] = [{"google_search": {}}]

One line. Dramatically more accurate factual content. The agent now
cites real sources before weaving them into the narrative.

The Google Cloud Stack

Service	What I Used It For
Gemini 3.1 Flash	Core text generation with image directives
Imagen 4	Scene illustration via Vertex AI
Cloud Text-to-Speech	Studio voice narration
Cloud Run	Serverless backend hosting
Cloud Storage	Asset persistence across sessions
Cloud Build	CI/CD with `cloudbuild.yaml`

One thing I want to highlight: Cloud Run's concurrency model was
perfect for SSE streaming. Each user gets a persistent async generator
that streams for up to 5 minutes — Cloud Run handles this gracefully
without the connection dropping.

The UX Decision That Changed Everything

Early in development, images appeared below the text after it
finished generating. It felt like "AI output."

The moment I moved images to appear inline — mid-paragraph,
exactly where the story described them — the experience shifted from
"AI output" to "living document."

That single UX change had more impact on how the product felt than
any technical improvement I made.

Challenges I Didn't Anticipate

Stream cancellation was deceptively hard. When a user hits
"Stop & Redirect" mid-generation, you can't just close the SSE
connection — there are async Imagen and TTS calls in flight on
GCP that need to be cleanly abandoned without leaving orphaned
uploads or billing surprises.

My solution: per-session cancellation flags checked at every
yield point in the async generator, combined with asyncio.shield()
for GCS cleanup tasks that must complete regardless.

Cloud TTS latency was the other beast. Studio voices take 2–4
seconds per paragraph. I solved this with a rolling generation pipeline:
generate audio for completed sentences while later sentences still stream.

What I'd Tell Someone Starting This Today

Use ADK, not raw SDK calls. The tool system makes multi-model
orchestration feel natural instead of spaghetti.
Design for streaming from day one. Adding SSE to a
request/response architecture later is painful. Build async generators
first.
Google Search grounding is a one-liner. Add it to any factual
mode. The quality difference is immediate and obvious.
The persona matters more than the features. OmniSence has a
distinct voice — warm, bold, cinematic. Users respond to that
personality more than any specific capability.
Deploy early to Cloud Run. Local dev hides async edge cases
that only appear under real network conditions.

Try It Yourself

🔗 GitHub: https://github.com/balaraj74/Omnisence

🔗 Live Demo: https://omnisence-518586257861.us-central1.run.app/

The entire project is open source. The README includes a one-command
deploy script — ./deploy.sh YOUR_PROJECT_ID YOUR_API_KEY — and
you'll have your own OmniSence instance running on Cloud Run in
under 10 minutes.

Built for the Gemini Live Agent Challenge · Powered by Google Gemini,
Imagen 4, and Google Cloud · #GeminiLiveAgentChallenge

DEV Community: Balaraj R