Building a Hybrid Edge/Cloud AI Workspace with Real-Time Voice & PII Shielding

Building Meridian: A PII-Safe, Hybrid Edge/Cloud AI Workspace with Real-Time Voice Ingestion 🎙️⚡
This is a submission for the MLH x DEV Writing Challenge

What I Built
In modern enterprise environments, developers face a constant dilemma: Privacy vs. Power. Cloud AI models (like Gemini) are incredibly powerful but require sending sensitive data to external servers. Local Edge models run privately and free, but often lack the heavy multimodal reasoning required for complex tasks.

My team and I asked ourselves: What if we didn't have to choose?

We built Meridian Workflow Hub, a self-hosted AI operating environment that acts as the brain for team workflows. It ingests multimodal inputs (voice, PDFs, emails), automatically shields sensitive PII, and intelligently routes tasks between local edge models and cloud endpoints based on the payload's complexity.

Here are the core engineering pillars we implemented:

Streaming Chunked Voice Pipeline: Bypassing slow, traditional Speech-to-Text (STT) pipelines, we built a speculative sliding-window audio ingest router. It takes live mic input, chunks it, and streams raw audio frames directly into the multimodal context. The model speculative-parses incoming phrases as words leave the user's mouth, reducing perceived latency to near-zero.

The Local Voice Stack: Pipecat + Gemma4-e2b + Kokoro-82M
Building a voice agent on an 8GB RAM laptop with an Intel i3 processor meant we couldn't just throw compute at the problem. We had to be ruthless about memory efficiency and Time-to-First-Token (TTFT).

The Orchestrator: Instead of building raw websockets from scratch, we used Pipecat, an open-source framework that flawlessly handles WebRTC streaming, Voice Activity Detection (VAD), and user interruptions.

The Brain (gemma4-e2b): We rejected heavy 7B/8B models like LLaVA or Llama-3 because they would instantly thrash the swap memory on our 8GB machine and freeze the system. gemma4-e2b is highly quantized, leaving plenty of memory overhead while delivering near-instant conversational reflexes.

The Voice (Kokoro-82M): This was our holy grail. Standard local TTS engines (like XTTSv2) take gigabytes of RAM and introduce seconds of delay. Cloud APIs (like ElevenLabs) violate our local-first privacy shield. Kokoro is an astonishingly tiny 82-million parameter model that runs locally on CPU with zero network latency, keeping our PII-shield intact while delivering natural, emotive speech in milliseconds.

Real-Time Markdown Demuxing: As the AI streams its response back, our parser demultiplexes the payload in real-time. It separates internal model logic (hidden inside tags) from the final Markdown output. This allows the user to watch the AI's "working process" right alongside the final answer.

The PII Shield (Reversible Tokenization): Before any data touches a cloud model, our local anonymizer scrubs sensitive data (emails, SSNs, phone numbers) and replaces them with deterministic tokens (e.g., ). When the LLM responds, the egress parser deanonymizes the output, ensuring zero sensitive data leakage to external APIs.

Dynamic Hybrid Task Routing: Fast/sensitive tasks are routed to local Ollama models (gemma2:2b, phi4-mini,nemotron-mini), while heavy visual reasoning (like reading event flyers) is routed to the cloud.

Autonomous Dispatch: Connects AI output directly to the real world by dispatching structured JSON to Discord Webhooks or rendering one-click Google Calendar injections directly in the UI.

The Voice Integration Workflow:
Why We Chose Speculative Chunking
When building the voice ingestion pipeline, we initially considered the standard route: record the full audio, transcribe it with a local Whisper model, and pass the text to the LLM.

We immediately scrapped that idea for two reasons: Constraints (running Whisper + an LLM simultaneously on an 8GB ram laptop causes massive swap-memory thrashing) and perceived latency (waiting for a 60-second audio clip to process takes 10-15 seconds of dead air).

Instead, we built a Streaming Chunked Pipeline with Rolling Context Synthesis. Here is why this was the absolute best workflow for our architecture:

Browser-Native STT: We offloaded the Speech-to-Text translation entirely to the browser using the Web Speech API. This costs zero local server memory and requires no API keys.

Speculative Sliding Windows: We break the inbound audio stream into overlapping 3–5 second chunks. The edge model speculative-parses these phrases as words leave the user's mouth, reducing the Time-to-First-Token (TTFT) to near-zero.

Map-Reduce Global Consolidation: The danger of chunking is "Context Boundary Loss" (e.g., the meeting time is in chunk 2, but the location is in chunk 8). To fix this, as soon as our Voice Activity Detection (VAD) registers silence, a lightweight final synthesis pass unites the aggregated transcript. gemma4:e2b-it runs a single-shot execution to reconcile overlapping boundaries, extract the final action items, and emit the Google Calendar link. It is connected to kokoro-82M to generate the combined output and is connected with gemma4:e2b-it with Pipecat

Demo

PROJECT 2.mp4 - Google Drive

drive.google.com

How it works in action:

Voice & Vision: You can upload an image of a hackathon flyer and literally speak to the app: "Extract the meeting details from this flyer and move the venue to the Microsoft Gurugram office."

Live Terminal: You immediately see the terminal chunking your voice input. If you watch closely, you can see the Demuxer splitting the AI's internal reasoning from its final output.

Actionable Output: It doesn't just give you text back. It autonomously injects a fully rendered, clickable 📅 Add Event to Google Calendar button right into the UI, or dispatches the summary directly to a Discord webhook.

Partner Technologies
We heavily utilized MLH partner technologies to make this architecture a reality, specifically focusing on Google Cloud (Gemini).

Google Gemini API (GenAI SDK)
While we love our local edge models, we needed serious multimodal firepower for visual document parsing. We integrated the Gemini 3.6 Flash model via the Google GenAI Python SDK.

Implementation: We built an in-browser Base64 image converter using HTML5 Canvas that sends raw image strings directly to our FastAPI backend. The backend passes this to Gemini to deconstruct flyers and schedules.

The Challenge: During testing, we hit 503 UNAVAILABLE high-demand spikes. We engineered a resilient fallback loop: if Gemini threw a 503, our Python backend would gracefully catch the exception and instantly reroute the Base64 image to a local Moondream (1.8B) vision model running on Ollama, ensuring zero downtime during our live demo.

Hackathon Experience
We built this at the USICT Hackathon, and the experience was nothing short of cinematic. Out of 1000+ participants, we were thrilled to be selected as one of the Top 29 Finalist Teams.

The absolute highlight was getting to do our final pitch and live run-through at the Microsoft Main Office in Gurugram. The atmosphere was electric. Teaming up with Mayank,Ytharth and Yashvardhan was a masterclass in collaboration. We were literally hardcoding execution bypasses and testing offline fallbacks moments before going on stage just in case the venue Wi-Fi dropped.

Knowing our app could survive a cloud outage by falling back to local edge nodes while we were standing in front of the judges gave us massive confidence. I will never forget the satisfaction of seeing that generated Google Calendar button pop up on the screen during the live demo flawlessly.

I’d love to hear your thoughts on edge vs. cloud computing!
Have you experimented with local models like Ollama, or do you prefer relying strictly on cloud APIs? Drop a comment below or ask any questions about how we handled the real-time audio streams!