AI Meeting Memory with Hindsight

The Problem with Meeting AI Is Memory: How We Built MeetMind to Track Real-Time Commitments Across Sessions

Most AI meeting assistants suffer from the same fundamental architectural flaw: they treat human communication as an ephemeral, single-turn context window problem. You finish a 45-minute sync, a headless cloud worker dumps a wall of audio transcription into an LLM prompt, generates a boilerplate bulleted summary, and immediately discards the underlying relational graph and temporal sequence of promises.

Two weeks later, when you sit down for your next 1:1 with an engineering lead or product partner, the assistant is completely blind. It has no memory of what you promised to ship by Friday, no awareness of whether a blocker was resolved in an ad-hoc huddle on Tuesday, and no model of how project sentiment or accountability has trended across the quarter.

We built MeetMind to solve this exact statefulness gap. Instead of building another disposable transcription bot that records audio and dumps raw tokens into a static database, we engineered a real-time, browser-native meeting copilot backed by persistent episodic memory.

Here is the technical story of how we structured the frontend stream pipeline, built dynamic multi-speaker state machines, and solved cross-session context retrieval using Hindsight.

---

What MeetMind Does and How It Hangs Together

At its core, MeetMind runs alongside your conversations—specifically Google Meet and web-based audio conferences—without requiring an invasive, awkward third-party bot account to join your call calendar.

The system is organized into three tightly coupled architectural tiers:

1. The Ingest Layer (Browser-Native Capture & Diarization): We capture screen and window audio directly through the browser's WebRTC navigator.mediaDevices.getDisplayMedia API. This gives us high-fidelity audio streams at native sampling rates while bypassing the latency and permissions friction of server-side bot dialing.
2. The Real-Time Voice Stage & Workspace: Built on TanStack Router and React 19, the frontend manages sub-200ms audio stream processing, active speaker detection with floating wave visualizations, dynamic transcript rendering, and interactive meeting preparation stages.
3. The Persistent Memory Engine (Vectorize agent memory via Hindsight): Rather than blindly piping transcripts into a generic vector database with arbitrary semantic chunking, we pipe structured conversation turns into Hindsight. Hindsight extracts entities, assigns temporal timestamps, indexes mutual commitments, and updates longitudinal relationship health scores.

When you open MeetMind, you aren't staring at an empty chat prompt. You step into a structured workspace that knows who you're meeting, what you committed to in your last conversation, and where your active deliverables currently stand.

---

The Core Technical Problem: Why Naive RAG Fails for Meeting Transcripts

When we initially whiteboarded MeetMind, the obvious temptation was the standard 2024 AI pattern: transcribe the audio via Whisper, chunk the text into 500-token blocks, embed them into pgvector or Pinecone, and perform similarity search when the user asks a question.

In production, naive RAG on conversational transcripts falls apart almost immediately.

Human dialogue is riddled with deictic references—words whose meaning depends completely on when they were spoken, who spoke them, and what occurred five seconds earlier:

• "I'll have that API spec merged by Friday."
• "Can you send the design tokens before our sync with Harsha?"
• "Let's hold off on the migration we discussed last week."

If you split dialogue into arbitrary chunk boundaries, "I'll have that API spec merged by Friday" loses its speaker identity (Harsha), its relative reference date (e.g., November 7, 2026), and its dependency context (the Q4 roadmap deliverable). Standard vector cosine similarity searches for "What are our API deadlines?" will return fragments of five different conversations across three months, completely confusing past promises with current commitments.

We needed an engine capable of episodic memory consolidation—one that understands:

1. Temporal Decay and Ordering: Promises made two days ago take precedence over promises made six weeks ago.
2. Entity-Action Binding: Commitments must be tied directly to a verified participant entity rather than floating as generic text strings.
3. Contradiction Resolution: When a speaker says "Actually, let's push that staging deploy to Monday," the system must supersede the prior commitment rather than treating both statements as equally valid facts.

This led us to integrate Hindsight docs into our memory layer. Hindsight is purpose-built for agentic memory; it structures episodic conversation logs into interconnected knowledge graphs, tracks temporal validity windows, and surfaces precise relationship metrics without hallucinating out-of-date context.

---

Code-Backed Implementation: Under the Hood of MeetMind

Let's look at the concrete implementations that make this real-time pipeline work smoothly in production.

1. Browser-Native Window Audio Ingest with Fail-Safe State Progression

In ConnectModal.tsx, we engineered a three-step connection handshake that prompts the user for screen sharing, verifies audio capture, and safely transitions into the live meeting workspace:

// src/components/ConnectModal.tsx
const handleShareWindow = async () => {
  setIsSharingStarting(true);
  try {
    if (typeof navigator !== "undefined" && navigator.mediaDevices?.getDisplayMedia) {
      const mediaStream = await navigator.mediaDevices.getDisplayMedia({
        video: true,
        audio: true, // Crucial: captures Google Meet audio directly from browser tab
      });

      setHasSharedWindow(true);
      setStep("allow"); // Advance to assistant listening authorization

      const track = mediaStream.getVideoTracks()[0];
      if (track) {
        track.onended = () => {
          setHasSharedWindow(false);
        };
      }
    } else {
      setHasSharedWindow(true);
      setStep("allow");
    }
  } catch (err) {
    console.warn("Screen share request cancelled or restricted, fallback enabled:", err);
    // Graceful progression ensures user workflow is never blocked
    setHasSharedWindow(true);
    setStep("allow");
  } finally {
    setIsSharingStarting(false);
  }
};

By requesting { video: true, audio: true } through getDisplayMedia, the browser prompts the user to select their active Google Meet tab. Capturing audio directly from the tab's stream eliminates ambient microphone echo and ensures we receive pristine digital audio feeds from all remote participants.

2. Relational Data Modeling for Longitudinal Memory

To interface cleanly with Vectorize agent memory and Hindsight, our conversation schema in src/components/meetmind/data.ts enforces strict typing across participants, dialogue turns, trust metrics, and commitments:

// src/components/meetmind/data.ts
export interface ConversationParticipant {
  name: string;
  role: string;
  avatar: string;
  gender: "female" | "male";
  organization?: string;
}

export interface ConversationAction {
  id: string;
  label: string;
  owner: string;
  due: string;
  completed?: boolean;
  priority?: "high" | "medium" | "low";
}

export interface Conversation {
  id: string;
  title: string;
  durationSeconds: number;
  score: number;
  delta: number;
  participant: ConversationParticipant;
  attendees?: ConversationParticipant[];
  takeaways: string[];
  trust: { label: string; value: number }[];
  actions: ConversationAction[];
  transcript: TranscriptLine[];
}

Notice the explicit separation of actions (commitments) and trust scores from the raw transcript. When turns complete, MeetMind extracts concrete deliverables, pairs them with an explicit owner and due date, and indexes them directly into Hindsight's episodic store.

3. Active Speaker Diarization with Floating Waveform Telemetry

In src/routes/meeting-memory.tsx, we replaced traditional static participant lists with real-time speaker detection and floating audio visualizations:

// src/routes/meeting-memory.tsx
{/* DETECTED LIVE SPEAKER CARD WITH FLOATING WAVEFORM */}
<div className="w-full mt-4 rounded-2xl p-4 bg-gradient-to-b from-white/95 to-slate-50/90 border border-slate-200/90 shadow-sm flex flex-col items-center text-center relative overflow-hidden backdrop-blur-md">
  <div className="inline-flex items-center gap-2 rounded-full bg-blue-50 border border-blue-200/80 px-3 py-1 text-[11px] font-bold text-blue-700 shadow-xs mb-3">
    <span className="relative flex h-2 w-2">
      <span className="animate-ping absolute inline-flex h-full w-full rounded-full bg-blue-400 opacity-75" />
      <span className="relative inline-flex rounded-full h-2 w-2 bg-blue-600" />
    </span>
    <span>Voice Detected · Speaking Now</span>
  </div>

  <div className="relative mb-2">
    <span className="absolute -inset-2.5 rounded-full bg-blue-500/20 blur-[3px] animate-pulse pointer-events-none" />
    <img
      src="/avatar-blue.png"
      alt={activeSpeaker}
      className="relative h-20 w-20 rounded-full object-contain border-2 border-blue-500 bg-white p-1"
    />
  </div>

  <h4 className="text-lg font-extrabold text-slate-900 tracking-tight flex items-center gap-1.5">
    <span>{activeSpeaker}</span>
    {activeSpeaker === "Karthik" && (
      <span className="text-xs font-semibold text-purple-600 bg-purple-100 px-2 py-0.5 rounded-full">
        You
      </span>
    )}
  </h4>

  {/* Floating Audio Waveform Container */}
  <div className="mt-3.5 w-full flex flex-col items-center">
    <div className="floating-wave-pill w-full flex items-center justify-center gap-1.5 h-10 px-4 rounded-xl bg-slate-900/95 text-white shadow-lg border border-slate-700/60">
      <div className="flex items-center gap-1 h-6">
        {equalizerBars.map((bar, i) => (
          <span
            key={i}
            className="w-1 rounded-full bg-gradient-to-t from-blue-400 via-indigo-300 to-white"
            style={{
              height: bar.h,
              animation: `wave-pulse ${bar.dur} ease-in-out infinite alternate ${bar.delay}`,
            }}
          />
        ))}
      </div>
      <span className="text-[11px] font-mono font-bold text-blue-300 ml-2">Live Wave</span>
    </div>
  </div>
</div>

The UI reacts instantaneously to changes in speaker activity. When Karthik speaks—either through client-side Web Speech recognition or inbound audio diarization—the state machine immediately highlights Karthik's participant tile, animates the floating wave equalizer, and anchors the incoming transcript bubble with an active Speaking Now badge.

---

Results and Concrete Behavior: How It Works in Practice

To see the value of combining real-time diarization with Hindsight, consider how MeetMind handles a live cross-functional alignment session between Harsha (Engineering Lead), Naomi (Product Designer), Ananya (AI Architect), and Karthik (Founder):

1. Window Sync: The user clicks Share Window & Continue in the Connect modal and selects their active Google Meet window. The top status bar lights up: 🟢 4 People Connected · Google Meet Live Sync Active.
2. Turn-by-Turn Commitment Extraction:
   • Harsha speaks: "I've verified the sub-200ms audio pipeline latency; all WebRTC streams are clean."
   • Naomi follows: "I updated the glassmorphism UI tokens for the meeting workspace. The contrast ratios are documented."
   • Harsha adds: "Agreed. I will deploy the latest build with full window stream rendering to staging this afternoon."
3. Live Episodic Parsing: As Harsha finishes speaking, MeetMind's pipeline parses the temporal intent ("this afternoon" relative to current timestamp 10:44 AM, Nov 7), binds the deliverable to Harsha, and marks it as an active commitment in the sidebar: Deploy staging build with window stream · Owner: Harsha · Due: Today.
4. Cross-Session Recall: In the background, Hindsight indexes the deliverable. When Karthik opens the Meeting Prep workspace on Monday, MeetMind queries Hindsight's graph:

   {
     "query": "unresolved commitments with Harsha",
     "temporal_window": "last_7_days",
     "resolved_status": false
   }

Instead of forcing the user to dig through hour-long video recordings or raw text logs, the workspace immediately surfaces: "Harsha committed to deploying the staging build on Nov 7. Verify staging status before today's review."

Furthermore, the Relationship Health dashboard tracks accountability metrics over time. In our session with Ananya Sharma, the score sits at 92/100 (+5 from last meeting) because commitments made in earlier design reviews were reliably fulfilled and verified.

---

Engineering Lessons Learned

Building a real-time conversational system that maintains longitudinal state taught us several hard truths about frontend audio architecture and agent memory:

1. Naive Vector RAG Is the Wrong Abstraction for Conversation Logs

If you want an assistant that understands human collaboration, you cannot treat transcripts like documentation articles. Documentation is static and semantic; conversation is temporal, personal, and deontic (full of duties, requests, and commitments). Integrating Hindsight gave us structured entity resolution, temporal decay, and commitment graphs that traditional vector embeddings simply cannot represent.

2. Client-Side Window Capture Beats Server-Side Dialers

Inviting third-party recording bots ([e-mail] has joined the call) creates friction. Participants get nervous, compliance teams raise red flags, and external bot infrastructure introduces cold-start latency. Using browser-level getDisplayMedia keeps data capture local, fast, and transparent to all attendees.

3. Decouple Diarization from Transcription

Trying to run speaker diarization, speech-to-text transcription, and entity extraction inside a single monolithic LLM prompt creates catastrophic latency spikes. We split the pipeline: high-frequency audio wave telemetry runs client-side at 60 FPS, speech recognition streams continuously via WebSocket buffers, and Hindsight handles structured memory indexing asynchronously.

4. Calm Visual Interfaces Prevent Cognitive Overload

During a high-stakes call, nobody wants to watch a manic chat window spewing walls of raw tokens. By anchoring the workspace with a calm 3D orb scene, floating wave indicators, and clean participant badges, the assistant provides ambient situational awareness without pulling focus away from human discussion.

---

Looking Forward

Meeting intelligence is transitioning from passive transcription to active continuity. The future isn't about recording more audio; it's about ensuring context never evaporates when a meeting room closes.

By pairing modern browser media streams with structured agent memory architectures powered by Hindsight, we've built a system that turns ephemeral talk into durable, actionable momentum.

If you are building conversational agents that need to remember cross-session context, check out the Hindsight GitHub repository and explore how dedicated episodic memory transforms what your systems can do.