Meeting Prep Agent

Why I Replaced Vector Search with Hindsight for Meeting Notes

Why I Stopped Relying on Vector Search for Meeting Prep Context
Preparing for client calls used to mean opening six tabs, scanning thousands of words of messy CRM notes, and praying I didn't miss a promise made three weeks ago. Most meeting prep tools attempt to solve this using standard Retrieval-Augmented Generation (RAG) over recent notes, but basic semantic similarity breaks down when you need temporal memory, cross-session continuity, and explicit constraint tracking.
To fix this, I built an automated Meeting Prep Agent using a persistent memory architecture powered by Hindsight's agent memory engine. Instead of throwing raw embeddings at a vector database and hoping for the best, the system ingests past interactions, extracts structural constraints, and generates time-blocked, executive-ready briefing dossiers in under two seconds.
Here is how the system hangs together, why traditional vector search failed, and how persistent long-term memory changed the architecture.

1. System Architecture: How It Hangs Together The Meeting Prep Agent operates as a single-turn orchestration pipeline that converts unstructured user inputs and persistent historical logs into a structured briefing dossier. +-----------------------------------------------------------------------+ | USER INTERFACE | | - Inputs: Meeting Title, Person/Company, Date, Purpose, Quick Notes | | - Quick Presets: Sales Pitch, 1-on-1 Review, Client Kickoff, etc. | +-----------------------------------------------------------------------+ | v +-----------------------------------------------------------------------+ | MEETING PREP AGENT ENGINE | | | | +-----------------------------------------------------------------+ | | | 1. Context Ingestion Layer | | | | - Extracts raw notes & meeting metadata | | | | - Queries Hindsight Memory Engine for historical context | | | +-----------------------------------------------------------------+ | | | | | v | | +-----------------------------------------------------------------+ | | | 2. Synthesis & Memory Resolution | | | | - Correlates historical promises against current objectives | | | | - Evaluates constraint shifts, past feedback, and commitments | | | +-----------------------------------------------------------------+ | | | | | v | | +-----------------------------------------------------------------+ | | | 3. Structural Dossier Formatter | | | | - Timed Agendas & Discussion Blocks | | | | - Discovery Questions & Strategic Talking Points | | | | - Risk Mitigation & Objection Prep | | | | - Post-Meeting Action Items & 3-Minute Readiness Checklist | | | +-----------------------------------------------------------------+ | +-----------------------------------------------------------------------+ | v +-----------------------------------------------------------------------+ | EXECUTIVE BRIEFING DOSSIER | +----------------------------------------------------------------------- The system takes four core inputs from the user: Meeting Title & Persona: e.g., Q4 Executive Budget Review with Apex Logistics (CTO Sarah Jenkins). Meeting Date & Purpose: The high-level objective, such as presenting a migration roadmap or resolving delivery friction. Previous Meeting Notes: Free-text past notes or action items. Preset Profile: Pre-configured strategy weights for Sales Pitches, 1-on-1s, Client Kickoffs, or Investor Reviews. Once triggered, the engine passes the user's intent alongside historical entity data into the memory pipeline, builds a structured JSON payload, and renders a complete dossier containing timed agendas, high-impact discovery questions, predicted objections with mitigations, and post-meeting execution items.
2. Deep Dive: Why Vector Search Failed and How Memory Fixed It When I built the first prototype, I used a standard naive RAG approach: chunk previous meeting notes, store them in a vector store, and fetch the top k most similar chunks when a new meeting was scheduled. It failed almost immediately in three edge cases: Temporal Drift: A vector search for "budget limits" returned notes from six months ago rather than the updated, scaled-up budget discussed last week. Implicit Commitments: Searching for "action items" missed sentences like "Let's revisit pricing once the dev team finishes the pilot." Context Overwrite: When a client reversed a previous decision, similarity search frequently pulled both the old decision and the new decision into the context window, causing the LLM to hallucinate contradictions. To resolve this, I replaced raw similarity search with persistent memory management using Vectorize's agent memory concepts via the Hindsight documentation. Hindsight allows the agent to treat past interactions not as isolated chunks, but as an evolving graph of stakeholder entity facts, temporal updates, and explicit promises. When synthesizing a brief, the agent queries Hindsight to understand what changed since the last call rather than just what sounds similar.
3. Code-Backed Implementation Here is how the pipeline is constructed in code.
4. Ingesting Meeting Inputs and Selecting Presets The entry point captures raw input and applies preset parameters depending on whether the call is a sales pitch, client kickoff, or investor review. export interface MeetingInputs { title: string; companyName: string; dateTime: string; purpose: string; previousNotes?: string; preset: 'sales' | 'one-on-one' | 'kickoff' | 'investor'; }

export function applyPreset(preset: MeetingInputs['preset']): Partial {
const presets = {
sales: {
purpose: "Present solution roadmap, address implementation concerns, and agree on pilot timeline.",
},
kickoff: {
purpose: "Strengthen partnership, review recent milestones, address friction, and align on upcoming goals.",
},
investor: {
purpose: "Review unit economics, demonstrate product velocity, and align on next funding milestones.",
}
};
return presets[preset] || {};
}

1. Querying Hindsight Memory Engine Before generating the dossier, the backend queries Hindsight to fetch historical context, past commitments, and entity-specific constraints related to the client. import { HindsightClient } from '@vectorize-io/hindsight';

const hindsight = new HindsightClient({
apiKey: process.env.HINDSIGHT_API_KEY,
});

export async function fetchEntityMemory(companyName: string, meetingPurpose: string) {
// Retrieve continuous contextual facts rather than simple keyword matches
const memoryContext = await hindsight.retrieveMemory({
entityId: companyName,
query: meetingPurpose,
includeUnresolvedPromises: true,
temporalOrdering: 'descending',
});

return {
historySummary: memoryContext.summary,
openCommitments: memoryContext.unresolvedPromises,
pastFrictionPoints: memoryContext.historicalRisks,
};
}

1. Synthesizing the Structured Dossier Payload The engine formats the retrieved memory along with current inputs to prompt the model for a strictly structured response. export async function generateDossier(inputs: MeetingInputs) { const memory = await fetchEntityMemory(inputs.companyName, inputs.purpose);

const prompt = `
You are an executive preparation engine. Generate a structured brief using this context:

Target: ${inputs.companyName}
Title: ${inputs.title}
Purpose: ${inputs.purpose}
Raw User Notes: ${inputs.previousNotes || 'None'}

Hindsight Memory Facts:
- Background: ${memory.historySummary}
- Open Promises: ${JSON.stringify(memory.openCommitments)}
- Past Friction: ${JSON.stringify(memory.pastFrictionPoints)}

Return a JSON payload with: objectives, companySummary, timedAgenda, questions, talkingPoints, objections, and postMeetingActionPlan.

`;

const response = await callLLM({
prompt,
responseFormat: 'json_object',
temperature: 0.2,
});

return JSON.parse(response.content);
}

1. Rendering the Timed Agenda Component The UI renders the structured output directly into a clean, actionable prep sheet. export function AgendaSection({ agendaItems }: { agendaItems: AgendaItem[] }) { return (

   Key Discussion Points

   {agendaItems.map((item, index) => ( {index + 1}. {item.title} {item.durationMins} mins

   {item.description}

   ))} ); }
2. Real-World Behavior & Example Interaction To see the system in action, consider a scenario where we are preparing for a follow-up call with ABC Technologies. Inputs Company: ABC Technologies Title: Project Discussion Meeting Purpose: Strengthen partnership, review recent milestones, address operational friction, and align on upcoming goals. Raw Notes: "Client mentioned interest in feature expansion but expressed worries about development time and budget adjustments during our last check-in." Generated Output Instead of returning a wall of text, the system generates a partitioned, executive-ready view: Meeting Objective: Strengthen partnership with ABC Technologies, review milestones, address friction, and align on upcoming goals. Success Criteria: Client feels heard regarding momentum; open deliverables have clear owners; expansion opportunities identified. Person / Company Summary: ABC Technologies wants a simple solution completed within a reasonable timeframe. Hindsight notes flag previous questions regarding development cost and timelines. Timed Agenda: 00:00 - 00:05 Welcome & Agenda Alignment 00:05 - 00:15 Review Past Action Items & Updates (addressing open notes explicitly) 00:15 - 00:35 Core Topic: Software Project Requirements & Deliverables 00:35 - 00:45 Strategic Context & Specific Constraints 00:45 - 00:50 Wrap-Up & Immediate Next Steps Objection Handling: Concern: Delays or misunderstandings on previous deliverables. Mitigation: Own the friction transparently, provide an immediate remediation timeline, and install updated check-in gates. Concern: Scope creep or pricing adjustment resistance. Mitigation: Show clear trade-offs between speed, scope, and quality. Provide modular options so the client stays in control.
3. Lessons Learned Memory beats context window size: Blindly dumping thousands of tokens of chat history into the prompt introduces noise. Extracting specific facts and unresolved commitments via persistent memory engines like Hindsight yields significantly cleaner LLM outputs. Deterministic structures enforce focus: LLMs tend to drift into conversational fluff when asked to "prepare a meeting notes summary." Forcing the output into explicit categories (Objectives, Timed Agendas, Objections, Next Steps) makes the agent immediately useful in real-life workflows. Pre-call checklists reduce operational panic: Technical prep is only half the battle. Adding a simple 3-minute readiness checklist (testing mic, opening browser tabs, reviewing recent updates) prevents avoidable last-minute stumbles. Time-boxing agendas improves call execution: Forcing the LLM to allocate fixed minute blocks to each agenda topic prevents meetings from spiraling into unstructured discussions that run over time.