DEV Community

Cover image for Whispers - A Voice Journaling App with Smart Memory Search (Algolia MCP)
VaishakhVipin
VaishakhVipin

Posted on

Whispers - A Voice Journaling App with Smart Memory Search (Algolia MCP)

#devchallenge #algoliachallenge #webdev #ai

Algolia MCP Server Challenge: Ultimate user Experience

Algolia MCP Server Challenge Submission

Whispers - A Contextual Voice Memory System

What I Built

Whispers is a voice-first journaling application that transforms spoken thoughts into searchable, contextual memories. Users speak naturally into their microphone, and the system captures, processes, and indexes their reflections with semantic understanding. The core innovation is using Algolia MCP Server to power intelligent search that goes beyond keyword matching—it understands context, emotional states, and temporal patterns in your personal narrative.

This isn't just a search engine for text. It's a second brain that remembers not just what you said, but when you said it, how you felt, and what patterns emerge across your thoughts over time.

Demo

🎥 Video Demo:
https://drive.google.com/file/d/1RHyqpW434EeTGdP6xMRYbZCfifNatZd7/view?usp=sharing

GitHub Repository

The complete source code is available at: (https://github.com/VaishakhVipin/whispers-final)

Key files demonstrating Algolia MCP integration:

  • backend/services/gemini.py - MCP search orchestration and query decomposition
  • backend/routes/stream.py - Algolia indexing and filtered search endpoints
  • frontend/src/components/SearchInterface.tsx - Natural language search interface
  • backend/services/algolia.py - Algolia MCP client implementation

How I Utilized the Algolia MCP Server

The Algolia MCP Server is the backbone of Whispers' contextual memory system. Here's how it transforms natural language queries into intelligent, filtered search results:

1. Structured Data Indexing with Rich Metadata

Each journal entry is indexed with comprehensive metadata that enables sophisticated filtering:

entry = {
    "user_id": user.id,           # User isolation
    "session_id": session_id,     # Session grouping
    "date": date,                 # Temporal filtering
    "timestamp": timestamp,       # Precise timing
    "title": title,               # Semantic search
    "summary": summary,           # Contextual understanding
    "tags": tags,                 # Emotional/topic classification
    "text": text,                 # Full content search
    "is_from_prompt": is_from_prompt  # Prompt-driven vs free-form
}
Enter fullscreen mode Exit fullscreen mode

2. Gemini-Powered Query Decomposition

When users ask questions like "When was I stuck?" or "What were my creative ideas last month?", Gemini breaks these into searchable components:

def mcp_search(query, user_id=None):
    """
    Multi-step MCP agent architecture for contextual journal search:

    Stage 1: Intent Extraction - Gemini analyzes query and extracts search terms
    Stage 2: Memory Retrieval - Check local memory for similar past queries
    Stage 3: Search Execution - Query Algolia with extracted terms and user filters
    Stage 4: Synthesis - Feed results back to Gemini for contextual insights
    Stage 5: Memory Storage - Store query and results for future reference
    """
    import json as pyjson
    import re
    import os
    from datetime import datetime

    # Initialize response structure
    response = {
        "original_query": query,
        "search_terms": [],
        "stage1_response": "",
        "algolia_hits": [],
        "final_summary": "",
        "memory_used": False,
        "timestamp": datetime.now().isoformat()
    }

    # ===== STAGE 1: Intent Extraction =====
    print("🔍 Stage 1: Extracting intent and search terms...")
    try:
        url = "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent"
        headers = {"Content-Type": "application/json"}

        extraction_prompt = (
            "You are an AI assistant for a journaling app. "
            "Analyze the user's query and extract intent and search terms. "
            "Return a JSON object with: "
            "1. 'is_search': 'yes' if this requires searching past entries, 'no' otherwise "
            "2. 'search_terms': array of specific, relevant search terms "
            "3. 'intent': brief description of what the user is looking for "
            "4. 'response': a helpful, natural response about what you'll search for "
            "Example: {\"is_search\": \"yes\", \"search_terms\": [\"productivity\", \"morning\"], \"intent\": \"finding productivity patterns\", \"response\": \"I'll search for entries about your productivity and morning routines.\"} "
            f"User query: {query}"
        )

        payload = {
            "contents": [{"parts": [{"text": extraction_prompt}]}],
            "generationConfig": {"maxOutputTokens": 512}
        }
        params = {"key": GEMINI_API_KEY}

        resp = requests.post(url, headers=headers, params=params, json=payload, timeout=15)
        resp.raise_for_status()
        data = resp.json()

        response_text = data.get("candidates", [{}])[0].get("content", {}).get("parts", [{}])[0].get("text", "")
Enter fullscreen mode Exit fullscreen mode

3. Contextual Relevance Scoring

Results are ranked by semantic relevance, not just keyword frequency:

def calculate_relevance(hit):
    relevance_score = 0
    for term in search_terms:
        term_lower = term.lower()
        if term_lower in hit.get("title", "").lower():
            relevance_score += 3  # Title matches are most important
        if term_lower in hit.get("summary", "").lower():
            relevance_score += 2  # Summary matches are important
        if any(term_lower in tag.lower() for tag in hit.get("tags", [])):
            relevance_score += 1  # Tag matches are good
    return relevance_score
Enter fullscreen mode Exit fullscreen mode

4. Real-World Search Examples

Query: "When did I feel burnt out?"

  • Gemini Decomposition: ["burnt", "out", "burnout", "exhausted"]
  • Algolia Filter: user_id:123 AND (burnt OR out OR burnout OR exhausted)
  • Result: Entries tagged with "burnout", "stress", or containing emotional context

Query: "What were my app ideas last month?"

  • Gemini Decomposition: ["app", "ideas", "startup", "project"]
  • Algolia Filter: user_id:123 AND date:2024-06* AND (app OR ideas OR startup OR project)
  • Result: Creative entries from June with relevant tags

Key Technical Achievements

Contextual Memory Recall

  • Semantic Understanding: Queries like "when I was struggling" find entries with emotional context, not just the word "struggling"
  • Temporal Intelligence: "Last week" automatically filters to recent entries
  • Pattern Recognition: Identifies recurring themes across multiple entries

Privacy-First Architecture

  • User Isolation: Every search is filtered by user_id ensuring complete data separation
  • Secure Indexing: No cross-user data leakage in the Algolia index
  • Audit Trail: All search queries are logged for transparency

Performance Optimization

  • Sub-200ms Search: Algolia's distributed search infrastructure delivers instant results
  • Smart Caching: Frequently accessed patterns are cached for faster retrieval
  • Efficient Filtering: User-specific filters reduce search space and improve performance

Key Takeaways

  1. MCP Enables Contextual Search: Traditional search engines match keywords. MCP with Gemini enables understanding of intent, emotion, and temporal context.

  2. Structured Data Powers Intelligence: Rich metadata (tags, dates, user context) transforms simple text search into intelligent memory recall.

  3. User Isolation is Critical: Multi-tenant applications require careful filter design to prevent data leakage while maintaining search performance.

  4. Natural Language Queries Need Decomposition: Complex questions require breaking down into searchable components while preserving semantic meaning.

  5. Relevance Scoring Matters: Beyond simple keyword matching, contextual relevance scoring ensures users find the most meaningful memories.

Technical Stack

Voice Processing:

  • AssemblyAI Universal Streaming for real-time transcription
  • WebSocket for low-latency audio streaming

AI & Search:

  • Google Gemini for query decomposition and content analysis
  • Algolia MCP Server for contextual search and filtering
  • FastAPI for backend orchestration

Data Architecture:

  • Supabase for user authentication and session management
  • Algolia for search indexing with rich metadata
  • React + TypeScript for responsive frontend

Deployment:

  • Vercel for frontend hosting
  • Vercel Functions for serverless backend
  • Environment-based security configuration

What's Next

Immediate Roadmap:

  • Implement semantic similarity search for finding related memories
  • Add emotional trend analysis across time periods
  • Create memory timelines with contextual insights

Future Enhancements:

  • Voice emotion detection for enhanced emotional context
  • Collaborative memory sharing with privacy controls
  • Integration with calendar and productivity apps
  • Advanced pattern recognition for personal growth insights

Final Note

Whispers demonstrates how Algolia MCP Server can transform simple text search into contextual memory recall. By combining structured data indexing, intelligent query decomposition, and semantic relevance scoring, it creates a second brain that understands not just what you said, but the context, emotion, and patterns in your thoughts over time.

The project showcases how MCP technology enables applications that feel like they understand you—not just search your data, but help you rediscover and reflect on your own thoughts and growth journey.

Top comments (0)