DEV Community

Ankit Arora
Ankit Arora

Posted on

Scaleable Multifaceted File System Search Architecture

#architecture #distributed #filesystem #search

To achieve an efficient "fast file lookup search, then deep content graph indexing," the best approach is a layered architecture including knowledge graph construction:

  • Layer 1 (Instant Lookup): Uses native OS commands to find files by name/metadata in milliseconds.

  • Layer 2 (Content Extraction): Parses the found files to extract plain text.

  • Layer 3 (Knowledge Memory): Stores extracted text as vectors and graph relationships, with built-in TTL (Time-To-Live) for automatic expiration.

Complete technical blueprint and architecture design

1. Fast File Lookup (Using OS Native Commands)

To find files instantly without scanning the entire hard drive, you must tap into the OS's pre-built indexes. Do not use os.walk() for the initial lookup.

OS Command / SDK How it works
Windows Everything SDK / es.exe Accesses the USN Journal and MFT (Master File Table). Searches are near-instantaneous (<100ms for millions of files).
macOS mdfind Uses the Spotlight metadata index. Run mdfind -name "keyword" for filenames, or mdfind "kMDItemContentType == 'pdf'" for types.
Linux plocate / mlocate Uses a pre-built database updated by updatedb. plocate is extremely fast and uses minimal I/O.

Python Implementation Snippet:

import subprocess
import platform

def os_file_lookup(query, file_type=None):
    system = platform.system()
    if system == "Windows":
        # Assuming Everything is installed
        cmd = ["es.exe", "-n", "100", query] 
    elif system == "Darwin": # macOS
        cmd = ["mdfind", "-name", query]
    else: # Linux
        cmd = ["plocate", query]

    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.stdout.splitlines()
Enter fullscreen mode Exit fullscreen mode

2. Content Extraction (Parsing PDF, Word, Excel, etc.)

Once you have the file paths from Layer 1, you pass them to an extraction engine. For a unified interface, use these Python libraries:

  • all2txt or doc-master: Provides a single parse(file_path) function that handles .docx, .xlsx, .pptx, .pdf, .odt, .epub, and .pages. It automatically chooses the right parser behind the scenes.
  • fscan: A multi-threaded Python search tool. You can borrow its extractor logic for speed.
  • For more improved performance in C++ *: Use *libpoppler (PDF), libxml2 (DOCX/XLSX are just ZIPs of XML), and xlsxioread (Excel).

3. Knowledge Graph + Vector Memory (With TTL)

To store this as "memory" and remove it after disuse or max capacity, you need a database that supports Graph relationships, Vector embeddings, and TTL (Time-To-Live) indices.

Tool Why it fits your requirement
Grafeo (with LlamaIndex) An embedded graph database (single .db file). It natively supports Vector Search (for semantic context) and GQL (Graph Query Language). Perfect for a single installable software.
Cognee Specifically built as an "AI Memory" engine. It automates the extraction of entities and relationships from your extracted text, builds a graph, and automatically prunes (deletes) old data based on usage frequency.
SQLite + sqlite-vec + TTL A simpler DIY approach: Store chunks of text in SQLite, enable the sqlite-vec extension for similarity search, and add a created_at column with an index. Run a background scheduler every hour to DELETE FROM memory WHERE created_at < datetime('now', '-7 days').

Memory Management (Your "Remove after unused or max increased" requirement):

  • Use a LRU (Least Recently Used) cache pattern on the graph nodes.
  • Store a last_accessed timestamp for every entity. When the database exceeds your defined size limit (e.g., 500MB), run a cleanup job that deletes the oldest 20% of nodes.
  • Alternatively, assign a "Hotness" score. Files opened via the chat interface get a +1 score; if a node's score drops to 0 and 30 days have passed, it is purged.

4. Hybrid Search Workflow (Efficient "Lookup First" Search)

Your system should not search the heavy graph database for every keystroke. Use this flow:

  1. User types a query in the chat interface.
  2. Fast Path: Fire the OS command (mdfind/es.exe) to get filenames matching the query. Display these immediately (under 200ms).
  3. Parallel Deep Path: Simultaneously, send the query to the Knowledge Graph (Vector + Graph search). This takes a bit longer (1-3 seconds).
  4. Aggregation: Show the user: "Found 15 files (OS lookup). Here are the top 3 knowledge contexts extracted from them (AI lookup)."

5. Suggested System Architecture Diagram 

flowchart TD
    UI[User Chat / Editor Interface] --> Router{Query Router};

    Router -->|"Filename/Path Query"| OS_Layer[OS Lookup Layer];
    OS_Layer --> Win[Windows: Everything SDK];
    OS_Layer --> Mac[macOS: mdfind];
    OS_Layer --> Lin[Linux: plocate];
    OS_Layer -->|"Paths found"| Extractor[Content Extractor];

    Router -->|"Semantic / Context Query"| Graph_DB[Knowledge Graph Memory];
    Graph_DB --> Vec[Vector Search for Context];
    Graph_DB --> Rel[Graph Traversal for Relationships];

    Extractor --> Chunker[Text Chunker + Embedder];
    Chunker --> Graph_DB;

    subgraph Memory_Management[Automated Memory Pruning]
        TTL[TTL Scheduler] -->|"Delete old/unused nodes"| Graph_DB;
        Size_Monitor[Size Monitor] -->|"If > 500MB, purge oldest"| Graph_DB;
    end

    OS_Layer --> Aggregator[Result Aggregator];
    Graph_DB --> Aggregator;
    Aggregator --> UI;
Enter fullscreen mode Exit fullscreen mode

6. Roadmap for the Installable Text Editor + Chat Solution

Since you want to eventually build this into a downloadable app in (C++ or Python), here is the recommended tech stack:

  • Backend Core (Python): Use FastAPI or Starlette to wrap your search and graph logic into a local REST API or WebSocket server.

  • Frontend UI (Cross-platform): Use Tauri (Rust + HTML/JS) or Electron for the text editor with a chat sidebar. Tauri is lighter, but Electron is easier to integrate Python via child_process.

  • Packaging: Use PyInstaller (for Python) or CPack (for C++) to bundle the Everything SDK (Windows DLLs) and the database into a single .exe / .dmg / .deb.

Why choose this approach?

  • Speed: OS commands give you file results in < 100ms, which feels "instant" to users.

  • Intelligence: The Graph DB provides "contextual memory" (e.g., finding an Excel number mentioned in a PDF).

  • Self-maintaining: The TTL and size monitors ensure the software won't bloat your user's hard drive, automatically removing memory that is no longer relevant.

Top comments (0)